Deep learning model training method

Abstract

The invention discloses a deep learning model training method, and relates to the field of data processing. The training method comprises a preparation stage and a training stage. In the preparation stage, aiming at vector and / or matrix operation which cannot be realized by a built-in function of a standard SQL (Structured Query Language) in different basic component calculation formulas of the deep learning model, a corresponding user-defined function is introduced and packaged into a database plug-in to be integrated into a database management system, and meanwhile, a deep learning model table is constructed; in the training stage, according to a training request input by a user, a directed acyclic graph of a basic component of a deep learning model in the training request is obtained, an SQL statement used for model training is generated, the SQL statement is executed in a database management system to obtain a final parameter assignment of the deep learning model, and the final parameter assignment is stored in a deep learning model table constructed in the preparation stage. According to the method, a database temporary table is used for achieving super memory training, a vector field storage parameter set is introduced, efficient training of a large-scale deep learning model is supported, and temporary data are automatically cleaned.

Description

Technical Field

[0001] This application relates to the field of data processing, and particularly to a method for training a deep learning model. Background Art

[0002] With the advent of the big data era, data analysis and machine learning technologies have been increasingly widely applied in various industries. As the core of data storage and management, the database system needs to be closely integrated with data analysis and machine learning technologies to support data-driven decision-making. In other words, modern database systems need to integrate machine learning algorithms into the database to achieve the integration of data storage, management, and analysis, thereby enhancing data analysis capabilities and simplifying the machine learning process. The traditional data analysis process usually includes steps such as data extraction, transformation, loading (ETL), and model training. In the traditional data analysis mode, these steps often require exporting data from the database and then analyzing it with the help of external tools or platforms. This process is not only time-consuming and laborious but also may reduce the analysis accuracy due to problems such as format conversion and loss during data migration. Integrating deep learning algorithms directly into the database, namely so-called "in-database deep learning", can change this situation. It allows users to directly execute analysis tasks at the location where the data is stored without cumbersome data migration steps, thus significantly improving the efficiency and real-time performance of data analysis. In addition, in-database deep learning can also utilize the optimization technology and parallel processing capabilities of the database itself to further accelerate the analysis process and make the processing of large-scale data sets easier and faster.

[0003] Currently, there are mainly two solutions to achieve in-database deep learning. One solution is to introduce user-defined functions (UDFs), abbreviated as the UDF solution. User-defined functions (UDFs) are an advanced feature provided by database management systems that give users great flexibility and allow them to create and execute custom logic code in the database environment. These custom functions can not only perform basic mathematical operations, string processing, and data transformation but also serve as a bridge to seamlessly connect the database with external high-performance computing resources and machine learning frameworks. Through UDFs, deep learning algorithms in popular deep learning frameworks such as PyTorch, TensorFlow, PaddlePaddle, and MindSpore can be integrated and called in the database. Another solution is to directly translate the training and application processes of deep learning algorithms into SQL statements, abbreviated as the translation solution. This solution means that users can directly execute deep learning tasks in the database without writing complex code or calling external tools. Its core idea is to convert the mathematical formulas and logic of deep learning algorithms into SQL statements.

[0004] So far, there are still some unsolved problems in both of these two solutions. On the one hand, a large amount of data required for training the deep learning model needs to be collected within the database query process and then submitted to the deep learning algorithm running in the same process for processing. It is easy for the system to run slowly or even crash due to insufficient physical memory to accommodate the training set. On the other hand, converting the mathematical formulas and logic of the deep learning algorithm into SQL statements requires the design of a procedural translation process to avoid introducing errors during the translation process. Summary of the Invention

[0005] This application provides a training method for a deep learning model, which is applicable to efficiently training different deep learning models in a database.

[0006] In a first aspect, this application provides a training method for a deep learning model, including: A preparation stage and a training stage; In the preparation stage, for vector and / or matrix operations in the calculation formulas of different basic components of the deep learning model that cannot be implemented by the built-in functions of standard SQL, corresponding user-defined functions for the vector and / or matrix operations are introduced and encapsulated into a database plug-in and integrated into the database management system; in addition, in the preparation stage, a deep learning model table is constructed to store the training request ID input by the user and the corresponding deep learning model parameters obtained in the training stage; In the training stage, according to the training request input by the user, the parameters of the corresponding deep learning model are generated and saved. The training request includes a training request ID, the network structure of the deep learning model, training hyperparameter assignments, and a training set; The training stage consists of a conversion link and an execution link; In the conversion link, the basic components of the deep learning model input by the user are determined, a directed acyclic graph of the basic components is obtained, and an SQL statement is output; each basic component has a pre-agreed operation process and corresponds to a subset of the parameters of the deep learning model input by the user. The operation process of the basic component is a calculation formula, and the calculation formula is constructed according to the built-in functions of standard SQL and the user-defined functions introduced in the preparation stage; the training hyperparameters input by the user include the number of iterations, the learning rate, etc.; The SQL statement output by the conversion link is defined step by step as follows: First, define the construction process of the training set temporary table; Second, define the construction process of the training hyperparameter assignment table; Third, define the construction process of the temporary tables involved in each iteration. The temporary tables involved in each iteration include: a temporary table composed of the assignment records of the deep learning model parameters obtained in each iteration, a temporary table of the basic component outputs obtained sequentially according to the forward traversal order of the directed acyclic graph of the basic components, a temporary table of the basic component gradients obtained sequentially according to the backward traversal order of the directed acyclic graph of the basic components, and a temporary table of the accumulated basic component gradients obtained by summarizing all the records in the training set; Fourth, define the construction process of the final parameter assignment table of the deep learning model input by the user; Finally, define the process of storing the final parameter assignment table of the deep learning model input by the user into the database; The execution link executes the SQL statements output by the conversion link in the database management system to obtain the final parameter assignment table of the deep learning model input by the user, and saves the final parameter assignment table into the deep learning model table constructed in the preparation stage.

[0007] In the above embodiments of the present application, the training data and the training parameters required by the model are converted into vector-type data and stored in the database form. A group of training parameters can be converted into a vector, that is, a vector can represent multiple parameters, so that the form can store a large number of parameters, thereby reducing the pressure of the training data on the physical memory, facilitating the training process to run through the database query process, and further supporting the use of training data exceeding the physical memory capacity in the database for deep learning model training, while avoiding data migration and improving the training efficiency of the deep learning model. In addition, by adopting the above technical solution of the present application, users can directly execute deep learning tasks in the database without writing complex codes or calling external tools, improving the development efficiency.

[0008] Based on the above solution, the obtaining of the directed acyclic graph of the basic components includes: Construct a directed acyclic graph with each basic component of the deep learning model input by the user as a node. There is an arc from node A to node B if and only if an output object of the basic component corresponding to node A is an input object of the basic component corresponding to node B.

[0009] In the above solution, the definition of the construction process of the training set temporary table includes: The training set input by the user is a set of training records. Each training record is composed of the assignments of multiple independent variables and the assignment of a single dependent variable. The assignments of all independent variables in each training record are combined into a vector, and together with the corresponding dependent variable assignment, they form a binary tuple, which is saved as a record in the training set temporary table in the training set temporary table.

[0010] In the above solution, the construction process of the defined training hyperparameter assignment table includes: Collect all the training hyperparameter assignments input by the user to form the only record of the training hyperparameter assignment table.

[0011] In the above solution, the construction process of the defined temporary table involved in each round of iteration includes: Divide all the parameters of each basic component in the network structure of the deep learning model input by the user into multiple parameter groups, and each parameter group corresponds to a vector; define a deep learning model parameter assignment table, where each field of the deep learning model parameter assignment table corresponds to a parameter group, and the field value is called the assignment vector of the corresponding parameter group. In addition, each record of the deep learning model parameter assignment table is called an assignment record, and the assignment record is composed of the iteration round number and the assignment vectors of all parameter groups obtained after this round of iteration, and is used to store the deep learning model parameters obtained in each round of iteration; define the following iterative operations, and the number of rounds of the iterative operations is equal to the number of iteration rounds input by the user: If this is the first round, add an assignment record to the deep learning model parameter assignment table, store the randomly initialized assignment vectors of all basic component parameter groups, and mark the assignment record as the most recent parameter assignment record. Otherwise, directly mark the assignment record of the deep learning model parameters in the previous round as the most recent parameter assignment record; Determine the forward traversal order of the directed acyclic graph of the basic components, ensuring that if there is an arc from the corresponding node of basic component A to the corresponding node of basic component B, then A is visited before B; define the output temporary tables of each basic component in sequence according to the forward traversal order, and each field of the basic component output temporary table corresponds to an output object; Determine the backward traversal order of the directed acyclic graph of the basic components, ensuring that if there is an arc from the corresponding node of basic component A to the corresponding node of basic component B, then B is visited before A; define the gradient temporary tables of each basic component in sequence according to the backward traversal order, and each field of the basic component gradient temporary table corresponds to a parameter group or an input object, and the value of each field is called the gradient vector of the corresponding parameter group or the corresponding input object; According to the basic component gradient temporary table generated last according to the backward traversal order, define a gradient accumulation temporary table to store the sum aggregation value of the parameter group gradient vectors of all basic components in the deep learning model input by the user. The sum aggregation value of the parameter group gradient vectors is the gradient accumulation vector obtained by performing a summation operation on the corresponding parameter group gradient vectors of all training records; According to the parameter update formula in the standard gradient descent method, that is, the assignment vector of the current round of parameter group is equal to the difference between the assignment vector of the previous round of parameter group and the gradient adjustment vector of the current round of parameter group, define the assignment record of the parameters of the deep learning model in the current round. The gradient adjustment vector of the current round of parameter group is the product of the learning rate input by the user and the gradient vector of the current round of parameter group.

[0012] In the above solution, the process of constructing the final parameter assignment table of the deep learning model input by the user includes: Take the last record of the deep learning model parameter assignment table as the only record of the final parameter assignment table.

[0013] In the above solution, the process of storing the final parameter assignment table of the deep learning model input by the user includes: Convert the only record of the final parameter assignment table of the deep learning model input by the user into an array of parameter group assignment vectors, and together with the training request ID input by the user, add it as a record to the deep learning model table constructed in the preparation stage.

[0014] In the above solution, it is characterized in that the process of sequentially defining the output temporary tables of each basic component according to the forward traversal order includes: The output temporary table of the first basic component stores the output vector of the first basic component traversed forward, as well as the training set and historical output vectors required in the subsequent traversal process. The stored content is defined based on the training set temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table; the output temporary table of each subsequent basic component stores the output vector of the current basic component, as well as the training set and historical output vectors required in the subsequent traversal process. The stored content is defined based on the output temporary table of the previous basic component, the most recent parameter assignment record, and the training hyperparameter assignment table.

[0015] In the above solution, it is characterized in that the process of sequentially defining the gradient temporary tables of each basic component according to the backward traversal order includes: The gradient temporary table of the first basic component stores the gradient vector of the first basic component traversed backward, as well as the training set, historical output vectors, and historical gradient vectors required in the subsequent traversal process. The stored content is defined based on the output temporary table generated at the end of the forward traversal, the most recent parameter assignment record, and the training hyperparameter assignment table; the gradient temporary table of each subsequent basic component stores the gradient vector of the current basic component, as well as the training set, historical output vectors, and historical gradient vectors required in the subsequent traversal process. The stored content is defined based on the gradient temporary table of the previous basic component, the most recent parameter assignment record, and the training hyperparameter assignment table.

[0016] Second aspect, the present application provides an electronic device, which includes a memory and one or more processors. Among them, one or more computer programs are stored in the memory, and the computer programs include instructions. When the instructions are executed by the processor, the electronic device can be made to execute the training method of the deep learning model in the first aspect.

[0017] Third aspect, the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions run on an electronic device, the electronic device is made to execute the training method of the deep learning model in the first aspect.

[0018] Fourth aspect, the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device is made to execute the training method of the deep learning model as described in the first aspect.

[0019] It can be understood that the beneficial effects that can be achieved by the above-provided training device, electronic device, computer-readable storage medium, and computer program product of the deep learning model can refer to the beneficial effects in the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic flowchart of the deep learning model training method provided by the embodiment of the present application; Figure 2 It is a schematic diagram of the network structure of the deep learning model input in a specific case of the deep learning model training method provided by the embodiment of the present application; Figure 3 It is a schematic diagram of the target SQL statement output in a specific case of the deep learning model training method provided by the embodiment of the present application; Figure 4 It is a schematic diagram of the structure of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] To facilitate a clear description of the technical solutions of the embodiments of this application, in the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with basically the same functions and roles. For example, the first chip and the second chip are only used to distinguish different chips, and no limitation is imposed on their sequence. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and the terms "first" and "second" do not necessarily limit being different. It should be noted that in the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design solution described as "exemplarily" or "for example" in this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "exemplarily" or "for example" aims to present relevant concepts in a specific manner. In the embodiments of this application, "at least one" means one or more, and "a plurality of" means two or more than two.

[0022] It should be noted that "when... " in the embodiments of this application can be at the instant when a certain situation occurs, or can be within a period of time after a certain situation occurs. The embodiments of this application do not make specific limitations on this.

[0023] The following will describe the implementation manners of this embodiment in detail with reference to the accompanying drawings.

[0024] This embodiment provides a training method for a deep learning model. By using database query statements to train the deep learning model, there is no need to perform data migration between different systems, which can improve the training efficiency of the deep learning model.

[0025] As an important branch of machine learning, deep learning has become a landmark technology in today's artificial intelligence with its excellent characteristics of automatic feature extraction and high-precision prediction. The operation of deep learning algorithms mainly includes two stages: one is the training stage (also known as the learning stage), in which a deep learning model is learned through a training set; the other is the application stage (also known as the prediction stage), in which new data is predicted through the trained model. Deep learning models generally adopt the representation of neural networks. A neural network consists of multiple layers, and each layer contains several components. A neural network can be simply regarded as a composite function, and the components that make up the neural network are equivalent to the basic functions that make up the composite function. The purpose of training a deep learning model is to adjust the parameters of the neural network so that the prediction results of the model on the training set are as close as possible to the true values, thereby improving the generalization ability of the model.

[0026] It can be understood that the method in this embodiment is not only applicable to the training stage of deep learning models, but also applicable to the application stage of deep learning models. Both the training stage and the application stage are executed through database statements, and there is no need to perform data migration.

[0027] Next, the specific process of the deep learning model training method provided in this embodiment will be introduced. The training method of this deep learning model can be applied to various electronic devices equipped with a database, such as a computer (PC), a tablet computer, a virtual reality / augmented reality device, a wearable device, an industrial computer, a vehicle-mounted computer, etc.; it can also be applied to a server, the cloud, a server cluster, etc. that support database services. This embodiment does not make special limitations on this.

[0028] Figure 1 The flowchart of the deep learning model training method provided in the embodiment of the present application is shown.

[0029] As Figure 1 shown, the training method of this deep learning model includes a preparation stage and a training stage; in the preparation stage, for vector and / or matrix operations in the calculation formulas of different basic components of the deep learning model that cannot be implemented by the built-in functions of standard SQL, the corresponding user-defined functions for the vector and / or matrix operations are introduced and encapsulated into a database plugin and integrated into the database management system; in addition, in the preparation stage, a deep learning model table is constructed to store the training request ID input by the user and the corresponding deep learning model parameters obtained in the training stage; in the training stage, according to the training request input by the user, the corresponding deep learning model parameters are generated and saved, and the training request includes a training request ID, the network structure of the deep learning model, training hyperparameter assignments, and a training set.

[0030] The training stage consists of a conversion link and an execution link; in the conversion link, the basic components of the deep learning model input by the user are determined, a directed acyclic graph of the basic components is obtained, and an SQL statement is output; each basic component has a pre-agreed operation process and corresponds to a subset of parameters of the deep learning model input by the user, and the operation process of the basic component is a calculation formula, and the calculation formula is constructed according to the built-in functions of standard SQL and the user-defined functions introduced in the preparation stage; the training hyperparameters input by the user include the number of iterations, the learning rate, etc.

[0031] The SQL statement output by the conversion link is defined step by step as follows: First, define the construction process of the training set temporary table; Second, define the construction process of the training hyperparameter assignment table; Third, define the construction process of the temporary tables involved in each iteration. The temporary tables involved in each iteration include: a temporary table composed of the assignment records of the deep learning model parameters obtained in each iteration, a temporary table of the basic component outputs obtained in sequence according to the forward traversal order of the directed acyclic graph of the basic components, a temporary table of the basic component gradients obtained in sequence according to the backward traversal order of the directed acyclic graph of the basic components, and a temporary table of the cumulative gradients of the basic components obtained by summarizing all the records in the training set; Fourth, define the construction process of the final parameter assignment table of the deep learning model input by the user; Finally, define the storage process of the final parameter assignment table of the deep learning model input by the user; The execution link executes the SQL statements output by the conversion link in the database management system, obtains the final parameter assignment table of the deep learning model input by the user, and saves the final parameter assignment table into the deep learning model table constructed in the preparation stage.

[0032] In the preparation stage, construct a database plug-in to provide the management function of vector type fields used in the training stage. In addition, construct a deep learning model table to store the training request ID and the corresponding deep learning model parameters obtained in the training stage. The database plug-in is implemented in C language and includes user-defined functions (UDFs) for vectors, matrices, and related operations on both. The database plug-in supports the management of vector type fields, where the matrix is represented as a vector in row-major order. Exemplarily, the database plug-in includes the following UDFs: Vector initialization: Generate a vector according to the vector length and initialization strategy. The initialization strategies include random initialization, all-zero initialization, and one-hot initialization, etc. Among them, one-hot initialization generates a one-hot vector with only one component being 1 and the rest being 0. When using the one-hot initialization strategy, the sequence number of the 1 component in the one-hot vector can be specified.

[0033] Four arithmetic operations between vectors: Add, subtract, multiply, or divide the corresponding components of two vectors with the same dimension, and return the result vector.

[0034] Four arithmetic operations between a vector and a scalar: Add, subtract, multiply, or divide each component in the vector by the scalar, and return the modified vector.

[0035] Euclidean distance between vectors: Calculate the Euclidean distance between two vectors with the same dimension, and return the result vector.

[0036] Dot product between vectors: Calculate the dot product of two vectors with the same dimension, and return the result scalar.

[0037] Cartesian product between vectors: Calculate the Cartesian product of two vectors and return the resulting matrix. The output matrix is represented as a vector with elements stored in row-major order.

[0038] Vector magnitude: Calculate the magnitude of a vector, which is the square root of the sum of the squares of all components, and return a scalar.

[0039] Vector normalization: Divide each component of the vector by the magnitude of the vector and return the normalized vector.

[0040] Vector transformation: Transform each component of the vector through activation functions such as sigmoid, tanh, ReLU, etc., and return the transformed vector.

[0041] Vector softmax: Use the softmax function to turn all components of the vector into real numbers between 0 and 1, satisfying the sum of all components being 1, and return the transformed vector.

[0042] Matrix transpose: Swap the rows and columns of the matrix and return the transposed matrix. Both the input matrix and the output matrix are represented as vectors with elements stored in row-major order.

[0043] Matrix-vector multiplication: Return the resulting vector of matrix-vector multiplication. Each component of the resulting vector is the dot product of a row vector of the input matrix and the input vector. The input matrix is represented as a vector with elements stored in row-major order.

[0044] Matrix-matrix multiplication: Return the resulting matrix of matrix-matrix multiplication. Each element of the resulting matrix is the dot product of a row vector of the first input matrix and a column vector of the second input matrix. The first input matrix, the second input matrix, and the output matrix are all represented as vectors with elements stored in row-major order.

[0045] Introduce a database plugin to store the parameters of the basic components of the deep learning model through vector-type fields, enabling a single form with a limited total number of fields in the database to store all the parameters of a large number of basic components. Through the UDFs of vectors, matrices, and their related operations, the mathematical formulas and logics of deep learning algorithms can be converted into UDFs, thereby simplifying the target SQL statement and improving the execution efficiency of the target SQL statement.

[0046] Obtaining the directed acyclic graph of the basic components specifically includes: constructing a directed acyclic graph with each basic component of the deep learning model input by the user as a node. There is an arc from node A to node B if and only if a certain output object of the basic component corresponding to node A is an input object of the basic component corresponding to node B.

[0047] Define the construction process of the training set temporary table, including: The training set input by the user is a set of training records, and each training record is composed of the assignments of multiple independent variables and the assignment of a single dependent variable. Combine the assignments of all independent variables in each training record into a vector, and together with the corresponding dependent variable assignment, form a binary tuple, which is saved as a record in the training set temporary table in the training set temporary table.

[0048] Define the construction process of the training hyperparameter assignment table, including: Collect all the training hyperparameter assignments input by the user to form the only record in the training hyperparameter assignment table.

[0049] Define the construction process of the temporary tables involved in each iteration, including: dividing all the parameters of each basic component in the network structure of the deep learning model input by the user into multiple parameter groups, with each parameter group corresponding to a vector; defining a deep learning model parameter assignment table, where each field of the deep learning model parameter assignment table corresponds to a parameter group, and the field value is called the assignment vector of the corresponding parameter group. In addition, each record of the deep learning model parameter assignment table is called an assignment record, and the assignment record consists of the iteration round number and the assignment vectors of all parameter groups obtained after this round of iteration, and is used to store the deep learning model parameters obtained in each iteration; define the following iterative operations, where the number of rounds of the iterative operation is equal to the number of iteration rounds input by the user: if this round is the first round, add an assignment record to the deep learning model parameter assignment table, store the randomly initialized assignment vectors of all basic component parameter groups, and mark the assignment record as the most recent parameter assignment record, otherwise directly mark the assignment record of the deep learning model parameters in the previous round as the most recent parameter assignment record; determine the forward traversal order of the directed acyclic graph of the basic components, ensuring that if there is an arc from the corresponding node of basic component A to the corresponding node of basic component B, then A is visited before B; define the output temporary tables of each basic component in sequence according to the forward traversal order, where each field of the basic component output temporary table corresponds to an output object; determine the backward traversal order of the directed acyclic graph of the basic components, ensuring that if there is an arc from the corresponding node of basic component A to the corresponding node of basic component B, then B is visited before A; define the gradient temporary tables of each basic component in sequence according to the backward traversal order, where each field of the basic component gradient temporary table corresponds to a parameter group or an input object, and the value of each field is called the gradient vector of the corresponding parameter group or the corresponding input object; based on the basic component gradient temporary table generated last according to the backward traversal order, define a gradient accumulation temporary table to store the sum aggregation value of the parameter group gradient vectors of all basic components in the deep learning model input by the user, and the sum aggregation value of the parameter group gradient vectors is the gradient accumulation vector obtained by performing a summation operation on the corresponding parameter group gradient vectors of all training records; according to the parameter update formula in the standard gradient descent method, that is, the assignment vector of the parameter group in this round is equal to the difference between the assignment vector of the parameter group in the previous round and the gradient adjustment vector of the parameter group in this round, define the assignment record of the deep learning model parameters in this round, and the gradient adjustment vector of the parameter group in this round is the product of the learning rate input by the user and the gradient vector of the parameter group in this round.

[0050] Define the construction process of the final parameter assignment table of the deep learning model input by the user, including: using the last record of the deep learning model parameter assignment table as the only record of the final parameter assignment table.

[0051] Define the storage process of the final parameter assignment table of the deep learning model with the user input, including: converting the unique record of the final parameter assignment table of the deep learning model with the user input into an array of parameter group assignment vectors, and adding it, together with the training request ID with the user input, as a record to the deep learning model table constructed in the preparation stage.

[0052] Define the output temporary tables of each basic component in turn according to the forward traversal order, including: the output temporary table of the first basic component stores the output vector of the first basic component traversed forward, as well as the training set and historical output vectors required in the subsequent traversal process, and the stored content is defined according to the training set temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table; the output temporary table of each subsequent basic component stores the output vector of the current basic component, as well as the training set and historical output vectors required in the subsequent traversal process, and the stored content is defined according to the output temporary table of the previous basic component, the most recent parameter assignment record, and the training hyperparameter assignment table.

[0053] Define the gradient temporary tables of each basic component in turn according to the backward traversal order, including: the gradient temporary table of the first basic component stores the gradient vector of the first basic component traversed backward, as well as the training set, historical output vectors, and historical gradient vectors required in the subsequent traversal process, and the stored content is defined according to the output temporary table generated at the end of the forward traversal, the most recent parameter assignment record, and the training hyperparameter assignment table; the gradient temporary table of each subsequent basic component stores the gradient vector of the current basic component, as well as the training set, historical output vectors, and historical gradient vectors required in the subsequent traversal process, and the stored content is defined according to the gradient temporary table of the previous basic component, the most recent parameter assignment record, and the training hyperparameter assignment table.

[0054] During the training phase, a training request is processed. The content of the training request includes a training request ID, a training set, the network structure of a deep learning model, and training hyperparameters. The training set is a specified form in the database, and all fields in the form have been preprocessed into numerical types. The deep learning model structure is a directed acyclic graph of the basic components of the deep learning model. In the directed acyclic graph, the nodes are basic components. If there is an arc from node A to node B, it means that a certain input of the basic component corresponding to B is a certain output of the basic component corresponding to A. The training hyperparameters include the learning rate, the number of iterations, the number of independent variables, etc. In deep learning algorithms, the parameters of the basic components of the deep learning model are trainable. The training process needs to perform forward numerical calculations and backward gradient propagation successively in multiple iterations to adjust these parameters so that the prediction results of the deep learning model on the training set are as close as possible to the true values of the training set. Training hyperparameters are parameters that need to remain unchanged during the training process. The forms of some deep learning model structures defined are not directed acyclic graphs of basic components. Therefore, when determining the content of the training request, the deep learning model structure needs to be represented as a directed acyclic graph of basic components. For example, the original structure of a recurrent deep learning model is the loop body of a single recurrent deep learning model component. Therefore, it is necessary to first expand the original structure of the recurrent deep learning model into a directed acyclic graph of recurrent deep learning model components at a finite number of time points, and then decompose the recurrent deep learning model components into directed acyclic subgraphs composed of basic components.

[0055] The following refers to Figure 2 , taking Figure 2 the training process of a deep learning model shown as an example, to illustrate how to use recursive common table expressions (CTE) to generate SQL statements for model training, and the SQL statements can complete the training and saving of the deep learning model when executed in the database.

[0056] Suppose the content of the training request is as follows: The training request ID is 1, the training set is the form aisql.iris, which contains 4 non-class label fields (i.e., independent variables) a1, a2, a3, a4 and 1 class label field (i.e., dependent variable) label. The deep learning model structure is a multi-layer feedforward neural network with one hidden layer shown in the following figure. The training hyperparameters are a learning rate of 0.001, the number of iterations of 1000, the number of independent variables of 4, and the number of neurons in the hidden layer of 10. The network structure consists of an input layer, a hidden layer, and an output layer. The hidden layer is a fully connected component with a sigmoid activation function, and the output layer is a fully connected component with a softmax activation function. According to the above training request content, the final generated SQL statement is as shown in Figure 3 .

[0057] Figure 3 The specific generation process of the SQL statement in Step 1: Define a training set temporary table to store the normalized training set. Each training record in the normalized training set is represented as a binary tuple consisting of an independent variable group vector and a dependent variable value. As shown in lines 1 - 2 of the SQL statement example, the input training set aisql.iris is represented as a normalized training set temporary table train, where each training record is a binary tuple consisting of an independent variable group vector X and a dependent variable y.

[0058] Step 2: Define a training hyperparameter assignment table to store the training hyperparameters. All training hyperparameters form a unique record in this temporary table. As shown in lines 3 - 4 of the SQL statement example, the training hyperparameters include the number of class labels nclass, the number of independent variables nfeature, the number of neurons in the hidden layer nhidden, the number of iterations niteration, and the learning rate alpha. They form a unique record in the training hyperparameter assignment table hyperparams, where the number of class labels nclass is obtained by counting the training set train.

[0059] Step 3: Divide all the parameters of each basic component in the network structure of the deep learning model into multiple parameter groups, and each of the said parameter groups corresponds to a vector. As shown in line 5 of the SQL statement example, the input deep learning model structure includes two fully connected layers, and each fully connected layer includes a weight matrix W and a bias vector b. Therefore, a total of four parameter groups W1, b1, W2, b2 are defined, where the weight matrices W1 and W2 are both represented as vectors in row - major order.

[0060] Step 4: Define a deep learning model parameter assignment table, where each field corresponds to a parameter group, the field value is called the assignment vector of the corresponding parameter group, and each record is called an assignment record, which consists of the iteration number and the assignment vectors of all parameter groups obtained after this round of iteration, and is used to store the deep learning model parameters obtained in each round of iteration. As shown in line 5 of the SQL statement example, the deep learning model parameter assignment table params_per_iteration is used to store the iteration number and the deep learning model parameters obtained after this round of iteration. Each round of iteration goes through two steps: forward numerical calculation and backward gradient propagation. The forward numerical calculation step obtains the output of each basic component in this round of iteration, and the backward gradient propagation obtains the gradient information of each basic component in this round of iteration. The gradient information is used to update the deep learning model parameters.

[0061] The specific process of translating the forward numerical calculation step and the backward gradient propagation step into SQL is as follows: i. If this is the first round, add an assignment record to the deep learning model parameter assignment table, store the random initialization assignment vectors of all basic component parameter groups, and mark this assignment record as the most recent parameter assignment record. Otherwise, directly mark the assignment record of the previous round of deep learning model parameters as the most recent parameter assignment record. As shown in line 6 of the SQL statement example, the random initialization assignment vectors of the basic component parameter groups are synthesized into the first assignment record and stored in the deep learning model parameter assignment table params_per_iteration. In line 6, random_vector is a function for generating random initialization assignment vectors. The input parameter of this function is the dimension of the vector, and the output parameter is a random initialization assignment vector of the specified dimension. As shown in lines 9 - 10 of the SQL statement example, the most recent parameter assignment record is stored in the most recent deep learning model parameter assignment table params_last_iteration and is the only record in this parameter assignment table.

[0062] ii. Determine the forward traversal order of the DAG of basic components, ensuring that if there is an arc from basic component A to basic component B, then A is visited before B. Define output temporary tables in sequence according to this forward traversal order to store the output vectors of each basic component. The first output temporary table stores the output vector of the first basic component visited in the forward traversal, which is defined based on the training set temporary table, the recent parameter assignment record, and the training hyperparameter assignment table, as well as the training set and historical output vectors required for subsequent traversal processes. Each subsequent output temporary table stores the output vector of the current basic component, which is defined based on the previous output temporary table, the recent parameter assignment table record, and the training hyperparameter assignment table, as well as the training set and historical output vectors required for subsequent traversal processes. If the calculation of the basic component output vector does not involve training hyperparameters, the corresponding output temporary table does not use the training hyperparameter assignment table during definition. As shown in lines 11 - 16 of the SQL statement example, the first basic component visited in the forward traversal is the fully connected layer with the sigmoid activation function. The output vector of this connection layer is defined based on the training set temporary table train and the deep learning model parameter assignment table params_last_iteration of the previous iteration, resulting in the first output temporary table layer1_out. layer1_out stores the output vector l1_out of the first basic component, as well as the training data X and y required for subsequent traversal processes; the second basic component visited in the forward traversal is the fully connected layer with the softmax activation function. The output vector of this connection layer is defined based on the first output temporary table layer1_out and the deep learning model parameter assignment table params_last_iteration of the previous iteration, resulting in the second output temporary table layer2_out. layer2_out stores the output vector l2_out of the second basic component, the training data X and y required for subsequent traversal processes, and the output vector l1_out of the first basic component. In lines 12 and 13, matrix_mul_vector is a function for multiplying a matrix by a vector, where the matrix is represented as a vector with elements stored in row-major order; in line 12, vector_sigmoid is the sigmoid activation function for a vector; in line 13, vector_softmax is the softmax activation function for a vector.

[0063] iii. Determine the reverse traversal order of the directed acyclic graph of the basic components, ensuring that if there is an arc from basic component A to basic component B, then B is visited before A. Define the gradient temporary tables in sequence according to this reverse traversal order to store the gradient vectors of each basic component, where each parameter group or input object of the basic component corresponds to a gradient vector. The first gradient temporary table stores the gradient vector of the first basic component in the reverse traversal defined based on the output temporary table generated last in the forward traversal, the recent parameter assignment record, and the training hyperparameter assignment table, as well as the training set, historical output vectors, and historical gradient vectors required in the subsequent traversal process. Each subsequent gradient temporary table stores the gradient vector of the current basic component defined based on the previous gradient temporary table, the recent parameter assignment record, and the training hyperparameter assignment table, as well as the training data, historical output vectors, and historical gradient vectors required in the subsequent traversal process. If the calculation of the basic component gradient vector does not involve training hyperparameters, the corresponding gradient temporary table does not use the training hyperparameter assignment table when being defined. If the calculation of the basic component gradient vector does not involve the deep learning model parameters of the previous iteration, the corresponding gradient temporary table does not use the deep learning model parameter assignment table of the previous iteration when being defined. In addition, if a basic component can be decomposed into smaller-grained components, and the gradient vectors of two or more parameter groups in the latter part depend on the gradient vector of a certain parameter group in the former part, then each part constituting the basic component corresponds to a gradient temporary table.As shown in lines 17 - 28 of the SQL statement example, the penultimate basic component is a fully connected layer with a softmax activation function. The gradients of the two parameter groups in the linear transformation part of this fully connected layer depend on the gradients of the parameter group in the softmax part. Therefore, two gradient temporary tables, grad_activation2 and grad_linear2, need to be generated. grad_activation2 is defined based on the output temporary table layer2_out and the training hyperparameter assignment table hyperparams generated at the end of the forward traversal, and stores the gradient vector g_act2 of the softmax part of the penultimate basic component, as well as the training data X and the output vector l1_out of the first basic component required for subsequent traversal. grad_linear2 is defined based on the previous gradient temporary table grad_activation2, the deep learning model parameter assignment table params_last_iteration in the previous iteration, and the training hyperparameter assignment table hyperparams, and stores the two gradient vectors g_W2 and g_b2 of the parameter groups in the linear transformation part of the penultimate basic component, the gradient vector g_in2 of the input layer of the penultimate basic component, as well as the training data X and the output vector l1_out of the first basic component required for subsequent traversal; the antepenultimate basic component is a fully connected layer with a sigmoid activation function. The gradients of the two parameter groups in the linear transformation part of this fully connected layer depend on the gradients of the parameter group in the sigmoid part. Therefore, two gradient temporary tables, grad_activation1 and grad_linear1, need to be generated. grad_activation1 is defined based on the previous gradient temporary table grad_linear2, and stores the gradient vector g_act2 of the sigmoid part of the antepenultimate basic component, as well as the training data X and the gradient vectors g_W2 and g_b2 of the penultimate basic component required for subsequent traversal. grad_linear1 is defined based on the previous gradient temporary table grad_activation1, and stores the two gradient vectors g_W1 and g_b1 of the parameter groups in the linear transformation part of the antepenultimate basic component, as well as the gradient vectors g_W2 and g_b2 of the penultimate basic component required for subsequent traversal. The one_hot_vector on line 18 is a one - hot vector generation function that returns a vector of a specified dimension with the specified serial number component being 1 and the rest being 0; the vector_cartesian_product on lines 21 and 27 is a Cartesian product function between vectors; the matrix_transpose on line 21 is a matrix transpose function that returns the transpose matrix of the input matrix, where both the input matrix and the output matrix are represented as vectors with each unit stored in row - major order.

[0064] iv. According to the gradient temporary table generated finally by backward traversal, define a gradient accumulation temporary table to store the summation aggregation value of the gradient vectors of all basic component parameter groups in the deep learning model, that is, for each parameter group, sum the corresponding gradient vectors of each training record to obtain the gradient accumulation vector of this parameter group. As shown in lines 29-30 of the SQL statement example, the gradient accumulation temporary table grad_sum is generated by cumulatively summing each parameter group field in the gradient temporary table grad_linear1 generated finally by backward traversal.

[0065] v. According to the parameter update formula in the gradient descent method, that is, the parameter group assignment vector in this round is equal to the difference between the parameter group assignment vector in the previous round and the parameter group gradient adjustment vector in this round, where the parameter group gradient adjustment vector in this round is the product of the learning rate input by the user and the parameter group gradient vector in this round, define the parameter assignment table for this round of deep learning model iteration to store the new iteration round and the new assignment vectors of all basic component parameter groups in the deep learning model. As shown in lines 31-33 of the SQL statement example, the parameter assignment table for this round of deep learning model iteration params_this_iteration is defined according to the parameter assignment table for the previous round of deep learning model iteration params_last_iteration, the gradient accumulation temporary table grad_sum, and the training hyperparameter assignment table hyperparams, and is used to save the updated values of each parameter group in the deep learning model after this round of iteration.

[0066] Step Five: After defining the parameter assignment table of the deep learning model composed of the specified number of iteration round assignment records, define the final parameter assignment table of the deep learning model to store the deep learning model parameters generated in the last round of iteration. As shown in lines 35-36 of the SQL statement example, the final parameter assignment table of the deep learning model final_params is defined according to the corresponding assignment records in the last round of iteration in the parameter assignment table of the deep learning model params_per_iteration, and is used to save the updated values of each parameter group in the deep learning model after the last round of iteration. Step Six: Save the training request ID and the deep learning model parameters stored in the final parameter assignment table of the deep learning model in the deep learning model table. As shown in lines 39-40 of the SQL statement example, the training request ID and the deep learning model parameters stored in the final parameter assignment table of the deep learning model final_params are saved in the deep learning model table aisql.dl_model_params, where the deep learning model parameters are stored in the form of a vector array, and each element in the vector array is a vector and corresponds to a parameter group of the deep learning model. In the training phase of this technical solution, the intermediate data generated by the deep learning model during the training phase is saved in the form of a temporary table. Relying on the efficient management ability of the temporary table that combines main memory and disk in modern databases, the training process of the deep learning model can still be efficiently executed when the size of the training set far exceeds the physical memory capacity. The temporary table generated during the training phase can be automatically deleted after the model training is completed, thus avoiding the physical storage space of the database being occupied by the temporary data generated during the training process. In addition, considering that generally only thousands of fields can be defined in a single form in the database, the technical solution of the present invention introduces a vector type field to store the parameter groups of the basic components of the deep learning model, ensuring that a single form with a limited total number of fields in the database can also store all the parameters of a large number of basic components. Therefore, the technical solution of the present invention is applicable to the training of deep learning models containing thousands of basic components.

[0067] Embodiments of the present disclosure also provide an electronic device. Figure 4 The structure diagram of the electronic device suitable for implementing the embodiments of the present disclosure is shown. Figure 4 The illustrated electronic device 600 is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0068] As Figure 4 shown, the electronic device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage section 608 into the random access memory (RAM) 603. In the RAM 603, various programs and data required for system operation are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.

[0069] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. The drive 610 is also connected to the I / O interface 605 as required. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as required so that the computer program read from it can be installed into the storage section 608 as required.

[0070] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product that includes a computer program carried on a computer-readable storage medium, and the computer program includes program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication section 609 and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above functions defined in the embodiments of the present application are performed.

[0071] It should be noted that the computer-readable medium shown in the embodiments of the present disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device. In the embodiments of the present disclosure, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination of the above.

[0072] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in an order different from that noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as combinations of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0073] The units described in the embodiments of the present disclosure can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not, in some cases, constitute a limitation on the unit itself.

[0074] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and the one or more programs include instructions that, when executed by the electronic device, cause the electronic device to implement the methods described in the above embodiments.

[0075] It should be noted that although several modules or units of devices for performing actions are mentioned in the above detailed description, such a division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0076] The above content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A training method for a deep learning model, characterized in that, Including: A preparation stage and a training stage; In the preparation stage, for vector and / or matrix operations in the calculation formulas of different basic components of a deep learning model that cannot be implemented by the built-in functions of standard SQL, corresponding user-defined functions for the vector and / or matrix operations are introduced, encapsulated into a database plugin and integrated into the database management system; in addition, in the preparation stage, a deep learning model table is constructed to store the training request ID input by the user and the corresponding deep learning model parameters obtained in the training stage; In the training stage, according to the training request input by the user, the parameters of the corresponding deep learning model are generated and saved, and the training request includes a training request ID, the network structure of the deep learning model, training hyperparameter assignments, and a training set; The training stage consists of a conversion link and an execution link; In the conversion link, the basic components of the deep learning model input by the user are determined, a directed acyclic graph of the basic components is obtained, and an SQL statement is output; each basic component has a pre-agreed operation process and corresponds to a subset of the parameters of the deep learning model input by the user, and the operation process of the basic component is a calculation formula, and the calculation formula is constructed according to the built-in functions of standard SQL and the user-defined functions introduced in the preparation stage; the training hyperparameters input by the user include the number of iterations, the learning rate, etc.; The SQL statement output by the conversion link is defined step by step as follows: First, define the construction process of the training set temporary table; Second, define the construction process of the training hyperparameter assignment table; Third, define the construction process of the temporary tables involved in each iteration. The temporary tables involved in each iteration include: a temporary table composed of the deep learning model parameter assignment records obtained in each iteration, a basic component output temporary table obtained in sequence according to the forward traversal order of the directed acyclic graph of the basic components, a basic component gradient temporary table obtained in sequence according to the backward traversal order of the directed acyclic graph of the basic components, and a basic component cumulative gradient temporary table obtained by summarizing all the records of the training set; Fourth, define the construction process of the final parameter assignment table of the deep learning model input by the user; Finally, define the storage process of the final parameter assignment table of the deep learning model input by the user into the database; In the execution link, the SQL statement output by the conversion link is executed in the database management system to obtain the final parameter assignment table of the deep learning model input by the user, and the final parameter assignment table is saved into the deep learning model table constructed in the preparation stage.

2. The training method of the deep learning model according to claim 1, characterized in that The obtaining of the directed acyclic graph of the basic components includes: Constructing a directed acyclic graph with each basic component of the deep learning model input by the user as a node. There is an arc from node A to node B if and only if an output object of the basic component corresponding to node A is an input object of the basic component corresponding to node B.

3. The training method of the deep learning model according to claim 1, wherein The defining of the construction process of the training set temporary table includes: The training set input by the user is a set of training records. Each training record consists of assignments of multiple independent variables and an assignment of a single dependent variable. The assignments of all independent variables in each training record are combined into a vector, which, together with the corresponding dependent variable assignment, forms a binary tuple and is saved as a record in the training set temporary table in the training set temporary table.

4. The training method of the deep learning model according to claim 1, wherein The construction process of defining the training hyperparameter assignment table includes: Collect all the training hyperparameter assignments input by the user to form the only record in the training hyperparameter assignment table.

5. The training method of the deep learning model according to claim 1, wherein The construction process of defining the temporary table involved in each round of iteration includes: Divide all the parameters of each basic component in the network structure of the deep learning model input by the user into multiple parameter groups, and each parameter group corresponds to a vector; define the deep learning model parameter assignment table. Each field in the deep learning model parameter assignment table corresponds to a parameter group, and the field value is called the assignment vector of the corresponding parameter group. In addition, each record in the deep learning model parameter assignment table is called an assignment record, and the assignment record consists of the iteration round number and the assignment vectors of all parameter groups obtained after this round of iteration, and is used to store the deep learning model parameters obtained in each round of iteration; define the following iteration operations, and the number of rounds of the iteration operation is equal to the number of iteration rounds input by the user: If this round is the first round, add an assignment record to the deep learning model parameter assignment table, store the randomly initialized assignment vectors of all basic component parameter groups, and mark the assignment record as the most recent parameter assignment record. Otherwise, directly mark the assignment record of the deep learning model parameters in the previous round as the most recent parameter assignment record; Determine the forward traversal order of the directed acyclic graph of the basic components, ensuring that if the corresponding node of basic component A has an arc pointing to the corresponding node of basic component B, then A is visited before B; define the output temporary tables of each basic component in turn according to the forward traversal order, and each field in the basic component output temporary table corresponds to an output object; Determine the backward traversal order of the directed acyclic graph of the basic components, ensuring that if the corresponding node of basic component A has an arc pointing to the corresponding node of basic component B, then B is visited before A; define the gradient temporary tables of each basic component in turn according to the backward traversal order, and each field in the basic component gradient temporary table corresponds to a parameter group or an input object, and each field value is called the gradient vector of the corresponding parameter group or the corresponding input object; According to the basic component gradient temporary table generated last according to the backward traversal order, define the gradient accumulation temporary table to store the sum aggregation value of the parameter group gradient vectors of all basic components in the deep learning model input by the user. The sum aggregation value of the parameter group gradient vectors is the gradient accumulation vector obtained by performing a summation operation on the corresponding parameter group gradient vectors of all training records; According to the parameter update formula in the standard gradient descent method, that is, the assignment vector of the current round of parameter group is equal to the difference between the assignment vector of the previous round of parameter group and the gradient adjustment vector of the current round of parameter group, define the assignment record of the deep learning model parameters in the current round. The gradient adjustment vector of the current round of parameter group is the product of the learning rate input by the user and the gradient vector of the current round of parameter group.

6. The training method of the deep learning model according to claim 1, wherein The construction process of defining the final parameter assignment table of the deep learning model input by the user includes: Taking the last record of the deep learning model parameter assignment table as the only record of the final parameter assignment table.

7. The training method of the deep learning model according to claim 1, wherein The storage process of defining the final parameter assignment table of the deep learning model input by the user includes: Converting the only record of the final parameter assignment table of the deep learning model input by the user into an array of parameter group assignment vectors, and adding it together with the training request ID input by the user as a record to the deep learning model table constructed in the preparation stage.

8. The training method of the deep learning model according to claim 5, wherein Defining the output temporary tables of each basic component in turn according to the forward traversal order, including: The output temporary table of the first basic component stores the output vector of the first basic component traversed forward, as well as the training set and historical output vectors required for the subsequent traversal process. The stored content is defined based on the training set temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table; the output temporary table of each subsequent basic component stores the output vector of the current basic component, as well as the training set and historical output vectors required for the subsequent traversal process. The stored content is defined based on the output temporary table of the previous basic component, the most recent parameter assignment record, and the training hyperparameter assignment table.

9. The training method of the deep learning model according to claim 5, wherein Defining the gradient temporary tables of each basic component in turn according to the backward traversal order, including: The gradient temporary table of the first basic component stores the gradient vector of the first basic component traversed backward, as well as the training set, historical output vector, and historical gradient vector required for the subsequent traversal process. The stored content is defined based on the output temporary table generated at the end of the forward traversal, the most recent parameter assignment record, and the training hyperparameter assignment table; the gradient temporary table of each subsequent basic component stores the gradient vector of the current basic component, as well as the training set, historical output vector, and historical gradient vector required for the subsequent traversal process. The stored content is defined based on the gradient temporary table of the previous basic component, the most recent parameter assignment record, and the training hyperparameter assignment table.

Images