Training methods for deep learning models

Abstract

A method for training a deep learning model relates to the field of data processing. The training method includes a preparation phase and a training phase. In the preparation phase, corresponding user-defined functions are introduced for vector and / or matrix operations in the calculation formulas of different basic components of the deep learning model that cannot be implemented by standard SQL built-in functions. These functions are encapsulated as database plug-ins and integrated into the database management system, while a deep learning model table is constructed. In the training phase, a directed acyclic graph of the basic components of the deep learning model in the training request input by the user is obtained, an SQL statement for model training is generated, and the SQL statement is executed in the database management system to obtain the final parameter assignment of the deep learning model, which is saved in the deep learning model table constructed in the preparation phase. The method utilizes database temporary tables to implement out-of-memory training, introduces vector fields to store parameter groups, supports efficient training of large-scale deep learning models, and automatically cleans up temporary data.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method for training a deep learning model. Background Art

[0002] With the advent of the big data era, data analysis and machine learning technologies are increasingly being used across industries. Database systems, as the core of data storage and management, need to be closely integrated with data analysis and machine learning technologies to support data-driven decision-making. In other words, modern database systems need to integrate machine learning algorithms into the database, achieving unified data storage, management, and analysis. This will enhance data analysis capabilities and streamline the machine learning process. Traditional data analysis processes typically include steps such as data extraction, transformation, and loading (ETL), and model training. These steps often require exporting data from the database and then analyzing it using external tools or platforms. This process is not only time-consuming and labor-intensive, but can also reduce analytical accuracy due to format conversion and data loss during data migration. Directly integrating deep learning algorithms into the database, known as "in-database deep learning," can change this situation. It allows users to perform analytical tasks directly where data is stored, eliminating the need for tedious data migration steps, significantly improving the efficiency and real-time nature of data analysis. Furthermore, in-database deep learning leverages the database's inherent optimization technologies and parallel processing capabilities to further accelerate the analysis process, making it easier and faster to process large datasets.

[0003] Currently, there are two main solutions for implementing in-database deep learning. One approach involves introducing user-defined functions (UDFs), also known as the UDF solution. A UDF is an advanced feature provided by database management systems that provides users with significant flexibility, allowing them to create and execute customized logic within the database environment. These custom functions not only perform basic mathematical operations, string processing, and data conversion, but also serve as a bridge to seamlessly connect the database with external advanced computing resources and machine learning frameworks. UDFs enable the integration and invocation of deep learning algorithms from popular deep learning frameworks such as PyTorch, TensorFlow, PaddlePaddle, and MindSpore within the database. Another approach involves directly translating the training and application processes of deep learning algorithms into SQL statements, also known as the translation solution. This approach allows users to execute deep learning tasks directly within the database without writing complex code or invoking external tools. The core concept is to convert the mathematical formulas and logic of deep learning algorithms into SQL statements.

[0004] To date, both approaches have unresolved issues. For one thing, the massive amount of data required for deep learning model training must be collected within the database query process and then submitted to the deep learning algorithm running in the same process. This can easily lead to system slowdowns or even crashes due to insufficient physical memory to accommodate the training set. Furthermore, converting the mathematical formulas and logic of deep learning algorithms into SQL statements requires a programmatic translation process to avoid errors introduced during translation. Summary of the Invention

[0005] This application provides a deep learning model training method suitable for efficiently training different deep learning models in a database.

[0006] In a first aspect, the present application provides a method for training a deep learning model, comprising:

[0007] preparation phase and training phase;

[0008] The preparation phase introduces corresponding user-defined functions for vector and / or matrix operations in the calculation formulas of different basic components of the deep learning model that cannot be implemented by standard SQL functions, encapsulates them into database plug-ins and integrates them into the database management system. In addition, the preparation phase constructs a deep learning model table for storing the training request ID input by the user and the corresponding deep learning model parameters obtained during the training phase.

[0009] The training phase generates and saves the parameters of the corresponding deep learning model according to the training request input by the user. The training request includes the training request ID, the network structure of the deep learning model, the training hyperparameter assignments and the training set;

[0010] The training phase consists of a conversion phase and an execution phase;

[0011] The conversion step determines the basic components of the deep learning model input by the user, obtains a directed acyclic graph of the basic components, and outputs an SQL statement; each basic component has a pre-agreed operation process and corresponds to a parameter subset of the deep learning model input by the user. The operation process of the basic component is a calculation formula, which is constructed based on standard SQL functions and user-defined functions introduced in the preparation stage; the training hyperparameters input by the user include the number of iterations and the learning rate, etc.

[0012] The SQL statements output by the conversion step are defined step by step as follows:

[0013] First, define the construction process of the training set temporary table;

[0014] Second, define the construction process of the training hyperparameter assignment table;

[0015] Third, define the construction process of the temporary tables involved in each round of iteration. The temporary tables involved in each round of iteration include: a temporary table composed of the deep learning model parameter assignment records obtained in each round of iteration, a basic component output temporary table obtained in sequence according to the forward traversal order of the directed acyclic graph of the basic components, a basic component gradient temporary table obtained in sequence according to the backward traversal order of the directed acyclic graph of the basic components, and a basic component cumulative gradient temporary table obtained by summarizing all records of the training set;

[0016] Fourth, defining the construction process of the final parameter assignment table of the deep learning model input by the user;

[0017] Finally, define the storage process of the final parameter assignment table of the deep learning model input by the user;

[0018] The execution link executes the SQL statement output by the conversion link in the database management system to obtain the final parameter assignment table of the deep learning model input by the user, and saves the final parameter assignment table to the deep learning model table constructed in the preparation stage.

[0019] In the above embodiment of the present application, the training data and the parameters to be trained required by the model are converted into vector-type data and stored in a database form. A set of parameters to be trained can be converted into a vector, that is, a vector can represent multiple parameters, so that the form can store a large number of parameters, thereby reducing the pressure of the data required for training on the physical memory, which is conducive to the training process running through the database query process, and then supporting the use of training data that exceeds the physical memory capacity in the database for deep learning model training, while avoiding data migration and improving the training efficiency of the deep learning model. In addition, by adopting the above technical solution of the present application, users can perform deep learning tasks directly in the database without writing complex code or calling external tools, thereby improving development efficiency.

[0020] Based on the above solution, obtaining the directed acyclic graph of the basic components includes:

[0021] A directed acyclic graph is constructed with each basic component of the deep learning model input by the user as a node, and node A has an arc pointing to node B if and only if an output object of the basic component corresponding to node A is an input object of the basic component corresponding to node B.

[0022] In the above solution, the construction process of defining the temporary table of the training set includes:

[0023] The training set input by the user is a set of training records, each of which is composed of the assignments of multiple independent variables and the assignment of a single dependent variable. All the independent variable assignments of each training record are synthesized into a vector, which together with the corresponding dependent variable assignment constitutes a tuple, which is saved in the training set temporary table as a record in the training set temporary table.

[0024] In the above solution, the construction process of defining the training hyperparameter assignment table includes:

[0025] All training hyperparameter values ​​input by the user are collected to form a unique record of the training hyperparameter value table.

[0026] In the above solution, the construction process of defining the temporary table involved in each iteration includes:

[0027] Divide all parameters of each basic component in the network structure of the deep learning model input by the user into multiple parameter groups, each parameter group corresponds to a vector; define a deep learning model parameter assignment table, each field of the deep learning model parameter assignment table corresponds to a parameter group, and the field value is called the assignment vector of the corresponding parameter group. In addition, each record of the deep learning model parameter assignment table is called an assignment record, and the assignment record consists of an iteration round and the assignment vectors of all parameter groups obtained after the iteration round, and is used to store the deep learning model parameters obtained in each iteration round; define the following iterative operation, and the number of rounds of the iterative operation is equal to the number of iteration rounds input by the user:

[0028] If this is the first round, add an assignment record to the deep learning model parameter assignment table to store the randomly initialized assignment vectors of all basic component parameter groups, and mark the assignment record as the most recent parameter assignment record. Otherwise, directly mark the assignment record of the deep learning model parameters in the previous round as the most recent parameter assignment record.

[0029] Determine a forward traversal order of the directed acyclic graph of the basic components, ensuring that if a corresponding node of basic component A has an arc pointing to a corresponding node of basic component B, then A is accessed before B; define each basic component output temporary table in sequence according to the forward traversal order, where each field of the basic component output temporary table corresponds to an output object;

[0030] Determine a backward traversal order of the directed acyclic graph of the basic components, ensuring that if a corresponding node of basic component A has an arc pointing to a corresponding node of basic component B, then B is accessed before A; define each basic component gradient temporary table in sequence according to the backward traversal order, where each field of the basic component gradient temporary table corresponds to a parameter group or an input object, and the value of each field is called a gradient vector of the corresponding parameter group or the corresponding input object;

[0031] Based on the basic component gradient temporary table generated last in the backward traversal order, a gradient accumulation temporary table is defined to store the summed aggregated value of the parameter group gradient vectors of all basic components in the deep learning model input by the user, where the summed aggregated value of the parameter group gradient vector is a gradient accumulation vector obtained by summing the corresponding parameter group gradient vectors of all training records;

[0032] According to the parameter update formula in the standard gradient descent method, that is, the parameter group assignment vector of this round is equal to the difference between the parameter group assignment vector of the previous round and the gradient adjustment vector of this round, the assignment record of the parameters of the current round deep learning model is defined. The gradient adjustment vector of the parameter group of this round is the product of the learning rate input by the user and the gradient vector of the parameter group of this round.

[0033] In the above solution, the process of constructing the final parameter assignment table defining the deep learning model input by the user includes:

[0034] The last record of the deep learning model parameter assignment table is used as the only record of the final parameter assignment table.

[0035] In the above solution, the process of storing the final parameter assignment table defining the deep learning model input by the user includes:

[0036] The unique record of the final parameter assignment table of the deep learning model input by the user is converted into a parameter group assignment vector array, and together with the training request ID input by the user, is added as a record to the deep learning model table constructed in the preparation stage.

[0037] The above solution is characterized in that the step of defining the output temporary tables of each basic component in sequence according to the forward traversal order includes:

[0038] The first basic component output temporary table saves the output vector of the forward traversal of the first basic component and the training set and historical output vectors required for the subsequent traversal process. The saved content is defined based on the training set temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table. Each subsequent basic component output temporary table saves the output vector of the current basic component and the training set and historical output vectors required for the subsequent traversal process. The saved content is defined based on the previous basic component output temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table.

[0039] The above solution is characterized in that the step of defining the temporary gradient tables of the basic components in sequence according to the backward traversal order includes:

[0040] The first basic component gradient temporary table saves the gradient vector of the first basic component in the backward traversal, as well as the training set, historical output vector, and historical gradient vector required for the subsequent traversal process. The saved content is defined based on the output temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table generated at the end of the forward traversal. Each subsequent basic component gradient temporary table saves the gradient vector of the current basic component as well as the training set, historical output vector, and historical gradient vector required for the subsequent traversal process. The saved content is defined based on the previous basic component gradient temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table.

[0041] In a second aspect, the present application provides an electronic device comprising a memory and one or more processors. The memory stores one or more computer programs, each of which includes instructions that, when executed by the processor, enable the electronic device to perform the deep learning model training method described in the first aspect.

[0042] In a third aspect, the present application provides a computer-readable storage medium having instructions stored therein. When the instructions are executed on an electronic device, the electronic device executes the training method of the deep learning model as in the first aspect.

[0043] In a fourth aspect, the present application provides a computer program product, which, when running on an electronic device, enables the electronic device to execute the training method of the deep learning model as described in the first aspect.

[0044] It can be understood that the beneficial effects that can be achieved by the training device, electronic device, computer-readable storage medium, and computer program product of the deep learning model provided above can be referred to the beneficial effects in the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a deep learning model training method provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the deep learning model network structure input in a specific case of the deep learning model training method provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of the target SQL statement output in a specific case of the deep learning model training method provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] To facilitate a clear description of the technical solutions in the embodiments of this application, terms such as "first" and "second" are used in the embodiments of this application to distinguish between identical or similar items with substantially the same functions and effects. For example, the terms "first chip" and "second chip" are used solely to distinguish between different chips and do not limit their order of precedence. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and do not necessarily imply differences. It should be noted that in the embodiments of this application, terms such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in this application as "exemplarily" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of terms such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner. In the embodiments of this application, "at least one" refers to one or more, and "more" refers to two or more.

[0050] It should be noted that the “at…” in the embodiments of the present application can be the instant when a certain situation occurs, or it can be a period of time after the occurrence of a certain situation. The embodiments of the present application do not make specific limitations on this.

[0051] The implementation of this embodiment will be described in detail below with reference to the accompanying drawings.

[0052] This embodiment provides a deep learning model training method, which trains the deep learning model through database query statements without the need for data migration between different systems, thereby improving the training efficiency of the deep learning model.

[0053] As a key branch of machine learning, deep learning has become a hallmark of artificial intelligence today, thanks to its advanced features for automatic feature extraction and highly accurate prediction. Deep learning algorithms operate in two main phases: the training phase (also known as the learning phase), where a deep learning model is acquired through learning from a training set; and the application phase (also known as the prediction phase), where the trained model is used to make predictions on new data. Deep learning models are generally represented using neural networks. Neural networks consist of multiple layers, each containing several components. A neural network can be simply viewed as a composite function, with the components of the neural network acting as the building blocks of the composite function. The goal of deep learning model training is to adjust the neural network parameters so that the model's predictions on the training set are as close to the true values ​​as possible, thereby improving the model's generalization ability.

[0054] It is understood that the method in this embodiment is applicable not only to the training phase of the deep learning model, but also to the application phase of the deep learning model. Both the training phase and the application phase are executed through database statements, without the need for data migration.

[0055] The following describes the specific process of the deep learning model training method provided in this embodiment. This deep learning model training method can be applied to various electronic devices equipped with databases, such as computers (PCs), tablets, virtual reality / augmented reality devices, wearable devices, industrial computers, and vehicle computers. It can also be applied to servers, cloud platforms, and server clusters that support database services, without any specific limitations in this embodiment.

[0056] Figure 1 A flow chart of the deep learning model training method provided in an embodiment of the present application is shown.

[0057] like Figure 1 As shown, the training method of the deep learning model includes a preparation stage and a training stage; in the preparation stage, for vector and / or matrix operations in the calculation formulas of different basic components of the deep learning model that cannot be implemented by standard SQL built-in functions, corresponding user-defined functions of the vector and / or matrix operations are introduced, and the functions are encapsulated into database plug-ins and integrated into the database management system; in addition, the preparation stage constructs a deep learning model table for storing the training request ID input by the user and the corresponding deep learning model parameters obtained in the training stage; in the training stage, the parameters of the corresponding deep learning model are generated and saved according to the training request input by the user, and the training request includes the training request ID, the network structure of the deep learning model, the training hyperparameter assignment and the training set.

[0058] The training phase consists of a conversion phase and an execution phase; the conversion phase determines the basic components of the deep learning model input by the user, obtains a directed acyclic graph of the basic components, and outputs an SQL statement; each basic component has a pre-agreed operation process and corresponds to a parameter subset of the deep learning model input by the user, and the operation process of the basic component is a calculation formula, which is constructed based on the standard SQL built-in functions and the user-defined functions introduced in the preparation phase; the training hyperparameters input by the user include the number of iterations and the learning rate, etc.

[0059] The SQL statements output by the conversion step are defined step by step as follows:

[0060] First, define the construction process of the training set temporary table;

[0061] Second, define the construction process of the training hyperparameter assignment table;

[0062] Third, define the construction process of the temporary tables involved in each round of iteration. The temporary tables involved in each round of iteration include: a temporary table composed of the deep learning model parameter assignment records obtained in each round of iteration, a basic component output temporary table obtained in sequence according to the forward traversal order of the directed acyclic graph of the basic components, a basic component gradient temporary table obtained in sequence according to the backward traversal order of the directed acyclic graph of the basic components, and a basic component cumulative gradient temporary table obtained by summarizing all records of the training set;

[0063] Fourth, defining the construction process of the final parameter assignment table of the deep learning model input by the user;

[0064] Finally, define the storage process of the final parameter assignment table of the deep learning model input by the user;

[0065] The execution link executes the SQL statement output by the conversion link in the database management system to obtain the final parameter assignment table of the deep learning model input by the user, and saves the final parameter assignment table to the deep learning model table constructed in the preparation stage.

[0066] During the preparation phase, a database plugin is built to provide vector field management functionality used during the training phase. Furthermore, a deep learning model table is constructed to store the training request ID and the corresponding deep learning model parameters obtained during the training phase. The database plugin is implemented in C and contains user-defined functions (UDFs) for vectors, matrices, and related operations. The database plugin supports vector field management, where matrices are represented as vectors in a row-major fashion. For example, the database plugin includes the following UDFs:

[0067] Vector initialization: Generates a vector based on the vector length and initialization strategy. Initialization strategies include random initialization, all-zero initialization, and one-hot initialization. One-hot initialization generates a one-hot vector with only one component set to 1 and all other components set to 0. When using the one-hot initialization strategy, you can specify the ordinal number of the 1 component in the one-hot vector.

[0068] Vector arithmetic operations: Add, subtract, multiply, or divide corresponding components of two vectors of the same dimension and return the resulting vector.

[0069] Four arithmetic operations between vectors and scalars: add, subtract, multiply, or divide each component of a vector by a scalar, and return the changed vector.

[0070] Euclidean distance between two vectors: Calculates the Euclidean distance between two vectors of the same dimension and returns the resulting vector.

[0071] Vector-to-Vector Dot Product: Calculates the dot product of two vectors of the same dimension and returns the result as a scalar.

[0072] Vector-Vector Cartesian Product: Computes the Cartesian product of two vectors and returns the result matrix. The output matrix is ​​represented as a vector with cells in row-major order.

[0073] Vector Magnitude: Calculates the magnitude of a vector, which is the square root of the sum of the squares of all its components, and returns a scalar.

[0074] Vector normalization: Divides each component of a vector by its magnitude and returns the normalized vector.

[0075] Vector transformation: Transform each component of a vector using activation functions such as sigmoid, tanh, and ReLU, and return the transformed vector.

[0076] Vector soft maximization: Use the softmax function to convert all components of the vector into real numbers between 0 and 1, so that the sum of all components is 1, and return the transformed vector.

[0077] Matrix Transpose: Swaps the rows and columns of a matrix and returns the transposed matrix. Both the input and output matrices are represented as vectors with cells in row-major order.

[0078] Matrix-Vector Multiply: Returns the result vector of the matrix-vector multiplication. Each component of the result vector is the dot product of a row vector of the input matrix and the input vector. The input matrix is ​​represented as a vector with cells in row-major order.

[0079] Matrix-Matrix Multiply: Returns the matrix result of the matrix-matrix multiplication. Each cell in the result matrix is ​​the dot product of a row vector of the first input matrix and a column vector of the second input matrix. The first input matrix, the second input matrix, and the output matrix are all represented as vectors with each cell in row-major order.

[0080] A database plug-in was introduced to store the parameters of deep learning model foundational components in vector-type fields, allowing a single table in the database, with a limited number of fields, to store all the parameters of a large number of foundational components. By using user-defined functions (UDFs) for vectors, matrices, and related operations, the mathematical formulas and logic of deep learning algorithms can be converted into UDFs, simplifying the target SQL statements and improving their execution efficiency.

[0081] Obtaining the directed acyclic graph of the basic components specifically includes: constructing a directed acyclic graph with each basic component of the deep learning model input by the user as a node, and node A has an arc pointing to node B if and only if an output object of the basic component corresponding to node A is an input object of the basic component corresponding to node B.

[0082] The construction process of defining a temporary table of a training set includes: the training set input by the user is a set of training records, each of the training records is composed of the assignment of multiple independent variables and the assignment of a single dependent variable, all the independent variable assignments of each training record are synthesized into a vector, and together with the corresponding dependent variable assignment, a two-tuple is formed, which is saved in the temporary table of the training set as a record of the temporary table of the training set.

[0083] The construction process of defining the training hyperparameter assignment table includes: collecting all the training hyperparameter assignments input by the user to form a unique record of the training hyperparameter assignment table.

[0084] The construction process of the temporary table involved in each round of iteration is defined, including: dividing all parameters of each basic component in the network structure of the deep learning model input by the user into multiple parameter groups, each parameter group corresponds to a vector; defining a deep learning model parameter assignment table, each field of the deep learning model parameter assignment table corresponds to a parameter group, and the field value is called the assignment vector of the corresponding parameter group. In addition, each record of the deep learning model parameter assignment table is called an assignment record, and the assignment record consists of the iteration round and the assignment vectors of all parameter groups obtained after the iteration round, which is used to store the deep learning model obtained in each round of iteration. Type parameter; define the following iterative operation, the number of rounds of the iterative operation is equal to the number of iterations input by the user: if this round is the first round, add an assignment record to the deep learning model parameter assignment table, store the randomly initialized assignment vectors of all basic component parameter groups, and mark the assignment record as the most recent parameter assignment record, otherwise directly mark the assignment record of the previous round of deep learning model parameters as the most recent parameter assignment record; determine the forward traversal order of the directed acyclic graph of the basic components, and ensure that if the corresponding node of basic component A has an arc pointing to the corresponding node of basic component B, then A is visited before B; according to the forward traversal The output temporary tables of each basic component are defined in sequence, and each field of the basic component output temporary table corresponds to an output object; the backward traversal order of the directed acyclic graph of the basic component is determined to ensure that if the corresponding node of basic component A has an arc pointing to the corresponding node of basic component B, then B is accessed before A; the gradient temporary tables of each basic component are defined in sequence according to the backward traversal order, and each field of the basic component gradient temporary table corresponds to a parameter group or an input object, and the value of each field is called the gradient vector of the corresponding parameter group or the corresponding input object; the gradient temporary tables of the basic component finally generated according to the backward traversal order are defined. A time table is defined, and a gradient accumulation temporary table is defined to store the summed aggregated values ​​of the parameter group gradient vectors of all basic components in the deep learning model input by the user. The summed aggregated value of the parameter group gradient vector is the gradient accumulation vector obtained by summing the corresponding parameter group gradient vectors of all training records; according to the parameter update formula in the standard gradient descent method, that is, the parameter group assignment vector of this round is equal to the difference between the parameter group assignment vector of the previous round and the parameter group gradient adjustment vector of this round, the assignment record of the parameters of the deep learning model of this round is defined, and the parameter group gradient adjustment vector of this round is the product of the learning rate input by the user and the parameter group gradient vector of this round.

[0085] The construction process of defining the final parameter assignment table of the deep learning model input by the user includes: using the last record of the deep learning model parameter assignment table as the only record of the final parameter assignment table.

[0086] The process of defining the warehousing of the final parameter assignment table of the deep learning model input by the user includes: converting the unique record of the final parameter assignment table of the deep learning model input by the user into a parameter group assignment vector array, and adding it together with the training request ID input by the user as a record to the deep learning model table constructed in the preparation stage.

[0087] The output temporary tables of each basic component are defined in sequence according to the forward traversal order, including: the first basic component output temporary table stores the output vector of the first basic component in the forward traversal and the training set and historical output vectors required for the subsequent traversal process, and the stored content is defined based on the training set temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table; each subsequent basic component output temporary table stores the output vector of the current basic component and the training set and historical output vectors required for the subsequent traversal process, and the stored content is defined based on the previous basic component output temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table.

[0088] The gradient temporary tables of each basic component are defined in sequence according to the backward traversal order, including: the first basic component gradient temporary table saves the gradient vector of the first basic component in the backward traversal and the training set, historical output vector and historical gradient vector required for the subsequent traversal process, and the saved content is defined based on the output temporary table, the most recent parameter assignment record and the training hyperparameter assignment table generated at the end of the forward traversal; each subsequent basic component gradient temporary table saves the gradient vector of the current basic component and the training set, historical output vector and historical gradient vector required for the subsequent traversal process, and the saved content is defined based on the previous basic component gradient temporary table, the most recent parameter assignment record and the training hyperparameter assignment table.

[0089] The training phase processes training requests. The training request content includes the training request ID, training set, deep learning model network structure, and training hyperparameters. The training set is a specified table in the database, and all fields in the table have been preprocessed into numeric types. The deep learning model structure is a directed acyclic graph of the deep learning model's basic components. Nodes in the directed acyclic graph are basic components. An arc from node A to node B indicates that an input of the corresponding basic component of B is an output of the corresponding basic component of A. Training hyperparameters include the learning rate, number of iterations, and number of independent variables. In deep learning algorithms, the parameters of the deep learning model's basic components are trainable. The training process requires multiple rounds of forward numerical computation and backward gradient propagation to adjust these parameters so that the deep learning model's predictions on the training set are as close as possible to the true values ​​of the training set. Training hyperparameters are parameters that need to remain constant during training. Some deep learning model structures are not defined as a directed acyclic graph of basic components. Therefore, when defining the content of the training request, it is necessary to represent the deep learning model structure as a directed acyclic graph of basic components. For example, the original structure of a cyclic deep learning model is the loop body of a single cyclic deep learning model component. Therefore, it is necessary to first expand the original structure of the cyclic deep learning model into a directed acyclic graph of cyclic deep learning model components at a finite time, and then decompose the cyclic deep learning model component into a directed acyclic subgraph composed of basic components.

[0090] Reference below Figure 2 ,by Figure 2 The following example shows the training process of a deep learning model, which illustrates how to use recursive common entrant expressions (CTEs) to generate SQL statements for model training. The SQL statements can be executed in the database to complete the training and storage of the deep learning model.

[0091] Assume the following training request: The training request ID is 1, the training set is in the form aisql.iris, and contains four non-class label fields (i.e., independent variables) a1, a2, a3, and a4, and one class label field (i.e., dependent variable) label. The deep learning model structure is a multi-layer feedforward neural network with one hidden layer, as shown in the figure below. The training hyperparameters are a learning rate of 0.001, 1000 iterations, 4 independent variables, and 10 hidden layer neurons. The network structure consists of an input layer, a hidden layer, and an output layer. The hidden layer is a fully connected component with a sigmoid activation function, and the output layer is a fully connected component with a softmax activation function. Based on the above training request, the final generated SQL statement is as follows: Figure 3 shown.

[0092] Figure 3 The specific generation process of SQL statements is as follows:

[0093] Step 1: Define a temporary training table to store the normalized training set. Each training record in the normalized training set is represented as a two-tuple consisting of an independent variable vector and a dependent variable value. As shown in lines 1-2 of the SQL statement example, the input training set aisql.iris is represented as a normalized temporary training table, train, where each training record is a two-tuple consisting of an independent variable vector X and a dependent variable y.

[0094] Step 2: Define a training hyperparameter assignment table to store training hyperparameters. All training hyperparameters are combined into a single record in this temporary table. As shown in lines 3-4 of the SQL statement example, the training hyperparameters include the number of class labels nclass, the number of independent variables nfeature, the number of neurons in the hidden layer nhidden, the number of iterations niteration, and the learning rate alpha. These are combined into a single record in the training hyperparameter assignment table hyperparams. The number of class labels nclass is obtained from the training set train.

[0095] Step 3: Divide all parameters of each basic component in the deep learning model's network structure into multiple parameter groups, each corresponding to a vector. As shown in line 5 of the SQL statement example, the input deep learning model structure includes two fully connected layers, each of which includes a weight matrix W and a bias vector b. Therefore, a total of four parameter groups W1, b1, W2, and b2 are defined, where the weight matrices W1 and W2 are both represented as vectors in a row-major manner.

[0096] Step 4: Define a deep learning model parameter assignment table, where each field corresponds to a parameter group, the field value is called the assignment vector of the corresponding parameter group, and each record is called an assignment record. It consists of the iteration round and the assignment vectors of all parameter groups obtained after that iteration round, and is used to store the deep learning model parameters obtained after each iteration round. As shown in line 5 of the SQL statement example, the deep learning model parameter assignment table params_per_iteration is used to store the iteration round and the deep learning model parameters obtained after that iteration round. Each iteration goes through two steps: forward numerical calculation and backward gradient propagation. The forward numerical calculation step obtains the output of each basic component during the iteration round, and the backward gradient propagation obtains the gradient information of each basic component during the iteration round. The gradient information is used to update the deep learning model parameters.

[0097] The specific process of translating the forward numerical calculation step and the backward gradient propagation step into SQL is as follows:

[0098] i. If this is the first round, an assignment record is added to the deep learning model parameter assignment table to store the randomly initialized assignment vectors of all basic component parameter groups, and the assignment record is marked as the most recent parameter assignment record. Otherwise, the assignment record of the previous round of deep learning model parameters is directly marked as the most recent parameter assignment record. As shown in line 6 of the SQL statement example, the randomly initialized assignment vectors of the basic component parameter groups are combined into the first assignment record and stored in the deep learning model parameter assignment table params_per_iteration. In line 6, random_vector is a random initialization assignment vector generation function. The input parameter of this function is the dimension of the vector, and the output parameter is a randomly initialized assignment vector of the specified dimension. As shown in lines 9-10 of the SQL statement example, the most recent parameter assignment record is stored in the most recent deep learning model parameter assignment table params_last_iteration and is the only record in this parameter assignment table.

[0099] ii. Determine the forward traversal order of the directed acyclic graph of basic components to ensure that if basic component A has an arc pointing to basic component B, A is accessed before B. Based on this forward traversal order, define the output temporary tables to store the output vectors of each basic component. The first output temporary table stores the output vector of the first basic component of the forward traversal defined based on the training set temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table, as well as the training set and historical output vectors required for subsequent traversal processes. Each subsequent output temporary table stores the output vector of the current basic component defined based on the previous output temporary table, the most recent parameter assignment table record, and the training hyperparameter assignment table, as well as the training set and historical output vectors required for subsequent traversal processes. If the calculation of the basic component output vector does not involve training hyperparameters, the corresponding output temporary table does not use the training hyperparameter assignment table when defining it. As shown in lines 11-16 of the SQL statement example, the first basic component of the forward traversal is a fully connected layer with a sigmoid activation function. The output vector of this connection layer is defined based on the training set temporary table train and the parameter assignment table params_last_iteration of the previous iterative deep learning model. The first output temporary table layer1_out is obtained. layer1_out stores the output vector l1_out of the first basic component and the training data X and y required for subsequent traversal processes. The second basic component of the forward traversal is a fully connected layer with a softmax activation function. The output vector of this connection layer is defined based on the first output temporary table layer1_out and the parameter assignment table params_last_iteration of the previous iterative deep learning model. The second output temporary table layer2_out is obtained. layer2_out stores the output vector l2_out of the second basic component, the training data X and y required for subsequent traversal processes, and the output vector l1_out of the first basic component. In lines 12 and 13, matrix_mul_vector is the matrix-vector multiplication function, where the matrix is ​​represented as a vector with each unit stored in row-first order; in line 12, vector_sigmoid is the sigmoid activation function of the vector; in line 13, vector_softmax is the softmax activation function of the vector.

[0100] iii. Determine the backward traversal order of the directed acyclic graph of basic components, ensuring that if basic component A has an arc pointing to basic component B, B is accessed before A. Based on this backward traversal order, define gradient temporary tables to store the gradient vectors for each basic component, with each parameter group or input object of the basic component corresponding to a gradient vector. The first gradient temporary table stores the gradient vector of the first basic component in the backward traversal, as defined by the final output temporary table generated by the forward traversal, the most recent parameter assignment record, and the training hyperparameter assignment table, as well as the training data, historical output vectors, and historical gradient vectors required for subsequent traversals. Each subsequent gradient temporary table stores the gradient vector of the current basic component, as defined by the previous gradient temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table, as well as the training data, historical output vectors, and historical gradient vectors required for subsequent traversals. If the calculation of the gradient vector of a basic component does not involve training hyperparameters, the corresponding gradient temporary table is defined without using the training hyperparameter assignment table. If the calculation of the basic component's gradient vector does not involve the deep learning model parameters of the previous iteration, the corresponding temporary gradient table does not use the parameter assignment table of the previous iteration when defining it. In addition, if the basic component can be decomposed into smaller parts, and two or more parameter group gradient vectors in the latter part depend on the gradient vector of a parameter group in the previous part, then each part of the basic component corresponds to a corresponding temporary gradient table.As shown in lines 17-28 of the SQL statement example, the penultimate basic component is a fully connected layer with a softmax activation function. The gradients of the two parameter groups in the linear transformation part of the fully connected layer depend on the gradients of the parameter group in the softmax part. Therefore, two gradient temporary tables grad_activation2 and grad_linear2 need to be generated. Gradient_activation2 is defined based on the output temporary table layer2_out and the training hyperparams assignment table hyperparams generated at the end of the forward traversal, and saves the gradient vector g_act2 of the softmax part of the penultimate basic component, as well as the training data X and the output vector l1_out of the first basic component required for subsequent traversal. Gradient_linear2 is defined based on the previous gradient temporary table grad_activation2, the parameter assignment table params_last_iteration of the previous iterative deep learning model, and the training hyperparams assignment table hyperparams, and saves the two parameter group gradient vectors g_W2 and g_b2 of the linear transformation part of the penultimate basic component. The gradient vector g_in2 of the input layer of the first basic component, as well as the training data X and the output vector l1_out of the first basic component required for the subsequent traversal process; the penultimate basic component is a fully connected layer with a sigmoid activation function. The gradients of the two parameter groups of the linear transformation part of the fully connected layer depend on the gradients of the parameter group of the sigmoid part. Therefore, two gradient temporary tables grad_activation1 and grad_linear1 need to be generated. grad_activation1 is defined based on the previous gradient temporary table grad_linear2, and saves the gradient vector g_act2 of the sigmoid part of the penultimate basic component, as well as the training data X and the gradient vectors g_W2 and g_b2 of the penultimate basic component required for the subsequent traversal process. grad_linear1 is defined based on the previous gradient temporary table grad_activation1, and saves the two parameter group gradient vectors g_W1 and g_b1 of the linear transformation part of the penultimate basic component, as well as the gradient vectors g_W2 and g_b2 of the penultimate basic component required for the subsequent traversal process. The one_hot_vector function on line 18 is a one-hot vector generator that returns a vector of the specified dimension with the specified index component set to 1 and the remaining components set to 0. The vector_cartesian_product function on lines 21 and 27 is a Cartesian product function between vectors. The matrix_transpose function on line 21 is a matrix transpose function that returns the transposed matrix of the input matrix. Both the input matrix and the output matrix are represented as vectors with each cell stored in row-major order.

[0101] iv. Based on the temporary gradient table generated at the end of the backward traversal, define a temporary gradient accumulation table to store the aggregated values ​​of the gradient vectors for all basic component parameter groups in the deep learning model. Specifically, for each parameter group, sum the corresponding gradient vectors for each training record to obtain the cumulative gradient vector for that parameter group. As shown in lines 29-30 of the SQL statement example, the temporary gradient accumulation table grad_sum is generated by cumulatively summing each parameter group field in the temporary gradient table grad_linear1 generated at the end of the backward traversal.

[0102] v. Based on the parameter update formula in the gradient descent method, the parameter assignment vector for this round is equal to the difference between the parameter assignment vector for the previous round and the gradient adjustment vector for this round. The gradient adjustment vector for this round is the product of the user-entered learning rate and the gradient vector for this round. Define the parameter assignment table for this iteration of the deep learning model to store the new iteration round and the new assignment vectors for all basic component parameter groups in the deep learning model. As shown in lines 31-33 of the SQL statement example, the deep learning model parameter assignment table for this iteration, params_this_iteration, is defined based on the deep learning model parameter assignment table params_last_iteration from the previous iteration, the gradient accumulation temporary table grad_sum, and the training hyperparams assignment table hyperparams. It is used to store the updated values ​​of each parameter group in the deep learning model after this iteration.

[0103] Step 5: After defining the deep learning model parameter assignment table consisting of the specified number of iterations, define the final deep learning model parameter assignment table to store the deep learning model parameters generated by the last iteration. As shown in lines 35-36 of the SQL statement example, the final deep learning model parameter assignment table, final_params, is defined based on the corresponding assignment records in the deep learning model parameter assignment table, params_per_iteration, for the last iteration. It is used to store the updated values ​​of each parameter group in the deep learning model after the last iteration.

[0104] Step 6: Save the training request ID and the deep learning model parameters stored in the final deep learning model parameter assignment table in the deep learning model table. As shown in lines 39-40 of the SQL statement example, the deep learning model table aisql.dl_model_params stores the training request ID and the deep learning model parameters stored in the final deep learning model parameter assignment table final_params. The deep learning model parameters are stored in the form of a vector array, where each element is a vector and corresponds to a parameter group of the deep learning model.

[0105] The training phase of the present technical solution saves the intermediate data generated by the deep learning model during the training phase in the form of temporary tables. Relying on the efficient management capabilities of temporary tables that combine main memory and disk in modern databases, the training process of the deep learning model can still be efficiently executed when the size of the training set far exceeds the physical memory capacity. The temporary tables generated during the training phase can be automatically deleted after the model training is completed, thereby avoiding the temporary data generated during the training process from occupying the physical storage space of the database. In addition, considering that a single form in the database can generally only define thousands of fields, the technical solution of the present invention introduces a vector type field to store the parameter group of the basic components of the deep learning model, ensuring that a single form with a limited total number of fields in the database can also store all the parameters of a large number of basic components. Therefore, the technical solution of the present invention is suitable for the training of deep learning models containing thousands of basic components.

[0106] The present disclosure also provides an electronic device, Figure 4 A schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure is shown. Figure 4 The electronic device 600 shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0107] like Figure 4 As shown, electronic device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for system operation are also stored in RAM 603. CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0108] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 608 including devices such as a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read from the removable media can be installed in the storage section 608 as needed.

[0109] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for executing the methods illustrated in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 609 and / or installed from removable media 611. When executed by the central processing unit (CPU) 601, the computer program performs the aforementioned functions defined in the embodiments of the present application.

[0110] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, and the units described may also be provided in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0113] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs, each of which includes instructions that, when executed by an electronic device, cause the electronic device to implement the methods described in the above embodiments.

[0114] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0115] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A training method for a deep learning model, characterized in that: include: preparation phase and training phase; The preparation phase introduces corresponding user-defined functions for vector and / or matrix operations in the calculation formulas of different basic components of the deep learning model that cannot be implemented by standard SQL functions, encapsulates them into database plug-ins and integrates them into the database management system. In addition, the preparation phase constructs a deep learning model table for storing the training request ID input by the user and the corresponding deep learning model parameters obtained during the training phase. The training phase generates and saves the parameters of the corresponding deep learning model according to the training request input by the user. The training request includes the training request ID, the network structure of the deep learning model, the training hyperparameter assignments and the training set; The training phase consists of a conversion phase and an execution phase; The conversion step determines the basic components of the deep learning model input by the user, obtains a directed acyclic graph of the basic components, and outputs an SQL statement; each basic component of the deep learning model input by the user has a pre-agreed operation process and corresponds to a parameter subset of the deep learning model input by the user. The operation process of the basic component is a calculation formula, which is constructed based on the standard SQL built-in functions and the user-defined functions introduced in the preparation stage; the training hyperparameters input by the user include the number of iterations and the learning rate; The SQL statements output by the conversion step are defined step by step as follows: First, define the construction process of the training set temporary table; Second, define the construction process of the training hyperparameter assignment table; Third, define the construction process of the temporary tables involved in each round of iteration. The temporary tables involved in each round of iteration include: a temporary table composed of the deep learning model parameter assignment records obtained in each round of iteration, a basic component output temporary table obtained in sequence according to the forward traversal order of the directed acyclic graph of the basic components, a basic component gradient temporary table obtained in sequence according to the backward traversal order of the directed acyclic graph of the basic components, and a basic component cumulative gradient temporary table obtained by summarizing all records of the training set; Fourth, defining the construction process of the final parameter assignment table of the deep learning model input by the user; Finally, define the storage process of the final parameter assignment table of the deep learning model input by the user; The execution link executes the SQL statement output by the conversion link in the database management system to obtain the final parameter assignment table of the deep learning model input by the user, and saves the final parameter assignment table to the deep learning model table constructed in the preparation stage.

2. The deep learning model training method according to claim 1, characterized in that The obtaining of the directed acyclic graph of the basic components includes: A directed acyclic graph is constructed with each basic component of the deep learning model input by the user as a node, and node A has an arc pointing to node B if and only if an output object of the basic component corresponding to node A is an input object of the basic component corresponding to node B.

3. The deep learning model training method according to claim 1, characterized in that The construction process of defining the temporary table of the training set includes: The training set input by the user is a set of training records, each of which is composed of the assignments of multiple independent variables and the assignment of a single dependent variable. All the independent variable assignments of each training record are synthesized into a vector, which together with the corresponding dependent variable assignment constitutes a tuple, which is saved in the training set temporary table as a record in the training set temporary table.

4. The deep learning model training method according to claim 1, characterized in that The construction process of defining the training hyperparameter assignment table includes: All training hyperparameter values ​​input by the user are collected to form a unique record of the training hyperparameter value table.

5. The deep learning model training method according to claim 1, characterized in that: The construction process of the temporary table involved in each iteration includes: Divide all parameters of each basic component in the network structure of the deep learning model input by the user into multiple parameter groups, each parameter group corresponds to a vector; define a deep learning model parameter assignment table, each field of the deep learning model parameter assignment table corresponds to a parameter group, and the field value is called the assignment vector of the corresponding parameter group. In addition, each record of the deep learning model parameter assignment table is called an assignment record, and the assignment record consists of an iteration round and the assignment vectors of all parameter groups obtained after the iteration round, and is used to store the deep learning model parameters obtained in each iteration round; define the following iterative operation, and the number of rounds of the iterative operation is equal to the number of iteration rounds input by the user: If this is the first round, add an assignment record to the deep learning model parameter assignment table to store the randomly initialized assignment vectors of all basic component parameter groups, and mark the assignment record as the most recent parameter assignment record. Otherwise, directly mark the assignment record of the deep learning model parameters in the previous round as the most recent parameter assignment record. Determine a forward traversal order of the directed acyclic graph of the basic components, ensuring that if a corresponding node of basic component A has an arc pointing to a corresponding node of basic component B, then A is accessed before B; define each basic component output temporary table in sequence according to the forward traversal order, where each field of the basic component output temporary table corresponds to an output object; Determine a backward traversal order of the directed acyclic graph of the basic components, ensuring that if a corresponding node of basic component A has an arc pointing to a corresponding node of basic component B, then B is accessed before A; define each basic component gradient temporary table in sequence according to the backward traversal order, where each field of the basic component gradient temporary table corresponds to a parameter group or an input object, and the value of each field is called a gradient vector of the corresponding parameter group or the corresponding input object; Based on the basic component gradient temporary table generated last in the backward traversal order, a gradient accumulation temporary table is defined to store the summed aggregated value of the parameter group gradient vectors of all basic components in the deep learning model input by the user, where the summed aggregated value of the parameter group gradient vector is a gradient accumulation vector obtained by summing the corresponding parameter group gradient vectors of all training records; According to the parameter update formula in the standard gradient descent method, that is, the parameter group assignment vector of this round is equal to the difference between the parameter group assignment vector of the previous round and the gradient adjustment vector of this round, the assignment record of the parameters of the current round deep learning model is defined. The gradient adjustment vector of the parameter group of this round is the product of the learning rate input by the user and the gradient vector of the parameter group of this round.

6. The deep learning model training method according to claim 1, characterized in that The process of constructing the final parameter assignment table defining the deep learning model input by the user includes: The last record of the deep learning model parameter assignment table is used as the only record of the final parameter assignment table.

7. The deep learning model training method according to claim 1, characterized in that The process of storing the final parameter assignment table defining the deep learning model input by the user includes: The unique record of the final parameter assignment table of the deep learning model input by the user is converted into a parameter group assignment vector array, and together with the training request ID input by the user, is added as a record to the deep learning model table constructed in the preparation stage.

8. The deep learning model training method according to claim 5, characterized in that: Defining each basic component output temporary table in sequence according to the forward traversal order includes: The first basic component output temporary table stores the output vector of the first basic component in the forward traversal, as well as the training set and historical output vectors required for the subsequent traversal process. The content stored in the first basic component output temporary table is defined based on the training set temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table. The subsequent output temporary table of each basic component stores the output vector of the current basic component, as well as the training set and historical output vectors required for the subsequent traversal process. The content stored in each subsequent output temporary table of each basic component is defined based on the previous basic component output temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table.

9. The deep learning model training method according to claim 5, characterized in that: Defining each basic component gradient temporary table in sequence according to the backward traversal order includes: The first basic component gradient temporary table stores the gradient vector of the first basic component in the backward traversal, as well as the training set, historical output vector, and historical gradient vector required for the subsequent traversal process. The content stored in the first basic component gradient temporary table is defined based on the output temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table generated at the end of the forward traversal. The subsequent gradient temporary table of each basic component stores the gradient vector of the current basic component, as well as the training set, historical output vector, and historical gradient vector required for the subsequent traversal process. The content stored in each subsequent gradient temporary table of each basic component is defined based on the gradient temporary table, the most recent parameter assignment record, and the training hyperparameter assignment table of the previous basic component.

Images