Training method for deep learning model, inference method, device, apparatus and medium

Abstract

To provide a training method for a deep learning model, an inference method, a device, an apparatus and a medium.SOLUTION: A training method comprises: determining a target loss for both a first model including a plurality of first parameters and a second model including a plurality of second parameters; adjusting parameter values of the plurality of first parameters and the plurality of second parameters based on the target loss; and, in response to determining that the target loss indicates that the parameter values of at least some of the second parameters included in the second model need to be adjusted, synchronously adjusting the parameter values of corresponding target parameters included in the first model.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to technical fields such as machine learning and deep learning, and specifically to a method for training a deep learning model, a method for inferring a deep learning model, a training device for a deep learning model, an inference device for a deep learning model, an electronic device, a computer-readable storage medium, and a computer program product. [Background technology]

[0002] Artificial intelligence is a field that studies how computers can imitate some of the human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, etc., while AI software technologies mainly include natural language processing technology, computer vision technology, voice recognition technology, machine learning / deep learning, big data processing technology, knowledge graph technology, and other directions.

[0003] The approaches described in this section are not necessarily approaches that have been previously conceived or adopted. Unless otherwise noted, it is not intended that any approach described in this section be considered prior art for inclusion in this section. Similarly, unless otherwise noted, the subject matter addressed in this section is not intended to be admitted as any prior art. Summary of the Invention

[0004] The present disclosure provides a method for training a deep learning model, a method for inferring a deep learning model, a training apparatus for a deep learning model, an inference apparatus for a deep learning model, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, a computer-implemented deep learning model training method is provided, in which a first deep learning model includes a plurality of first parameters, a second deep learning model includes a plurality of second parameters, the plurality of second parameters are initialized to parameter values ​​of a plurality of target parameters corresponding to the plurality of second parameters in the plurality of first parameters, and a number of the plurality of second parameters is less than the plurality of first parameters, the training method includes determining target losses of the first deep learning model and the second deep learning model, and adjusting the parameter values ​​of the plurality of first parameters and the plurality of second parameters based on the target losses to obtain the first deep learning model and the second deep learning model after training. synchronously adjusting parameter values ​​of second parameters included in the second deep learning model corresponding to at least some of the target parameters in response to determining that the target loss indicates that parameter values ​​of at least some of the target parameters included in the first deep learning model need to be adjusted; and synchronously adjusting parameter values ​​of target parameters included in the second deep learning model corresponding to at least some of the second parameters included in the first deep learning model in response to determining that the target loss indicates that parameter values ​​of at least some of the second parameters included in the second deep learning model need to be adjusted.

[0006] According to another aspect of the present disclosure, there is provided a computer-implemented deep learning model inference method, in which a first deep learning model and a second deep learning model are obtained by training using the above training method, and the inference method includes: using the second deep learning model to sequentially generate a plurality of predicted tokens; using the first deep learning model to generate a reliability of each of the plurality of predicted tokens based on the plurality of predicted tokens; checking the plurality of predicted tokens based on a generation order of the plurality of predicted tokens to obtain an inference result, in response to determining that the reliability of a currently checked predicted token is lower than a predetermined threshold, using the first deep learning model to generate a corrective token at a position of the predicted token and replace the predicted token as a checked predicted token; in response to determining that the predetermined token generation condition is satisfied, using the second deep learning model to sequentially generate new predicted tokens from a position next to the corrective token based on the corrective token, and using the first deep learning model to generate and check a reliability of the newly generated predicted token; and obtaining an inference result based on the checked predicted token.

[0007] According to another aspect of the present disclosure, there is provided an apparatus for training a deep learning model, in which a first deep learning model includes a plurality of first parameters, a second deep learning model includes a plurality of second parameters, the plurality of second parameters are initialized to parameter values ​​of a plurality of target parameters in the plurality of first parameters, and a number of the plurality of second parameters is less than the plurality of first parameters, the training apparatus includes: a determination unit configured to determine a target loss of the first deep learning model and the second deep learning model; and a parameter adjustment unit configured to adjust parameter values ​​of the plurality of first parameters and the plurality of second parameters based on the target loss to obtain a first deep learning model and a second deep learning model after training, and a parameter adjustment unit including: a first parameter adjustment subunit configured to synchronously adjust parameter values ​​of second parameters corresponding to at least some of the target parameters included in the second deep learning model in response to determining that the target loss indicates that parameter values ​​of at least some of the target parameters included in the second deep learning model need to be adjusted; and a second parameter adjustment subunit configured to synchronously adjust parameter values ​​of target parameters corresponding to at least some of the second parameters included in the first deep learning model in response to determining that the target loss indicates that parameter values ​​of at least some of the second parameters included in the second deep learning model need to be adjusted.

[0008] According to another aspect of the present disclosure, there is provided an inference device for a deep learning model, in which a first deep learning model and a second deep learning model are obtained by training using the training device, and the inference device includes a first generation unit configured to sequentially generate a plurality of predicted tokens using the second deep learning model, a second generation unit configured to generate a reliability of each of the plurality of predicted tokens based on the plurality of predicted tokens using the first deep learning model, and a check unit configured to check the plurality of predicted tokens based on a generation order of the plurality of predicted tokens to obtain an inference result, and a check unit configured to check the reliability of the currently checked predicted token based on a generation order of the plurality of predicted tokens to obtain an inference result, and the checking unit includes a replacement subunit configured to, in response to determining that the predicted token is lower than the predicted token, generate a corrective token at the position of the predicted token using a first deep learning model, and replace the predicted token as a checked predicted token; a generation subunit configured to, in response to determining that a predetermined token generation condition is satisfied, use a second deep learning model to sequentially generate a new predicted token from a position next to the corrective token based on the corrective token, and use the first deep learning model to generate and check a confidence level of the newly generated predicted token; and an inference subunit configured to obtain an inference result based on the checked predicted token.

[0009] According to another aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively coupled to the at least one processor, where the memory stores instructions executable by the at least one processor, which instructions, when executed by the at least one processor, enable the at least one processor to perform the method described above.

[0010] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, where the computer instructions are used to cause a computer to perform the above-described method.

[0011] According to another aspect of the present disclosure there is provided a computer program product comprising a computer program which, when executed by a processor, implements the above method.

[0012] According to one or more embodiments of the present disclosure, the present disclosure first determines the target losses of a first deep learning model and a second deep learning model constructed based on some parameters of the first deep learning model, and determines the parameters of the two models that need to be adjusted based on the target losses, and then synchronously adjusts the parameter values ​​of the corresponding parameters in the other model based on the corresponding relationship between the parameters of the two models, thereby realizing the creation of multiple models with different levels in one training, and meeting the needs of various scenarios, various performances, and various target effects. In addition, by training the first deep learning model and the second deep learning model as a whole, knowledge and information can be transferred in both directions during the training process, which can accelerate the convergence of the models and provide both of the two models with better predictive capabilities.

[0013] It should be understood that the subject matter described in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the disclosure will be readily apparent from the following specification. [Brief description of the drawings]

[0014] The drawings illustratively illustrate the examples and constitute a part of the specification and, together with the textual description of the specification, are used to explain exemplary embodiments of the examples. The illustrated examples are for purposes of illustration and not for limiting the scope of the claims. In all the drawings, the same reference numerals refer to similar, but not necessarily identical, elements. [Figure 1] 1 is a schematic diagram of an example system capable of implementing various methods described herein, according to embodiments of the present disclosure. [Diagram 2] 1 is a flowchart of a method for training a deep learning model according to an embodiment of the present disclosure. [Diagram 3] 1 is a schematic diagram of a first deep learning model and a second deep learning model according to an embodiment of the present disclosure. [Figure 4] 1 is a flowchart of a process for determining target losses for a first deep learning model and a second deep learning model according to an embodiment of the present disclosure. [Diagram 5] 1 is a flowchart of an inference method for a deep learning model according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a structural block diagram of a deep learning model training apparatus according to an embodiment of the present disclosure. [Figure 7] FIG. 2 is a structural block diagram of an inference device for a deep learning model according to an embodiment of the present disclosure. [Figure 8] FIG. 1 is a structural block diagram of an example electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, exemplary embodiments of the present disclosure will be described in conjunction with the drawings, and in order to facilitate understanding, various details of the embodiments of the present disclosure will be included therein, but they should be regarded as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for clarity and conciseness, the following description omits the description of known functions and structures.

[0016] In this disclosure, unless otherwise specified, the terms "first," "second," and the like, used to describe various elements are not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some examples, a first element and a second element may refer to the same instance of the element, or in some cases, may refer to different instances based on the contextual description.

[0017] The terms used in the description of various examples of the present disclosure are intended only to describe specific examples and are not intended to be limiting. Unless otherwise clearly indicated by the context, an element may be one or more, unless the number of elements is specifically limited. In addition, the term "and / or" used in the present disclosure covers any one of the listed items and all possible combinations.

[0018] In the related art, different deep learning models are trained independently, resulting in low training efficiency.

[0019] In order to solve the above problem, the present disclosure first determines the target losses of a first deep learning model and a second deep learning model constructed based on some parameters of the first deep learning model, and determines the parameters of the two models that need to be adjusted based on the target losses, and then synchronously adjusts the parameter values ​​of the corresponding parameters in the other model based on the corresponding relationship between the parameters of the two models, thereby realizing the creation of multiple models with different levels in one training, and meeting the needs of various scenarios, various performances, and various target effects. In addition, by training the first deep learning model and the second deep learning model as a whole, knowledge and information can be transferred in both directions during the training process, which can accelerate the convergence of the models and provide both of the two models with better predictive capabilities.

[0020] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0021] 1 illustrates a schematic diagram of an example system 100 in which various methods and apparatus described herein may be implemented, according to an embodiment of the present disclosure. With reference to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0022] In an embodiment of the present disclosure, the server 120 is operable to perform one or more services or software applications of the methods of the present disclosure.

[0023] In some embodiments, server 120 may further provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and / or 106 in a Software as a Service (SaaS) model.

[0024] In the arrangement shown in Figure 1, the server 120 may include one or more assemblies that implement the functions performed by the server 120. These assemblies may include software assemblies, hardware assemblies, or a combination thereof that may be executed by one or more processors. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 may utilize the services provided by these assemblies by, in turn, utilizing one or more client applications to interact with the server 120. It should be understood that a variety of different system configurations are possible and may differ from system 100. As such, Figure 1 is an example of a system for implementing various methods described herein and is not intended to be limiting.

[0025] A user may use client devices 101, 102, 103, 104, 105 and / or 106 to perform man-machine interaction. The client devices may provide an interface to the user of the client device through which interaction with the client device may occur. The client devices may also output information to the user via the interface. Although only six client devices are shown in FIG. 1, one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0026] The client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computing devices, such as portable handheld devices, general purpose computers (e.g., personal computers or laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (e.g., GOOGLE Chrome OS), etc., and may include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android, etc. Portable handheld devices may include cellular phones, smartphones, tablet PCs, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, etc. The client devices may run a variety of different applications, such as various Internet-related applications, communication applications (eg, email applications), Short Message Service (SMS) applications, and may use a variety of communication protocols.

[0027] Network 110 may be any type of network known to those skilled in the art that may use any one of a number of available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.) to support data communications. By way of example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token loop, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0028] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may also include one or more virtual machines running a virtual operating system, or other computing architecture involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain virtual storage devices for the server). In various embodiments, server 120 may execute one or more services or software applications that provide the functionality described below.

[0029] The computing units in server 120 may run one or more operating systems, including any of the operating systems listed above, as well as any commercially available server operating system. Server 120 may further run any one of a variety of additional server applications and / or mid-tier applications, including an HTTP server, an FTP server, a CGI server, a JAVA server, a database server, etc.

[0030] In some embodiments, server 120 may include one or more applications for integrating with analytics data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may further include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0031] In some embodiments, the server 120 may be a server of a distributed system or a server incorporating blockchain. The server 120 may be a cloud server, or a smart cloud computing server or a smart cloud host equipped with artificial intelligence technology. The cloud server is a host product in the cloud computing service system, which solves the deficiencies of traditional physical host and virtual private server (VPS) services, such as high management difficulty and low business scalability.

[0032] System 100 may further include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of databases 130 may be used to store information for audio and video files. Databases 130 may be located in a variety of locations. For example, a database used by server 120 may be local to server 120 or may be separate from server 120 and in communication with server 120 over a network or dedicated connection. Databases 130 may be of various types. In some embodiments, a database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from the database and from the database in response to instructions.

[0033] In some embodiments, one or more of databases 130 may also be used by an application to store data for the application. The databases used by the application may be different types of databases, such as a key-value repository, an object repository, or a general-purpose repository supported by a file system.

[0034] The system 100 of FIG. 1 can be configured and operated in a variety of ways to accommodate the various methods and apparatus described in accordance with this disclosure.

[0035] According to one aspect of the present disclosure, a computer-implemented method for training a deep learning model is provided, in which a first deep learning model includes a plurality of first parameters, and a second deep learning model includes a plurality of second parameters, and the plurality of second parameters are initialized to parameter values ​​of a plurality of target parameters corresponding to the plurality of second parameters in the plurality of first parameters. The number of the plurality of second parameters is less than the number of the plurality of first parameters.

[0036] FIG. 2 illustrates a flowchart of a method 200 for training a deep learning model according to an embodiment of the present disclosure. The method 200 includes a step S201 of determining a target loss for a first deep learning model and a second deep learning model; and a step S202 of adjusting parameter values ​​of a plurality of first parameters and a plurality of second parameters based on the target loss to obtain a first deep learning model and a second deep learning model after training, the step S202 including: in response to determining that the target loss indicates that a parameter value of at least a portion of the plurality of target parameters included in the first deep learning model needs to be adjusted, a step S2021 of synchronously adjusting a parameter value of a second parameter corresponding to at least a portion of the target parameters included in the second deep learning model; and in response to determining that the target loss indicates that a parameter value of at least a portion of the second parameters included in the second deep learning model needs to be adjusted, a step S2022 of synchronously adjusting a parameter value of a target parameter corresponding to at least a portion of the second parameters included in the first deep learning model.

[0037] Thus, for the first deep learning model and the second deep learning model constructed based on some parameters of the first deep learning model, the target losses of both are first determined, and the parameters of the two models that need to be adjusted based on the target losses are determined, and then the parameter values ​​of the corresponding parameters in the other model are synchronously adjusted based on the corresponding relationship between the parameters of the two models, so as to realize the creation of multiple models with different levels in one training, and to meet the needs of various scenarios, various performances, and various target effects. In addition, by training the first deep learning model and the second deep learning model as a whole, knowledge and information can be transferred in both directions during the training process, which can accelerate the convergence of the models and provide both of the two models with better predictive capabilities.

[0038] In some embodiments, the first deep learning model and the second deep learning model may be based on various types of deep learning models, such as convolutional neural networks, recurrent neural networks, multi-layer perceptrons, fully connected networks, Transformer structures or Transformer-like structures (extensions of each type of Transformer structure), or other types of deep learning models. The first deep learning model and the second deep learning model may be end-to-end complete models or may be parts of models for specific tasks, not limited here. It should be noted that the first deep learning model and the second deep learning model belong to the same type of model.

[0039] In some embodiments, a first model architecture and a second model architecture may be used to describe the model architecture and model configuration information of the first deep learning model and the second deep learning model, including, for example, the number of layers the model includes, the number of intermediate dimensions of the model layers, etc. The scale of the second model architecture is smaller than the first model architecture, i.e., the model scale / size / amount of parameters of the second deep learning model is smaller than the first deep learning model.

[0040] In some embodiments, based on the number of the second parameters in the plurality of first parameters, some of the parameters (i.e., a plurality of target parameters) may be selected to be used for constructing the second deep learning model. In the plurality of first parameters, a plurality of target parameters may be selected in the same number as the plurality of second parameters, and the plurality of second parameters may be initialized to the parameter values ​​of the plurality of target parameters. In other words, the parameter value of each second parameter in the second deep learning model after initialization is the same as the parameter value of the target parameter corresponding to the second parameter in the first deep learning model. Thereafter, in the process of training the first deep learning model and the second deep learning model, the parameter values ​​of the plurality of target parameters in the first deep learning model and the parameter values ​​of the plurality of second parameters in the second deep learning model are always maintained to be consistent, as will be introduced below.

[0041] In some embodiments, the multiple target parameters may be obtained by randomly selecting among the multiple first parameters based on the number of the multiple second parameters, or may combine several heuristic policies, such as uniform distribution, or adopt a feature selection / feature engineering method to select among the multiple first parameters to obtain multiple target parameters that can achieve a better training effect.

[0042] According to some embodiments, the first deep learning model may include a first layer of a first number of layers, and the first layers of the first number of layers may each include a first intermediate dimension of a first intermediate dimensionality. The second deep learning model may include a second layer of a second number of layers, and the second layers may each include a second intermediate dimension of a second intermediate dimensionality. The second number of layers of the second deep learning model may be less than the first number of layers of the first deep learning model, and the second intermediate dimensionality of the second layer in the second deep learning model may be less than the first intermediate dimensionality of the first layer in the first deep learning model.

[0043] The plurality of target parameters may include a first parameter in at least one target intermediate dimension for each of the at least one target layer, the at least one target layer may be selected in a first layer of the first number of layers based on a second number of layers, and the at least one target intermediate dimension may be selected in a first intermediate dimension of the first number of intermediate dimensionalities based on the second number of intermediate dimensionalities.

[0044] As a result, the above method enables selection from a plurality of first parameters contained in the first deep learning model to obtain a plurality of target parameters that satisfy the parameter amount of the second deep learning model, while maintaining the overall model architecture of the first deep learning model as much as possible, avoiding excessive concentration of the selected parameters, and improving the predictive ability of the second deep learning model.

[0045] According to some embodiments, the first deep learning model and the second deep learning model may be based on a Transformer structure or a class Transformer structure. The first deep learning model may include a first hidden dimension of a first number of hidden dimensions, and a first layer of the first number of layers may include a first attention head of a first number of attention heads, respectively. The second deep learning model may include a second hidden dimension of a second number of hidden dimensions, and a second layer of the second number of layers may include a second attention head of a second number of attention heads, respectively.

[0046] The target parameters may include first parameters corresponding to the target hidden dimensions and may include a first parameter in at least one target attention head. The target hidden dimensions may be selected in a first hidden dimension of the first number of hidden dimensions based on a second number of hidden dimensions, and the at least one target attention head may be selected in a first attention head of the first number of attention heads based on the second number of attention heads.

[0047] As a result, the above method can select from a plurality of first parameters included in the first deep learning model to obtain a plurality of target parameters that satisfy the parameter amount of the second deep learning model, while maintaining the overall model architecture of the first deep learning model based on the Transformer structure as much as possible, avoiding excessive concentration of the selected parameters, and improving the predictive ability of the second deep learning model.

[0048] In some embodiments, the first layer may be, for example, a Transformer layer, and the first intermediate dimension may be, for example, an intermediate dimension in Feed Forward Networks (FFN) in the Transformer layer.

[0049] 3 shows a schematic diagram of a first deep learning model and a second deep learning model according to an embodiment of the present disclosure. As shown in FIG. 3, the first deep learning model 300 includes an embedding input 302, a hidden dimension is 6, and includes three Transformer layers, each of which includes a set of four attention heads 304 and a feedforward neural network 306, and the intermediate dimension of the feedforward neural network is 8. The second deep learning model 310 includes an embedding input 312, a hidden dimension is 3, and includes two Transformer layers, each of which includes a set of two attention heads 314 and a feedforward neural network 316, and the intermediate dimension of the feedforward neural network is 4.

[0050] As can be seen from the figure, the multiple target parameters selected in the multiple first parameters include the first parameters in four intermediate dimensions, 1, 2, 4, and 5, of the first Transformer layer, and the first parameters in four intermediate dimensions, 3, 5, 7, and 8, of the third Transformer layer. The multiple target parameters include first parameters corresponding to three hidden dimensions, 1, 4, and 6, and include first parameters in two attention heads, 2 and 4, of the first Transformer layer, and include first parameters in two attention heads, 1 and 2, of the third Transformer layer. Based on these target parameters, a second deep learning model can be constructed.

[0051] In some embodiments, the second model architecture (i.e., the number of layers, the number of intermediate dimensions, etc. included in the second deep learning model) may be determined based on the first model architecture and a preset proportion. In one exemplary embodiment, the preset proportion may be 50%, in which case the second number of layers of the second deep learning model may be half the first number of layers of the first deep learning model, and the second number of intermediate dimensions of the second deep learning model may be half the first number of intermediate dimensions of the first deep learning model. As can be understood, the second model architecture may be determined based on a number of preset hyperparameter values, which are not limited herein.

[0052] According to some embodiments, the at least one target layer, the at least one target intermediate dimension, the plurality of target hidden dimensions and / or the at least one target attention head may be randomly selected.

[0053] According to some embodiments, the first deep learning model and the second deep learning model may both be configured to perform at least one of the following tasks: a text processing task, an image processing task, and a voice processing task. The first deep learning model and the second deep learning model may be configured to perform the same task. Although the two models perform the same task, the two models have different parameter amounts, so that the inference speeds and accuracies of the two models are different. Therefore, the needs of various scenarios, various performances, and various target effects can be met.

[0054] According to some embodiments, the first deep learning model may be a pre-trained large model. The pre-trained large model has good performance when processing various text processing tasks, image processing tasks and voice processing tasks, but its inference cost is high and its deployment difficulty is high. If another "small" model is trained independently, the knowledge acquired by the large model cannot be utilized, and the process of training the "small" model cannot further improve the ability of the large model. Therefore, in the pre-trained large model, a plurality of target parameters are obtained by a selection method, and a "small" model is further constructed based on the plurality of target parameters, and the large model and the "small" model are trained as a whole, so that knowledge and information can be transferred in both directions, the convergence of the model can be accelerated, and both models can have better prediction ability.

[0055] In some embodiments, the target parameters can be selected for the first parameters by initializing one or more fixed mask matrices. The mask matrix only includes 0 and 1, where 0 represents training only the first deep learning model, and 1 represents training both the first deep learning model and the second deep learning model. For a deep learning model based on a Transformer structure, the mask matrix may include a mask matrix of a hidden dimension, a mask matrix of a feedforward neural network, a mask matrix of an attention head, and a mask matrix of a Transformer layer. As can be understood, for different types of deep learning models, the mask matrix may include mask matrices corresponding to different model structures, and is not limited herein.

[0056] In some embodiments, in step S201, a target loss in the training phase of the first deep learning model and the second deep learning model may be determined. As can be understood, various samples, various loss functions, and various training policies may be adopted to calculate the target loss, which are not limited herein.

[0057] According to some embodiments, FIG. 4 illustrates a flowchart of a process 400 for determining target losses of a first deep learning model and a second deep learning model according to embodiments of the present disclosure. The process 400 may be used to implement step S201 in the method 200. The process 400 may include a step S401 of determining a first loss of the first deep learning model for a first sample, a step S402 of determining a second loss of the second deep learning model for a second sample, and a step S403 of determining a target loss based on the first loss and the second loss.

[0058] In this way, by determining the first loss and the second loss of the first deep learning model and the second deep learning model for the first sample and the second sample, respectively, and then obtaining a target loss based on the two, the parameters of the first deep learning model and the second deep learning model can be adjusted simultaneously in a single training session, thereby realizing synchronous training of the two models.

[0059] In some embodiments, a batch training method may be adopted to train the first deep learning model and the second deep learning model, and the first sample and the second sample may both be batch data. In one exemplary embodiment, the first deep learning model may be trained using one batch of data (first sample), and the second deep learning model may be trained using another batch of data (second sample), and an overall target loss may be obtained based on the two losses, thereby realizing joint training for the first deep learning model and the second deep learning model.

[0060] According to some embodiments, the first sample and the second sample may be the same. By making the first deep learning model and the second deep learning model use completely the same training sample, the second deep learning model can learn the attention matrix and probability distribution of the top layer of the first deep learning model, so that the training results of the second deep learning model and the first deep learning model are more fitting, and the joint training of the two models is accelerated, and the prediction ability of the two models after training is improved.

[0061] In some embodiments, in steps S401 and S402, a corresponding loss function may be selected as necessary to obtain the first loss and the second loss. In step S403, the first loss and the second loss may be added to obtain the target loss. In addition to the above methods, the target loss may be obtained based on the first loss and the second loss by other methods, which are not limited here.

[0062] In some embodiments, in step S202, the method may first determine which of the first and second parameters need to be adjusted based on the target loss by backpropagation, and determine the corresponding adjustment amount. Further, in step S2021, in response to determining that the target loss indicates that a parameter value of at least a portion of the plurality of target parameters included in the first deep learning model needs to be adjusted, the method may adjust a parameter value of the portion of the target parameters in the first deep learning model based on the adjustment amount determined for the portion of the target parameters, and synchronously adjust a parameter value of a second parameter corresponding to the portion of the target parameters in the second deep learning model based on the similar adjustment amount. If the target loss indicates that a parameter value of a first parameter other than the plurality of target parameters in the first deep learning model needs to be adjusted, the method may adjust a parameter value of the first parameter in the first deep learning model based only on the adjustment amount determined for the first parameter, without adjusting a parameter of the second deep learning model.

[0063] Similarly, in step S2022, in response to determining that the target loss indicates that parameter values ​​of at least a portion of the second parameters among the plurality of second parameters included in the second deep learning model need to be adjusted, adjust the parameter values ​​of the second parameters of the portion in the second deep learning model based on the adjustment amount determined for the second parameters of the portion, and synchronously adjust the parameter value of a target parameter (i.e., the first parameter) in the first deep learning model corresponding to the second parameters of the portion based on the similar adjustment amount.

[0064] In some embodiments, two adjustment amounts may be determined for the same parameter in the same training (although in the method introduced below, there may be more adjustment amounts), and one of the adjustment amounts may be used to synchronously adjust the parameter values ​​of the corresponding parameters in the two models, or two adjustment amounts may be used to synchronously adjust the parameter values ​​of the corresponding parameters in the two models, which is not limited here.

[0065] In some embodiments, the method 200 may be extended to training for more models. The model to be trained may further include a third deep learning model. The third deep learning model includes a plurality of third parameters, and the plurality of third parameters are initialized to parameter values ​​of a plurality of shared parameters corresponding to the plurality of third parameters in the plurality of target parameters, where the number of the plurality of third parameters is less than the plurality of second parameters. In other words, the plurality of target parameters corresponding to the plurality of second parameters are a subset of the plurality of first parameters, and the plurality of shared parameters corresponding to the plurality of third parameters are a subset of the plurality of target parameters. The structure and operation of the third model may refer to the above description for the second model.

[0066] In the training stage, the target loss of three models may be determined first, and the parameter value of the parameter of each model may be adjusted according to the target loss. Specifically, in response to determining that the target loss indicates that the parameter value of one of the parameters in one of the models needs to be adjusted, if the other two models include a parameter corresponding to the parameter, the parameter value of the corresponding parameter is adjusted synchronously. In this manner, three models with different levels can be obtained.

[0067] As can be understood, with reference to the above description, the method 200 may be extended to training four or more models, and the description is omitted here. The method of the present disclosure can extend the number of models to be trained without increasing the training cost, and since each round of training trains a different model, extending the number of models does not increase the number of training rounds.

[0068] According to another aspect of the present disclosure, a computer-implemented method for inferring a deep learning model is provided. Figure 5 shows a flowchart of a method 500 for inferring a deep learning model according to an embodiment of the present disclosure. The first and second deep learning models according to the method 500 may be obtained by training using the method 200 described above. The method 500 includes a step S501 of sequentially generating a plurality of predicted tokens using a second deep learning model; a step S502 of generating a reliability of each of the plurality of predicted tokens based on the plurality of predicted tokens using a first deep learning model; and a step S503 of checking the plurality of predicted tokens based on a generation order of the plurality of predicted tokens to obtain an inference result, in which in response to determining that the reliability of the currently checked predicted token is lower than a predetermined threshold, a step S5031 of generating a corrective token at the position of the predicted token using the first deep learning model and replacing the predicted token as a checked predicted token, in response to determining that the predetermined token generation condition is satisfied, a step S5032 of sequentially generating a new predicted token from a position next to the corrective token using the second deep learning model based on the corrective token, and generating and checking the reliability of the newly generated predicted token using the first deep learning model; and a step S5033 of obtaining an inference result based on the checked predicted token.

[0069] Prediction of a deep learning model generally includes two modes, one is a decoding mode, which requires prediction for each token, and takes longer. The other is a check mode, which is to give multiple generated tokens and obtain the confidence of each token in the result (i.e., the probability distribution of the top layer). The check mode can process the entire generation sequence in one go, so it takes significantly less time than the decoding mode. The present disclosure combines the decoding process of a smaller second deep learning model with the check process of a larger first deep learning model, which can quickly obtain high-quality results and further improve performance and inference effect.

[0070] In some embodiments, in step S501, the second deep learning model may be used in a decoding mode to sequentially generate multiple predicted tokens. Because the scale of the second deep learning model is small, the decoding mode can be used to quickly obtain a prediction result sequence, i.e., multiple predicted tokens.

[0071] In one exemplary embodiment, the input token sequence may be input to a second deep learning model to obtain a first predicted token output from the model, and the input token sequence and the first predicted token may be further input to the model to obtain a second predicted token output from the model, and so on until the model outputs an [End] token indicating the end, or until the number of tokens output reaches a preset number. It can be understood that in addition to the above manner, the second deep learning model may be used in other manners to sequentially generate multiple predicted tokens, and is not limited thereto.

[0072] In some embodiments, in step S502, the first deep learning model may be used in a check mode to directly generate confidence values ​​for each of the multiple predicted tokens based on the multiple predicted tokens. Because the first deep learning model is large in scale, it may take a long time to obtain a prediction result sequence when using the decode mode, while the check mode may quickly obtain the top-layer probability distributions corresponding to the multiple predicted tokens generated by the second deep learning model.

[0073] In some embodiments, in step S503, the predicted tokens may be checked based on the generation order of the predicted tokens, and a corresponding operation may be performed based on the checking result.

[0074] In some embodiments, step S5031 indicates that in response to determining that the confidence of the currently checked predicted token is lower than the preset threshold, the first larger deep learning model determines that the currently checked predicted token is inaccurate. As such, the first deep learning model may be used in a decoding mode to generate a corrective token at the location of the predicted token to replace the predicted token generated by the second deep learning model. The corrective token may be used as the checked predicted token in generating the final inference result.

[0075] In one exemplary embodiment, in response to determining that the confidence level of the currently checked predicted token is lower than a preset threshold, all predicted tokens prior to the currently checked predicted token may be input to a first deep learning model to obtain a token for the current position, i.e., a corrective token, generated by the first deep learning model.

[0076] In some embodiments, in response to determining that the preset token generation condition is satisfied in step S5032, the second deep learning model is used in a decoding mode to sequentially generate at least one predicted token again from the position next to the corrective token based on the corrective token. Thereafter, when the generation of the at least one predicted token is completed or another generation stopping condition is satisfied, the first deep learning model is used in a check mode to generate a confidence value for the at least one predicted token based on the newly generated at least one predicted token, and further check the at least one predicted token. The above process may be repeated until checking of all predicted tokens is completed.

[0077] According to some embodiments, the preset token generation condition may include the number of generated tokens being less than a preset number. In other words, a single preset number may be set, and predicted tokens may be generated and checked in units of the preset number, and after the checking is completed, the next preset tokens may be generated and checked in units of the preset number until the generation process is completed, for example, until a highly reliable [End] token is generated.

[0078] This allows the scheme to control the upper limit of the length of predicted token sequences being generated and checked, thereby avoiding performance degradation caused by generating and checking predicted token sequences that are too long.

[0079] According to some embodiments, the preset token generation condition may include a corrective token indicating that the token generation is not finished. In one exemplary embodiment, if the corrective token is not an [End] token, the preset generation condition is not satisfied and a second deep learning model is required to generate a new set of predicted tokens in sequence.

[0080] According to some embodiments, the step S503 of checking the multiple predicted tokens to obtain an inference result based on the generation order of the multiple predicted tokens may further include retaining the currently checked predicted token as a checked predicted token in response to determining that the reliability of the currently checked predicted token is equal to or greater than a preset threshold.

[0081] In this way, the above method can ensure that all checked predicted tokens that are ultimately retained have a high degree of reliability, thereby ensuring the quality of the generated inference results.

[0082] In some embodiments, in step S5033, content corresponding to each checked prediction token, such as text, image, voice, etc., may be determined, and these contents may then be matched to obtain corresponding inference results.

[0083] This allows high-quality inference results to be obtained quickly by using a smaller-scale second deep learning model to generate a predicted token sequence based on a decoding mode, using a larger-scale first deep learning model to check each generated predicted token based on a check mode, and if the check does not pass (e.g., is less than the confidence level), using the second deep learning model to generate a corrective token for the current position based on the decoding mode, and using the second deep learning model to generate a new predicted token each time from the next position.

[0084] In some embodiments, the model used for inference further includes a target deep learning model that is larger than the first deep learning model, i.e., the first parameters included in the first deep learning model are a subset of the parameters included in the target deep learning model. In step S5033, after obtaining the check tokens checked by the first deep learning model, the target deep learning model may be used to generate confidences for the check tokens, and the confidences may be used to further check the check tokens. If a check token does not pass the check of the target deep learning model, the target deep learning model may generate a new token at the position to replace the check token, and the second deep learning model may generate a subsequent token after the token, which is then checked again by the first deep learning model, and the target deep learning model may then recheck the result after the check by the first deep learning model.

[0085] In some embodiments, the second deep learning model generates N1 predicted tokens each time, and the generated results are checked and regenerated by the second deep learning model. The results generated and checked by the first deep learning model and the second deep learning model may be regarded as a whole, and each time the first deep learning model and the second deep learning model complete N2 rounds of generation and checking, the obtained N1×N2 checked predicted tokens may be passed to the target deep learning large model for checking and regeneration.

[0086] As can be understood, the above generating and checking scheme can be further extended to four or more models, which is not described here.

[0087] According to another aspect of the present disclosure, there is provided an apparatus for training a deep learning model, wherein a first deep learning model includes a plurality of first parameters, a second deep learning model includes a plurality of second parameters, the plurality of second parameters are initialized to parameter values ​​of a plurality of target parameters in the plurality of first parameters, and the number of the plurality of second parameters is less than the number of the plurality of first parameters.

[0088] FIG. 6 shows a structural block diagram of a deep learning model training apparatus 600 according to an embodiment of the present disclosure. The apparatus 600 includes a determining unit 610 configured to determine a target loss for the first deep learning model and the second deep learning model; and a parameter adjusting unit 620 configured to adjust parameter values ​​of a plurality of first parameters and a plurality of second parameters based on the target loss to obtain a first deep learning model and a second deep learning model after training, the parameter adjusting unit 620 including: a first parameter adjusting subunit 622 configured to synchronously adjust parameter values ​​of second parameters corresponding to at least some of the target parameters included in the second deep learning model in response to determining that the target loss indicates that parameter values ​​of at least some of the target parameters included in the first deep learning model need to be adjusted; and a second parameter adjusting subunit 624 configured to synchronously adjust parameter values ​​of target parameters corresponding to at least some of the second parameters included in the first deep learning model in response to determining that the target loss indicates that parameter values ​​of at least some of the second parameters included in the second deep learning model need to be adjusted.

[0089] It can be understood that the operations of the units 610 to 620 and their sub-units in the apparatus 600 may refer to the above description of steps S201 to S202 and their sub-steps in the method 200, and the description will be omitted here.

[0090] According to some embodiments, the first deep learning model may include a first layer of a first number of layers, and the first layers of the first number of layers may each include a first intermediate dimension of a first intermediate dimensionality.

[0091] The second deep learning model may include a second number of second layers, and each of the second layers may include a second intermediate dimension of the second intermediate dimensionality. The second number of layers may be less than the first number of layers, and the second intermediate dimensionality may be less than the first intermediate dimensionality.

[0092] The plurality of target parameters may include a first parameter in at least one target intermediate dimension for each of the at least one target layer, the at least one target layer may be selected in a first layer of the first number of layers based on a second number of layers, and the at least one target intermediate dimension may be selected in a first intermediate dimension of the first number of intermediate dimensionalities based on the second number of intermediate dimensionalities.

[0093] According to some embodiments, the first deep learning model and the second deep learning model may be based on a Transformer structure or a Transformer-like structure.

[0094] Indicated that the first deep learning model may include a first hidden dimension of a first number of hidden dimensions, and each first layer of the first number of layers may include a first attention head of a first number of attention heads.

[0095] Indicated that the second deep learning model may include a second hidden dimension of a second number of hidden dimensions, and each of the second layers of the second number of layers may include a second attention head of a second number of attention heads.

[0096] The target parameters may include first parameters corresponding to the target hidden dimensions and may include a first parameter in at least one target attention head. The target hidden dimensions may be selected in a first hidden dimension of the first number of hidden dimensions based on a second number of hidden dimensions, and the at least one target attention head may be selected in a first attention head of the first number of attention heads based on the second number of attention heads.

[0097] According to some embodiments, the at least one target layer, the at least one target intermediate dimension, the plurality of target hidden dimensions and / or the at least one target attention head may be randomly selected.

[0098] According to some embodiments, both the first deep learning model and the second deep learning model may be configured to perform at least one of the following tasks: a text processing task, an image processing task, and a voice processing task.

[0099] According to some embodiments, the first deep learning model may be a pre-trained large model.

[0100] According to some embodiments, the determination unit may include a first determination subunit configured to determine a first loss of a first deep learning model for a first sample, a second determination subunit configured to determine a second loss of a second deep learning model for a second sample, and a third determination subunit configured to determine a target loss based on the first loss and the second loss.

[0101] According to some embodiments, the first sample and the second sample may be the same.

[0102] According to another aspect of the present disclosure, there is provided an inference device for a deep learning model, wherein the first deep learning model and the second deep learning model may be obtained by training using the device 600.

[0103] 7 shows a structural block diagram of an inference device 700 of a deep learning model according to an embodiment of the present disclosure. The device 700 includes a first generating unit 710 configured to use a second deep learning model to sequentially generate a plurality of predicted tokens, a second generating unit 720 configured to use the first deep learning model to generate a confidence level for each of the plurality of predicted tokens based on the plurality of predicted tokens, and a checking unit 730 configured to check the plurality of predicted tokens based on a generation order of the plurality of predicted tokens to obtain an inference result, in which in response to determining that the confidence level of the currently checked predicted token is lower than a preset threshold, a checking unit 730 is configured to check the position of the predicted token using the first deep learning model to obtain an inference result. The checking unit 730 includes a replacement subunit 732 configured to generate a correct token and replace the predicted token as a checked predicted token, a generation subunit 734 configured to, in response to determining that a preset token generation condition is satisfied, use a second deep learning model to sequentially generate a new predicted token from the position next to the corrective token based on the corrective token, and use the first deep learning model to generate and check the reliability of the newly generated predicted token, and an inference subunit 736 configured to obtain an inference result based on the checked predicted token.

[0104] It can be understood that the operations of units 710 to 730 and their sub-units in the apparatus 700 may refer to the description of steps S501 to S503 and their sub-steps in the method 500 above, and the description will be omitted here.

[0105] According to some embodiments, the preset token generation condition may include the number of generated tokens being less than a preset number.

[0106] According to some embodiments, the pre-defined Token generation condition may include a corrective token indicating that Token generation is not complete.

[0107] According to some embodiments, the checking unit 730 (not shown) may include a retaining subunit configured to retain the currently checked predicted token as a checked predicted token in response to determining that the reliability of the predicted token is equal to or greater than a preset threshold.

[0108] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved shall all comply with the provisions of relevant laws and regulations and shall not violate public order and morals.

[0109] According to embodiments of the present disclosure, an electronic device, a readable storage medium, and a computer program product are further provided.

[0110] Next, referring to FIG. 8, a structural block diagram of an electronic device 800 that can be used as a server or client of the present disclosure will be described, which is an example of a hardware device that can be applied to each aspect of the present disclosure. The electronic device represents various types of digital electronic computing devices, such as laptop computers, desktop computers, stages, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various types of mobile devices, such as personal digital processing, cellular phones, smartphones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0111] 8, the electronic device 800 includes a computing unit 801, which can perform various suitable operations and processes based on a computer program stored in a read-only memory (ROM) 802 or loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may further store various programs and data required for the operation of the electronic device 800. The computing unit 801, the ROM 802 and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0112] A number of components in the electronic device 800 are connected to an I / O interface 805, and include an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 may be any type of device capable of inputting information to the electronic device 800, and the input unit 806 may receive input numeric or character information and generate key signal input for user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a track board, a track ball, a control lever, a microphone, and / or a remote control. The output unit 807 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 808 may include, but is not limited to, a magnetic disk, an optical disk. The communication unit 809 enables the electronic device 800 to exchange information / data with other devices via a computer network, e.g., the Internet, and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, e.g., a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0113] The computing unit 801 may be various general-purpose and / or dedicated processing assemblies having processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs each of the methods, processes, and / or operations described above. For example, in some embodiments, the methods, processes, and / or operations may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, some or all of the computer program may be loaded and / or installed in the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, it may perform one or more steps of the methods, processes, and / or operations described above. Alternatively, in other embodiments, the computing unit 801 may be configured to perform these methods, processes and / or operations in any other suitable manner (eg, firmware).

[0114] Various embodiments of the systems and techniques described herein above may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: embodied in one or more computer programs that may be executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor, and may receive data and instructions from a storage system, at least one input device, at least one output device, and may transmit data and instructions to the storage system, the at least one input device, and the at least one output device;

[0115] The program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer or other programmable data processing apparatus, so that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The program codes may be entirely executed on a machine, partly executed on a machine, partly executed on a machine and partly executed on a remote machine as a separate software package, or entirely executed on a remote machine or server.

[0116] In the context of this disclosure, a machine-readable medium may be a tangible medium, which may contain or store a program for use in or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include an electrical connection by one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0117] To provide interaction with a user, the systems and techniques described herein may be implemented in a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which a user may provide input to the computer. Other types of devices may also be provided for providing interaction with a user. For example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form (including audio input, voice input, tactile input).

[0118] The systems and techniques described herein may be implemented in a computing system including a background component (e.g., a data server), a middleware component (e.g., an application server), a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or any combination of such background, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0119] The computer system may include a client and a server. The client and the server are generally remote from each other and usually interact with each other via a communication network. The client-server relationship is created by running computer programs in the corresponding computers that have a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0120] It should be understood that steps may be reordered, added, or removed using the various types of flows shown above. For example, each step described in this disclosure may be performed in parallel, sequentially, or in a different order, and is not limited herein, as long as the technical solution disclosed in this disclosure can achieve the desired results.

[0121] Although the embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the scope of the claims and their equivalents after grant. Various elements in the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in a different order from the order described in this disclosure. Furthermore, various elements in the embodiments or examples may be combined in various ways. In essence, with the evolution of technology, many elements described herein may be replaced by equivalent elements appearing later in this disclosure.

Claims

1. A computer-implemented method for training a deep learning model, wherein a first deep learning model includes a plurality of first parameters, and a second deep learning model includes a plurality of second parameters, the plurality of second parameters being initialized to parameter values ​​of a plurality of target parameters in the plurality of first parameters that correspond to the plurality of second parameters, the number of the plurality of second parameters being less than the number of the first parameters, the method comprising: determining a target loss for the first deep learning model and the second deep learning model; Adjusting parameter values ​​of the plurality of first parameters and the plurality of second parameters based on the target loss to obtain a first deep learning model and a second deep learning model after training; synchronously adjusting parameter values ​​of second parameters, which correspond to the at least some of the target parameters, included in the second deep learning model, in response to determining that the target loss indicates that parameter values ​​of at least some of the target parameters included in the first deep learning model need to be adjusted; and and in response to determining that the target loss indicates that parameter values ​​of at least a portion of the second parameters among the plurality of second parameters included in the second deep learning model need to be adjusted, synchronously adjusting parameter values ​​of target parameters included in the first deep learning model that correspond to the at least a portion of the second parameters.

2. Determining a target loss for the first deep learning model and the second deep learning model includes: determining a first loss of the first deep learning model for a first sample; determining a second loss of the second deep learning model for a second sample; and and determining the target loss based on the first loss and the second loss.

3. The method of claim 2 , wherein the first sample and the second sample are the same.

4. The first deep learning model includes a first layer of a first number of layers, each of the first layers of the first number of layers including a first intermediate dimension of a first intermediate dimensionality; The second deep learning model includes a second layer of a second number of layers, and the second layers each include a second intermediate dimension of a second intermediate dimensionality, where the second number of layers is smaller than the first number of layers, and the second intermediate dimensionality is smaller than the first intermediate dimensionality; 2. The method of claim 1 , wherein the plurality of target parameters includes a first parameter in at least one target intermediate dimension for each of the at least one target layer, wherein the at least one target layer is selected in a first layer of the first number of layers based on the second number of layers, and the at least one target intermediate dimension is selected in a first intermediate dimension of the first number of intermediate dimensionality based on the second number of intermediate dimensionality.

5. The first deep learning model and the second deep learning model are based on a Transformer structure or a class Transformer structure; The first deep learning model includes a first hidden dimension of a first number of hidden dimensions, and each of the first layers of the first number of layers includes a first attention head of a first number of attention heads; The second deep learning model includes a second hidden dimension of a second number of hidden dimensions, and the second layers of the second number of layers each include a second attention head of a second number of attention heads; 5. The method of claim 4, wherein the plurality of target parameters include a first parameter corresponding to a plurality of target hidden dimensions and a first parameter at at least one target attention head, wherein the plurality of target hidden dimensions are obtained by selecting a first hidden dimension of the first number of hidden dimensions based on the second number of hidden dimensions, and the at least one target attention head is obtained by selecting a first attention head of the first number of attention heads based on the second number of attention heads.

6. The method of claim 5 , wherein the at least one target stratum, the at least one target intermediate dimension, the plurality of target hidden dimensions and / or the at least one target attention head are randomly selected.

7. The method of any one of claims 1 to 6, wherein the first deep learning model is a pre-trained large model.

8. 7. The method of claim 1, wherein both the first deep learning model and the second deep learning model are configured to perform at least one of the following tasks: a text processing task, an image processing task, and a voice processing task.

9. A computer-implemented method for inference of deep learning models, the first and second deep learning models being obtained by training using a method according to any one of claims 1 to 8, the method comprising: sequentially generating a plurality of predicted tokens using the second deep learning model; and generating a confidence level for each of the predicted tokens based on the predicted tokens using the first deep learning model; and Checking the plurality of predicted tokens based on a generation order of the plurality of predicted tokens to obtain an inference result, in response to determining that a confidence level of a currently checked predicted token is lower than a preset threshold, utilizing the first deep learning model to generate a corrective token at a position of the currently checked predicted token to replace the currently checked predicted token; In response to determining that a predetermined token generation condition is satisfied, using the second deep learning model, based on the corrective token, a new predicted token is sequentially generated from a position next to the corrective token, and using the first deep learning model, a reliability of the newly generated predicted token is generated and checked. and obtaining the inference result based on the checked predicted token.

10. Checking the plurality of predicted tokens based on a generation order of the plurality of predicted tokens to obtain an inference result, 10. The method of claim 9, comprising, in response to determining that a confidence level of a currently checked predicted token is equal to or exceeds a preset threshold, retaining the predicted token as a checked predicted token.

11. The method according to claim 9 or 10, wherein the preset token generation condition includes that the number of already generated tokens is less than a preset number.

12. 11. The method of claim 9 or 10, wherein the pre-set token generation condition comprises the corrective token indicating that token generation is not finished.

13. A deep learning model training apparatus, wherein a first deep learning model includes a plurality of first parameters, and a second deep learning model includes a plurality of second parameters, the plurality of second parameters are initialized to parameter values ​​of a plurality of target parameters in the plurality of first parameters, and the number of the plurality of second parameters is less than the number of the plurality of first parameters; a determination unit configured to determine a target loss for the first deep learning model and the second deep learning model; a parameter adjustment unit configured to adjust parameter values ​​of the plurality of first parameters and the plurality of second parameters based on the target loss to obtain a first deep learning model and a second deep learning model after training, a first parameter adjustment subunit configured to synchronously adjust parameter values ​​of second parameters, which correspond to the at least some target parameters, included in the second deep learning model, in response to determining that the target loss indicates that parameter values ​​of at least some target parameters of the plurality of target parameters included in the first deep learning model need to be adjusted; and and a second parameter adjustment subunit configured to synchronously adjust parameter values ​​of target parameters corresponding to at least a portion of the second parameters included in the first deep learning model in response to determining that the target loss indicates that parameter values ​​of at least a portion of the second parameters among the plurality of second parameters included in the second deep learning model need to be adjusted.

14. The determination unit is a first determination subunit configured to determine a first loss of the first deep learning model for a first sample; a second determination subunit configured to determine a second loss of the second deep learning model for a second sample; and The apparatus of claim 13 , further comprising: a third determining subunit configured to determine the target loss based on the first loss and the second loss.

15. The apparatus of claim 14 , wherein the first sample and the second sample are the same.

16. The first deep learning model includes a first layer of a first number of layers, each of the first layers of the first number of layers including a first intermediate dimension of a first intermediate dimensionality; The second deep learning model includes a second layer of a second number of layers, and the second layers each include a second intermediate dimension of a second intermediate dimensionality, where the second number of layers is smaller than the first number of layers, and the second intermediate dimensionality is smaller than the first intermediate dimensionality; 14. The apparatus of claim 13, wherein the plurality of target parameters include a first parameter in at least one target intermediate dimension for each of the at least one target layer, wherein the at least one target layer is selected in a first layer of the first number of layers based on the second number of layers, and the at least one target intermediate dimension is selected in a first intermediate dimension of the first number of intermediate dimensionality based on the second number of intermediate dimensionality.

17. The first deep learning model and the second deep learning model are based on a Transformer structure or a class Transformer structure; The first deep learning model includes a first hidden dimension of a first number of hidden dimensions, and each of the first layers of the first number of layers includes a first attention head of a first number of attention heads; The second deep learning model includes a second hidden dimension of a second number of hidden dimensions, and the second layers of the second number of layers each include a second attention head of a second number of attention heads; 17. The apparatus of claim 16, wherein the plurality of target parameters include a first parameter corresponding to a plurality of target hidden dimensions and a first parameter at at least one target attention head, wherein the plurality of target hidden dimensions are obtained by selecting a first hidden dimension of the first number of hidden dimensions based on the second number of hidden dimensions, and the at least one target attention head is obtained by selecting a first attention head of the first number of attention heads based on the second number of attention heads.

18. 20. The apparatus of claim 17, wherein the at least one target stratum, the at least one target intermediate dimension, the plurality of target hidden dimensions and / or the at least one target attention head are randomly selected.

19. The apparatus of any one of claims 13 to 18, wherein the first deep learning model is a pre-trained large model.

20. 19. The apparatus of claim 13, wherein both the first deep learning model and the second deep learning model are configured to perform at least one of the following tasks: a text processing task, an image processing task, and a voice processing task.

21. A deep learning model inference device, the first deep learning model and the second deep learning model being obtained by training using the device according to any one of claims 13 to 20, the device comprising: a first generation unit configured to sequentially generate a plurality of predicted tokens using the second deep learning model; a second generating unit configured to generate a confidence level for each of the predicted tokens based on the predicted tokens using the first deep learning model; A checking unit configured to check the plurality of predicted tokens based on a generation order of the plurality of predicted tokens to obtain an inference result, a replacement subunit configured to, in response to determining that a confidence level of a currently checked predicted token is lower than a preset threshold, generate a corrective token at a position of the currently checked predicted token by utilizing the first deep learning model to replace the currently checked predicted token as the checked predicted token; a generation subunit configured to, in response to determining that a preset token generation condition is satisfied, use the second deep learning model to sequentially generate a new predicted token based on the corrective token from a position next to the corrective token, and use the first deep learning model to generate and check a reliability of the newly generated predicted token; and an inference subunit configured to obtain the inference result based on the checked prediction token.

22. The checking unit includes:

22. The apparatus of claim 21, further comprising a retaining subunit configured to retain a currently checked predicted token as a checked predicted token in response to determining that the reliability of the currently checked predicted token is equal to or greater than a preset threshold.

23. 23. The apparatus according to claim 21 or 22, wherein the preset token generation condition includes that the number of generated tokens is less than a preset number.

24. 23. The apparatus of claim 21 or 22, wherein the pre-set token generation condition includes the corrective token indicating that token generation is not finished.

25. An electronic device, At least one processor; and a memory communicatively coupled to the at least one processor, wherein: The electronic device, characterized in that the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 12.

26. A non-transitory computer readable storage medium having stored thereon computer instructions, the computer instructions being used to cause the computer to carry out a method according to any one of claims 1 to 12.

27. A computer program product comprising a computer program, said computer program being adapted to implement the method according to any one of claims 1 to 12 when said computer program is executed by a processor.

Images