Model training method and device, model deployment method and device, electronic equipment and storage medium

Abstract

The invention provides a model training method, and relates to the technical field of artificial intelligence, in particular to the technical field of large models and the technical field of deep learning. According to the specific implementation scheme, at least one piece of module scale information is determined according to preset model scale information; according to the at least one piece of module scale information and a plurality of initial modules of the initial model, at least one to-be-trained module is determined, and the to-be-trained module comprises to-be-trained parameters determined from the initial modules according to the module scale information. And training at least one to-be-trained module according to training sample data to obtain at least one trained module, the training sample data being obtained from at least one of text sample data, image sample data, audio sample data and video sample data. The invention further provides a model deployment method and device, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of large model technology and deep learning technology. More specifically, the present disclosure provides a model training method, a model deployment method, an apparatus, an electronic device, and a storage medium. Background Art

[0002] With the development of artificial intelligence technology, the scale of deep learning models is continuously increasing. As the model scale increases, the performance of deep learning models is also continuously improving. In order to balance the model inference ability and the hardware resource overhead, the model scale can be adjusted when deploying the deep learning model to an application scenario. Summary of the Invention

[0003] The present disclosure provides a model training method, a model deployment method, an apparatus, a device, and a storage medium.

[0004] According to one aspect of the present disclosure, there is provided a model training method, the method comprising: determining at least one module scale information according to preset model scale information; determining at least one module to be trained according to the at least one module scale information and a plurality of initial modules of an initial model, wherein the module to be trained includes trained parameters determined from the initial modules according to the module scale information. Training the at least one module to be trained according to training sample data to obtain at least one trained module, wherein the training sample data is obtained from at least one of text sample data, image sample data, audio sample data, and video sample data.

[0005] According to another aspect of the present disclosure, there is provided a model deployment method, the method comprising: determining a model to be deployed from a target model according to preset model scale information. Deploying the model to be deployed to a hardware device for the model to be deployed, wherein the target model is obtained by training an initial model through the following operations: determining at least one module scale information according to preset model scale information; determining at least one module to be trained according to the at least one module scale information and a plurality of initial modules of the initial model, wherein the module to be trained includes trained parameters determined from the initial modules according to the module scale information; training the at least one module to be trained according to training sample data to obtain at least one trained module, wherein the training sample data is obtained from at least one of text sample data, image sample data, audio sample data, and video sample data.

[0006] According to another aspect of the present disclosure, there is provided a model training apparatus, which includes: a first determination module configured to determine at least one module scale information according to preset model scale information; a second determination module configured to determine at least one module to be trained according to the at least one module scale information and a plurality of initial modules of an initial model, wherein the module to be trained includes trained parameters determined from the initial modules according to the module scale information; a training module configured to train the at least one module to be trained according to training sample data to obtain at least one trained module, wherein the training sample data is obtained from at least one of text sample data, image sample data, audio sample data, and video sample data.

[0007] According to another aspect of the present disclosure, there is provided a model deployment apparatus, which includes: a third determination module configured to determine a model to be deployed from a target model according to preset model scale information; a deployment module configured to deploy the model to be deployed to a hardware device for the model to be deployed, wherein the target model is obtained by training the initial model through the following modules performing related operations: a first determination module configured to determine at least one module scale information according to preset model scale information; a second determination module configured to determine at least one module to be trained according to the at least one module scale information and a plurality of initial modules of the initial model, wherein the module to be trained includes trained parameters determined from the initial modules according to the module scale information; a training module configured to train the at least one module to be trained according to training sample data to obtain at least one trained module, wherein the training sample data is obtained from at least one of text sample data, image sample data, audio sample data, and video sample data.

[0008] According to another aspect of the present disclosure, there is provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided by the present disclosure.

[0009] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method provided by the present disclosure.

[0010] According to another aspect of the present disclosure, there is provided a computer program product including a computer program, where the computer program implements the method provided by the present disclosure when executed by a processor.

[0011] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. Description of the Drawings

[0012] The drawings are used to better understand the present solution and do not constitute a limitation to the present disclosure. Among them:

[0013] Figure 1 is a flowchart of a model training method according to an embodiment of the present disclosure;

[0014] Figure 2A is a schematic diagram of an initial model according to an embodiment of the present disclosure;

[0015] Figure 2B is a schematic diagram of determining a model to be trained according to an embodiment of the present disclosure;

[0016] Figure 3A is a schematic diagram of a plurality of initial attention data blocks according to an embodiment of the present disclosure;

[0017] Figure 3B is a schematic diagram of an intermediate query parameter matrix according to an embodiment of the present disclosure;

[0018] Figure 4 is a schematic diagram of an initial feed-forward parameter matrix according to an embodiment of the present disclosure;

[0019] Figure 5 is a flowchart of a model deployment method according to another embodiment of the present disclosure;

[0020] Figure 6 is a block diagram of a model training apparatus according to an embodiment of the present disclosure;

[0021] Figure 7 is a block diagram of a model deployment apparatus according to an embodiment of the present disclosure; and

[0022] Figure 8 is a block diagram of an electronic device that can apply at least one of the model training method and the model deployment method according to an embodiment of the present disclosure. Detailed Embodiments

[0023] The following describes exemplary embodiments of the present disclosure with reference to the drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, descriptions of well-known functions and structures are omitted below for clarity and conciseness.

[0024] Large-scale deep learning models can also be referred to as large models. Large models can be, for example, large language models (LLMs). The large models with huge scale require high computing resources and storage resources for inference and training, and the inference latency is also large, making it difficult to be deployed in application scenarios with limited hardware resources. Therefore, for different application scenarios, models of different scales can be deployed to balance the inference speed and accuracy.

[0025] To obtain models of different scales, multiple models of different scales can be trained separately to meet the requirements of various application scenarios. However, the method of separate training will increase the model training cost. If large-scale pre-training of the model is required, the computing resource overhead will be very large.

[0026] Large models can be constructed based on the Transformer architecture. To reduce the computing resource overhead, the nested Transformer (Matryoshka Transformer, MatFormer) architecture can be used to obtain multiple small-scale sub-models from a trained model. The nested Transformer architecture makes a structured nested design for the multi-head self-attention (MHA) layer and the feed-forward network (FFN) layer of the Transformer architecture, and the smaller sub-models are included in the larger sub-models. Based on the nested Transformer architecture, a sub-model can be randomly selected for training during the training process. The parameters of the sub-models based on the nested Transformer architecture can be predetermined. For example, for multiple attention heads of the multi-head self-attention layer, based on the nested Transformer architecture, the parameters of the 1st to 3rd attention heads can be used as the parameters of the first sub-model, and the parameters of the 4th to 6th attention heads can be used as the parameters of the second sub-model. If the first sub-model is to be trained, the parameters of the 1st to 3rd attention heads are adjusted.

[0027] Based on the nested Transformer architecture, effective elastic training can be achieved to a certain extent. However, the parameters of the sub-models based on the nested Transformer architecture are preset, which may lead to some parameters not being fully trained, unable to fully utilize the training ability of the large model, and may also affect the final performance of the large model. In addition, during the forward calculation process, based on the nested Transformer architecture, calculations can be performed based on multiple sub-models, resulting in an increase in training cost.

[0028] Therefore, to efficiently deploy the model to the application scenario, the present disclosure provides a model training method, which will be described below.

[0029] Figure 1 It is a flowchart of a model training method according to an embodiment of the present disclosure.

[0030] As Figure 1 shown, the method 100 may include operation S110 to operation S130.

[0031] In operation S110, according to the preset model scale information, at least one module scale information is determined.

[0032] In the embodiments of the present disclosure, the preset model scale information may indicate determining the scale of the model to be trained from the initial model. For example, the preset model scale information may indicate that the number of parameters of the model to be trained is 25% of the number of parameters of the initial model. Another example, the preset model scale information may indicate that the number of parameters of the model to be trained is 50% of the number of parameters of the initial model.

[0033] In the embodiments of the present disclosure, the initial model may be a model with a relatively large scale. The initial model may include a plurality of initial modules. The initial model may also be referred to as a full-scale model. For example, the initial model may be the above-mentioned large model. One transformer block of the large model may be used as an initial module. In addition, the embedding module of the large model may also be used as an initial module.

[0034] In the embodiments of the present disclosure, the module scale information may indicate the scale of the parameters to be trained in the initial module. For example, if the preset model scale information indicates that the number of parameters of the model to be trained is 25% of the number of parameters of the initial model, the module scale information may indicate that the number of parameters to be trained in an initial module is 25% of the number of parameters of this initial module.

[0035] In operation S120, according to at least one module scale information and a plurality of initial modules of the initial model, at least one module to be trained is determined.

[0036] In the embodiments of the present disclosure, the module to be trained includes the parameters to be trained determined from the initial module according to the module scale information. For example, taking the module scale information indicating that the number of parameters to be trained in the initial module is 25% of the number of parameters of this initial module as an example, a plurality of initial parameters can be randomly determined from all the initial parameters of this initial module as the parameters to be trained. The number of parameters of the parameters to be trained is 25% of the number of parameters of all the initial parameters.

[0037] In the embodiments of the present disclosure, the number of modules to be trained may be less than or equal to the number of initial modules. At least one module to be trained may be used as the model to be trained. The model to be trained may be a sub-model of the initial model or the entire initial model.

[0038] In operation S130, at least one module to be trained is trained according to training sample data to obtain at least one trained module.

[0039] In the embodiments of the present disclosure, the training sample data is obtained from at least one of text sample data, image sample data, audio sample data, and video sample data. For example, the training sample data can be unimodal sample data or multimodal sample data. In the case where the training sample data is unimodal sample data, one of the text sample data, image sample data, and audio sample data can be used as the training sample data. For another example, a video includes data of multiple modalities such as images and audio. In the case where the training sample data is multimodal sample data, the video sample data can be used as the training sample data.

[0040] In the embodiments of the present disclosure, the training sample data is input into at least one module to be trained to obtain a sample result. According to the sample result and the label of the training sample data, loss information can be determined. According to the loss information, at least one module to be trained can be adjusted to obtain at least one trained module.

[0041] Through the embodiments of the present disclosure, the module scale information is determined according to the preset model scale information, and the module to be trained is determined according to the module scale information and the initial module. Thus, in multiple training rounds, even if the preset model scale information is the same, the module scale information may be different, and the parameters included in multiple modules to be trained determined according to the same initial module can also be different. In different training rounds, at least one module to be trained corresponding to the preset model scale information is not fixed but dynamically determined, which can make all parameters of the full-scale model possible to be training parameters, enabling the parameters of the model to be fully trained and helping to improve the performance of the model.

[0042] In addition, through the embodiments of the present disclosure, the module to be trained is obtained based on the initial module of the initial model. Thus, only the parameters of the initial model need to be stored in the storage unit, without storing the parameters of each of the multiple sub-models, which can effectively reduce the storage resource overhead. During the training process, forward calculation is performed based on at least one module to be trained determined according to the preset model scale information, which is equivalent to performing forward calculation based on one module to be trained, avoiding forward calculation of multiple sub-models simultaneously, reducing the computational resource overhead of each training round, and lowering the hardware requirement threshold for model training.

[0043] It can be understood that the method of the present disclosure has been described above, and the preset model scale information of the present disclosure will be described below.

[0044] In some embodiments, the preset model scale information includes multiple preset model scale parameters. The preset number of parameters indicated by the preset model scale parameter is less than or equal to the initial number of parameters of the initial model. For example, the multiple preset model scale parameters may include a first preset model scale parameter, a second preset model scale parameter, a third preset model scale parameter, and a fourth preset model scale parameter. The first preset model scale parameter may be 25%, which may indicate that the number of parameters of the model to be trained is 25% of the initial model. The second preset model scale parameter may be 50%, which may indicate that the number of parameters of the model to be trained is 50% of the initial model. The third preset model scale parameter may be 75%, which may indicate that the number of parameters of the model to be trained is 75% of the initial model. The fourth preset model scale parameter may be 100%, which may indicate using the initial model as the model to be trained.

[0045] It can be understood that the preset model scale information of the present disclosure has been described above. Next, some methods for determining the module scale information of the present disclosure will be described.

[0046] In some embodiments, in some implementation manners of the above operation S110, determining at least one module scale information according to the preset model scale information includes: determining a target model scale parameter from multiple preset model scale parameters according to multiple preset parameter probability values for the multiple preset model scale parameters. The preset number of parameters indicated by the target model scale parameter is consistent with the number of parameters to be trained for at least one module to be trained. For example, the first preset parameter probability value for the above first preset model scale parameter may be 16.6%, indicating that the probability of using 25% of the initial model's number of parameters as the parameters of the model to be trained is 16.6%. The second preset parameter probability value for the above second preset model scale parameter may be 16.6%, indicating that the probability of using 50% of the initial model's number of parameters as the parameters of the model to be trained is 16.6%. The third preset parameter probability value for the above third preset model scale parameter may be 16.8%, indicating that the probability of using 75% of the initial model's number of parameters as the parameters of the model to be trained is 16.8%. The fourth preset parameter probability value for the above fourth preset model scale parameter may be 50%, indicating that the probability of using the initial model as the model to be trained is 50%. Taking the second preset model scale parameter as an example of the target model scale parameter, the number of parameters of the model to be trained determined from the initial model may be 50% of the number of parameters of the initial model.

[0047] In some embodiments, the number of modules to be trained is less than or equal to the number of initial modules. Thus, the number of Transformer blocks can be reduced to achieve lightweight deployment. For example, if the target scale parameter is 50%, the initial number of modules is N, and N is an even number greater than 1, N / 2 initial modules can be used as the modules to be trained. Another example is that if the target scale parameter is 50% and the initial number of modules is N, N initial modules can be used as N modules to be trained, and then the scale of each module to be trained can be determined. It can be understood that the number of modules to be trained can be randomly determined based on the target module scale parameter. It can be understood that through the embodiments of the present disclosure, models with different depths (the number of different Transformer blocks) can be effectively trained, improving the training efficiency. The trained models with different depths can adapt to the deployment requirements of different hardware resources and also help improve the model deployment efficiency.

[0048] In some embodiments, in some implementations of the above operation S110, determining at least one module scale information according to the preset model scale information further includes: determining at least one module scale information for at least one initial module according to the target model scale parameter. For example, any module scale information can be the same as the target model scale information. Taking the target model scale parameter being 50% as an example, the module scale information can indicate that the number of parameters of the module to be trained is 50% of the number of parameters of the corresponding initial module. Another example is that when there are multiple pieces of module scale information, different pieces of module scale information can be different. The multiple pieces of module scale information can include first module scale information and second module scale information. The first module scale information can be 40%, indicating that the number of parameters of the module to be trained is 40% of the number of parameters of the corresponding initial module. The second module scale information can be 60%, indicating that the number of parameters of the module to be trained is 60% of the number of parameters of the corresponding initial module.

[0049] In some embodiments, determining at least one module scale information for at least one initial module according to the target model scale parameter includes: determining the latent variable scale information and at least one module scale information for at least one initial module according to the target model scale parameter. The latent variable scale information can indicate the scale of the latent variables used for the module to be trained. For example, if the target model scale parameter indicates that the number of parameters of the model to be trained is 50% of the number of parameters of the corresponding initial model, the latent variable scale information can indicate that the number of parameters of the latent variables used for the model to be trained is 50% of the number of parameters of the latent variables used for the corresponding initial model.

[0050] It can be understood that the above describes the manner of determining the module scale information of the present disclosure. Next, some manners of determining the model to be trained will be described.

[0051] In some embodiments, in some implementations of the above operation S120, determining at least one module to be trained based on at least one module scale information and multiple initial modules of the initial model includes: determining multiple training parameters included in the module to be trained from the initial modules based on the module scale information for the initial modules and at least one of multiple sampling modes. The multiple sampling modes include a random sampling mode. For example, taking the above first module scale information being 40% and the first module scale information being used for the first initial module as an example, 40% of the parameters can be randomly sampled from the multiple initial parameters of the first initial module as the training parameters. In one example, if the number of parameters of the first initial module is 100 megabytes (MB), 40 megabytes of parameters can be randomly sampled as the training parameters. The following will be further described in conjunction with Figure 2A and Figure 2B for further illustration.

[0052] Figure 2A is a schematic diagram of an initial model according to an embodiment of the present disclosure.

[0053] As Figure 2A shown, the initial model m20 may include multiple initial modules. The multiple initial modules may include an initial embedding module emb201, an initial processing module tb201, an initial processing module tb202,..., an initial processing module tb203. The initial processing module may be the above-mentioned transformer block.

[0054] Figure 2B is a schematic diagram of determining a model to be trained according to an embodiment of the present disclosure.

[0055] As Figure 2B shown, by sampling parameters according to the target scale, a model to be trained can be determined from the initial model m20. Next, taking the target model scale parameter being 50% as an example, an explanation will be given.

[0056] As Figure 2B shown, the initial embedding data of the initial embedding module emb201 may include multiple initial embedding parameter blocks. The initial embedding parameter blocks may include multiple initial embedding parameters. Among the multiple initial embedding parameter blocks, the initial embedding parameter block emb2011 and the initial embedding parameter block emb2012 may be used as two training embedding parameter blocks. The sum of the number of parameters of the initial embedding parameter block emb2011 and the number of parameters of the initial embedding parameter block emb2012 may be 50% of the number of parameters of the above-mentioned initial embedding module emb201.

[0057] As Figure 2BAs shown, the initial processing data of the initial processing module tb201 includes multiple initial processing parameter blocks. The initial processing parameter blocks can include multiple initial processing parameters. Among the multiple initial processing parameter blocks, the initial processing parameter block tb2011 and the initial processing parameter block tb2012 can be used as two to-be-trained processing parameter blocks. The sum of the number of parameters of the initial processing parameter block tb2011 and the number of parameters of the initial processing parameter block tb2012 can be 40% of the number of parameters of the above-mentioned initial processing module tb201. It can be understood that the initial processing module tb201 can be the above-mentioned first initial module, and the module scale information for the initial processing module tb201 can be the above-mentioned first module scale information.

[0058] As Figure 2B shown, the initial processing data of the initial processing module tb202 includes multiple initial processing parameter blocks. The multiple initial processing parameter blocks can include the initial processing parameter block tb2021. The initial processing parameter block tb2021 can be used as a to-be-trained processing parameter block. The number of parameters of the initial processing parameter block tb2021 can be 50% of the number of parameters of the above-mentioned initial processing module tb202.

[0059] As Figure 2B shown, the initial processing data of the initial processing module tb203 includes multiple initial processing parameter blocks. The multiple initial processing parameter blocks can include the initial processing parameter block tb2031. The initial processing parameter block tb2031 can be used as a to-be-trained processing parameter block. The number of parameters of the initial processing parameter block tb2031 can be 60% of the number of parameters of the above-mentioned initial processing module tb203. It can be understood that the initial processing module tb203 can be the above-mentioned second initial module, and the module scale information for the initial processing module tb203 can be the above-mentioned second module scale information.

[0060] It can be understood that some methods for determining the to-be-trained module have been described above, and some methods for determining the to-be-trained module will be further described below.

[0061] In some embodiments, the initial module includes at least one of an initial attention layer and an initial feed-forward layer. The module to be trained includes at least one of a to-be-trained attention layer and a to-be-trained feed-forward layer. For example, multiple initial modules may include the above-mentioned initial embedding module and initial processing module. The initial processing module includes at least one of an initial attention layer and an initial feed-forward layer. Multiple initial parameters may include multiple initial processing parameters. Multiple initial processing parameters may include initial attention parameters and initial feed-forward parameters. Multiple to-be-trained parameters may include multiple to-be-trained processing parameters. Multiple to-be-trained processing parameters may include to-be-trained attention parameters and to-be-trained feed-forward parameters. For example, the initial module may include an initial processing module. The initial attention layer may be the above-mentioned multi-head self-attention layer. The initial feed-forward layer may be the above-mentioned feed-forward network layer. The to-be-trained attention layer may be obtained based on the initial attention layer. The to-be-trained feed-forward layer may be obtained based on the initial feed-forward layer.

[0062] In some embodiments, the to-be-trained parameters of the module to be trained may be determined from at least one of the initial attention layer and the initial feed-forward layer. It can be understood that the attention scale information for the initial attention layer and the feed-forward scale information for the initial feed-forward layer may be determined according to the module scale information for the initial processing module. The sum of the number of parameters of the to-be-trained feed-forward parameters indicated by the feed-forward scale information and the number of parameters of the to-be-trained attention parameters indicated by the attention scale information may be the number of initial processing parameters indicated by the module scale information. This will be described below in conjunction with the initial attention layer.

[0063] In some embodiments, determining at least one to-be-trained module according to at least one module scale information and multiple initial modules of the initial model includes: determining the to-be-trained attention data of the to-be-trained attention layer from the initial attention layer according to the module scale information for the initial module. The to-be-trained attention data includes multiple to-be-trained attention parameters.

[0064] In some embodiments, the initial attention layer includes the initial attention data of multiple initial attention heads respectively, and the initial attention data includes an initial query parameter matrix, an initial key parameter matrix, and an initial value parameter matrix. Determining the to-be-trained attention data of the to-be-trained attention layer from the initial attention layer includes: determining at least one intermediate attention data of at least one to-be-trained attention head from at least one initial attention data of at least one initial attention head. Determining at least one to-be-trained attention data from at least one intermediate attention data. This will be further described below in conjunction with Figure 3A for further illustration.

[0065] Figure 3A is a schematic diagram of multiple initial attention data blocks according to an embodiment of the present disclosure.

[0066] As Figure 3AAs shown, the initial attention layer includes a plurality of initial attention data blocks. The plurality of initial attention data blocks include an initial attention data block tba3001, an initial attention data block tba3002, and an initial attention data block tba3003. The initial attention layer of the initial processing module may include 16 initial attention heads. The module scale information for the initial processing module may be 50%, and 8 initial attention heads may be randomly determined as the attention heads to be trained. The initial attention data block tba3001, the initial attention data block tba3002, and the initial attention data block tba3003 respectively include initial attention data of 4 initial attention heads, initial attention data of 2 initial attention heads, and initial attention data of 2 initial attention heads. The initial attention data of each of the 8 initial attention heads may be used as 8 intermediate attention data. In one example, the 8 intermediate attention data may be used as 8 attention data to be trained.

[0067] Through the embodiments of the present disclosure, any one or more attention heads can be determined from a plurality of attention heads as the attention heads to be trained, which improves the flexibility of training and can effectively improve the training efficiency.

[0068] It can be understood that the intermediate attention data is used as the attention data to be trained above. However, the present disclosure is not limited thereto, and a part of the intermediate attention data may also be used as the attention data to be trained, which will be further described below.

[0069] In some embodiments, the intermediate attention data may include an intermediate query parameter matrix, an intermediate key parameter matrix, and an intermediate value parameter matrix. The attention data to be trained includes a query parameter matrix to be trained, a key parameter matrix to be trained, and a value parameter matrix to be trained.

[0070] In some embodiments, determining at least one attention data to be trained from at least one intermediate attention data includes at least one of the following: determining at least one intermediate query parameter sub-matrix in the intermediate query parameter matrix as at least one query parameter sub-matrix to be trained in the query parameter matrix to be trained; determining at least one intermediate key parameter sub-matrix in the intermediate key parameter matrix as at least one key parameter sub-matrix to be trained in the key parameter matrix to be trained; determining at least one intermediate value parameter sub-matrix in the intermediate value parameter matrix as at least one value parameter sub-matrix to be trained in the value parameter matrix to be trained. For example, taking the query parameter matrix as an example, the initial query parameter matrix may be used as the intermediate query parameter matrix. A part or all of the intermediate query parameter matrix may be used as the query parameter matrix to be trained. The following will be combined with Figure 3B for further description.

[0071] Figure 3B is a schematic diagram of an intermediate query parameter matrix according to an embodiment of the present disclosure.

[0072] As shown Figure 3B in the figure, the intermediate query parameter matrix tbaq30 may include multiple intermediate query parameters. The intermediate query parameter sub-matrices tbaq301, tbaq302, and tbaq303 can be randomly determined from the intermediate query parameter matrix tbaq30 as three to-be-trained query parameter sub-matrices. In one example, the matrix elements in the intermediate query parameter matrix tbaq30 other than the intermediate query parameter sub-matrices tbaq301, tbaq302, and tbaq303 can also be set to a preset value (0) to obtain the to-be-trained query parameter matrix. It can be understood that the methods for obtaining the to-be-trained key parameter matrix and the to-be-trained value parameter matrix are the same as or similar to the method for obtaining the to-be-trained parameter matrix according to the intermediate query parameter matrix tbaq30, and the present disclosure will not elaborate herein. Through the embodiments of the present disclosure, any part of the attention parameters can be used as the to-be-trained attention parameters, which can effectively improve the training efficiency.

[0073] It can be understood that according to the attention scale information, at least one initial attention data of at least one initial attention head is randomly determined as at least one intermediate attention data of at least one to-be-trained attention head.

[0074] It can be understood that some methods for determining the to-be-trained attention parameters are described above, and below, some methods for determining the to-be-trained feed-forward parameters will be described.

[0075] In some embodiments, according to the module scale information for the initial module, the to-be-trained feed-forward data of the to-be-trained feed-forward layer can be determined from the initial feed-forward layer.

[0076] In some embodiments, the initial feed-forward parameter matrix of the initial feed-forward layer includes multiple initial feed-forward parameters, and the to-be-trained feed-forward data includes at least one to-be-trained feed-forward parameter sub-matrix. Determining the to-be-trained feed-forward data of the to-be-trained feed-forward layer from the initial feed-forward layer includes: determining at least one initial feed-forward parameter sub-matrix of the initial feed-forward parameter matrix as at least one to-be-trained feed-forward parameter sub-matrix. Below, it will be described in conjunction with Figure 4 for illustration.

[0077] Figure 4 is a schematic diagram of the initial feed-forward parameter matrix according to an embodiment of the present disclosure.

[0078] As Figure 4As shown, the initial feed-forward parameter matrix tbffn40 may include multiple initial feed-forward parameters. The initial feed-forward parameter sub-matrices tbffn401, tbffn402, and tbffn403 can be randomly determined from the initial feed-forward parameter matrix tbffn40 as three to-be-trained feed-forward parameter sub-matrices. In one example, the matrix elements in the initial feed-forward parameter matrix tbffn40 other than the initial feed-forward parameter sub-matrices tbffn401, tbffn402, and tbffn403 can also be set to a preset value (0) to obtain the to-be-trained feed-forward parameter matrix. The total number of parameters of the multiple to-be-trained feed-forward parameter sub-matrices can be consistent with the number of parameters indicated by the feed-forward scale information for the initial feed-forward layer.

[0079] It can be understood that the present disclosure has been described above in connection with the initial attention layer and the initial feed-forward layer of the initial processing module. However, the present disclosure is not limited thereto. The initial processing module may further include multiple initial expert units, which will be described below.

[0080] In some embodiments, determining at least one to-be-trained module according to at least one module scale information and multiple initial modules of the initial model includes: according to the module scale information for the initial module, determining at least one initial expert unit among the multiple initial expert units as at least one to-be-trained expert unit of the to-be-trained module. The expert unit can be a mixture of experts (MoE). For example, taking the above first module scale information as an example, if there are 16 initial expert units, 4 initial expert units can be randomly selected as the to-be-trained expert units. Through the embodiments of the present disclosure, the number of expert units can be adjusted, which can further enable the model to be efficiently deployed in various application scenarios with different hardware resources.

[0081] Through the embodiments of the present disclosure, it is possible to train attention layers, feed-forward layers, latent variables, expert units, and processing modules of different scales, achieving all-round elastic training and providing richer elastic strategies for model training.

[0082] It can be understood that some ways of determining the to-be-trained model have been described above. Below, some ways of training the to-be-trained model will be described.

[0083] In some embodiments, in some embodiments of the above operation S130, the text sample data can be tokenized to obtain text sample token data. The text sample token data is input into at least one module to be trained to process the text sample token data by using at least one module to be trained, and a text sample result is obtained. At least one module to be trained can be trained according to the text sample result and the label of the text sample data. In one example, the text sample data can be question text sample data. The text sample result can be answer text sample result. The label of the text sample data can be the true answer text manually annotated for the question text sample data.

[0084] It can be understood that the above has described the method of the present disclosure in combination with text sample data. However, the present disclosure is not limited thereto, and at least one module to be trained can also be trained based on one or more of text sample data, image sample data, audio sample data, and video sample data. For example, taking the image sample data as an example, the above initial embedding module can perform patch embedding processing on the image sample data for subsequent module processing. The image sample data is input into at least one module to be trained to process the image sample data by using at least one module to be trained, and an image sample result is obtained. At least one module to be trained can be trained according to the image sample result and the label of the image sample result. The image sample data can be, for example, a photographed photo. The image sample result can be, for example, an animated image. The label of the image sample result can be, for example, an animated image manually drawn corresponding to the image sample data. Again, for example, the audio sample data can be converted into text, and then referring to the training method of at least one module to be trained for the text sample data, at least one module to be trained can be trained.

[0085] It can be understood that the above has described the model training method of the present disclosure. Next, the model deployment method of the present disclosure will be described.

[0086] Figure 5 is a flowchart of a model deployment method according to another embodiment of the present disclosure.

[0087] As Figure 5 shown, the method 500 may include operation S510 to operation S520.

[0088] In operation S510, according to the preset model scale information, a model to be deployed is determined from the target model.

[0089] It can be understood that the above description of the preset model scale information also applies to this embodiment, and the present disclosure will not repeat it here.

[0090] In operation S520, the model to be deployed is deployed to the hardware device for the model to be deployed.

[0091] In the embodiments of the present disclosure, the target model is obtained by training the initial model through the following operations: determining at least one module scale information according to the preset model scale information; determining at least one module to be trained according to the at least one module scale information and multiple initial modules of the initial model. The module to be trained includes the parameters to be trained determined from the initial modules according to the module scale information; training at least one module to be trained according to the training sample data to obtain at least one trained module. The training sample data is obtained from at least one of text sample data, image sample data, audio sample data, and video sample data. For example, the target model is obtained by training the initial model through the above method 100.

[0092] In some embodiments, the preset model scale information includes multiple preset model scale parameters, and the preset number of parameters indicated by the preset model scale parameters is less than or equal to the initial number of parameters of the initial model. Determining at least one module scale information according to the preset model scale information includes: determining a target model scale parameter from the multiple preset model scale parameters according to the multiple preset parameter probability values for the multiple preset model scale parameters. The preset number of parameters indicated by the target model scale parameter is consistent with the number of parameters to be trained for at least one module to be trained. Determining at least one module scale information for at least one initial module according to the target model scale parameter.

[0093] In some embodiments, determining at least one module scale information for at least one initial module according to the target model scale parameter includes: determining the latent variable scale information and at least one module scale information for at least one initial module according to the target model scale parameter. The latent variable scale information is used to indicate the scale of the latent variables for the module to be trained.

[0094] In some embodiments, determining at least one module to be trained according to the at least one module scale information and multiple initial modules of the initial model includes: determining multiple parameters to be trained included in the module to be trained from the initial modules based on at least one of multiple sampling modes according to the module scale information for the initial modules. The multiple sampling modes include a random sampling mode.

[0095] In some embodiments, the initial module includes at least one of an initial attention layer and an initial feed-forward layer, and the module to be trained includes at least one of a to-be-trained attention layer and a to-be-trained feed-forward layer. Determining at least one module to be trained according to at least one module scale information and a plurality of initial modules of the initial model includes at least one of the following: determining to-be-trained attention data of the to-be-trained attention layer from the initial attention layer according to the module scale information for the initial module, where the to-be-trained attention data includes a plurality of to-be-trained attention parameters; determining to-be-trained feed-forward data of the to-be-trained feed-forward layer from the initial feed-forward layer according to the module scale information for the initial module, where the to-be-trained feed-forward data includes a plurality of to-be-trained feed-forward parameters.

[0096] In some embodiments, the initial attention layer includes initial attention data of respective multiple initial attention heads, and the initial attention data includes an initial query parameter matrix, an initial key parameter matrix, and an initial value parameter matrix. Determining the to-be-trained attention data of the to-be-trained attention layer from the initial attention layer includes: determining at least one intermediate attention data of at least one to-be-trained attention head from at least one initial attention data of at least one initial attention head. Determining at least one to-be-trained attention data from the at least one intermediate attention data.

[0097] In some embodiments, the intermediate attention data includes an intermediate query parameter matrix, an intermediate key parameter matrix, and an intermediate value parameter matrix, and the to-be-trained attention data includes a to-be-trained query parameter matrix, a to-be-trained key parameter matrix, and a to-be-trained value parameter matrix. Determining at least one to-be-trained attention data from the at least one intermediate attention data includes at least one of the following: determining at least one to-be-trained query parameter sub-matrix of the to-be-trained query parameter matrix from at least one intermediate query parameter sub-matrix in the intermediate query parameter matrix; determining at least one to-be-trained key parameter sub-matrix of the to-be-trained key parameter matrix from at least one intermediate key parameter sub-matrix in the intermediate key parameter matrix; determining at least one to-be-trained value parameter sub-matrix of the to-be-trained value parameter matrix from at least one intermediate value parameter sub-matrix in the intermediate value parameter matrix.

[0098] In some embodiments, the initial feed-forward parameter matrix of the initial feed-forward layer includes a plurality of initial feed-forward parameters, and the to-be-trained feed-forward data includes at least one to-be-trained feed-forward parameter sub-matrix. Determining the to-be-trained feed-forward data of the to-be-trained feed-forward layer from the initial feed-forward layer includes: determining at least one to-be-trained feed-forward parameter sub-matrix of the initial feed-forward parameter matrix as at least one to-be-trained feed-forward parameter sub-matrix.

[0099] In some embodiments, the initial module includes a plurality of initial expert units. Determining at least one module to be trained based on at least one module scale information and a plurality of initial modules of the initial model includes: determining at least one initial expert unit among the plurality of initial expert units as at least one expert unit to be trained of the module to be trained according to the module scale information for the initial module.

[0100] It can be understood that the method of the present disclosure has been described above, and the apparatus of the present disclosure will be described below.

[0101] Figure 6 It is a block diagram of a model training apparatus according to an embodiment of the present disclosure.

[0102] As Figure 6 shown, the apparatus 600 may include a first determination module 610, a second determination module 620, and a training module 630.

[0103] The first determination module 610 is configured to determine at least one module scale information according to preset model scale information.

[0104] The second determination module 620 is configured to determine at least one module to be trained according to at least one module scale information and a plurality of initial modules of the initial model. The module to be trained includes training parameters determined from the initial module according to the module scale information.

[0105] The training module 630 is configured to train at least one module to be trained according to training sample data to obtain at least one trained module. The training sample data is obtained from at least one of text sample data, image sample data, audio sample data, and video sample data.

[0106] In some embodiments, the preset model scale information includes a plurality of preset model scale parameters, and the preset number of parameters indicated by the preset model scale parameters is less than or equal to the initial number of parameters of the initial model. The first determination module includes: a first determination sub-module, configured to determine a target model scale parameter from the plurality of preset model scale parameters according to a plurality of preset parameter probability values for the plurality of preset model scale parameters. The preset number of parameters indicated by the target model scale parameter is consistent with the number of training parameters for at least one module to be trained. A second determination sub-module, configured to determine at least one module scale information for at least one initial module according to the target model scale parameter.

[0107] In some embodiments, the second determination sub-module is further configured to: determine latent variable scale information and at least one module scale information for at least one initial module according to the target model scale parameter. The latent variable scale information is used to indicate the scale of the latent variable for the module to be trained.

[0108] In some embodiments, the second determination module includes: a third determination sub-module, configured to determine, according to the module scale information for the initial module, a plurality of to-be-trained parameters included in the to-be-trained module from the initial module based on at least one of multiple sampling modes. The multiple sampling modes include a random sampling mode.

[0109] In some embodiments, the initial module includes at least one of an initial attention layer and an initial feed-forward layer, and the to-be-trained module includes at least one of a to-be-trained attention layer and a to-be-trained feed-forward layer. The second determination module includes at least one of the following: a fourth determination sub-module, configured to determine, according to the module scale information for the initial module, the to-be-trained attention data of the to-be-trained attention layer from the initial attention layer. The to-be-trained attention data includes a plurality of to-be-trained attention parameters. A fifth determination sub-module, configured to determine, according to the module scale information for the initial module, the to-be-trained feed-forward data of the to-be-trained feed-forward layer from the initial feed-forward layer. The to-be-trained feed-forward data includes a plurality of to-be-trained feed-forward parameters.

[0110] In some embodiments, the initial attention layer includes the initial attention data of respective multiple initial attention heads, and the initial attention data includes an initial query parameter matrix, an initial key parameter matrix, and an initial value parameter matrix. The fourth determination sub-module includes: a first determination unit, configured to determine at least one initial attention data of at least one initial attention head as at least one intermediate attention data of at least one to-be-trained attention head. A second determination unit, configured to determine at least one to-be-trained attention data from the at least one intermediate attention data.

[0111] In some embodiments, the intermediate attention data includes an intermediate query parameter matrix, an intermediate key parameter matrix, and an intermediate value parameter matrix, and the to-be-trained attention data includes a to-be-trained query parameter matrix, a to-be-trained key parameter matrix, and a to-be-trained value parameter matrix. The second determination unit includes at least one of the following: a first determination sub-unit, configured to determine at least one intermediate query parameter sub-matrix in the intermediate query parameter matrix as at least one to-be-trained query parameter sub-matrix in the to-be-trained query parameter matrix. A second determination sub-unit, configured to determine at least one intermediate key parameter sub-matrix in the intermediate key parameter matrix as at least one to-be-trained key parameter sub-matrix in the to-be-trained key parameter matrix. A third determination sub-unit, configured to determine at least one intermediate value parameter sub-matrix in the intermediate value parameter matrix as at least one to-be-trained value parameter sub-matrix in the to-be-trained value parameter matrix.

[0112] In some embodiments, the initial feed-forward parameter matrix of the initial feed-forward layer includes a plurality of initial feed-forward parameters, and the to-be-trained feed-forward data includes at least one to-be-trained feed-forward parameter sub-matrix. The fifth determination sub-module includes: a third determination unit, configured to determine at least one initial feed-forward parameter sub-matrix of the initial feed-forward parameter matrix as at least one to-be-trained feed-forward parameter sub-matrix.

[0113] In some embodiments, the initial module includes a plurality of initial expert units. The second determination module includes: a sixth determination sub-module, configured to determine, according to the module scale information for the initial module, at least one initial expert unit from the plurality of initial expert units as at least one to-be-trained expert unit of the to-be-trained module.

[0114] Figure 7 It is a block diagram of a model deployment device according to an embodiment of the present disclosure.

[0115] As Figure 7 shown, the device 700 may include a third determination module 710 and a deployment module 720.

[0116] The third determination module 710 is configured to determine a to-be-deployed model from the target model according to the preset model scale information.

[0117] The deployment module 720 is configured to deploy the to-be-deployed model to the hardware device for the to-be-deployed model.

[0118] In some embodiments, the target model is obtained by training the initial model through the following modules performing related operations: a first determination module, configured to determine at least one module scale information according to the preset model scale information; a second determination module, configured to determine at least one to-be-trained module according to the at least one module scale information and a plurality of initial modules of the initial model; the to-be-trained module includes to-be-trained parameters determined from the initial module according to the module scale information; a training module, configured to train the at least one to-be-trained module according to the training sample data to obtain at least one trained module; the training sample data is obtained from at least one of text sample data, image sample data, audio sample data, and video sample data. For example, the target model may be obtained by training the initial model through the above device 600. The above description of the device 600 is equally applicable to this embodiment, and the present disclosure will not be elaborated herein.

[0119] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0120] According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0121] Figure 8FIG. 0 shows a schematic block diagram of an exemplary electronic device 800 that may be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, for example, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as, for example, personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementations of the present disclosure described and / or claimed herein.

[0122] As Figure 8 shown, the device 800 includes a computing unit 801 that may perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 may also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0123] A plurality of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as, for example, a keyboard, a mouse, etc.; an output unit 807, such as, for example, various types of displays, speakers, etc.; a storage unit 808, such as, for example, a magnetic disk, an optical disk, etc.; and a communication unit 809, such as, for example, a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0124] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a Central Processing Unit (CPU), a Graph Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 executes each of the methods and processes described above, such as at least one of the model training method and the module deployment method. For example, in some embodiments, at least one of the model training method and the module deployment method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of at least one of the model training method and the module deployment method described above can be executed. Alternatively, in other embodiments, the computing unit 801 can be configured to execute at least one of the model training method and the module deployment method by any other suitable means (e.g., by means of firmware).

[0125] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip (SOC) systems, complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0126] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine, or entirely on the remote machine or server.

[0127] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM) or flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) monitor or a liquid crystal display (LCD)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0129] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0130] A computer system may include a client and a server. The client and the server are generally far away from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs that run on the respective computers and have a client-server relationship with each other.

[0131] It should be understood that various forms of the processes shown above can be used, steps can be reordered, added or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and no limitation is made herein.

[0132] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A model training method, comprising: Determining at least one module scale information according to preset model scale information; Determining at least one module to be trained according to the at least one module scale information and multiple initial modules of the initial model, wherein the module to be trained includes to-be-trained parameters determined from the initial modules according to the module scale information; Training the at least one module to be trained according to training sample data to obtain at least one trained module, wherein the training sample data is obtained from at least one of text sample data, image sample data, audio sample data, and video sample data.

2. The method according to claim 1, wherein, The preset model scale information includes multiple preset model scale parameters, and the preset number of parameters indicated by the preset model scale parameters is less than or equal to the initial number of parameters of the initial model. The determining at least one module scale information according to the preset model scale information includes: Determining a target model scale parameter from the multiple preset model scale parameters according to multiple preset parameter probability values for the multiple preset model scale parameters, wherein the preset number of parameters indicated by the target model scale parameter is consistent with the to-be-trained number of parameters for at least one of the modules to be trained; Determining at least one of the module scale information for at least one of the initial modules according to the target model scale parameter.

3. The method according to claim 2, wherein The determining at least one of the module scale information for at least one of the initial modules according to the target model scale parameter includes: Determining latent variable scale information and at least one of the module scale information for at least one of the initial modules according to the target model scale parameter, wherein the latent variable scale information is used to indicate the scale of the latent variables for the module to be trained.

4. The method according to claim 1, wherein The determining at least one module to be trained according to the at least one module scale information and multiple initial modules of the initial model includes: Determining multiple to-be-trained parameters included in the module to be trained from the initial modules based on at least one of multiple sampling modes according to the module scale information for the initial modules, wherein the multiple sampling modes include a random sampling mode.

5. The method according to claim 1, wherein, The initial module includes at least one of an initial attention layer and an initial feed-forward layer, and the module to be trained includes at least one of a to-be-trained attention layer and a to-be-trained feed-forward layer. The determining at least one module to be trained according to the at least one module scale information and multiple initial modules of the initial model includes at least one of the following: Determining to-be-trained attention data of the to-be-trained attention layer from the initial attention layer according to the module scale information for the initial module, wherein the to-be-trained attention data includes multiple to-be-trained attention parameters; Determining to-be-trained feed-forward data of the to-be-trained feed-forward layer from the initial feed-forward layer according to the module scale information for the initial module, wherein the to-be-trained feed-forward data includes multiple to-be-trained feed-forward parameters.

6. The method according to claim 5, wherein, The initial attention layer includes initial attention data of multiple initial attention heads respectively, and the initial attention data includes an initial query parameter matrix, an initial key parameter matrix, and an initial value parameter matrix. Determining the attention data to be trained for the attention layer to be trained from the initial attention layer includes: Determining at least one intermediate attention data of at least one of the attention heads to be trained from at least one of the initial attention data of at least one of the initial attention heads; Determining at least one of the attention data to be trained from at least one of the intermediate attention data.

7. The method according to claim 6, wherein, The intermediate attention data includes an intermediate query parameter matrix, an intermediate key parameter matrix, and an intermediate value parameter matrix, and the attention data to be trained includes a query parameter matrix to be trained, a key parameter matrix to be trained, and a value parameter matrix to be trained. Determining at least one of the attention data to be trained from at least one of the intermediate attention data includes at least one of the following: Determining at least one intermediate query parameter sub-matrix in the intermediate query parameter matrix as at least one query parameter sub-matrix to be trained in the query parameter matrix to be trained; Determining at least one intermediate key parameter sub-matrix in the intermediate key parameter matrix as at least one key parameter sub-matrix to be trained in the key parameter matrix to be trained; Determining at least one intermediate value parameter sub-matrix in the intermediate value parameter matrix as at least one value parameter sub-matrix to be trained in the value parameter matrix to be trained.

8. The method according to claim 6, wherein The initial feed-forward parameter matrix of the initial feed-forward layer includes a plurality of initial feed-forward parameters, and the feed-forward data to be trained includes at least one feed-forward parameter sub-matrix to be trained. Determining the feed-forward data to be trained for the feed-forward layer to be trained from the initial feed-forward layer includes: Determining at least one initial feed-forward parameter sub-matrix of the initial feed-forward parameter matrix as at least one feed-forward parameter sub-matrix to be trained.

9. The method according to claim 1, wherein The initial module includes a plurality of initial expert units. Determining at least one module to be trained based on at least one of the module scale information and a plurality of initial modules of the initial model includes: Based on the module scale information for the initial module, determining at least one of the initial expert units among the plurality of initial expert units as at least one expert unit to be trained in the module to be trained.

10. A model deployment method, including: Determining a model to be deployed from a target model according to preset model scale information; Deploying the model to be deployed to a hardware device for the model to be deployed, where the target model is obtained by training an initial model through the following operations: Determining at least one module scale information according to the preset model scale information; Determining at least one module to be trained based on at least one of the module scale information and a plurality of initial modules of the initial model, where the module to be trained includes trained parameters determined from the initial module according to the module scale information; Training at least one of the modules to be trained according to training sample data to obtain at least one trained module, where the training sample data is obtained from at least one of text sample data, image sample data, audio sample data, and video sample data.

11. A model training device, including: A first determination module for determining at least one module scale information according to preset model scale information; A second determination module, configured to determine at least one module to be trained according to at least one of the module scale information and a plurality of initial modules of the initial model, where the module to be trained includes training parameters determined from the initial modules according to the module scale information; A training module, configured to train at least one of the modules to be trained according to training sample data to obtain at least one trained module, where the training sample data is obtained from at least one of text sample data, image sample data, audio sample data, and video sample data.

12. A module deployment device, comprising: A third determination module, configured to determine a model to be deployed from a target model according to preset model scale information; A deployment module, configured to deploy the model to be deployed to a hardware device for the model to be deployed, where the target model is obtained by training the initial model through the following modules performing related operations: A first determination module, configured to determine at least one module scale information according to the preset model scale information; A second determination module, configured to determine at least one module to be trained according to at least one of the module scale information and a plurality of initial modules of the initial model, where the module to be trained includes training parameters determined from the initial modules according to the module scale information; A training module, configured to train at least one of the modules to be trained according to training sample data to obtain at least one trained module, where the training sample data is obtained from at least one of text sample data, image sample data, audio sample data, and video sample data.

13. An electronic device, comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

15. A computer program product, comprising a computer program, where the computer program, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images