Data processing method and apparatus, computing device

Abstract

Embodiments of the present application provide a data processing method and device, and a computing device. The data processing method comprises: determining a first network module whose model parameters are known; obtaining a to-be-trained module newly added for the first network module; wherein the model parameters of the to-be-trained module are unknown; constructing a second network module according to the first network module and the to-be-trained module; and training the model parameters of the to-be-trained module in the second network module based on a plurality of training data. The embodiments of the present application improve the training efficiency of a model.

Description

Technical Field

[0001] This application relates to the field of computing device technology, and in particular to a data processing method and apparatus, and a computing device. Background Technology

[0002] In recent years, natural language processing (NLP) tasks have become increasingly widespread. NLP neural network models are generally used to perform these tasks. Typically, to ensure the accuracy of the neural network model in processing natural language, deep neural network models are employed. BERT (Bidirectional Encoder Representations from Transformers) is a common deep neural network model used in NLP, applicable to scenarios such as word prediction, sentence association prediction, automatic question answering, and content recommendation.

[0003] Typically, a BERT model can include an input layer, a vector representation layer, an encoding layer, and a classification layer. The key to this model is its use of the Transformer encoder. The model parameters of a BERT model can include the parameters of each encoder in the encoding layer. To obtain these model parameters, multiple training datasets are typically used, and the model is trained multiple times to obtain the parameters of the pre-built BERT model.

[0004] As described above, the encoding layer in deep neural network models such as BERT typically consists of multiple encoder modules. During training, representation calculations need to be performed on all modules to obtain accurate model parameters, which involves a very large amount of computation and consumes a lot of time, resulting in low training efficiency. Summary of the Invention

[0005] In view of this, the present application provides a method to solve the technical problems in the prior art.

[0006] In a first aspect, embodiments of this application provide a data processing method, including:

[0007] Identify the first network module whose model parameters are known;

[0008] Obtain the newly added training module for the first network module; wherein the model parameters of the training module are unknown;

[0009] Based on the first network module and the module to be trained, a second network module is constructed;

[0010] Based on multiple training data, the model parameters of the module to be trained in the second network module are obtained.

[0011] Secondly, embodiments of this application provide a data processing apparatus, including:

[0012] The model determination module is used to determine the first network module whose model parameters are known.

[0013] The model adds a module, which is used to obtain a new training module to be added to the first network module; wherein the model parameters of the training module are unknown.

[0014] A new construction module is added to construct a second network module based on the first network module and the module to be trained;

[0015] The parameter training module is used to train and obtain the model parameters of the module to be trained in the second network module based on multiple training data.

[0016] Thirdly, embodiments of this application provide a computing device, including: a storage component and a processing component; the storage component is used to store one or more computer instructions; the one or more computer instructions are invoked by the processing component to execute any data processing method provided in embodiments of this application.

[0017] Fourthly, embodiments of this application provide a storage medium, including: a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer, performs any of the data processing methods provided in embodiments of this application.

[0018] In this embodiment, after determining a first network module with known model parameters, a newly added trainable module can be obtained, whose model parameters are unknown. Then, a second network module can be constructed based on the first network module and the newly added trainable module. At this point, the model parameters of the first network module in the second network module are known, while the model parameters of the trainable module are unknown. Based on multiple training data sets, the model parameters of the trainable module in the second network module can be obtained through training. In this embodiment, a trainable module is added to the first network module with known model parameters, and training is performed only on the model parameters of this trainable module, thereby reducing the number of parameters required for model training, lowering training complexity, and improving training efficiency. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of one embodiment of a data processing method provided in this application;

[0021] Figure 2 A flowchart illustrating yet another embodiment of a data processing method provided in this application;

[0022] Figure 3 A flowchart illustrating yet another embodiment of a data processing method provided in this application;

[0023] Figure 4 A flowchart illustrating yet another embodiment of a data processing method provided in this application;

[0024] Figure 5 An example diagram illustrating a data processing method provided in an embodiment of this application;

[0025] Figure 6 Another example diagram of a data processing method provided in an embodiment of this application;

[0026] Figure 7 A schematic diagram of one embodiment of a data processing apparatus provided in this application;

[0027] Figure 8 This is a schematic diagram of one embodiment of a computing device provided in this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “said,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0030] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0031] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to identification.” Similarly, depending on the context, the phrases “if determination” or “if identification (of the condition or event of the statement)” can be interpreted as “when determination” or “in response to determination” or “when identification (of the condition or event of the statement)” or “in response to identification (of the condition or event of the statement).”

[0032] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0033] The technical solution of this application embodiment can be applied to the scenario of neural network model optimization. By training the model parameters in the neural network in batches, targeted training is performed only on newly added modules in the neural network model each time, thereby improving training efficiency.

[0034] In existing technologies, the model parameters of neural network models need to be obtained through training. Neural network models can be, for example, deep neural network models like BERT. Due to the complexity of neural network models, obtaining model parameters through training is quite complex for deep neural network models. Taking the BERT model as an example, the encoding layer of the BERT model can consist of 12 or 24 encoder modules. Each encoder, according to its connection order, constitutes a sub-encoding layer of that encoding layer. The assembled encoding layers can be referred to as... and ,in, Includes 12 encoder modules, It includes 24 encoder modules. The encoder layer of the BERT model can contain many neurons, with 12 encoder modules being a common configuration. For example, one encoder module includes 768 neural units and 12 self-attention heads, with a corresponding number of parameters of 110M. Taking 24 encoder modules as an example... For example, one Encoder module includes 1024 neurons and 16 self-attention heads, with a corresponding number of parameters of 340M. As can be seen from the BERT model example, the network structure of neural network models is very complex, and the number of parameters that need to be trained during the training process is very large, resulting in a very complex training process and low training efficiency.

[0035] In this embodiment, the model architecture of a neural network model can be layered by dividing network modules with similar or identical structures. Taking the decoding layer of the BERT model as an example, whether there are 12 or 24 encoder modules, it can be divided into multiple training modules. Based on the original network and network parameters, the parameters of newly added training modules are continuously trained to obtain the model parameters of the newly added training modules, thereby reducing the number of parameters in each training iteration and improving the training efficiency of the neural network model.

[0036] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0037] like Figure 1 The diagram shown is a flowchart of one embodiment of a data processing method provided in this application. The method may include the following steps:

[0038] 101: Determine the first network module whose model parameters are known.

[0039] The data processing method provided in this application can be applied to computing devices. In some embodiments, the computing device may include, for example, a personal computer, a super mobile personal computer, a laptop, or other electronic device with computing processing capabilities. Furthermore, the computing device may also include, for example, a server or a cloud server. This application does not impose excessive limitations on the specific type of computing device.

[0040] Optionally, the first network module can be a portion of the network structure in the target network model that has already been trained to obtain model parameters. The target network model can be a deep neural network model, specifically a machine learning model applied in the field of natural language processing, such as the BERT model. The target network model can include a multi-layer network structure, for example, it can include a vector representation layer, an encoding layer, and a classification layer, where the classification layer generally includes a decoder and a classifier. The classifier can decode and classify the data output from the encoding layer to obtain the network computation results corresponding to the target network model.

[0041] The model parameters of the first network module can be obtained through training. The first network module may include the original basic training module as well as newly added pre-trained modules.

[0042] 102: Obtain the newly added training module for the first network module.

[0043] The model parameters of the module to be trained are unknown.

[0044] 103: Construct a second network module based on the first network module and the module to be trained.

[0045] Optionally, the module to be trained can be added to the first network module to form a second network module. The model parameters of the module to be trained need to be obtained through training. The second network module includes the original network modules in the first network module as well as the newly added module to be trained.

[0046] 104: Based on multiple training data, the model parameters of the module to be trained in the second network module are obtained through training.

[0047] Optionally, the model parameters of the module to be trained in the second network module can be obtained by training based on multiple training data. During the training process, the model parameters of the original network modules in the first network module are known and do not participate in parameter training.

[0048] In this embodiment, after determining the first network module with known model parameters, a newly added trainable module can be obtained, whose model parameters are unknown. Then, a second network module can be constructed based on the first network module and the newly added trainable module. At this point, the model parameters of the first network module in the second network module are known, while the model parameters of the trainable module are unknown. Based on multiple training data, the model parameters of the trainable module in the second network module can be trained and obtained. In this embodiment, based on the first network module with known model parameters, a trainable module for parameter training is added, and training is performed only on the model parameters of this trainable module, thereby reducing the number of parameters in the model training, lowering the training complexity, and improving training efficiency.

[0049] In order to train the entire neural network model, it can be determined whether the newly obtained second network module meets the network construction conditions. If the network construction conditions are not met, training efficiency can be improved by adding new modules to be trained multiple times and performing targeted individual training.

[0050] refer to Figure 2 ,and Figure 1 The difference in the illustrated embodiment is that, after step 104, the method may further include:

[0051] 105: Determine whether the second network module meets the model construction conditions.

[0052] Optionally, determining whether the second network module meets the model construction conditions may include: obtaining the number of encoders in the encoding layer of the second network module, and determining whether the number of encoders is the same as a preset encoder number threshold. Taking the BERT model as an example, the encoder number threshold can be set to 12 or 24. Obtaining the number of encoders in the second network module, and when the obtained number of encoders is 12 or 24, indicates that the second network module meets the model construction conditions.

[0053] 106: If so, then the second network module is determined to be the target network model.

[0054] When the second network module meets the model construction conditions, the second network module can be used as the target network model. At this time, the model parameters of the first network module and the module to be trained in the second network module are known parameters, and the target model parameters of the target network model are also known.

[0055] The target model parameters of the target network model are obtained by training multiple modules in multiple batches. This means the model parameters may not be obtained from the same batch of training. To improve the performance of the target model parameters, in some embodiments, the target model parameters can be fine-tuned to obtain more accurate target network parameters. Optionally, when fine-tuning the target model parameters, a parameter adjustment strategy can be used to adjust the target model parameters to obtain the final target model parameters.

[0056] 107: If not, then the second network module is used as the new first network module, and the process returns to step 102 to continue execution.

[0057] The second network module is used as the new first network module, and the process continues to obtain newly added training modules for this new first network module.

[0058] In this embodiment, the step of adding a new training module based on the first network module can be performed multiple times to continuously adjust the network model of the first network module and add new network modules. In practical applications, the model construction conditions of the second network module can be determined.

[0059] exist Figure 2 Step 106 of the illustrated embodiment: If so, after determining that the second network module is the target network model, the method may further include:

[0060] The model parameters of the first network module and the model parameters of the module to be trained are determined as the target model parameters of the target network model.

[0061] The target model parameters include the original model parameters of the first network module and the model parameters of the module to be trained. After obtaining the model parameters of the module to be trained, since the model parameters of the first network module are known, the original network parameters of the first network module can be directly determined.

[0062] In practical applications, the target network model can specifically be a deep neural network model, such as the BERT model, ALBERT (ALitBidirectional Encoder Representations from Transformers) model, or Struct BERT (Structural Bidirectional Encoder Representations) model. Some common application scenarios for target network models include:

[0063] Method 1:

[0064] Obtain the target model parameters of the target network model;

[0065] Detect the pending statement input by the second user;

[0066] The statement to be processed is input into the target network model corresponding to the target model parameters, and the target information corresponding to the statement to be processed is calculated.

[0067] The target information is output to the second user.

[0068] Optionally, the statement to be processed can be a statement input by a second user that needs to be processed by the parameters of the target model. The statement type can be one of several types, and the statement type can be set based on the function of the target network model. For example, if the main function of the target network model is statement inference, the statement to be processed can be of the inference type; if the main function is voice question answering, the statement to be processed can be of the question answering type; if the main function is translation, the statement to be processed can be of the translation type; if the main function is statement classification, the statement to be processed can be of the classification type. The target network model performs calculations on the statement to be processed to obtain the target information corresponding to the information to be processed, which is also determined based on the statement type of the statement to be processed. For example, if the statement to be processed is of the inference type, the target information corresponding to the statement to be processed can be the target inference word or phrase; if the statement to be processed is of the question answering type, the target information corresponding to the statement to be processed can be the target answer statement; if the statement to be processed is of the translation type, the target information corresponding to the statement to be processed can be the text information obtained after translating the statement to be processed from the source language to the target language.

[0069] In practical use, a second user can input a statement to be processed using a second user terminal. After detecting the statement input by the second user, the second user terminal can send the statement to a computing device configured with the data processing method provided in this application embodiment. At this time, the computing device can confirm that it has detected the statement input by the second user. By inputting the statement to be processed into the target network model corresponding to the target model parameters, the target information corresponding to the information to be processed can be calculated. The computing device outputting the target information to the second user can specifically include: sending the target information to the second user terminal. After receiving the target information, the second user terminal can output the target information.

[0070] Optionally, when sending target information to the second user terminal, the computing device may send the target information via a short message, long link, or web link.

[0071] Method 2:

[0072] Obtain the target model parameters of the target network model;

[0073] Determine the user information of the third user;

[0074] The user information is input into the target network model corresponding to the target model parameters, and the first recommended content matching the third user is calculated.

[0075] The first recommended content is output to the third user.

[0076] The target network model can also retrieve relevant content recommendations based on user information. User information may include, for example, user identity information, browsing history of products, historical clicks, and historical purchase information. User identity information may include, for example, the user's age, identity, occupation, and / or interests.

[0077] In one possible design, the computing device can retrieve user information based on the identity information of a third user. The user's identity information may include at least one of the following: identity identifier, mobile phone number, account information, employee number, etc.

[0078] Optionally, the first recommended content can be recommended content that matches the user information of a third user, such as recommended products, books, multimedia content, or advertisements.

[0079] In another possible design, the computing device can obtain the user information of the third user based on the recommendation request initiated by the third user, and input the user information into the target network model corresponding to the target model parameters to calculate the first recommendation content that matches the third user.

[0080] In practical applications, the computing device outputting the first recommended content to the third user may specifically include: sending the first recommended content to the third user's third user terminal, so that the third user terminal outputs the first recommended content to the third user.

[0081] Optionally, when sending the first recommended content to the third user terminal, the computing device may send the first recommended content to the third user terminal in the form of a short message, a long link, or a web link.

[0082] Method 3:

[0083] Obtain the target model parameters of the target network model;

[0084] Determine the image to be processed input by the fourth user;

[0085] The image to be processed is input into the target network model corresponding to the target model parameters, and second recommended content matching the image to be processed is calculated.

[0086] The second recommended content is output to the fourth user.

[0087] The fourth user can also input an image to be processed, and use a target network model to extract features from the image to be processed, and obtain second recommended content consisting of matching product objects, commodity objects, multimedia information or advertising information based on the image to be processed, so as to realize image-based content recommendation.

[0088] In some embodiments, the fourth user terminal can obtain historical images stored in an image library by the fourth user to obtain the image to be processed. The fourth user terminal can also detect camera requests initiated by the fourth user and take real-time photos to obtain the image to be processed.

[0089] In some other embodiments, the image to be processed can be extracted from product information, videos, or advertising materials provided by a fourth user, so as to make subsequent relevant content recommendations based on the image to be processed provided by the fourth user.

[0090] Optionally, when sending the second recommended content to the fourth user terminal, the computing device may send a short message, long link information, or webpage link corresponding to the second recommended content to the fourth user terminal. The fourth user terminal can obtain the short message, long link information, or webpage link information corresponding to the second recommended content, and output the second recommended content to the fourth user through message output, information output, or webpage display.

[0091] refer to Figure 2 Step 105: Determining whether the second network module meets the model construction conditions may specifically include:

[0092] Determine whether the number of encoders in the coding layer of the second network module has reached the preset number of network layers.

[0093] The second network module can include a vector representation layer, an encoding layer, and a classification layer. The encoding layer in the second network module can include multiple encoders. When the number of encoders reaches a certain number of network layers, it means that the network depth of the encoding layer can meet the requirements. At this point, adding new modules to be trained can be stopped to obtain the final target network model.

[0094] In this embodiment, the number of encoders in the coding layer is used as the basis for terminating model training. When the preset number of network layers is reached, the addition of new encoders to the coding layer is stopped, so as to quickly complete the training of the network model and improve training efficiency.

[0095] In one possible design, the target network model can be divided into a vector representation layer, an encoding layer, and a classification layer. The classification layer can include a decoder and a classifier.

[0096] As one embodiment, constructing a second network module based on the first network module and the module to be trained may include:

[0097] The vector representation layer, the first encoding layer, and the classification layer in the first network module are determined; wherein, the vector parameters of the vector representation layer, the first encoding parameters of the first encoder, and the classification parameters of the classification layer are known;

[0098] The module to be trained is added to the first encoding layer to obtain the second encoding layer;

[0099] The second network module, consisting of the vector representation layer, the second encoding layer, and the classification layer, is determined.

[0100] The process of training the model parameters of the module to be trained in the second network module based on multiple training data may include:

[0101] The model parameters of the module to be trained in the encoding layer of the second network module are trained based on multiple training data.

[0102] In this embodiment, the module to be trained can be a portion of the encoding layer. This module can be added to the encoding layer of the first network module to obtain the second network module. By selectively adding the module to the encoding layer, parameter training is performed only on the newly added module. The model parameters of the original vector representation layer, the original first encoding layer, and the classification layer in the first network module are known and do not participate in training. By reducing the number of modules involved in training, the training of the second network model can be completed quickly, improving training efficiency. Furthermore, in this embodiment, adding the module to be trained to the encoding layer of the first network module has a significant effect on improving the training efficiency of neural network models with high encoding layer depth, thereby increasing the overall training efficiency of the network model.

[0103] In some target network models, especially the BERT model, the encoding layer consists of multiple cascaded encoders. A common example of an encoder is the Encoder in the Transform model.

[0104] To reduce computational complexity during training, the number of encoders can be set according to the total number of encoders in the target network model's encoding layer, or the training phases can be divided. For example, for an encoding layer consisting of 12 encoders, a step size of 4 can be set, training the model parameters of 4 encoders each time, and determining the encoder whose model parameters are known at the end of training. Alternatively, when pre-dividing the training phases, multiple training phases can be set, and the number of encoders in each training phase can be set accordingly. Again, for an encoding layer consisting of 12 encoders, 3 training phases can be designed: the first phase trains 5 encoders, the second phase trains 5 encoders, and the third phase trains 2 encoders.

[0105] Therefore, as an example, the first encoding layer may include: at least one first encoder; the first encoding parameters of the at least one first encoder are known;

[0106] The module to be trained includes: at least one second encoder; wherein, the second encoding parameters of the at least one second encoder are unknown;

[0107] Adding the module to be trained to the first coding layer to obtain the second coding layer includes:

[0108] By placing the at least one second encoder after the at least one first encoder, a second encoding layer is obtained, consisting of the at least one first encoder and the at least one second encoder located after the at least one first encoder.

[0109] The first network module may include a vector representation layer, a first encoding layer, and a classification layer. The vector parameters of the vector representation layer, the first encoding parameters of the first encoding layer, and the classification parameters of the classification layer are known.

[0110] The second network module may include a vector representation layer, a second encoder, and a classification layer. The second encoder layer consists of at least one first encoder from the original first encoder layer and at least one newly added second encoder.

[0111] At least one encoder is connected in series to form a coding layer according to the number of its respective coding sub-layers.

[0112] Step 103: Constructing the second network module based on the first network module and the module to be trained may specifically include:

[0113] The second network module is defined as consisting of the vector representation layer, at least one first encoder following the vector representation layer, at least one second encoder following the at least one first encoder, and a classification layer following the at least one second encoder in the first network module.

[0114] At least one second encoder may be connected in series after at least one first encoder.

[0115] When at least one second encoder is the same encoder, since the encoders are the same and the parameters are unknown, at least one second encoder can be randomly connected in series.

[0116] After identifying at least one first encoder and at least one second encoder following the first encoder, the cascade order of each encoder can be recorded to facilitate the management of the model parameters of each encoder and avoid parameter matching errors.

[0117] As another embodiment, after setting the at least one second encoder after the at least one first encoder, and obtaining the second encoding layer consisting of the at least one first encoder and the at least one second encoder located after the at least one first encoder, the method may further include:

[0118] Based on multiple training data, the second encoding parameters of at least one second encoder in the encoding layer of the second network module are obtained through training.

[0119] When the module to be trained is at least one second encoder, the model parameters of at least one second encoder are trained during the training process to reduce training complexity and improve training efficiency.

[0120] As another embodiment, the acquisition is a newly added training module to the first network module; wherein, the unknown model parameters of the training module may include:

[0121] Obtain the preset encoder quantity step size;

[0122] A training module consisting of at least one second encoder that matches the number step size of the encoder is determined; wherein the second encoding parameter of the at least one second encoder is unknown.

[0123] The encoder's step size can be preset, for example, based on the quotient of the network depth of the encoding layer in the target network model and the number of training iterations. Assuming the network depth of the encoding layer in the target model is 12, and the number of training iterations or training stages is set to 3, then the encoder's step size can be 4.

[0124] In practical applications, during the initial training phase of a neural network model, the parameters of each network layer may be unknown. In this case, it is necessary to train the basic network model in the target network model to obtain the first training process.

[0125] like Figure 3 The diagram shown is a flowchart of another embodiment of a data processing method provided in this application. The method may include the following steps:

[0126] 301: Determine if there is a first network module with known model parameters; if yes, proceed to step 101; if no, proceed to step 302.

[0127] 302: Obtain the basic training module with unknown parameters as the training module in the second network module. Upon completion of step 302, proceed to step 104.

[0128] When training the network model begins, there may not be a first network module with known model parameters; that is, the first network module is empty at this point. When acquiring a new trainable module for the first network module, only this newly acquired trainable module is acquired; this trainable module is the base training module in the target network model. The base training module serves as the trainable module, and subsequent training based on multiple training data is performed to obtain the model parameters of the trainable module in the second network module. At this point, the model parameters of the obtained trainable module are the model parameters of the base training module.

[0129] The base network module may include a vector representation layer, an initial encoding layer, and a classification layer. At this stage, the model parameters of the vector representation layer, the initial encoding layer, and the classification layer in the base network module are unknown. The base network module can be used as the training module to train the model parameters and obtain the model parameters within the base network module.

[0130] refer to Figure 3 In the illustrated embodiment, the basic training module may include: a vector representation layer, an encoding layer composed of a basic encoder, and a classification layer; wherein, the vector parameters of the vector representation layer, the first encoding parameters of the encoding layer composed of at least one first encoder, and the classification parameters of the classification layer are unknown.

[0131] If not, the step of obtaining the unknown basic training module as the training module in the second network module and then proceeding to the step of training the model parameters of the training module in the second network module based on multiple training data may include:

[0132] If not, the basic training module with unknown parameters is obtained as the training module in the second network module;

[0133] Based on multiple training data, the vector parameters of the vector representation layer in the basic training module, the encoding parameters of the basic encoding layer composed of at least one encoder, and the classification parameters of the classification layer are obtained through training.

[0134] The basic coding layer includes at least one encoder. This encoder can be the same as the first encoder and the second encoder. The terms "first" and "second" used in this application embodiment are merely for distinguishing different encoders by name, and do not distinguish them by encoder structure or encoder principle.

[0135] In this embodiment, after obtaining the basic training module with unknown parameters, the basic training module can be used as the training module in the second network module. Based on multiple training data, the vector parameters of the vector representation layer, the encoding layer consisting of at least one encoder, and the classification parameters of the classification layer in the basic training module are trained. The model parameters of the basic training module can be obtained through multiple training data. If the basic training module training is completed, the basic training module is used as the first network module to begin the subsequent steps of adding training modules to the first network module. At this time, the at least one encoder in the basic training module is the at least one first encoder mentioned above.

[0136] If the basic training module has finished training, and the first network module contains encoders other than those in the basic training module, then at least one first encoder in the first network module contains, in addition to at least one encoder in the basic training module, other newly added encoders that have finished training.

[0137] In other words, based on the different training phases, in the first training phase, the basic training module is used as the module to be trained for parameter training, and the first network module is empty at this time. In the second training phase, the basic training module is used as the first network module, and new modules to be trained are added to the first network module, that is, new modules to be trained are added to the basic training module. In the third and subsequent training phases, the second network module obtained in the previous training is used as the first network module, and new modules to be trained are added to the newly obtained first network module, until the number of encoders in the coding layer reaches the preset number of network layers.

[0138] In the foregoing embodiments, the module to be trained can be at least one newly added second decoder, or a basic training module consisting of a vector representation layer, at least one first encoder, and a classification layer. When training the parameters of the module to be trained, the model parameters of the module to be trained in the second network module can be specifically obtained based on multiple training data, including:

[0139] Parameter determination steps: Determine the reference training parameters of the module to be trained in the second network module, so as to obtain the network module to be trained composed of the reference training parameters and the second network module composed of the first network module whose model parameters are known;

[0140] The multiple training data are sequentially input into the second network module, and the prediction results corresponding to the multiple training data are calculated respectively.

[0141] If the prediction results corresponding to the multiple training data satisfy the preset training objective, then the reference training parameters are determined as the model parameters of the module to be trained, so as to obtain all the model parameters of the second network module;

[0142] If the prediction results corresponding to the multiple training data do not meet the preset training objective, the reference training parameters of the module to be trained in the second network module are adjusted based on the prediction results corresponding to the multiple training data, and the process proceeds to the parameter determination step.

[0143] During initial training, since the model parameters of the trainable module in the second network module are unknown, these parameters can be initialized. The reference training parameters for the trainable module in the second network module then become the initial training parameters. During training, the model parameters of the trainable module in the second network module need to be continuously adjusted. At this point, the reference model parameters for the trainable module in the second network module become the intermediate training parameters generated during the training process.

[0144] In some embodiments, after sequentially inputting the plurality of training data into the second network module and calculating the prediction results corresponding to the plurality of training data respectively, the method further includes:

[0145] Based on the prediction results corresponding to the multiple training data and the result identifiers corresponding to the multiple training data, the current convergence information of the second network module is determined.

[0146] If the prediction results corresponding to the multiple training data satisfy the preset training objective, then the reference training parameters are determined as the model parameters of the module to be trained, so that all model parameters of the second network module are obtained, including:

[0147] If the current convergence information of the second network module satisfies the convergence condition, then the reference training parameters are determined as the model parameters of the module to be trained, so as to obtain all the model parameters of the second network module;

[0148] If the prediction results corresponding to the multiple training data do not meet the preset training objective, then the reference training parameters of the module to be trained in the second network module are adjusted based on the prediction results corresponding to the multiple training data, and the process proceeds to the parameter determination step, including:

[0149] If the current convergence information of the second network module does not meet the convergence condition, the reference training parameters of the module to be trained in the second network module are adjusted based on the prediction results corresponding to the multiple training data, and the process proceeds to the parameter determination step.

[0150] In the specific training process, if the module to be trained is the basic training module, then the first network module does not exist at this time. That is, the reference training parameters of the module to be trained are determined to obtain the second network module composed of the network module to be trained by the reference training parameters. The subsequent training steps are the same.

[0151] In one possible design, determining the current convergence information of the second network module based on the prediction results corresponding to the multiple training data and the result identifiers corresponding to the multiple training data may specifically include:

[0152] For any training data, the result identifier corresponding to the training data and the training data are input into the objective function to calculate the loss data corresponding to the training data, so as to obtain the loss data corresponding to the multiple training data respectively.

[0153] Based on the loss data corresponding to the multiple training data, the loss result of the second network module is determined;

[0154] Specifically, determining whether the current convergence information of the second network module meets the convergence conditions may include: judging whether the loss result meets a preset loss threshold.

[0155] If the loss result meets the preset loss threshold, then the current convergence information of the second network module is determined to meet the convergence condition; if the loss result does not meet the loss threshold, then the current convergence information of the second network module is determined to not meet the convergence condition.

[0156] The loss threshold can be preset. If the loss result is the loss error data, it is generally better to have as small an error data as possible to ensure the accuracy of the model. A loss result satisfying the preset loss threshold can actually mean that the loss error data is less than the loss threshold. The loss threshold can be set to, for example, 0.1, 0.05, etc. If the loss result is the accuracy, it is generally better to have as high an accuracy as possible to ensure the accuracy of the model. A loss result satisfying the preset loss threshold can actually mean that the accuracy is greater than the loss threshold. The loss threshold can be set to, for example, 0.98, 0.95, etc.

[0157] In practical applications, training of model parameters can be triggered through user interaction.

[0158] As an example, step 101: determining the first network module with known model parameters may specifically include:

[0159] Detect the training request triggered by the first user;

[0160] In response to the training request, a first network module with known model parameters is identified.

[0161] In some embodiments, after obtaining the model parameters of the module to be trained, the model parameters can be output to a first user, and output regarding the model parameters can be performed to prompt the first user to confirm the model parameters. After obtaining the model parameters of the module to be trained in the second network module based on multiple training data, the method may further include:

[0162] The model parameters of the module to be trained in the second network module are output to the first user so that the first user can confirm the model parameters of the module to be trained in the second network module.

[0163] like Figure 4 The diagram shown is a flowchart of another embodiment of a data processing method provided in this application. The method may include:

[0164] 401: Determine if there is a first network module with known model parameters. If yes, proceed to step 402; otherwise, proceed to step 403.

[0165] 402: Determine the first network module with known model parameters, then proceed to step 404.

[0166] 403: Obtain the basic training module with unknown parameters as the training module in the second network module, and then proceed to step 406.

[0167] 404: Obtain the newly added training module for the first network module.

[0168] 405: Construct a second network module based on the first network module and the module to be trained.

[0169] 406: Based on multiple training data, the model parameters of the module to be trained in the second network module are obtained through training.

[0170] 407: Determine whether the second network module meets the model construction conditions. If yes, proceed to step 408; otherwise, proceed to step 409.

[0171] 408: The second network module is determined to be the target network model.

[0172] 409: Treat the second network module as the new first network module and return to step 404.

[0173] For ease of understanding, please refer to Figure 5 The technical solution of this application embodiment will be described in detail using a BERT model with a target network module of 12 layers as an example.

[0174] The first user triggers a training request using computer M1. Computer M1 can execute operation 501, sending the training request to server M2. Server M2 receives the training request and executes operation 502: responding to the training request and starting the training of the BERT model parameters.

[0175] Assuming the encoder step size is set to 4, the BERT model's encoding layer contains 12 encoders, and the BERT model can be trained in three separate steps.

[0176] Phase 1 training:

[0177] Server M2 can execute operation 503 to obtain the basic training module with unknown parameters as the trainable module in the second network module. This basic training module may include a vector representation layer, an encoding layer consisting of four encoders, and a classification layer. The parameters of the basic training module are unknown. Server M2 can execute operation 504 to train and obtain the model parameters of the trainable module in the second network module based on multiple training data.

[0178] Four encoders did not meet the model construction conditions, so the process entered the second training phase:

[0179] Server M2 can execute operation 505, using the second network module as a new first network module, that is, using the base training module with known model parameters as the first network module. The four encoders in the encoding layer of this first network module are four first encoders. Next, server M1 executes operation 506, obtaining the newly added training module for this first network module. Since the encoder stride is 4, the training module can be four second encoders. Continuing with operation 507, the training module, i.e., the four new second encoders, is added to the encoding layer of the first network module, resulting in a second network module consisting of an encoding layer composed of four first encoders and four second encoders. The model parameters of the four first encoders are known, while the model parameters of the four second encoders are unknown. Server M1 executes operation 508, training the four second encoders in the second network module based on multiple training data to obtain the model parameters of these second encoders. At this point, the model parameters of the encoding layer composed of the eight encoders in the second network module are all known.

[0180] Eight encoders did not meet the model construction conditions, so the training proceeded to the third stage.

[0181] Server M2 can execute operation 509, treating the second network module as a new first network module, which includes an encoding layer consisting of 8 first encoders. Then, server M2 can execute operation 510, obtaining the newly added training module for the first network module. Since the encoder stride is 4, the training module can consist of 4 second encoders. Server M2 can execute operation 511, adding the training module, i.e., the 4 new second encoders, to the encoding layer of the first network module, obtaining a second network module containing an encoding layer consisting of 8 first encoders and 4 second encoders. The model parameters of the 8 first encoders are known, while the model parameters of the 4 second encoders are unknown. Server M1 can execute operation 512, training the model parameters of the 4 second encoders in the second network module based on multiple training data. At this point, the model parameters of the encoding layer consisting of 12 encoders in the second network module are known. Since the 12 encoders satisfy the model composition conditions, the second network module can be determined as the target network model, thus completing the training of the target network model's model parameters.

[0182] In practical applications, refer to Figure 6 Taking a tablet computer as an example, this application describes in detail the usage process of the target network model in this embodiment. The mobile phone M3 can execute operation 601 to detect the pending statement input by the second user, for example, "What's the weather like today?". The server M2 can obtain the pending statement, execute operation 602 to obtain the target model parameters of the trained target network model; and execute operation 602 again to input the pending statement into the target network model corresponding to the target model parameters, and calculate the target information corresponding to the pending statement, for example, "Today is a sunny day". Afterwards, the server M2 can execute operation 604 to send the target information "Today is a sunny day" to the mobile phone M3, and the mobile phone M3 can output the target information "Today is a sunny day" through its display screen.

[0183] like Figure 7 The diagram shown is a structural schematic of one embodiment of a data processing apparatus provided in this application. The apparatus may include:

[0184] Model determination module 701: Used to determine the first network module whose model parameters are known;

[0185] Model addition module 702: used to obtain a new trainable module to be added to the first network module; wherein, the model parameters of the trainable module are unknown;

[0186] A new construction module 703 is added: used to construct a second network module based on the first network module and the module to be trained;

[0187] Parameter training module 704: used to train and obtain the model parameters of the module to be trained in the second network module based on multiple training data.

[0188] As one embodiment, the device may further include:

[0189] The first judgment module is used to determine whether the second network module meets the model construction conditions;

[0190] The first result module is used to determine the second network module as the target network model if the result is true.

[0191] The second result module is used to, if not, treat the second network module as the new first network module and return it to the model addition module.

[0192] In some embodiments, the first result module is further configured to: determine the model parameters of the first network module and the model parameters of the module to be trained as the target model parameters of the target network model.

[0193] In some embodiments, the first determination module may specifically be used to: determine whether the number of encoders in the coding layer of the second network module reaches a preset number of network layers.

[0194] In some embodiments, the added building module may include:

[0195] The first determining unit is used to determine the vector representation layer, the first encoding layer, and the classification layer in the first network module; wherein the vector parameters of the vector representation layer, the first encoding parameters of the first encoder, and the classification parameters of the classification layer are known.

[0196] The first addition unit is used to add the module to be trained to the first coding layer to obtain the second coding layer;

[0197] The second determining unit is used to determine the second network module consisting of the vector representation layer, the second encoding layer, and the classification layer.

[0198] The parameter training module may include:

[0199] The first training unit is used to train the model parameters of the module to be trained in the encoding layer of the second network module based on multiple training data.

[0200] As yet another embodiment, the model determination module may include:

[0201] The first judgment unit is used to determine whether there is a first network module with known model parameters;

[0202] The first result unit is used to execute the first network module step of determining that the model parameters are known if the result is true.

[0203] The second result unit is used to, if not, obtain the unknown basic training module as the training module in the second network module, and proceed to the step of training the model parameters of the training module in the second network module based on multiple training data.

[0204] In some embodiments, the basic training module includes: a vector representation layer, an encoding layer composed of a basic encoder, and a classification layer; wherein the vector parameters of the vector representation layer, the first encoding parameters of the basic encoding layer composed of at least one encoder, and the classification parameters of the classification layer are unknown.

[0205] The second result unit can be specifically used for:

[0206] If not, the basic training module with unknown parameters is obtained as the training module in the second network module; based on multiple training data, the vector parameters of the vector representation layer, the encoding parameters of the basic encoding layer composed of at least one encoder, and the classification parameters of the classification layer in the basic training module are trained.

[0207] As another embodiment, the parameter training module may include:

[0208] A parameter determination unit is used to determine reference training parameters of the module to be trained in the second network module, so as to obtain the network module to be trained composed of the reference training parameters and the second network module composed of the first network module whose model parameters are known.

[0209] The result prediction unit is used to input the plurality of training data into the second network module in sequence, and calculate the prediction results corresponding to the plurality of training data respectively.

[0210] The third determining unit is used to determine the reference training parameters as the model parameters of the module to be trained if the prediction results corresponding to the plurality of training data satisfy the preset training objective, so as to obtain all the model parameters of the second network module.

[0211] The fourth determining unit is used to adjust the reference training parameters of the module to be trained in the second network module based on the prediction results corresponding to the multiple training data if the prediction results corresponding to the multiple training data do not meet the preset training objective, and then proceed to the parameter determining step to continue execution.

[0212] As yet another embodiment, the model determination module may include:

[0213] The request detection unit is used to detect training requests triggered by the first user.

[0214] A request-response unit is used to respond to the training request and identify a first network module whose model parameters are known.

[0215] In some embodiments, the device may further include:

[0216] The parameter determination module is used to output the model parameters of the module to be trained in the second network module to the first user, so that the first user can confirm the model parameters of the module to be trained in the second network module.

[0217] As one embodiment, the device may further include:

[0218] The parameter acquisition module is used to obtain the target model parameters of the target network model.

[0219] The statement detection module is used to detect the statement to be processed input by the second user;

[0220] The first calculation module is used to input the statement to be processed into the target network model corresponding to the target model parameters, and calculate the target information corresponding to the statement to be processed.

[0221] The first output module is used to output the target information to the second user.

[0222] As yet another embodiment, the device may further include:

[0223] The parameter acquisition module is used to obtain the target model parameters of the target network model.

[0224] The user information module is used to determine the user information of a third user.

[0225] The second calculation module is used to input the user information into the target network model corresponding to the target model parameters, and calculate the first recommended content that matches the third user.

[0226] The second output module is used to output the first recommended content to the third user.

[0227] As yet another embodiment, the device may further include:

[0228] The parameter acquisition module is used to obtain the target model parameters of the target network model.

[0229] The image acquisition module is used to determine the image to be processed input by the fourth user;

[0230] The third calculation module is used to input the image to be processed into the target network model corresponding to the target model parameters, and calculate the second recommended content that matches the image to be processed.

[0231] The third output module is used to output the second recommended content to the fourth user.

[0232] Figure 7 The data processing device can perform Figures 1-4 The implementation principle and technical effects of the data processing method described in the illustrated embodiments will not be repeated here. The specific methods by which each module and unit of the data processing device in the above embodiments performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0233] In practical applications, Figure 7 The illustrated embodiment can be configured as a computing device, see reference. Figure 8 This is a schematic diagram of the structure of one embodiment of a computing device provided in this application. The device may include: a storage component 801 and a processing component 802; the storage component 801 is used to store one or more computer instructions; the one or more computer instructions are invoked by the processing component 802 to execute. Figures 1-4 Any of the data processing methods described herein.

[0234] The processing component 802 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0235] Storage component 801 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0236] Of course, computing devices may also include other components, such as input / output interfaces and communication components. Input / output interfaces provide an interface between processing components and peripheral interface modules, which can be output devices, input devices, etc. Communication components are configured to facilitate wired or wireless communication between the computing device and other devices.

[0237] Furthermore, embodiments of this application also provide a storage medium, including: a computer-readable storage medium storing a computer program, which, when executed by a computer, can perform the above-described functions. Figures 1-4 Any of the data processing methods provided in the embodiments shown.

[0238] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0239] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. This application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0240] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Identify the first network module whose model parameters are known; Obtain the newly added training module for the first network module; wherein the model parameters of the training module are unknown; Based on the first network module and the module to be trained, a second network module is constructed; Based on multiple training data, the model parameters of the module to be trained in the second network module are obtained; wherein, the training data includes any one of the following: the statement to be processed, user information, and the image to be processed; The step of constructing the second network module based on the first network module and the module to be trained includes: The vector representation layer, the first encoding layer, and the classification layer in the first network module are determined; wherein the vector parameters of the vector representation layer, the first encoding parameters of the first encoding layer, and the classification parameters of the classification layer are known; The module to be trained is added to the first encoding layer to obtain the second encoding layer; The second network module, consisting of the vector representation layer, the second encoding layer, and the classification layer, is determined. The process of training the model parameters of the module to be trained in the second network module based on multiple training data includes: The model parameters of the module to be trained in the second encoding layer of the second network module are trained based on multiple training data.

2. The method according to claim 1, characterized in that, Also includes: Determine whether the second network module meets the model construction conditions; If so, then the second network module is determined to be the target network model; If not, the second network module is used as the new first network module, and the process returns to the step of obtaining the newly added training module for the first network module.

3. The method according to claim 2, characterized in that, If so, after determining that the second network module is the target network model, the method further includes: The model parameters of the first network module and the model parameters of the module to be trained are determined as the target model parameters of the target network model.

4. The method according to claim 2, characterized in that, The determination of whether the second network module meets the model construction conditions includes: Determine whether the number of encoders in the second coding layer of the second network module has reached the preset number of network layers.

5. The method according to claim 1, characterized in that, The first network module whose model parameters are known includes: Determine if a first network module with known model parameters exists; If so, then execute the step of determining that the first network module has known model parameters; If not, the basic training module with unknown parameters is obtained as the training module in the second network module, and the process proceeds to the step of training the model parameters of the training module in the second network module based on multiple training data.

6. The method according to claim 5, characterized in that, The basic training module includes: a vector representation layer, an encoding layer composed of a basic encoder, and a classification layer; wherein, the vector parameters of the vector representation layer, the first encoding parameters of the basic encoding layer composed of at least one encoder, and the classification parameters of the classification layer are unknown; If not, then the step of obtaining the unknown basic training module as the training module in the second network module, and proceeding to train the model parameters of the training module in the second network module based on multiple training data specifically includes: If not, the basic training module with unknown parameters is obtained as the training module in the second network module; Based on multiple training data, the vector parameters of the vector representation layer in the basic training module, the first encoding parameters of the basic encoding layer composed of at least one encoder, and the classification parameters of the classification layer are obtained through training.

7. The method according to claim 1, characterized in that, The process of training the model parameters of the module to be trained in the second network module based on multiple training data includes: Parameter determination steps: Determine the reference training parameters of the module to be trained in the second network module, so as to obtain the module to be trained composed of the reference training parameters and the second network module composed of the first network module whose model parameters are known; The multiple training data are sequentially input into the second network module, and the prediction results corresponding to the multiple training data are calculated respectively. If the prediction results corresponding to the multiple training data satisfy the preset training objective, then the reference training parameters are determined as the model parameters of the module to be trained, so as to obtain all the model parameters of the second network module; If the prediction results corresponding to the multiple training data do not meet the preset training objective, the reference training parameters of the module to be trained in the second network module are adjusted based on the prediction results corresponding to the multiple training data, and the process proceeds to the parameter determination step.

8. The method according to claim 1, characterized in that, The first network module whose model parameters are known includes: Detect the training request triggered by the first user; In response to the training request, a first network module with known model parameters is identified.

9. The method according to claim 8, characterized in that, After obtaining the model parameters of the module to be trained in the second network module based on multiple training data, the process further includes: The model parameters of the module to be trained in the second network module are output to the first user so that the first user can confirm the model parameters of the module to be trained in the second network module.

10. The method according to claim 3, characterized in that, include: Obtain the target model parameters of the target network model; Detect the pending statement input by the second user; The statement to be processed is input into the target network model corresponding to the target model parameters, and the target information corresponding to the statement to be processed is calculated. The target information is output to the second user.

11. The method according to claim 3, characterized in that, include: Obtain the target model parameters of the target network model; Determine the user information of the third user; The user information is input into the target network model corresponding to the target model parameters, and the first recommended content matching the third user is calculated. The first recommended content is output to the third user.

12. The method according to claim 3, characterized in that, include: Obtain the target model parameters of the target network model; Determine the image to be processed input by the fourth user; The image to be processed is input into the target network model corresponding to the target model parameters, and second recommended content matching the image to be processed is calculated. The second recommended content is output to the fourth user.

13. A data processing apparatus, characterized in that, include: The model determination module is used to determine the first network module whose model parameters are known. The model adds a module, which is used to obtain a new training module to be added to the first network module; wherein the model parameters of the training module are unknown. A new construction module is added to construct a second network module based on the first network module and the module to be trained. The parameter training module is used to train and obtain the model parameters of the module to be trained in the second network module based on multiple training data; wherein, the training data includes any one of the following: the statement to be processed, user information, and the image to be processed; The new build modules include: The first determining unit is used to determine the vector representation layer, the first encoding layer, and the classification layer in the first network module; wherein the vector parameters of the vector representation layer, the first encoding parameters of the first encoding layer, and the classification parameters of the classification layer are known. The first addition unit is used to add the module to be trained to the first coding layer to obtain the second coding layer; The second determining unit is used to determine the second network module consisting of the vector representation layer, the second encoding layer, and the classification layer; The parameter training module includes: The first training unit is used to train the model parameters of the module to be trained in the second encoding layer of the second network module based on multiple training data.

14. A computing device, characterized in that, include: A storage component and a processing component; the storage component is used to store one or more computer instructions; the one or more computer instructions are invoked by the processing component to execute the data processing method according to any one of claims 1 to 12.

15. A storage medium, characterized in that, include: A computer-readable storage medium storing a computer program that, when executed by a computer, implements the data processing method according to any one of claims 1 to 12.

Images