Natural language prediction task processing method and device, equipment and storage medium

Abstract

The invention provides a natural language prediction task processing method and device, equipment and a storage medium, and can be applied to the technical field of artificial intelligence. According to the method, to-be-processed data of a natural language prediction task is obtained, the to-be-processed data is input into a language prediction model, a processing result output by the language prediction model is obtained, and a pre-trained depth tree intersection model is deployed in the language prediction model. The depth tree intersection model is used for determining the coding parameters of the current input data of the language prediction model, and according to the method, the parameter information output by the depth tree intersection model is adopted to replace the original coding parameters of the language prediction model, so that the resource demand participating in calculation in the processing process of the to-be-processed data is reduced; the accuracy of language prediction model data processing is improved, and the processing efficiency of the natural language prediction task is also improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for processing natural language prediction tasks. Background Art

[0002] A language model is an abstract mathematical model of a language based on objective language facts. It is a correspondence relationship whose task is to predict the probability of a sentence appearing in a language. Therefore, language models are widely used in many fields such as machine translation, information retrieval, question answering, and abstracts. The stronger the language model's ability to predict language samples, the more accurate the prediction results it outputs.

[0003] At present, language prediction models have been widely used in natural language processing. The basis of language prediction models is Bidirectional Encoder Representations from Transformers (BERT). Each layer of the existing BERT model is fully connected, and all training parameters of each layer will be applied to all inputs of this layer. The parameters are huge, the computing pressure of the equipment is also high, and the processing efficiency of natural language prediction tasks is low. Summary of the invention

[0004] The present application provides a natural language prediction task processing method, apparatus, device and storage medium to solve the problem of low processing efficiency in the prior art.

[0005] In a first aspect, the present application provides a method for processing a natural language prediction task, the method comprising:

[0006] Get the data to be processed for the natural language prediction task;

[0007] Inputting the data to be processed into a language prediction model to obtain a processing result output by the language prediction model;

[0008] Among them, a pre-trained deep tree intersection model is deployed in the language prediction model, and the deep tree intersection model is used to determine the encoding parameters of the current input data of the language prediction model.

[0009] Optionally, the inputting the data to be processed into a language prediction model to obtain a processing result output by the language prediction model includes:

[0010] Acquire a first encoding parameter of at least one first encoding layer in the language prediction model, and perform data processing on the data to be processed according to the first encoding parameter to obtain processed data;

[0011] According to the deep tree intersection model, a second encoding parameter of the language prediction model is determined, and data processing is performed on the processed data according to the second encoding parameter to obtain a processing result of the natural language prediction task, wherein the second encoding parameter is an encoding parameter of the current input data of the language prediction model.

[0012] Optionally, determining the second encoding parameter of the language prediction model according to the deep tree intersection model includes:

[0013] Inputting the data to be processed into the input node of each tree structure in the deep tree intersection model to obtain multiple leaf node scores of each tree structure;

[0014] Based on the scores of multiple leaf nodes of each tree structure, select the leaf node with the highest score of each leaf node of each tree structure as the target leaf node;

[0015] The parameter corresponding to the target leaf node is used as the second encoding parameter corresponding to the language prediction model at the current moment.

[0016] Optionally, inputting the data to be processed into an input node of each tree structure in the deep tree intersection model to obtain multiple leaf node scores of each tree structure includes:

[0017] The data to be processed is input into the input node of each tree structure, and based on linear transformation and Gumbel distribution function, multiple leaf node scores of each tree structure are obtained.

[0018] Optionally, the parameters corresponding to the target leaf node of each tree structure include a weight parameter and a bias parameter; and using the parameters corresponding to the target leaf node as the second encoding parameters corresponding to the language prediction model at the current moment includes:

[0019] Performing a sum or product operation on the weight parameters corresponding to the target leaf nodes of each tree structure to obtain a first logical parameter;

[0020] Performing a summation or product operation on the deviation parameters corresponding to the target leaf nodes of each tree structure to obtain a second logic parameter;

[0021] According to the first logical parameter and the second logical parameter, a target parameter corresponding to the target leaf node is determined, and the target parameter is used as a second encoding parameter corresponding to the language prediction model at the current moment.

[0022] In a second aspect, the present application provides a processing device for a natural language prediction task, the device comprising:

[0023] An acquisition module is used to obtain the data to be processed for the natural language prediction task;

[0024] A processing module, used for inputting the data to be processed into a language prediction model to obtain a processing result output by the language prediction model;

[0025] Among them, a pre-trained deep tree intersection model is deployed in the language prediction model, and the deep tree intersection model is used to determine the historical weight of the historical input data of the language prediction model or determine the encoding parameters of the current input data of the language prediction model.

[0026] Optionally, the acquisition module is further used to acquire a first encoding parameter of at least one first encoding layer in the language prediction model;

[0027] The processing module is further used to perform data processing on the data to be processed according to the first encoding parameter to obtain processed data;

[0028] The processing module is also used to determine a second encoding parameter of the language prediction model according to the deep tree intersection model, and perform data processing on the processed data according to the second encoding parameter to obtain a processing result of the natural language prediction task.

[0029] Optionally, the processing module is further used to input the data to be processed into an input node of each tree structure in the deep tree intersection model to obtain scores of multiple leaf nodes of each tree structure;

[0030] The processing module is further configured to select, based on the scores of the multiple leaf nodes of each tree structure, a leaf node with the highest leaf node score of each tree structure as a target leaf node;

[0031] The processing module is further used to use the parameter corresponding to the target leaf node as the second encoding parameter corresponding to the language prediction model at the current moment.

[0032] Optionally, the processing module is further used to input the data to be processed into the input node of each tree structure, and obtain multiple leaf node scores of each tree structure based on linear transformation and Gumbel distribution function.

[0033] Optionally, the parameters corresponding to the target leaf node of each tree structure include a weight parameter and a bias parameter;

[0034] The processing module is further used to perform a sum or product operation on the weight parameters corresponding to the target leaf nodes of each tree structure to obtain a first logical parameter;

[0035] The processing module is further used to perform a sum or product operation on the deviation parameters corresponding to the target leaf nodes of each tree structure to obtain a second logic parameter;

[0036] The processing module is further used to determine a target parameter corresponding to the target leaf node according to the first logical parameter and the second logical parameter, and use the target parameter as a second encoding parameter corresponding to the language prediction model at a current moment.

[0037] In a third aspect, the present application provides a processing device for a natural language prediction task, the device comprising:

[0038] Memory;

[0039] processor;

[0040] Wherein, the memory stores computer-executable instructions;

[0041] The processor executes the computer-executable instructions stored in the memory to implement the method for processing the natural language prediction task as described in the first aspect and various possible implementations of the first aspect.

[0042] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement a method for processing a natural language prediction task as described in the first aspect and various possible implementations of the first aspect.

[0043] The processing method of the natural language prediction task provided in the present application obtains the data to be processed of the natural language prediction task, inputs the data to be processed into a language prediction model, and obtains the processing result output by the language prediction model. A pre-trained deep tree intersection model is deployed in the language prediction model, and the deep tree intersection model is used to determine the encoding parameters of the current input data of the language prediction model. The method uses the parameter information output by the deep tree intersection model to replace the original encoding parameters of the language prediction model, thereby reducing the resource requirements involved in the calculation during the processing of the data to be processed, improving the accuracy of the language prediction model data processing, and also improving the processing efficiency of the natural language prediction task. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] Figure 1 The structure of the language prediction model in the prior art provided by this application is shown as follows Figure 1 ;

[0046] Figure 2 Schematic diagram of the process of processing the natural language prediction task provided in this application Figure 1 ;

[0047] Figure 3 The structure of the language prediction model in the prior art provided by this application is shown as follows Figure 2 ;

[0048] Figure 4 Schematic diagram of the process of processing the natural language prediction task provided in this application Figure 2 ;

[0049] Figure 5 Schematic diagram of the process of processing the natural language prediction task provided in this application Figure 3 ;

[0050] Figure 6 A schematic diagram of a tree structure of a three-level single tree full binary structure provided in this application;

[0051] Figure 7 A schematic diagram of the structure of the two-dimensional deep tree intersection model provided in this application;

[0052] Figure 8 A schematic diagram of the structure of a processing device for a natural language prediction task provided in an embodiment of the present application;

[0053] Fig. 9 A schematic diagram of the structure of a processing device for a natural language prediction task provided in an embodiment of the present application.

[0054] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments made by ordinary technicians in this field under the enlightenment of the embodiments belong to the scope of protection of the present application.

[0056] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0057] Next, we will first introduce the language prediction model that is widely used in natural language prediction tasks in the prior art.

[0058] Figure 1 A schematic diagram of the structure of the language prediction model in the prior art provided in the embodiment of the present application Figure 1 .like Figure 1 As shown in the figure, the input sequence of the language prediction model is a sentence pair consisting of two sentences: Sentence 1 and Sentence 2. [CLS] is the start mark, which is used to indicate the summary information of the entire sequence. Tok1...TokN are multiple word units in the input sequence of Sentence 1, and Tok1...TokM are multiple word units in the input sequence of Sentence 2. SEP is used as a sentence separation mark to separate and identify the two sentences Sentence 1 and Sentence 2. Then Tok1...TokN and Tok1...TokM will pass through the embedding layer. There are three layers of embedding information in the structure of the language prediction model: word embedding information, sentence embedding information, and position embedding information, and E is obtained. [CLS] 、E 1 、E N 、E [SEP] 、E 1 ′、E M '. The language prediction model transmits the multiple embedding information to the corresponding multiple encoder layers in the language prediction model. Finally, the masked word in the input sequence of the language prediction model is output, and whether sentence 2 is the next sentence of sentence 1, completing the natural language prediction task.

[0059] The language prediction model consists of 12 layers of encoding, namely Z x=12, all parameters corresponding to the encoder of each layer are applied to the input of this layer, so as to obtain the output value of this layer, and the input data is transferred from bottom to top for calculation, and the result is transferred through the feedforward neural network and handed over to the next encoder. Therefore, in the prior art, in the process of natural language processing, the number of model parameters is huge, and the number of parameters involved in the calculation is large, the operation efficiency is low, and a more powerful central processing unit (CPU) is required.

[0060] In view of this, the present application proposes a method for processing natural language prediction tasks. The method deploys a pre-trained deep tree intersection model in a language prediction model so that when the language prediction model processes data, it uses the parameter information output by the deep tree intersection model to replace the original encoding parameters, thereby reducing the resource requirements involved in the calculation during the processing of the data to be processed, improving the accuracy of the language prediction model data processing, and also improving the processing efficiency of the natural language prediction task.

[0061] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0062] Figure 2 Schematic diagram of the process of processing the natural language prediction task provided in the embodiment of the present application Figure 1 .like Figure 2 As shown, the processing method of the natural language prediction task provided by this application includes:

[0063] S101: Obtaining data to be processed for a natural language prediction task;

[0064] The data to be processed may be, for example, speech data for model training. The data to be processed may be obtained from a database or through a network. This application does not limit this.

[0065] S102: Inputting the data to be processed into a language prediction model to obtain a processing result output by the language prediction model.

[0066] Among them, a pre-trained deep tree intersection model is deployed in the language prediction model, and the deep tree intersection model is used to determine the encoding parameters of the current input data of the language prediction model.

[0067] The deep tree intersection model includes multiple tree structures, each of which includes an input node, a transformation node and a leaf node. The input node of each tree structure is used to input data to be processed, the transformation node of each tree structure is used to perform linear transformation on the data to be processed, and the leaf node of each tree structure is used to output a leaf node score based on the data after the linear transformation. The parameters corresponding to the leaf node with the highest score in the leaf nodes of each tree structure can be used as encoding parameters of the current input data of the language prediction model.

[0068] It should be understood that the tree structure in the deep tree intersection model has at least three levels. The top is the first level, which serves as the input of the tree structure and is composed of an input node; the middle level can be one level or multiple levels, and each middle level is composed of multiple transformation nodes; the bottom is the last level, which is composed of multiple leaf nodes.

[0069] The processing method of the natural language prediction task provided in this embodiment obtains the data to be processed of the natural language prediction task, inputs the data to be processed into the language prediction model, and obtains the processing result output by the language prediction model. The language prediction model is deployed with a pre-trained deep tree intersection model, and the deep tree intersection model is used to determine the encoding parameters of the current input data of the language prediction model. The method uses the parameter information output by the deep tree intersection model to replace the original encoding parameters of the language prediction model, thereby reducing the resource requirements involved in the calculation during the processing of the data to be processed, improving the accuracy of the language prediction model data processing, and also improving the processing efficiency of the natural language prediction task.

[0070] Figure 3 The structure of the language prediction model in the prior art provided in this embodiment is shown as follows: Figure 2 .like Figure 3 As shown in the figure, the process of language prediction model outputting data is as follows: input data such as Sentence 1 and Sentence 2 are divided into multiple word units Tok1...TokN and Tok1...TokM, which are input into the input layer of the language prediction model. Then Tok1...TokN and Tok1...TokM are passed through the embedding layer. There are three layers of embedding information in the language prediction model structure: word embedding information, sentence embedding information, and position embedding information. [CLS] 、E 1 、E N 、E [SEP] 、E 1 ′、E M ′. The language prediction model transmits multiple words from bottom to top to multiple encoder layers N according to multiple embedding information. X, the encoder is responsible for mapping the natural language sequence into a hidden layer, which is a mathematical expression containing the natural language sequence. Among them, the multiple encoder layers N of the language prediction model X The last encoder layer in the deep tree intersection model is deployed with a pre-trained deep tree intersection model. Since the language prediction model can obtain the vector representation of all word units corresponding to the input data, the last encoder layer of the language prediction model can obtain the vector features of all word units, and then use the parameter information output by the deep tree intersection model to replace the parameters of the original encoder to reduce the number of encoder layers, that is, N X <Z x .

[0071] The multi-layer encoders in this language prediction model are superimposed and transmitted layer by layer. The last layer of encoders has the feature value of the input data, which is sent to the corresponding decoder. The decoder maps the hidden layer into a natural language sequence, allowing us to complete various natural language prediction tasks, such as sentiment classification, named entity recognition, semantic relationship extraction, summary generation, machine translation, etc.

[0072] It can be understood that the existing encoding parameters are all fixed values, and their determination can be, for example, preset or default. In the language prediction model of this embodiment, a deep tree intersection model is deployed, and the deep tree intersection model is used to determine the current encoding parameters. That is, the deep tree intersection model can determine different encoding parameters based on the input data, thereby improving the accuracy of the language prediction model data processing. Since the data processing is more accurate, the purpose of reducing the resource requirements involved in the calculation can be further achieved.

[0073] The processing method of the natural language prediction task provided in this embodiment obtains the data to be processed of the natural language prediction task; inputs the data to be processed into a recurrent neural network model to determine the parameter information of the current input data according to a pre-trained deep tree intersection model deployed in the language prediction model, and then uses the parameter information output by the deep tree intersection model to replace the original encoding parameters, thereby reducing the resource requirements involved in the calculation during the processing of the data to be processed, improving the accuracy of the language prediction model data processing, and also improving the processing efficiency of the natural language prediction task.

[0074] Figure 4 Schematic diagram of the process of processing the natural language prediction task provided in the embodiment of the present application Figure 2 This embodiment is in Figure 2 Based on the embodiment, the processing method of the natural language prediction task is described in detail. Figure 4 As shown, the processing method of the natural language prediction task provided by this application includes:

[0075] S201: Obtaining data to be processed for a natural language prediction task.

[0076] Step S201 is similar to the above step S101 and will not be described again.

[0077] S202: Acquire a first encoding parameter of at least one first encoding layer in the language prediction model, and perform data processing on the data to be processed according to the first encoding parameter to obtain processed data.

[0078] Among them, the language prediction model has multiple encoding layers, and the data to be processed flows from bottom to top after being input into the language prediction model. The number of first encoding layers can be one or more. When the first encoding layer is located at the bottom layer of the language prediction model, it is used to process the data to be processed according to the first encoding parameter, that is, to map the natural language sequence of the data to be processed into a mathematical expression, for example, the data to be processed is represented in a vector form including tag embedding information, fragment embedding information, and position embedding information according to the first encoding parameter and the natural language sequence. The first encoding parameter can be, for example, a weight parameter or a bias parameter, and the number of first encoding parameters can be one or more.

[0079] In this step, the at least one first coding layer performs hierarchical superposition processing on the data to be processed to obtain data processed by each coding layer.

[0080] S203: Determine a second encoding parameter of the language prediction model according to the deep tree intersection model, and perform data processing on the processed data according to the second encoding parameter to obtain a processing result of the natural language prediction task.

[0081] The deep tree intersection model is deployed in the last coding layer of the language prediction model, and the second coding parameter is a coding parameter determined by the deep tree intersection model according to current input data of the language prediction model.

[0082] Since the language prediction model extracts features of the data to be processed through multiple coding layers, the last coding layer has all the features of the data to be processed. If the original coding parameters of the language prediction model are used for data processing, all the original coding parameters will be weighted according to a certain probability distribution, which reduces the processing efficiency of the natural language prediction task.

[0083] Therefore, it is necessary to process the data processed by at least one first coding layer using the second coding parameters output by the deep tree intersection model to reduce the number of coding parameters of the language prediction model.

[0084] The processing method of the natural language prediction task provided in this embodiment obtains the data to be processed of the natural language prediction task, obtains the first coding parameter of at least one first coding layer in the language prediction model, and processes the data to be processed according to the first coding parameter to obtain the processed data, determines the second coding parameter of the language prediction model according to the deep tree intersection model, and processes the processed data according to the second coding parameter to obtain the processing result of the natural language prediction task. The method uses the output result of the deep tree intersection model to replace the current coding parameter in the language prediction model, so that the result output by the language prediction model is more accurate, thereby saving the computing power of the device, reducing the resource requirements involved in the calculation during the processing of the data to be processed, and improving the processing efficiency of the natural language prediction task.

[0085] Figure 5 Schematic diagram of the process of processing the natural language prediction task provided in the embodiment of the present application Figure 2 The deep tree intersection model in this embodiment is used to determine the encoding parameters of the current input data of the language prediction model. Figure 5 As shown, the processing method of the natural language prediction task provided by this application includes:

[0086] S301: Obtaining data to be processed for a natural language prediction task.

[0087] Step S301 is similar to the above step S201 and will not be described again.

[0088] S302: Acquire a first encoding parameter of at least one first encoding layer in the language prediction model, and perform data processing on the data to be processed according to the first encoding parameter to obtain processed data.

[0089] Step S302 is similar to the above step S202 and will not be described again.

[0090] S303: Input the data to be processed into the input node of each tree structure in the deep tree intersection model to obtain multiple leaf node scores of each tree structure.

[0091] In a possible implementation, the data to be processed is input into the input node of each tree structure, and based on linear transformation and the Gumbel distribution function, multiple leaf node scores of each tree structure are obtained.

[0092] S304: Based on the scores of the multiple leaf nodes of each tree structure, select a leaf node with the highest score in each tree structure as a target leaf node.

[0093] S305: Using the parameter corresponding to the target leaf node as the second encoding parameter corresponding to the language prediction model at the current moment.

[0094] S306: Perform data processing on the processed data according to the second encoding parameter to obtain a processing result of the natural language prediction task.

[0095] Step S306 is similar to the above step S203 and will not be described again.

[0096] Figure 6 This is a tree structure diagram of a three-level single tree full binary structure, such as Figure 6 As shown in , the data of the input node is transmitted to the transformation node through linear transformation and activation function, and the data of the transformation node is transmitted to the leaf node through linear transformation and activation function. Figure 6 In the tree structure shown, the input node is the 0th layer, the transformation node is the first layer, and the leaf node is the second layer. Figure 6 , taking the data to be processed in the natural language prediction task as vector data as an example, the method provided in the embodiment of the present application is described in detail.

[0097] In one possible implementation, the transformation node can be represented as The output of the transformation node satisfies the following formula:

[0098]

[0099] Where X is the data input to the input node, i represents the transformation layer number of the current node, j represents the position of the current node in the current layer, and t represents the number of the current tree structure. is the weight parameter when the tree structure numbered t is linearly transformed to the jth position of the i-th layer, is the deviation parameter when the tree structure numbered t is linearly transformed to the jth position of the i-th layer.

[0100] In one possible implementation, a leaf node can be represented as The output of the leaf node satisfies the following formula:

[0101]

[0102] in, is the value of the transformation node in the previous layer of the leaf node, i represents the layer number of the current node, j represents the position of the current node in the current layer, and t represents the number of the current tree structure. is the weight parameter when the tree structure numbered t is linearly transformed to the jth position of the i-th layer, is the deviation parameter when the tree structure numbered t is linearly transformed to the jth position of the i-th layer.

[0103] In one possible implementation, represents the score (LeafScore) of the jth leaf node of the tth tree. The scores of the multiple leaf nodes each satisfy the following formula:

[0104]

[0105] Among them, L t represents the number of leaf nodes in the tree numbered t, represents the data of the jth leaf node in the tree numbered t, g j =-log(-log(u j )),u j ~U(0,1) is the Gumbel distribution. The smaller τ is (τ→0), the sampled vector tends to be a one-hot vector; the larger τ is (τ→∞), the sampled vector tends to be uniformly distributed.

[0106] It should be understood that the Gumbel-Softmax function can make the calculated leaf node score In the form of a one-hot vector, that is, a vector with only one element being 1 and the rest being 0, the following formula can be used to select the leaf node with the highest score as the weight parameter C corresponding to the target leaf node t and the deviation parameter d t :

[0107]

[0108] in, and Leaf nodes The corresponding weight parameters and bias parameters, is the leaf node score calculated by the above formula.

[0109] It should be understood that the weight parameters and bias parameters involved in the embodiments of the present application are obtained after training the deep tree intersection model, and the training process is the conventional training process for the deep learning model in the prior art, which will not be repeated here.

[0110] The deep tree intersection model provided in this embodiment includes multiple tree structures, each of which can calculate a target leaf node. The weight parameters and deviation parameters of the target leaf node obtained according to the data to be processed are a set of target physical parameters selected for each tree structure.

[0111] Figure 7 It is a structural diagram of the two-dimensional deep tree intersection model, such as Figure 7As shown, the model includes 2 tree structures, each tree structure includes 4 leaf nodes, that is, each tree has 4 sets of physical parameters, and the physical parameters of the 8 leaf nodes of the 2 numbers can be composed as follows Figure 5 The 16 logic parameters are shown.

[0112] The above step 305 can, for example, determine the second encoding parameter of the language prediction model according to the weight parameter and the bias parameter corresponding to the target leaf node of each tree structure. The specific implementation is as follows:

[0113] The weight parameters corresponding to the target leaf nodes of each tree structure are summed or multiplied to obtain a first logic parameter; the deviation parameters corresponding to the target leaf nodes of each tree structure are summed or multiplied to obtain a second logic parameter; according to the first logic parameter and the second logic parameter, a target parameter corresponding to the target leaf node is determined, and the target parameter is used as the second encoding parameter corresponding to the language prediction model at the current moment.

[0114] The first logic parameter and the second logic parameter are target logic parameters selected according to the data to be processed. Figure 7 As shown, although the two-dimensional deep tree intersection model can have 16 combinations of logic parameters, the logic parameter combinations selected by tree 1 and tree 2 are 5 logic parameters, that is, only one set needs to be selected for the final result operation.

[0115] It should be understood that Figure 7 The two-dimensional deep tree intersection model shown is only exemplary, and the present application does not specifically limit the number of tree structures involved in the deep tree intersection model.

[0116] In a possible implementation, the first logic parameter A and the second logic parameter b satisfy the following formula:

[0117]

[0118] Among them, T is the number of tree structures in the deep tree comparison model.

[0119] In the deep tree intersection model for classification tasks, the processing result y obtained by the to-be-processed data x based on the first logical parameter A and the second logical parameter b satisfies the probability distribution obtained by the following formula.

[0120] y=softmax(A•x+b)

[0121] The processing method of the natural language prediction task provided in this embodiment replaces the current encoding parameters in the language prediction model with the output results of the deep tree intersection model, thereby making the results output by the recurrent neural network model more accurate. At the same time, since the above method does not require every physical parameter of the leaf node to participate in the result calculation, it can save equipment computing power, reduce the resource requirements involved in the calculation during the processing of the data to be processed, and also improve the processing efficiency of the natural language prediction task.

[0122] Figure 8 Schematic diagram of the structure of the processing device for natural language prediction task provided by this application. Figure 8 As shown, the processing device 400 for the natural language prediction task includes:

[0123] An acquisition module 401 is used to acquire data to be processed for a natural language prediction task;

[0124] The processing module 402 is used to input the data to be processed into the language prediction model to obtain the processing result output by the language prediction model;

[0125] Among them, a pre-trained deep tree intersection model is deployed in the language prediction model, and the deep tree intersection model is used to determine the historical weight of the historical input data of the language prediction model or determine the encoding parameters of the current input data of the language prediction model.

[0126] Optionally, the acquisition module 401 is further used to acquire a first encoding parameter of at least one first encoding layer in the language prediction model;

[0127] The processing module 402 is further configured to perform data processing on the data to be processed according to the first encoding parameter to obtain processed data;

[0128] The processing module 402 is further used to determine a second encoding parameter of the language prediction model according to the deep tree intersection model, and perform data processing on the processed data according to the second encoding parameter to obtain a processing result of the natural language prediction task.

[0129] Optionally, the processing module 402 is further configured to input the data to be processed into an input node of each tree structure in the deep tree intersection model to obtain scores of multiple leaf nodes of each tree structure;

[0130] The processing module 402 is further configured to select, based on the scores of the multiple leaf nodes of each tree structure, a leaf node with the highest leaf node score of each tree structure as a target leaf node;

[0131] The processing module 402 is further configured to use the parameter corresponding to the target leaf node as the second encoding parameter corresponding to the language prediction model at the current moment.

[0132] Optionally, the processing module 402 is further used to input the data to be processed into the input node of each tree structure, and obtain multiple leaf node scores of each tree structure based on linear transformation and Gumbel distribution function.

[0133] Optionally, the parameters corresponding to the target leaf node of each tree structure include a weight parameter and a bias parameter;

[0134] The processing module 402 is further configured to perform a sum or product operation on the weight parameters corresponding to the target leaf nodes of each tree structure to obtain a first logic parameter;

[0135] The processing module 402 is further configured to perform a summation or product operation on the deviation parameters corresponding to the target leaf nodes of each tree structure to obtain a second logic parameter;

[0136] The processing module 402 is further configured to determine a target parameter corresponding to the target leaf node according to the first logical parameter and the second logical parameter, and use the target parameter as a second encoding parameter corresponding to the language prediction model at a current moment.

[0137] Fig. 9 A schematic diagram of the structure of the processing device for the natural language prediction task provided in this application. Fig. 9 As shown, the present application provides a processing device for a natural language prediction task. The processing device 500 for the natural language prediction task includes: a receiver 501 , a transmitter 502 , a processor 503 and a memory 504 .

[0138] Receiver 501, used for receiving instructions and data;

[0139] A transmitter 502, used for sending instructions and data;

[0140] Memory 504, used to store computer-executable instructions;

[0141] The processor 503 is used to execute the computer-executable instructions stored in the memory 504 to implement the various steps executed by the method for processing the natural language prediction task in the above embodiment. For details, please refer to the relevant description in the embodiment of the method for processing the natural language prediction task.

[0142] Optionally, the memory 504 may be independent or integrated with the processor 503 .

[0143] When the memory 504 is independently provided, the electronic device further includes a bus for connecting the memory 504 and the processor 503 .

[0144] The present application also provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, a processing method for a natural language prediction task as performed by the processing device for the natural language prediction task described above is implemented.

[0145] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0146] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0147] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for processing a natural language prediction task, characterized in that: include: Get the data to be processed for the natural language prediction task; Inputting the data to be processed into a language prediction model to obtain a processing result output by the language prediction model; Among them, a pre-trained deep tree intersection model is deployed in the language prediction model, and the deep tree intersection model is used to determine the encoding parameters of the current input data of the language prediction model.

2. The method according to claim 1, characterized in that The step of inputting the data to be processed into a language prediction model to obtain a processing result output by the language prediction model includes: Acquire a first encoding parameter of at least one first encoding layer in the language prediction model, and perform data processing on the data to be processed according to the first encoding parameter to obtain processed data; According to the deep tree intersection model, a second encoding parameter of the language prediction model is determined, and data processing is performed on the processed data according to the second encoding parameter to obtain a processing result of the natural language prediction task, wherein the second encoding parameter is an encoding parameter of the current input data of the language prediction model.

3. The method according to claim 2, characterized in that Determining a second encoding parameter of the language prediction model according to the deep tree intersection model includes: Inputting the data to be processed into the input node of each tree structure in the deep tree intersection model to obtain multiple leaf node scores of each tree structure; Based on the scores of multiple leaf nodes of each tree structure, select the leaf node with the highest score of each leaf node of each tree structure as the target leaf node; The parameter corresponding to the target leaf node is used as the second encoding parameter corresponding to the language prediction model at the current moment.

4. The method according to claim 3, characterized in that The step of inputting the data to be processed into the input node of each tree structure in the deep tree intersection model to obtain multiple leaf node scores of each tree structure includes: The data to be processed is input into the input node of each tree structure, and based on linear transformation and Gumbel distribution function, multiple leaf node scores of each tree structure are obtained.

5. The method according to claim 4, characterized in that The parameters corresponding to the target leaf nodes of each tree structure include a weight parameter and a bias parameter; and using the parameters corresponding to the target leaf nodes as the second encoding parameters corresponding to the language prediction model at the current moment includes: Performing a sum or product operation on the weight parameters corresponding to the target leaf nodes of each tree structure to obtain a first logical parameter; Performing a summation or product operation on the deviation parameters corresponding to the target leaf nodes of each tree structure to obtain a second logic parameter; According to the first logical parameter and the second logical parameter, a target parameter corresponding to the target leaf node is determined, and the target parameter is used as a second encoding parameter corresponding to the language prediction model at the current moment.

6. A processing device for natural language prediction tasks, characterized in that: The device comprises: An acquisition module is used to obtain the data to be processed for the natural language prediction task; A processing module, used for inputting the data to be processed into a language prediction model to obtain a processing result output by the language prediction model; Among them, a pre-trained deep tree intersection model is deployed in the language prediction model, and the deep tree intersection model is used to determine the historical weight of the historical input data of the language prediction model or determine the encoding parameters of the current input data of the language prediction model.

7. The device according to claim 6, characterized in that The acquisition module is further used to acquire a first encoding parameter of at least one first encoding layer in the language prediction model; The processing module is further used to perform data processing on the data to be processed according to the first encoding parameter to obtain processed data; The processing module is also used to determine a second encoding parameter of the language prediction model according to the deep tree intersection model, and perform data processing on the processed data according to the second encoding parameter to obtain a processing result of the natural language prediction task.

8. The device according to claim 6, characterized in that The processing module is further used to input the data to be processed into the input node of each tree structure in the deep tree intersection model to obtain multiple leaf node scores of each tree structure; The processing module is further configured to select, based on the scores of the multiple leaf nodes of each tree structure, a leaf node with the highest leaf node score of each tree structure as a target leaf node; The processing module is further used to use the parameter corresponding to the target leaf node as the second encoding parameter corresponding to the language prediction model at the current moment.

9. A processing device for natural language prediction tasks, characterized in that: include: Memory; processor; Wherein, the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the processing method of the natural language prediction task according to any one of claims 1 to 5.

10. A computer storage medium, characterized in that: The computer storage medium stores computer execution instructions, which, when executed by a processor, are used to implement the processing method for the natural language prediction task as described in any one of claims 1 to 5.

Images