Multi-head microstack-based reasoning language model training method and model

Abstract

The invention provides an inference language model training method and model based on a multi-head microstack, and the method comprises the steps: calling an ith feature extraction layer to obtain first input data and second input data, and carrying out the feature extraction of the first input data and the second input data, and obtaining an output semantic feature of the ith feature extraction layer; the first input data are output semantic features of the (i-1) th feature extraction layer; the second input data is a splicing result of a plurality of semantic features stored in a plurality of storage stacks; when the ith feature extraction layer is the last layer of the plurality of feature extraction layers, calculating a loss value between an output semantic feature of the ith feature extraction layer and a preset standard semantic feature; and if the loss value meets a preset convergence condition, determining the initial language model as an inference language model based on the multi-head microstack. According to the embodiment of the invention, the video memory overhead during model training can be effectively reduced, the model training speed is increased, and the modeling effect of the model is improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and specifically to a method and model for training an inference language model based on a multi-head differentiable stack. Background Art

[0002] As a mainstream natural language processing (NLP) model, the Transformer model can effectively capture long-distance dependencies. However, in the current training and application of Transformer models, the performance and expressiveness of the Transformer model are poor when processing formal languages ​​such as programming languages ​​and logical languages. Formal languages ​​typically have strict structures and rules. Existing Transformer models often cannot accurately understand the grammatical structure or logical relationships of the code in code generation or logical reasoning tasks, resulting in generation results that do not meet expectations.

[0003] However, in order to enable the Transformer model to improve the semantic recognition effect of formal language, the dimensions of each model layer are usually increased during the model training phase. However, this processing method will lead to a significant increase in the number of model parameters, significantly increase the usage of video memory, and reduce the training speed of the model. Summary of the Invention

[0004] This application proposes a method and model for training an inference language model based on a multi-head differentiable stack to solve the problem in the prior art of reduced model training speed due to increased video memory usage.

[0005] The first embodiment of the present application proposes a method for training an inference language model based on a multi-head differentiable stack, wherein the initial language model includes multiple feature extraction layers and multiple storage stacks, and the method includes:

[0006] For the i-th feature extraction layer, calling the i-th feature extraction layer to obtain first input data and second input data respectively, and performing feature extraction on the first input data and the second input data to obtain output semantic features of the i-th feature extraction layer; wherein the first input data is the output semantic features of the i-1-th feature extraction layer; the second input data is the concatenation result of multiple semantic features stored in the multiple storage stacks; the semantic features stored in each storage stack refer to the fragmented semantic features obtained after the output semantic features of the i-1-th feature extraction layer are subjected to dimensionality reduction and segmentation processing; the i is greater than or equal to 2;

[0007] When the i-th feature extraction layer is the last layer of the plurality of feature extraction layers, calculating a loss value between an output semantic feature of the i-th feature extraction layer and a preset standard semantic feature;

[0008] If the loss value satisfies a preset convergence condition, the initial language model is determined to be an inference language model based on a multi-head differentiable stack, and the inference language model based on the multi-head differentiable stack has a function of recognizing the semantics of a formal language.

[0009] In an embodiment of the present application, when the i-th feature extraction layer is not the last layer of the plurality of feature extraction layers, the method further includes:

[0010] Get the output semantic features of the i-1th feature extraction layer;

[0011] Performing dimensionality reduction processing on the output semantic features to obtain output semantic features after dimensionality reduction;

[0012] Splitting the output semantic features after dimensionality reduction into a plurality of slice semantic features;

[0013] For any slice semantic feature among the multiple slice semantic features, the slice semantic feature is stored in a corresponding storage stack; the multiple partial semantic features correspond one-to-one to the multiple storage stacks.

[0014] In an embodiment of the present application, performing dimensionality reduction processing on the output semantic feature to obtain the reduced-dimensional output semantic feature includes: adjusting the dimension value of the output semantic feature from 2048 dimensions to 64 dimensions;

[0015] The output semantic features after dimensionality reduction are divided into a plurality of slice semantic features, including: dividing the 64-dimensional output semantic features into four 16-dimensional slice semantic features.

[0016] In the embodiment of the present application, calling the i-th feature extraction layer to obtain second input data includes:

[0017] Reading a plurality of shard semantic features from the plurality of storage stacks respectively;

[0018] Identifying a plurality of sequence tags of the plurality of fragment semantic features;

[0019] splicing the plurality of fragment semantic features into a spliced ​​semantic feature according to the order of the high bits and the low bits indicated by the plurality of bit sequence labels;

[0020] The splicing semantic feature is subjected to dimensionality-upgrading processing to obtain a dimensionality-upgraded splicing semantic feature, and the dimensionality-upgraded splicing semantic feature is used as the second input data; the dimension value of the dimensionality-upgrading is the same as the dimension value of the dimensionality-reducing.

[0021] In an embodiment of the present application, splicing the plurality of slice semantic features into a spliced ​​semantic feature includes: splicing four 16-dimensional slice semantic features into a 64-dimensional semantic feature;

[0022] Performing dimensionality-upgrading processing on the spliced ​​semantic features to obtain dimensionality-upgraded spliced ​​semantic features includes: adjusting the dimension value of the 64-dimensional semantic features from 64 dimensions to 2048 dimensions.

[0023] In an embodiment of the present application, the dimension value of the output semantic feature after dimensionality reduction is determined according to the number of the multiple storage stacks and the storage space size of each storage stack.

[0024] In an embodiment of the present application, each storage stack includes multiple storage bits, and storing the shard semantic features in the corresponding storage stack includes:

[0025] For any storage bit among the multiple storage bits, based on the shard semantic features stored in the storage bit at the previous calculation moment, the shard semantic features stored in the adjacent storage bits of the storage bit, and the preset probability, determine the shard semantic features of the storage bit at the current calculation moment; the adjacent storage bits include the low-order storage bits and the high-order storage bits adjacent to the storage bit; the calculation moment refers to the moment when the corresponding feature extraction layer performs calculation and stores the output result to the storage stack, and one calculation moment corresponds to one feature extraction layer;

[0026] If the storage bit is the lowest storage bit among the multiple storage bits, the data of the lower storage bit in the adjacent storage position is the output semantic feature of the feature extraction layer corresponding to the previous calculation moment;

[0027] If the storage bit is the highest storage bit among the multiple storage bits, the data stored in the higher storage bits in the adjacent storage positions is empty.

[0028] In an embodiment of the present application, the preset probability includes a write probability, an unchanged probability, and a deletion probability. The determining of the shard semantic feature of the storage bit at the current calculation moment based on the shard semantic feature stored in the storage bit at the previous calculation moment, the shard semantic feature stored in the adjacent storage bit of the storage bit, and the preset probability includes:

[0029] Calculating a first semantic feature based on the shard semantic feature stored in the low-order storage bit at the last calculation moment and the write probability;

[0030] Calculate a second semantic feature based on the slice semantic feature stored in the storage bit at the last calculation moment and the invariant probability;

[0031] Calculate a third semantic feature based on the shard semantic feature stored in the high-order storage bit at the last calculation moment and the deletion probability;

[0032] A weighted sum is performed on the first semantic feature, the second semantic feature, and the third semantic feature to obtain the shard semantic feature stored in the storage location at the current calculation moment.

[0033] In an embodiment of the present application, the method further includes: if i=1, the first input data obtained by the i-th feature extraction layer is the initial semantic feature of the language to be recognized, and the second input data obtained by the i-th feature extraction layer is empty.

[0034] An embodiment of the second aspect of the present application provides an inference language model based on a multi-head differentiable stack, comprising an input layer, a recognition layer, and an output layer, wherein the recognition layer comprises multiple feature extraction layers and multiple storage stacks; wherein the i-th feature extraction layer is configured to obtain first input data and second input data, respectively, and perform feature extraction on the first input data and the second input data to obtain output semantic features of the i-th feature extraction layer; wherein the first input data is the output semantic features of the i-1-th feature extraction layer; the second input data is the concatenation of multiple semantic features stored in the multiple storage stacks; the semantic features stored in each storage stack are sliced ​​semantic features obtained by subjecting the output semantic features of the i-1-th feature extraction layer to dimensionality reduction and segmentation; and wherein i is greater than or equal to 2.

[0035] An embodiment of the third aspect of the present application provides a computer device, which includes a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the inference language model training method based on a multi-head differentiable stack described in the first aspect by executing the computer instructions.

[0036] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to enable a computer to execute the inference language model training method based on a multi-head differentiable stack described in the first aspect above.

[0037] Additional aspects and advantages of the present application will be given in part in the description below and in part will become apparent from the description below or learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. Throughout the accompanying drawings, the same reference numerals are used to denote the same components.

[0039] In the attached figure:

[0040] Figure 1 A schematic diagram of the structure of an inference language model based on a multi-head differentiable stack provided in one embodiment of the present application is shown;

[0041] Figure 2 A schematic diagram of the structure of the recognition layer in the inference language model based on a multi-head differentiable stack provided in one embodiment of the present application is shown;

[0042] Figure 3 A flowchart of a method for training an inference language model based on a multi-head differentiable stack provided by an embodiment of the present application is shown;

[0043] Figure 4 A flow chart of another method for training an inference language model based on a multi-head differentiable stack provided in one embodiment of the present application is shown;

[0044] Figure 5 A schematic diagram of a storage stack provided in one embodiment of the present application is shown;

[0045] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application is shown;

[0046] Figure 7 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] The following describes exemplary embodiments of the present application in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0048] It should be noted that, unless otherwise specified, the technical or scientific terms used in this application should have the common meanings understood by those skilled in the art to which this application belongs.

[0049] The following describes the technical names involved in the embodiments of this application.

[0050] The Transformer model is a deep learning model based on the self-attention mechanism. However, its effectiveness in actual training is limited, as it cannot model and complete tasks requiring higher-level formal language, which affects its performance and expressiveness.

[0051] The structure of the Transformer model mainly consists of two parts: the encoder and the decoder. Each part enhances the model's representation and learning capabilities by stacking multiple identical layers.

[0052] The encoder consists of N identical layers stacked together, each of which consists of two sublayers: a multi-head self-attention mechanism and a feedforward neural network. Residual connections and layer normalization are applied after each sublayer in the encoder to stabilize the training process. The encoder layer is primarily responsible for processing the input sequence and generating a series of encoded representations. These encoded representations capture the contextual information in the input sequence.

[0053] The decoder is similarly composed of N identical layers stacked together, but each layer contains the same two sublayers as the encoder, plus an additional multi-head attention sublayer to process the encoder output: a multi-head self-attention mechanism, a multi-head encoder-decoder attention mechanism (sublayer), and a feedforward neural network. Residual connections and layer normalization are also applied after each sublayer in the decoder. The decoder layer is responsible for generating the next output based on the encoder output and the previously generated output sequence (in the case of generation tasks).

[0054] Formal language theory is a branch of computer science that studies the abstract structure and rules of languages. The Formal Language Hierarchy describes different types of languages ​​and their generative capabilities.

[0055] According to an embodiment of the present application, an embodiment of a method for training an inference language model based on a multi-head differentiable stack is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] The embodiment of the present disclosure proposes a reasoning language model based on a multi-head differentiable stack, such as Figure 1 As shown;

[0057] The inference language model based on the multi-head differentiable stack of the embodiment of the present disclosure includes an input layer 101, a recognition layer 102 and an output layer 103;

[0058] Specifically, the input layer 101 can be used to extract preliminary semantic features from the input language to be recognized. For example, the language to be recognized can include languages ​​with abstract semantics, such as mathematical formulas, computer programming languages, and chemical formulas. The input layer 101 can convert the language to be recognized directly input by the user into initial semantic features in the form of semantic vectors, etc. The initial semantic features can be converted into vectors, etc.

[0059] The recognition layer 102 is used to extract the initial semantic features and then realize semantic recognition. Figure 2The recognition layer 102 shown includes multiple feature extraction layers and multiple storage stacks. Specifically, taking the Transformer model as an example, the recognition layer 102 can include the encoder layer and decoder layer in the Transformer model, as well as the corresponding multiple storage stacks, wherein the encoder layer can serve as the first feature extraction layer and the decoder layer as the second feature extraction layer; the output data of the first feature extraction layer can be divided into multiple partial data, and then the multiple partial data are respectively stored in the corresponding storage stacks. When the second feature extraction layer performs feature extraction, the input data is composed of the output data of the first feature extraction layer and the data obtained from the multiple storage stacks.

[0060] More specifically, the i-th feature extraction layer is used to obtain first input data and second input data respectively, and perform feature extraction on the first input data and the second input data to obtain the output semantic features of the i-th feature extraction layer; wherein the first input data is the output semantic features of the i-1-th feature extraction layer; the second input data is the concatenation result of multiple semantic features stored in the multiple storage stacks; the semantic features stored in each storage stack refer to the fragmented semantic features obtained after the output semantic features of the i-1-th feature extraction layer are subjected to dimensionality reduction and segmentation processing; and i is greater than or equal to 2;

[0061] Taking the Transformer model as an example, both the encoder and the decoder include multiple sub-layers, wherein any sub-layer of the encoder includes a multi-head self-attention mechanism, a feedforward neural network, and a residual connection layer and a layer normalization layer for stabilization. In combination with the embodiments of the present application, for the layers in the two sub-layers of the multi-head self-attention mechanism and the feedforward neural network in the encoder, the multiple output data in the multi-head self-attention mechanism can be stored in multiple storage stacks, and the input data in the feedforward neural network is composed of the output data of the multi-head self-attention mechanism and the data in the storage stack. Similarly, the decoder is similar and will not be described in detail here. The recognition layer 102 performs feature extraction, and the storage and extraction process of the storage stack is introduced in the embodiments later and will not be described in detail here.

[0062] The output layer 103 is used to integrate and output the data in the recognition layer 102.

[0063] In this embodiment, a method for training an inference language model based on a multi-head differentiable stack is provided, wherein the initial language model includes multiple feature extraction layers and multiple storage stacks; Figure 3 Flowchart of the inference language model training method based on a multi-head differentiable stack according to an embodiment of the present application, such as Figure 3 As shown, the process includes the following steps:

[0064] Step S101: for the i-th feature extraction layer, call the i-th feature extraction layer to obtain first input data and second input data respectively, and perform feature extraction through the first input data and the second input data to obtain output semantic features of the i-th feature extraction layer.

[0065] Specifically, the first input data is the output semantic feature of the i-1th feature extraction layer; the second input data is the splicing result of multiple semantic features stored in the multiple storage stacks; the semantic feature stored in each storage stack refers to the sliced ​​semantic feature obtained after the output semantic feature of the i-1th feature extraction layer is processed by dimensionality reduction and segmentation; the i is greater than or equal to 2.

[0066] In the embodiments of the present application, the trained model is invoked to perform multiple feature extractions on the initial semantic features of the language layer by layer, obtaining the output semantic features of the final layer of the multiple feature extraction layers. The semantic features of the language can be either natural language or formal language. The embodiments of the present application do not limit the specific form of the language, and those skilled in the art can determine the specific form based on actual circumstances.

[0067] In the embodiment of the present application, the initial language model can be understood as the above Figure 1 The inference language model based on a multi-head differentiable stack is constructed; when calling the initial language model to extract the initial semantic features, each feature extraction layer further extracts the output data of the previous feature extraction layer. On this basis, the "output data of the previous feature extraction layer" and the "data extracted from the storage stack" are used as the input data of the next feature extraction layer, realizing multi-dimensional feature extraction of the language to be recognized and improving the accuracy of feature extraction.

[0068] In some specific embodiments, when the feature extraction layer is the first layer among multiple feature extraction layers, that is, i=1, the first input data obtained by the i-th feature extraction layer is the initial semantic feature of the language to be recognized, and the second input data obtained by the i-th feature extraction layer is empty.

[0069] In some specific embodiments, when the i-th feature extraction layer is not the last layer of the plurality of feature extraction layers, the method further includes steps a1 to a4:

[0070] Step a1: Obtain the output semantic features of the i-1th feature extraction layer.

[0071] Step a2: performing dimensionality reduction processing on the output semantic features to obtain output semantic features after dimensionality reduction.

[0072] Specifically, for example, the dimension value of the output semantic feature is adjusted from 2048 dimensions to 64 dimensions, for example Figure 4Although the dimension of the output semantic features is reduced, the output semantic features after dimensionality reduction retain most of the core features of the output semantic features before dimensionality reduction.

[0073] Step a3: dividing the output semantic features after dimensionality reduction into multiple slice semantic features.

[0074] Step a4: for any slice semantic feature among the multiple slice semantic features, store the slice semantic feature in a corresponding storage stack; the multiple partial semantic features correspond one-to-one to the multiple storage stacks.

[0075] In the above steps a2 and a3, a multi-head mechanism is used to divide the output semantic features after dimensionality reduction into multiple slice semantic features according to the preset dimension value, and each slice semantic feature is stored in the corresponding storage stack. Figure 4 As shown: the 64-dimensional output semantic features are divided into four 16-dimensional slice semantic features, and the four 16-dimensional slice semantic features are stored in four storage stacks head_0, head_1, head_2, and head_3 respectively, that is: slice semantic feature 0 is stored in storage stack head_0, slice semantic feature 1 is stored in storage stack head_1, slice semantic feature 2 is stored in storage stack head_2, and slice semantic feature 3 is stored in storage stack head_3.

[0076] In the above steps a2 and a3, the dimension value of the output semantic feature after dimensionality reduction is determined based on the number of the multiple storage stacks and the storage space size of each storage stack. For example, assuming that the dimension value of the output semantic feature before dimensionality reduction is 2048, the initial language model has 4 storage stacks, and the storage space of each storage stack is 16 dimensions, then the dimension value of the output semantic feature after dimensionality reduction is: 4*16=64 dimensions, that is, the dimension value of the output semantic feature needs to be reduced from 2048 to 64.

[0077] In some specific embodiments, calling the i-th feature extraction layer to obtain second input data includes:

[0078] Step b1: Read multiple shard semantic features from the multiple storage stacks respectively.

[0079] Specifically, for example Figure 4 As shown: shard semantic feature 0 is read from storage stack head_0, shard semantic feature 1 is read from storage stack head_1, shard semantic feature 2 is read from storage stack head_2, shard semantic feature 3 is read from storage stack head_3, ..., shard semantic feature n is read from storage stack head_n.

[0080] Step b2: identifying multiple sequence labels of the multiple fragment semantic features.

[0081] Specifically, for example, the bit sequence label of slice semantic feature 0 is S0, indicating that slice semantic feature 0 is the lowest bit in the spliced ​​semantic feature, and the bit sequence label of slice semantic feature n is Sn, indicating that slice semantic feature n is the highest bit in the spliced ​​semantic feature.

[0082] Step b3: splicing the multiple fragment semantic features into a spliced ​​semantic feature according to the order of the high bits and the low bits indicated by the multiple sequence labels.

[0083] Specifically, the positional label of each fragment semantic feature can be used to determine the corresponding position of each fragment semantic feature in the spliced ​​semantic feature, so that each fragment semantic feature is spliced ​​at the corresponding position to obtain the spliced ​​semantic feature.

[0084] Step b4: performing dimensionality-upgrading processing on the splicing semantic features to obtain dimensionality-upgraded splicing semantic features, and using the dimensionality-upgraded splicing semantic features as the second input data.

[0085] Specifically, for example, the dimension value of the concatenated semantic feature is adjusted from 64 to 2048. The dimension value of the dimension increase is the same as the dimension value of the dimension reduction. For example, if the dimension reduction is 2048→64, then the dimension increase is 64→2048.

[0086] In some specific embodiments, the above step a4 includes steps a41 to a43:

[0087] Step a41, for any storage bit among the multiple storage bits, determine the shard semantic feature of the storage bit at the current calculation moment based on the shard semantic feature stored in the storage bit at the previous calculation moment, the shard semantic feature stored in the adjacent storage bit of the storage bit and the preset probability.

[0088] The adjacent storage bits include low-order storage bits and high-order storage bits adjacent to the storage bits; the calculation moment refers to the moment when the corresponding feature extraction layer performs calculations and stores the output results in the storage stack, and one calculation moment corresponds to one feature extraction layer.

[0089] In the embodiment of the present application, each storage stack includes multiple storage bits, such as Figure 5 shown.

[0090] When storing each shard semantic feature in the corresponding preset storage stack, the preset probability includes a write probability, an unchanged probability, and a deletion probability. Taking any storage bit as an example, the weighted result of the first semantic feature, the second semantic feature, and the third semantic feature at the previous calculation moment is determined as the shard semantic feature stored in the storage bit at the current calculation moment. Among them, the first semantic feature is the product of the shard semantic feature stored in the low-order storage bit and the write probability; the second semantic feature is the product of the shard semantic feature stored in the storage bit and the unchanged probability; the third semantic feature is the product of the shard semantic feature stored in the high-order storage bit and the deletion probability.

[0091] Specifically, the shard semantic features corresponding to the storage bits can be calculated using the following formula:

[0092] stack t [n]=stack t-1 [n-1]×action[push] t +stack t-1 [n]×action[noop] t

[0093] +stack t-1 [n+1]×action[pop] t

[0094] Among them, t is the calculation time, n is the index of the storage bit, stack t [n] is the shard semantic feature stored in the nth storage position at the current calculation moment, stack t-1 [n-1] is the shard semantic feature stored in the n-1th storage bit (low storage bit) at the previous calculation moment, action [push] t is the write probability at the current calculation time, stack t-1 [n] is the shard semantic feature stored in the current storage location at the previous calculation moment, action[noop] t is the constant probability at the current calculation moment, stack t-1 [n+1] is the shard semantic feature stored in the n+1th storage bit (high storage bit) at the previous calculation moment, action [pop] t is the constant probability at the current calculation moment.

[0095] Among them, the three probabilities in the preset probabilities can be pre-set with corresponding probability values, or can be predicted based on a neural model, etc. The embodiment of the present application does not limit the method of obtaining the preset probabilities, and those skilled in the art can determine it according to actual conditions.

[0096] Step a42: If the storage bit is the lowest storage bit among the multiple storage bits, the data of the lower storage bits in the adjacent storage positions are the output semantic features of the feature extraction layer corresponding to the previous calculation moment.

[0097] Step a43: If the storage bit is the highest storage bit among the multiple storage bits, the data stored in the higher storage bits in the adjacent storage positions is empty.

[0098] In some embodiments, each feature extraction layer may perform feature extraction in the following manner:

[0099] When the i-th feature extraction layer is the first layer of multiple feature extraction layers, the i-th feature extraction layer performs feature extraction according to the initial semantic features of the language to be recognized to obtain the output semantic features of the j-th feature extraction layer.

[0100] When the i-th feature extraction layer is neither the last layer nor the first layer of multiple feature extraction layers, feature extraction can be performed based on the i-th feature extraction layer and the fused semantic features to obtain the output semantic features of the j-th feature extraction layer. The fused semantic features refer to semantic features that are fused based on the output semantic features of the i-1-th feature extraction layer and the splicing and dimensionality-raising results of multiple slice semantic features read from multiple storage stacks (i.e., first splicing four 16-dimensional slice semantic features to obtain a 64-dimensional spliced ​​semantic feature, and then adjusting the dimensionality of the spliced ​​semantic feature from 64 to 2048).

[0101] Exemplarily, for any feature extraction layer, the input data of the feature extraction layer is a fused semantic feature obtained by fusing the output semantic features of the previous feature extraction layer and the spliced ​​semantic features of multiple fragmented semantic features stored in multiple storage stacks. Specifically, the fused semantic feature can be obtained by directly splicing the two data or by weighted summation of the two data. The embodiment of the present application does not limit the combination form of the fused features, and those skilled in the art can determine it according to actual conditions.

[0102] In some embodiments, the fused feature can also be generated in the following manner: the output semantic feature of the previous feature extraction layer and the spliced ​​semantic feature are spliced ​​to obtain a third semantic feature; the third semantic feature is point multiplied with a preset weight matrix to obtain the probability of each element in the third semantic feature; based on each element in the third semantic feature and the corresponding probability of each element, a fused semantic feature is obtained.

[0103] For example, when performing feature fusion, the output semantic features of the previous feature extraction layer and the concatenated semantic features can be concatenated to obtain a third semantic feature. After concatenating the third semantic feature, the probability of each element in the third semantic feature is calculated. Specifically, the probability of each element obtained by passing the third semantic feature through a fully connected layer is calculated. The weighted sum of each element and the corresponding probability is then taken to obtain the fused feature.

[0104] In some embodiments, regularization can be performed after the weighted summation, and the fused feature can be obtained by averaging. Specifically, each element in the third semantic feature is multiplied by the corresponding probability to obtain a probabilistic fused feature; the probabilistic fused features corresponding to each element in the third semantic feature are added to obtain an initial fused feature; and the initial fused feature is divided by the number of elements to obtain a fused semantic feature.

[0105] Step S102: when the i-th feature extraction layer is the last layer of the plurality of feature extraction layers, calculating a loss value between an output semantic feature of the i-th feature extraction layer and a preset standard semantic feature;

[0106] Step S103: If the loss value satisfies a preset convergence condition, the initial language model is determined to be an inference language model based on a multi-head differentiable stack, and the inference language model based on the multi-head differentiable stack has a function of recognizing the semantics of a formal language.

[0107] Exemplarily, the loss function can be mean square error, cross entropy loss, etc. The specific choice depends on the representation form of the semantic feature (continuous value or discrete category). The embodiment of this application does not limit the specific form of the loss function, and those skilled in the art can determine it according to actual conditions.

[0108] The model is trained using an optimization algorithm (such as Adam, SGD, etc.) to minimize the loss function. During the training process, the weights of the model are updated through the backpropagation algorithm.

[0109] Set a predefined convergence condition, such as when the loss falls below a certain threshold or when the loss stops decreasing significantly after a certain number of training epochs. Use the validation set to monitor model performance to avoid overfitting and complete the training of the inference language model based on the multi-head differentiable stack.

[0110] The embodiment of the present application provides an inference language model training method based on a multi-head differentiable stack, which obtains the first input data and the second input data respectively by calling the i-th feature extraction layer in the initial language model, and performs feature extraction through the first input data and the second input data to obtain the output semantic features of the i-th feature extraction layer, wherein the first input data is the output semantic features of the i-1-th feature extraction layer; the second input data is the splicing result of the multiple semantic features stored in the multiple storage stacks; the semantic features stored in each storage stack refer to the sliced ​​semantic features obtained after the output semantic features of the i-1-th feature extraction layer are processed by dimensionality reduction and segmentation. Since the input data of the feature extraction layer during feature extraction is increased by the data stored in the storage stack on the basis of the original output data of the previous feature extraction layer, the accuracy of language recognition is improved; furthermore, the target inference language model based on the differentiable stack obtained by training according to subsequent steps can process languages ​​with a certain degree of complexity.

[0111] Preferably, in the embodiment of the present application, by performing dimensionality reduction processing on the output semantic features of the feature extraction layer, the memory overhead of the semantic model during model training can be reduced and the training speed can be faster; and by segmenting the output semantic features after dimensionality reduction processing, the model modeling effect can be improved, that is, the modeling ability of the formal language and the modeling ability of the general language are improved.

[0112] The present application also provides a computer device to execute the above-mentioned inference language model training method based on a multi-head differentiable stack. Figure 6 , which shows a schematic diagram of a computer device provided by some embodiments of the present application. Figure 6 As shown, the computer device 6 includes: a processor 600, a memory 601, a bus 602 and a communication interface 603, and the processor 600, the communication interface 603 and the memory 601 are connected via the bus 602; the memory 601 stores a computer program that can be run on the processor 600, and when the processor 600 runs the computer program, it executes the inference language model training method based on the multi-head differentiable stack provided in the aforementioned embodiment of the present application.

[0113] The memory 601 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element and at least one other network element are connected via at least one communication interface 603 (which may be wired or wireless), and may use the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0114] Bus 602 can be an ISA bus, a PCI bus, or an EISA bus. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 601 is used to store programs. The processor 600 executes the programs after receiving execution instructions. The inference language model training method based on a multi-head differentiable stack disclosed in the above embodiment can be applied to the processor 600 or implemented by the processor 600.

[0115] The processor 600 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 600 or by software instructions. The above processor 600 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 601 , and the processor 600 reads the information in the memory 601 and completes the steps of the above method in combination with its hardware.

[0116] The computer device provided in the embodiment of the present application and the reasoning language model training method based on a multi-head differentiable stack provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.

[0117] The present application also provides a computer-readable storage medium corresponding to the multi-head differentiable stack-based reasoning language model training method provided in the above embodiment. Figure 7 , the computer-readable storage medium shown is a CD 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by the processor, it will execute the reasoning language model training method based on a multi-head differentiable stack provided by any of the aforementioned embodiments.

[0118] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0119] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the reasoning language model training method based on a multi-head differentiable stack provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0120] It should be noted that:

[0121] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0122] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present application.

[0123] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this application and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0124] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for training an inference language model based on a multi-head differentiable stack, characterized in that: The initial language model includes multiple feature extraction layers and multiple storage stacks. The method includes: For the i-th feature extraction layer, calling the i-th feature extraction layer to obtain first input data and second input data respectively, and performing feature extraction on the first input data and the second input data to obtain output semantic features of the i-th feature extraction layer; wherein the first input data is the output semantic features of the i-1-th feature extraction layer; the second input data is the concatenation result of multiple semantic features stored in the multiple storage stacks; the semantic features stored in each storage stack refer to the fragmented semantic features obtained after the output semantic features of the i-1-th feature extraction layer are subjected to dimensionality reduction and segmentation processing; the i is greater than or equal to 2; When the i-th feature extraction layer is the last layer of the plurality of feature extraction layers, calculating a loss value between an output semantic feature of the i-th feature extraction layer and a preset standard semantic feature; If the loss value satisfies a preset convergence condition, the initial language model is determined to be an inference language model based on a multi-head differentiable stack, and the inference language model based on the multi-head differentiable stack has a function of recognizing the semantics of a formal language.

2. The method according to claim 1, characterized in that In a case where the i-th feature extraction layer is not the last layer of the plurality of feature extraction layers, the method further includes: Get the output semantic features of the i-1th feature extraction layer; Performing dimensionality reduction processing on the output semantic features to obtain output semantic features after dimensionality reduction; Splitting the output semantic features after dimensionality reduction into a plurality of slice semantic features; For any slice semantic feature among the multiple slice semantic features, the slice semantic feature is stored in a corresponding storage stack; the multiple partial semantic features correspond one-to-one to the multiple storage stacks.

3. The method according to claim 2, characterized in that Performing dimensionality reduction processing on the output semantic feature to obtain a reduced-dimensional output semantic feature, including: adjusting the dimension value of the output semantic feature from 2048 dimensions to 64 dimensions; The output semantic features after dimensionality reduction are divided into a plurality of slice semantic features, including: dividing the 64-dimensional output semantic features into four 16-dimensional slice semantic features.

4. The method according to claim 2 or 3, characterized in that Calling the i-th feature extraction layer to obtain second input data includes: Reading a plurality of shard semantic features from the plurality of storage stacks respectively; Identifying a plurality of sequence tags of the plurality of fragment semantic features; splicing the plurality of fragment semantic features into a spliced ​​semantic feature according to the order of the high bits and the low bits indicated by the plurality of bit sequence labels; The splicing semantic feature is subjected to dimensionality-upgrading processing to obtain a dimensionality-upgraded splicing semantic feature, and the dimensionality-upgraded splicing semantic feature is used as the second input data; the dimension value of the dimensionality-upgrading is the same as the dimension value of the dimensionality-reducing.

5. The method according to claim 4, characterized in that Splicing the plurality of slice semantic features into a spliced ​​semantic feature, comprising: splicing four 16-dimensional slice semantic features into a 64-dimensional semantic feature; Performing dimensionality-upgrading processing on the spliced ​​semantic features to obtain dimensionality-upgraded spliced ​​semantic features includes: adjusting the dimension value of the 64-dimensional semantic features from 64 dimensions to 2048 dimensions.

6. The method according to claim 4, characterized in that The dimension value of the output semantic feature after dimensionality reduction is determined according to the number of the multiple storage stacks and the storage space size of each storage stack.

7. The method according to claim 2, characterized in that Each storage stack includes multiple storage bits, and stores the shard semantic features in the corresponding storage stack, including: For any storage bit among the multiple storage bits, based on the shard semantic features stored in the storage bit at the previous calculation moment, the shard semantic features stored in the adjacent storage bits of the storage bit, and the preset probability, determine the shard semantic features of the storage bit at the current calculation moment; the adjacent storage bits include the low-order storage bits and the high-order storage bits adjacent to the storage bit; the calculation moment refers to the moment when the corresponding feature extraction layer performs calculation and stores the output result to the storage stack, and one calculation moment corresponds to one feature extraction layer; If the storage bit is the lowest storage bit among the multiple storage bits, the data of the lower storage bit in the adjacent storage position is the output semantic feature of the feature extraction layer corresponding to the previous calculation moment; If the storage bit is the highest storage bit among the multiple storage bits, the data stored in the higher storage bits in the adjacent storage positions is empty.

8. The method according to claim 7, characterized in that The preset probabilities include a write probability, an unchanged probability, and a deletion probability. Determining the shard semantic features of the storage bit at the current calculation moment based on the shard semantic features stored in the storage bit at the previous calculation moment, the shard semantic features stored in adjacent storage bits of the storage bit, and the preset probabilities includes: Calculating a first semantic feature based on the shard semantic feature stored in the low-order storage bit at the last calculation moment and the write probability; Calculate a second semantic feature based on the slice semantic feature stored in the storage bit at the last calculation moment and the invariant probability; Calculate a third semantic feature based on the shard semantic feature stored in the high-order storage bit at the last calculation moment and the deletion probability; A weighted sum is performed on the first semantic feature, the second semantic feature, and the third semantic feature to obtain the shard semantic feature stored in the storage location at the current calculation moment.

9. The method according to claim 1, characterized in that The method further comprises: If i=1, the first input data obtained by the i-th feature extraction layer is the initial semantic feature of the language to be recognized, and the second input data obtained by the i-th feature extraction layer is empty.

10. A reasoning language model based on a multi-head differentiable stack, characterized in that: It includes an input layer, a recognition layer and an output layer, and the recognition layer includes multiple feature extraction layers and multiple storage stacks; wherein the i-th feature extraction layer is used to obtain the first input data and the second input data respectively, and perform feature extraction through the first input data and the second input data to obtain the output semantic features of the i-th feature extraction layer; wherein the first input data is the output semantic features of the i-1-th feature extraction layer; the second input data is the splicing result of multiple semantic features stored in the multiple storage stacks; the semantic features stored in each storage stack refer to the sliced ​​semantic features obtained after the output semantic features of the i-1-th feature extraction layer are processed by dimensionality reduction and segmentation; the i is greater than or equal to 2.

Images