Semantic extraction method based on model computing power dynamic adjustment, model and computer equipment

Abstract

The invention provides a semantic extraction method and model based on model computing power dynamic adjustment and computer equipment, and the method comprises the steps: calling a plurality of feature extraction layers to carry out multiple times of feature extraction on initial semantic features, and obtaining predicted semantic features; judging a matching degree between the predicted semantic feature and a preset expected semantic feature; if the matching degree of the predicted semantic feature and the preset expected semantic feature meets a preset matching threshold value, generating an output result of the language model through the predicted semantic feature; and if the matching degree of the predicted semantic features and the preset expected semantics does not meet a preset matching threshold value, taking the predicted semantic features as new initial semantic features, and recalling the plurality of feature extraction layers to carry out multiple times of feature extraction on the new initial semantic features until an output result of the language model is generated. According to the embodiment of the invention, different computing resources can be allocated for different tasks, and the purpose of reasonably using the computing resources is achieved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and specifically to a semantic extraction method, model and computer equipment based on dynamic adjustment of model computing power. Background Art

[0002] The Transformer model is used for natural language processing (NLP) and relies on the self-attention mechanism to capture long-range dependencies in sequence data.

[0003] However, traditional Transformer models allocate the same computing resources to each token when processing different tasks. For example, when predicting simple math problems (such as "1+1=?") and complex math problems (such as "321*321="), the model allocates the same computing power to each token. This "one-size-fits-all" approach is overly complex for simple tasks and may not capture all necessary information for complex ones. Summary of the Invention

[0004] In view of this, the present application proposes a semantic extraction method, model and computer device based on dynamic adjustment of model computing power to solve the problem of irrational resource utilization in related technologies due to the same computing resources allocated to different tasks.

[0005] The first embodiment of the present application proposes a semantic extraction method based on dynamic adjustment of model computing power, including:

[0006] Calling the multiple feature extraction layers to perform multiple feature extractions on the initial semantic features to obtain predicted semantic features;

[0007] Determining the degree of match between the predicted semantic features and the preset expected semantic features;

[0008] If the matching degree between the predicted semantic feature and the preset expected semantic feature meets a preset matching threshold, generating an output result of the language model using the predicted semantic feature;

[0009] If the degree of matching between the predicted semantic feature and the preset expected semantics does not meet the preset matching threshold, the predicted semantic feature is used as the new initial semantic feature, and the multiple feature extraction layers are re-called to perform multiple feature extractions on the new initial semantic feature until the output result of the language model is generated.

[0010] In the embodiment of the present application, determining the degree of matching between the predicted semantic feature and the preset expected semantic feature includes:

[0011] Calculating a first category probability and a second category probability corresponding to the predicted semantic feature; the first category probability represents a probability that the predicted semantic feature is used to generate an output result of the language model; the second category probability represents a probability that the predicted semantic feature is used as a new initial semantic feature; the sum of the first category probability and the second category probability is 1;

[0012] If the first category probability is greater than the second category probability, determining whether the degree of matching between the predicted semantic feature and the preset expected semantic feature meets a preset matching threshold;

[0013] If the first category probability is less than or equal to the second category probability, it is determined that the degree of matching between the predicted semantic feature and the preset expected semantic feature does not meet a preset matching threshold.

[0014] In the embodiment of the present application, calculating the first category probability and the second category probability corresponding to the predicted semantic feature includes:

[0015] Calculating a score vector based on the predicted semantic features and a preset weight matrix; each element in the score vector corresponds to a score of a different category; the preset weight matrix is ​​learned by the language model during the model training process;

[0016] The score vector is converted into a probability distribution according to an activation function; the probability distribution includes the first category probability and the second category probability.

[0017] In an embodiment of the present application, converting the score vector into a probability distribution according to an activation function includes:

[0018] Circle=softmax(AT)

[0019] Among them, Circle represents probability distribution, AT represents score vector, softmax(.) represents activation function, A represents preset weight, and T represents predicted semantic feature.

[0020] In an embodiment of the present application, the language model further includes multiple storage stacks; calling the multiple feature extraction layers to perform multiple feature extractions on the initial semantic features to obtain predicted semantic features includes:

[0021] Calling the i-th feature extraction layer to obtain first input data and second input data respectively; wherein the first input data is the output semantic feature of the i-1-th feature extraction layer; the second input data is the concatenation result of multiple semantic features stored in the multiple storage stacks; the semantic feature stored in each storage stack is a sliced ​​semantic feature obtained after dimensionality reduction and segmentation of the output semantic feature of the i-1-th feature extraction layer, where i is greater than or equal to 2;

[0022] Performing feature extraction on the first input data and the second input data to obtain output semantic features of the i-th feature extraction layer;

[0023] When the i-th feature extraction layer is the last layer of the multiple feature extraction layers, the output semantic features of the i-th feature extraction layer are used as the predicted semantic features.

[0024] In an embodiment of the present application, a semantic judgment layer is provided between any two adjacent feature extraction layers; when i is greater than or equal to 2, after obtaining the output semantic features of the i-th feature extraction layer, the method further includes:

[0025] Calling a target semantic judgment layer to judge whether the output semantic features of the i-th feature extraction layer meet preset semantic requirements; the target semantic judgment layer is set between the i-th feature extraction layer and the i+1-th feature extraction layer;

[0026] If the output semantic features of the i-th feature extraction layer meet the preset semantic requirements, the output semantic features of the i-th feature extraction layer are used as the predicted semantic features.

[0027] In an embodiment of the present application, when the i-th feature extraction layer is not the last layer of the plurality of feature extraction layers, the method further includes:

[0028] Get the output semantic features of the i-1th feature extraction layer;

[0029] Performing dimensionality reduction processing on the output semantic features to obtain dimensionality-reduced output semantic features;

[0030] Splitting the output semantic features after dimensionality reduction into a plurality of slice semantic features;

[0031] For any slice semantic feature among the multiple slice semantic features, the slice semantic feature is stored in a corresponding storage stack; the multiple partial semantic features correspond one-to-one to the multiple storage stacks.

[0032] In the embodiment of the present application, calling the i-th feature extraction layer to obtain second input data includes:

[0033] Reading a plurality of shard semantic features from the plurality of storage stacks respectively;

[0034] Identifying a plurality of sequence tags of the plurality of fragment semantic features;

[0035] splicing the plurality of fragment semantic features into a spliced ​​semantic feature according to the order of the high bits and the low bits indicated by the plurality of bit sequence labels;

[0036] The splicing semantic feature is subjected to dimensionality-upgrading processing to obtain a dimensionality-upgraded splicing semantic feature, and the dimensionality-upgraded splicing semantic feature is used as the second input data; the dimension value of the dimensionality-upgrading is the same as the dimension value of the dimensionality-reducing.

[0037] The embodiment of the third aspect of the present application provides a language model based on dynamic adjustment of computing power, including an input layer, a recognition layer, and an output layer, wherein the recognition layer includes a computing power judgment layer and multiple feature extraction layers;

[0038] The multiple feature extraction layers are used to perform multiple feature extractions on the initial semantic features layer by layer to obtain predicted semantic features;

[0039] The computing power judgment layer is used to judge the degree of matching between the predicted semantic feature and the preset expected semantic feature; if the degree of matching between the predicted semantic feature and the preset expected semantic feature meets the preset matching threshold, the output result of the language model is generated through the predicted semantic feature; if the degree of matching between the predicted semantic feature and the preset expected semantic does not meet the preset matching threshold, the predicted semantic feature is used as a new initial semantic feature, and the multiple feature extraction layers are re-called to perform multiple feature extractions on the new initial semantic feature until the output result of the language model is generated.

[0040] An embodiment of the third aspect of the present application provides a computer device, which includes a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the semantic extraction method based on dynamic adjustment of model computing power described in the first aspect by executing the computer instructions.

[0041] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to enable a computer to execute the semantic extraction method based on dynamic adjustment of model computing power as described in the first aspect above.

[0042] Additional aspects and advantages of the present application will be given in part in the description below and in part will become apparent from the description below or learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. Throughout the accompanying drawings, the same reference numerals are used to denote the same components.

[0044] In the attached figure:

[0045] Figure 1A schematic diagram of the structure of a language model based on dynamic adjustment of computing power provided by an embodiment of the present application is shown;

[0046] Figure 2 A schematic diagram of the structure of the recognition layer in a language model based on dynamic adjustment of computing power provided in one embodiment of the present application is shown;

[0047] Figure 3 A flowchart of a semantic extraction method based on dynamic adjustment of model computing power provided by an embodiment of the present application is shown;

[0048] Figure 4 A schematic diagram of a process for extracting features from data stored in multiple storage stacks using output semantic features of a previous feature extraction layer is shown in one embodiment of the present application;

[0049] Figure 5 A schematic diagram of a storage stack provided in one embodiment of the present application is shown;

[0050] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application is shown;

[0051] Figure 7 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0052] The following describes exemplary embodiments of the present application in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0053] It should be noted that, unless otherwise specified, the technical or scientific terms used in this application should have the common meanings understood by those skilled in the art to which this application belongs.

[0054] The following describes the technical scenarios involved in the embodiments of this application.

[0055] The Transformer model is a deep learning model based on the self-attention mechanism. However, its effectiveness in actual training is limited, and it cannot model and complete higher-level formal language tasks, which affects its performance and expressiveness.

[0056] The structure of the Transformer model mainly consists of two parts: the encoder and the decoder. Each part enhances the model's representation and learning capabilities by stacking multiple identical layers.

[0057] The encoder consists of N identical layers stacked together, each of which consists of two sublayers: a multi-head self-attention mechanism and a feedforward neural network. Residual connections and layer normalization are applied after each sublayer in the encoder to stabilize the training process. The encoder layer is primarily responsible for processing the input sequence and generating a series of encoded representations. These encoded representations capture the contextual information in the input sequence.

[0058] The decoder is similarly composed of N identical layers stacked together, but each layer contains the same two sublayers as the encoder, plus an additional multi-head attention sublayer to process the encoder output: a multi-head self-attention mechanism, a multi-head encoder-decoder attention mechanism (sublayer), and a feedforward neural network. Residual connections and layer normalization are also applied after each sublayer in the decoder. The decoder layer is responsible for generating the next output based on the encoder output and the previously generated output sequence (in the case of generation tasks).

[0059] Formal language theory is a branch of computer science that studies the abstract structure and rules of languages. The Formal Language Hierarchy describes different types of languages ​​and their generative capabilities.

[0060] According to an embodiment of the present application, an embodiment of a semantic extraction method based on dynamic adjustment of model computing power is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0061] The embodiment of the present disclosure proposes a language model based on dynamic adjustment of computing power, such as Figure 1 As shown;

[0062] The language model based on dynamic adjustment of computing power includes an input layer 101, a recognition layer 102, and an output layer 103. The recognition layer includes a computing power judgment layer, multiple feature extraction layers, and multiple storage stacks.

[0063] The input layer 101 can be used to extract preliminary semantic features from the input language to be recognized. For example, the language to be recognized can be a language with abstract semantics, such as mathematical formulas, computer programming languages, and chemical formulas. The input layer 101 can convert the language to be recognized directly input by the user into initial semantic features in the form of semantic vectors, etc. The initial semantic features can be converted into vectors, etc.

[0064] The recognition layer 102 is used to extract the initial semantic features and then realize semantic recognition. Figure 2 The illustrated recognition layer 102 includes multiple feature extraction layers and multiple storage stacks.

[0065] The multiple feature extraction layers are used to perform multiple feature extractions on the initial semantic features layer by layer to obtain predicted semantic features;

[0066] Specifically, taking the Transformer model as an example, the recognition layer 102 may include the encoder layer and the decoder layer in the Transformer model, as well as corresponding multiple storage stacks, wherein the encoder layer may serve as the first feature extraction layer, and the decoder layer may serve as the second feature extraction layer; the output data of the first feature extraction layer may be divided into multiple partial data, and then the multiple partial data are respectively stored in corresponding storage stacks, and when feature extraction is performed in the second feature extraction layer, the input data is composed of the output data of the first feature extraction layer and the data obtained from multiple storage stacks.

[0067] More specifically, when i is equal to 1, the first feature extraction layer is used to perform feature extraction on the initial semantic features of the language to be recognized to obtain the output semantic features of the first feature extraction layer; when i is greater than or equal to 2, the i-th feature extraction layer is used to obtain the first input data and the second input data respectively, and perform feature extraction on the first input data and the second input data to obtain the output semantic features of the i-th feature extraction layer; wherein, the first input data is the output semantic features of the i-1-th feature extraction layer; the second input data is the splicing result of the multiple semantic features stored in the multiple storage stacks; the semantic features stored in each storage stack refer to the sliced ​​semantic features obtained after the output semantic features of the i-1-th feature extraction layer are subjected to dimensionality reduction and segmentation processing; when the i-th feature extraction layer is the last layer of the multiple feature extraction layers, the output semantic features of the i-th feature extraction layer are used as the predicted semantic features.

[0068] Taking the Transformer model as an example, both the encoder and the decoder include multiple sub-layers, wherein any sub-layer of the encoder includes a multi-head self-attention mechanism, a feedforward neural network, and a residual connection layer and a layer normalization layer for stabilization. In combination with the embodiments of the present application, for the layers in the two sub-layers of the multi-head self-attention mechanism and the feedforward neural network in the encoder, the multiple output data in the multi-head self-attention mechanism can be stored in multiple storage stacks, and the input data in the feedforward neural network is composed of the output data of the multi-head self-attention mechanism and the data in the storage stack. Similarly, the decoder is similar and will not be described in detail here. The recognition layer 102 performs feature extraction, and the storage and extraction process of the storage stack is introduced in the embodiments later and will not be described in detail here.

[0069] The computing power judgment layer is used to judge the degree of matching between the predicted semantic feature and the preset expected semantic feature; if the degree of matching between the predicted semantic feature and the preset expected semantic feature meets the preset matching threshold, the output result of the language model is generated through the predicted semantic feature; if the degree of matching between the predicted semantic feature and the preset expected semantic does not meet the preset matching threshold, the predicted semantic feature is used as a new initial semantic feature, and the multiple feature extraction layers are re-called to perform multiple feature extractions on the new initial semantic feature until the output result of the language model is generated.

[0070] The output layer 103 is used to integrate and output the data in the recognition layer 102.

[0071] In this embodiment, a semantic extraction method based on dynamic adjustment of model computing power is provided. Figure 3 is a flowchart of a semantic extraction method based on dynamic adjustment of model computing power according to an embodiment of the present application, such as Figure 3 As shown, the process includes the following steps:

[0072] Step S101 : Calling the multiple feature extraction layers to perform multiple feature extractions on the initial semantic features to obtain predicted semantic features.

[0073] In some specific embodiments, the above step S101 includes steps S1011 to S1013:

[0074] Step S1011: calling the i-th feature extraction layer to obtain first input data and second input data respectively.

[0075] Specifically, the first input data is the output semantic feature of the i-1th feature extraction layer; the second input data is the concatenation result of multiple semantic features stored in the multiple storage stacks; the semantic feature stored in each storage stack refers to the sliced ​​semantic feature obtained after the output semantic feature of the i-1th feature extraction layer is processed by dimensionality reduction and segmentation, and i is greater than or equal to 2;

[0076] In the embodiments of the present application, a language model is invoked to perform multiple feature extractions on the initial semantic features of a language layer by layer to obtain predicted semantic features, i.e., the output semantic features of the final layer of the multiple feature extraction layers. The semantic features of the language can be either natural or formal. The embodiments of the present application do not limit the specific form of the language, and those skilled in the art can determine the specific form based on actual circumstances.

[0077] In the embodiment of the present application, the language model can be understood as the above Figure 1The language model in the language is dynamically adjusted based on computing power; when calling the language model to extract the initial semantic features, each feature extraction layer further extracts the output data of the previous feature extraction layer. On this basis, the "output data of the previous feature extraction layer" and the "data extracted from the storage stack" are used as the input data of the next feature extraction layer, realizing multi-dimensional feature extraction of the language to be recognized and improving the accuracy of feature extraction.

[0078] In some specific embodiments, when the feature extraction layer is the first layer among multiple feature extraction layers, that is, i=1, the first input data obtained by the i-th feature extraction layer is the initial semantic feature of the language to be recognized, and the second input data obtained by the i-th feature extraction layer is empty.

[0079] In some specific embodiments, when the i-th feature extraction layer is not the last layer of the plurality of feature extraction layers, the method further includes steps a1 to a4:

[0080] Step a1: Obtain the output semantic features of the i-1th feature extraction layer.

[0081] Step a2: performing dimensionality reduction processing on the output semantic features to obtain output semantic features after dimensionality reduction.

[0082] Specifically, for example, the dimension value of the output semantic feature is adjusted from 2048 dimensions to 64 dimensions, for example Figure 4 Although the dimension of the output semantic features is reduced, the output semantic features after dimensionality reduction retain most of the core features of the output semantic features before dimensionality reduction.

[0083] Step a3: dividing the output semantic features after dimensionality reduction into multiple slice semantic features.

[0084] Step a4: for any slice semantic feature among the multiple slice semantic features, store the slice semantic feature in a corresponding storage stack; the multiple partial semantic features correspond one-to-one to the multiple storage stacks.

[0085] In the above steps a2 and a3, a multi-head mechanism is used to divide the output semantic features after dimensionality reduction into multiple slice semantic features according to the preset dimension value, and each slice semantic feature is stored in the corresponding storage stack. Figure 4As shown: the 64-dimensional output semantic features are divided into four 16-dimensional slice semantic features, and the four 16-dimensional slice semantic features are stored in four storage stacks head_0, head_1, head_2, and head_3 respectively, that is: slice semantic feature 0 is stored in storage stack head_0, slice semantic feature 1 is stored in storage stack head_1, slice semantic feature 2 is stored in storage stack head_2, and slice semantic feature 3 is stored in storage stack head_3.

[0086] In the above steps a2 and a3, the dimension value of the output semantic feature after dimensionality reduction is determined based on the number of the multiple storage stacks and the storage space size of each storage stack. For example, assuming that the dimension value of the output semantic feature before dimensionality reduction is 2048, the initial language model has 4 storage stacks, and the storage space of each storage stack is 16 dimensions, then the dimension value of the output semantic feature after dimensionality reduction is: 4*16=64 dimensions, that is, the dimension value of the output semantic feature needs to be reduced from 2048 to 64.

[0087] In some specific embodiments, calling the i-th feature extraction layer to obtain second input data includes:

[0088] Step b1: Read multiple shard semantic features from the multiple storage stacks respectively.

[0089] Specifically, for example Figure 4 As shown: shard semantic feature 0 is read from storage stack head_0, shard semantic feature 1 is read from storage stack head_1, shard semantic feature 2 is read from storage stack head_2, shard semantic feature 3 is read from storage stack head_3, ..., shard semantic feature n is read from storage stack head_n.

[0090] Step b2: identifying multiple sequence labels of the multiple fragment semantic features.

[0091] Specifically, for example, the bit sequence label of slice semantic feature 0 is S0, indicating that slice semantic feature 0 is the lowest bit in the spliced ​​semantic feature, and the bit sequence label of slice semantic feature n is Sn, indicating that slice semantic feature n is the highest bit in the spliced ​​semantic feature.

[0092] Step b3: splicing the multiple fragment semantic features into a spliced ​​semantic feature according to the order of the high bits and the low bits indicated by the multiple sequence labels.

[0093] Specifically, the positional label of each fragment semantic feature can be used to determine the corresponding position of each fragment semantic feature in the spliced ​​semantic feature, so that each fragment semantic feature is spliced ​​at the corresponding position to obtain the spliced ​​semantic feature.

[0094] Step b4: performing dimensionality-upgrading processing on the splicing semantic features to obtain dimensionality-upgraded splicing semantic features, and using the dimensionality-upgraded splicing semantic features as the second input data.

[0095] Specifically, for example, the dimension value of the concatenated semantic feature is adjusted from 64 to 2048. The dimension value of the dimension increase is the same as the dimension value of the dimension reduction. For example, if the dimension reduction is 2048→64, then the dimension increase is 64→2048.

[0096] In some specific embodiments, the above step a4 includes steps a41 to a43:

[0097] Step a41, for any storage bit among the multiple storage bits, determine the shard semantic feature of the storage bit at the current calculation moment based on the shard semantic feature stored in the storage bit at the previous calculation moment, the shard semantic feature stored in the adjacent storage bit of the storage bit and the preset probability.

[0098] The adjacent storage bits include low-order storage bits and high-order storage bits adjacent to the storage bits; the calculation moment refers to the moment when the corresponding feature extraction layer performs calculations and stores the output results in the storage stack, and one calculation moment corresponds to one feature extraction layer.

[0099] In the embodiment of the present application, each storage stack includes multiple storage bits, such as Figure 5 shown.

[0100] When storing each shard semantic feature in the corresponding preset storage stack, the preset probability includes a write probability, an unchanged probability, and a deletion probability. Taking any storage bit as an example, the weighted result of the first semantic feature, the second semantic feature, and the third semantic feature at the previous calculation moment is determined as the shard semantic feature stored in the storage bit at the current calculation moment. Among them, the first semantic feature is the product of the shard semantic feature stored in the low-order storage bit and the write probability; the second semantic feature is the product of the shard semantic feature stored in the storage bit and the unchanged probability; the third semantic feature is the product of the shard semantic feature stored in the high-order storage bit and the deletion probability.

[0101] Specifically, the shard semantic features corresponding to the storage bits can be calculated using the following formula:

[0102] stack t [n]=stack t-1 [n-1]×action[push] t +stack t-1 [n]×action[noop] t

[0103] +stack t-1 [n+1]×action[pop] t

[0104] Among them, t is the calculation time, n is the index of the storage bit, stack t [n] is the shard semantic feature stored in the nth storage position at the current calculation moment, stack t-1 [n-1] is the shard semantic feature stored in the n-1th storage bit (low storage bit) at the previous calculation moment, action [push] t is the write probability at the current calculation time, stack t-1 [n] is the shard semantic feature stored in the current storage location at the previous calculation moment, action[noop] t is the constant probability at the current calculation moment, stack t-1 [n+1] is the shard semantic feature stored in the n+1th storage bit (high storage bit) at the previous calculation moment, action [pop] t is the constant probability at the current calculation moment.

[0105] Among them, the three probabilities in the preset probabilities can be pre-set with corresponding probability values, or can be predicted based on a neural model, etc. The embodiment of the present application does not limit the method for obtaining the preset probabilities, and those skilled in the art can determine it according to actual conditions.

[0106] Step a42: If the storage bit is the lowest storage bit among the multiple storage bits, the data of the lower storage bits in the adjacent storage positions is the output semantic feature of the feature extraction layer corresponding to the previous calculation moment.

[0107] Step a43: If the storage bit is the highest storage bit among the multiple storage bits, the data stored in the higher storage bits in the adjacent storage positions is empty.

[0108] Step S1012: performing feature extraction on the first input data and the second input data to obtain output semantic features of the i-th feature extraction layer.

[0109] In an embodiment of the present application, feature extraction can be performed on the fusion data of the first input data and the second input data to obtain output semantic features of the i-th feature extraction layer; wherein the fusion method includes but is not limited to: directly concatenating the first input data and the second input data, or performing a weighted summation of the first input data and the second input data. The embodiment of the present application does not limit the fusion method, and those skilled in the art can determine it based on actual circumstances.

[0110] In some embodiments, the fused feature can also be generated in the following manner: the output semantic feature of the previous feature extraction layer and the spliced ​​semantic feature are spliced ​​to obtain a third semantic feature; the third semantic feature is point multiplied with a preset weight matrix to obtain the probability of each element in the third semantic feature; based on each element in the third semantic feature and the corresponding probability of each element, a fused semantic feature is obtained.

[0111] For example, when performing feature fusion, the output semantic features of the previous feature extraction layer and the concatenated semantic features can be concatenated to obtain a third semantic feature. After concatenating the third semantic feature, the probability of each element in the third semantic feature is calculated. Specifically, the probability of each element obtained by passing the third semantic feature through a fully connected layer is calculated. The weighted sum of each element and the corresponding probability is then taken to obtain the fused feature.

[0112] In some embodiments, regularization can be performed after the weighted summation, and the fused feature can be obtained by averaging. Specifically, each element in the third semantic feature is multiplied by the corresponding probability to obtain a probabilistic fused feature; the probabilistic fused features corresponding to each element in the third semantic feature are added to obtain an initial fused feature; and the initial fused feature is divided by the number of elements to obtain a fused semantic feature.

[0113] Step S1013: When the i-th feature extraction layer is the last layer of the multiple feature extraction layers, the output semantic features of the i-th feature extraction layer are used as the predicted semantic features.

[0114] In some specific embodiments, after the output semantic features of the i-th feature extraction layer are used as the predicted semantic features, the method also includes a model training step, namely: calculating the loss value between the output semantic features of the i-th feature extraction layer and the preset standard semantic features; if the loss value meets the preset convergence condition, the language model is determined to be an inference language model based on a multi-head differentiable stack, and the inference language model based on the multi-head differentiable stack has the function of recognizing the semantics of a formal language.

[0115] The embodiment of the present application obtains the first input data and the second input data respectively by calling the i-th feature extraction layer in the language model, and performs feature extraction through the first input data and the second input data to obtain the output semantic features of the i-th feature extraction layer, wherein the first input data is the output semantic features of the i-1-th feature extraction layer; the second input data is the concatenation result of the multiple semantic features stored in the multiple storage stacks; the semantic features stored in each storage stack refer to the sliced ​​semantic features obtained after the output semantic features of the i-1-th feature extraction layer are processed by dimensionality reduction and segmentation. Since the input data of the feature extraction layer during feature extraction is increased by the data stored in the storage stack based on the original output data of the previous feature extraction layer, the accuracy of language recognition is improved; furthermore, the target inference language model based on the differentiable stack obtained by training according to subsequent steps can process languages ​​with a certain degree of complexity.

[0116] Preferably, in the embodiment of the present application, by performing dimensionality reduction processing on the output semantic features of the feature extraction layer, the memory overhead of the semantic model during model training can be reduced and the training speed can be faster; and by segmenting the output semantic features after dimensionality reduction processing, the model modeling effect can be improved, that is, the modeling ability of the formal language and the modeling ability of the general language are improved.

[0117] Step S102: determining the degree of matching between the predicted semantic feature and the preset expected semantic feature.

[0118] In an embodiment of the present application, by calculating the degree of matching between the predicted semantic feature and the preset expected semantic feature, it can be determined based on the degree of matching whether to use the predicted semantic feature to generate the output result of the language model, or to use the predicted semantic feature as a new initial semantic feature, and re-call the multiple feature extraction layers to perform multiple feature extractions on the new initial semantic feature until the output result of the language model is generated.

[0119] In some specific embodiments, the above step S102 includes steps S1021 to S1023:

[0120] Step S1021 , calculating the first category probability and the second category probability corresponding to the predicted semantic feature.

[0121] Specifically, the first category probability represents the probability that the predicted semantic feature is used to generate the output result of the language model; the second category probability represents the probability of using the predicted semantic feature as a new initial semantic feature; the sum of the first category probability and the second category probability is 1.

[0122] In some specific embodiments, the above step S1021 includes steps c1 and c2:

[0123] Step c1, calculating a score vector based on the predicted semantic features and a preset weight matrix.

[0124] Specifically, each element in the score vector corresponds to a score of a different category; the preset weight matrix is ​​learned by the language model during the model training process;

[0125] Step c2: converting the score vector into a probability distribution according to an activation function; the probability distribution includes the first category probability and the second category probability.

[0126] In the embodiment of the present application, as shown in the following formula:

[0127] Circle=softmax(AT)

[0128] Among them, Circle represents probability distribution, AT represents score vector, softmax(.) represents activation function, A represents preset weight, and T represents predicted semantic feature.

[0129] Furthermore, the above formula can be transformed into:

[0130] Circle=softmax(AT i )

[0131] Where i represents the number of cycles, for example:

[0132] T1=Transformer(input)

[0133] T2=Transformer(T1) ...

[0135] T1 represents the predicted semantic features output by the language model after a cycle of the initial semantic features. A cycle refers to the step of calling multiple feature extraction layers to extract multiple features from the initial semantic features.

[0136] T2 represents the new predicted semantic features obtained through the second cycle, that is, the predicted semantic features output by the language model after the above cycle are re-input into the model, and multiple feature extraction layers of the model are called to perform multiple feature extractions on the predicted semantic features again to obtain new predicted semantic features.

[0137] In the embodiment of the present application, the first category probability Circle[exit] can be calculated by the above method. i and the second category probability Circle[next] i On this basis, the probability of the first category is determined by Circle[exit] i and the second category probability Circle[next]i The size relationship is determined based on the size relationship to determine whether to enter the next cycle.

[0138] Step S1022: If the first category probability is greater than the second category probability, it is determined that the degree of matching between the predicted semantic feature and the preset expected semantic feature meets a preset matching threshold.

[0139] Step S1023: If the first category probability is less than or equal to the second category probability, it is determined that the degree of matching between the predicted semantic feature and the preset expected semantic feature does not meet a preset matching threshold.

[0140] In the above steps S1022-S1023,

[0141] If Circle[exit] i >Circle[next] i , then the output result of the language model is generated by the predicted semantic features; otherwise, if Circle[exit] i ≤Circle[next] i , the predicted semantic feature is used as a new initial semantic feature, and the multiple feature extraction layers are re-called to perform multiple feature extractions on the new initial semantic feature until the output result of the language model is generated.

[0142] In some specific embodiments, a semantic judgment layer is provided between any two adjacent feature extraction layers; when i is greater than or equal to 2, after obtaining the output semantic features of the i-th feature extraction layer, the method further includes:

[0143] Calling a target semantic judgment layer to judge whether the output semantic features of the i-th feature extraction layer meet preset semantic requirements; the target semantic judgment layer is set between the i-th feature extraction layer and the i+1-th feature extraction layer;

[0144] If the output semantic features of the i-th feature extraction layer meet the preset semantic requirements, the output semantic features of the i-th feature extraction layer are used as the predicted semantic features.

[0145] In an embodiment of the present application, a language model includes multiple feature extraction layers, wherein a semantic judgment layer is provided between any two adjacent feature extraction layers, for example: feature extraction layer 1 → semantic judgment layer → feature extraction layer 2 → semantic judgment layer → feature extraction layer 3 → ...; each semantic judgment layer is used to judge the output semantic features of the previous feature extraction layer to determine whether the output semantic features meet the preset semantic requirements. If they meet the requirements, the output semantic features of the previous feature extraction layer are stored in multiple storage stacks, and the feature extraction actions of several feature extraction layers after the semantic judgment layer are suspended, and the semantic features stored in the multiple storage stacks are used as the recognition features in the language model. The recognition results of other layers are used to enable the output layer to generate the output results of the language model according to the recognition results; on the contrary, if the output semantic features do not meet the preset semantic requirements, the output semantic features of the previous feature extraction layer are stored in multiple storage stacks, and the output semantic features of the previous feature extraction layer are sent to the next feature extraction layer, so that the next feature extraction layer performs feature extraction according to the output semantic features of the previous feature extraction layer and the semantic features stored in multiple storage stacks to obtain the output semantic features of the next feature extraction layer, and calls the semantic judgment layer connected to the next feature extraction layer to judge whether the output semantic features of the next feature extraction layer meet the preset semantic requirements.

[0146] Among them, the specific implementation method of "the semantic judgment layer judges the output semantic features of the previous feature extraction layer to determine whether the output semantic features meet the preset semantic requirements" can be "judging the degree of matching between the output semantic features and the preset expected semantic features; if the degree of matching between the output semantic features and the preset expected semantic features meets the preset matching threshold, then determining that the output semantic features meet the preset semantic requirements; if the degree of matching between the output semantic features and the preset expected semantic features does not meet the preset matching threshold, then determining that the output semantic features do not meet the preset semantic requirements", and other methods can also be adopted, which are not specifically limited here.

[0147] The present application also provides a training method corresponding to the semantic extraction method based on dynamic adjustment of model computing power. The training method adopts recursive logic to judge the output semantic features of each round to determine whether to enter the next cycle. The detailed steps are as follows:

[0148] Step d1, calculate the hidden state of the complete Transformer.

[0149] In an embodiment of the present application, the current hidden state (hidden_states) is first processed through the Transformer layer, and then the processing result is normalized to obtain a new hidden state.

[0150] (hidden_current).

[0151] T i =Transformer(T i-1 )

[0152] hidden_current=Norm(T i )

[0153] Among them, T i-1 represents the current hidden state, T i Represents the new hidden state obtained after a complete Transformer is performed on the current hidden state.

[0154] Step d2: Generate the current output result logits of the language model based on the new hidden state.

[0155] logits_current=LM_Head(hidden_current)

[0156] Step d3: Use the routing gating mechanism to determine whether to proceed to the next cycle.

[0157] Use the routing network (router) to calculate whether each token enters the next cycle or exits the cycle, and obtain two weights:

[0158] Circle[exit] i Indicates that the current token tends to exit the loop; Circle[next] i Indicates that the current token tends to continue to the next cycle.

[0159] Router output weight formula:

[0160] Circle[next] i , Circle[exit] i =Router(hidden_current)

[0161] Then, the two weights are normalized:

[0162] weights=Softmax(Circle[next] i , Circle[exit] i )

[0163] exit_weights=weight[1]=Circle[exit] i

[0164] Step d4, exit logic processing.

[0165] For tokens that have been confirmed to be exited, their logits are weighted and accumulated to the final output:

[0166] final_logits[t]=final_logits[t]+logits_current[t]*exit_weights[t]

[0167] And update the token for the next loop, thereby implementing a parallel strategy:

[0168] T i =where(exit_mask, T i-1 , T i )

[0169] If all tokens have exited the loop, subsequent calculations are terminated; if not all have exited, updates are continued until the maximum number of iterations max_iter is reached.

[0170] For tokens that have not exited after the maximum number of iterations, directly use the logits of the last round:

[0171] final_logits=logits_current*exit_weinghts

[0172] The final training uses the cross entropy loss function to calculate the target token prediction loss, using final_logits, and is used for backpropagation:

[0173] L=CrossEntropy(final_logits, targets)

[0174] The embodiment of the present application performs semantic judgment on the predicted semantic features output by the multiple feature extraction layers of the model, that is, judges the degree of matching between the predicted semantic features and the preset expected semantic features. When the matching degree meets the preset matching threshold, the output result of the language model is generated through the predicted semantic features; when the matching degree between the predicted semantic features and the preset expected semantics does not meet the preset matching threshold, the predicted semantic features are used as new initial semantic features, and the multiple feature extraction layers are re-called to perform multiple feature extractions on the new initial semantic features until the output result of the language model is generated. In this way, different computing resources can be allocated to different tasks, thereby achieving the purpose of rational use of computing resources.

[0175] The present application also provides a computer device to execute the above semantic extraction method based on dynamic adjustment of model computing power. Figure 6 , which shows a schematic diagram of a computer device provided by some embodiments of the present application. Figure 6As shown, the computer device 6 includes: a processor 600, a memory 601, a bus 602 and a communication interface 603, and the processor 600, the communication interface 603 and the memory 601 are connected via the bus 602; the memory 601 stores a computer program that can be run on the processor 600, and when the processor 600 runs the computer program, it executes the semantic extraction method based on dynamic adjustment of model computing power provided by any of the aforementioned embodiments of the present application.

[0176] The memory 601 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element and at least one other network element are connected via at least one communication interface 603 (which may be wired or wireless), and may use the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0177] Bus 602 can be an ISA bus, a PCI bus, or an EISA bus. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 601 is used to store programs, and processor 600 executes the programs after receiving execution instructions. The semantic extraction method based on dynamic adjustment of model computing power disclosed in any of the aforementioned embodiments can be applied to processor 600 or implemented by processor 600.

[0178] The processor 600 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 600 or by software instructions. The above processor 600 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 601 , and the processor 600 reads the information in the memory 601 and completes the steps of the above method in combination with its hardware.

[0179] The computer device provided in the embodiment of the present application and the semantic extraction method based on dynamic adjustment of model computing power provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented by them.

[0180] The present application also provides a computer-readable storage medium corresponding to the semantic extraction method based on dynamic adjustment of model computing power provided in the above embodiment. Figure 7 The computer-readable storage medium shown is a CD 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by the processor, it will execute the semantic extraction method based on dynamic adjustment of model computing power provided by any of the aforementioned embodiments.

[0181] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0182] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the semantic extraction method based on dynamic adjustment of model computing power provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0183] It should be noted that:

[0184] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0185] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present application.

[0186] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this application and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0187] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A semantic extraction method based on dynamic adjustment of model computing power, characterized in that: The language model includes multiple feature extraction layers; the method includes: Calling the multiple feature extraction layers to perform multiple feature extractions on the initial semantic features to obtain predicted semantic features; Determining the degree of match between the predicted semantic features and the preset expected semantic features; If the matching degree between the predicted semantic feature and the preset expected semantic feature meets a preset matching threshold, generating an output result of the language model using the predicted semantic feature; If the degree of matching between the predicted semantic feature and the preset expected semantics does not meet the preset matching threshold, the predicted semantic feature is used as the new initial semantic feature, and the multiple feature extraction layers are re-called to perform multiple feature extractions on the new initial semantic feature until the output result of the language model is generated.

2. The method according to claim 1, characterized in that Determining the degree of matching between the predicted semantic feature and the preset expected semantic feature includes: Calculating a first category probability and a second category probability corresponding to the predicted semantic feature; the first category probability represents a probability that the predicted semantic feature is used to generate an output result of the language model; the second category probability represents a probability that the predicted semantic feature is used as a new initial semantic feature; the sum of the first category probability and the second category probability is 1; If the first category probability is greater than the second category probability, determining whether the degree of matching between the predicted semantic feature and the preset expected semantic feature meets a preset matching threshold; If the first category probability is less than or equal to the second category probability, it is determined that the degree of matching between the predicted semantic feature and the preset expected semantic feature does not meet a preset matching threshold.

3. The method according to claim 2, characterized in that Calculating the first category probability and the second category probability corresponding to the predicted semantic feature includes: Calculating a score vector based on the predicted semantic features and a preset weight matrix; each element in the score vector corresponds to a score of a different category; the preset weight matrix is ​​learned by the language model during the model training process; The score vector is converted into a probability distribution according to an activation function; the probability distribution includes the first category probability and the second category probability.

4. The method according to claim 3, characterized in that The score vector is converted into a probability distribution according to an activation function, including: Circle=softmax(AT) Among them, Circle represents probability distribution, AT represents score vector, softmax(.) represents activation function, A represents preset weight, and T represents predicted semantic feature.

5. The method according to claim 1, wherein The language model further includes a plurality of storage stacks; the calling of the plurality of feature extraction layers to perform multiple feature extractions on the initial semantic features to obtain predicted semantic features includes: Calling the i-th feature extraction layer to obtain first input data and second input data respectively; wherein the first input data is the output semantic feature of the i-1-th feature extraction layer; the second input data is the concatenation result of multiple semantic features stored in the multiple storage stacks; the semantic feature stored in each storage stack is a sliced ​​semantic feature obtained after dimensionality reduction and segmentation of the output semantic feature of the i-1-th feature extraction layer, where i is greater than or equal to 2; Performing feature extraction on the first input data and the second input data to obtain output semantic features of the i-th feature extraction layer; When the i-th feature extraction layer is the last layer of the multiple feature extraction layers, the output semantic features of the i-th feature extraction layer are used as the predicted semantic features.

6. The method according to claim 5, characterized in that A semantic judgment layer is provided between any two adjacent feature extraction layers; when i is greater than or equal to 2, after obtaining the output semantic features of the i-th feature extraction layer, the method further includes: Calling a target semantic judgment layer to judge whether the output semantic features of the i-th feature extraction layer meet preset semantic requirements; the target semantic judgment layer is set between the i-th feature extraction layer and the i+1-th feature extraction layer; If the output semantic features of the i-th feature extraction layer meet the preset semantic requirements, the output semantic features of the i-th feature extraction layer are used as the predicted semantic features.

7. The method according to claim 5, characterized in that In the case where the i-th feature extraction layer is not the last layer of the plurality of feature extraction layers, the method further includes: Get the output semantic features of the i-1th feature extraction layer; Performing dimensionality reduction processing on the output semantic features to obtain dimensionality-reduced output semantic features; Splitting the output semantic features after dimensionality reduction into a plurality of slice semantic features; For any slice semantic feature among the multiple slice semantic features, the slice semantic feature is stored in a corresponding storage stack; the multiple partial semantic features correspond one-to-one to the multiple storage stacks.

8. The method according to claim 7, characterized in that Calling the i-th feature extraction layer to obtain second input data includes: Reading a plurality of shard semantic features from the plurality of storage stacks respectively; Identifying a plurality of sequence tags of the plurality of fragment semantic features; splicing the plurality of fragment semantic features into a spliced ​​semantic feature according to the order of the high bits and the low bits indicated by the plurality of bit sequence labels; The splicing semantic feature is subjected to dimensionality-upgrading processing to obtain a dimensionality-upgraded splicing semantic feature, and the dimensionality-upgraded splicing semantic feature is used as the second input data; the dimension value of the dimensionality-upgrading is the same as the dimension value of the dimensionality-reducing.

9. A language model based on dynamic adjustment of computing power, characterized in that: It includes an input layer, a recognition layer, and an output layer. The recognition layer includes a computing power judgment layer and multiple feature extraction layers. The multiple feature extraction layers are used to perform multiple feature extractions on the initial semantic features layer by layer to obtain predicted semantic features; The computing power judgment layer is used to judge the degree of matching between the predicted semantic feature and the preset expected semantic feature; if the degree of matching between the predicted semantic feature and the preset expected semantic feature meets the preset matching threshold, the output result of the language model is generated through the predicted semantic feature; if the degree of matching between the predicted semantic feature and the preset expected semantic does not meet the preset matching threshold, the predicted semantic feature is used as a new initial semantic feature, and the multiple feature extraction layers are re-called to perform multiple feature extractions on the new initial semantic feature until the output result of the language model is generated.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the semantic extraction method based on dynamic adjustment of model computing power according to any one of claims 1 to 8 by executing the computer instructions.

Images