Semantic extraction method based on model computing power dynamic adjustment, and construction method of language model

Abstract

The application provides a semantic extraction method based on model computing power dynamic adjustment, a model and computer equipment. The method comprises the following steps: calling a plurality of feature extraction layers to perform multiple feature extractions on initial semantic features to obtain predicted semantic features; judging the matching degree of the predicted semantic features and preset expected semantic features; if the matching degree of the predicted semantic features and the preset expected semantic features meets a preset matching threshold, generating an output result of a language model through the predicted semantic features; and if the matching degree of the predicted semantic features and the preset expected semantic features does not meet the preset matching threshold, taking the predicted semantic features as new initial semantic features, re-calling the plurality of feature extraction layers to perform multiple feature extractions on the new initial semantic features until the output result of the language model is generated. The embodiments of the application can allocate different computing resources for different tasks, so that the purpose of reasonably using computing resources is achieved.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, specifically to a semantic extraction method, model, and computer device based on dynamic adjustment of model computing power. Background Technology

[0002] Transformer models are used in Natural Language Processing (NLP) and rely on self-attention mechanisms to capture long-distance dependencies in sequential data.

[0003] However, traditional Transformer models allocate the same computational resources to each token when handling different tasks. For example, in predicting simple mathematical problems (such as "1+1=?") and complex mathematical problems (such as "321*321="), the model schedules the same computational power for each token. This "one-size-fits-all" approach is overly complex when handling simple tasks, while it may be insufficient to capture all the necessary information when handling complex tasks. Summary of the Invention

[0004] In view of this, this application proposes a semantic extraction method, model, and computer device based on dynamic adjustment of model computing power, in order to solve the problem of unreasonable resource use caused by the same computing resources allocated to different tasks in related technologies.

[0005] The first aspect of this application proposes a semantic extraction method based on dynamic adjustment of model computing power, including:

[0006] The multiple feature extraction layers are invoked to extract features from the initial semantic features multiple times to obtain the predicted semantic features;

[0007] Determine the degree of matching between the predicted semantic features and the preset expected semantic features;

[0008] If the degree of matching between the predicted semantic features and the preset expected semantic features meets the preset matching threshold, then the output result of the language model is generated through the predicted semantic features;

[0009] If the degree of matching between the predicted semantic features and the preset expected semantics does not meet the preset matching threshold, then the predicted semantic features are used as new initial semantic features, and the multiple feature extraction layers are called again to extract features from the new initial semantic features multiple times until the output result of the language model is generated.

[0010] In this embodiment of the application, determining the degree of matching between the predicted semantic features and the preset expected semantic features includes:

[0011] Calculate the first category probability and the second category probability corresponding to the predicted semantic feature; the first category probability represents the probability that the predicted semantic feature is used to generate the output result of the language model; the second category probability represents the probability that the predicted semantic feature is used as a new initial semantic feature; the sum of the first category probability and the second category probability is 1;

[0012] If the probability of the first category is greater than the probability of the second category, then the degree of matching between the predicted semantic feature and the preset expected semantic feature is determined to meet the preset matching threshold.

[0013] If the probability of the first category is less than or equal to the probability of the second category, then it is determined that the degree of matching between the predicted semantic feature and the preset expected semantic feature does not meet the preset matching threshold.

[0014] In this embodiment of the application, calculating the first category probability and the second category probability corresponding to the predicted semantic feature includes:

[0015] A score vector is calculated based on the predicted semantic features and a preset weight matrix; each element in the score vector corresponds to a score for a different category; the preset weight matrix is ​​learned by the language model during model training.

[0016] The score vector is converted into a probability distribution according to the activation function; the probability distribution includes the first category probability and the second category probability.

[0017] In this embodiment of the application, converting the score vector into a probability distribution according to the activation function includes:

[0018]

[0019] in, Represents a probability distribution. Represents the score vector. This represents the activation function. Indicates the preset weight. This represents the predicted semantic features.

[0020] In this embodiment of the application, the language model further includes multiple storage stacks; the step of calling the multiple feature extraction layers to perform multiple feature extractions on the initial semantic features to obtain predicted semantic features includes:

[0021] The i-th feature extraction layer is invoked to obtain the first input data and the second input data respectively; wherein, the first input data is the output semantic feature of the (i-1)-th feature extraction layer; the second input data is the concatenation result of multiple semantic features stored in the multiple storage stacks; the semantic feature stored in each storage stack refers to the segmented semantic feature obtained after the output semantic feature of the (i-1)-th feature extraction layer has undergone dimensionality reduction and segmentation processing, where i is greater than or equal to 2;

[0022] The output semantic features of the i-th feature extraction layer are obtained by performing feature extraction using the first input data and the second input data.

[0023] When the i-th feature extraction layer is the last of the plurality of feature extraction layers, the output semantic features of the i-th feature extraction layer are used as the predicted semantic features.

[0024] In this embodiment of the application, a semantic judgment layer is provided between any two adjacent feature extraction layers; after obtaining the output semantic features of the i-th feature extraction layer when i is greater than or equal to 2, the method further includes:

[0025] The target semantic judgment layer is invoked to determine whether the output semantic features of the i-th feature extraction layer meet the preset semantic requirements; the target semantic judgment layer is set between the i-th feature extraction layer and the (i+1)-th feature extraction layer.

[0026] If the output semantic features of the i-th feature extraction layer meet the preset semantic requirements, then the output semantic features of the i-th feature extraction layer are used as the predicted semantic features.

[0027] In this embodiment of the application, when the i-th feature extraction layer is not the last layer of multiple feature extraction layers, the method further includes:

[0028] Obtain the output semantic features of the (i-1)th feature extraction layer;

[0029] The output semantic features are then subjected to dimensionality reduction processing to obtain the dimensionality-reduced output semantic features;

[0030] The dimensionality-reduced output semantic features are divided into multiple segmented semantic features;

[0031] For any one of the plurality of segmented semantic features, the segmented semantic feature is stored in the corresponding storage stack; the plurality of partial semantic features correspond one-to-one with the plurality of storage stacks.

[0032] In this embodiment of the application, calling the i-th feature extraction layer to obtain the second input data includes:

[0033] Multiple slice semantic features are read from the multiple storage stacks respectively;

[0034] Identify multiple positional labels of the multiple segmented semantic features;

[0035] According to the order of the high and low bits indicated by the multiple positional tags, the multiple segmented semantic features are concatenated into a concatenated semantic feature;

[0036] The concatenated semantic features are subjected to dimensionality-upgrading processing to obtain dimensionality-upgraded concatenated semantic features, which are then used as the second input data; the dimensionality value of the dimensionality-upgrading process is the same as that of the dimensionality-reducing process.

[0037] An embodiment of the third aspect of this application provides a language model based on dynamic adjustment of computing power, including an input layer, a recognition layer and an output layer, wherein the recognition layer includes a computing power judgment layer and multiple feature extraction layers;

[0038] The multiple feature extraction layers are used to extract features from the initial semantic features layer by layer to obtain the predicted semantic features;

[0039] The computing power judgment layer is used to determine the degree of matching between the predicted semantic features and the preset expected semantic features. If the degree of matching between the predicted semantic features and the preset expected semantic features meets the preset matching threshold, the output result of the language model is generated through the predicted semantic features. If the degree of matching between the predicted semantic features and the preset expected semantic features does not meet the preset matching threshold, the predicted semantic features are used as new initial semantic features, and the multiple feature extraction layers are called again to extract features from the new initial semantic features multiple times until the output result of the language model is generated.

[0040] An embodiment of the third aspect of this application provides a computer device including a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the semantic extraction method based on dynamic adjustment of model computing power described in the first aspect.

[0041] An embodiment of the fourth aspect of this application provides a computer-readable storage medium storing computer instructions, the computer instructions being used to cause a computer to execute the semantic extraction method based on dynamic adjustment of model computing power as described in the first aspect above.

[0042] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0043] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0044] Figure 1 This illustration shows a schematic diagram of the structure of a language model based on dynamic adjustment of computing power according to an embodiment of this application;

[0045] Figure 2 This illustration shows a schematic diagram of the structure of the recognition layer in a language model based on dynamic adjustment of computing power provided in an embodiment of this application;

[0046] Figure 3 A schematic flowchart of a semantic extraction method based on dynamic adjustment of model computing power provided in an embodiment of this application is shown.

[0047] Figure 4 This illustration shows a flowchart of the process of extracting features from the output semantic features of the previous feature extraction layer and from data stored in multiple storage stacks, according to an embodiment of this application.

[0048] Figure 5 A schematic diagram of a storage stack provided in an embodiment of this application is shown;

[0049] Figure 6 This invention provides a schematic diagram of the structure of a computer device according to an embodiment of the present application.

[0050] Figure 7 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0051] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0052] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0053] The technical scenarios involved in the embodiments of this application are described below.

[0054] The Transformer model is a deep learning model based on a self-attention mechanism. However, its effectiveness is limited in practical training; it cannot model and complete higher-level formal language tasks, thus affecting its performance and expressive power.

[0055] The Transformer model is mainly composed of two parts: an encoder and a decoder. Each part enhances the model's representation and learning capabilities by stacking multiple identical layers.

[0056] The encoder consists of N identical stacked layers, each primarily composed of two sub-layers: a multi-head self-attention mechanism and a feedforward neural network. Residual connections and layer normalization are applied after each sub-layer in the encoder to stabilize the training process. The encoder layer is mainly responsible for processing the input sequence and generating a series of encoded representations. These encoded representations capture the contextual information within the input sequence.

[0057] The decoder also consists of N identical layers stacked together, but each layer, in addition to containing the same two sub-layers as the encoder, adds an extra multi-head attention sub-layer to process the encoder output: multi-head self-attention mechanism, multi-head encoder-decoder attention mechanism (sub-layer), and feedforward neural network. Residual connections and layer normalization are applied after each sub-layer in the decoder. The decoder layer is responsible for generating the next output based on the encoder output and the previously generated output sequence (in the generation task).

[0058] Formal language theory is a branch of computer science that studies the abstract structure and rules of languages. The formal language hierarchy describes different types of languages ​​and their generative capabilities.

[0059] According to an embodiment of this application, a semantic extraction method based on dynamic adjustment of model computing power is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0060] This disclosure proposes a language model based on dynamic adjustment of computing power, such as... Figure 1 As shown;

[0061] The language model based on dynamic adjustment of computing power includes an input layer 101, a recognition layer 102 and an output layer 103. The recognition layer includes a computing power judgment layer, multiple feature extraction layers and multiple storage stacks.

[0062] The input layer 101 can be used to extract preliminary semantic features from the input language to be recognized. For example, the language to be recognized can be a language with abstract semantics, including mathematical formulas, computer programming languages, and chemical formulas. The input layer 101 can convert the language directly input by the user into initial semantic features in the form of semantic vectors, etc., and these initial semantic features can be vector transformations, etc.

[0063] The recognition layer 102 is used to extract features from the initial semantic features to achieve semantic recognition. For example... Figure 2 The identification layer 102 shown includes multiple feature extraction layers and multiple storage stacks.

[0064] The multiple feature extraction layers are used to extract features from the initial semantic features layer by layer to obtain the predicted semantic features.

[0065] Specifically, taking the Transformer model as an example, the recognition layer 102 may include the encoder layer and decoder layer of the Transformer model, as well as multiple corresponding storage stacks. The encoder layer can serve as the first feature extraction layer, and the decoder layer can serve as the second feature extraction layer. The output data of the first feature extraction layer can be divided into multiple parts, and then the multiple parts are stored in the corresponding storage stacks. When performing feature extraction in the second feature extraction layer, the input data consists of the output data of the first feature extraction layer and the data obtained from the multiple storage stacks.

[0066] More specifically, when i equals 1, the first feature extraction layer is used to extract features from the initial semantic features of the language to be recognized to obtain the output semantic features of the first feature extraction layer; when i is greater than or equal to 2, the i-th feature extraction layer is used to obtain the first input data and the second input data respectively, and to extract features from the first input data and the second input data to obtain the output semantic features of the i-th feature extraction layer; wherein, the first input data is the output semantic features of the (i-1)-th feature extraction layer; the second input data is the concatenation result of multiple semantic features stored in the multiple storage stacks; the semantic features stored in each storage stack refer to the segmented semantic features obtained after dimensionality reduction and segmentation of the output semantic features of the (i-1)-th feature extraction layer; when the i-th feature extraction layer is the last layer of the multiple feature extraction layers, the output semantic features of the i-th feature extraction layer are used as the predicted semantic features.

[0067] Taking the Transformer model as an example, both the encoder and decoder include multiple sub-layers. Each sub-layer of the encoder includes a multi-head self-attention mechanism, a feedforward neural network, and residual connection layers and layer normalization layers for stabilization. In the embodiments of this application, for the layers within the multi-head self-attention mechanism and feedforward neural network sub-layers of the encoder, multiple output data from the multi-head self-attention mechanism can be stored in multiple memory stacks. The input data in the feedforward neural network consists of the output data from the multi-head self-attention mechanism and the data in the memory stacks. Similarly, the decoder is similar, and will not be elaborated upon here. The feature extraction process of the recognition layer 102, as well as the storage and retrieval process of the memory stacks, will be described in the embodiments below, and will not be repeated here.

[0068] The computing power judgment layer is used to determine the degree of matching between the predicted semantic features and the preset expected semantic features. If the degree of matching between the predicted semantic features and the preset expected semantic features meets the preset matching threshold, the output result of the language model is generated through the predicted semantic features. If the degree of matching between the predicted semantic features and the preset expected semantic features does not meet the preset matching threshold, the predicted semantic features are used as new initial semantic features, and the multiple feature extraction layers are called again to extract features from the new initial semantic features multiple times until the output result of the language model is generated.

[0069] Output layer 103 is used to integrate and output the data in recognition layer 102.

[0070] This embodiment provides a semantic extraction method based on dynamic adjustment of model computing power. Figure 3 This is a flowchart of a semantic extraction method based on dynamic adjustment of model computing power according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps:

[0071] Step S101: Call the multiple feature extraction layers to perform multiple feature extractions on the initial semantic features to obtain the predicted semantic features.

[0072] In some specific embodiments, step S101 above includes steps S1011-S1013:

[0073] Step S1011: Call the i-th feature extraction layer to obtain the first input data and the second input data respectively.

[0074] Specifically, the first input data is the output semantic feature of the (i-1)th feature extraction layer; the second input data is the concatenation result of multiple semantic features stored in the multiple storage stacks; the semantic feature stored in each storage stack refers to the segmented semantic feature obtained after the output semantic feature of the (i-1)th feature extraction layer has undergone dimensionality reduction and segmentation processing, where i is greater than or equal to 2;

[0075] In this embodiment, a language model is invoked to extract initial semantic features of the language layer by layer multiple times to obtain predicted semantic features, which are the output semantic features of the last layer of multiple feature extraction layers. The semantic features of the language can be natural language or formal language; this embodiment does not limit the specific form of the language, which can be determined by those skilled in the art based on the actual situation.

[0076] In this application embodiment, the language model can be understood as described above. Figure 1 The language model is dynamically adjusted based on computing power. When the language model is called to extract features from the initial semantic features, each feature extraction layer further extracts the output data of the previous feature extraction layer. On this basis, the "output data of the previous feature extraction layer" and "data extracted from the storage stack" are used as the input data of the next feature extraction layer, realizing multi-dimensional feature extraction of the language to be recognized and improving the accuracy of feature extraction.

[0077] In some specific embodiments, when the feature extraction layer is the first layer among multiple feature extraction layers, that is, when i=1, the first input data obtained by the i-th feature extraction layer is the initial semantic features of the language to be recognized, and the second input data obtained by the i-th feature extraction layer is empty.

[0078] In some specific embodiments, when the i-th feature extraction layer is not the last layer of multiple feature extraction layers, the method further includes steps a1-a4:

[0079] Step a1: Obtain the output semantic features of the (i-1)th feature extraction layer.

[0080] Step a2: Perform dimensionality reduction processing on the output semantic features to obtain the dimensionality-reduced output semantic features.

[0081] Specifically, for example, the dimension value of the output semantic features can be adjusted from 2048 dimensions to 64 dimensions, for example. Figure 4 As shown, although the dimensionality of the output semantic features is reduced, the dimensionality-reduced output semantic features retain most of the core features of the original output semantic features.

[0082] Step a3: Divide the dimensionality-reduced output semantic features into multiple segmented semantic features.

[0083] Step a4: For any one of the multiple segmented semantic features, store the segmented semantic feature into the corresponding storage stack; the multiple partial semantic features correspond one-to-one with the multiple storage stacks.

[0084] In steps a2-a3 above, a multi-head mechanism is used to divide the dimensionality-reduced output semantic features into multiple segmented semantic features according to a preset dimension value, and each segmented semantic feature is stored in its corresponding storage stack. For example... Figure 4 As shown: The 64-dimensional output semantic features are divided into four 16-dimensional segmented semantic features. The four 16-dimensional segmented semantic features are stored in four storage stacks head_0, head_1, head_2, and head_3 respectively. That is: segmented semantic feature 0 is stored in storage stack head_0, segmented semantic feature 1 is stored in storage stack head_1, segmented semantic feature 2 is stored in storage stack head_2, and segmented semantic feature 3 is stored in storage stack head_3.

[0085] In steps a2-a3 above, the dimension value of the output semantic features after dimensionality reduction is determined based on the number of multiple storage stacks and the storage space size of each storage stack. For example, assuming the dimension value of the output semantic features before dimensionality reduction is 2048, and the initial language model has 4 storage stacks, with each storage stack having a storage space of 16 dimensions, then the dimension value of the output semantic features after dimensionality reduction is 4 * 16 = 64 dimensions. In other words, the dimension value of the output semantic features needs to be reduced from 2048 to 64.

[0086] In some specific embodiments, calling the i-th feature extraction layer to obtain the second input data includes:

[0087] Step b1: Read multiple slice semantic features from the multiple storage stacks respectively.

[0088] Specifically, for example Figure 4 As shown: read the shard semantic feature 0 from the storage stack head_0, read the shard semantic feature 1 from the storage stack head_1, read the shard semantic feature 2 from the storage stack head_2, read the shard semantic feature 3 from the storage stack head_3, ..., read the shard semantic feature n from the storage stack head_n.

[0089] Step b2: Identify multiple positional labels of the multiple segmented semantic features.

[0090] Specifically, for example, the position label of the segmented semantic feature 0 is S0, which indicates the lowest position of the segmented semantic feature 0 in the concatenated semantic features, and the position label of the segmented semantic feature n is Sn, which indicates the highest position of the segmented semantic feature n in the concatenated semantic features.

[0091] Step b3: According to the order of the high and low bits indicated by the multiple positional tags, the multiple segmented semantic features are concatenated into a concatenated semantic feature.

[0092] Specifically, the positional label of each segment semantic feature can determine the corresponding position of each segment semantic feature in the concatenated semantic feature, thereby concatenating each segment semantic feature in the corresponding position to obtain the concatenated semantic feature.

[0093] Step b4: Perform dimensionality upscaling on the concatenated semantic features to obtain dimensionality upscaled concatenated semantic features, and use the dimensionality upscaled concatenated semantic features as the second input data.

[0094] Specifically, for example, the dimension value of the concatenated semantic features can be adjusted from 64 to 2048. The dimension value of the dimensionality increase is the same as that of the dimensionality decrease. For example, if the dimensionality decrease is 2048→64, then the dimensionality increase is 64→2048.

[0095] In some specific embodiments, step a4 above includes steps a41-a43:

[0096] Step a41: For any one of the plurality of storage bits, determine the fragmentation semantic features of the storage bit at the current calculation time based on the fragmentation semantic features stored in the storage bit at the previous calculation time, the fragmentation semantic features stored in the adjacent storage bits of the storage bit, and a preset probability.

[0097] The adjacent storage bits include the low-order storage bits and the high-order storage bits adjacent to the storage bit; the calculation time refers to the time when the corresponding feature extraction layer performs calculation and stores the output result to the storage stack, and one calculation time corresponds to one feature extraction layer.

[0098] In this embodiment of the application, each storage stack includes multiple storage bits, for example Figure 5 As shown.

[0099] When storing each segment semantic feature into its corresponding preset storage stack, the preset probabilities include the write probability, the invariance probability, and the deletion probability. Taking any storage bit as an example, the weighted result of the first, second, and third semantic features from the previous calculation time is determined as the segment semantic feature stored in the storage bit at the current calculation time. Specifically, the first semantic feature is the product of the segment semantic feature stored in the lower storage bits and the write probability; the second semantic feature is the product of the segment semantic feature stored in the storage bit and the invariance probability; and the third semantic feature is the product of the deletion probabilities of the segment semantic features stored in the higher storage bits.

[0100] Specifically, the segmentation semantic features corresponding to the storage bits can be calculated using the following formula:

[0101]

[0102] Where t is the computation time and n is the index of the storage location. The segmentation semantic features stored in the nth storage bit at the current computation time. The segmentation semantic features stored in the (n-1)th storage bit (lower storage bit) of the previous calculation time. This represents the write probability at the current calculation moment. The segmentation semantic features stored in the current storage bit at the previous calculation time. Let be the invariant probability at the current calculation time. The segmentation semantic features are stored in the (n+1)th storage bit (high-order storage bit) of the previous calculation time. This represents the constant probability at the current calculation time.

[0103] Among them, the three probabilities in the preset probabilities can be preset with corresponding probability values, or they can be predicted based on neural models, etc. The embodiments of this application do not limit the method of obtaining the preset probabilities, and those skilled in the art can determine it according to the actual situation.

[0104] Step a42: If the storage bit is the lowest storage bit among the plurality of storage bits, then the data of the lowest storage bit in the adjacent storage location is the output semantic feature of the feature extraction layer corresponding to the previous calculation time.

[0105] Step a43: If the storage bit is the highest storage bit among the plurality of storage bits, then the data stored in the high storage bit in the adjacent storage location is empty.

[0106] Step S1012: Perform feature extraction using the first input data and the second input data to obtain the output semantic features of the i-th feature extraction layer.

[0107] In this embodiment, feature extraction can be performed on the fused data of the first input data and the second input data to obtain the output semantic features of the i-th feature extraction layer. The fusion method includes, but is not limited to, directly concatenating the first input data and the second input data, or performing a weighted summation of the first input data and the second input data. This embodiment does not limit the fusion method; those skilled in the art can determine it according to the actual situation.

[0108] In some embodiments, the fusion feature can also be generated in the following manner: concatenating the output semantic feature of the previous feature extraction layer and the concatenated semantic feature to obtain a third semantic feature; performing a dot product operation between the third semantic feature and a preset weight matrix to obtain the probability of each element in the third semantic feature; and obtaining the fusion semantic feature based on each element in the third semantic feature and the probability corresponding to each element.

[0109] For example, during feature fusion, the output semantic features of the previous feature extraction layer and the concatenated semantic features can first be concatenated to obtain a third semantic feature. After obtaining the third semantic feature, the probability of each element in the third semantic feature is calculated. Specifically, the probability of each element can be obtained by passing the third semantic feature through a fully connected layer. The fused feature is obtained by weighted summation of each element and its corresponding probability.

[0110] In some embodiments, regularization can be performed after weighted summation, and the fusion feature can be obtained after averaging. Specifically, each element in the third semantic feature is multiplied by its corresponding probability to obtain a probabilistic fusion feature; the probabilistic fusion features corresponding to each element in the third semantic feature are added together to obtain an initial fusion feature; the initial fusion feature is divided by the number of elements to obtain the fusion semantic feature.

[0111] Step S1013: When the i-th feature extraction layer is the last layer of the plurality of feature extraction layers, the output semantic features of the i-th feature extraction layer are used as the predicted semantic features.

[0112] In some specific embodiments, after using the output semantic features of the i-th feature extraction layer as the predicted semantic features, the method further includes a model training step, namely: calculating the loss value between the output semantic features of the i-th feature extraction layer and the preset standard semantic features; if the loss value satisfies the preset convergence condition, the language model is determined as a reasoning language model based on a multi-head differentiable stack, and the reasoning language model based on a multi-head differentiable stack has the function of recognizing the semantics of formal languages.

[0113] This embodiment of the application obtains first input data and second input data by calling the i-th feature extraction layer in the language model, and extracts the output semantic features of the i-th feature extraction layer by performing feature extraction on the first input data and second input data. The first input data is the output semantic features of the (i-1)-th feature extraction layer; the second input data is the concatenation result of multiple semantic features stored in multiple storage stacks; the semantic features stored in each storage stack refer to the segmented semantic features obtained after dimensionality reduction and segmentation of the output semantic features of the (i-1)-th feature extraction layer. Because the input data for feature extraction is expanded from only the output data of the previous feature extraction layer to include data stored in storage stacks, the accuracy of language recognition is improved. Furthermore, the target inference language model based on a differentiable stack, trained according to subsequent steps, can handle languages ​​of a certain complexity.

[0114] Preferably, in this embodiment of the application, by performing dimensionality reduction processing on the output semantic features of the feature extraction layer, the memory overhead of the semantic model during model training can be reduced, and the training speed can be faster; and by segmenting the output semantic features after dimensionality reduction, the modeling effect can be improved, that is, the modeling ability of formal language and general language can be improved.

[0115] Step S102: Determine the degree of matching between the predicted semantic features and the preset expected semantic features.

[0116] In this embodiment of the application, by calculating the degree of matching between the predicted semantic features and the preset expected semantic features, it can be determined whether to use the predicted semantic features to generate the output result of the language model, or to use the predicted semantic features as new initial semantic features and re-call the multiple feature extraction layers to extract features from the new initial semantic features multiple times until the output result of the language model is generated.

[0117] In some specific embodiments, step S102 above includes steps S1021-S1023:

[0118] Step S1021: Calculate the first category probability and the second category probability corresponding to the predicted semantic features.

[0119] Specifically, the first category probability represents the probability that the predicted semantic feature is used to generate the output of the language model; the second category probability represents the probability that the predicted semantic feature is used as a new initial semantic feature; and the sum of the first category probability and the second category probability is 1.

[0120] In some specific embodiments, step S1021 above includes steps c1-c2:

[0121] Step c1: Calculate the score vector based on the predicted semantic features and the preset weight matrix.

[0122] Specifically, each element in the score vector corresponds to a score for a different category; the preset weight matrix is ​​learned by the language model during model training.

[0123] Step c2: Convert the score vector into a probability distribution according to the activation function; the probability distribution includes the first category probability and the second category probability.

[0124] In the embodiments of this application, as shown in the following formula:

[0125]

[0126] in, Represents a probability distribution. Represents the score vector. This represents the activation function. Indicates the preset weight. This represents the predicted semantic features.

[0127] Furthermore, the above formula can be transformed into:

[0128]

[0129] in, i Indicates the number of loops, for example:

[0130]

[0131] ...

[0133] This represents the predicted semantic features output by the language model after one iteration on the initial semantic features. One iteration refers to the step of calling multiple feature extraction layers to extract features from the initial semantic features multiple times.

[0134] This means that the new predicted semantic features are obtained through a second loop, that is: the predicted semantic features output by the language model after the first loop are re-input into the model, and multiple feature extraction layers of the model are called to extract features from the predicted semantic features multiple times to obtain new predicted semantic features.

[0135] In this embodiment of the application, the probability of the first category can be calculated using the above method. Second category probability Based on this, determine the probability of the first category. Second category probability The size relationship is used to determine whether to proceed to the next iteration.

[0136] Step S1022: If the probability of the first category is greater than the probability of the second category, then it is determined that the degree of matching between the predicted semantic feature and the preset expected semantic feature meets the preset matching threshold.

[0137] Step S1023: If the probability of the first category is less than or equal to the probability of the second category, then it is determined that the degree of matching between the predicted semantic feature and the preset expected semantic feature does not meet the preset matching threshold.

[0138] In the above steps S1022-S1023,

[0139] if If the predicted semantic features are used, then the output of the language model is generated; otherwise, if Then, the predicted semantic features are used as new initial semantic features, and the multiple feature extraction layers are called again to extract features from the new initial semantic features multiple times until the output of the language model is generated.

[0140] In some specific embodiments, a semantic judgment layer is set between any two adjacent feature extraction layers; after obtaining the output semantic features of the i-th feature extraction layer when i is greater than or equal to 2, the method further includes:

[0141] The target semantic judgment layer is invoked to determine whether the output semantic features of the i-th feature extraction layer meet the preset semantic requirements; the target semantic judgment layer is set between the i-th feature extraction layer and the (i+1)-th feature extraction layer.

[0142] If the output semantic features of the i-th feature extraction layer meet the preset semantic requirements, then the output semantic features of the i-th feature extraction layer are used as the predicted semantic features.

[0143] In this embodiment, the language model includes multiple feature extraction layers, with a semantic judgment layer positioned between any two adjacent feature extraction layers, for example: Feature Extraction Layer 1 → Semantic Judgment Layer → Feature Extraction Layer 2 → Semantic Judgment Layer → Feature Extraction Layer 3 → ...; Each semantic judgment layer judges the output semantic features of the previous feature extraction layer to determine whether the output semantic features meet preset semantic requirements. If they do, the output semantic features of the previous feature extraction layer are stored in multiple storage stacks, and the feature extraction operations of several feature extraction layers following the semantic judgment layer are paused. The semantic features stored in the multiple storage stacks are used as the semantic features in the language model. The recognition results of the previous feature extraction layer are used to enable the output layer to generate the output of the language model based on the recognition results. Conversely, if the output semantic features do not meet the preset semantic requirements, the output semantic features of the previous feature extraction layer are stored in multiple storage stacks, and the output semantic features of the previous feature extraction layer are sent to the next feature extraction layer. The next feature extraction layer then performs feature extraction based on the output semantic features of the previous feature extraction layer and the semantic features stored in multiple storage stacks to obtain the output semantic features of the next feature extraction layer. Finally, the semantic judgment layer connected to the next feature extraction layer is called to determine whether the output semantic features of the next feature extraction layer meet the preset semantic requirements.

[0144] The specific implementation of "the semantic judgment layer judges the output semantic features of the previous feature extraction layer to determine whether the output semantic features meet the preset semantic requirements" can be implemented by "judging the degree of matching between the output semantic features and the preset expected semantic features; if the degree of matching between the output semantic features and the preset expected semantic features meets the preset matching threshold, then the output semantic features are determined to meet the preset semantic requirements; if the degree of matching between the output semantic features and the preset expected semantic features does not meet the preset matching threshold, then the output semantic features are determined not to meet the preset semantic requirements". Other methods can also be adopted, which are not specifically limited here.

[0145] This application also provides a training method corresponding to the semantic extraction method based on dynamic adjustment of model computing power. The training method adopts recursive logic, and judges the output semantic features of each round to determine whether to enter the next round of the loop. The detailed steps are as follows:

[0146] Step d1: Calculate the hidden state of a complete Transformer.

[0147] In this embodiment, the current hidden states are first processed through the Transformer layer, and then the processing result is normalized to obtain a new hidden state. ).

[0148]

[0149]

[0150] in, Indicates the current hidden state. This represents the new hidden state obtained after performing a complete Transformer operation on the current hidden state.

[0151] Step d2: Generate the current output logits of the language model based on the new hidden state.

[0152]

[0153] Step d3: Use the routing gating mechanism to determine whether to proceed to the next loop.

[0154] The routing network is used to calculate whether each token should enter the next round of the loop or exit the loop, obtaining two weights:

[0155] This indicates that the current token is likely to exit the loop; This indicates that the current token is likely to continue to the next round of the cycle.

[0156] Router output weight formula:

[0157]

[0158] Then, normalize the two weights:

[0159]

[0160]

[0161] Step d4: Exit logic processing.

[0162] For tokens that have been confirmed to have exited, their logits are weighted and accumulated in the final output:

[0163]

[0164] And update the token for the next iteration, thus implementing a parallel strategy:

[0165]

[0166] If all tokens have exited the loop, the subsequent calculations will terminate; if not all tokens have exited, the update will continue until the maximum number of iterations, max_iter, is reached.

[0167] For tokens that have not exited after the maximum number of iterations, use the logits from the last round directly:

[0168]

[0169] The final training uses the cross-entropy loss function to calculate the target token prediction loss, uses final_logits, and is used for backpropagation:

[0170]

[0171] This application embodiment performs semantic judgment on the predicted semantic features output by multiple feature extraction layers of the model, that is, it judges the degree of matching between the predicted semantic features and preset expected semantic features. When the degree of matching meets a preset matching threshold, the output result of the language model is generated through the predicted semantic features; when the degree of matching between the predicted semantic features and the preset expected semantic features does not meet the preset matching threshold, the predicted semantic features are used as new initial semantic features, and the multiple feature extraction layers are called again to extract features from the new initial semantic features multiple times until the output result of the language model is generated. In this way, different computing resources can be allocated for different tasks, achieving the goal of rational use of computing resources.

[0172] This application also provides a computer device for performing the above-described semantic extraction method based on dynamic adjustment of model computing power. Please refer to... Figure 6 This illustrates a schematic diagram of a computer device provided by some embodiments of this application. For example... Figure 6 As shown, the computer device 6 includes: a processor 600, a memory 601, a bus 602, and a communication interface 603. The processor 600, the communication interface 603, and the memory 601 are connected via the bus 602. The memory 601 stores a computer program that can run on the processor 600. When the processor 600 runs the computer program, it executes the semantic extraction method based on dynamic adjustment of model computing power provided in any of the foregoing embodiments of this application.

[0173] The memory 601 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 603 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0174] Bus 602 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 601 is used to store programs. After receiving an execution instruction, the processor 600 executes the program. The semantic extraction method based on dynamic adjustment of model computing power disclosed in any of the foregoing embodiments can be applied to the processor 600, or implemented by the processor 600.

[0175] The processor 600 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 600 or by instructions in software form. The processor 600 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 601. Processor 600 reads the information in memory 601 and, in conjunction with its hardware, completes the steps of the above method.

[0176] The computer device provided in this application embodiment and the semantic extraction method based on dynamic adjustment of model computing power provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, run or implement.

[0177] This application also provides a computer-readable storage medium corresponding to the semantic extraction method based on dynamic adjustment of model computing power provided in the foregoing embodiments. Please refer to [link / reference]. Figure 7 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the semantic extraction method based on dynamic adjustment of model computing power provided in any of the foregoing embodiments.

[0178] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0179] The computer-readable storage medium provided in the above embodiments of this application and the semantic extraction method based on dynamic adjustment of model computing power provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0180] It should be noted that:

[0181] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0182] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0183] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0184] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A semantic extraction method based on dynamic adjustment of model computing power, characterized in that, The language model includes multiple feature extraction layers; the method includes: Preliminary semantic features are extracted from the input language to be recognized to obtain initial semantic features; The multiple feature extraction layers are invoked to extract features from the initial semantic features multiple times to obtain predicted semantic features; Determine the degree of matching between the predicted semantic features and the preset expected semantic features; If the degree of matching between the predicted semantic features and the preset expected semantic features meets the preset matching threshold, then the output result of the language model is generated through the predicted semantic features; If the degree of matching between the predicted semantic features and the preset expected semantics does not meet the preset matching threshold, then the predicted semantic features are used as new initial semantic features, and the multiple feature extraction layers are called again to extract features from the new initial semantic features multiple times until the output result of the language model is generated. The language model also includes multiple storage stacks; the step of calling the multiple feature extraction layers to perform multiple feature extractions on the initial semantic features to obtain predicted semantic features includes: The i-th feature extraction layer is called to obtain the first input data and the second input data respectively; wherein, the first input data is the output semantic feature of the (i-1)-th feature extraction layer; the second input data is the concatenation result of multiple semantic features stored in the multiple storage stacks; the semantic feature stored in each storage stack refers to the segmented semantic feature obtained after the output semantic feature of the (i-1)-th feature extraction layer has undergone dimensionality reduction and segmentation processing, and i is greater than or equal to 2; The output semantic features of the i-th feature extraction layer are obtained by performing feature extraction using the first input data and the second input data. When the i-th feature extraction layer is the last layer of the plurality of feature extraction layers, the output semantic features of the i-th feature extraction layer are used as the predicted semantic features; When the i-th feature extraction layer is not the last of multiple feature extraction layers, the method further includes: Obtain the output semantic features of the (i-1)th feature extraction layer; The output semantic features are then subjected to dimensionality reduction processing to obtain the dimensionality-reduced output semantic features; The dimensionality-reduced output semantic features are divided into multiple segmented semantic features; For any one of the plurality of segmented semantic features, the segmented semantic feature is stored in the corresponding storage stack; the plurality of segmented semantic features correspond one-to-one with the plurality of storage stacks.

2. The method according to claim 1, characterized in that, Determining the degree of matching between the predicted semantic features and the preset expected semantic features includes: Calculate the first category probability and the second category probability corresponding to the predicted semantic feature; the first category probability represents the probability that the predicted semantic feature is used to generate the output result of the language model; the second category probability represents the probability that the predicted semantic feature is used as a new initial semantic feature; the sum of the first category probability and the second category probability is 1; If the probability of the first category is greater than the probability of the second category, then the degree of matching between the predicted semantic feature and the preset expected semantic feature is determined to meet the preset matching threshold. If the probability of the first category is less than or equal to the probability of the second category, then it is determined that the degree of matching between the predicted semantic feature and the preset expected semantic feature does not meet the preset matching threshold.

3. The method according to claim 2, characterized in that, Calculating the first category probability and the second category probability corresponding to the predicted semantic features includes: A score vector is calculated based on the predicted semantic features and a preset weight matrix; each element in the score vector corresponds to a score for a different category; the preset weight matrix is ​​learned by the language model during model training. The score vector is converted into a probability distribution according to the activation function; the probability distribution includes the first category probability and the second category probability.

4. The method according to claim 3, characterized in that, The score vector is transformed into a probability distribution according to the activation function, including: in, Represents a probability distribution. Represents the score vector. This represents the activation function. Indicates the preset weight. This represents the predicted semantic features.

5. The method according to claim 1, characterized in that, A semantic judgment layer is set between any two adjacent feature extraction layers; after obtaining the output semantic features of the i-th feature extraction layer when i is greater than or equal to 2, the method further includes: The target semantic judgment layer is invoked to determine whether the output semantic features of the i-th feature extraction layer meet the preset semantic requirements; the target semantic judgment layer is set between the i-th feature extraction layer and the (i+1)-th feature extraction layer. If the output semantic features of the i-th feature extraction layer meet the preset semantic requirements, then the output semantic features of the i-th feature extraction layer are used as the predicted semantic features.

6. The method according to claim 1, characterized in that, Calling the i-th feature extraction layer to obtain the second input data includes: Multiple slice semantic features are read from the multiple storage stacks respectively; Identify multiple positional labels of the multiple segmented semantic features; According to the order of the high and low bits indicated by the multiple positional tags, the multiple segmented semantic features are concatenated into a concatenated semantic feature; The concatenated semantic features are subjected to dimensionality-upgrading processing to obtain dimensionality-upgraded concatenated semantic features, which are then used as the second input data; the dimensionality value of the dimensionality-upgrading process is the same as that of the dimensionality-reducing process.

7. A method for constructing a language model based on dynamic adjustment of computing power, characterized in that, The construction method includes an input layer, a recognition layer, and an output layer. The recognition layer includes a computing power judgment layer and multiple feature extraction layers. The input layer is used to perform preliminary semantic feature extraction on the input language to be recognized, and obtain initial semantic features; The multiple feature extraction layers are used to extract features from the initial semantic features layer by layer to obtain predicted semantic features; The computing power judgment layer is used to determine the degree of matching between the predicted semantic features and the preset expected semantic features. If the degree of matching between the predicted semantic features and the preset expected semantic features meets the preset matching threshold, the output result of the language model is generated through the predicted semantic features. If the degree of matching between the predicted semantic features and the preset expected semantic features does not meet the preset matching threshold, the predicted semantic features are used as new initial semantic features, and the multiple feature extraction layers are called again to extract features from the new initial semantic features multiple times until the output result of the language model is generated. The language model also includes multiple storage stacks; the step of calling the multiple feature extraction layers to perform multiple feature extractions on the initial semantic features to obtain predicted semantic features includes: The i-th feature extraction layer is called to obtain the first input data and the second input data respectively; wherein, the first input data is the output semantic feature of the (i-1)-th feature extraction layer; the second input data is the concatenation result of multiple semantic features stored in the multiple storage stacks; the semantic feature stored in each storage stack refers to the segmented semantic feature obtained after the output semantic feature of the (i-1)-th feature extraction layer has undergone dimensionality reduction and segmentation processing, and i is greater than or equal to 2; The output semantic features of the i-th feature extraction layer are obtained by performing feature extraction using the first input data and the second input data. When the i-th feature extraction layer is the last layer of the plurality of feature extraction layers, the output semantic features of the i-th feature extraction layer are used as the predicted semantic features; In the case where the i-th feature extraction layer is not the last of multiple feature extraction layers, the following is also included: Obtain the output semantic features of the (i-1)th feature extraction layer; The output semantic features are then subjected to dimensionality reduction processing to obtain the dimensionality-reduced output semantic features; The dimensionality-reduced output semantic features are divided into multiple segmented semantic features; For any one of the plurality of segmented semantic features, the segmented semantic feature is stored in the corresponding storage stack; the plurality of segmented semantic features correspond one-to-one with the plurality of storage stacks.

8. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory stores computer instructions, and the processor executes the computer instructions to perform the semantic extraction method based on dynamic adjustment of model computing power as described in any one of claims 1 to 6.

Images