Natural language semantic recognition method and device, equipment and medium

Abstract

The invention discloses a natural language semantic recognition method, device and equipment and a medium, and the method comprises the steps: extracting semantic features according to an input natural language, inputting the semantic features into an initial hybrid expert model, and analyzing the semantic features through a Transform model to obtain a hidden state matrix; inputting the hidden state matrix into a gating network to generate a routing weight matrix; on the basis of rank estimation of randomized singular value decomposition, generating the number K of expert networks activated by the current layer in combination with the hidden state matrix; selecting first K expert networks according to a sorting result of the routing weight matrix, generating a mask matrix, executing sparse forward calculation, and optimizing model parameters of the expert networks based on sparse gradient propagation and load balancing loss to obtain a target hybrid expert model; and the target hybrid expert model is adopted to recognize the semantic features to obtain a semantic recognition result, and the semantic recognition accuracy and efficiency are improved by adopting the semantic recognition method and device.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a natural language semantic recognition method, device, computer equipment and medium. Background Art

[0002] In the field of natural language processing, a technological approach to improving large language models is the Mixture of Experts (MOE) model. Its core approach is to divide the entire model into multiple independent expert sub-networks and use a gating mechanism to dynamically route different input data, thereby activating only some experts for computation. This allows large-parameter models to only activate a subset of parameters during inference, reducing hardware consumption while maintaining the accuracy of the model's inference answers.

[0003] The existing mainstream technologies of hybrid expert models in expert settings mainly include:

[0004] 1. Fixed Top-K routing;

[0005] 2. Top-K routing with noise and auxiliary loss;

[0006] 3. Hypernetwork-based expert transfer (HyperMoE);

[0007] 4. Dynamically select a different number of experts at each level.

[0008] In traditional intensive models, all parameters are involved in the calculation, and the amount of calculation increases linearly with the model parameters;

[0009] MoE, on the other hand, decouples parameter size and computational load by selectively activating experts. Existing MoE models often use a fixed Top-K strategy and introduce noise and auxiliary losses to balance the expert load. However, these models still suffer from cumbersome hyperparameter adjustment, insufficient dynamic adaptability, and a need for improved robustness, resulting in natural language recognition accuracy failing to meet preset requirements. Summary of the Invention

[0010] Embodiments of the present invention provide a natural language semantic recognition method, apparatus, computer equipment, and storage medium to improve the accuracy of natural language recognition.

[0011] In order to solve the above technical problems, the present invention provides a method for natural language semantic recognition, including:

[0012] Extracting semantic features based on the input natural language and inputting the semantic features into an initial hybrid expert model, which includes a Transformer model, a gating network, a dynamic routing, and an expert network;

[0013] The semantic features are parsed through the Transformer model to obtain the hidden state matrix;

[0014] Input the hidden state matrix into the gating network to generate the routing weight matrix;

[0015] The number K of expert networks activated in the current layer is generated by combining the hidden state matrix with the rank estimation of randomized singular value decomposition;

[0016] The first K expert networks are selected according to the sorting results of the routing weight matrix, a mask matrix is ​​generated, a sparse forward calculation is performed, and the model parameters of the expert networks are optimized based on sparse gradient propagation and load balancing loss to obtain the target hybrid expert model.

[0017] The target mixture expert model is used to identify semantic features and obtain semantic recognition results.

[0018] Optionally, semantic features extracted from the input natural language include:

[0019] Obtain the natural language input by the user and convert it into standard format text;

[0020] Convert standard format text into token sequence and use the token sequence as semantic feature.

[0021] Optionally, the semantic features are parsed through the Transformer model to obtain a hidden state matrix including:

[0022] Through the embedding layer of the Transformer model, the semantic features are mapped into a vector feature matrix;

[0023] Superimpose the vector feature matrix and the position encoding matrix output by the Transformer model to generate the initial hidden state matrix;

[0024] The self-attention layer and Norm layer of the Transformer network are used to parse the initial hidden state matrix to obtain the hidden state matrix.

[0025] Optionally, the number of experts K for generating the current layer activation based on the rank estimation of randomized singular value decomposition combined with the hidden state matrix includes:

[0026] Perform low-rank approximation analysis on the routing weight matrix to determine the effective rank value r, which represents the complexity of the input data;

[0027] The effective rank value r is normalized to the preset interval, and combined with the dynamic upper and lower limit constraints, the number of experts K activated in the current layer is generated through linear interpolation mapping.

[0028] Optionally, the model parameters of the expert network are optimized based on sparse gradient propagation and load balancing loss to obtain the target hybrid expert model including:

[0029] When sparse gradient propagation is back propagation, the parameters of the activated expert sub-network are updated, and the gradients of the inactivated expert sub-network are set to zero;

[0030] Introduce expert activation frequency statistics to calculate the load balancing loss L_balance:

[0031]

[0032] Among them, f i is the activation frequency of expert network i, μ is the average frequency, λ is the weight coefficient, and N is the total number of expert networks in the current layer;

[0033] Based on the load balancing loss L_balance, a load balancing weight is determined.

[0034] In order to solve the above technical problems, the embodiment of the present application further provides a natural language semantic recognition device, comprising:

[0035] A semantic feature extraction module is used to extract semantic features based on the input natural language and input the semantic features into an initial hybrid expert model, wherein the initial hybrid expert model includes a Transformer model, a gating network, a dynamic routing and an expert network;

[0036] A state matrix generation module is used to parse the semantic features through a Transformer model to obtain a hidden state matrix;

[0037] A weight matrix generation module, configured to input the hidden state matrix into a gating network to generate a routing weight matrix;

[0038] A dynamic quantity adjustment module is used to generate the number K of expert networks activated in the current layer based on the rank estimation of randomized singular value decomposition and in combination with the hidden state matrix;

[0039] A model parameter optimization module is used to select the top K expert networks according to the ranking result of the routing weight matrix, generate a mask matrix, perform a sparse forward calculation, and optimize the model parameters of the expert networks based on sparse gradient propagation and load balancing loss to obtain a target hybrid expert model;

[0040] The semantic feature recognition module is used to recognize the semantic features using the target hybrid expert model to obtain a semantic recognition result.

[0041] Optionally, the semantic feature extraction module includes:

[0042] The standardization unit is used to obtain the natural language input by the user and convert it into a standard format text;

[0043] A serialization unit is used to convert the standard format text into a token sequence and use the token sequence as the semantic feature.

[0044] Optionally, the state matrix generation module includes:

[0045] The mapping unit is used to map semantic features into vector feature matrices through the embedding layer of the Transformer model;

[0046] A superposition unit, configured to superimpose the vector feature matrix and the position encoding matrix output by the Transformer model to generate an initial hidden state matrix;

[0047] A parsing unit is used to parse the initial hidden state matrix using the self-attention layer and the Norm layer of the Transformer network to obtain the hidden state matrix.

[0048] Optionally, the quantity dynamic adjustment module includes:

[0049] an effective rank value determining unit, configured to perform a low-rank approximation analysis on the routing weight matrix to determine an effective rank value r, wherein the effective rank value r represents the complexity of the input data;

[0050] The normalization unit is used to normalize the effective rank value r to a preset interval, and generate the number of experts K activated in the current layer through linear interpolation mapping in combination with dynamic upper and lower limit constraints.

[0051] Optionally, the model parameter optimization module includes:

[0052] The parameter adjustment unit is used to update the parameters of the activated expert sub-network when the sparse gradient propagation is back propagation, and the gradient of the sub-network of the inactivated expert is set to zero;

[0053] The loss calculation unit is used to introduce expert activation frequency statistics and calculate the load balancing loss L_balance:

[0054]

[0055] Among them, f i is the activation frequency of expert network i, μ is the average frequency, λ is the weight coefficient, and N is the total number of expert networks in the current layer;

[0056] The weight adjustment unit is used to determine the load balancing weight based on the load balancing loss L_balance.

[0057] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above natural language semantic recognition method when executing the computer program.

[0058] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above natural language semantic recognition method are implemented.

[0059] The natural language semantic recognition method, apparatus, computer device, and storage medium provided by the embodiments of the present invention extract semantic features from input natural language and input the semantic features into an initial hybrid expert model, which includes a Transformer model, a gating network, dynamic routing, and an expert network. The semantic features are parsed using the Transformer model to obtain a hidden state matrix. The hidden state matrix is ​​input into the gating network to generate a routing weight matrix. The number K of expert networks activated in the current layer is generated based on rank estimation of randomized singular value decomposition and the hidden state matrix. The top K expert networks are selected based on the sorting result of the routing weight matrix to generate a mask matrix, a sparse forward calculation is performed, and the model parameters of the expert networks are optimized based on sparse gradient propagation and load balancing loss to obtain a target hybrid expert model. The target hybrid expert model is used to identify semantic features and obtain semantic recognition results. The target hybrid expert model implements a dynamic routing mechanism based on matrix rank analysis to dynamically adjust the number of expert networks, improving computational efficiency while reducing complexity. This achieves adaptive matching between task difficulty and computing resources, ensuring that the network as a whole has end-to-end adaptive constraints, simplifying training and parameter adjustment, and improving model robustness, thereby improving the accuracy and efficiency of semantic recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0061] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0062] Figure 2 is a flowchart of an embodiment of the natural language semantic recognition method of the present application;

[0063] Figure 3is a schematic structural diagram of an embodiment of a natural language semantic recognition device according to the present application;

[0064] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0066] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0067] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0068] See also Figure 1 ,like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0069] Users can use terminal devices 101 , 102 , 103 to interact with server 105 via network 104 to receive or send messages, etc.

[0070] Terminal devices 101, 102, and 103 can be various electronic devices with display screens and support web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), laptop computers, desktop computers, etc.

[0071] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0072] It should be noted that the natural language semantic recognition method provided in the embodiment of the present application is executed by a server, and accordingly, the natural language semantic recognition device is set in the server.

[0073] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only illustrative. According to the implementation requirements, there can be any number of terminal devices, networks and servers. The terminal devices 101, 102, 103 in the embodiment of the present application can specifically correspond to application systems in actual production.

[0074] See also Figure 2 , Figure 2 A natural language semantic recognition method provided by an embodiment of the present invention is shown. Figure 1 The server in the example is used as an example, and the details are as follows:

[0075] S201: Extracting semantic features based on the input natural language and inputting the semantic features into an initial hybrid expert model, which includes a Transformer model, a gating network, a dynamic routing, and an expert network.

[0076] In a specific optional implementation, extracting semantic features based on the input natural language includes:

[0077] Obtain the natural language input by the user and convert it into standard format text;

[0078] Convert standard format text into token sequence and use the token sequence as semantic feature.

[0079] (1) The user enters the text content of their symptoms (e.g., "persistent fever and cough with blood in sputum for the past three days");

[0080] (2) Word segmentation and embedding processing:

[0081] A medical-enhanced word segmenter (integrating clinical terminology and ICD-10 coding) is used to convert text into token sequences.

[0082] S202: Analyze the semantic features through the Transformer model to obtain a hidden state matrix.

[0083] In a specific optional implementation, semantic features are parsed using a Transformer model to obtain a hidden state matrix including:

[0084] Through the embedding layer of the Transformer model, the semantic features are mapped into a vector feature matrix;

[0085] Superimpose the vector feature matrix and the position encoding matrix output by the Transformer model to generate the initial hidden state matrix;

[0086] The self-attention layer and Norm layer of the Transformer network are used to parse the initial hidden state matrix to obtain the hidden state matrix.

[0087] Specifically, the medical Mixture of Experts (MOE) model is applied to the scenario of medical knowledge question and answer. Due to the limitations of the hardware scenario, the model size is capped. For example, a large MOE language model of 8B can be deployed on some hardware. Using the mixed expert model of this embodiment can achieve better accuracy and more accurate answers in the medical knowledge question and answer scenario. For example, if a user asks which department to consult for a symptom, the MOE model will give the department to consult. The data input process is as follows:

[0088] The semantic feature token is mapped to a 768-dimensional vector E through the embedding layer. The dimension is {(L×768), L is the token length

[0089] The superposition learnable position encoding matrix P has a dimension of {(L×768)}, generating the initial hidden state H0=E+P

[0090] The model consists of 32 layers, i.e., 32 mixed expert layers, each containing 128 expert networks (Expert = FeedForward Network);

[0091] The feature H0 passes through the self-attention layer and the Norm layer of the Transformer network to obtain the hidden state matrix H.

[0092] In this embodiment, the specific format of the hidden state matrix H of the input data is batch size B = 32 and feature dimension D = 512. In actual applications, the batch size can be adjusted according to the size of the data set and hardware resources, such as 16, 64, etc.

[0093] Normalize the input data to ensure that the data is within the same scale range to avoid numerical instability affecting model training.

[0094] S203: Input the hidden state matrix into the gating network to generate a routing weight matrix.

[0095] In this embodiment, the gated network adopts two fully connected layers. The number of neurons in the first fully connected layer is 1024, and the activation function is ReLU. The number of neurons in the second fully connected layer is the total number of experts N=128, and the activation function is Softmax.

[0096] Initialize the learnable parameter matrix W_g∈{D×N} and the bias term b_g, and use the Xavier initialization method to ensure that the initial values ​​of the parameters are within a reasonable range and accelerate model convergence.

[0097] Specifically, the routing weight matrix generation process is as follows:

[0098] The hidden state matrix H∈{B×D} (B is the batch size, D is the feature dimension) is input, and the routing weight matrix G∈{B×N} (B is the batch size, N is the total number of experts) is generated through the gating network (fully connected layer + activation function).

[0099] The calculation formula of routing weight matrix G is:

[0100] G=Softmax(H·W_g+b_g)

[0101] Among them, W_g∈{D×N} is the learnable parameter matrix and b_g is the bias term.

[0102] S204: Based on the rank estimation of the randomized singular value decomposition and combined with the hidden state matrix, the number K of expert networks activated in the current layer is generated.

[0103] In a specific optional implementation, based on the rank estimation of randomized singular value decomposition, the number of experts K activated in the current layer is generated in combination with the hidden state matrix, including:

[0104] Perform low-rank approximation analysis on the routing weight matrix to determine the effective rank value r, which represents the complexity of the input data;

[0105] The effective rank value r is normalized to the preset interval, and combined with the dynamic upper and lower limit constraints, the number of experts K activated in the current layer is generated through linear interpolation mapping.

[0106] Specifically, randomized singular value decomposition (rSVD) is performed on the matrix to obtain an approximate value of the matrix rank. Taking the matrix A of dimension {m×n} as an example, the goal is to approximate its first k singular values ​​and singular vectors k<<min(m,n). The specific calculation process is:

[0107] A) Generate a random projection matrix.

[0108] Construct a random Gaussian matrix Ω of dimension {N×(k+p)}, where p is the oversampling amount (usually p=5-10) to improve stability.

[0109] The matrix elements follow the standard normal distribution N(0,1).

[0110] Example: If k = 10, then the dimension of Ω is n × 15.

[0111] B) Construct a low-dimensional approximate subspace.

[0112] Compute the matrix product Y = AΩ, and obtain Y of dimension {m×(k+p)}, which captures the main column space of matrix A through random projection.

[0113] C) Orthogonalize the subspace basis.

[0114] Perform QR decomposition on Y to obtain the orthogonal basis matrix Q with dimensions {m×(k+p)}, Y=QR, where QQ T =I.

[0115] D) Small matrix SVD calculation.

[0116] Construct a small matrix B = Q T A, whose dimension is {(k+p)×n}.

[0117] Perform accurate SVD on the small matrix B, that is, the traditional singular value decomposition SVD algorithm:

[0118]

[0119] matrix Dimension {(k+p)×(k+p)}ΣDimension {(k+p)×n}V T Dimension {n×n}.

[0120] E) Truncate the B matrix to get the SVD result.

[0121] Retain the first k singular values ​​{σ_1,...,σ_k} so that the smallest k is obtained when the corresponding principal component energy accounts for more than 95%, which is a rough estimate of the rank.

[0122] The corresponding principal component energy accounts for more than 95% means that the sum of the first k singular values ​​accounts for 95% of the sum of all singular values. Its implicit mathematical feature is that after singular value decomposition, the middle diagonal matrix Σ characteristic has the main components in the front of the diagonal matrix, and the values ​​are smaller the further back they are.

[0123] In this embodiment, randomized singular value decomposition (rSVD) is used instead of the traditional SVD algorithm, significantly reducing the computational complexity of the matrix rank while maintaining approximate accuracy and avoiding the additional computational bottleneck introduced by dynamic routing. This innovation is achieved by applying the rSVD method to this feature to reduce the amount of computation. This avoids the large additional computational overhead associated with the SVD process when calculating the matrix rank.

[0124] It should be noted that this embodiment proposes dynamically adjusting the number of activated experts by analyzing the rank of the pre-Softmax routing matrix (hidden state matrix). The rank of the matrix is ​​related to the complexity of the input data: when the rank is low, the number of experts is reduced (for simple tasks), while when the rank is high, the number of experts is increased (for complex tasks), achieving an adaptive match between task difficulty and computing resources.

[0125] Preferably, the present invention adopts adaptive expert (network) quantity decision to normalize the rank value r to the interval [0,1]:

[0126] r_norm=(r-r_min) / (r_max-r_min)

[0127] Among them, r_min and r_max are the minimum / maximum rank statistics in the training data.

[0128] The number of experts K is determined by the linear interpolation function:

[0129]

[0130] Among them, K_min and K_max are the upper and lower limits of the number of experts preset in each layer (such as K_min = 1, K_max = 6), which are determined by the network structure. The value K_max is determined when the network structure is determined, and therefore will not change with changes in downstream tasks.

[0131] Here, r_norm is defined as the normalized matrix rank, which ranges from [0 to 1]. It is calculated as above, normalizing the rank. It represents the complexity of the input data (the higher the rank, the more complex the data).

[0132] r_min and r_max are the minimum and maximum rank values ​​counted in the training data, which are used to normalize the rank values ​​to a uniform scale.

[0133] p is a custom threshold. The number of experts in each layer is accumulated until the sum of the confidence scores is greater than p. This method does not set an upper limit on the number of experts. For example, if there are 128 experts in a layer, the number of choices in each layer is actually [1, 128], ranging from 1 to 128.

[0134] Linear interpolation is a relatively simple method for this module. It mainly stipulates the upper and lower limits of the number of experts [K_min, K_max]. It will not select too many experts in extreme cases, making the upper limit controllable and unchanged with the task. There is no need to readjust the parameters when switching tasks, ensuring the sparsity advantage of the MOE model.

[0135] S205: Select the top K expert networks according to the sorting result of the routing weight matrix, generate a mask matrix, perform sparse forward calculation, and optimize the model parameters of the expert network based on sparse gradient propagation and load balancing loss to obtain the target hybrid expert model.

[0136] Specifically, according to the weight ranking of the routing weight matrix G, the first K expert networks are selected to generate a mask matrix M∈{B×N}, where M is a binary value 0 or 1.

[0137] Perform the sparsification calculation and activate only the experts corresponding to the mask:

[0138] Output=Σ_{i=1}^NM[:,i]·Expert_i(H).

[0139] Preferably, selecting the top K expert networks according to the ranking result of the routing weight matrix, generating a mask matrix, and performing the sparse forward calculation includes the following steps:

[0140] (1) Routing weight ranking and expert selection

[0141] Input: Routing matrix G of dimension {B×N} (B = batch size, N = total number of experts);

[0142] Rule: Sort by sample, sort the routing weight G[i,:] of each sample from high to low;

[0143] Dynamically select Top-K: Based on the number of experts K in the current layer decision obtained in step 3, select the top K expert networks with the highest weights.

[0144] (2) Mask matrix generation, the generation logic is:

[0145] For each sample i, generate a binary mask M[i,:]∈{0,1} where

[0146]

[0147] Constraint: Each row mask contains only K 1s and the rest are 0s.

[0148] (3) Sparse calculation

[0149] Input distribution: Sample input is only passed to activated experts (experts with mask 1).

[0150] Output aggregation: Weighted sum: The output of the activated experts is weighted summed by the routing weight G[i,j].

[0151] In this embodiment, dynamic upper and lower limits on the number of experts are introduced (e.g., a minimum of one expert and a maximum of K_max experts are activated per layer). The number of experts is generated based on the normalized rank, replacing fixed thresholds or hyperparameters. This improves model robustness and reduces the need for manual parameter tuning. This eliminates the hassle and limitations of manually specifying the hyperparameter p; the entire method eliminates the need for manual hyperparameter adjustment for data with different distributions, resulting in greater robustness. The upper limit on the number of expert networks is set, and it does not change with the input data.

[0152] Furthermore, gradient propagation optimization includes:

[0153] Sparse gradient: Only the parameters of the activated experts are updated, and the gradients of the inactivated experts are set to zero.

[0154] Implementation: Masking inactive experts through automatic differentiation of deep learning framework.

[0155] In a specific optional implementation, optimizing the model parameters of the expert network based on sparse gradient propagation and load balancing loss to obtain the target hybrid expert model includes:

[0156] When sparse gradient propagation is back propagation, the parameters of the activated expert sub-network are updated, and the gradients of the inactivated expert sub-network are set to zero;

[0157] Introduce expert activation frequency statistics to calculate the load balancing loss L_balance:

[0158]

[0159] Among them, f i is the activation frequency of expert network i, μ is the average frequency, λ is the weight coefficient, and N is the total number of expert networks in the current layer;

[0160] Based on the load balancing loss L_balance, a load balancing weight is determined.

[0161] S206: Using the target hybrid expert model to identify semantic features, and obtain semantic recognition results.

[0162] In this embodiment, semantic features are extracted based on input natural language and input into an initial hybrid expert model, which includes a Transformer model, a gating network, dynamic routing, and an expert network. The semantic features are parsed using the Transformer model to obtain a hidden state matrix. The hidden state matrix is ​​input into the gating network to generate a routing weight matrix. The number K of expert networks activated in the current layer is generated based on rank estimation of randomized singular value decomposition and the hidden state matrix. The top K expert networks are selected based on the sorting result of the routing weight matrix to generate a mask matrix, a sparse forward calculation is performed, and the model parameters of the expert networks are optimized based on sparse gradient propagation and load balancing loss to obtain a target hybrid expert model. The target hybrid expert model is used to identify semantic features and obtain semantic recognition results. The dynamic routing mechanism of matrix rank analysis is implemented to dynamically adjust the number of expert networks, thereby improving computational efficiency while reducing complexity, achieving adaptive matching between task difficulty and computing resources, and ensuring that the network as a whole has end-to-end adaptive constraints, simplifying training and parameter adjustment, and improving model robustness, thereby improving the accuracy and efficiency of semantic recognition.

[0163] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0164] Figure 3 The principle block diagram of the natural language semantic recognition device corresponding to the natural language semantic recognition method in the above embodiment is shown. Figure 3 As shown, the natural language semantic recognition device includes a semantic feature extraction module 31, a state matrix generation module 32, a weight matrix generation module 33, a quantity dynamic adjustment module 34, a model parameter optimization module 35 and a semantic feature recognition module 36. The functional modules are described in detail as follows:

[0165] A semantic feature extraction module 31 is used to extract semantic features based on the input natural language and input the semantic features into an initial hybrid expert model, wherein the initial hybrid expert model includes a Transformer model, a gating network, a dynamic routing and an expert network;

[0166] A state matrix generation module 32 is used to parse the semantic features through a Transformer model to obtain a hidden state matrix;

[0167] A weight matrix generation module 33 is used to input the hidden state matrix into the gating network to generate a routing weight matrix;

[0168] A number dynamic adjustment module 34 is used to generate the number K of expert networks activated in the current layer based on the rank estimation of randomized singular value decomposition and in combination with the hidden state matrix;

[0169] A model parameter optimization module 35 is configured to select the top K expert networks based on the ranking result of the routing weight matrix, generate a mask matrix, perform a sparse forward calculation, and optimize the model parameters of the expert networks based on sparse gradient propagation and load balancing loss to obtain a target hybrid expert model;

[0170] The semantic feature recognition module 36 is used to recognize the semantic features using the target hybrid expert model to obtain a semantic recognition result.

[0171] Optionally, the semantic feature extraction module 31 includes:

[0172] The standardization unit is used to obtain the natural language input by the user and convert it into a standard format text;

[0173] A serialization unit is used to convert the standard format text into a token sequence and use the token sequence as the semantic feature.

[0174] Optionally, the state matrix generating module 32 includes:

[0175] The mapping unit is used to map semantic features into vector feature matrices through the embedding layer of the Transformer model;

[0176] A superposition unit, configured to superimpose the vector feature matrix and the position encoding matrix output by the Transformer model to generate an initial hidden state matrix;

[0177] A parsing unit is used to parse the initial hidden state matrix using the self-attention layer and the Norm layer of the Transformer network to obtain the hidden state matrix.

[0178] Optionally, the quantity dynamic adjustment module 34 includes:

[0179] an effective rank value determining unit, configured to perform a low-rank approximation analysis on the routing weight matrix to determine an effective rank value r, wherein the effective rank value r represents the complexity of the input data;

[0180] The normalization unit is used to normalize the effective rank value r to a preset interval, and generate the number of experts K activated in the current layer through linear interpolation mapping in combination with dynamic upper and lower limit constraints.

[0181] Optionally, the model parameter optimization module 35 includes:

[0182] The parameter adjustment unit is used to update the parameters of the activated expert sub-network when the sparse gradient propagation is back propagation, and the gradient of the sub-network of the inactivated expert is set to zero;

[0183] The loss calculation unit is used to introduce expert activation frequency statistics and calculate the load balancing loss L_balance:

[0184]

[0185] Among them, f i is the activation frequency of expert network i, μ is the average frequency, λ is the weight coefficient, and N is the total number of expert networks in the current layer;

[0186] The weight adjustment unit is used to determine the load balancing weight based on the load balancing loss L_balance.

[0187] For the specific definition of the natural language semantic recognition device, please refer to the definition of the natural language semantic recognition method above, which will not be repeated here. The various modules in the above-mentioned natural language semantic recognition device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0188] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0189] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components connected to the memory 41, the processor 42, and the network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0190] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0191] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or D interface display memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as the program code of the natural language semantic recognition method. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0192] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute program code stored in the memory 41 or process data, such as executing program code for a natural language semantic recognition method.

[0193] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0194] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to perform the steps of the natural language semantic recognition method as described above.

[0195] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0196] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A natural language semantic recognition method, characterized in that: include: Extracting semantic features based on input natural language, and inputting the semantic features into an initial hybrid expert model, wherein the initial hybrid expert model includes a Transformer model, a gating network, a dynamic routing, and an expert network; Parsing the semantic features through the Transformer model to obtain a hidden state matrix; Inputting the hidden state matrix into a gating network to generate a routing weight matrix; Based on the rank estimation of randomized singular value decomposition, the number K of expert networks activated in the current layer is generated in combination with the hidden state matrix; Selecting the top K expert networks according to the ranking result of the routing weight matrix, generating a mask matrix, performing a sparse forward calculation, and optimizing the model parameters of the expert networks based on sparse gradient propagation and load balancing loss to obtain a target hybrid expert model; The target hybrid expert model is used to identify the semantic features to obtain a semantic recognition result.

2. The natural language semantic recognition method according to claim 1, wherein: The step of extracting semantic features based on the input natural language includes: Obtain the natural language input by the user and convert it into standard format text; The standard format text is converted into a token sequence, and the token sequence is used as the semantic feature.

3. The natural language semantic recognition method according to claim 1, wherein: The semantic features are parsed by the Transformer model to obtain a hidden state matrix including: Through the embedding layer of the Transformer model, the semantic features are mapped into a vector feature matrix; Superimposing the vector feature matrix and the position encoding matrix output by the Transformer model to generate an initial state matrix; The self-attention layer and the Norm layer of the Transformer network are used to parse the initial state matrix to obtain the hidden state matrix.

4. The natural language semantic recognition method according to claim 1, wherein: The rank estimation based on randomized singular value decomposition and the number of experts K activated in the current layer generated in combination with the hidden state matrix include: Performing a low-rank approximation analysis on the routing weight matrix to determine an effective rank value r, where the effective rank value r represents the complexity of the input data; The effective rank value r is normalized to a preset interval, and combined with dynamic upper and lower limit constraints, the number of experts K activated in the current layer is generated through linear interpolation mapping.

5. The natural language semantic recognition method according to claim 1, wherein: The method of optimizing the model parameters of the expert network based on sparse gradient propagation and load balancing loss to obtain the target hybrid expert model includes: When sparse gradient propagation is back propagation, the parameters of the activated expert sub-network are updated, and the gradients of the inactivated expert sub-network are set to zero; Introduce expert activation frequency statistics to calculate the load balancing loss L_balance: Among them, f i is the activation frequency of expert network i, μ is the average frequency, λ is the weight coefficient, and N is the total number of expert networks in the current layer; Based on the load balancing loss L_balance, a load balancing weight is determined.

6. A natural language semantic recognition device, characterized in that: include: A semantic feature extraction module is used to extract semantic features based on the input natural language and input the semantic features into an initial hybrid expert model, wherein the initial hybrid expert model includes a Transformer model, a gating network, a dynamic routing and an expert network; A state matrix generation module is used to parse the semantic features through a Transformer model to obtain a hidden state matrix; A weight matrix generation module, configured to input the hidden state matrix into a gating network to generate a routing weight matrix; A dynamic quantity adjustment module, configured to generate the number K of expert networks activated in the current layer based on the rank estimation of randomized singular value decomposition and in combination with the hidden state matrix; A model parameter optimization module is used to select the top K expert networks according to the ranking result of the routing weight matrix, generate a mask matrix, perform a sparse forward calculation, and optimize the model parameters of the expert networks based on sparse gradient propagation and load balancing loss to obtain a target hybrid expert model; The semantic feature recognition module is used to recognize the semantic features using the target hybrid expert model to obtain a semantic recognition result.

7. The natural language semantic recognition device according to claim 6, characterized in that: The semantic feature extraction module includes: The standardization unit is used to obtain the natural language input by the user and convert it into a standard format text; A serialization unit is used to convert the standard format text into a token sequence and use the token sequence as the semantic feature.

8. The natural language semantic recognition device according to claim 6, wherein: The state matrix generation module includes: The mapping unit is used to map semantic features into vector feature matrices through the embedding layer of the Transformer model; A superposition unit, configured to superimpose the vector feature matrix and the position encoding matrix output by the Transformer model to generate an initial hidden state matrix; A parsing unit is used to parse the initial hidden state matrix using the self-attention layer and the Norm layer of the Transformer network to obtain the hidden state matrix.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the natural language semantic recognition method according to any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the natural language semantic recognition method according to any one of claims 1 to 5 is implemented.

Images