Speech coding method and electronic device

Abstract

The application discloses a speech coding method and an electronic device. The method comprises the following steps: acquiring an acoustic token sequence corresponding to input speech and a basic routing probability distribution of a current acoustic token; calculating acoustic similarity between the current acoustic token and a historical acoustic token; weighting and aggregating a historical expert routing probability distribution by taking the acoustic similarity as a weight to generate a historical expert prior distribution; correcting the basic routing probability distribution by using the historical expert prior distribution to obtain a corrected expert routing probability distribution; and determining an activated expert according to the corrected distribution and performing feature coding and fusion. The application can effectively solve the problems of unstable expert allocation and discontinuous coding representation time caused by independent token routing, and improve the learning ability of experts for complete acoustic patterns and the representation stability of the encoder.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and more specifically, to a speech coding method and electronic device. Background Technology

[0002] Automatic speech recognition technology, as a core component of human-computer interaction, has been widely applied in various scenarios such as smart terminals, in-vehicle systems, meeting transcription, and customer service. In current deep learning architectures, speech encoders are typically responsible for converting continuous speech signals into high-level acoustic or semantic representations for subsequent decoders to generate text results. With increasing demands for recognition accuracy and the growing complexity of application scenarios, the size of models continues to expand. To increase model capacity while controlling computational resource consumption, the Mixture of Experts (MoE) architecture has been introduced into speech encoders. This architecture achieves a significant increase in model parameters while keeping the actual computational load relatively controllable by dynamically selecting a small number of expert networks to activate for each input unit. Existing hybrid expert routing mechanisms typically calculate routing probabilities independently based on the hidden state of the current input unit and select the top-ranked expert for processing based on these probabilities.

[0003] Because speech signals are highly correlated over time, adjacent acoustic segments often belong to the same phoneme or a stable acoustic environment. However, traditional frame-by-frame independent routing is sensitive to local perturbations in routing scores. When the scores of two experts are close, two adjacent and highly similar acoustic tokens are assigned to different expert paths. This discrete fluctuation in routing results not only disrupts the continuity of acoustic patterns over time, making it difficult for experts to learn complete and stable acoustic feature representations, but may also cause jumps in the encoded features output by the encoder that do not match the actual acoustic changes, interfering with the decoding judgment of subsequent decoding modules. Summary of the Invention

[0004] In view of the above problems, this application is proposed to provide a speech coding method and electronic device to solve the problems of unstable expert allocation of adjacent acoustic tokens and incomplete acoustic pattern learning in existing hybrid expert speech coding. The specific solution is as follows:

[0005] Firstly, a speech coding method is provided, applied to a speech encoder comprising a hybrid expert coding layer, wherein the hybrid expert coding layer includes multiple parallel experts, and the speech coding method includes:

[0006] Obtain the acoustic token sequence corresponding to the input speech, and determine the basic routing probability distribution of the current acoustic token in the acoustic token sequence;

[0007] Obtain several historical acoustic tokens preceding the current acoustic token, and calculate the acoustic similarity between the current acoustic token and each of the historical acoustic tokens respectively;

[0008] Using the acoustic similarity corresponding to each historical acoustic token as the weight, the expert routing probability distribution corresponding to each historical acoustic token is weighted and aggregated to generate the historical expert prior distribution;

[0009] The basic route probability distribution is corrected by using the historical expert prior distribution to obtain the corrected expert route probability distribution of the current acoustic token;

[0010] Based on the corrected expert routing probability distribution, at least one active expert is determined from each expert in the hybrid expert coding layer. The current acoustic token is then input into each of the active experts for feature encoding and the encoding results are fused to obtain the encoding result of the current acoustic token.

[0011] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of determining the basic routing probability distribution of the current acoustic token includes:

[0012] Obtain the hidden state of the current acoustic token;

[0013] Based on the hidden state of the current acoustic token, calculate the matching score between the current acoustic token and each expert in the hybrid expert coding layer;

[0014] Normalization is performed on each of the matching scores to obtain the basic route probability distribution.

[0015] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of calculating the acoustic similarity between the current acoustic token and each of the historical acoustic tokens includes:

[0016] The acoustic feature representations of the current acoustic token and each of the historical acoustic tokens are projected onto the continuity discrimination space, and the projection results are normalized.

[0017] The cosine similarity between the current acoustic token and each of the historical acoustic tokens is calculated based on the normalized features, and is used as the acoustic similarity.

[0018] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of using the acoustic similarity corresponding to each of the historical acoustic tokens as weights includes:

[0019] The acoustic similarity corresponding to each of the historical acoustic tokens is converted into continuous weights through a gating function. The continuous weights are positively correlated with the acoustic similarity, and the value range of the continuous weights is limited to between 0 and 1.

[0020] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of correcting the basic route probability distribution using the historical expert prior distribution includes:

[0021] The overall continuity strength corresponding to the current acoustic token is calculated based on the acoustic similarity between the current acoustic token and each of the historical acoustic tokens.

[0022] The correction magnitude of the historical expert prior distribution is controlled by the overall continuity strength, thereby correcting the basic route probability distribution.

[0023] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of correcting the basic route probability distribution by controlling the correction magnitude of the historical expert prior distribution with the overall continuity strength includes:

[0024] The aforementioned historical expert prior distribution is transformed into a correction term in logarithmic space;

[0025] The correction term is weighted by the overall continuity strength to obtain a weighted correction term. The weighted correction term is added to the routing score logits corresponding to the basic routing probability distribution to obtain the corrected routing score logits. After normalization, the corrected expert routing probability distribution is obtained.

[0026] In one possible design, in another implementation of the first aspect of the present application, the number of hybrid expert coding layers is multiple, and the multiple hybrid expert coding layers are arranged sequentially in the depth direction of the speech encoder. Each expert in each hybrid expert coding layer is divided into an acoustic expert pool and a semantic expert pool. The experts in the acoustic expert pool are used to implement acoustic pattern coding, and the experts in the semantic expert pool are used to implement semantic pattern coding.

[0027] In one possible design, in another implementation of the first aspect of the embodiments of this application, after obtaining the corrected expert routing probability distribution of the current acoustic token, the method further includes:

[0028] Determine the layer depth position of the current hybrid expert coding layer in the encoder depth direction;

[0029] The layer depth prior is determined based on the layer depth position, and the layer depth prior is used to characterize the degree of bias of the current hybrid expert coding layer towards the acoustic expert pool and the semantic expert pool;

[0030] The expert routing probability distribution of the current acoustic token is further modified based on the layer depth prior to obtain the final modified expert routing probability distribution.

[0031] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of determining the layer depth prior based on the layer depth position includes:

[0032] A layer depth coefficient is calculated based on the layer depth position, and the layer depth coefficient is positively correlated with the depth of the current coding layer;

[0033] The first deep prior corresponding to the experts in the acoustic expert pool and the second deep prior corresponding to the experts in the semantic expert pool are determined based on the depth coefficient.

[0034] Wherein, when the layer depth coefficient is less than the first threshold, the correction magnitude of the first layer depth prior to the expert routing probability is greater than that of the second layer depth prior; when the layer depth coefficient is greater than the second threshold, the correction magnitude of the second layer depth prior to the expert routing probability is greater than that of the first layer depth prior, and the second threshold is greater than the first threshold.

[0035] In one possible design, in another implementation of the first aspect of the present application, the correction strength when using the historical expert prior distribution to correct the basic route probability distribution varies with the depth of the hybrid expert coding layer, wherein the correction strength of the hybrid expert coding layer located at the first depth is greater than the correction strength of the hybrid expert coding layer located at the second depth, and the first depth is shallower than the second depth.

[0036] In one possible design, in another implementation of the first aspect of the embodiments of this application, the number of experts in the semantic expert pool is greater than or equal to the number of experts in the acoustic expert pool.

[0037] In one possible design, in another implementation of the first aspect of the embodiments of this application, the loss function of the speech encoder training phase includes the load balancing loss of each hybrid expert coding layer, wherein, for hybrid expert coding layers whose layer depth coefficient is less than a first threshold, the load balancing loss within the acoustic expert pool is calculated; for hybrid expert coding layers whose layer depth coefficient is greater than a second threshold, the load balancing loss within the semantic expert pool is calculated; and for hybrid expert coding layers whose layer depth coefficient is between the first threshold and the second threshold, the load balancing losses of the acoustic expert pool and the semantic expert pool are weighted and summed according to the first layer depth prior and the second layer depth prior.

[0038] In one possible design, in another implementation of the first aspect of the embodiments of this application, the speech encoder and speech decoder constitute a speech recognition model, and the total loss function during the training phase of the speech recognition model includes speech recognition loss, acoustic continuity loss and expert load balancing loss.

[0039] The acoustic continuity loss is used to constrain the degree of difference in the expert routing probability distributions of adjacent acoustic tokens, using the acoustic similarity between them as the weight.

[0040] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of inputting the current acoustic token into each activation expert for feature encoding includes:

[0041] Reorder all acoustic tokens within the batch containing the current acoustic token into contiguous memory groups according to the activation expert's number;

[0042] Based on the acoustic tokens corresponding to each expert, feature encoding calculations for each expert are performed in parallel through group matrix multiplication.

[0043] In a second aspect, an electronic device is provided, comprising: a memory and a processor;

[0044] The memory is used to store programs;

[0045] The processor is configured to execute the program to implement the various steps of the speech coding method described in any of the first aspects above.

[0046] By employing the aforementioned technical solution, this application dynamically corrects the current route by introducing a historical expert prior distribution based on acoustic similarity. When the speech signal is within a stable and continuous range, the historical expert prior distribution enhances the probability of adjacent tokens selecting the same or similar experts, thereby smoothing the expert path and reducing frequent switching caused by local disturbances. When phoneme boundaries or acoustic abrupt changes are detected, the similarity weight automatically decreases, restoring the route to a basic route probability distribution primarily based on the current token's own features, preserving the necessary expert switching capability. This mechanism effectively balances route stability and boundary sensitivity, thus effectively solving the technical problems of unstable expert allocation between adjacent acoustic tokens, expert path jumps, and incomplete acoustic pattern learning in existing hybrid expert speech coding. Therefore, it avoids large fluctuations in expert gradient sources and jumps in encoder representation that do not match real acoustic changes, improving the speech encoder's ability to model complex acoustic patterns and the robustness and consistency of the overall recognition system. Attached Figure Description

[0047] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0048] Figure 1 A schematic diagram of an implementation system architecture for the speech coding method provided in this application embodiment;

[0049] Figure 2 This is a schematic flowchart of a speech coding method provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of another speech coding method provided in an embodiment of this application;

[0051] Figure 4 A schematic diagram of the layered structure of a speech encoder, comprising a shallow acoustic expert, a transitional layer hybrid expert, and a deep semantic expert, provided for embodiments of this application;

[0052] Figure 5 A schematic diagram of the overall processing flow of a speech recognition method based on acoustic continuity sensing routing and hierarchical expert encoder provided for embodiments of this application;

[0053] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0056] Existing speech recognition encoders typically employ dense encoders. In a dense encoder, all tokens pass through the same set of feedforward network parameters. When it is necessary to increase the model capacity, the number of encoding layers, hidden dimensions, or intermediate dimensions of the feedforward network are usually increased, which leads to an increase in forward computation, backward computation, and memory usage.

[0057] To address this, existing technologies have proposed hybrid expert encoders. These encoders replace some dense feedforward networks with multiple expert feedforward networks and configure the router to calculate the routing probabilities from tokens to each expert. Existing routers typically perform linear projection and softmax independently on each token, then select the top-k experts and fuse their outputs according to routing weights. While this approach partially decouples the total number of model parameters from the computational cost of single-token activation, its routing function itself does not distinguish between text tokens and acoustic tokens with strong temporal continuity.

[0058] Existing token-independent routing methods are highly sensitive to local perturbations in router routing scores. When the scores of two experts are close, adjacent acoustic tokens may repeatedly switch between different experts. Due to the discrete nature of top-k selection, this switching causes highly similar consecutive speech frames to traverse different expert paths, resulting in fragmented acoustic segments received by the experts, unstable expert gradient sources, and path jumps in adjacent coded representations that do not correspond to real acoustic changes. Simply forcing adjacent tokens to use the same expert would suppress the expert transformations required at real acoustic boundaries (such as phoneme boundaries, silence transitions, or abrupt changes in acoustic scenes).

[0059] This application incorporates an acoustic continuity-aware router within the speech encoder. This router retains the basic routing probability distribution of the current acoustic token while extracting historical acoustic tokens acoustically similar to the current token from a historical window. The expert routing probability distributions of these historical acoustic tokens are then aggregated into a temporal prior (historical expert prior distribution). The temporal prior is injected into the basic routing probability distribution, and its influence is dynamically determined by acoustic similarity. When speech is within a stable, continuous range, similarity gating strengthens the historical expert prior distribution, causing adjacent acoustic tokens to tend towards consistent or similar expert paths. When phoneme boundaries, silence changes, or other significant acoustic changes are detected, the similarity gating value decreases, and the current acoustic token re-selects an expert primarily based on its own representation.

[0060] The speech coding method provided in this application can distinguish between the "continuous intervals" and "real boundaries" of the input speech, and realize adaptive routing constraints.

[0061] The speech coding method of this application can be used in scenarios such as self-supervised speech pre-training, supervised speech recognition training, and streaming or offline speech recognition inference.

[0062] The speech coding method of this application can be applied to in-vehicle voice assistants, smart cockpit voice interaction, input method voice input, meeting transcription, intelligent customer service, speech recognition cloud services, large-scale ASR base model training platforms, and edge-cloud collaborative speech recognition systems. This application is highly applicable to tasks involving long speech, noisy speech, fast-paced speech, and streaming recognition, where the continuity of adjacent acoustic frames is clear but the scene boundaries require timely switching of experts.

[0063] This application provides a speech coding method that can be applied to, for example... Figure 1 The system architecture shown may include a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 1 (This example uses a server as an illustration).

[0064] Either terminal 100 or server 200 can be used independently to execute the speech coding method provided in the embodiments of this application. Alternatively, terminal 100 and server 200 can also be used collaboratively to execute the speech coding method provided in the embodiments of this application.

[0065] The following description Figure 1 The product form of the mid-terminal 100;

[0066] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, teaching large screen, wearable device, vehicle-mounted device, conference terminal, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0067] This application provides a speech coding method, which is illustrated by applying the method to a computer device. Specifically, the computer device may be... Figure 1 The system consists of terminal 100 or a combination of terminal 100 and server 200. (Refer to...) Figure 2 The speech coding method specifically includes the following steps:

[0068] Step S100: Obtain the acoustic token sequence corresponding to the input speech, and determine the basic routing probability distribution of the current acoustic token in the acoustic token sequence.

[0069] The input speech undergoes feature extraction and downsampling processes sequentially to obtain an acoustic token sequence. This acoustic token sequence is a discretized representation of the speech signal in the time dimension, where each token corresponds to an acoustic feature vector at a specific time step. For the current acoustic token in the acoustic token sequence, the speech encoder extracts its hidden state. This hidden state is a high-dimensional feature representation of the current acoustic token after processing by the encoder's preceding layers (such as self-attention layers or convolutional layers), containing the acoustic information at that moment.

[0070] Based on the hidden state of the current acoustic token, a matching score is calculated between it and each expert in the Mixture of Experts (MoE) coding layer. The MoE layer contains multiple parallel expert networks, each typically an independent feedforward neural network, designed to handle specific types of feature patterns. The matching score reflects the degree of fit between the current token and the expertise of each expert. For example, the original matching score for each expert can be obtained by linearly transforming the hidden state with the expert's weight matrix, or by nonlinearly mapping the hidden state using a multilayer perceptron. Normalization (e.g., using the Softmax function) is then applied to each matching score, converting it into probability values ​​and ensuring the sum of all probabilities is 1, thus generating the basic routing probability distribution. This basic routing probability distribution represents the initial probability of the current acoustic token being assigned to each expert without considering historical information.

[0071] In this embodiment of the application, the acoustic token sequence obtained after feature extraction and downsampling of the input speech is defined as follows: Where T is the time length after downsampling. D represents the hidden dimension of the acoustic token, and the specific value of D depends on the hidden layer dimension configuration of the speech encoder.

[0072] The speech encoder consists of L coding layers, and the output of the l-th layer is denoted as . The hybrid expert coding layer comprises several dimensionally consistent expert feedforward networks. The router selects K experts for each acoustic token, and the expert outputs are weighted according to the normalized route weights.

[0073] In one possible implementation, the process of determining the underlying routing probability distribution of the current acoustic token may include:

[0074] S1. Get the hidden state of the current acoustic token.

[0075] S2. Based on the hidden state of the current acoustic token, calculate the matching score between the current acoustic token and each expert in the hybrid expert coding layer.

[0076] The calculation process for the matching score can be formalized as follows:

[0077]

[0078] in, Indicates the first The matching score between the t-th token and the e-th expert in the layer. This indicates the hidden state of the currently obtained acoustic token. and These represent the corresponding numbers in the router. The learnable weight vectors and bias terms of each expert are used. The consistency between the hidden state vector direction and the expert weight vector direction is evaluated using dot product operations. The higher the score, the greater the potential for the current token to be processed by that expert.

[0079] S3. Perform normalization on each matching score to obtain the basic route probability distribution.

[0080] The matching scores in real form are converted into probability assignments among experts, ensuring that the sum of the probabilities of all experts is 1. A normalized exponential function is used to transform the matching scores, and its formal expression is as follows:

[0081]

[0082] In the formula, Indicates the first The basic routing probability of an expert This represents the total number of experts in this layer. After this processing, each matching score is mapped to the (0,1) interval, forming a basic routing probability distribution that characterizes the current acoustic token's preference for selecting each expert in the hybrid expert coding layer.

[0083] Step S110: Obtain several historical acoustic tokens preceding the current acoustic token, and calculate the acoustic similarity between the current acoustic token and each historical acoustic token.

[0084] Specifically, given the strong temporal continuity of speech signals, adjacent or nearby acoustic tokens often belong to the same phoneme or are in a stable acoustic environment. This step aims to quantify the correlation between the current acoustic token and the historical context in terms of acoustic features.

[0085] In this step, several historical acoustic tokens preceding the current acoustic token can be obtained based on a preset historical window length W. This historical window length W can be set according to the sampling rate of the speech signal and the temporal dependency range of the acoustic features, or it can be dynamically adjusted based on the statistical analysis results of the temporal dependency on the training set.

[0086] After obtaining historical acoustic tokens, the acoustic similarity between the current acoustic token and each historical acoustic token is calculated to measure the degree of closeness between them in the acoustic feature space.

[0087] This embodiment provides an optional method for calculating the acoustic similarity between the current acoustic token and each historical acoustic token, including the following steps:

[0088] S1. Project the acoustic feature representations of the current acoustic token and each historical acoustic token onto the continuous discrimination space, and normalize the projection result (continuous feature representation).

[0089] Before calculating acoustic similarity, the acoustic feature representations are first subjected to spatial transformation and standardization to eliminate the interference of absolute feature amplitudes on similarity measurement. The continuity discrimination space is a mapping space specifically constructed to extract acoustic continuity features. Its dimension is usually lower than that of the original hidden state, aiming to remove redundant information unrelated to temporal continuity (such as global energy fluctuations) while retaining core feature components that can reflect the direction of acoustic pattern evolution.

[0090] This projection process is achieved by introducing a projection matrix. During system initialization or model training, a linear projection layer or a dimensionality-reducing neural network can be used as the initialization form of the projection matrix, and its parameters can be optimized as trainable parameters of the model. The projection operation maps the original high-dimensional acoustic feature representation to a continuous discriminative space, allowing subsequent similarity calculations to be performed in this more discriminative subspace.

[0091] Normalizing the projection results aims to unify the magnitudes of the feature vectors to the same order of magnitude, ensuring that similarity calculations depend only on the angle between the vectors, rather than their lengths. For example, L2 norm normalization can be used, which involves calculating the L2 norm of the projected feature vectors and dividing each element by this norm value to obtain a unit vector. In engineering practice, this normalization process eliminates interference from signal strength differences caused by variations in recording gain, distance, or channel attenuation, ensuring that only the morphological characteristics (i.e., orientation) of the acoustic pattern itself participate in subsequent measurements.

[0092] This step can be formally expressed as the following formula:

[0093] set up For a trainable projection matrix or a fixed-dimensionality reduction mapping, the normalized feature representation is as follows:

[0094]

[0095] in, It is a very small positive number to prevent the denominator from being divided by zero.

[0096] S2. Calculate the cosine similarity between the current acoustic token and each historical acoustic token based on the normalized features, and use it as the acoustic similarity.

[0097] After obtaining the normalized features in the continuous discriminant space, the similarity between the current acoustic token and each historical acoustic token is quantified by calculating the vector dot product. Since the features have been L2 normalized, the dot product between two vectors is numerically equivalent to the cosine of their angle, which is strictly limited to the interval [-1, 1]. A value closer to 1 indicates that the two acoustic tokens have more consistent directions in the continuous discriminant space, meaning their acoustic features are more similar; a value closer to -1 indicates that the features are completely opposite; and a value close to 0 indicates that the features are orthogonal or unrelated.

[0098] This calculation process can be formally represented as: the normalized feature of the current acoustic token in the continuous discriminant space is... , No. A historical acoustic token (i.e., distance from the current position) The normalized feature of the token at each time step is: The acoustic similarity between the two is... Determined by the following formula:

[0099]

[0100] In the formula, Representing vectors transpose, This represents the scalar similarity score. By introducing cosine similarity as a metric, it is possible to effectively capture the continuous changes in the feature space direction between temporally adjacent frames in a speech signal. Compared to directly calculating the Euclidean distance, cosine similarity is invariant to the feature scale and can more robustly handle nonlinear fluctuations in signal intensity, thus providing an accurate and reliable weighting basis for subsequent similarity-based weighted aggregation of historical expert priors.

[0101] By projecting features onto a continuity discriminant space for normalization and calculating cosine similarity, the complex time-frequency information contained in the original acoustic features is transformed into a scalar metric that accurately reflects the strength of acoustic continuity. This directional similarity-based calculation method allows subsequent steps to adaptively adjust the influence weight of historical expert selections on the current route based on the similarity level, thereby enhancing route consistency within stable acoustic intervals and maintaining route flexibility at acoustic boundaries.

[0102] Step S120: Using the acoustic similarity corresponding to each historical acoustic token as the weight, perform weighted aggregation on the expert routing probability distribution corresponding to each historical acoustic token to generate the historical expert prior distribution.

[0103] Each historical acoustic token has had its corresponding expert routing probability distribution determined during previous processing. This distribution records the probability of a historical token being assigned to each expert. To utilize historical information to guide the routing of the current token, the system performs weighted aggregation of the expert routing probability distributions of these historical acoustic tokens, using the acoustic similarity to each token as the weight. Higher acoustic similarity indicates a closer similarity between the corresponding historical token and the current token's acoustic characteristics, resulting in greater reference value for the expert routing selection of the current token; therefore, it is assigned a higher weight during aggregation.

[0104] In some optional implementations, to prevent unstable weight values, the acoustic similarity corresponding to each historical acoustic token can be converted into continuous weights using a gating function (such as the sigmoid function). These continuous weights are positively correlated with acoustic similarity; that is, the higher the acoustic similarity, the closer the continuous weight is to 1, and vice versa. The value range of the continuous weights is limited to between 0 and 1.

[0105] The process of converting the acoustic similarity corresponding to each historical acoustic token into continuous weights through a gating function can be formally represented as follows:

[0106]

[0107] in, This represents the continuity weight between the t-th token in layer l and a historical token at a distance d. It is a sigmoid-gated function. A preset similarity threshold is used to determine the degree of similarity of acoustic features. Control the steepness of the gating curve.

[0108] When adjacent acoustic representations are close It is close to 1; when the characteristic changes significantly, the value is close to 0.

[0109] In some optional implementations, a distance attenuation factor can be set to increase the influence of neighboring frames, such as... .

[0110] After obtaining the continuity weights for each historical acoustic token, a weighted sum of the historical expert routing probability distributions for each historical acoustic token can be calculated based on these weights to generate a historical expert prior distribution. This historical expert prior distribution is a probability vector with the same dimension as the number of experts, reflecting the distribution of experts that the historical context tends to choose under acoustic continuity constraints. For example, if historical tokens with high similarity to the current token tend to choose a specific expert, the probability value corresponding to that expert in the historical expert prior distribution will be significantly higher.

[0111] The process of generating the prior distribution of historical experts can be formally represented as:

[0112]

[0113] Step S130: Correct the basic route probability distribution using the historical expert prior distribution to obtain the corrected expert route probability distribution of the current acoustic token.

[0114] Specifically, this application integrates the historical expert prior distribution generated in step S120 into the basic route probability distribution obtained in step S100 to achieve a soft correction to the current route decision. This correction process, while preserving the current token's own characteristic judgment, makes adjustments based on historical priors, and can be implemented through operations in the probability space or logarithmic probability space.

[0115] This step involves modifying the basic route probability distribution using the prior distribution of historical experts, which constitutes the operation of imposing acoustic continuity constraints on the current acoustic token routing decision.

[0116] Step S140: Based on the corrected expert routing probability distribution, determine at least one active expert from each expert in the hybrid expert coding layer, input the current acoustic token into each active expert for feature encoding and fuse the encoding results to obtain the encoding result of the current acoustic token.

[0117] Specifically, based on the corrected expert routing probability distribution, at least one active expert is determined from among the experts in the hybrid expert coding layer. For example, a Top-K strategy can be adopted, which selects the K experts with the highest probability values ​​as active experts, where K is a preset positive integer (such as 1, 2, or 4). By using Top-K selection, it is ensured that each token is processed only by the few most relevant experts, thereby effectively controlling the amount of computation while maintaining model capacity.

[0118] After the activation experts are determined, the current acoustic token is input into each activation expert for feature encoding. Each activation expert performs an independent feedforward neural network calculation on the received current acoustic token and outputs the corresponding feature encoding result.

[0119] In some optional implementations, to improve computational efficiency, all acoustic tokens within a batch can be rearranged into contiguous memory groups according to the activation expert's number, ensuring that all tokens processed by the same expert are arranged consecutively in memory. This arrangement facilitates the parallel execution of feature encoding calculations for each expert using Grouped Matrix Multiplication, reducing the startup overhead of the computation kernel.

[0120] Finally, the feature encoding results output by each activated expert are fused. The fusion method, for example, involves weighted summation of the expert outputs based on the probability values ​​corresponding to each activated expert in the corrected expert routing probability distribution. Specifically, each expert's output vector is multiplied by its corresponding routing weight, and all weighted vectors are summed to obtain the final encoding result of the current acoustic token. This encoding result integrates the perspectives of multiple experts, utilizing their specialized abilities while ensuring routing continuity through historical prior corrections, thereby improving the speech encoder's ability to model continuous acoustic patterns and the overall stability of the representation.

[0121] The method provided in this application obtains the expert routing probability distribution of historical acoustic tokens, and generates a historical expert prior distribution by weighting and aggregating it with acoustic similarity as the weight. This prior distribution is then used to correct the basic routing probability distribution of the current token, making the routing results of each frame within a continuous sound segment more consistent. This effectively avoids repeated expert switching caused by minor perturbations in routing scores, ensuring that acoustically similar consecutive frames are more stably assigned to the same or similar experts, rather than being fragmented and scattered among different experts. Each expert can receive temporally continuous and pattern-complete samples, and the gradient update source tends to be stable, which is beneficial for experts to form a stable modeling ability for complete acoustic patterns.

[0122] Furthermore, this application uses acoustic similarity as the weight of historical expert priors. When the acoustic features of consecutive frames are highly similar, the historical expert prior distribution has a significant effect on route correction, and the route remains stable. When acoustic features change significantly at phoneme boundaries, silence transitions, or abrupt changes in the acoustic scene, acoustic similarity decreases, the correction effect of the historical expert prior distribution automatically weakens, and the route reverts to a basic route probability distribution based primarily on the features of the current token itself. This application achieves an adaptive balance between route stability and boundary sensitivity. This adaptive constraint effectively solves the problem of route instability in speech coding using hybrid expert models, improves the encoder's ability to model acoustic continuity, and enhances the final speech recognition performance.

[0123] In some embodiments of this application, an optional implementation of the aforementioned step S130 is described, which uses the prior distribution of historical experts to correct the probability distribution of the basic route.

[0124] In this embodiment, the overall continuity strength corresponding to the current acoustic token can be calculated, and the correction magnitude of the historical expert prior distribution can be controlled by the overall continuity strength to correct the basic route probability distribution. Specifically:

[0125] First, the overall continuity strength corresponding to the current acoustic token is calculated based on the acoustic similarity between the current acoustic token and each historical acoustic token.

[0126] To quantify the stability of the current acoustic token over time, this embodiment obtains the acoustic similarity between the current acoustic token and each historical acoustic token within the historical window. These acoustic similarities reflect the proximity of the current moment to past moments in the acoustic feature space. A high overall acoustic similarity within the historical window indicates that the current acoustic token is within a continuous range of stable acoustic feature changes; conversely, a generally low acoustic similarity suggests that the current moment may be at the boundary of phoneme switching, sudden silence, or acoustic scene transition. Statistical processing of these acoustic similarities generates a numerical index characterizing the strength of local continuity at the current time point, i.e., the overall continuity strength. This overall continuity strength directly determines the influence weight of historical experience (i.e., the prior distribution of historical experts) on the current routing decision in subsequent steps.

[0127] The process of calculating the overall continuity strength corresponding to the current acoustic token can be formally represented as:

[0128]

[0129] in, This represents the overall continuity strength of the t-th acoustic token in the l-th hybrid expert coding layer. Here, clip represents the truncation function, used to restrict the overall continuity strength between 0 and 1.

[0130] Furthermore, the basic route probability distribution is modified by controlling the correction magnitude of the historical expert prior distribution with the overall continuity strength.

[0131] Specifically, utilizing the aforementioned overall continuity strength As a correction coefficient, the basic route probability distribution is adaptively adjusted. This correction coefficient is positively correlated with the correction magnitude of the historical expert prior distribution on the basic route probability distribution. When the overall continuity strength value is large (close to 1), it indicates that the current acoustic token is highly similar to the historical acoustic token. In this case, the correction magnitude of the historical expert prior distribution is increased, that is, the historical experts are given a higher weight in the route calculation, so that the current acoustic token tends to maintain the expert path consistent with the historical acoustic token, thereby utilizing the stability of expert assignment to handle continuous acoustic patterns. When the overall continuity strength value is small (close to 0), it indicates that the current acoustic token has undergone significant acoustic changes. In this case, the correction magnitude of the historical expert prior distribution is reduced, reducing the interference of historical experience on the current route, so that the basic route probability distribution calculated based on the characteristics of the current token itself plays a dominant role in the decision-making, thereby allowing the routing mechanism to quickly switch to experts more suitable for new acoustic features. Through this dynamic adjustment mechanism based on the overall continuity strength, the technical effect of maintaining stable expert assignment within continuous acoustic segments and maintaining the flexibility of expert switching at the real acoustic boundary is achieved.

[0132] In some embodiments of this application, an optional implementation method is provided to correct the basic route probability distribution by controlling the correction magnitude of the historical expert prior distribution with the overall continuity strength, specifically including the following steps:

[0133] S1. Transform the prior distribution of historical experts into a correction term in logarithmic space.

[0134] Based on the historical expert prior distribution corresponding to the current acoustic token obtained through the aforementioned steps, this distribution is first mapped to logarithmic space to avoid numerical overflow or insufficient smoothness issues that may arise from direct manipulation in the probability space. The historical expert prior distribution represents the expert selection tendency aggregated based on the feature similarity of historical acoustic tokens, characterizing the degree of preference for each expert at a historical moment. By introducing logarithmic operations into the route correction process, the multiplicative relationship in the probability space can be transformed into an additive relationship in the logarithmic space, facilitating subsequent fusion with the basic route score. In specific implementation, logarithmic operations are performed on each dimension of the probability value in the historical expert prior distribution. To prevent numerical anomalies when the input value is zero, a very small smoothing constant can be added to the input term of the logarithmic operation. The logarithmic space result obtained after this transformation is the correction term, whose dimension is consistent with the total number of experts. It is used to adjust the base route score additively in subsequent steps.

[0135] S2. The correction term is weighted by the overall continuity strength to obtain the weighted correction term. The weighted correction term is added to the routing score logits corresponding to the basic routing probability distribution to obtain the corrected routing score logits. After normalization, the corrected expert routing probability distribution is obtained.

[0136] This step can be formally represented as:

[0137]

[0138]

[0139] in, The acoustic continuity constraint strength is used to control the maximum effect of the weighted correction term (i.e., the continuity prior) in the l-th layer. This represents the final routing probability of the t-th acoustic token being assigned to the e-th expert in the l-th hybrid expert coding layer.

[0140] It should be noted that the overall continuity strength in this application Instead of imposing binarization control on the routing correction term, it provides a continuous, differentiable, soft adjustment mechanism.

[0141] when When the value approaches 0, it indicates that the overall acoustic similarity between the current acoustic token and all acoustic tokens within its historical window is low, meaning the current frame is at an acoustic boundary or scene transition point. At this point, the reference value of the historical expert prior distribution is significantly reduced, and the correction term... As the decay approaches zero, the corrected routing score... Degenerates into a basic routing score determined solely by the hidden state of the current acoustic token itself. The routing mechanism reverts to frame-by-frame independent decision-making mode.

[0142] when A larger value indicates that the current frame is within a continuous and stable acoustic range, and the historical expert selection information has high reference value. However, this application does not force the current token to choose the exact same expert as the historical token. The correction term is only added to the base routing score in logarithmic probability form. Therefore, even if the prior distribution of historical experts strongly favors a certain expert, if the base routing score of the current token shows a low matching degree for that expert, the corrected routing probability will still remain at a reasonable level, and there will be no situation where it is rigidly locked to a certain expert and cannot switch. In other words, the historical expert prior only plays an enhancing role when the current token's own routing judgment is consistent with the prior direction. When the two conflict, the acoustic characteristics of the current token itself still retain the dominant role in the final decision.

[0143] The aforementioned soft adjustment mechanism enables the routing scheme of this application to effectively suppress unnecessary expert oscillations within a stable continuous range, while retaining sufficient switching flexibility at the acoustic reality boundary, thus avoiding damage to the model's response to rapid acoustic changes due to excessive constraints.

[0144] Ultimately from Select the K experts with the highest probabilities, and then locally renormalize the weights of the selected experts:

[0145]

[0146]

[0147]

[0148] in, This represents the set of K experts selected for the l-th acoustic token. This represents the route weights after renormalization within the selected K experts. This means inputting the hidden state of the t-th acoustic token in layer l into K activation experts, each activation expert... Output an encoded result, using normalized route weights. The final encoded output of the acoustic token is obtained by weighted summing of the encoded results from each activated expert. .

[0149] This application also provides a training strategy for an acoustic encoder. The acoustic encoder and decoder constitute a speech recognition model. During the model training phase, its loss function may include speech recognition loss, acoustic continuity loss, and expert load balancing loss.

[0150] The speech recognition loss can be CTC, attention cross-entropy, self-supervised prediction loss, or a combination thereof.

[0151] Acoustic continuity loss is used to constrain the difference in expert route probability distributions between adjacent acoustic tokens, using their acoustic similarity as weights, thereby avoiding unconditional smoothing of the true boundary. The acoustic continuity loss can be measured using the Jensen-Shannon divergence metric, and the following provides an optional calculation method:

[0152]

[0153] The expert load balancing loss is used to constrain the average routing probability of each expert and the actual allocation ratio, so as to avoid continuous priors causing long-term concentration of large segments of speech by a few experts.

[0154] To limit the excessive increase of router logits, a z-loss can be added. The total loss function can be expressed as:

[0155]

[0156] in, Indicates speech recognition loss, Indicates acoustic continuity loss, This indicates the expert load balancing loss. and These are the weights corresponding to the losses.

[0157] In the early stages of model training, the routers have not yet formed a stable and reliable division of labor among experts, and the reference value of historical experts' prior knowledge is limited. Applying strong continuity constraints too early may cause routing decisions to fall into a localized pattern dominated by the initial random state, affecting the balance of expert division of labor. Therefore, this application adjusts the strength of the continuity constraints during the training startup phase. and continuity loss weights The algorithm gradually increases from a preset initial value to a target value, allowing the router to first establish basic expert matching relationships in a frame-by-frame independent decision-making mode. Then, as the training progresses, the constraints of historical expert priors are gradually introduced. This incremental process can be implemented using various scheduling methods such as linear interpolation, exponential decay, or piecewise functions.

[0158] Furthermore, during gradient backpropagation, the continuous feature extraction branch (the process of projecting the acoustic feature representations of the current acoustic token and each historical acoustic token onto the continuous discriminant space to obtain continuous features) can choose to block or retain its gradient propagation based on the actual needs of training stability. Specifically, if the continuous feature branch remains trainable, the model theoretically has the potential to reduce continuity loss and thus circumvent continuity constraints by adjusting the acoustic similarity of the hidden states. To suppress this tendency, gradient backpropagation of this branch can be blocked in the early stages of training or throughout the entire training process, keeping the continuous feature representation used for similarity calculation relatively stable. This forces the model to satisfy continuity constraints by optimizing the routing decision itself rather than manipulating acoustic similarity, thereby ensuring that the original intention of the continuous training objective is effectively implemented.

[0159] It should be noted that the aforementioned auxiliary training objectives, such as continuity loss and load balancing loss, only participate in parameter optimization during the model training phase. During the inference deployment phase, all auxiliary loss functions are removed. The inference path only retains forward computation processes such as acoustic similarity calculation, gating transformation, historical prior aggregation, and routing score correction, without involving any loss calculation or gradient solving operations. This ensures that inference efficiency is not additionally affected by the auxiliary training modules.

[0160] In some embodiments of this application, a layered structure for a hybrid expert coding layer in a speech encoder is provided. Specifically:

[0161] There are multiple hybrid expert coding layers, which are arranged sequentially in the depth direction of the speech encoder. Each expert in each hybrid expert coding layer is divided into an acoustic expert pool and a semantic expert pool. The experts in the acoustic expert pool are used to implement acoustic pattern coding, and the experts in the semantic expert pool are used to implement semantic pattern coding.

[0162] Multiple hybrid expert coding layers are stacked sequentially along the depth direction of the speech encoder, constructing a vertical hierarchical architecture. As the network depth increases, different coding layers undertake feature extraction tasks with varying levels of abstraction. Through this hierarchical stacking structure, speech signals can be progressively transformed from the raw acoustic features at the bottom layer into abstract semantic representations at higher levels, thus achieving multi-level, progressive parsing of speech signals. Within each hybrid expert coding layer, multiple experts operating in parallel are explicitly divided into acoustic expert pools and semantic expert pools. This partitioning mechanism aims to match the representation learning patterns of the speech encoder at different depth stages, ensuring that the functional characteristics of the experts are adapted to the data distribution and feature attributes of their respective layers.

[0163] The acoustic expert pool is configured to perform acoustic pattern coding, focusing on processing and extracting the physical acoustic properties of speech signals. This type of coding focuses on the physical characteristics of speech, encompassing dimensions such as local spectral morphology, phoneme articulation states, speaker timbre, recording channel characteristics, background noise, and reverberation conditions. Experts in the acoustic expert pool focus on modeling short-term, local acoustic variations, with their input primarily derived from downsampled low-level acoustic representations. In the shallow regions of the encoder, the acoustic expert pool plays a dominant role, responsible for capturing subtle changes and stable patterns in the acoustic signal, such as continuous features within the same phoneme or noise patterns under specific conditions. Through refined processing of this local acoustic information, the acoustic expert pool lays a solid foundation for subsequent semantic understanding, ensuring the model has a full capacity to perceive the physical details of the speech signal.

[0164] The semantic expert pool is configured to perform semantic pattern coding, focusing on processing and extracting high-level semantic information from speech signals. This type of coding focuses on the linguistic meaning and contextual relationships carried by speech, including phoneme combination rules, word-level structure, long-range dependencies, and semantic distinguishing information. Experts in the semantic expert pool focus on modeling the hidden states that incorporate long-range context, with their input primarily derived from deep representations after multiple layers of coding. In the deeper regions of the encoder, the semantic expert pool plays a dominant role, responsible for further abstracting acoustic features into semantic features that can express lexical meaning and syntactic relationships. Through processing high-level semantic information, the model can accurately understand the actual content corresponding to speech segments, thereby improving the accuracy of speech recognition systems in understanding complex linguistic phenomena and long sentence contexts.

[0165] By dividing the experts within the hybrid expert coding layer into acoustic expert pools and semantic expert pools, and configuring them hierarchically according to the encoder depth, experts in different coding layers can perform their respective duties. Shallow coding layers primarily rely on the acoustic expert pool to perform refined modeling of local acoustic patterns, ensuring the ability to capture acoustic details; deep coding layers primarily rely on the semantic expert pool to perform macroscopic modeling of global semantic patterns, ensuring a deep understanding of language content. This hierarchical expert structure effectively avoids gradient interference and functional redundancy caused by using homogeneous experts in all coding layers, prompting experts to form a clear specialization during training, thereby improving the overall modeling efficiency and recognition accuracy of the speech encoder.

[0166] In some embodiments of this application, based on the structure of the hybrid expert coding layer in the speech encoder described above, another optional implementation scheme of the speech coding method of this application is introduced, referring to... Figure 3 As shown:

[0167] After correcting the basic route probability distribution using the historical expert prior distribution in step S130 to obtain the corrected expert route probability distribution for the current acoustic token, the following steps are also included:

[0168] Step S200: Determine the layer depth position of the current hybrid expert coding layer in the encoder depth direction.

[0169] In the architecture design of a speech encoder, the layer depth position in the encoder's depth direction is a key parameter determining the representation modeling task that the current layer should undertake. The layer depth position of the current hybrid expert coding layer is determined by the overall layer structure of the encoder. For example, if the encoder contains L coding layers, then the layer depth position of the l-th layer is l. This layer depth position is used to distinguish whether the current layer belongs to a shallow layer that focuses on local acoustic feature extraction, an intermediate layer that undertakes the transition from acoustic to semantics, or a deep layer that focuses on long-range context and semantic understanding.

[0170] Step S210: Determine the layer depth prior based on the layer depth position. The layer depth prior is used to characterize the degree of bias of the current hybrid expert coding layer towards the acoustic expert pool and the semantic expert pool.

[0171] The layer depth prior is a distributed bias term constructed based on the specific position of the current layer in the encoder depth direction. It is used to guide the routing mechanism to show a preference for the acoustic expert pool and the semantic expert pool at different depths.

[0172] In one alternative implementation, the process of constructing the layer-depth prior may include:

[0173] First, define the end point of the shallow acoustic phase. and the starting position of the deep semantic stage And the percentage of semantic experts that varies with coding depth Percentage of semantic experts This is used to represent the degree of bias of the semantic expert pool S in the expert routing process of layer l.

[0174] Percentage of semantic experts This can be set as a function that varies linearly with depth, such that the coefficient is close to 0 in shallow layers and close to 1 in deep layers. For example, the semantic expert percentage coefficient... This can be expressed as the following formula:

[0175]

[0176] when hour, This layer is primarily calculated by experts from acoustic expert pool A; when hour, Acoustic expert pool A and semantic expert pool S jointly participate in the computation; when hour, This layer is primarily computed by experts from the semantic expert pool S. Therefore, the expert function changes continuously with the coding depth, rather than discretely switching at a fixed coding layer.

[0177] Subsequently, based on this semantic expert ratio coefficient The layer depth prior is determined, which includes the prior weights corresponding to the acoustic expert pool A and the semantic expert pool S. This can be formally represented as:

[0178]

[0179] in, A very small positive number set to prevent numerical anomalies in logarithmic operations.

[0180] In another alternative implementation, the process of constructing the layer-depth prior may include:

[0181] S1. Calculate the layer depth coefficient based on the layer depth position. The layer depth coefficient is positively correlated with the depth of the current coding layer.

[0182] Layer depth coefficients are used to characterize the relative position of the hybrid expert coding layer in the depth direction of the speech encoder. During calculation, the index of the current hybrid expert coding layer is first obtained and mapped to the total number of encoder layers or a preset layer range. The numerical construction of the layer depth coefficients follows a monotonically increasing logic with increasing encoder depth, which can be implemented mathematically through linear interpolation or a non-linear sigmoid function. For example, using the start and end positions of the shallow acoustic stage and the start position of the deep semantic stage as reference points, the layer depth coefficients are dynamically calculated based on the relative position of the current layer index within the aforementioned range.

[0183] In one alternative implementation, the layer depth position number can be directly used as the layer depth coefficient.

[0184] S2. Determine the first deep prior corresponding to the experts in the acoustic expert pool and the second deep prior corresponding to the experts in the semantic expert pool based on the depth coefficient.

[0185] Wherein, when the layer depth coefficient is less than the first threshold, the correction magnitude of the first layer depth prior to the expert routing probability is greater than that of the second layer depth prior.

[0186] The first and second layer deep priors are quantitative indicators generated based on the layer depth coefficient, designed to adjust the bias of expert routing. The first layer deep prior acts on the acoustic expert pool to increase the selection probability of acoustic experts when the current coding layer is shallow; the second layer deep prior acts on the semantic expert pool to increase the selection probability of semantic experts when the current coding layer is deep. A first threshold is introduced as a discrimination criterion, which defines the boundary of the shallow acoustic dominance region. When the layer depth coefficient is less than the first threshold, it indicates that the current coding layer is shallow, and the generated first layer deep prior value is significantly greater than the second layer deep prior. This numerical difference makes the correction magnitude from the acoustic expert pool dominant when correcting the expert routing probability, thereby guiding the routing mechanism to prioritize the activation of acoustic experts and ensuring that the shallow network focuses on capturing acoustic features such as local spectrum and phoneme pronunciation.

[0187] When the layer depth coefficient is greater than the second threshold, the correction magnitude of the second layer depth prior to the expert routing probability is greater than that of the first layer depth prior, and the second threshold is greater than the first threshold.

[0188] When the layer depth coefficient exceeds the second threshold, it indicates that the current coding layer has entered the higher-level region of the network, and the focus of the representation task shifts from local acoustic patterns to long-range context and semantic understanding. At this stage, the determined second-layer deep prior value surpasses the first-layer deep prior, making the correction magnitude corresponding to the semantic expert pool dominant. By setting a second threshold greater than the first threshold, an intermediate transition interval is constructed between the shallow and deep dominant regions. Within this transition interval (i.e., the layer depth coefficient is between the first and second thresholds), the correction magnitudes of the first and second-layer deep priors exhibit a smooth inverse relationship, avoiding discrete jumps. This dual-threshold control mechanism allows the switching of expert functions to be completed gradually with increasing coding depth, effectively preventing abrupt changes in hidden state distribution or gradient oscillations caused by drastic reversals in expert preferences between adjacent coding layers, and ensuring the coherence of feature evolution in the depth direction of the speech encoder.

[0189] Step S220: Based on the layer depth prior, perform a second correction on the corrected expert routing probability distribution of the current acoustic token to obtain the final corrected expert routing probability distribution.

[0190] Specifically, at the numerical computation level, this correction operation involves adding and fusing the numerical elements in the layer-depth prior vector with the routing score logits corresponding to the expert routing probability distribution after the current acoustic token correction.

[0191] Specifically, the corrected expert routing probability distribution is typically obtained by normalizing the routing score logits using the Softmax function. Therefore, during the second correction, the layer depth prior is added as a bias term to the original routing score logits before normalization. If the current layer is in a shallow position, the layer depth prior will increase the routing score of experts in the acoustic expert pool, causing the probability distribution after the second correction to tilt towards acoustic experts; conversely, if the current layer is in a deep position, the layer depth prior will enhance the competitive advantage of experts in the semantic expert pool. The routing score after addition is then normalized again using Softmax to generate the final corrected expert routing probability distribution. This final distribution simultaneously contains correction information based on acoustic continuity and structured guidance information based on layer depth, enabling the hybrid expert coding layer to adaptively select the expert type most suitable for the current representation task at different depths. This achieves a technical shift in expert routing from purely data-driven to data and structure-driven collaboratively.

[0192] Depth Prior After incorporating the expert routing score, we obtain the final routing score for the t-th acoustic token in layer l to the e-th expert:

[0193]

[0194] in, This represents the final route score after incorporating depth priors. This represents the route score after acoustic continuity prior correction. This indicates the strength of the prior influence of the depth in the l-th layer.

[0195] Performing Softmax normalization on the final route score yields the final corrected expert route probability that the t-th acoustic token in layer l is assigned to the e-th expert:

[0196]

[0197] in, This represents the final corrected expert route probability. Let represent the total number of experts at layer l who can participate in routing, and j represent the expert summation index. Then, the expert with the highest probability is selected based on the routing probability. One expert:

[0198]

[0199] in, This represents the set of experts who are actually activated by the t-th acoustic token in the l-th layer. This indicates the number of experts actually activated by a single token at this layer.

[0200] Layer depth priors are soft constraints; the basic routing score generated by the current acoustic token itself still retains a decisive role. Even when a shallow acoustic token has a significantly high matching degree with a specific semantic expert, Even if the number is small, the semantic expert may still be included in the set of activation experts. Similarly, deep acoustic tokens can also select acoustic experts when necessary, thereby avoiding the degradation of model expressive power caused by hard-shielding a certain expert pool.

[0201] In one possible implementation, to avoid When the value is 0 or 1, the other expert pool is almost completely blocked. A minimum open ratio for the expert pool can be set. And the actual proportion of semantic experts used is limited to:

[0202]

[0203] in, This represents the percentage of semantic experts after the minimum openness limit has been applied. This represents the minimum open percentage that any expert pool needs to retain, satisfying the following conditions:

[0204]

[0205] by Alternative Then, the layer depth prior can be expressed as:

[0206]

[0207] This approach allows the shallow layer to retain a small amount of semantic expert selection capability, while the deep layer retains a small amount of acoustic expert selection capability, thus improving the model's adaptability in abnormal acoustic scenarios and complex contextual conditions.

[0208] This embodiment of the method determines the layer depth and generates a layer depth prior accordingly to perform a secondary correction on the routing probability. This causes the shallow encoder to tend to activate the experts responsible for acoustic pattern coding, thereby stably extracting local spectral and phoneme features. Simultaneously, it causes the deep encoder to tend to activate the experts responsible for semantic pattern coding, in order to model long-range context and semantic associations. This hierarchical guidance mechanism effectively binds coding layers of different depths to corresponding expert function pools, avoiding the randomization and hybridization of expert functions, and improving the professionalism of the speech encoder's representations at different levels of abstraction and its overall modeling capability.

[0209] In some embodiments of this application, the correction strength when using historical expert prior distribution to correct the basic route probability distribution in the foregoing embodiments can vary with the depth of the hybrid expert coding layer. The correction strength of the hybrid expert coding layer located at the first depth is greater than the correction strength of the hybrid expert coding layer located at the second depth, and the first layer is shallower than the second layer.

[0210] In the hierarchical expert encoder architecture, the correction strength characterizes the degree to which the prior distribution of historical experts influences the current basic route probability distribution. This degree is quantified and controlled by continuity constraint parameters. Speech signals exhibit differentiated representational characteristics at different coding depths: the shallow hybrid expert coding layer at the first depth mainly processes low-level acoustic information with strong temporal correlation, such as local spectral morphology and phoneme articulation states; while the deep hybrid expert coding layer at the second depth integrates long-range context, focusing on high-level abstract information such as phoneme combinations, word-level structure, and semantic distinctions. Given the high similarity and relative stability of features between adjacent acoustic tokens in the shallow layer, a large correction strength is needed to suppress random route jitter and ensure that acoustic experts can process complete continuous acoustic segments. As the coding layer depth increases, the semantic abstraction of tokens increases, and adjacent positions may not correspond to the same semantic function. If an excessively large correction strength is maintained at this point, it will limit the model's ability to flexibly switch experts based on context. Therefore, the correction strength is configured as a dynamic variable that varies with the depth of the hybrid expert coding layer.

[0211] In one optional implementation, the correction strength when using the prior distribution of historical experts to correct the basic route probability distribution is... Smooth interpolation can be performed using the following formula:

[0212] ,satisfy, .

[0213] in, Indicates the first The strength of the continuity constraint actually used in the layer (i.e., the strength of the correction when the basic route probability distribution is corrected using the prior distribution of historical experts). This indicates the strength of the continuous constraint used in the shallow acoustic stage. This indicates the strength of the continuity constraints used in the deep semantic stage. and It can be pre-configured to satisfy the size relationship in the example above.

[0214] When the When the layer is in the shallow stage, Since it is close to 0, therefore:

[0215]

[0216] At this point, the expert prior knowledge formed by historical acoustic tokens has a strong influence on the current route. When the layer is in the deep stage, Since it is close to 1, therefore:

[0217]

[0218] At this point, the routing result is largely determined by the high-level context representation of the current acoustic token. In the transition layer, the strength of the continuity constraint... The coding layer depth gradually decreases. Therefore, this application achieves linkage between expert pool type, layer depth routing prior, and temporal continuity strength.

[0219] The embodiments described above enable adaptive matching between the continuity constraint strength and the hierarchical representation rules of the speech encoder. This allows the hybrid expert encoder to have strong continuity constraints in the shallow layers to maintain the stability of acoustic modeling, while having weak continuity constraints in the deep layers to retain the flexibility of semantic modeling. This hierarchically differentiated strength control mechanism effectively avoids the problems of insufficient local feature smoothing or global semantic rigidity that may be caused by uniform strength across all layers, thereby improving the comprehensive processing performance of the speech encoder for complex acoustic patterns and variable semantic environments.

[0220] The acoustic expert pool A and the semantic expert pool S can be configured with different numbers of experts and different dimensions for the intermediate expert layer. Let the number of experts in the acoustic expert pool be... The number of experts in the semantic expert pool is Then it can be expressed as:

[0221]

[0222]

[0223] Shallow layers can be configured with a smaller number of acoustic experts with relatively stable workloads, allowing each acoustic expert to obtain sufficient and continuous training samples. Deep layers can be configured with a larger number or greater capacity of semantic experts to improve the model's capacity to model complex contextual patterns and long-tail semantic phenomena. That is, the number of experts in the semantic expert pool can be greater than or equal to the number of experts in the acoustic expert pool.

[0224] In one implementation, each layer can maintain the same number of activated experts:

[0225]

[0226] in, This indicates the number of activation experts uniformly adopted across all coding layers.

[0227] Even with deeper configuration of more semantic experts, As the encoding depth increases, a single acoustic token still only executes a fixed number of times. The feedforward computation of experts does not directly lead to a linear increase in the activation computation of a single token as the total number of experts increases.

[0228] For the The first in the layer Each token, and its multi-expert output, can be represented as:

[0229]

[0230] in, Indicates the first The first in the layer The hidden state of each token Indicates the first The first in the layer Each expert corresponds to a feedforward network. Represents the normalized expert fusion weights This indicates the output of the multi-expert module.

[0231] The expert fusion weight can be expressed as:

[0232]

[0233] In some embodiments of this application, a strategy is provided for grouping and parallel execution of multiple experts using GMM operators to ensure that the hierarchical expert structure maintains high training efficiency while expanding model capacity. Therefore, step S140 in the aforementioned embodiments, which involves inputting the current acoustic token into each activated expert for feature encoding, includes:

[0234] All acoustic tokens within the batch containing the current acoustic token are rearranged into contiguous memory groups according to the activated expert's number; based on the acoustic token group corresponding to each expert, the feature encoding calculation of each expert is performed in parallel through group matrix multiplication.

[0235] After determining the activation expert number corresponding to each acoustic token within a batch through the routing mechanism of the hybrid expert coding layer, the data layout is first reorganized to fully utilize the parallel computing capabilities of the hardware. The core of this process lies in breaking the original time series or batch order and rearranging all acoustic tokens routed to the same expert into adjacent contiguous blocks in the physical memory space.

[0236] For input data of batch size B and sequence length T, the set of activated experts (e.g., Top-K experts) assigned to each token is first determined. Based on the activated expert ID of each token, all tokens with target expert 1 are extracted and arranged together, followed by tokens with target expert 2, and so on. This rearrangement operation aggregates acoustic tokens that were originally scattered across different samples and time steps but required processing by the same expert network parameters into contiguous memory groups. This data organization method allows subsequent matrix multiplication operations to read contiguous memory blocks at once when calculating the same expert, thus greatly reducing the randomness of memory access and improving data read bandwidth utilization. Simultaneously, corresponding index mapping information is generated to restore the results to the original token order after calculation, ensuring that subsequent residual connections or hierarchical processing correctly correspond to the input data.

[0237] After the acoustic tokens are rearranged and grouped, the core feature encoding calculation is performed. This process abandons the conventional approach of sequentially calling each expert, and instead uses the Grouped Matrix Multiplication (GMM) operator for highly parallel batch processing.

[0238] For each expert, the acoustic token group generated in the above steps corresponding to that expert is treated as an independent matrix multiplication task. Since the tokens corresponding to the same expert are continuously distributed in memory, the input matrix of that expert can be directly constructed. The GMM operator is configured to concurrently process the matrix multiplication tasks of multiple experts within a single computation kernel startup cycle. That is, the weight matrices corresponding to all experts are bound to their corresponding input token groups and submitted to the computation unit at once. During execution, the computation unit performs a series of operations in parallel, such as weights of expert A multiplied by token groups of expert A, weights of expert B multiplied by token groups of expert B, etc., without explicit looping and waiting at the software level. Through this grouped matrix multiplication method, the originally scattered and frequent small-scale matrix operations are transformed into intensive parallel computation, significantly reducing the number of computation kernel startups and device idle waiting time, thereby greatly improving the computational throughput of the multi-expert coding layer and supporting the efficient training and inference of large-scale hybrid expert speech coding models.

[0239] In some embodiments of this application, as described above, the speech recognition model training loss function includes expert load balancing loss. In this embodiment, to avoid conflicts between layer depth prior and the traditional global load balancing objective, the load balancing loss can be calculated separately based on the expert pool. Specifically:

[0240] The loss function during the speech encoder training phase can include the load balancing loss of each hybrid expert coding layer. Specifically, for hybrid expert coding layers with a layer depth coefficient less than a first threshold, the load balancing loss within the acoustic expert pool is calculated. For hybrid expert coding layers with a layer depth coefficient greater than a second threshold, the load balancing loss within the semantic expert pool is calculated. For hybrid expert coding layers with a layer depth coefficient between the first and second thresholds, the load balancing losses of the acoustic expert pool and the semantic expert pool are weighted and summed according to the first and second layer depth priors.

[0241] For example, both the first-layer deep prior and the second-layer deep prior are related to the aforementioned semantic expert proportion coefficient. Correspondingly, for a hybrid expert coding layer where the layer depth coefficients are between the first and second thresholds, the process of weighted summation of the load balancing loss of the acoustic expert pool S and the semantic expert pool A can be expressed as:

[0242]

[0243] in, Indicates the first Overall load balancing loss of the layer Acoustics expert pool Internal load balancing losses, Semantic expert pool Internal load balancing losses.

[0244] This approach can prevent optimization conflicts caused by the layer depth prior requiring shallow layers to prioritize acoustic experts, while the global load balancing objective requires all semantic experts to be activated on an average basis in shallow layers. It also reduces the risk of individual experts being overloaded or not being trained for a long time.

[0245] During model training, the acoustic expert pool, semantic expert pool, basic router parameters, layer depth prior strength, and continuity constraint parameters can be jointly optimized. The coding layer position and layer depth coefficients can be determined before forward computation, and the layer depth prior can be pre-generated and cached. Expert logits are directly added during each layer routing, thus avoiding the introduction of significant additional matrix calculations. The inference phase adopts the same expert pool partitioning and layer depth routing mechanism as the training phase. Each acoustic token still selects a fixed number of Top-K experts according to the final routing probability, thus not changing the encoder input / output interface and ensuring compatibility with existing CTC, Attention, or Encoder-Decoder recognition structures.

[0246] Through the hierarchical expert encoder described above, shallow acoustic tokens more easily form stable expert assignments under acoustic continuity constraints, enabling acoustic experts to learn relatively complete local spectra, phonemes, speaker and noise patterns. The intermediate layer, through the joint participation of two expert pools, achieves a smooth transition from acoustic representation to contextual representation. The deep layer, under weaker continuity constraints, uses a larger semantic expert pool, allowing the model to retain the ability to flexibly select experts based on long-range context and semantic differences. This structure avoids the expert function mixing caused by all coding layers sharing the same expert function and routing constraints, which is beneficial for improving expert specialization, routing stability, and the speech recognition encoder's ability to jointly model complex acoustic and semantic information.

[0247] Reference Figure 4 The example illustrates the hierarchical structure of the speech encoder of this application, including the shallow acoustic expert, the transitional layer mixing expert, and the deep semantic expert.

[0248] like Figure 4 As shown, the encoder is divided into a shallow acoustic stage, a transition stage, and a deep semantic stage along the depth direction. In the shallow acoustic stage, the coding layer mainly configures experts from the acoustic expert pool A to participate in the routing calculation, while a small number of experts can be configured in the acoustic expert pool. The continuity constraint strength is relatively large, which ensures that the local acoustic tokens maintain a stable expert allocation path in the time direction, thereby facilitating the acoustic experts to professionally learn low-level acoustic patterns such as spectral details, phoneme internal structure, and speaker characteristics.

[0249] In the deep semantic stage, the encoding layer mainly configures experts from the semantic expert pool S to participate in routing calculations. The semantic expert pool can be configured with a large number of experts. The continuity constraint strength is relatively small to weaken the temporal continuity constraint, enabling the deep hidden state, which incorporates long-range contextual information, to flexibly select experts based on semantic differences, thereby improving the model's ability to model word-level structures and long-range semantic dependencies.

[0250] During the transition phase, experts from both the acoustic expert pool A and the semantic expert pool S participate in the routing computation, and the strength of the continuity constraint smoothly transitions with increasing layer depth. This transition phase design ensures that the transformation of the encoding representation from local acoustic modeling to contextual semantic modeling is gradual rather than abrupt, avoiding jumps in hidden state distribution and discontinuous gradient update paths caused by sudden switching of expert types between adjacent encoding layers.

[0251] Reference Figure 5 This is a schematic diagram of the overall processing flow of the speech recognition method based on acoustic continuity-aware routing and hierarchical expert encoder in this application.

[0252] like Figure 5 As shown, the input speech signal is first processed by feature extraction and convolutional downsampling to be converted into an acoustic token sequence. This acoustic token sequence is then input into the encoder sequentially along the time dimension.

[0253] The encoder consists of L coding layers, each including an attention sublayer and a multi-expert feedforward sublayer. The multi-expert feedforward sublayer contains multiple parallel expert networks used for feature encoding of the current acoustic token. For each acoustic token in each coding layer, the routing process of the multi-expert feedforward sublayer is as follows:

[0254] First, a basic routing score is calculated based on the hidden state of the current acoustic token to obtain the basic matching relationship between the current token and each expert. Simultaneously, the continuous-aware routing module reads the feature representations and historical expert routing probability distributions of several historical acoustic tokens preceding the current token from the historical cache, calculates the acoustic similarity between the current token and each historical token, and weights and aggregates the expert routing probability distributions of each historical token using acoustic similarity as the weight to generate a historical expert prior distribution. Then, the basic routing score is corrected using the historical expert prior distribution to obtain the corrected expert routing probability distribution. Based on the corrected expert routing probability distribution, the K experts with the highest probabilities are selected as activation experts from the candidate expert set of the current coding layer.

[0255] In the hierarchical expert encoder, shallow coding layers preferentially select experts from the acoustic expert pool A, biased towards acoustic modeling. Deep coding layers preferentially select experts from the semantic expert pool S, biased towards semantic modeling. The intermediate transition layers involve both expert pools, with the participation of the semantic expert pool gradually increasing with layer depth. After each acoustic token is assigned to a corresponding activation expert, parallel feature encoding calculations are performed on all tokens routed to the same expert using the GMM grouping matrix multiplication operator. The outputs of each expert are then weighted and fused according to their routing weights to obtain the encoded output of the current layer.

[0256] After all coding layers have been processed sequentially, the encoder outputs a high-level acoustic representation. This representation can be directly fed into the CTC recognition head, attention decoder, or other sequence decoding modules to generate the final speech-to-text result.

[0257] The acoustic continuity sensing routing process runs through the shallow, transition, and deep layers, and dynamically adjusts the continuity constraint strength according to the coding depth. From the shallow to the deep layer, the continuity constraint strength can be adjusted from strong to weak.

[0258] The parallel computation process of GMM grouped matrix multiplication can run through shallow, transitional and deep layers.

[0259] Applying the speech coding method of this application, during the self-supervised pre-training stage, the speech encoder learns a general representation through large-scale unlabeled speech. Acoustic continuity loss and load balancing loss jointly constrain the router, enabling experts to form a stable acoustic division of labor while maintaining controllable load. The hierarchical expert structure allows shallow experts to prioritize learning local acoustic patterns, while deeper experts gradually learn contextual and semantically relevant patterns. The GMM execution path is used to complete the expert forward and backward computations without altering the original self-supervised objective.

[0260] During the supervised fine-tuning phase, the speech encoder integrates with CTC, Attention, or Encoder-Decoder training frameworks, using labeled speech for further optimization. Continuity constraint parameters, depth-gated parameters, and expert parameters can all be trained, or, when the data volume is small, some continuity projections and shallow experts can be frozen to preserve the acoustic specialization formed during pre-training. The input and output dimensions of the speech encoder are consistent with those of a regular encoder, facilitating compatibility with existing masking, loss calculation, checkpoint storage, and decoding processes.

[0261] During the offline inference phase, the base route score, acoustic similarity, and historical expert priors for the entire speech segment can be computed in parallel. During the streaming inference phase, a historical state cache of length W is maintained for each layer. The cache contains continuous feature representations of acoustic tokens and base expert route probabilities. When a new token arrives, only the acoustic similarity is calculated with the cached state to update the prior; future speech frames are not required. Expert outputs can still be executed via the GMM in inference batches.

[0262] In summary, the speech coding method adopted in this application has the following advantages:

[0263] Compared to a conventional MoE router that performs top-k selection independently for each acoustic token, this application explicitly utilizes the temporal continuity of the speech signal. Historical expert selection only affects the current route when acoustic features are similar, thus reducing repeated expert switching caused by fractional perturbations within the same sustained pronunciation, allowing similar acoustic tokens to more stably enter the same or similar experts. A stable expert input distribution facilitates experts learning complete acoustic patterns, reduces random fluctuations in expert gradient sources, and minimizes path jumps in the encoder representation that are unrelated to actual acoustic changes.

[0264] Compared to fixed smoothing or forced reuse of experts from the previous frame, this application dynamically controls historical priors through similarity gating. When phoneme boundaries, silence transitions, sudden noise, or other significant changes occur, the continuity gating weakens, allowing the router to select a new expert in a timely manner. Therefore, this method achieves an adaptive balance between routing stability and boundary sensitivity, avoiding over-smoothing of short sounds and rapid pronunciation changes.

[0265] Compared to structures that uniformly use the same type of experts and the same continuity parameters across all coding layers, this application organizes experts based on the gradual evolution of the speech encoder from local acoustic representations to high-level semantic representations. Shallow acoustic experts learn local acoustic patterns such as spectrum, phonemes, speakers, and noise under strong continuity constraints, while deep semantic experts model long-range context and semantic differences under weaker continuity constraints. The transition layer utilizes layer-depth priors to achieve smooth transitions. This structure reduces expert functional overlap, allowing model capacity to be used for effective specialization at different levels.

[0266] Regarding training efficiency, this application does not rely on continuous routing for direct acceleration. Instead, it uses the Gaussian Mixture Model (GMM) to change the multi-expert feedforward computation from a per-expert loop to grouped parallel execution. Thus, recognition performance and routing stability are handled by the acoustic continuity perception mechanism and the hierarchical expert structure, while expert computation efficiency is handled by the GMM execution path. Streaming inference only caches limited historical states and does not use future frames, thus allowing it to be applied while maintaining causality.

[0267] This application also provides an electronic device in its embodiments. (See reference...) Figure 6 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, tablets, learning machines, translators, large teaching screens, in-vehicle terminals, wearable devices, etc. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0268] like Figure 6 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 2 or a program loaded from a storage device 8 into a random access memory (RAM) 3, to implement the voice encoding method of the foregoing embodiments of this application. When the electronic device is powered on, the RAM 3 also stores various programs and data required for the operation of the electronic device. The processing unit 1, ROM 2, and RAM 3 are interconnected via a bus 4. An input / output (I / O) interface 5 is also connected to the bus 4.

[0269] Typically, the following devices can be connected to I / O interface 5: input devices 6 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 7 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 8 including, for example, memory cards, hard drives, etc.; and communication devices 9. Communication device 9 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0270] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the speech coding methods provided in this application.

[0271] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the speech coding methods provided in this application.

[0272] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0273] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0274] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0275] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0276] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

Claims

1. A speech coding method, characterized in that, A speech coding method applied to a speech encoder comprising a hybrid expert coding layer, wherein the hybrid expert coding layer includes multiple parallel experts, includes: Obtain the acoustic token sequence corresponding to the input speech, and determine the basic routing probability distribution of the current acoustic token in the acoustic token sequence; Obtain several historical acoustic tokens preceding the current acoustic token, and calculate the acoustic similarity between the current acoustic token and each of the historical acoustic tokens respectively; Using the acoustic similarity corresponding to each historical acoustic token as the weight, the expert routing probability distribution corresponding to each historical acoustic token is weighted and aggregated to generate the historical expert prior distribution; The basic route probability distribution is corrected by using the historical expert prior distribution to obtain the corrected expert route probability distribution of the current acoustic token; Based on the corrected expert routing probability distribution, at least one active expert is determined from each expert in the hybrid expert coding layer. The current acoustic token is then input into each of the active experts for feature encoding and the encoding results are fused to obtain the encoding result of the current acoustic token.

2. The method according to claim 1, characterized in that, The process of determining the underlying routing probability distribution of the current acoustic token includes: Obtain the hidden state of the current acoustic token; Based on the hidden state of the current acoustic token, calculate the matching score between the current acoustic token and each expert in the hybrid expert coding layer; Normalization is performed on each of the matching scores to obtain the basic route probability distribution.

3. The method according to claim 1, characterized in that, The process of calculating the acoustic similarity between the current acoustic token and each of the historical acoustic tokens includes: The acoustic feature representations of the current acoustic token and each of the historical acoustic tokens are projected onto the continuity discrimination space, and the projection results are normalized. The cosine similarity between the current acoustic token and each of the historical acoustic tokens is calculated based on the normalized features, and is used as the acoustic similarity.

4. The method according to claim 1, characterized in that, The process, weighted by the acoustic similarity corresponding to each of the aforementioned historical acoustic tokens, includes: The acoustic similarity corresponding to each of the historical acoustic tokens is converted into continuous weights through a gating function. The continuous weights are positively correlated with the acoustic similarity, and the value range of the continuous weights is limited to between 0 and 1.

5. The method according to claim 1, characterized in that, The process of correcting the basic route probability distribution using the historical expert prior distribution includes: The overall continuity strength corresponding to the current acoustic token is calculated based on the acoustic similarity between the current acoustic token and each of the historical acoustic tokens. The correction magnitude of the historical expert prior distribution is controlled by the overall continuity strength, thereby correcting the basic route probability distribution.

6. The method according to claim 5, characterized in that, The process of correcting the basic route probability distribution by controlling the correction magnitude of the historical expert prior distribution with the overall continuity strength includes: The aforementioned historical expert prior distribution is transformed into a correction term in logarithmic space; The correction term is weighted by the overall continuity strength to obtain a weighted correction term. The weighted correction term is added to the routing score logits corresponding to the basic routing probability distribution to obtain the corrected routing score logits. After normalization, the corrected expert routing probability distribution is obtained.

7. The method according to claim 1, characterized in that, The number of hybrid expert coding layers is multiple, and the multiple hybrid expert coding layers are arranged sequentially in the depth direction of the speech encoder. Each expert in each hybrid expert coding layer is divided into an acoustic expert pool and a semantic expert pool. The experts in the acoustic expert pool are used to implement acoustic pattern coding, and the experts in the semantic expert pool are used to implement semantic pattern coding.

8. The method according to claim 7, characterized in that, After obtaining the corrected expert routing probability distribution for the current acoustic token, the following is also included: Determine the layer depth position of the current hybrid expert coding layer in the encoder depth direction; The layer depth prior is determined based on the layer depth position, and the layer depth prior is used to characterize the degree of bias of the current hybrid expert coding layer towards the acoustic expert pool and the semantic expert pool; The expert routing probability distribution of the current acoustic token is further modified based on the layer depth prior to obtain the final modified expert routing probability distribution.

9. The method according to claim 8, characterized in that, The process of determining the layer depth prior based on the layer depth position includes: A layer depth coefficient is calculated based on the layer depth position, and the layer depth coefficient is positively correlated with the depth of the current coding layer; The first deep prior corresponding to the experts in the acoustic expert pool and the second deep prior corresponding to the experts in the semantic expert pool are determined based on the depth coefficient. Wherein, when the layer depth coefficient is less than the first threshold, the correction magnitude of the first layer depth prior to the expert routing probability is greater than that of the second layer depth prior; when the layer depth coefficient is greater than the second threshold, the correction magnitude of the second layer depth prior to the expert routing probability is greater than that of the first layer depth prior, and the second threshold is greater than the first threshold.

10. The method according to claim 7, characterized in that, The correction strength when using the historical expert prior distribution to correct the basic route probability distribution varies with the depth of the hybrid expert coding layer. The correction strength of the hybrid expert coding layer located at the first depth is greater than that of the hybrid expert coding layer located at the second depth, and the first depth is shallower than the second depth.

11. The method according to claim 7, characterized in that, The number of experts in the semantic expert pool is greater than or equal to the number of experts in the acoustic expert pool.

12. The method according to claim 9, characterized in that, The loss function during the training phase of the speech encoder includes the load balancing loss of each hybrid expert coding layer. Specifically, for hybrid expert coding layers with a layer depth coefficient less than a first threshold, the load balancing loss within the acoustic expert pool is calculated. For hybrid expert coding layers with a layer depth coefficient greater than a second threshold, the load balancing loss within the semantic expert pool is calculated. For hybrid expert coding layers with a layer depth coefficient between the first and second thresholds, the load balancing losses of the acoustic expert pool and the semantic expert pool are weighted and summed according to the first and second layer depth priors.

13. The method according to claim 1, characterized in that, The speech encoder and speech decoder together form a speech recognition model. The total loss function during the training phase of the speech recognition model includes speech recognition loss, acoustic continuity loss, and expert load balancing loss. The acoustic continuity loss is used to constrain the degree of difference in the expert routing probability distributions of adjacent acoustic tokens, using the acoustic similarity between them as the weight.

14. The method according to any one of claims 1-13, characterized in that, The process of inputting the current acoustic token into each activation expert for feature encoding includes: Reorder all acoustic tokens within the batch containing the current acoustic token into contiguous memory groups according to the activation expert's number; Based on the acoustic tokens corresponding to each expert, feature encoding calculations for each expert are performed in parallel through group matrix multiplication.

15. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the various steps of the speech coding method as described in any one of claims 1 to 14.

Images