Speech enhancement method and system based on large language model semantic prior guidance

Abstract

The application provides a speech enhancement method and system based on large language model semantic prior guidance, and relates to the technical field of speech enhancement, wherein the method comprises: obtaining a noisy speech signal, combining a large language model to obtain an aligned semantic embedding aligned with an acoustic feature time; obtaining an environmental compensation parameter by associating and matching the aligned semantic embedding in an acoustic knowledge graph; generating an initial modulation parameter according to the aligned semantic embedding, correcting the initial modulation parameter by using the environmental compensation parameter to obtain a corrected modulation parameter, and performing feature modulation processing on the acoustic feature by using the corrected modulation parameter to obtain a modulated acoustic feature; performing context modeling processing on the modulated acoustic feature to generate an acoustic mask, and performing masking processing on the acoustic feature by using the acoustic mask to output an enhanced speech signal. The application improves the intelligibility and clarity of the speech signal in a complex acoustic environment, especially in a sudden noise scene.

Description

Technical Field

[0001] This application relates to the field of speech enhancement technology, and in particular to a speech enhancement method and system based on semantic prior guidance from a large language model. Background Technology

[0002] Speech enhancement technology aims to recover clean speech from noisy speech signals. It is a key preprocessing step in fields such as voice communication, human-computer interaction, and intelligent hearing devices. With the rapid development of deep learning technology, neural network-based speech enhancement methods are constantly improving their ability to model noise interference, providing a new technical path for high-fidelity speech pickup in complex acoustic environments.

[0003] Existing neural network-based speech enhancement methods typically extract acoustic features from noisy speech signals and suppress noise components using techniques such as masking estimation, spectral mapping, or time-frequency modeling to recover the target speech. These methods achieve good enhancement results under stationary noise or partially known noise conditions, but they primarily focus on acoustic distortion suppression and do not adequately consider the loss of semantic information in complex environments. Especially in sudden noise scenarios, noise is often characterized by its strong transient nature, concentrated energy, and random location, easily masking local speech segments and causing short-term loss of some semantic embeddings, resulting in incomplete semantic information or impaired contextual continuity. Most existing methods lack mechanisms for effectively compensating for masked semantic content; therefore, when facing sudden noise interference, it is often difficult to simultaneously achieve noise suppression and semantic integrity restoration.

[0004] To overcome the limitations of relying solely on acoustic feature enhancement, some methods have begun to introduce semantic prior information to assist enhancement processing using prior knowledge at the semantic embedding level. These methods typically extract semantic representations from noisy speech and combine semantic information with acoustic features to improve the intelligibility of the enhancement results. However, in this processing architecture, the correlation between semantic prior information and noise characteristics and speech distortion patterns under different acoustic environments remains insufficient, and the combination of semantic information and acoustic features is usually relatively fixed, making it difficult for the system to accurately distinguish semantic content from environmental interference when facing varying acoustic conditions. Especially when sudden noise causes local semantic cues to be masked, existing methods struggle to effectively compensate for damaged semantics based on context, ultimately affecting the naturalness, intelligibility, and semantic integrity of the enhanced speech. Therefore, existing technologies suffer from insufficient adaptability between semantic priors and environmental features, and limited ability to recover local semantic losses caused by sudden noise. Summary of the Invention

[0005] This application provides a speech enhancement method and system based on semantic prior guidance of a large language model, in order to solve the problems of poor clarity and low intelligibility of speech signals in complex acoustic environments, especially in sudden noise scenarios, in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a speech enhancement method based on semantic prior guidance from a large language model, comprising:

[0007] Noisy speech signals are acquired, and semantic embeddings are semantically modeled using a large language model. The noisy speech signals are enhanced and predicted using the context-based next word prediction capability to obtain aligned semantic embeddings that are time-aligned with acoustic features.

[0008] The environmental compensation parameters are obtained by associating and matching the aligned semantic embedding with the pre-constructed acoustic knowledge graph. The acoustic knowledge graph stores the mapping relationship between noise features and speech distortion patterns under different acoustic environments.

[0009] Initial modulation parameters are generated based on the aligned semantic embedding. The initial modulation parameters are then corrected using the environmental compensation parameters to obtain corrected modulation parameters. The acoustic features are then subjected to feature modulation processing using the corrected modulation parameters to obtain modulated acoustic features that fuse semantic priors and environmental compensation information.

[0010] The modulated acoustic features are subjected to context modeling to generate an acoustic mask, and the acoustic features are masked using the acoustic mask to output an enhanced speech signal.

[0011] Secondly, this application provides a speech enhancement system based on semantic prior guidance from a large language model, comprising:

[0012] The acquisition module is used to acquire noisy speech signals and perform semantic modeling on the semantic embedding by combining a large language model. It uses its context-based next-word prediction capability to enhance and predict the noisy speech signals to obtain aligned semantic embeddings that are time-aligned with acoustic features.

[0013] The matching module is used to perform association matching between the aligned semantic embedding and the pre-constructed acoustic knowledge graph to obtain environmental compensation parameters. The acoustic knowledge graph stores the mapping relationship between noise features and speech distortion patterns under different acoustic environments.

[0014] The generation module is used to generate initial modulation parameters based on the aligned semantic embedding, correct the initial modulation parameters using the environmental compensation parameters to obtain corrected modulation parameters, and perform feature modulation processing on the acoustic features using the corrected modulation parameters to obtain modulated acoustic features that fuse semantic prior and environmental compensation information.

[0015] The processing module is used to perform context modeling on the modulated acoustic features to generate an acoustic mask, and to use the acoustic mask to mask the acoustic features, thereby outputting an enhanced speech signal.

[0016] Thirdly, this application provides an electronic device, comprising:

[0017] Memory, used to store computer programs;

[0018] A processor, configured to implement the steps of the speech enhancement method based on semantic prior guidance of a large language model as described in the first aspect above when executing the computer program.

[0019] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the speech enhancement method based on semantic prior guidance of a large language model as described in the first aspect above.

[0020] The technical solution provided in this application has the following beneficial effects:

[0021] This application obtains a noisy speech signal and combines it with a large language model to obtain an aligned semantic embedding that is time-aligned with the acoustic features. This enables the semantic information to correspond precisely with the acoustic features in time, providing a joint feature basis for more accurate semantic expression in subsequent processing. Secondly, the aligned semantic embedding is matched with a pre-constructed acoustic knowledge graph to obtain environmental compensation parameters. This knowledge graph stores the mapping relationship between noise features and speech distortion patterns under different acoustic environments, thereby obtaining compensation information for the current acoustic environment.

[0022] Then, initial modulation parameters are generated based on the aligned semantic embedding and corrected using environmental compensation parameters to obtain corrected modulation parameters. The corrected modulation parameters are then used to perform feature modulation processing on the acoustic features to obtain modulated acoustic features that integrate semantic priors and environmental compensation information, enabling the acoustic features to adapt to different environmental changes under semantic guidance. Finally, context modeling is performed on the modulated acoustic features to generate an acoustic mask, and this acoustic mask is used to mask the acoustic features to output an enhanced speech signal, completing the entire process of recovering clean speech from noise interference.

[0023] Furthermore, this application extracts semantic embeddings and acoustic features from noisy speech signals. The semantic embeddings are then dimensionally adapted by a pre-projection network and input into a large language model. The large language model performs context prediction processing based on a self-attention mechanism to obtain context features. The context features are then mapped back to the speech semantic space by a post-projection network to obtain enhanced semantic embeddings. Finally, the enhanced semantic embeddings and acoustic features are temporally aligned to obtain aligned semantic embeddings. This process realizes the enhanced propagation of semantic information in the deep network and precise temporal alignment with acoustic features, providing a joint feature representation with richer semantic expression and more accurate temporal matching for subsequent processing. The context modeling capability of the large language model improves the expression quality of semantic embeddings, and temporal alignment ensures the accurate correspondence between semantic information and acoustic features at each moment, enabling semantic priors to more accurately guide subsequent speech enhancement processing.

[0024] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating a speech enhancement method based on semantic prior guidance from a large language model, provided for embodiments of this application;

[0027] Figure 2 A schematic diagram illustrating a specific implementation of speech semantic embedding enhancement and time alignment based on a large language model, provided in an embodiment of this application.

[0028] Figure 3 This is a schematic diagram of the structure of a speech enhancement system based on semantic prior guidance of a large language model, provided as an embodiment of this application.

[0029] Figure 4 This is a schematic diagram of a practical scenario of this application: speech enhancement in a sudden noise masking scenario. Detailed Implementation

[0030] To address the problems of existing technologies, this application proposes a speech enhancement method based on semantic prior guidance from a large language model. The core of this method lies in establishing a dynamic association mechanism between semantic information and environmental characteristics by introducing an acoustic knowledge graph. First, the large language model's context-based next-word prediction capability is used to enhance and predict the completion of the noisy speech signal, resulting in an aligned semantic embedding that is time-aligned with acoustic features. This semantic embedding is then used for association matching within a pre-constructed acoustic knowledge graph to obtain environmental compensation parameters specific to the current acoustic environment. This knowledge graph stores noise features and speech distortion under different acoustic environments. The method first establishes the mapping relationship of the true mode, then generates initial modulation parameters based on the aligned semantic embedding and corrects them using environmental compensation parameters. Subsequently, it modulates the acoustic features to obtain modulated acoustic features that fuse semantic prior and environmental compensation information. Finally, it generates an acoustic mask through context modeling and masks the acoustic features to output an enhanced speech signal. This method enables the semantic prior to adapt to different acoustic conditions by introducing environmental compensation parameters, fundamentally solving the problem of insufficient semantic guidance and environmental characteristic adaptation in existing technologies. It improves the intelligibility of enhanced speech in different noise environments while maintaining the naturalness of the speech.

[0031] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] The core of this application is to provide a speech enhancement method based on semantic prior guidance from a large language model. A flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0033] Step 101: Acquire noisy speech signals and perform semantic modeling on the semantic embedding using a large language model. Utilize its context-based next-word prediction capability to enhance and predict the noisy speech signals, thereby obtaining an aligned semantic embedding that is time-aligned with the acoustic features.

[0034] Among them, noisy speech signal refers to the original audio waveform data containing the target speech component and background noise interference, which is collected by the sound pickup device in the actual acoustic environment. This signal is usually stored in the form of time-domain sampling points. Acoustic features refer to the parameterized representations extracted from the noisy speech signal to characterize the acoustic properties of the speech, such as filter bank energy or Mel frequency cepstral coefficients. These features can reflect the changes in the spectral envelope of the speech signal over time. Aligned semantic embedding refers to the semantic vector representation that has been enhanced by a large language model and is precisely corresponding to the acoustic features in the time dimension. This embedding contains the semantic information of the speech content and maintains a synchronous correspondence with the acoustic features in each time frame, providing a joint feature basis for temporal matching for subsequent semantic guidance enhancement processing.

[0035] The large language model adopts a Transformer-based decoder architecture, which consists of an input embedding layer, a positional encoding layer, and multiple stacked decoder layers. Each decoder layer contains a multi-head self-attention sub-layer, a feedforward network sub-layer, and two normalization sub-layers. The multi-head self-attention sub-layer uses eight attention heads, each with a dimension of 96, keeping the total dimension at 768. The feedforward network sub-layer uses a two-layer fully connected structure, with the intermediate layer dimension expanded to 3072 and using the GELU activation function. Each sub-layer is followed by a combination of residual connections and layer normalization.

[0036] The training process of this large language model is divided into two stages: pre-training and fine-tuning. In the pre-training stage, a large-scale general corpus, such as text data containing hundreds of billions of words, is used. The model is trained through an autoregressive language modeling task, which is to predict the probability of the next word given a sequence of preceding words. The loss function is cross-entropy loss, the optimizer is AdamW optimizer, the learning rate is set to 3e-4 and cosine annealing is used to adjust the learning rate, the batch size is set to 512 and the number of training steps is 500,000.

[0037] In the fine-tuning stage, domain-specific speech semantic alignment data is used. This data includes text descriptions corresponding to speech signals and semantic embedding vectors. Training is performed through a contrastive learning task, which brings positive sample pairs with the same speech content closer in the semantic space while pushing negative sample pairs further apart. The loss function is InfoNCE loss, and the optimizer is still AdamW optimizer. The learning rate is reduced to 5e-5, the batch size is set to 64, and the number of training rounds is 10. Through the above training process, the large language model can effectively understand the semantic content of speech signals and generate context-enhanced semantic representations.

[0038] It should be noted that the above structure is exemplary. This application does not impose specific limitations on the internal structure design of the large language model, and corresponding settings can be made according to the actual situation.

[0039] In this embodiment, step 101 includes the following process, such as... Figure 2 As shown:

[0040] Step 1011: Extract semantic embeddings and acoustic features from the noisy speech signal.

[0041] In step 1011, semantic embedding refers to the vectorized representation of high-level semantic information of speech content extracted from noisy speech signals. This embedding is usually obtained through a pre-trained semantic coding model, which can capture semantic content such as words and sentences in the speech signal.

[0042] In this embodiment, a segment of time-domain waveform data is first acquired using a microphone array as a noisy speech signal. This noisy speech signal is then input into two parallel processing branches. In the first branch, the noisy speech signal is processed using a pre-trained semantic coding model. This semantic coding model is built on a deep neural network and trained using a large amount of paired data of clean speech and corresponding text. It can map the input speech signal into a fixed-dimensional semantic embedding vector. Each element in the semantic embedding vector takes a value between -1 and 1 to represent the position of the speech content in the semantic space.

[0043] In the second branch, the noisy speech signal is processed by the acoustic feature extraction module to perform framing, windowing, and Fourier transform to obtain the spectral data of each frame. Then, the energy features of the filter bank are calculated. These energy features of the filter bank are arranged in chronological order to form an acoustic feature sequence. Each frame of the acoustic feature sequence corresponds to a time segment in the original speech signal to describe the change of the spectral envelope of the speech over time.

[0044] Step 1012: Input the semantic embedding into the pre-projection network, and the pre-projection network maps the dimension of the semantic embedding to match the input dimension of the large language model to obtain the projected semantic embedding.

[0045] In step 1012, the pre-projection network refers to a neural network module composed of fully connected layers, which is used to transform the dimension of the input semantic embedding vector to match the input dimension expected by the large language model; the post-projection semantic embedding refers to the semantic embedding after dimension transformation, whose dimension is consistent with the input dimension of the large language model, so as to facilitate subsequent input into the large language model for processing.

[0046] This application does not impose specific limitations on the type of front projection network, the structural design of its internal structure, the parameter design, the training process, etc., and can be set accordingly based on the actual situation.

[0047] In this embodiment, the semantic embedding extracted in step 1011 is input to a pre-projection network. The pre-projection network contains a fully connected layer. The number of input neurons in the fully connected layer is the same as the dimension of the semantic embedding, and the number of output neurons is the same as the input dimension required by the large language model. The original semantic embedding vector is mapped to a dimension-adapted projected semantic embedding through the linear transformation of the fully connected layer. The projected semantic embedding retains the core semantic information of the original semantic embedding, but the dimensional form meets the input requirements of the large language model, ensuring that the semantic information can be effectively received and processed by the large language model.

[0048] Step 1013: Input the projected semantic embedding into the large language model, and the large language model performs context prediction processing based on the self-attention mechanism to obtain context features.

[0049] Among them, contextual features refer to the enhanced representation obtained by processing the projected semantic embedding with a large language model, which incorporates contextual information. This feature not only contains the semantic information at the current moment but also integrates the semantic connections between the preceding and following texts, making the semantic expression richer and more accurate.

[0050] Step 1013 may specifically include the following steps:

[0051] A1: The projected semantic embedding is input into a large language model. The projected semantic embedding and the positional encoding are superimposed through the positional encoding layer of the large language model to obtain an initial embedding sequence. The initial embedding sequence is used to represent speech semantic features with temporal information.

[0052] In step A1, position encoding refers to the vector representation used to add position information to the input sequence. Since the self-attention mechanism itself does not have temporal awareness, position encoding needs to be explicitly added so that the model can distinguish elements at different positions in the sequence. The initial embedding sequence refers to the sequence formed by superimposing the projected semantic embedding and the position encoding. Each element in this sequence contains both the original semantic information and the position information of the element in the sequence.

[0053] In this embodiment, the projected semantic embedding obtained in step 1012 is input into the large language model. First, the projected semantic embedding is sent to the positional encoding layer. The positional encoding layer generates a set of positional encoding vectors with the same dimension as the projected semantic embedding. These positional encoding vectors are arranged in the order of the input sequence, and each position corresponds to a unique positional encoding. The positional encoding layer adds the projected semantic embedding to the positional encoding of the corresponding position element by element to obtain an initial embedding sequence. Each vector in the initial embedding sequence carries both semantic content and temporal positional information, providing a basic input representation for the subsequent self-attention processing of the large language model.

[0054] A2: The initial embedding sequence is processed by multiple decoder layers of the large language model, wherein each decoder layer performs a linear transformation on the initial embedding sequence through an internal multi-head self-attention sublayer to generate a query vector, a key vector, and a value vector.

[0055] In step A2, the decoder layer is the core component of the large language model, which is usually composed of a multi-head self-attention sub-layer, a feedforward network sub-layer, and a normalization sub-layer stacked together. The multi-head self-attention sub-layer maps the input sequence into a query vector, a key vector, and a value vector through linear transformation. These three vectors are the basis for self-attention calculation. The query vector is used to represent the information demand at the current attention position, the key vector is used to represent the information index provided by each position, and the value vector is used to represent the actual information content at each position.

[0056] In this embodiment, the initial embedding sequence obtained in step A1 is input to the first decoder layer of the large language model. Inside this decoder layer, the initial embedding sequence is first fed into a multi-head self-attention sub-layer. The multi-head self-attention sub-layer contains multiple sets of independent linear transformation matrices. For each set, each vector in the initial embedding sequence is multiplied by the corresponding query weight matrix, key weight matrix, and value weight matrix to generate the corresponding query vector, key vector, and value vector. Due to the use of a multi-head mechanism, multiple heads can focus on information from different representation subspaces, enabling the model to capture richer dependencies. After sequential processing by multiple decoder layers, each layer further extracts and integrates contextual information based on the output of the previous layer.

[0057] A3: The similarity between the query vector and the key vector is calculated through the weight calculation sublayer inside the decoder layer to obtain the attention weight. The value vector is then weighted and aggregated using the attention weight to obtain an aggregated vector. The aggregated vector is used to represent the semantic information of the current moment predicted based on the historical context.

[0058] In step A3, attention weight refers to the normalized coefficient obtained by calculating the similarity between the query vector and the key vector, which is used to measure the importance of each position in the sequence to the current position of interest; the aggregation vector refers to the vector obtained by weighted summation of the value vector according to the attention weight. This vector integrates the information of each position in the sequence, and its weight is determined by the attention weight, so that the model can focus on the contextual information related to the current query.

[0059] In this embodiment, the weight calculation sublayer within the multi-head self-attention sublayer first calculates the similarity between the query vector at the current time step and the key vectors at all times in the sequence for each attention head. Typically, a dot product operation is used to obtain the original attention score. This original attention score is then scaled and normalized using the Softmax function to obtain attention weights. The sum of these weights is 1, representing the contribution ratio of each position to the current time step. Then, these attention weights are used to perform a weighted summation of the value vectors at all times in the sequence, resulting in the aggregated vector at the current time step under the current attention head. This process is repeated for each of the eight attention heads to obtain eight aggregated vectors. Finally, these eight aggregated vectors are concatenated and fused using a linear transformation to obtain the aggregated vector output by the decoder layer. This aggregated vector integrates contextual information from different aspects attended by multiple heads and is used to characterize the semantic information of the current time step predicted based on historical context.

[0060] A4: The aggregation vector is added to the initial embedding sequence and then normalized through the normalization sub-layer inside the decoder layer to obtain the first intermediate sequence.

[0061] In step A4, the normalization sublayer refers to the network module that performs layer normalization on the input data. By calculating and adjusting the mean and variance of the feature dimensions, the data distribution becomes more stable, which is beneficial for model training and convergence. The first intermediate sequence refers to the sequence obtained after residual connection and layer normalization. This sequence retains the original information of the initial embedding and incorporates the contextual information extracted by the self-attention mechanism. At the same time, the data distribution is more stable and suitable for subsequent processing.

[0062] In this embodiment, within the decoder layer, the aggregated vector obtained in step A3 is first residually concatenated with the vector at the corresponding time step in the initial embedding sequence obtained in step A1, i.e., element-wise addition. This operation helps alleviate the gradient vanishing problem in deep networks. Subsequently, the result of addition is input to the normalization sublayer. The normalization sublayer calculates the mean and variance of the input data in the feature dimension and then performs normalization processing, so that the features of each sample have zero mean and unit variance. Finally, a linear transformation is performed through learnable scaling and translation parameters to obtain the first intermediate sequence. This first intermediate sequence serves as the input to the subsequent feedforward network sublayer, preserving the information after residual concatenation and stabilizing the data distribution through normalization.

[0063] A5: The first intermediate sequence is nonlinearly transformed by the feedforward network sublayer inside the decoder layer to obtain the second intermediate sequence, and the second intermediate sequence is used as the output sequence of the current decoder layer.

[0064] In step A5, the feedforward sublayer refers to a neural network module consisting of two fully connected layers and a nonlinear activation function, used to perform nonlinear transformations on the input data to enhance the model's expressive power; the second intermediate sequence refers to the sequence obtained after the first intermediate sequence has been processed by the feedforward sublayer. Each vector in this sequence has undergone a deep nonlinear transformation, containing richer semantic features. For example, vectors that originally mainly represent local semantic information or contextual fragment information can form more abstract and complex semantic feature representations after the nonlinear transformation of the feedforward sublayer, thereby enhancing the model's ability to express semantic relationships and contextual dependencies in speech content.

[0065] In this embodiment, the first intermediate sequence obtained in step A4 is input to the feedforward sub-layer inside the decoder layer. The feedforward sub-layer first expands the dimension of the input vector to a higher dimension through the first fully connected layer, then transforms it through a non-linear activation function such as GELU, and then restores the dimension to the original input dimension through the second fully connected layer. This non-linear transformation process enables the model to learn more complex feature representations. After processing by the feedforward sub-layer, the second intermediate sequence is obtained. The second intermediate sequence is passed to the next decoder layer as the input as the output sequence of the current decoder layer, or as the final output of the large language model if the current is the last decoder layer.

[0066] A6: The output sequence of the last decoder layer is used as contextual features through the output layer of the large language model.

[0067] In this embodiment, the large language model includes multiple stacked decoder layers, such as 12 layers. Steps A1 to A5 describe the processing of a decoder layer. The initial embedding sequence is processed by each decoder layer in sequence, and the output of each layer is used as the input of the next layer. After the last decoder layer is processed, the sequence output by the last decoder layer is directly used as the output of the large language model. This sequence is the context feature. Each vector in the context feature contains the semantic information at the corresponding time and the context dependency from other times in the sequence, making the semantic expression more complete and accurate.

[0068] Step 1014: Input the context features into the post-projection network, and the post-projection network maps the context features back to the speech semantic space to obtain the enhanced semantic embedding.

[0069] In step 1014, the post-projection network refers to a fully connected layer that transforms the 768-dimensional context feature vector back into a 512-dimensional semantic embedding vector. For example, the 768-dimensional vector after large language model enhancement is reduced to 512 dimensions through linear transformation. The enhanced semantic embedding refers to the 512-dimensional vector obtained after processing by the post-projection network. This vector contains richer contextual information than the original 512-dimensional semantic embedding. For example, the original semantic embedding may only recognize the speech content as "disconnect the circuit breaker", while the enhanced semantic embedding, combined with the context, can more accurately determine whether the instruction is issued in normal operation or in a fault handling scenario.

[0070] In this embodiment, the contextual features obtained in step 1013 are input into a post-projection network. This post-projection network contains a fully connected layer whose input neurons have the same number as the output dimension of the large language model, and whose output neurons have the same number as the dimension of the semantic embedding in step 1011. Through the linear transformation of this fully connected layer, each vector in the contextual features is mapped back to the original speech semantic space dimension to obtain an enhanced semantic embedding sequence. This enhanced semantic embedding is consistent with the original semantic embedding in terms of dimensional form, but incorporates contextual information in terms of content, thus improving the semantic quality.

[0071] Step 1015: Perform time alignment processing on the enhanced semantic embedding and the acoustic features to obtain the aligned semantic embedding.

[0072] In this embodiment, the enhanced semantic embedding sequence obtained in step 1014 and the acoustic feature sequence extracted in step 1011 are first obtained. Each frame of the enhanced semantic embedding sequence corresponds to the semantic representation at a certain moment, and each frame of the acoustic feature sequence corresponds to the acoustic representation at the same moment. Since both originate from the same noisy speech signal and use the same framing parameters, they naturally have a correspondence in the time dimension. By checking the consistency of the frame index, the enhanced semantic embedding sequence and the acoustic feature sequence are paired in the same frame order to ensure that the enhanced semantic embedding of each frame corresponds to the acoustic feature of the same frame. After this verification and pairing process, the semantic embedding is aligned. The embedding sequence and the acoustic feature sequence are completely aligned in time, providing a joint feature basis for temporal matching for subsequent semantic guidance enhancement processing.

[0073] This application combines the contextual enhancement capabilities of a large language model with the temporal representation of acoustic features to obtain an enhanced semantic embedding that is precisely time-aligned with the acoustic features. This enables semantic information to maintain temporal synchronization with the acoustic features, providing a feature foundation for subsequent semantic guidance enhancement processing that is richer in semantic expression and more accurate in temporal matching.

[0074] Step 102: Based on the alignment of the semantic embedding and the pre-constructed acoustic knowledge graph, environmental compensation parameters are obtained by association matching. The acoustic knowledge graph stores the mapping relationship between noise features and speech distortion patterns under different acoustic environments.

[0075] Among them, the acoustic knowledge graph refers to a knowledge base organized in the form of a graph structure. This knowledge base contains multiple nodes, each corresponding to a typical acoustic environment, such as a train station waiting hall environment, a subway car environment, an outdoor plaza environment, etc. Each node stores the node feature vector, noise features, and speech distortion mode of that environment. The environment compensation parameter refers to the adjustment parameters used to adapt the speech enhancement process to the environment. This parameter includes the noise suppression intensity and speech distortion compensation coefficient for the current acoustic environment, so that the subsequent speech enhancement processing can be adjusted according to the actual environmental characteristics.

[0076] In this embodiment, step 102 includes the following process:

[0077] Step 1021: Obtain supplementary environmental data corresponding to the current time, extract auxiliary feature values ​​from the supplementary environmental data, and encode the auxiliary feature values ​​into an auxiliary feature vector.

[0078] In step 1021, supplementary environmental data refers to auxiliary information related to the current acquisition environment that can be obtained in addition to the speech signal itself, such as geographical location, acquisition time period, and seasonal information. This data can provide additional discrimination basis for acoustic environment identification. Auxiliary feature values ​​refer to specific numerical values ​​extracted from the supplementary environmental data, such as encoding geographical location as latitude and longitude coordinates and encoding time period as hourly values. Auxiliary feature vectors refer to the vector representation obtained after numerically encoding these auxiliary feature values, which are used to perform environmental matching in conjunction with acoustic features.

[0079] In this embodiment, supplementary environmental data corresponding to the current time is acquired. For example, the current geographical location information is obtained through a positioning module, and the current acquisition time is obtained through a system clock; these data are used as supplementary environmental data. Auxiliary feature values ​​are extracted from this data, such as converting the geographical location information into latitude and longitude coordinates, and converting the acquisition time into hourly and monthly values. These auxiliary feature values ​​are then encoded to form a fixed-dimensional auxiliary feature vector, which is used to supplement the auxiliary conditions describing the current acoustic environment.

[0080] In practical applications, the current geographical location is obtained as 39.9 degrees north latitude and 116.4 degrees east longitude, and the current collection time is 3 pm; this information is used as supplementary environmental data. Latitude and longitude are extracted from the geographical location as two auxiliary feature values, and the hour value 15 is extracted from the collection time as a third auxiliary feature value; these three values ​​are encoded into a three-dimensional auxiliary feature vector [39.9, 116.4, 15].

[0081] It should be understood that the above-mentioned longitude and latitude values, as well as other related values ​​given later, are all virtual and hypothetical geographical information, and are not real geographical information.

[0082] Step 1022: Extract environmental feature vectors from the aligned semantic embedding, the environmental feature vectors being used to characterize the noise characteristics of the current acoustic environment.

[0083] In step 1022, the environmental feature vector refers to the vector representation extracted from the aligned semantic embedding that reflects the noise characteristics of the current acoustic environment. Since the aligned semantic embedding not only contains the semantic information of the speech content but also implicitly contains the distribution characteristics of the background noise, the components related to the noise environment can be separated from it through specific feature extraction operations. For example, the type and intensity of noise can be inferred by statistically analyzing the energy distribution of each dimension in the semantic embedding.

[0084] In this embodiment, the aligned semantic embedding sequence obtained in step 101 is first obtained, which contains semantic embedding vectors at multiple time points. Then, statistical feature extraction is performed on these semantic embedding vectors, such as calculating the mean vector and variance vector of the semantic embedding vectors at all time points, or processing the semantic embedding sequence through a pre-trained environment recognition network to extract a fixed-dimensional environment feature vector. This environment feature vector integrates information such as the energy distribution and spectral characteristics of background noise in the current acoustic environment and is used to characterize the overall noise characteristics of the current acoustic environment.

[0085] In practical applications, taking the aligned semantic embeddings obtained in step 101 as an example, the semantic embedding vector dimension at each time step is 512. By calculating the mean of these 299 vectors in the time dimension, a 512-dimensional mean vector is obtained, and the variance is calculated to obtain a 512-dimensional variance vector. The mean vector and the variance vector are concatenated to obtain a 1024-dimensional environmental feature vector, which is used to characterize the noise statistical characteristics of the current acoustic environment.

[0086] Step 1023: Transform the dimension of the environmental feature vector to match the dimension of the node features in the acoustic knowledge graph to obtain the mapped feature vector, and concatenate the mapped feature vector with the auxiliary feature vector to obtain the joint environmental feature vector.

[0087] In step 1023, the mapped feature vector refers to the vector whose dimension is consistent with the feature dimension of the acoustic knowledge graph node after linear transformation of the environmental feature vector. For example, the 1024-dimensional environmental feature vector is converted into a 256-dimensional vector through a fully connected layer. The joint environmental feature vector refers to the comprehensive feature representation that integrates the statistical characteristics of the acoustic environment and supplementary environmental information, which is used for subsequent matching with nodes in the acoustic knowledge graph.

[0088] In this embodiment, the environmental feature vector extracted in step 1022 is first obtained, with a dimension of D1. Simultaneously, the dimension of the node feature vector stored in each node of the acoustic knowledge graph is obtained, denoted as D2. A fully connected layer is used to convert the dimension of the environmental feature vector from D1 to D2, resulting in a mapped feature vector. This mapped feature vector retains the core information of the environmental features but its dimensional form is consistent with the node features of the knowledge graph. Then, the auxiliary feature vector obtained in step 1021 is obtained, and this auxiliary feature vector is concatenated with the mapped feature vector, i.e., a new vector is formed by concatenating the auxiliary feature vector first and the mapped feature vector last. This vector is the joint environmental feature vector, which integrates environmental statistical characteristics and supplementary environmental information to more accurately match the acoustic environment type.

[0089] In practical applications, the environmental feature vector extracted in step 1022 has a dimension of 1024, and the node feature vector in the acoustic knowledge graph has a dimension of 256. The environmental feature vector is linearly transformed by a fully connected layer with an input dimension of 1024 and an output dimension of 256 to obtain a 256-dimensional mapped feature vector. The auxiliary feature vector obtained in step 1021 has a dimension of 3. The 3-dimensional auxiliary feature vector is concatenated with the 256-dimensional mapped feature vector to obtain a 259-dimensional joint environmental feature vector.

[0090] Step 1024: Compare the joint environment feature vector with the node feature vector stored in each node of the acoustic knowledge graph, and determine the node that matches the joint environment feature vector as the matching node. The acoustic knowledge graph contains multiple nodes, and each node stores a node feature vector, noise feature and speech distortion mode corresponding to an acoustic environment.

[0091] Among them, the node feature vector refers to the feature vector stored in each node in the acoustic knowledge graph, which is used to characterize the type of acoustic environment corresponding to the node. For example, the node feature vector corresponding to the waiting hall of the train station is [0.23, -0.15, 0.67, ...] with a total of 256 dimensions; the matching node refers to the node that is closest to the current joint environment feature vector determined by similarity calculation. The noise features and speech distortion patterns stored in the node are considered to be most suitable for the current acoustic environment.

[0092] Step 1024 may specifically include the following steps:

[0093] B1: Calculate the Euclidean distance between the joint environment feature vector and the node feature vector stored in each node of the acoustic knowledge graph to obtain the distance value corresponding to each node.

[0094] In step B1, Euclidean distance is a metric that measures the straight-line distance between two vectors in the vector space; the smaller the distance value, the more similar the two vectors are, that is, the closer the current joint environmental feature vector is to the acoustic environment type represented by the node.

[0095] In this embodiment, the joint environment feature vector obtained in step 1023 is first obtained, denoted as V, with dimension D; simultaneously, the number of all nodes in the acoustic knowledge graph is obtained, denoted as N, and the node feature vector stored in each node i is denoted as... Where i ranges from 1 to N, it should be noted that, in order for the joint environment feature vector to be effectively compared with the node feature vectors in the knowledge graph, the dimension of the node feature vectors of each node in the acoustic knowledge graph needs to be pre-expanded to match the dimension D of the joint environment feature vector. This expansion can be achieved by uniformly adding learnable embedding components with the same dimension as the auxiliary feature vectors to all nodes, or by dynamically concatenating the auxiliary feature vectors with the original node feature vectors before matching calculation. After dimension unification, The dimension of V is the same as that of D; then for each node i, the joint environment feature vector V and the node feature vector are calculated. The Euclidean distance between them is calculated using the following formula: ,in Represents the joint environmental feature vector. The system consists of N distance components; each distance value corresponds to a node and is used to represent the similarity between the current environment and various known acoustic environment types.

[0096] In practical applications, taking a joint environmental feature vector V as a 259-dimensional vector as an example; the acoustic knowledge graph has 50 nodes, and each node's feature vector is 256-dimensional; for the first node, its node feature vector U1, V is calculated with... The squares of the differences in each dimension yield 259 squared difference values. These squared difference values ​​are summed to obtain a sum S1. The square root of S1 is then calculated to obtain the first distance value. This operation is repeated for all 50 nodes to obtain 50 distance values.

[0097] B2: Select the smallest distance value from all the distance values ​​as the minimum distance value, and select the node corresponding to the minimum distance value as a candidate node.

[0098] In step B2, the minimum distance value refers to the minimum value among the N distance values ​​calculated in step B1. The node corresponding to the minimum value is closest to the current joint environment feature vector in the vector space. The candidate node refers to the node corresponding to the minimum distance value. This node is the most likely matching object of the current acoustic environment, but it still needs to be further verified whether the matching degree is high enough.

[0099] In this embodiment of the application, the smallest distance value is first found from the N distance values ​​calculated in step B1 using a sorting algorithm, and this distance value is denoted as... Then determine the relationship with the The corresponding node number is used to identify the node as a candidate node. The acoustic environment type represented by the candidate node is the closest to the current environment in the feature space.

[0100] In practical applications, taking the 50 distance values ​​calculated in step B1 as an example, these distance values ​​are 5.2, 6.8, 3.4, 7.1... By comparison, it is found that the smallest distance value is 3.4, and the node corresponding to 3.4 is the 3rd node; the 3rd node is determined as a candidate node.

[0101] B3: Compare the minimum distance value with a preset matching threshold.

[0102] In step B3, the matching threshold refers to a pre-set distance limit value used to determine whether two vectors are sufficiently similar. This threshold is set according to the coverage and accuracy requirements of the acoustic knowledge graph, for example, it is set to 5.0. When the minimum distance value is less than the threshold, it is considered that the candidate node is sufficiently matched with the current environment. When the minimum distance value is greater than or equal to the threshold, it is considered that the current environment does not have sufficiently similar known types in the knowledge graph.

[0103] In this embodiment of the application, a preset matching threshold is obtained and denoted as T, which is a value determined in advance through experiments; the minimum distance value obtained in step B2 is... Compare with the matching threshold T to determine Whether it is less than T, the comparison result will determine the subsequent processing branch.

[0104] In practical applications, the preset matching threshold T is set to 5.0, and the minimum distance value obtained in step B2 is... The value is 3.4, and the comparison result is that 3.4 is less than 5.0.

[0105] B4: When the minimum distance value is less than the matching threshold, the candidate node is determined as the matching node.

[0106] In this embodiment of the application, when the comparison result of step B3 is that the minimum distance value is less than the matching threshold, it indicates that the candidate node is sufficiently similar to the current joint environment feature vector, and the acoustic environment type represented by the candidate node can well describe the current environment; therefore, the candidate node determined in step B2 is directly used as the matching node for subsequent parameter reading.

[0107] In practical applications, since the minimum distance value of 3.4 is less than the matching threshold of 5.0, the third node is determined as the matching node.

[0108] B5: When the minimum distance value is greater than or equal to the matching threshold, the joint environment feature vector is used as the new node feature vector, the new node feature vector is added to the acoustic knowledge graph as a new node, and a preset initial noise feature and initial speech distortion mode are assigned to the new node, and the new node is determined as a matching node.

[0109] In step B5, the new node feature vector refers to the current joint environment feature vector itself. Since the distance between this vector and all known nodes exceeds the matching threshold, it indicates that the current environment is a new acoustic environment type that has not yet been included in the knowledge graph. The new node refers to the newly added node whose new node feature vector is added as a node feature to the acoustic knowledge graph. The initial noise feature refers to the default noise feature pre-set for the new node, such as a general background noise spectrum. The initial speech distortion mode refers to the default distortion mode pre-set for the new node, such as including default time-domain distortion parameters and frequency-domain attenuation coefficients.

[0110] In this embodiment, when the comparison result of step B3 is that the minimum distance value is greater than or equal to the matching threshold, it indicates that the current acoustic environment does not have enough similar known types in the knowledge graph, and the environment needs to be added to the knowledge graph as a new type. First, the joint environment feature vector obtained in step 1023 is used as the new node feature vector. Then, a new node is created in the acoustic knowledge graph, and the new node feature vector is stored in the new node. At the same time, a preset initial noise feature and an initial speech distortion mode are assigned to the new node. These initial values ​​are set based on general environmental experience. Finally, the newly created node is determined as the matching node for subsequent parameter reading.

[0111] In practical applications, assuming the minimum distance value is 6.2 and the matching threshold is 5.0, the comparison result is that 6.2 is greater than 5.0. Then, the joint environment feature vector obtained in step 1023 is used as the new node feature vector, and the 51st node is created in the acoustic knowledge graph. The new node feature vector is stored in the node, and a preset initial noise feature, such as a general Gaussian white noise spectrum, and a preset initial speech distortion mode, such as including the default time-domain distortion parameter of 0.05 and the frequency-domain attenuation coefficient of 0.8, are assigned to the node. Then, the 51st node is determined as the matching node.

[0112] Step 1025: Read the noise features and the speech distortion mode from the matching node. The speech distortion mode includes multiple distortion types and time-domain distortion parameters and frequency-domain attenuation coefficients corresponding to each distortion type.

[0113] In step 1025, the speech distortion mode refers to the data set stored in the matching node that describes the distortion effect of the acoustic environment on the pure speech signal. The mode includes multiple distortion types such as reverberation distortion, frequency-selective attenuation distortion, nonlinear distortion, etc. Each distortion type corresponds to a time-domain distortion parameter such as reverberation time T60, and a frequency-domain attenuation coefficient such as the attenuation ratio of different frequency bands.

[0114] In this embodiment, firstly, based on the matching node determined in step 1024, the data stored in that node in the acoustic knowledge graph is accessed; pre-stored noise features are read from that node, which typically represent noise energy in different frequency bands in vector form; simultaneously, speech distortion patterns are read from that node, which contain a list, each item in the list corresponding to a distortion type, and each distortion type entry contains the time-domain distortion parameter and frequency-domain attenuation coefficient of that type; these read noise features and speech distortion patterns are used as input data for subsequently generating environmental compensation parameters.

[0115] In practical applications, taking the third node determined in step 1024 as an example, the noise feature read from this node is a 256-dimensional vector, which represents the noise energy distribution in each frequency band from 20 Hz to 8000 Hz; the read speech distortion pattern includes three distortion types. The first distortion type is reverberation distortion, with a time-domain distortion parameter of 1.2 seconds and a frequency-domain attenuation coefficient of uniform attenuation. The second distortion type is high-frequency attenuation distortion, with a time-domain distortion parameter of no frequency domain attenuation coefficient of 0.3 for high-frequency band attenuation. The third distortion type is low-frequency enhancement distortion, with a time-domain distortion parameter of no frequency domain attenuation coefficient of 1.5 for low-frequency band gain.

[0116] Step 1026: Input the noise characteristics, the time-domain distortion parameters, and the frequency-domain attenuation coefficient into the conditional random field for structured prediction to obtain the environmental compensation parameters.

[0117] In step 1026, the Conditional Random Field (CRF) is a probabilistic graphical model for structured prediction. This model can consider the dependencies between input variables and output structured prediction results. In this application, it is used to integrate noise features and multiple distortion parameters into a unified environmental compensation parameter.

[0118] This application does not impose specific restrictions on the model type, internal structure design, parameter design, training process, etc. of conditional random fields, and corresponding settings can be made according to the actual situation.

[0119] In this embodiment, the noise features, time-domain distortion parameters corresponding to multiple distortion types, and frequency-domain attenuation coefficients corresponding to multiple distortion types read in step 1025 are first organized and combined into an input feature set. Then, the input feature set is input into a pre-trained conditional random field model. This conditional random field model is trained on a large amount of speech data under different acoustic environments and can learn the correlation between noise features and distortion parameters. The conditional random field model performs structured reasoning on the input features, considers the mutual influence between various distortion types, and outputs a structured environmental compensation parameter. This environmental compensation parameter is represented in vector form and contains various compensation coefficients used for subsequent speech enhancement processing.

[0120] In practical applications, the 256-dimensional noise feature vector read in step 1025, the three time-domain distortion parameters (1.2 seconds, none, none), and the three frequency-domain attenuation coefficients (uniform attenuation, 0.3, 1.5) are combined to obtain a 262-dimensional input feature vector. This input feature vector is then input into a conditional random field model, which contains a linear chain conditional random field structure. The optimal compensation parameter sequence is obtained by decoding using the Viterbi algorithm. The model outputs a 128-dimensional environmental compensation parameter vector, which contains 64 tap coefficients of the time-domain filter and 64 gain values ​​of the frequency-domain equalizer, used to guide subsequent speech enhancement processing.

[0121] This application extracts environmental features from aligned semantic embeddings and generates auxiliary features by combining supplementary environmental data. These features are then matched and dynamically updated in an acoustic knowledge graph. Furthermore, structured prediction is performed using conditional random fields. This yields environmental compensation parameters that can accurately adapt to the current acoustic environment, enabling speech enhancement processing to adaptively adjust according to the characteristics of the actual environment.

[0122] Step 103: Generate initial modulation parameters based on the aligned semantic embedding, correct the initial modulation parameters using the environmental compensation parameters to obtain corrected modulation parameters, and use the corrected modulation parameters to perform feature modulation processing on the acoustic features to obtain modulated acoustic features that fuse semantic prior and environmental compensation information.

[0123] The initial modulation parameters refer to the basic parameters generated from the aligned semantic embedding for adjusting the acoustic features. These parameters include a scaling factor and a bias term. The scaling factor controls the amplification or reduction of each channel of the acoustic feature, while the bias term controls the numerical shift of each channel of the acoustic feature. The corrected modulation parameters refer to the final modulation parameters obtained by fusing the environmental compensation parameters with the initial modulation parameters. These parameters include a corrected scaling factor and a corrected bias term. The modulated acoustic features refer to the enhanced features obtained by applying the corrected modulation parameters to the original acoustic features. These features are adaptively adjusted based on semantic priors and environmental information on the basis of the original acoustic features.

[0124] In this embodiment, step 103 includes the following process:

[0125] Step 1031: Perform a linear transformation on the aligned semantic embedding to generate initial modulation parameters.

[0126] In this embodiment of the application, the aligned semantic embedding sequence obtained in step 101 is first obtained, and there is a semantic embedding vector at each time step in the sequence. Then, the semantic embedding vector at each time step is input into a pre-trained fully connected layer. The input dimension of the fully connected layer is the same as the dimension of the semantic embedding, and the output dimension is set to twice the number of acoustic feature channels. After the linear transformation of the fully connected layer, the initial modulation parameters corresponding to each time step are obtained. The first half of the dimension of the initial modulation parameters is used as the initial value of the scaling coefficient, and the second half of the dimension is used as the initial value of the bias term.

[0127] Step 1032: Add the environmental compensation parameters to the initial modulation parameters element by element to obtain the corrected modulation parameters, which include the corrected scaling factor and the corrected bias term.

[0128] In this embodiment, the initial modulation parameters generated in step 1031 are first obtained, which include a scaling factor and a bias term. Simultaneously, the environmental compensation parameters generated in step 102 are also obtained, which similarly include a compensation scaling factor and a compensation bias term, and the dimensions of these two parts are identical to the corresponding dimensions in the initial modulation parameters. Then, the scaling factor of the initial modulation parameters is added element-wise to the compensation scaling factor of the environmental compensation parameters to obtain the corrected scaling factor. The bias term of the initial modulation parameters is added element-wise to the compensation bias term of the environmental compensation parameters to obtain the corrected bias term. Finally, the corrected scaling factor and the corrected bias term are combined to form the corrected modulation parameters.

[0129] Step 1033: Multiply the corrected scaling factor and the acoustic feature element-wise along the channel dimension to obtain the intermediate feature.

[0130] In this embodiment of the application, the acoustic feature vector corresponding to the current time obtained in step 101 is first obtained. This vector has the same number of channels as the corrected scaling factor. At the same time, the corrected scaling factor obtained in step 1032 is obtained. Then, the corrected scaling factor and the acoustic feature vector are multiplied element-wise along the channel dimension. That is, for each channel, the corresponding element of the corrected scaling factor is multiplied by the corresponding element of the acoustic feature vector to obtain a new value. After performing this operation on all channels, a new vector is obtained, which is the intermediate feature.

[0131] Step 1034: Add the corrected bias term and the intermediate feature element-wise along the channel dimension to obtain the modulated acoustic feature.

[0132] In this embodiment, the intermediate feature vector generated in step 1033 is first obtained; the corrected bias term obtained in step 1032 is also obtained; then the corrected bias term and the intermediate feature vector are added element-wise along the channel dimension, that is, for each channel, the corresponding element of the corrected bias term is added to the corresponding element of the intermediate feature vector to obtain a new value; after performing this operation on all channels, a new vector is obtained, which is the modulation acoustic feature.

[0133] This application generates initial modulation parameters by aligning semantic embedding, corrects them using environmental compensation parameters, and finally applies the corrected modulation parameters to acoustic features, thereby achieving a deep fusion of semantic priors and environmental information, enabling the modulated acoustic features to simultaneously reflect the characteristics of speech content and the characteristics of the current acoustic environment.

[0134] Step 104: Perform context modeling on the modulated acoustic features to generate an acoustic mask, and use the acoustic mask to mask the acoustic features to output an enhanced speech signal.

[0135] The acoustic mask is a two-dimensional matrix with the same dimensions as the acoustic features. Each element in the matrix takes a value between 0 and 1, which represents the proportion of speech to be preserved in the corresponding time frame and frequency channel. For example, a value close to 1 indicates that the main component at that position is speech and should be preserved, while a value close to 0 indicates that the main component at that position is noise and should be suppressed. The enhanced speech signal refers to the time-domain waveform data recovered after masking. In this signal, background noise is effectively suppressed and the target speech component is enhanced.

[0136] In this embodiment, step 104 includes the following process:

[0137] Step 1041: Input the modulation acoustic features into the first branch and the second branch of the dual-path Transformer module respectively. The first branch is used to perform local context modeling of the modulation acoustic features, and the second branch is used to perform global context modeling of the modulation acoustic features.

[0138] In step 1041, the dual-path Transformer module refers to a neural network structure containing two parallel processing branches. The first branch adopts a Transformer structure with a local attention mechanism. The self-attention calculation in this branch is limited to the range of adjacent finite frames, for example, only focusing on the range of 5 frames before and after the current frame, and is used to capture short-term dependencies in the speech signal, such as spectral changes within phonemes. The second branch adopts a Transformer structure with a global attention mechanism. The self-attention calculation in this branch covers all frames of the entire input sequence and is used to capture long-term dependencies in the speech signal, such as the semantic coherence of the entire sentence.

[0139] It should be noted that the above structure is exemplary. This application does not impose specific limitations on the internal structure design of the dual-channel Transformer module, and corresponding settings can be made according to the actual situation.

[0140] Local context modeling refers to the short-term spectral change features extracted through the first branch, while global context modeling refers to the structural features of the entire speech segment extracted through the second branch.

[0141] In this embodiment, the modulation acoustic feature sequence obtained in step 103 is first acquired. This sequence contains feature vectors at T time points, each vector having a dimension of C. Then, the modulation acoustic feature sequence is simultaneously input into two parallel branches of the dual-path Transformer module. In the first branch, the modulation acoustic features are processed using a local window attention mechanism. Each attention head in this branch only calculates the attention weight between the current time point and each of the previous L time points, thereby extracting local spectral change features. In the second branch, the modulation acoustic features are processed using a global attention mechanism. Each attention head in this branch calculates the attention weight between the current time point and all T time points, thereby extracting the global structural features of the entire sequence. The two branches process in parallel without interfering with each other, and each outputs its own modeling results.

[0142] Step 1042: Fuse the output features of the first branch with the output features of the second branch to obtain contextual acoustic features.

[0143] In step 1042, the output features of the first branch refer to the feature sequence obtained after local context modeling, and the output features of the second branch refer to the feature sequence obtained after global context modeling. The contextual acoustic features refer to the comprehensive feature representation that integrates local detail information and global structural information. This feature includes both the fine changes in the short-time spectrum and the contextual dependencies of the entire speech segment.

[0144] In this embodiment, the output feature sequence after the first branch processing and the output feature sequence after the second branch processing in step 1041 are first obtained. These two sequences have the same number of time frames and feature dimensions. Then, the two feature sequences are added element-wise at the corresponding time points. That is, for each time point, the output feature vector of the first branch is added to the output feature vector of the second branch at the corresponding dimension to obtain the fused feature vector at that time point. After the addition operation is completed for all time points, a new feature sequence is obtained. This sequence is the context acoustic feature. Each frame in this feature contains both local spectral details and global context information.

[0145] Step 1043: Input the contextual acoustic features into the mask generation network, and have the mask generation network output an acoustic mask with the same dimension as the acoustic features.

[0146] In this embodiment, the contextual acoustic features obtained in step 1042 are first input into a mask generation network. This mask generation network contains multiple two-dimensional convolutional layers, which perform convolution operations on the contextual acoustic features in both time and frequency dimensions to gradually extract higher-level features. After processing by several convolutional layers, the network finally compresses the output value to the range of 0 to 1 using a Sigmoid activation function. Finally, the mask generation network outputs a two-dimensional matrix, the number of rows of which is the same as the number of time frames of the input modulated acoustic features, and the number of columns is the same as the number of frequency channels of the original acoustic features. This matrix is ​​the acoustic mask.

[0147] Step 1044: Multiply the acoustic mask element-wise with the acoustic features to obtain the masked acoustic features, and convert the masked acoustic features into a time-domain waveform to obtain an enhanced speech signal.

[0148] In this embodiment, the acoustic mask matrix generated in step 1043 is first obtained, and the original acoustic feature matrix extracted in step 101 is also obtained. The dimensions of these two matrices are exactly the same. Then, the acoustic mask and the original acoustic feature are multiplied element-wise. That is, for each frequency channel of each time frame, the corresponding element value in the acoustic mask is multiplied by the corresponding element value in the original acoustic feature to obtain a new value. After performing the multiplication operation on all time and frequency points, the masked acoustic feature matrix is ​​obtained. Finally, the masked acoustic feature matrix is ​​input to a vocoder or converted from frequency domain representation to time domain waveform data through short-time Fourier transform. This time domain waveform data is the enhanced speech signal, in which background noise is effectively suppressed and the clarity of the target speech is improved.

[0149] This application uses a dual-channel Transformer module to perform local and global context modeling of modulated acoustic features, and after fusion, generates an acoustic mask and masks the original acoustic features, ultimately obtaining an enhanced speech signal with effectively suppressed background noise and improved target speech clarity.

[0150] Figure 3 A schematic diagram of the structure of a speech enhancement system based on semantic prior guidance of a large language model is provided for an embodiment of this application, as shown below. Figure 3 As shown, the system includes:

[0151] The acquisition module 31 is used to acquire noisy speech signals and perform semantic modeling on the semantic embedding by combining a large language model. It uses its context-based next-word prediction capability to enhance and predict the noisy speech signals to obtain aligned semantic embeddings that are time-aligned with acoustic features.

[0152] The matching module 32 is used to perform association matching between the aligned semantic embedding and the pre-constructed acoustic knowledge graph to obtain environmental compensation parameters. The acoustic knowledge graph stores the mapping relationship between noise features and speech distortion patterns under different acoustic environments.

[0153] The generation module 33 is used to generate initial modulation parameters based on the aligned semantic embedding, correct the initial modulation parameters using the environmental compensation parameters to obtain corrected modulation parameters, and use the corrected modulation parameters to perform feature modulation processing on the acoustic features to obtain modulated acoustic features that fuse semantic prior and environmental compensation information.

[0154] The processing module 34 is used to perform context modeling processing on the modulated acoustic features to generate an acoustic mask, and to use the acoustic mask to mask the acoustic features and output an enhanced speech signal.

[0155] Another specific implementation example, such as Figure 4 As shown:

[0156] It should be noted that this embodiment is only used to illustrate the working process of the technical solution of this application in a specific scenario, and does not constitute any limitation on the scope of protection of this application. In actual applications, the parameters and model selection can be reasonably adjusted according to the specific noise environment and hardware conditions.

[0157] Please refer to the following embodiment, which focuses on describing how, in a real telephone call scenario, when a sudden car horn completely obscures the key speech segment "click", this application utilizes the context prediction capability of a large language model and combines it with the environmental compensation parameters of an acoustic knowledge graph to ultimately fully recover the semantically coherent enhanced speech "meeting at 3 pm".

[0158] First, the user says "Meeting at 3 PM." While uttering the word "click," a loud car horn suddenly sounds in the surrounding environment, completely obscuring this short speech segment in the temporal domain. After acquiring this noisy speech signal, the system extracts semantic embeddings from the signal and inputs them into a large language model for contextual modeling. Based on the semantic logic of "3 PM" and "meeting," the large language model infers that the most reasonable semantic content for the obscured position should be "click," thus generating an enhanced semantic embedding.

[0159] On the other hand, the system extracts acoustic features from noisy speech in parallel and extracts environmental feature vectors from aligned semantic embeddings, then performs association matching with a pre-constructed acoustic knowledge graph. Since horn noise is a typical sudden impulse noise, the corresponding nodes in the knowledge graph store the features of this type of noise and speech distortion patterns, such as temporal distortion parameters describing the brief impulse characteristics of impulse noise and frequency domain attenuation coefficients describing its masking effect on mid-to-high frequency bands.

[0160] After obtaining the environmental compensation parameters through matching, the system adds them element-wise to the initial modulation parameters generated based on semantic embedding and then uses the modified modulation parameters to perform feature modulation processing on the acoustic features, thus obtaining modulated acoustic features that integrate semantic prior and environmental compensation information.

[0161] Finally, through local and global context modeling of the dual-path Transformer module, an acoustic mask is generated and the original acoustic features are masked. The final output enhanced speech signal is no longer "afternoon three...meeting" obtained by the traditional method, but "meeting at three o'clock in the afternoon" with complete semantics.

[0162] As can be seen, this embodiment not only utilizes the ability of a large language model to complete missing semantics, but also introduces an environmental compensation mechanism for sudden noise through an acoustic knowledge graph, so that the enhanced speech is both semantically correct and acoustically natural, thereby improving the clarity and intelligibility of speech signals in complex acoustic environments.

[0163] The scenarios and parameter settings described above are for illustrative purposes only. Those skilled in the art can select different large language models or adjust the matching threshold of the knowledge graph according to the specific noise type and hardware conditions during actual implementation.

[0164] The speech enhancement system based on semantic prior guidance of a large language model in this application is used to implement the aforementioned speech enhancement method based on semantic prior guidance of a large language model. Therefore, the specific implementation of the speech enhancement system based on semantic prior guidance of a large language model can be found in the embodiment section of the speech enhancement method based on semantic prior guidance of a large language model above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0165] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described speech enhancement methods based on semantic prior guidance of a large language model.

[0166] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described speech enhancement methods based on semantic prior guidance of a large language model.

[0167] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0168] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the speech enhancement method based on semantic prior guidance of a large language model.

[0169] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0171] The above provides a detailed description of the speech enhancement method and system based on semantic prior guidance of a large language model provided in this application.

[0172] To facilitate verification of whether others have used the speech enhancement method described in this application, this section provides an operable infringement detection test scheme. By analyzing the output characteristics of the system under test under specific test inputs, it can be determined whether it has adopted the technical solution described in this application.

[0173] (a) Test Input Construction

[0174] Construct test speech samples: In the sentence "The meeting is scheduled to start at 3 PM," mask the word "click" with strong noise; in the sentence "Please buy me a bottle of mineral water," mask the word "bottle of mineral water." Various noise types, including non-stationary noise and impulse noise, are used.

[0175] (II) Output Feature Analysis

[0176] The enhanced speech output by the system under test is analyzed for the following features:

[0177] Semantic completion capability: If the system under test can correctly restore "The meeting is scheduled to start at 3 PM" to "The meeting is scheduled to start at 3 PM", it indicates that it has semantic prior guidance capability.

[0178] Environmental adaptive characteristics: If the system exhibits differentiated enhancement effects under different noise environments and can reasonably supplement the acoustic evidence based on environmental characteristics when it is missing, it indicates that it has adopted an environmental compensation mechanism.

[0179] (III) Infringement Determination Standards

[0180] When the system under test simultaneously meets the following conditions, it can be preliminarily determined that it uses the technical solution described in this application:

[0181] It has semantic completion capabilities: in scenarios where semantic information is missing, it can complete the masked words or syllables based on contextual semantic information, and the completion results are semantically reasonable.

[0182] It has environmental adaptability: Under different acoustic environments, the enhancement effect of the system exhibits environmental adaptability, and can dynamically adjust the processing strategy according to the environmental type.

[0183] A semantic and environmental joint modulation mechanism is adopted: during the enhancement process, semantic information and environmental compensation information jointly modulate the acoustic features, and the modulation method conforms to the parameter correction and feature modulation mechanism described in this application.

[0184] Technical solution consistency: The overall architecture, module composition, and data flow of the system under test are substantially the same as or equivalent to the technical solution described in this application, including core aspects such as semantic embedding alignment, knowledge graph matching, parameter correction, feature modulation, and context modeling.

[0185] Based on this framework, the use of additional information, such as but not limited to speaker voiceprint characteristics, to further enhance the speech enhancement effect is also within the scope of this patent protection.

[0186] (iv) Precautions

[0187] This testing method is for technical reference only. Actual infringement determination should be based on a comprehensive assessment of the scope of protection of the patent claims, the technical solution of the accused infringing product, and relevant laws and regulations.

[0188] When conducting infringement detection, it is recommended to entrust a qualified judicial appraisal institution to conduct technical appraisal in accordance with legal procedures.

[0189] This testing method does not constitute an infringement claim against any third-party products or services, but is only used to illustrate the technical features of this application and possible infringement verification methods.

Claims

1. A speech enhancement method based on semantic prior guidance from a large language model, characterized in that, include: Noisy speech signals are acquired, and semantic modeling of semantic embedding is performed by combining a large language model. The noisy speech signal is enhanced and predicted by using its context-based next word prediction capability to obtain aligned semantic embedding that is time-aligned with acoustic features. The aligned semantic embedding refers to the semantic vector representation that has been enhanced by the large language model and is precisely corresponding to the acoustic features in the time dimension. The environmental compensation parameters are obtained by associating and matching the aligned semantic embedding with the pre-constructed acoustic knowledge graph. The acoustic knowledge graph stores the mapping relationship between noise features and speech distortion patterns under different acoustic environments. Initial modulation parameters are generated based on the aligned semantic embedding. The initial modulation parameters are then corrected using the environmental compensation parameters to obtain corrected modulation parameters. The acoustic features are then subjected to feature modulation processing using the corrected modulation parameters to obtain modulated acoustic features that fuse semantic priors and environmental compensation information. The modulated acoustic features are subjected to context modeling to generate an acoustic mask, and the acoustic features are masked using the acoustic mask to output an enhanced speech signal. The step of performing association matching between the aligned semantic embedding and the pre-constructed acoustic knowledge graph to obtain environmental compensation parameters includes: Acquire supplementary environmental data corresponding to the current time, extract auxiliary feature values ​​from the supplementary environmental data, and encode the auxiliary feature values ​​into an auxiliary feature vector. The supplementary environmental data refers to auxiliary information related to the current acquisition environment that can be obtained, other than the speech signal itself. An environmental feature vector is extracted from the aligned semantic embedding, and the environmental feature vector is used to characterize the noise characteristics of the current acoustic environment; The dimensions of the environmental feature vector are transformed to match the dimensions of the node features in the acoustic knowledge graph to obtain the mapped feature vector. The mapped feature vector is then concatenated with the auxiliary feature vector to obtain the joint environmental feature vector. The joint environment feature vector is compared with the node feature vector stored in each node of the acoustic knowledge graph, and the node that matches the joint environment feature vector is determined as the matching node. The acoustic knowledge graph contains multiple nodes, and each node stores a node feature vector, noise feature and speech distortion mode corresponding to an acoustic environment. The noise features and the speech distortion pattern are read from the matching node. The speech distortion pattern includes multiple distortion types and time-domain distortion parameters and frequency-domain attenuation coefficients corresponding to each distortion type. The noise characteristics, the time-domain distortion parameters, and the frequency-domain attenuation coefficients are input into a conditional random field for structured prediction to obtain environmental compensation parameters.

2. The method according to claim 1, characterized in that, The process involves semantic modeling of the noisy speech signal using a large language model, leveraging its context-based next-word prediction capability to enhance and predictively complete the noisy speech signal, resulting in an aligned semantic embedding that is temporally aligned with the acoustic features. This includes: Extract semantic embeddings and acoustic features from the noisy speech signal; The semantic embedding is input into a pre-projection network, which maps the dimensions of the semantic embedding to match the input dimensions of the large language model, thus obtaining the projected semantic embedding. The projected semantic embedding is input into a large language model, which then performs context prediction processing based on a self-attention mechanism to obtain context features. The context features are input into a post-projection network, which maps the context features back to the speech semantic space to obtain an enhanced semantic embedding. The enhanced semantic embedding is time-aligned with the acoustic features to obtain the aligned semantic embedding.

3. The method according to claim 2, characterized in that, The step of inputting the projected semantic embedding into a large language model, and having the large language model perform context prediction processing based on a self-attention mechanism to obtain context features, includes: The projected semantic embedding is input into a large language model, and the projected semantic embedding is superimposed with the positional encoding through the positional encoding layer of the large language model to obtain an initial embedding sequence. The initial embedding sequence is used to represent speech semantic features with temporal information. The initial embedding sequence is processed by multiple decoder layers of the large language model, wherein each decoder layer performs a linear transformation on the initial embedding sequence through an internal multi-head self-attention sublayer to generate a query vector, a key vector, and a value vector. The similarity between the query vector and the key vector is calculated by the weight calculation sublayer inside the decoder layer to obtain the attention weight. The value vector is then weighted and aggregated using the attention weight to obtain an aggregate vector. The aggregate vector is used to represent the semantic information of the current moment predicted based on the historical context. The first intermediate sequence is obtained by adding the aggregated vector to the initial embedded sequence and then normalizing it through the normalization sub-layer inside the decoder layer. The first intermediate sequence is nonlinearly transformed by the feedforward sub-layer inside the decoder layer to obtain the second intermediate sequence, and the second intermediate sequence is used as the output sequence of the current decoder layer. The output layer of the large language model uses the sequence output by the last decoder layer as contextual features.

4. The method according to claim 1, characterized in that, The step of comparing the joint environment feature vector with the node feature vector stored in each node of the acoustic knowledge graph, and determining the node that matches the joint environment feature vector as the matching node, includes: Calculate the Euclidean distance between the joint environment feature vector and the node feature vector stored in each node of the acoustic knowledge graph to obtain the distance value corresponding to each node; The smallest distance value is selected from all the distance values ​​as the minimum distance value, and the node corresponding to the minimum distance value is selected as the candidate node; The minimum distance value is compared with a preset matching threshold; When the minimum distance value is less than the matching threshold, the candidate node is determined as the matching node; When the minimum distance value is greater than or equal to the matching threshold, the joint environment feature vector is used as the new node feature vector, the new node feature vector is added to the acoustic knowledge graph as a new node, and a preset initial noise feature and initial speech distortion mode are assigned to the new node, and the new node is determined as a matching node.

5. The method according to claim 1, characterized in that, The process of generating initial modulation parameters based on the aligned semantic embedding, correcting the initial modulation parameters using the environmental compensation parameters to obtain corrected modulation parameters, and then using the corrected modulation parameters to perform feature modulation processing on the acoustic features to obtain modulated acoustic features that fuse semantic prior and environmental compensation information includes: A linear transformation is performed on the aligned semantic embedding to generate initial modulation parameters; The environmental compensation parameters are added element by element to the initial modulation parameters to obtain the corrected modulation parameters, which include the corrected scaling factor and the corrected bias term. The modified scaling factor is multiplied element-wise with the acoustic feature along the channel dimension to obtain the intermediate feature; The modified bias term and the intermediate feature are added element-wise along the channel dimension to obtain the modulated acoustic feature.

6. The method according to claim 1, characterized in that, The step of performing contextual modeling on the modulated acoustic features to generate an acoustic mask, and using the acoustic mask to mask the acoustic features to output an enhanced speech signal, includes: The modulated acoustic features are input to the first branch and the second branch of the dual-path Transformer module, respectively. The first branch is used to perform local context modeling on the modulated acoustic features, and the second branch is used to perform global context modeling on the modulated acoustic features. The output features of the first branch and the output features of the second branch are fused to obtain the contextual acoustic features; The contextual acoustic features are input into a mask generation network, which then outputs an acoustic mask with the same dimension as the acoustic features. The acoustic mask is multiplied element-wise with the acoustic features to obtain the masked acoustic features, and the masked acoustic features are converted into time-domain waveforms to obtain the enhanced speech signal.

7. A speech enhancement system based on semantic prior guidance of a large language model, used to execute the speech enhancement method based on semantic prior guidance of a large language model as described in any one of claims 1 to 6, characterized in that, include: The acquisition module is used to acquire noisy speech signals and perform semantic modeling on the semantic embedding by combining a large language model. It uses its context-based next word prediction capability to enhance and predict the noisy speech signals to obtain aligned semantic embeddings that are time-aligned with acoustic features. The aligned semantic embeddings refer to semantic vector representations that have been enhanced by the large language model and are precisely corresponding to the acoustic features in the time dimension. The matching module is used to perform association matching between the aligned semantic embedding and the pre-constructed acoustic knowledge graph to obtain environmental compensation parameters. The acoustic knowledge graph stores the mapping relationship between noise features and speech distortion patterns under different acoustic environments. The generation module is used to generate initial modulation parameters based on the aligned semantic embedding, correct the initial modulation parameters using the environmental compensation parameters to obtain corrected modulation parameters, and perform feature modulation processing on the acoustic features using the corrected modulation parameters to obtain modulated acoustic features that fuse semantic prior and environmental compensation information. The processing module is used to perform context modeling on the modulated acoustic features to generate an acoustic mask, and to use the acoustic mask to mask the acoustic features, thereby outputting an enhanced speech signal.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the speech enhancement method based on semantic prior guidance of a large language model as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the speech enhancement method based on semantic prior guidance of a large language model as described in any one of claims 1 to 6.

Images