Speech processing method and apparatus, electronic device, and computer-readable storage medium

Abstract

Embodiments of the present application disclose a speech processing method and device, electronic equipment and computer readable storage medium, relating to the technical field of voice separation; the method comprises: performing feature extraction and fusion on audio data and motion sensing data to obtain environmental context vectors and environmental perception audio features; inputting the environmental perception audio features into a plurality of separation blocks connected in turn to obtain convolution features and query vectors of each separation block; determining target memory features matched with each query vector from a speaker memory library, and generating an attention key and an attention value based on the target memory features; performing attention calculation on the attention key and the attention value based on the environmental context vectors and the query vectors to obtain an attention map of the current separation block; performing gate modulation on the convolution features of the current separation block based on the attention map to obtain gate enhanced features of the current separation block; and obtaining an enhanced speech signal of a target speaker according to the gate enhanced features of the plurality of separation blocks.

Description

Technical Field

[0001] This application relates to the field of human voice separation technology, specifically to a speech processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the increasing prevalence of remote work, online meetings, and multi-person voice interaction scenarios, speech separation and speaker-specific enhancement technologies have become crucial for improving the voice interaction experience. Related speech separation technologies mainly fall into two categories: methods based on traditional signal processing and methods based on deep learning. Methods based on traditional signal processing typically rely on the spatial location information of the microphone array and sound source for analysis, making them sensitive to device placement and environmental noise, and their performance degrades significantly when the speaker moves. While deep learning-based methods can effectively separate overlapping speech, the models often involve high computational costs and numerous parameters, making them difficult to run in real-time on resource-constrained edge devices such as headphones and speakers. Summary of the Invention

[0003] This application provides a speech processing method, apparatus, electronic device, and computer-readable storage medium that can perform speech separation in real time on resource-constrained devices.

[0004] In a first aspect, embodiments of this application provide a voice processing method, including: Acquire audio data and motion sensing data, extract and fuse features from the audio data and motion sensing data to obtain environmental context vectors and environmentally perceptual audio features; The environmentally aware audio features are input into a series of sequentially connected separation blocks to obtain the convolutional features and query vectors of each separation block. Target memory features matching each query vector are determined from the speaker's memory bank, and attention keys and attention values ​​are generated based on the target memory features; Based on the environment context vector and the query vector, attention is calculated on the attention key and attention value to obtain the attention graph of the current separation block; Based on the attention map, the convolutional features of the current separation block are gated and modulated to obtain the gated enhanced features of the current separation block; The enhanced speech signal of the target speaker is obtained based on the gating enhancement features of the multiple separation blocks.

[0005] Secondly, embodiments of this application provide a voice processing device, including: The data acquisition module is used to acquire audio data and motion sensing data, and to extract and fuse features from the audio data and motion sensing data to obtain environmental context vectors and environmentally perceived audio features. The feature input module is used to sequentially connect multiple separation blocks of the environmentally aware audio feature input to obtain the convolutional features and query vectors of each separation block; The feature matching module is used to determine the target memory features that match each of the query vectors from the speaker's memory bank, and to generate attention keys and attention values ​​based on the target memory features; The attention calculation module is used to perform attention calculation on the attention key and attention value based on the environment context vector and the query vector to obtain the attention graph of the current separation block; The gated modulation module is used to perform gated modulation on the convolutional features of the current separation block based on the attention map to obtain the gated enhanced features of the current separation block; The speech acquisition module is used to obtain the enhanced speech signal of the target speaker based on the gating enhancement features of the multiple separation blocks.

[0006] Thirdly, embodiments of this application also provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the above-described voice processing method.

[0007] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the above-described speech processing method. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a schematic diagram of the steps of a speech processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a separation network provided in an embodiment of this application; Figure 3 This is a schematic diagram of the process for updating the speaker's memory bank provided in one embodiment of this application; Figure 4 This is a schematic diagram of the data flow of a speech processing method provided in an embodiment of this application; Figure 5 This is a schematic diagram of a multimodal feature fusion strategy provided in an embodiment of this application; Figure 6This is a schematic diagram of the structure of a voice processing device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0011] In one embodiment, such as Figure 1 As shown, a voice processing method is provided. Although the logical order is illustrated in the step diagram, in some cases, the steps shown or described can be performed in a different order than that shown in the diagram. Specifically, this voice processing method can be applied to devices with audio acquisition and motion sensing capabilities, such as resource-constrained edge devices; it can also be applied to other terminal devices or servers that are communicatively connected to the aforementioned devices. These devices can be one or more of the following, including but not limited to smart headsets, smart speakers, conference machines, smartphones, tablets, laptops, desktop computers, and in-vehicle computers.

[0012] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0013] according to Figure 1 The speech processing method shown includes at least steps S110 to S160, which are described in detail below: In step S110, audio data and motion sensing data are acquired, and feature extraction and fusion are performed on the audio data and motion sensing data to obtain environmental context vectors and environmental perception audio features.

[0014] In step S120, the environmentally perceptual audio features are input into multiple sequentially connected separation blocks to obtain the convolutional features and query vectors of each separation block.

[0015] In step S130, target memory features that match each query vector are determined from the speaker's memory bank, and attention keys and attention values ​​are generated based on the target memory features.

[0016] In step S140, attention is calculated on the attention key and attention value based on the environment context vector and query vector to obtain the attention map of the current separation block.

[0017] In step S150, the convolutional features of the current separation block are gated and modulated based on the attention map to obtain the gated enhancement features of the current separation block.

[0018] In step S160, the enhanced speech signal of the target speaker is obtained based on the gating enhancement features of multiple separation blocks.

[0019] Audio data refers to time-domain audio signals acquired by one or more microphones on a device. It may contain a mixture of the target speaker's voice, interference from other speakers, and environmental noise. Motion sensing data refers to sensor data acquired by the device's built-in Inertial Measurement Unit (IMU). The IMU typically includes an accelerometer and gyroscope, reflecting the device's current motion state, such as stationary, walking, turning, or running. Environmental context vectors are compact vector representations obtained by feature extraction and mapping based on motion sensing data, used to characterize the device's current motion state. Environmentally aware audio features are feature representations obtained by fusing environmental context vectors and audio features, simultaneously containing information about both audio content and the device's motion state.

[0020] In one embodiment, step S110 may include: time-aligning audio data and motion sensing data; extracting features from the time-aligned audio data to obtain audio features; extracting features from the time-aligned motion sensing data to obtain motion sensing features; processing the motion sensing features based on a multilayer perceptron to obtain an environmental context vector; and fusing the environmental context vector and audio features to obtain environmentally perceptual audio features.

[0021] The audio data and motion sensing data are first time aligned to ensure that the two data streams are consistent in the time dimension, thereby eliminating timing deviations caused by sampling rate differences and transmission delays.

[0022] Feature extraction is performed on the time-aligned audio data, for example, through frame segmentation, windowing, and short-time fourier transform (STFT) or temporal convolutional coding, to obtain audio features. Motion sensing features are then extracted from the time-aligned motion sensing data after preprocessing such as filtering and calibration.

[0023] Motion sensing features can be processed using a Multi-Layer Perceptron (MLP) to generate an environmental context vector. A MLP is a feedforward neural network consisting of multiple stacked fully connected layers, which can be used to map motion sensing features into a compact environmental context vector. The environmental context vector and audio features are concatenated along their feature dimensions, and then dimensionality reduction and feature fusion are performed through a fusion convolutional layer to obtain the environmentally perceptual audio features.

[0024] Through the above fusion strategy, the environmentally perceptible audio features carry information on both audio content and device motion state, enabling subsequent processing to perceive changes in device motion state and improve robustness in dynamic scenarios such as changes in wearing method and user movement.

[0025] Figure 2 This is a schematic diagram of the structure of a separation network provided in an embodiment of this application; the separation network includes an encoder, a separator, and a decoder. The separator can query and update the speaker's memory bank. A separation block refers to a processing unit with the same structure that is connected in series within the separator; multiple separation blocks are stacked to form a separator. The receptive field of each separation block increases progressively with the order of arrangement, meaning that each subsequent separation block can capture dependencies over a larger time range than the previous one.

[0026] Using environmentally perceptual audio features as the input features for the first separation block, each block performs causal convolution on the input features to obtain convolutional features. Causal convolution is a structure that uses only information from the current and previous time steps for convolution operations, ensuring the causality of the processing and making it suitable for real-time speech processing scenarios. Different expansion factors are used in each separation block, for example, expansion factors of 1, 2, 4, and 8, to exponentially expand the receptive field, thereby modeling the long-term dependencies of speech signals at different time scales. Convolutional features refer to the feature representation obtained after convolution processing the input features of the separation block. The convolutional features are mapped through a linear layer to generate a query vector, the dimension of which can be the same as or different from the dimension of the convolutional features. The query vector is a vector generated based on the convolutional features after linear mapping, used for subsequent interactive retrieval with the speaker's memory. Through the hierarchical concatenation of multiple separation blocks, the input features are progressively refined, with each separation block refining the target speaker features at a larger time scale.

[0027] A speaker memory bank is a data structure used to store and manage the memory features of learned speakers. The speaker's memory features contain structured data of feature signatures that characterize the speaker's timbre information. These feature signatures are fixed-dimensional embedding vectors extracted from speech audio through a voiceprint model and are used for speaker similarity comparison.

[0028] The system can determine target memory features from the speaker's memory bank using either an automatic retrieval mode or a specified mode. Target memory features refer to the memory features retrieved from the speaker's memory bank that match the current query vector. In automatic retrieval mode, the cosine similarity between the query vector and each memory feature in the speaker's memory bank is calculated. The K most similar memory features (e.g., K=2) are selected as target memory features, and the K target memory features are adaptively weighted and fused with the query vector based on similarity to generate the attention key and attention value for the current query. In specified mode, the user specifies the target speaker identifier via physical buttons, voice commands, or an application. The system directly retrieves the corresponding identifier's memory features from the speaker's memory bank as target memory features, and generates the attention key and attention value accordingly. The two modes can be flexibly switched according to the actual application scenario. The attention key and attention value are concepts in the attention mechanism; the attention key is used to calculate the similarity with the query vector, and the attention value is used to provide a content representation of the target information in the attention calculation.

[0029] A similarity score is calculated based on the query vector and attention key, and the context vector is used as a bias term to correct the similarity score, resulting in a corrected attention score. The context vector is linearly mapped and then additively superimposed on the original similarity score, allowing motion state information to guide the attention calculation to focus more on sound source features that match the current motion scene. The corrected attention score and attention value are then weighted and summed to obtain the attention map of the current separation block. The attention map is the feature representation obtained after attention calculation, reflecting the degree of correlation between the input features at each time step and the target speaker features.

[0030] The attention map is activated, for example using the sigmoid activation function, mapping the values ​​of the attention map to between 0 and 1, resulting in a gating mask. A gating mask is a mask vector with values ​​between 0 and 1, used to control the degree to which different dimensions of the features are preserved. The gating mask is then multiplied element-wise with the convolutional features of the current separation block to achieve gating modulation, resulting in gated enhanced features. Gating modulation is a processing method that selectively controls the strength of features through a gating mask. Gated enhanced features are features that have undergone gating modulation, where the target speaker's features are enhanced, while non-target speaker features and noise are suppressed. In this process, time steps related to the target speaker receive a gating value close to 1 and are thus preserved and enhanced, while time steps related to interfering speakers and noise receive a gating value close to 0 and are thus suppressed.

[0031] The gated enhancement features of each delimiter block are residually added to the input features of that block, and then normalized by layers to obtain the input features of the next delimiter block. This process achieves stepwise feature propagation and refinement. Residual addition refers to directly adding the output of a delimiter block to its input, which helps maintain gradient flow and training stability. Layer normalization normalizes the features along the channel dimension to accelerate convergence and improve training stability. The output features of the last delimiter block are determined as the final features. A decoder is used to decode the final features, for example, by reconstructing the time-domain waveform using a one-dimensional transposed convolutional layer, to obtain the enhanced speech signal of the target speaker.

[0032] In one specific application scenario, users wear wireless headsets to participate in multi-person conference calls. The headsets have built-in dual microphones and a six-axis IMU (including a three-axis accelerometer and a three-axis gyroscope). The headset end is equipped with... Figure 2 The separation network is shown. First, dual microphones acquire multi-channel audio data at a sampling rate of 16kHz, and the IMU acquires motion sensing data at a sampling rate of 100Hz. The audio data is segmented into frames (512 sampling points per frame, 128 sampling points per frame shift), processed with a Hanning window and STFT, resulting in audio features of dimension 257×T, where T is the frame number. The motion sensing data is low-pass filtered and calibrated to extract pose features, which are then used to generate a 32-dimensional environment context vector through a two-layer MLP (64 hidden layer dimensions). The environment context vector and audio features are concatenated along the feature dimension, and then dimensionality-reduced and fused through a one-dimensional fusion convolutional layer (input dimension 257+32=289, output dimension 128) to obtain 128×T environment-aware audio features.

[0033] The environmentally perceptual audio features are input into a separator consisting of four concatenated decoupling blocks. The causal convolution dilation factors of the four decoupling blocks are 1, 2, 4, and 8, respectively, with a kernel size of 3 and a feature dimension of 128. Taking the first decoupling block as an example: a causal convolution with a dilation factor of 1 is performed on the input features to obtain convolutional features of dimension 128×T; a query vector Q of dimension 128×T is generated through a linear layer. The cosine similarity between Q and all memory features in the speaker's memory bank is calculated, and the two memory features with the highest similarity are selected. The key parts of the two memory features are weighted and summed based on similarity to obtain the attention key K_t, and the value parts are weighted and summed based on similarity to obtain the attention value V_t.

[0034] The context vector is linearly mapped and then used as an additive bias, which is superimposed on the dot product similarity score of Q and K_t to obtain the corrected attention score. The corrected attention score and V_t are then weighted and summed to obtain the attention map A. The attention map A is activated by Sigmoid to obtain a gating mask G with values ​​between 0 and 1. G is then multiplied element-wise with the convolutional features to obtain the gated enhancement features.

[0035] The gated enhancement features are added to the residual input features of the first separation block, and then normalized to obtain the input features of the second separation block. The second, third, and fourth separation blocks perform the same operation as the first separation block, with their receptive fields increasing progressively with the expansion factor. Finally, the fourth separation block outputs the final feature V', with a dimension of 128×T. V' is then reconstructed back into a time-domain waveform through a one-dimensional transposed convolutional layer to obtain the enhanced speech signal of the target speaker, which is then played to the user through the headphone speaker. The inference latency of the entire processing flow is low, which can meet the requirements of real-time voice communication.

[0036] The technical solution of this application embodiment extracts and fuses features from audio data and motion sensing data to obtain environmental context vectors and environmentally aware audio features. This achieves early fusion of motion state information and audio information, enabling subsequent processing to perceive changes in the user's motion state and improving the robustness of speech separation in dynamic scenarios such as changes in wearing method and user movement. The environmentally aware audio features are input into multiple sequentially connected separation blocks to obtain convolutional features and query vectors for each separation block. Utilizing the hierarchical concatenation structure of multiple separation blocks, the receptive field gradually expands with processing depth, enabling the modeling of speech signal dependencies at different time scales. Simultaneously, each separation block progressively refines features, ensuring separation accuracy while maintaining low computational overhead. Target memory features matching each query vector are determined from the speaker memory bank, and attention keys and attention values ​​are generated based on these target memory features. The speaker memory bank is then used for each query... It provides prior information related to the target speaker, enabling adaptive speaker recognition and enhancement without pre-training or pre-registration of the target speaker. Based on the environmental context vector and query vector, it performs attention calculation on attention keys and attention values ​​to obtain the attention map of the current segmentation block. Motion environment information is introduced into the attention calculation process, guiding the model to accurately focus on the features of the target speaker even in complex motion scenarios. Based on the attention map, it performs gated modulation on the convolutional features of the current segmentation block to obtain the gated enhancement features of the current segmentation block. This enhances the features of the target speaker and suppresses noise and irrelevant speaker features at the feature level, improving the signal-to-noise ratio. Based on the gated enhancement features of multiple segmentation blocks, it obtains the enhanced speech signal of the target speaker. The result of the step-by-step purification of multiple segmentation blocks is restored to the clean speech of the target speaker, achieving high-quality real-time speech separation and enhancement on resource-constrained edge devices such as headphones and speakers.

[0037] Based on the above technical solution, as an embodiment, step S120 may include: using environmentally perceptible audio features as input features of the first separation block; performing causal convolution processing on the input features of each separation block to obtain convolution features of each separation block; wherein the receptive field of each separation block increases progressively with the order of the separation blocks; and generating query vectors for each separation block based on the convolution features of each separation block.

[0038] Multiple separation blocks are stacked in series. The environmental audio features are input only to the first separation block, rather than to each separation block individually. The input features of each subsequent separation block are derived from the output of the previous separation block, thus forming a chain-like processing structure that passes through and refines at each level.

[0039] Causal convolution is applied to the input features of each deconvolution block to obtain the convolutional features of each block. Each deconvolution block employs causal convolution with different dilation factors. The dilation factor refers to the multiple of the spacing between adjacent kernel elements in causal convolution. Increasing the dilation factor can exponentially expand the receptive field of the convolution without increasing the number of parameters. The receptive field refers to the temporal range of the input sequence that each output element in the convolution operation can perceive. The receptive field of each deconvolution block increases progressively with the order of the blocks. For example, when four deconvolution blocks are set, the dilation factors can be 1, 2, 4, and 8 respectively. With a kernel size of 3, the receptive fields corresponding to each deconvolution block are 3, 5, 9, and 17 time steps, respectively. Through this progressively expanding receptive field design, the preceding deconvolution blocks focus on capturing short-term local dependencies of the speech signal, while the subsequent deconvolution blocks can model long-term dependencies at a larger time scale. This allows multiple deconvolution blocks to refine the target speaker features layer by layer at different temporal granularities.

[0040] Convolutional features are mapped through linear layers to generate query vectors with the same or different dimensions as the convolutional features. Linear layers are fully connected layers without non-linear activation functions, used to perform linear transformations on the input features. The query vectors are used in subsequent steps for interactive retrieval with the speaker's memory bank; their similarity to the memory features in the memory bank determines the selection of the target memory features. Since the convolutional features of each deconvolutional block have different receptive fields, the query vectors generated by different deconvolutional blocks encode the input features at their respective time scales. The target memory features retrieved from the speaker's memory bank also correspond to different time granularities, allowing multiple deconvolutional blocks to acquire prior information related to the target speaker at different scales.

[0041] By adopting the technical solution of this application embodiment, environmentally perceptual audio features are input only into the first separation block and passed step by step in a chain structure. With the causal convolution of each separation block using different expansion factors to gradually increase the receptive field, layer-by-layer modeling of the dependence of speech signals at different time scales is realized. The preceding separation block captures local features in detail, and the subsequent separation block grasps the global context. Multiple separation blocks work together, which effectively improves the accuracy of speech separation and the effect of feature purification while ensuring real-time processing.

[0042] Based on the above technical solution, as an embodiment, step S130 may include: selecting at least one target memory feature from the speaker's memory bank according to the similarity between the query vector and each memory feature in the speaker's memory bank; and performing adaptive weighted fusion of the target memory feature and the query vector based on the similarity corresponding to the target memory feature to obtain the attention key and attention value.

[0043] At least one target memory feature can be selected from the speaker's memory bank based on the similarity between the query vector and various memory features in the speaker's memory bank. The similarity is used to measure the degree of matching between the query vector and the memory feature. Cosine similarity can be used as the metric. Cosine similarity refers to the cosine of the angle between two vectors, ranging from -1 to +1. The closer the value is to +1, the more consistent the directions of the two vectors are in the feature space, indicating a higher degree of matching. It should be noted that this application does not limit the specific method of similarity measurement; other methods such as Euclidean distance or dot product can also be used.

[0044] After calculating the cosine similarity between the query vector and each memory feature in the speaker's memory bank, the features are sorted in descending order of similarity. The K most similar memory features are selected as target memory features, where K is a preset positive integer, such as K=2. By selecting multiple target memory features instead of a single nearest neighbor, prior information from multiple similar speakers can be comprehensively utilized, reducing matching bias caused by subtle differences between a single memory feature and the current speaker.

[0045] Based on the similarity between the target memory features and the query vector, an adaptive weighted fusion is performed to obtain the attention key and attention value. Adaptive weighted fusion refers to a fusion method that uses the similarity between each target memory feature and the query vector as a weight to perform a weighted summation of the target memory features and the query vector. The weights are automatically adjusted as the similarity changes, without the need to manually set a fixed fusion ratio.

[0046] Specifically, the similarity of each of the K target memory features is first normalized using Softmax to obtain normalized weights that sum to 1. Each memory feature contains a pair of key and value parts. The key parts of each of the K target memory features are weighted and summed using the normalized weights to obtain the attention key for the current query. The value parts of each of the K target memory features are weighted and summed using the same normalized weights to obtain the attention value for the current query.

[0047] In the aforementioned weighted fusion process, memory features with higher similarity to the query vector receive greater weight and contribute more to the final generated attention keys and attention values, thus making them closer to the speaker's true feature distribution. Simultaneously, information from the query vector itself is incorporated into the generation of attention keys and attention values, ensuring that the final result not only includes prior voiceprint information from the speaker's memory bank but also retains the current segmentation block's immediate perception of the input features, enhancing the representational ability of attention keys and attention values ​​for the current input.

[0048] The technical solution adopted in this application selects multiple target memory features from the speaker memory bank based on similarity, and generates attention keys and attention values ​​by adaptive weighted fusion based on similarity. This can comprehensively utilize the prior voiceprint information of multiple similar speakers, improving the accuracy and robustness of target speaker feature retrieval. At the same time, the adaptive weighting mechanism enables the system to automatically identify and enhance the target speaker without pre-registration, adapting to changing interpersonal interaction scenarios and improving the adaptive capability of the speech separation system.

[0049] Based on the above technical solution, as an embodiment, step S130 may include: determining the target memory features corresponding to the specified instruction from the speaker's memory bank according to the received specified instruction; and determining the attention key and attention value based on the target memory features.

[0050] The system allows users to specify a target speaker. Based on the received specified instruction, the system determines the target memory features corresponding to the instruction from the speaker memory bank. A specified instruction is an operation command initiated by the user to explicitly identify the target speaker, carrying the target speaker's identification information. The triggering methods for a specified instruction include, but are not limited to, any of the following: the user selects the target speaker via a physical button on the device, the user specifies the target speaker via voice command, or the user selects the target speaker through the accompanying application interface.

[0051] Upon receiving a specified instruction, the system directly searches the speaker memory database for the memory feature corresponding to the target speaker identifier carried in the instruction, and identifies it as the target memory feature. Unlike the automatic retrieval mode, the specified mode does not require calculating the similarity between the query vector and the memory feature; instead, the user directly specifies the target speaker. Therefore, the determination of the target memory feature is more accurate and is not affected by similarity calculation errors.

[0052] In the designated mode, since the target memory features are explicitly specified by the user, the key portion of the target memory features can be directly determined as the attention key, and the value portion as the attention value, without the need for additional calculations such as weighted fusion. This direct determination method has minimal computational overhead and is suitable for scenarios where the identity of the target speaker is known, such as when a user explicitly selects to enhance the voice of a participant in a conference call.

[0053] The technical solution adopted in this application directly determines the target memory features by specifying instructions and generates attention keys and attention values ​​accordingly, avoiding matching errors caused by similarity calculation. It achieves more accurate speaker feature retrieval in scenarios where the identity of the target speaker is known. At the same time, the calculation process is simple and the overhead is extremely low, which is conducive to achieving low-latency real-time speech separation on resource-constrained edge devices.

[0054] Based on the above technical solution, as an embodiment, step S140 may include: calculating a similarity score based on the query vector and the attention key, and correcting the similarity score by using the environment context vector as a bias term to obtain a corrected attention score; and performing a weighted summation of the corrected attention score and the attention value to obtain the attention map of the current separation block.

[0055] The similarity score measures the degree of matching between the query vector and the attention key, reflecting the correlation between the input features of the current separation block and the target memory features in the speaker's memory bank. The similarity score is calculated using a scaled dot product, where the query vector and attention key are multiplied by the square root of their dimensions to prevent the dot product from becoming too large and causing gradient vanishing. It should be noted that this application does not limit the specific calculation method for similarity; other similarity calculation methods such as additive attention and bilinear attention can also be used.

[0056] The similarity score is corrected by using the context vector as a bias term to obtain the corrected attention score. The bias term is an additional amount superimposed on the original calculation result to introduce additional conditional information to adjust the original result. In this embodiment, the context vector, after linear mapping, is superimposed as an additive bias on the dot product similarity score of the query vector and the attention key.

[0057] For example, when the device detects that a user is turning their head, the bias value corresponding to the environmental context vector guides the model to pay more attention to the similarity score of sound source features that are consistent with the spatial information of the head-turning direction, thus increasing their relative value; while for the similarity score of interfering sound source features that are inconsistent with the head-turning direction, the bias value decreases their relative value. In this way, motion state information is injected into the similarity evaluation process of attention calculation, so that the corrected attention score not only reflects the inherent correlation between the query vector and the attention key, but also incorporates the influence of the current device motion state on the spatial distribution of sound sources.

[0058] The corrected attention scores are subjected to Softmax normalization, transforming each similarity score into a probability distribution with a sum of 1 as a weighting coefficient. Then, the attention values ​​are weighted and summed using this weighting coefficient to obtain the attention map of the current separation block. The values ​​of each dimension in the attention map reflect the correlation strength between each time step in the input features and the target speaker features. Time steps with higher correlation strength receive larger response values ​​in subsequent attention maps.

[0059] By using the technical solution of this application embodiment, the similarity score between the query vector and the attention key is corrected by using the environmental context vector as a bias term, so that motion state information can directly affect the attention mechanism's attention weight allocation for each sound source feature. In scenarios where the device's motion state changes, the attention calculation can adaptively adjust the focusing direction, thereby improving the extraction accuracy of target speaker features and the robustness of speech separation in dynamic scenarios such as walking and turning the head.

[0060] Based on the above technical solution, as an embodiment, step S150 may include: performing activation processing on the attention map to obtain a gating mask; multiplying the gating mask element-wise with the convolutional features of the current separation block to obtain the gating enhancement features corresponding to the current separation block; wherein, in the gating enhancement features, the target speaker features are enhanced and the non-target speaker features are suppressed.

[0061] The attention map is activated to obtain a gating mask. Activation processing refers to the transformation operation that maps input data to a specific range of values ​​using an activation function. In this embodiment, the activation processing can use the Sigmoid activation function, which is a non-linear function that maps any real number to a range between 0 and 1. The closer the output value is to 1, the more important the corresponding feature is; the closer it is to 0, the less important the corresponding feature is. After the attention map is processed by the Sigmoid activation function, the values ​​of each dimension are normalized to between 0 and 1, resulting in a gating mask. The values ​​of each element in the gating mask reflect the degree of preservation of the corresponding time step: time steps related to the target speaker, due to the high attention response value obtained during the attention calculation, have a gating value close to 1 after Sigmoid activation; time steps related to non-target speakers and noise, due to the low attention response value, have a gating value close to 0 after Sigmoid activation. It should be noted that the specific method of activation processing is not limited in the embodiments of this application. Other activation functions such as Hard Sigmoid and Swish, which can map the output to the range of 0 to 1, can also be used.

[0062] The gating mask is multiplied element-wise with the convolutional features of the current splitting block to obtain the gated enhancement features corresponding to the current splitting block. Element-wise multiplication refers to the operation of multiplying the element-wise values ​​at each corresponding position of two feature vectors with the same dimension. The gating mask and the convolutional features have the same dimension, and the gating value at each position in the gating mask is multiplied with the feature value at the corresponding position in the convolutional feature. For the time step where the target speaker features are located, the gating value is close to 1, and the feature value remains basically unchanged or even gains preservation enhancement after multiplication; for the time steps where non-target speaker features and noise are located, the gating value is close to 0, and the feature value is significantly attenuated or even approaches zero after multiplication.

[0063] Through the element-wise multiplication operation described above, components related to the target speaker in the convolutional features are selectively preserved and enhanced, while components related to interfering speakers and environmental noise are effectively suppressed, thus achieving enhanced separation of the target speaker at the feature level. The resulting gated enhanced features have a higher target speaker signal-to-noise ratio compared to the original convolutional features, providing a cleaner feature input for subsequent residual connections and progressive purification.

[0064] The technical solution of this application generates a gating mask with values ​​between 0 and 1 by activating the attention map, and then multiplies the gating mask element-wise with the convolutional features. At the feature level, it achieves selective enhancement of the target speaker features and effective suppression of noise and irrelevant speaker features. The calculation method is simple and efficient, and does not require the introduction of additional network parameters, which is conducive to achieving low-latency real-time speech separation processing on resource-constrained edge devices.

[0065] Based on the above technical solution, as an embodiment, step S160 may include: adding the residuals of the gated enhancement features of each separation block to the input features of the separation block and performing layer normalization to obtain the input features of the next separation block; determining the output features of the last separation block as the final features; and reconstructing the final features back to the time domain waveform to obtain the enhanced speech signal of the target speaker.

[0066] The gated enhancement features of each splitting block are residually summed with the input features of that block, followed by layer normalization, to obtain the input features of the next splitting block. Residual summation refers to the element-wise addition of the output of the current processing module to its input. Its purpose is to preserve useful information from the original input while mitigating the vanishing gradient problem in deep network training, ensuring that gradients flow smoothly between multiple splitting blocks. Layer normalization involves normalizing the mean and variance of the features along the channel dimension, stabilizing the feature distribution. Layer normalization accelerates the model's convergence process and improves training stability.

[0067] In this process, the gated enhancement feature of each separation block is first added to the residual of its input feature. The result is then normalized layer by layer and used as the input feature for the next separation block. This creates a chain-like transmission structure between the separation blocks: the input feature of the first separation block is the environmentally aware audio feature. The first separation block performs convolution, memory retrieval, attention calculation, and gated modulation on the input feature to obtain the gated enhancement feature. This gated enhancement feature is then added to the residual of the first separation block's input feature and normalized layer by layer to obtain the input feature for the second separation block. The second separation block repeats the same process for its input feature, and so on, until the last separation block completes its processing. Through this chain-like transmission method using residual connections, each separation block further refines the result based on the previous separation block, enhancing the target speaker component in the input feature layer by layer and suppressing interference components layer by layer.

[0068] The output feature of the last separation block is determined as the final feature. The final feature refers to the feature representation obtained after step-by-step purification by all separation blocks. It combines the layer-by-layer enhancement effect of multiple separation blocks on the target speaker features at different receptive field scales and has higher target speaker purity than the output of the first separation block.

[0069] The final features are reconstructed back to the time-domain waveform to obtain the enhanced speech signal of the target speaker. Reconstruction refers to the process of inversely transforming the feature representation in the feature space back to the original signal domain. In one embodiment, a one-dimensional transposed convolutional layer is used to reconstruct the final features. The one-dimensional transposed convolutional layer is the inverse operation of the one-dimensional convolutional layer in the encoding stage. Through a learnable filter, the high-dimensional features are accurately mapped back to the time-domain waveform space to obtain the enhanced speech signal of the target speaker. It should be noted that the embodiments of this application do not limit the reconstruction method, and other time-domain reconstruction methods such as inverse short-time fourier transform (iSTFT) can also be used.

[0070] The technical solution of this application achieves chain-like feature transfer between separation blocks through residual addition and layer normalization, which not only ensures the training stability and gradient flow of the deep network, but also continuously refines the target speaker features in the step-by-step transfer. The output of the last separation block is determined as the final feature, which combines the purification effects of multiple separation blocks at different time scales. The final feature is reconstructed into a time-domain waveform through a one-dimensional transposed convolutional layer, and an enhanced speech signal that can be played directly is directly output, realizing end-to-end speech separation from the feature space to the signal domain.

[0071] Based on the above technical solution, as an embodiment, the speech separation method may further include: in response to detecting a speech segment that meets preset speech quality conditions and preset novelty conditions, performing secondary verification on the speech segment, the secondary verification including extending the observation window and comparing it again with the speaker memory bank; after the secondary verification is passed, extracting the feature signature of the speech segment; generating memory features based on the feature signature, and storing the memory features in the speaker memory bank.

[0072] Figure 3 This is a schematic diagram of a process for updating the speaker's memory bank according to an embodiment of this application; see reference. Figure 3When a sustained clear speech segment is detected, it can be determined whether the speech segment meets preset speech quality conditions and preset novelty conditions. The preset speech quality conditions refer to the criteria used to determine whether a speech segment is clear enough and suitable for extracting reliable voiceprint features. In one embodiment, the preset speech quality conditions include: detecting a clear speech segment lasting a certain duration (e.g., 1 second), and the signal-to-noise ratio (SNR) and harmonic-to-noise ratio (HNR) of the speech segment are both higher than preset quality thresholds. SNR is the ratio of the target speech signal power to the background noise power, used to measure the clarity of the speech; HNR is the ratio of the harmonic component power to the non-harmonic noise power in the speech, used to measure the purity of the speech quality. The preset novelty conditions refer to the criteria used to determine whether the speaker corresponding to the speech segment already exists in the speaker's memory bank. In one embodiment, the preset novelty condition is: the maximum cosine similarity between the feature signature of the speech segment and all existing memory features in the speaker's memory bank is lower than a preset novelty threshold (e.g., 0.75). When the maximum cosine similarity is below this threshold, it indicates that the speaker corresponding to the speech segment differs significantly from existing speakers in the feature space, belonging to a new speaker not yet included in the memory. Only speech segments that simultaneously meet the preset speech quality condition and the preset novelty condition will trigger the subsequent secondary confirmation process.

[0073] Secondary verification refers to a review process conducted after a speech segment has been initially determined to meet the registration requirements. This is done by extending the observation period and re-comparing the segments. The purpose is to avoid erroneous registration of new speakers due to transient noise interference or variations in the speech signal. Extending the observation window involves increasing the duration of continuous speech observation from the initial short duration to a longer duration (e.g., from 1 second to 3-5 seconds) to obtain more stable and sufficient speech information. After extending the observation window, the feature signature of the speech segment is re-extracted, and its similarity to all existing memory features in the speaker's memory bank is recalculated. If the recalculated maximum similarity is still below the novelty threshold, the secondary verification passes; if the recalculated maximum similarity is above the novelty threshold, the speech segment is deemed to have too high a similarity to existing speakers, the secondary verification fails, and registration is abandoned.

[0074] After secondary confirmation, feature signatures of the speech segments are extracted. A feature signature is a fixed-dimensional embedding vector extracted from a speech segment, uniquely representing the speaker's voiceprint characteristics. In one embodiment, the feature signature is extracted by a lightweight feature extraction network from stable segments of the encoder output features. These stable segments are determined by Voice Activity Detection (VAD), and the feature signature has a 128-dimensional dimension. This lightweight feature extraction network is trained together with the main separation network, enabling the feature signature to have good discriminative power in the embedding space. The feature signature is used as the core content of the memory features, and auxiliary information related to the feature signature can also be associated and stored, all written into the speaker's memory bank. After storage, the new speaker's voiceprint characteristics can be retrieved and utilized in subsequent speech separation processing.

[0075] The technical solution adopted in this application ensures that only clear speech segments belonging to new speakers enter the registration process through dual gating of preset speech quality conditions and preset novelty conditions; by extending the observation window for secondary confirmation, it effectively avoids false registration caused by transient noise or speech variation, ensuring the quality and reliability of memory features in the speaker's memory bank, and enabling the system to automatically and accurately learn the voiceprint features of new speakers during dialogue.

[0076] Based on the above technical solution, as an embodiment, a confidence score and usage frequency count can also be maintained for each memory feature in the speaker's memory bank; when the speaker's memory bank is full, the memory features are eliminated according to the usage frequency count.

[0077] The confidence score is a numerical indicator used to characterize the reliability of a corresponding memory feature, reflecting the credibility of that memory feature as a speaker's voiceprint representation. The initial value of the confidence score is determined when the memory feature is added to the database based on the speech quality indicators (such as signal-to-noise ratio and harmonic noise ratio) of the speech segment from which the feature signature is extracted. The higher the speech quality indicator, the higher the initial confidence score. In subsequent use, whenever the memory feature is retrieved and used to generate attention keys and attention values, if the quality assessment of the separation result is positive, the confidence score of the memory feature is increased accordingly; if the separation result is abnormal or actively corrected by the user, the confidence score of the memory feature is decreased accordingly.

[0078] Usage frequency count refers to a counter that records the cumulative number of times a corresponding memory feature is retrieved and hit during speech separation processing. Each time a memory feature is selected as a target memory feature in automatic retrieval mode or specified by the user as a target memory feature in designated mode, its usage frequency count is incremented by one. The usage frequency count reflects the activity level of the speaker corresponding to that memory feature in the current interactive scenario; the higher the usage frequency of a memory feature, the more frequently the speaker has appeared in recent interactions.

[0079] The speaker memory bank employs a fixed-capacity design, meaning it can only hold a preset number of memory feature entries, such as a maximum of 20 speaker signatures. This fixed-capacity design is to accommodate the limited storage space of resource-constrained edge devices like headphones and speakers, preventing the memory bank from expanding indefinitely over time. When a new speaker feature needs to be added but the memory bank is full, the system sorts the memory features based on their usage frequency counts, removing the memory feature with the lowest usage frequency count to free up storage space for the new feature. The speaker corresponding to the lowest usage frequency feature is the least active in the current interaction scenario, and its removal has the least impact on the system's separation performance. Through this frequency-based removal mechanism, the speaker memory bank can always retain the most active speaker features in the current interaction scenario, automatically adapting to the dynamic changes of the interaction group.

[0080] By adopting the technical solution of this application embodiment, by maintaining confidence scores and usage frequency counts for each memory feature, quantitative management of the quality and activity of each entry in the memory bank is realized; when the memory bank is full, it is eliminated based on the usage frequency count, and the most active speaker features are automatically retained under the constraint of fixed storage space, so that the memory bank can dynamically adapt to the changes of the interaction group, while avoiding the infinite growth of storage space and adapting to the limited resource conditions of edge devices.

[0081] Based on the above technical solution, as an embodiment, the training steps of the separation network may include at least: training a basic separation network without the speaker memory bank based on the signal-to-noise ratio loss function; and, under the condition that the parameters of the basic separation network are fixed, introducing a speaker memory bank module and fine-tuning the basic separation network based on the memory contrast loss function to obtain the trained separation network.

[0082] The separation block is a module of the separation network, which includes an encoder, a separator composed of multiple stacked separation blocks, and a decoder. During the training phase, the input to the separation network is the mixed speech signal from the training dataset, and the output is the separated target speaker's speech signal. The training data is constructed using a large-scale simulated overlapping speech dataset. During the simulation, various environmental noises, reverberation conditions, and simulated IMU motion pattern data are superimposed to enable the trained separation network to generalize in diverse acoustic scenarios and motion states.

[0083] The signal-to-noise ratio (SNR) loss function is a loss function used to measure the difference in SNR between the separated speech signal and the target reference signal. In this embodiment, scale-invariant signal-to-noise ratio (SI-SNR) loss is used. SI-SNR loss is an SNR metric that is insensitive to signal amplitude scaling. Its calculation process is as follows: First, both the separated output signal and the target reference signal are zero-mean processed. Then, the projection component of the target reference signal onto the direction of the separated output signal is calculated as the target signal component. The difference between the separated output signal and the target signal component is taken as the noise component. Finally, the negative logarithm of the ratio of the target signal component power to the noise component power is taken as the loss value. The smaller the loss value, the better the separation effect.

[0084] In this training phase, the speaker memory module in the separation network is removed, and the attention calculation of the separation block degenerates into a standard self-attention mechanism. This means that the query vector, attention key, and attention value are all generated by the input features themselves, without interaction with an external memory. By minimizing the scale-invariant signal-to-noise ratio loss on large-scale training data, all parameters of the encoder, separator, and decoder of the basic separation network are trained end-to-end, enabling the basic separation network to learn the fundamental ability to separate the target speaker's speech from mixed speech. The training objective of this phase is to achieve stable speech separation performance in the basic separation network, providing a good parameter initialization foundation for the subsequent introduction of the speaker memory module.

[0085] With the parameters of the basic separation network fixed, a speaker memory module is introduced. The basic separation network is then fine-tuned using a memory-contrast loss function to obtain a trained separation network. Parameter fixing means that during the fine-tuning training phase, the parameters of the encoder and decoder in the basic separation network are no longer updated; only the parameters related to the newly introduced speaker memory module and the parameters in the separator related to memory interaction are updated. This partial parameter freezing fine-tuning strategy maintains the stable separation capability already learned by the basic separation network, while allowing the new module to focus on learning memory retrieval and update strategies, avoiding parameter conflicts and performance regressions that may occur during full-parameter joint training. The memory-contrast loss function is a loss function used to constrain the distribution characteristics of feature signatures in the speaker memory in the embedding space.

[0086] Specifically, the optimization objective of the memory contrastive loss function includes two aspects: making the feature signatures generated by the same speaker in different sentences as close as possible in the embedding space, and making the feature signatures of different speakers as far apart as possible in the embedding space. In this embodiment of the invention, the memory contrastive loss function adopts the framework of contrastive learning, treating the feature signatures extracted from different speech segments of the same speaker as positive sample pairs and the feature signatures of different speakers as negative sample pairs. The parameters of the feature extraction network are optimized by minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs. Through training with the memory contrastive loss function, the feature signatures in the speaker memory bank form a clear clustering structure in the embedding space. Features of the same speaker cluster in adjacent regions, while features of different speakers are separated from each other. This enables the accurate finding of the memory feature that best matches the current speaker from the memory bank when performing memory retrieval using cosine similarity during the inference stage. After fine-tuning the training, a complete separation network including the speaker memory bank module is obtained. This network has both the speech separation capability of the basic separation network and the ability to perform adaptive speaker recognition and enhancement through the speaker memory bank.

[0087] The technical solution adopted in this application uses a phased training strategy. In the first phase, the basic separation network is trained using scale-invariant signal-to-noise ratio loss, enabling the network to first obtain stable basic speech separation capabilities. In the second phase, under the condition of fixed basic network parameters, a speaker memory bank module is introduced and fine-tuned using memory contrast loss. This avoids the instability of full-parameter joint training and enables the feature signature to form a good discriminative distribution in the embedding space, improving the accuracy of memory retrieval. The resulting separation network has both high-quality speech separation capabilities and adaptive speaker enhancement capabilities.

[0088] Based on the above technical solutions, as an example, an optimized design was carried out for the deployment and online adaptive operation of the separation network at the edge device, including three aspects: model lightweighting, low-power operation, and online fine-tuning.

[0089] In terms of model lightweighting, knowledge distillation can be used to compress the separation network. Knowledge distillation refers to a model compression method that transfers knowledge learned by a teacher network with a large number of parameters to a student network with a smaller number of parameters. The teacher network is a separation network with a large number of parameters and high separation performance, while the student network is a lightweight separation network in this embodiment. During the distillation process, the distribution difference between the output of the student network and the output of the teacher network is used as the distillation loss, and the student network is trained using a combined signal-to-noise ratio loss function, so that the student network approximates the separation performance of the teacher network under the condition of significantly reducing the number of parameters. Based on knowledge distillation, model pruning and quantization techniques can be further used to compress the model. Model pruning refers to the operation of removing redundant parameters or channels in the network that contribute little to the output to reduce the model size. Quantization refers to the technique of reducing the numerical precision of model parameters and calculation processes from high-precision floating-point numbers to low-precision integers. In the embodiments of this application, INT8 quantization can be used, which converts the model parameters from 32-bit floating-point numbers to 8-bit integer representations, compressing the model size to one-quarter of the original size. At the same time, inference calculations can be accelerated by utilizing the integer arithmetic units of the digital signal processor (DSP) or neural processing unit (NPU) on the edge device, significantly reducing inference latency and power consumption.

[0090] For low-power operation, an adaptive computing mechanism was designed. This mechanism dynamically adjusts the allocation of system computing resources based on the complexity of the current acoustic environment. Specifically, the system monitors the current acoustic environment in real time. When it detects a silent environment or only one person speaking, the system enters a low-power monitoring mode. In this mode, only speech activity detection and simple feature extraction are performed, while the full separation network remains dormant, significantly reducing system power consumption. When multiple overlapping voices or a new speaker are detected, the system automatically activates the full separation network for speech separation processing. Through this adaptive computing mechanism, the system operates in low-power monitoring mode most of the time during daily use, only activating full computing resources when speech separation is required, effectively extending the battery life of edge devices.

[0091] Regarding online fine-tuning, the system supports personalized parameter optimization on the device based on limited user feedback. Online fine-tuning refers to the operation of updating some parameters of the model on a small scale using local data after deploying the separation network on the device. Specifically, when users report unsatisfactory enhancement effects through interactive means, the system, while protecting user privacy, uses a small amount of local speech data to fine-tune the feature signatures in the speaker's memory bank or some parameters of the fusion network. During the fine-tuning process, only a small number of parameters are updated, not all network parameters, thus achieving personalized optimization for specific users or usage scenarios within a very short training time. Since the fine-tuning process is completed entirely locally on the device, user speech data does not need to be uploaded to the cloud, ensuring user privacy and security. Through online fine-tuning, the system can continuously adapt to users' personalized needs during use, gradually improving the speech separation effect in specific scenarios.

[0092] Figure 4 This is a schematic diagram of the data flow of a speech processing method provided in an embodiment of this application; see reference. Figure 4 The system's data input includes two signals. The first signal is audio data, acquired by the device's built-in microphone array. This audio data undergoes audio preprocessing, which includes framing, windowing, and short-time Fourier transform to convert the time-domain audio signal into a frequency-domain representation. The second signal is motion sensing data, acquired by the device's built-in IMU sensor. This motion sensing data undergoes IMU preprocessing, which includes low-pass filtering to remove high-frequency noise, calibration of multi-axis data to eliminate sensor bias, and time alignment with the audio data to eliminate timing discrepancies between the two signals.

[0093] After preprocessing, the two data streams enter the feature extraction and environment analysis stage. In this stage, feature extraction is performed on the audio features output from the audio preprocessing and the motion sensing features output from the IMU preprocessing, and the environmental context vector is fused with the audio features to obtain the environmentally perceptual audio features. The feature extraction and environment analysis stage is a crucial step in multimodal data fusion, enabling early fusion of motion state information and audio information.

[0094] The environmentally aware audio features and environmental context vectors output from the feature extraction and environmental analysis stages are fed into a lightweight separation network. The separation network consists of an encoder, a separator containing multiple separation blocks, and a decoder. Within the separation network, each separation block performs memory enhancement processing through bidirectional interaction with the speaker's memory. Specifically, the separation block sends a query vector to the speaker's memory to retrieve matching target memory features. The speaker's memory returns the retrieved target memory features to the separation block for generating attention keys and attention values. The separation block then performs attention calculations and gating modulation based on the attention keys and attention values. The speaker's memory supports dynamic updates during inference. When a new speaker's speech segment that meets the registration criteria is detected, its feature signature is written into the memory, achieving adaptive learning for new speakers.

[0095] The final features output by the separation network enter the post-processing and waveform reconstruction stage. Post-processing and waveform reconstruction include necessary smoothing and spectral enhancement operations on the final features. Then, a one-dimensional transposed convolutional layer reconstructs the high-dimensional features back into a time-domain waveform, obtaining the enhanced speech signal of the target speaker. The final output enhanced speech signal can be directly played to the user through the device's speakers or headphones, realizing real-time separation and enhancement of the target speaker's speech on resource-constrained edge devices such as headphones and speakers.

[0096] Figure 5 This is a schematic diagram of a multimodal feature fusion strategy provided in an embodiment of this application; Figure 5 This demonstrates the fusion paths of environmental context vectors at three different levels within the system. The top-level input is the environmental context vector obtained from IMU analysis, which enters three pathways: feature fusion, model fusion, and decision fusion. Feature fusion concatenates and fuses the context vector with audio features in the encoder stage to produce environmentally perceptive audio features. Model fusion injects the context vector as a bias into the attention mechanism within the separation network to enhance focus on the target speaker. Decision fusion dynamically adjusts processing parameters based on the context in the post-processing stage. The three pathways ultimately converge on the "scene classification and policy matching" module, employing corresponding processing strategies for different motion scenarios (stationary, walking, head turning, and changes in wearing).

[0097] In the feature fusion stage, the environmental context vector and the audio features output by the encoder are concatenated along the feature dimension. Then, dimensionality reduction and feature fusion are performed through a fusion convolutional layer to obtain environmentally perceptive audio features. This stage belongs to early fusion, introducing motion state information at the very beginning of the speech separation process. This ensures that the input features of all subsequent processing modules carry contextual information about the device's motion state, providing a motion-aware feature foundation for the entire separation network.

[0098] In the model fusion stage, the environmental context vector, after linear mapping, is injected as a bias term into the attention calculation process of each separation block in the separation network. Specifically, when calculating the similarity score between the query vector and the attention key, the environmental context vector is used as an additive bias to correct the similarity score, thereby guiding the attention mechanism to adjust the attention weights for different sound source features according to the current motion state. For example, when the device detects that the user is running and generates high-frequency jitter, this bias will prompt the network to be more robust to phase changes and adjust the attention mechanism's dependence weights on spatial cues, enabling the separation network to accurately focus on the target speaker even under motion interference. This stage belongs to model-level fusion, deeply embedding motion state information into the core calculation of the separation network. Compared with the method of fusion only at the input end, it can continuously utilize motion context information in each layer of processing within the model.

[0099] During the decision fusion stage, post-processing parameters are dynamically adjusted based on the environmental context vector to achieve adaptive output-level fusion. Specific strategies can be matched according to scene classification results; for example, in a stationary state, standard processing weights are used without additional parameter adjustments; in a walking motion state, the utilization weights of spatial cues are enhanced to compensate for changes in the relative position of sound sources caused by walking; in a rapid head-turning scenario, the similarity threshold for memory bank retrieval is temporarily increased to avoid misclassifying feature variations caused by angle changes in the same speaker as a new speaker; when the wearing status changes, changes in the acoustic transfer function are inferred from gyroscope data, and calibration parameters are switched to the corresponding wearing mode to fine-tune the speech reconstruction processing. Through the above scene classification and strategy matching mechanism, the system can adaptively select the optimal processing parameters according to different motion scenarios, further improving the speech separation effect in complex scenarios.

[0100] Through the coordinated efforts of feature fusion, model fusion, and decision fusion, the environmental context vector is fully utilized from the input end to the model and then to the output end, forming a complete multimodal fusion strategy system. Feature fusion provides motion-aware input features for the separation network, model fusion continuously guides attention focus during the separation process, and decision fusion performs scene-adaptive optimization in the post-processing stage. The three work together to enable the system to maintain high-quality speech separation performance under various motion states.

[0101] To facilitate better implementation of the speech processing method of this application, this application also provides a speech processing apparatus based on the above-described speech processing method. The meanings of the terms used are the same as in the speech processing method described above, and specific implementation details can be found in the descriptions of the method embodiments.

[0102] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of the voice processing device provided in the embodiments of this application, wherein the voice processing device includes: The data acquisition module 601 is used to acquire audio data and motion sensing data, and to extract and fuse features from the audio data and motion sensing data to obtain environmental context vectors and environmentally perceived audio features. The feature input module 602 is used to sequentially connect multiple separation blocks of the environmentally aware audio feature input to obtain the convolutional features and query vectors of each separation block; The feature matching module 603 is used to determine the target memory features that match each of the query vectors from the speaker's memory bank, and to generate attention keys and attention values ​​based on the target memory features; Attention calculation module 604 is used to perform attention calculation on attention key and attention value based on the environment context vector and the query vector to obtain the attention graph of the current separation block; The gate modulation module 605 is used to perform gate modulation on the convolutional features of the current separation block based on the attention map to obtain the gated enhancement features of the current separation block; The speech acquisition module 606 is used to obtain the enhanced speech signal of the target speaker based on the gating enhancement features of the multiple separation blocks.

[0103] In one embodiment, the attention calculation module 604 is specifically used to perform: A similarity score is calculated based on the query vector and the attention key, and the environmental context vector is used as a bias term to correct the similarity score, resulting in a corrected attention score. The attention map of the current separated block is obtained by weighted summation of the corrected attention score and the attention value.

[0104] In one embodiment, the gated modulation module 605 is specifically used to perform: The attention map is activated to obtain a gating mask; The gate mask is multiplied element-wise with the convolutional features of the current separation block to obtain the gated enhancement features corresponding to the current separation block; wherein, in the gated enhancement features, the target speaker features are enhanced and the non-target speaker features are suppressed.

[0105] In one embodiment, the voice acquisition module 606 is specifically used to perform: The gated enhancement features of each separation block are added to the input features of that separation block by residual summation and layer normalization to obtain the input features of the next separation block. The output feature of the last separated block is determined as the final feature; The final features are reconstructed back into a time-domain waveform to obtain the enhanced speech signal of the target speaker.

[0106] In one embodiment, the feature input module 602 is specifically used to perform: The environmentally aware audio features are used as the input features for the first separation block; The input features of each of the separation blocks are subjected to causal convolution processing to obtain the convolution features of each of the separation blocks; wherein, the receptive field of each of the separation blocks increases progressively with the order of the separation blocks. Based on the convolutional features of each of the separated blocks, a query vector for each of the separated blocks is generated.

[0107] In one embodiment, the feature matching module 603 is specifically used to perform: Based on the similarity between the query vector and each memory feature in the speaker's memory bank, at least one target memory feature is selected from the speaker's memory bank; Based on the similarity corresponding to the target memory feature, the target memory feature and the query vector are adaptively weighted and fused to obtain the attention key and the attention value.

[0108] In one embodiment, the feature matching module 603 is specifically used to perform: Based on the received specified instruction, determine the target memory feature corresponding to the specified instruction from the speaker's memory bank; The attention key and the attention value are determined based on the target memory features.

[0109] In one embodiment, the data acquisition module 601 is specifically used to perform: Time alignment is performed on the audio data and the motion sensing data; Feature extraction is performed on the time-aligned audio data to obtain audio features; Feature extraction is performed on the time-aligned motion sensing data to obtain motion sensing features; The motion sensing features are processed using a multilayer perceptron to obtain the environmental context vector; The environmental context vector and the audio features are fused to obtain the environmentally perceptive audio features.

[0110] In one embodiment, the voice processing device further includes: A secondary confirmation module is used to perform secondary confirmation on a speech segment that meets preset speech quality conditions and preset novelty conditions in response to the detection of such a segment. The secondary confirmation includes extending the observation window and comparing it again with the speaker's memory bank. The signature extraction module is used to extract the feature signature of the voice segment after the secondary confirmation is passed; The feature storage module is used to generate memory features based on the feature signature and store the memory features in the speaker's memory bank.

[0111] In one embodiment, the voice processing device further includes: The maintenance module is used to maintain confidence scores and usage frequency counts for each memory feature in the speaker's memory bank; The elimination module is used to eliminate the memory features based on the usage frequency count when the speaker's memory bank is full.

[0112] In one embodiment, the separation block is a module of a separation network; the training steps of the separation network include: A basic separation network that does not include the speaker's memory bank is trained based on the signal-to-noise ratio loss function; With the parameters of the basic separation network fixed, a speaker memory bank module is introduced, and the basic separation network is fine-tuned based on the memory contrast loss function to obtain the trained separation network.

[0113] The technical solution of this application embodiment extracts and fuses features from audio data and motion sensing data to obtain environmental context vectors and environmentally aware audio features. This achieves early fusion of motion state information and audio information, enabling subsequent processing to perceive changes in the user's motion state and improving the robustness of speech separation in dynamic scenarios such as changes in wearing method and user movement. The environmentally aware audio features are input into multiple sequentially connected separation blocks to obtain convolutional features and query vectors for each separation block. Utilizing the hierarchical concatenation structure of multiple separation blocks, the receptive field gradually expands with processing depth, enabling the modeling of speech signal dependencies at different time scales. Simultaneously, each separation block progressively refines features, ensuring separation accuracy while maintaining low computational overhead. Target memory features matching each query vector are determined from the speaker memory bank, and attention keys and attention values ​​are generated based on these target memory features. The speaker memory bank is then used for each query... It provides prior information related to the target speaker, enabling adaptive speaker recognition and enhancement without pre-training or pre-registration of the target speaker. Based on the environmental context vector and query vector, it performs attention calculation on attention keys and attention values ​​to obtain the attention map of the current segmentation block. Motion environment information is introduced into the attention calculation process, guiding the model to accurately focus on the features of the target speaker even in complex motion scenarios. Based on the attention map, it performs gated modulation on the convolutional features of the current segmentation block to obtain the gated enhancement features of the current segmentation block. This enhances the features of the target speaker and suppresses noise and irrelevant speaker features at the feature level, improving the signal-to-noise ratio. Based on the gated enhancement features of multiple segmentation blocks, it obtains the enhanced speech signal of the target speaker. The result of the step-by-step purification of multiple segmentation blocks is restored to the clean speech of the target speaker, achieving high-quality real-time speech separation and enhancement on resource-constrained edge devices such as headphones and speakers.

[0114] For specific limitations regarding the speech processing device, please refer to the limitations on the speech processing method above, which will not be repeated here. Each module in the aforementioned speech processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0115] In addition, this application also provides an electronic device, such as Figure 7 As shown, it illustrates the structural diagram of the electronic device involved in this application, specifically: The electronic device may include components such as a processor 701 with one or more processing cores and a memory 702 with one or more computer-readable storage media. Those skilled in the art will understand that... Figure 7The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 701 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 702, and by calling data stored in the memory 702, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 701 may include one or more processing cores; preferably, the processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 701.

[0116] The memory 702 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 702 may also include a memory controller to provide the processor 701 with access to the memory 702.

[0117] In one embodiment, the electronic device further includes a power supply 703 that supplies power to the various components. Preferably, the power supply 703 can be logically connected to the processor 701 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 703 may also include one or more DC or AC power supplies, recharging systems, power equipment debugging circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0118] In one embodiment, the electronic device may further include an input unit 704, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0119] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 701 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 702 according to the following instructions, and the processor 701 runs the applications stored in the memory 702, thereby implementing the steps in any of the voice processing methods provided in the embodiments of this application.

[0120] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0121] In one embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the methods described in any embodiment of this application.

[0122] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of this application.

[0123] In some embodiments, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the methods described in any embodiment of this application.

[0124] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0125] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0126] Therefore, this application provides a computer-readable storage medium storing a computer program that can be loaded by a processor to execute the steps of any of the speech processing methods provided in this application.

[0127] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0128] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0129] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the speech processing methods provided in this application, the beneficial effects that any of the speech processing methods provided in this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0130] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0131] The above provides a detailed description of a speech processing method, apparatus, electronic device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A speech processing method, characterized in that, include: Acquire audio data and motion sensing data, extract and fuse features from the audio data and motion sensing data to obtain environmental context vectors and environmentally perceptual audio features; The environmentally aware audio features are input into a series of sequentially connected separation blocks to obtain the convolutional features and query vectors of each separation block. Target memory features matching each query vector are determined from the speaker's memory bank, and attention keys and attention values ​​are generated based on the target memory features; Based on the environment context vector and the query vector, attention is calculated on the attention key and attention value to obtain the attention graph of the current separation block; Based on the attention map, the convolutional features of the current separation block are gated and modulated to obtain the gated enhanced features of the current separation block; The enhanced speech signal of the target speaker is obtained based on the gating enhancement features of the multiple separation blocks.

2. The speech processing method according to claim 1, characterized in that, The step of performing attention calculation on the attention key and attention value based on the environment context vector and the query vector to obtain the attention graph of the current separation block includes: A similarity score is calculated based on the query vector and the attention key, and the environmental context vector is used as a bias term to correct the similarity score, resulting in a corrected attention score. The attention map of the current separated block is obtained by weighted summation of the corrected attention score and the attention value.

3. The speech processing method according to claim 1, characterized in that, The step of gating and modulating the convolutional features of the current split block based on the attention map to obtain the gated enhanced features of the current split block includes: The attention map is activated to obtain a gating mask; The gate mask is multiplied element-wise with the convolutional features of the current separation block to obtain the gated enhancement features corresponding to the current separation block; wherein, in the gated enhancement features, the target speaker features are enhanced and the non-target speaker features are suppressed.

4. The speech processing method according to claim 1, characterized in that, The step of obtaining the enhanced speech signal of the target speaker based on the gated enhancement features of multiple separated blocks includes: The gated enhancement features of each separation block are added to the input features of that separation block by residual summation and layer normalization to obtain the input features of the next separation block. The output feature of the last separated block is determined as the final feature; The final features are reconstructed back into a time-domain waveform to obtain the enhanced speech signal of the target speaker.

5. The speech processing method according to claim 4, characterized in that, The process of sequentially concatenating the environmentally aware audio features into multiple separate blocks to obtain the convolutional features and query vectors of each separate block includes: The environmentally aware audio features are used as the input features for the first separation block; The input features of each of the separation blocks are subjected to causal convolution processing to obtain the convolution features of each of the separation blocks; wherein, the receptive field of each of the separation blocks increases progressively with the order of the separation blocks. Based on the convolutional features of each of the separated blocks, a query vector for each of the separated blocks is generated.

6. The speech processing method according to claim 1, characterized in that, The step of determining target memory features matching each query vector from the speaker's memory bank and generating attention keys and attention values ​​based on the target memory features includes: Based on the similarity between the query vector and each memory feature in the speaker's memory bank, at least one target memory feature is selected from the speaker's memory bank; Based on the similarity corresponding to the target memory feature, the target memory feature and the query vector are adaptively weighted and fused to obtain the attention key and the attention value.

7. The speech processing method according to claim 1, characterized in that, The step of determining target memory features matching each query vector from the speaker's memory bank and generating attention keys and attention values ​​based on the target memory features includes: Based on the received specified instruction, determine the target memory feature corresponding to the specified instruction from the speaker's memory bank; The attention key and the attention value are determined based on the target memory features.

8. The speech processing method according to claim 1, characterized in that, The step of extracting and fusing features from the audio data and the motion sensing data to obtain environmental context vectors and environmentally perceptual audio features includes: Time alignment is performed on the audio data and the motion sensing data; Feature extraction is performed on the time-aligned audio data to obtain audio features; Feature extraction is performed on the time-aligned motion sensing data to obtain motion sensing features; The motion sensing features are processed using a multilayer perceptron to obtain the environmental context vector; The environmental context vector and the audio features are fused to obtain the environmentally perceptive audio features.

9. The speech processing method according to any one of claims 1 to 8, characterized in that, The speech processing method further includes: In response to the detection of a speech segment that meets preset speech quality conditions and preset novelty conditions, the speech segment is subjected to secondary verification, which includes extending the observation window and comparing it again with the speaker's memory bank; After the secondary confirmation is passed, the feature signature of the voice segment is extracted; Memory features are generated based on the feature signature, and the memory features are stored in the speaker's memory bank.

10. The speech processing method according to claim 9, characterized in that, The speech processing method further includes: Maintain confidence scores and frequency counts for each memory feature in the speaker's memory bank; When the speaker's memory bank is full, the memory features are eliminated based on the frequency of use count.

11. The speech processing method according to any one of claims 1 to 8, characterized in that, The separation block is a module of the separation network; The training steps for the separation network include: A basic separation network that does not include the speaker's memory bank is trained based on the signal-to-noise ratio loss function; With the parameters of the basic separation network fixed, a speaker memory bank module is introduced, and the basic separation network is fine-tuned based on the memory contrast loss function to obtain the trained separation network.

12. A voice processing device, characterized in that, include: The data acquisition module is used to acquire audio data and motion sensing data, and to extract and fuse features from the audio data and motion sensing data to obtain environmental context vectors and environmentally perceived audio features. The feature input module is used to sequentially connect multiple separation blocks of the environmentally aware audio feature input to obtain the convolutional features and query vectors of each separation block; The feature matching module is used to determine the target memory features that match each of the query vectors from the speaker's memory bank, and to generate attention keys and attention values ​​based on the target memory features; The attention calculation module is used to perform attention calculation on the attention key and attention value based on the environment context vector and the query vector to obtain the attention graph of the current separation block; The gated modulation module is used to perform gated modulation on the convolutional features of the current separation block based on the attention map to obtain the gated enhanced features of the current separation block; The speech acquisition module is used to obtain the enhanced speech signal of the target speaker based on the gating enhancement features of the multiple separation blocks.

13. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the speech processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the speech processing method as described in any one of claims 1 to 11.

Images