Multi-channel sound source separation method, device and equipment based on sound event guidance

Abstract

The application provides a multi-channel sound source separation method, device and equipment based on sound event guidance, and relates to the technical field of sound source separation and state monitoring.The application extracts time-frequency feature embedding as a first feature set through a layered self-attention mechanism after dividing windows, aligning features, Fourier transform and Mel filter conversion of collected multi-channel audio data;obtains a second feature set through a two-dimensional convolution layer mapping;extracts a third feature set through down-sampling of a residual encoder;then embeds the second and first feature sets into the third feature set in sequence to obtain a fused third feature set, and obtains a fourth feature set through up-sampling, and then embeds the second and first feature sets into the fourth feature set in sequence, so as to obtain a sound source separation result.The application can effectively solve the problems of low multi-channel audio processing precision and insufficient information utilization in the prior art, and significantly improves the performance of equipment state monitoring and fault early warning.

Description

Technical Field

[0001] This invention relates to the field of sound source separation and industrial equipment operation status monitoring technology, and more specifically, to a multi-channel sound source separation method, apparatus and equipment based on sound event guidance. Background Technology

[0002] In modern industrial production, monitoring the operational status and providing early warning of faults for industrial equipment are crucial for ensuring production efficiency and equipment safety. With the rapid development of industrial automation, multi-channel microphone data, capable of simultaneously acquiring acoustic signals from different spatial directions, has gradually become an important means of audio monitoring for industrial equipment. Compared to single-channel microphones, multi-channel microphones effectively avoid signal loss due to angle limitations, thus providing a richer source of information for monitoring the operational status of industrial equipment. However, despite the significant advantages of multi-channel audio data, existing deep learning-based methods for industrial equipment audio monitoring still exhibit obvious limitations in handling multi-channel audio source separation and equipment operational status monitoring.

[0003] Current technical solutions primarily rely on traditional audio processing methods or single-task deep learning models. These methods often struggle to fully utilize the spatiotemporal information in multi-channel audio data, resulting in insufficient accuracy in sound source separation and low accuracy in equipment operation status detection. Furthermore, existing sound source separation models typically fail to adequately consider the temporal characteristics of sound events and their impact on separation performance, limiting the robustness and stability of these models in complex industrial environments. Simultaneously, in equipment operation status monitoring tasks, existing methods do not comprehensively utilize multi-channel information, making it difficult to achieve refined analysis and real-time monitoring of the operation status of multiple devices. These problems severely restrict the development and application of industrial equipment audio monitoring technology.

[0004] In view of the above, this application is hereby submitted. Summary of the Invention

[0005] The present invention aims to provide a method, apparatus and device for multi-channel audio source separation based on sound event guidance, so as to solve the problems of low accuracy and insufficient information utilization in the existing technology in multi-channel audio processing, and to significantly improve the performance and stability of the model in tasks such as equipment status monitoring and fault early warning.

[0006] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0007] A multi-channel sound source separation method based on sound event guidance includes:

[0008] S1, divide the multi-channel audio data of the collected industrial equipment into windows one by one, and perform Fourier transform and Mel filtering on the feature-aligned data in each window to obtain the Mel spectrum;

[0009] S2, extract time-frequency feature embeddings from the Mel spectrum through a hierarchical self-attention mechanism as the first feature set; then gradually compress the length of the time-frequency feature embeddings and increase the feature dimension to obtain a time-frequency feature embedding token set;

[0010] S3, the time-frequency features are embedded into the dimension of the token set and mapped to the number of categories of device operating status through a two-dimensional convolutional layer, and then averaged and pooled in the frequency dimension to obtain a device operating status sound event map per event and frame, i.e., the second feature set.

[0011] S4, the multi-channel audio data is transformed by short-time Fourier transform to obtain a multi-channel time-frequency map, and the global low-frequency latent features of the multi-channel time-frequency map are extracted by downsampling of the residual encoder as a third feature set;

[0012] S5, a set of time-varying dynamic scaling parameters and offset parameters are generated for the second feature set through a linear layer with shared parameters, so as to embed the second feature set into the third feature set to obtain the third feature set after sound event embedding;

[0013] S6, the first feature set is weighted and fused by a learnable adaptive weighted neural network, and the weighted first feature set is projected onto the last dimension of the third feature set after the sound event is embedded. The fusion is performed by adding elements one by one to obtain the third feature set after the sound event is embedded and the feature is embedded.

[0014] S7. Input the third feature set after sound event embedding and feature embedding into the residual decoder. Extract local high-frequency latent features through upsampling as the fourth feature set. Then, fuse the second feature set and the first feature set with the fourth feature set in the same way as the third feature set to obtain the fourth feature set after sound event embedding and feature embedding, which is the sound source separation result.

[0015] Preferably, S1 specifically comprises:

[0016] The data of each channel of the multi-channel audio data is divided into windows according to a preset time length;

[0017] The data within each window is used to calculate the time difference of multi-channel audio using a cross-correlation function, and the data within each channel window is time-synchronized and aligned.

[0018] Perform a Fourier transform on the aligned time-domain data in each window to obtain preliminary frequency-domain features;

[0019] By using a set number of Mel filter banks, the linear frequencies of the initial frequency domain characteristics are converted into nonlinear Mel frequencies, generating a Mel spectrum to highlight key frequency components related to equipment failure.

[0020] Preferably, S2 specifically comprises:

[0021] The Mel spectrum is divided into feature blocks, and a position embedding is added to each feature block to obtain blocks of size (T / P*F / P,D); where T is the time dimension, P is the window size, F is the number of frequencies, and D is the feature dimension.

[0022] Each feature block is input into a multi-layer self-attention network, and time-frequency feature embeddings are extracted layer by layer as the first feature set.

[0023] The time-frequency feature embedding of each layer gradually compresses the time series length through feature reconstruction, while using a linear layer to increase the feature dimension, resulting in a time-frequency feature embedding token set.

[0024] Preferably, S5 specifically includes:

[0025] Let the second feature set be Where R is a real number, For the time dimension Classification of equipment operating status;

[0026] Let the third feature set be ,in, The time dimension of the third feature set The frequency count of the third feature set. For feature dimensions;

[0027] The resolution of the sound timemap for the corresponding device operating status category in the second feature set is adjusted using bilinear interpolation to align with the third feature set.

[0028] A time-varying dynamic scaling parameter is learned from a resolution-adjusted second feature set through a linear layer with shared parameters. With offset parameter These are used to control the intensity of the feature and adjust the feature distribution baseline, respectively. use Adjust the amplitude of the third feature set, using The feature distribution of the third feature set is corrected to embed the state probability information of the second feature into the third feature set, resulting in the third feature set after sound event embedding.

[0029] ; ;

[0030] in, Let R be the third feature set after embedding the sound events, and R be a real number. transpose

[0031] Preferably, time-varying dynamic scaling parameters are learned from the resolution-adjusted second feature set through a linear layer with shared parameters. With offset parameter Specifically:

[0032] ;

[0033] in, Represents a linear layer with shared parameters; This is the second feature set after resolution adjustment.

[0034] Preferably, the learnable adaptive weighted neural network adopts a single-layer fully connected layer structure. It learns an importance score for each feature in the first feature set through a linear layer, and after obtaining weights based on softmax normalization, calculates a weighted sum of features. Finally, it interpolates the weighted feature dimensions two-dimensionally to the feature dimensions corresponding to the third feature set after the sound event embedding, and fuses them element-wise to obtain the third feature set after the sound event embedding and feature embedding; specifically:

[0035] The first feature set The input is a fully connected layer, from which the importance score of each feature is obtained. The expression is:

[0036] ;

[0037] in, For the set of importance scores, This represents the score of the j-th feature in the i-th sample; This is the weight matrix; For bias terms;

[0038] The score vector of each sample is normalized using softmax to obtain the weight vector:

[0039] ;

[0040] in, This represents the weight of the j-th feature in the i-th sample; The feature dimension of the first feature set; This represents the importance score of the k-th feature dimension in the i-th sample;

[0041] The first feature set is multiplied element-wise by the weight vector to obtain the weighted features. ;

[0042] Weighted features The dimension is adjusted to match the dimension of the third feature set after the sound event is embedded through two-dimensional interpolation;

[0043] The interpolated weighted features are added element-wise to the third feature set after the sound event embedding, resulting in the third feature set after the sound event embedding and feature embedding.

[0044] Preferably, it further includes: performing multiple iterative optimizations on the sound source separation results, and adding an additional channel for feedback during the iterative optimization training process;

[0045] The additional channel is the (M+1)th channel, where M is the number of channels in the multi-channel audio data;

[0046] The input signal of the additional channel changes dynamically with the number of iterations, specifically:

[0047] During the initial iteration, the input signal of the additional channel is a mute signal, and the sound source is separated through the channels of the multi-channel audio data;

[0048] In subsequent iterations, the input signal of the additional channel is the historical sound source separation result obtained from the previous iteration; based on the historical sound source separation result and the multi-channel audio data input in the current iteration, the weight of feature extraction is adaptively adjusted to gradually enhance the recognition and separation capability of multi-channel audio data.

[0049] Preferably, it is also used for monitoring the operating status of industrial equipment, specifically:

[0050] Calculate the signal-to-noise ratio based on the sound source separation results;

[0051] When the signal-to-noise ratio reaches the set threshold, fault features are extracted from the sound source separation results to obtain the operation status monitoring results; wherein, the operation status monitoring results include normal or fault.

[0052] The present invention also provides a multi-channel audio source separation device based on sound event guidance, comprising:

[0053] The data acquisition and feature preprocessing unit is used to divide the multi-channel audio data of the acquired industrial equipment into windows one by one, and perform Fourier transform and Mel filtering on the feature-aligned data in each window to obtain the Mel spectrum.

[0054] The time-frequency feature extraction unit is used to extract time-frequency feature embeddings from the Mel spectrum through a hierarchical self-attention mechanism, which serve as the first feature set; then, the length of the time-frequency feature embeddings is gradually compressed and the feature dimension is increased to obtain a time-frequency feature embedding token set.

[0055] The sound event feature extraction unit is used to embed the time-frequency features into the dimension of the token set and map it to the number of categories of device operating status through a two-dimensional convolutional layer, and then average pool the frequency dimension to obtain a device operating status sound event map per event and frame, i.e., the second feature set.

[0056] The latent feature extraction unit is used to obtain a multi-channel time-frequency map from the multi-channel audio data through short-time Fourier transform, and to extract the global low-frequency latent features of the multi-channel time-frequency map by downsampling of the residual encoder, as a third feature set.

[0057] The feature-guided embedding unit uses a linear layer with shared parameters to generate a set of time-varying dynamic scaling and offset parameters for the second feature set, so as to embed the second feature set into the third feature set, resulting in the third feature set after sound event embedding;

[0058] The feature fusion unit is used to perform weighted fusion on the first feature set through a learnable adaptive weighted neural network, and project the weighted first feature set onto the last dimension of the third feature set after the sound event embedding. The fusion is performed by adding elements one by one to obtain the third feature set after the sound event embedding and the feature embedding.

[0059] The result unit is used to output the third feature set after sound event embedding and feature embedding, and the fourth feature set after sound event embedding, which is the sound source separation result. The third feature set after sound event embedding and feature embedding is input into the residual encoder, and local high-frequency latent features are extracted by upsampling as the fourth feature set. The second feature set and the first feature set are fused with the fourth feature set in the same way as the third feature set to obtain the fourth feature set after sound event embedding and feature embedding, which is the sound source separation result.

[0060] The present invention also provides a multi-channel audio source separation method based on sound event guidance, including a processor and a memory. The memory stores a computer program that can be executed by the processor to realize the multi-channel audio source separation method based on sound event guidance as described above.

[0061] The present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device on which the computer-readable storage medium is located, implement the multi-channel audio source separation method based on sound event guidance as described above.

[0062] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0063] This invention solves the problems of low accuracy and insufficient information utilization in multi-channel audio processing by introducing a self-attention mechanism, sound event feature guidance and multi-feature fusion mechanism. It significantly improves the accuracy and stability of the model in tasks such as equipment status monitoring and fault early warning, and has broad application prospects.

[0064] Specifically, this invention extracts multi-dimensional time-frequency feature embeddings through a hierarchical self-attention mechanism to generate a frame-level strong predictive sound event map, effectively capturing the time and frequency domain features of the device's operating state.

[0065] By embedding sound event features into the extracted global low-frequency latent features and local high-frequency latent features through interpolation matching and learnable adaptive weights, the spatial information of multi-channel audio is fully utilized to improve the accuracy of sound source separation.

[0066] This invention uses the sound source separation result from the previous iteration as the input signal for the model's additional channels during the iterative optimization process, thereby avoiding misidentification and enhancing the robustness of the algorithm.

[0067] This invention enables the separation of independent audio signals from multiple devices in complex industrial environments, facilitating subsequent analysis and processing, real-time monitoring of device operating status, and early warning of potential faults.

[0068] This invention is applicable to terminal devices with data processing capabilities such as desktop computers, laptops, servers, and workstations, and can be extended to cloud deployment to meet the audio monitoring needs of large-scale industrial equipment. Cloud deployment enables concurrent processing of multiple devices through a distributed computing framework, supporting efficient analysis and feedback of real-time data streams. Attached Figure Description

[0069] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0070] Figure 1This is a flowchart illustrating a multi-channel audio source separation method based on sound events, as provided in Embodiment 1.

[0071] Figure 2 This is a schematic diagram of the framework of a multi-channel audio source separation method based on sound events, provided in Embodiment 1.

[0072] Figure 3 This is a schematic diagram of a multi-channel audio source separation device based on sound event guidance, provided in Embodiment 2.

[0073] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0075] Example 1

[0076] Embodiment 1 of the present invention provides a multi-channel audio source separation method based on sound events, which can be implemented by a multi-channel audio source separation device based on sound events (hereinafter referred to as the audio source separation device), and in particular, executed by one or more processors within the audio source separation device.

[0077] In this embodiment, the audio source separation device may be an electronic device equipped with a processor. The processor carries a computer program for the multi-channel audio source separation method guided by sound events, and the computer program can be executed. Examples include computers, smartphones, smart tablets, workstations, etc., which are not limited here.

[0078] like Figures 1-2 As shown, a multi-channel sound source separation method based on sound events includes steps S1 to S7.

[0079] S1 divides the multi-channel audio data of the acquired industrial equipment into windows channel by channel, and performs Fourier transform and Mel filtering on the feature-aligned data in each window to obtain the Mel spectrum.

[0080] like Figure 2 As shown, this invention first acquires multi-channel industrial audio data from industrial equipment using a multi-channel microphone acquisition device. To ensure efficient subsequent processing, the audio data of each channel is divided into windows according to a preset time length. Then, the data within each window is used to calculate the time difference of the multi-channel audio using a cross-correlation function, and the data within each channel window are time-synchronized and aligned.

[0081] Then, the aligned time-domain data in each window are subjected to Fourier transform to obtain preliminary frequency-domain features;

[0082] By setting a number of Mel filter banks, the linear frequencies of the initial frequency domain characteristics are converted into nonlinear Mel frequencies, generating a Mel spectrum to highlight key frequency components related to equipment faults.

[0083] For example, the time dimension T is set to 1024, the frequency number F is set to 64, and the window size P is set to 16. These parameter choices allow for efficient use of computing resources while preserving sufficient time-frequency resolution to capture subtle changes in the operation of industrial equipment.

[0084] In this embodiment, the cross-correlation function is a mathematical function describing the similarity of two signals at different time offsets, reflecting the degree of overlap between one signal and another after a time delay. This embodiment quantifies the change in similarity between different channel audio signals with time offset to accurately locate the time difference of signal propagation, providing a basis for subsequent multi-channel data time synchronization alignment. Specifically, for two channels of audio data within the same time window (such as discrete data with a preset window length of 100ms), the discrete cross-correlation function Rxy is calculated under different discrete time offsets; the optimal offset that maximizes the discrete cross-correlation function Rxy is found. Based on the optimal offset and the audio sampling frequency, the actual time difference is calculated.

[0085] S2, through a hierarchical self-attention mechanism, extract time-frequency feature embeddings from the Mel spectrum as the first feature set; then gradually compress the length of the time-frequency feature embeddings and increase the feature dimension to obtain a time-frequency feature embedding token set.

[0086] like Figure 2 The self-attention sound event detection model based on token semantics shown includes a hierarchical self-attention module and a token semantics module.

[0087] In the hierarchical self-attention module, the Mel spectrum is divided into feature blocks (according to time-frequency-frame window) and a position embedding is added to each feature block to obtain a block of size (T / P*F / P,D); where T is the time dimension, P is the window size, F is the number of frequencies, and D is the feature dimension.

[0088] Each feature block is input into a multi-layer self-attention network, and time-frequency feature embeddings are extracted layer by layer as the first feature set.

[0089] Each layer of the self-attention mechanism includes feature reconstruction and linear layer operations. The time-frequency feature embedding in each layer gradually compresses the temporal length through feature reconstruction, while simultaneously increasing the feature dimension using linear layers, resulting in a set of time-frequency feature embedding tokens with dimensions (T / (NP), F / (NP), ND), where N is the number of layers in the multi-layer self-attention network.

[0090] The hierarchical design in this process not only improves the depth of feature extraction, but also enhances the ability to capture local time-frequency relationships.

[0091] S3, the time-frequency features are embedded into the dimension of the token set and mapped to the number of categories of device operating status through a two-dimensional convolutional layer, and then averaged and pooled in the frequency dimension to obtain a device operating status sound event map per event and frame, i.e., the second feature set.

[0092] To further generate a sound event map of device operating status, time-frequency features embedded in a token set are processed through a token semantic module. The token semantic module employs a two-dimensional convolution with a kernel size of (3,F,8P) and padding size of 10, mapping the feature dimension to the dimension of the number of device operating status categories, C. This convolutional design effectively captures the local time-frequency relationships in the time-frequency features and ensures, through padding, that the size of the output feature map matches the input feature map. Figure 1 The final step is to generate a device operation status sound event map, which is then used as the second feature set, on an event-by-event, frame-by-frame basis. This module is designed to strengthen the correlation between sound events and device operation status, thereby improving the accuracy of subsequent sound source separation.

[0093] S4, the multi-channel audio data is transformed by short-time Fourier transform to obtain a multi-channel time-frequency map, and the global low-frequency latent features of the multi-channel time-frequency map are extracted by downsampling of the residual encoder as a third feature set.

[0094] In this step, the window length can be set to 2048, the hop size to 160, and the sampling rate to 32kHz during the short-time Fourier transform to ensure high resolution of the time-frequency plot. Subsequently, the multi-channel time-frequency plot is fed into the downsampled residual blocks in the residual encoder to extract latent features. For example, latent features can be extracted layer by layer using four sets of downsampled residual codes as a third feature set.

[0095] The design of residual encoders enables the capture of global and local features at different levels, providing rich information support for sound source separation.

[0096] S5, a set of time-varying dynamic scaling parameters and offset parameters are generated for the second feature set through a linear layer with shared parameters, so as to embed the second feature set into the third feature set, and obtain the third feature set after the sound event is embedded.

[0097] A set of time-varying dynamic scaling and offset parameters are generated for the second feature set using a dedicated CNN network, which is used to guide the third feature set and the fourth feature set in step S7.

[0098] Specifically, let the second feature set be... Where R is a real number, In terms of time dimension, For frequency numbers, This refers to the equipment's operating status category.

[0099] Let the third feature set be ,in, The time dimension of the third feature set The frequency count of the third feature set. For feature dimensions.

[0100] Let the fourth feature set be ,in, The time dimension of the fourth feature set The frequency count of the fourth feature set.

[0101] The resolution of the sound timemap for the corresponding device operating status category in the second feature set is adjusted to align with the third feature set by bilinear interpolation.

[0102] A time-varying dynamic scaling parameter is learned from a resolution-adjusted second feature set through a linear layer with shared parameters. With offset parameter These are used to control the intensity of the feature and adjust the feature distribution baseline, respectively. use Adjust the amplitude of the third feature set, using Correct the feature distribution of the third feature set to embed the state probability information of the second feature into the third feature set. The formula is:

[0103] ;

[0104] in, This is the third feature set after the sound event is embedded.

[0105] A time-varying dynamic scaling parameter is learned from a resolution-adjusted second feature set through a linear layer with shared parameters. With offset parameter Specifically:

[0106] ;

[0107] in, Represents a linear layer with shared parameters; This is the second feature set after resolution adjustment.

[0108] This design enables the separation model to dynamically adjust its behavior based on the temporal fluctuations of sound events, thereby adapting to complex changes in real-world scenarios.

[0109] S6, the first feature set is weighted and fused by a learnable adaptive weighted neural network, and the weighted first feature set is projected onto the last dimension of the third feature set after the sound event is embedded. The feature set is fused by adding elements one by one to obtain the third feature set after the sound event is embedded and the feature is embedded.

[0110] To embed the features of the sound event monitoring model into the sound source separation model, the learnable adaptive weighted neural network adopts a single-layer fully connected layer structure. It learns an importance score for each feature in the first feature set through a linear layer, and after obtaining the weights based on softmax normalization, it calculates the weighted sum of the features. Finally, it interpolates the weighted feature dimensions to the feature dimensions corresponding to the third feature set after sound event embedding, and fuses them by adding them element by element to obtain the third feature set after sound event embedding and feature embedding.

[0111] Specifically, the first feature set The input is a fully connected layer, from which the importance score of each feature is obtained. The expression is:

[0112] ;

[0113] in, For the set of importance scores, This represents the score of the j-th feature in the i-th sample; This is the weight matrix; For bias terms;

[0114] The score vector of each sample is normalized using softmax to obtain the weight vector:

[0115] ;

[0116] in, This represents the weight of the j-th feature in the i-th sample; The feature dimension of the first feature set; This represents the importance score of the k-th feature dimension in the i-th sample;

[0117] The first feature set is multiplied element-wise by the weight vector to obtain the weighted features. ;

[0118] Weighted features The dimension is adjusted to match the dimension of the third feature set after the sound event is embedded through two-dimensional interpolation;

[0119] The interpolated weighted features are added element-wise to the third feature set after the sound event embedding, resulting in the third feature set after the sound event embedding and feature embedding.

[0120] This design achieves deep integration of sound event features and sound source separation features, making full use of the spatial information of multi-channel audio and significantly improving the accuracy of sound source separation.

[0121] S7. Input the third feature set after sound event embedding and feature embedding into the residual decoder. Extract local high-frequency latent features through upsampling as the fourth feature set. Then, fuse the second feature set and the first feature set with the fourth feature set in the same way as the third feature set to obtain the fourth feature set after sound event embedding and feature embedding, which is the sound source separation result.

[0122] In this step, the third feature set, formed by the sound event embedding and feature embedding, is fed into the residual decoder for upsampling to extract latent features. For example, latent features can be extracted layer by layer through four sets of upsampled residual decoding to form the fourth feature set.

[0123] Next, the fourth feature set is fused using sound event embedding and feature embedding of the third feature set, respectively. Adjust the amplitude of the fourth feature set, using The feature distribution of the fourth feature set is corrected to embed the state probability information of the second feature into the fourth feature set, as shown in the formula:

[0124] ;

[0125] in, This is the fourth feature set after embedding the sound events. This is the fourth feature set.

[0126] Then, the feature dimensions of the weighted first feature set are interpolated in two dimensions to the feature dimensions of the fourth feature set after the sound event embedding through the linear layer of the learnable adaptive weighted neural network. Finally, the feature sets are fused by adding elements one by one to obtain the fourth feature set after the sound event embedding and the feature embedding.

[0127] In this embodiment, the sound source separation result obtained by fusing four feature sets can combine time events, time-frequency features, and latent features, breaking through the limitations of single features. It can still accurately distinguish target sound sources in scenarios such as "instantaneous overlap, similar frequencies, and complex coupling", ultimately improving the sound source separation evaluation index and providing higher quality single sound source signals for tasks such as industrial equipment fault diagnosis and speech separation, which can better guide the separation effect of the model.

[0128] In another preferred embodiment, the method further includes iteratively optimizing and training the model corresponding to the multi-channel audio source separation method (i.e., the multi-channel audio source separation model) to enhance the robustness of the algorithm. During the iterative optimization and training process, an additional channel for feedback is added as an additional channel input signal to the multi-channel audio source separation model. This additional channel is the (M+1)th channel, where M is the number of channels in the multi-channel audio data.

[0129] The input signal of the additional channel changes dynamically with the number of iterations, specifically:

[0130] During the initial iteration, the input signal of the additional channel is a mute signal, and the sound source is separated through the channels of the multi-channel audio data;

[0131] In subsequent iterations, the number of iterations can be randomly sampled from 1 to Nmax times, and extended to a set number (e.g., 10 times) during inference. The input signal of the additional channel is the historical sound source separation result obtained from the previous iteration; based on the historical sound source separation result and the multi-channel audio data input in the current iteration, the weight of feature extraction is adaptively adjusted to gradually enhance the recognition and separation capability of multi-channel audio data.

[0132] The essence of iterative optimization is to "guide the current separation with historical separation results" to solve the problem of "target sound source features being masked by interference signals" in traditional single-round separation.

[0133] Original multi-channel (M-channel): refers to the audio channels collected by microphones deployed in different parts of industrial equipment (e.g., M=4, corresponding to 4 collection points at both ends of the motor and the inlet and outlet of the pump). Its core function is to capture the sound characteristics of different locations of the equipment, but it is limited by "multi-device audio aliasing" and cannot directly distinguish the signals of the target device from those of interfering devices.

[0134] Additional channel (M+1th channel): A new "feedback channel" added during the iteration process, whose input changes dynamically with the number of iterations.

[0135] Initial iteration (1st time): The input is a silence signal (to avoid introducing false interference since there are no historical separation results). At this time, the separation network only relies on the raw data of the M channel to initially separate the sound sources of each device (the accuracy is low and there may be residual interference).

[0136] Subsequent iterations (2nd to Nmaxth iterations): The input is the target audio source signal separated in the previous iteration (e.g., the M+1 channel of the 2nd iteration inputs the "motor A audio" separated in the 1st iteration). This signal already contains the core frequency characteristics of the target device (e.g., the motor's power frequency of 50Hz, the bearing's characteristic frequency of 1kHz-8kHz), which can be used as a "reference template" to guide the separation network to enhance the target characteristics and suppress interference characteristics.

[0137] During the iteration process, the feature extraction weights are adjusted based on the historical target sound source information obtained through the "M+1 channel".

[0138] For example, after the first separation, 10% of the "pump body B noise" still remains in the "motor A audio". In the second iteration, the "motor A audio with a small amount of noise" is input into the M+1 channel. The separation network will compare the original data of the M channel with the reference signal of the M+1 channel, automatically identify the frequency characteristics of the "pump body B noise" (such as the 100Hz characteristic frequency of the pump body), and suppress it in a targeted manner during separation, so that the residual noise of the "motor A audio" in the second separation is reduced to less than 5%.

[0139] As the number of iterations increases, the accuracy of the reference signal in the M+1 channel continuously improves, and the separation network's ability to identify the target sound source gradually strengthens, ultimately achieving "high-purity target sound source separation" (residual interference ≤3%, meeting the needs of industrial equipment condition monitoring).

[0140] During iterative optimization training, a random sampling strategy can be used to randomly generate the number of iterations, thereby improving the model's generalization ability.

[0141] By introducing different iteration numbers to simulate changes in data distribution in real-world scenarios, the model's generalization ability is enhanced. A feedback mechanism feeds the sound source separation results back to the model as input, further optimizing model performance.

[0142] The core objective of the training phase is to enable the model to adapt to complex situations in industrial scenarios, such as "changes in the number of devices, fluctuations in noise intensity, and differences in the degree of sound source aliasing," and to avoid overfitting to scenarios with a fixed number of iterations.

[0143] Iteration range (1 to Nmax): Nmax is typically set to 5-8 iterations (to balance training efficiency and accuracy; too many iterations will increase training costs). During training, for each training sample (e.g., the aliased audio of "motor + pump + fan"), 1 to Nmax iterations are randomly selected (e.g., 3 iterations in one training session, 5 iterations in another) to simulate the following changes in a real-world scenario:

[0144] Differences in aliasing severity: Simple aliasing (2 devices) may be separated in 1-2 iterations, while complex aliasing (5 or more devices) requires 4-5 iterations. Random sampling can cover different aliasing scenarios.

[0145] Equipment start-up and shutdown dynamics: Equipment in the workshop may be started and stopped temporarily (e.g., after a certain pump is stopped, the number of cascaded equipment decreases). Random iterations can allow the model to adapt to the situation of "changes in the number of separation targets";

[0146] Noise intensity fluctuations: Background noise in industrial environments (such as workshop ventilation noise) changes over time. Random iteration allows the model to learn "separation strategies under different noise intensities" (such as needing more iterations to strengthen target features when the noise is high).

[0147] In another preferred embodiment, the multi-channel sound source separation method of the present invention can also be used for monitoring the operating status of industrial equipment, specifically:

[0148] Calculate the signal-to-noise ratio (SNR) based on the sound source separation results;

[0149] When the signal-to-noise ratio reaches the set threshold, fault features are extracted from the sound source separation results to obtain the operation status monitoring results; wherein, the operation status monitoring results include normal or fault.

[0150] After calculation, the SNR of each device's audio source is calculated. If the SNR is ≥ 20dB, the separation is deemed effective, and the process proceeds to the subsequent "feature extraction - status monitoring" process. If the SNR is < 20dB (e.g., due to sudden strong interference), "re-iteration" or "alarm prompt" is triggered (notifying maintenance personnel to check the acquisition equipment).

[0151] Based on high-purity separated sound sources, fault-sensitive features (such as time domain RMS and frequency domain characteristic frequency peaks) are extracted. Through preset thresholds or intelligent models, the operating status of each device (normal / abnormal, abnormal type, severity) is output.

[0152] In practical applications, this invention is applicable to terminal devices with data processing capabilities, such as desktop computers, laptops, servers, and workstations, and can be extended to cloud deployment. Cloud deployment enables concurrent processing of multiple devices through a distributed computing framework, supporting efficient analysis and feedback of real-time data streams. For example, in a large factory, multiple industrial devices operate simultaneously, and the resulting audio signals are transmitted to the system via a multi-channel microphone collector 1. The system completes sound event detection, sound source separation, and iterative optimization through the above process, ultimately outputting the independent audio signal of each device and its operating status detection results. This solution not only enables the separation of independent audio signals from multiple devices, facilitating subsequent analysis and processing, but also allows for real-time monitoring of equipment operating status and early warning of potential faults.

[0153] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0154] This invention addresses the problems of low accuracy and insufficient information utilization in existing technologies for multi-channel audio processing by introducing a token-based self-attention mechanism, a sound source separation model guided by sound event features, and an iterative optimization strategy. In tasks such as device status monitoring and fault early warning, this invention significantly improves the accuracy and stability of the model and has broad application prospects.

[0155] This invention addresses the core pain points of traditional separation methods:

[0156] Pain Point 1: Low single-wheel separation accuracy: Traditional methods rely solely on raw multi-channel data and cannot distinguish between similar frequency characteristics of the target and interference (such as the 50Hz power frequency interference between the motor and the 55Hz power frequency interference between the pump body). Iterative feedback can enhance the ability to distinguish between targets by using historical target characteristics.

[0157] Pain Point 2: Poor Model Generalization Ability: Traditional models with a fixed number of iterations are prone to failure when the number of devices or noise intensity changes. Random sampling during training can help the model adapt to complex scenarios.

[0158] This invention has the following practical value in real-world industrial applications:

[0159] Improve the accuracy of status monitoring: High-purity separated sound sources can reduce "misjudgments caused by interference signals" (such as avoiding misjudging pump noise as motor failure), thus improving the accuracy of equipment status monitoring to over 95%.

[0160] Reduced operation and maintenance costs: By accurately separating the audio of each device, it is possible to "monitor multiple devices with a single microphone array" (without the need to deploy sensors for each device separately), reducing hardware investment;

[0161] Adaptable to dynamic industrial environments: The combination of random iterative training and inference can adapt to dynamic scenarios such as equipment start-up and shutdown, load changes, and noise fluctuations in the workshop, ensuring long-term stable operation.

[0162] Example 2

[0163] like Figure 3 As shown, the second embodiment of the present invention also provides a multi-channel audio source separation device based on sound event guidance, comprising:

[0164] The data acquisition and feature preprocessing unit is used to divide the multi-channel audio data of the acquired industrial equipment into windows one by one, and perform Fourier transform and Mel filtering on the feature-aligned data in each window to obtain the Mel spectrum.

[0165] The time-frequency feature extraction unit is used to extract time-frequency feature embeddings from the Mel spectrum through a hierarchical self-attention mechanism, which serve as the first feature set; then, the length of the time-frequency feature embeddings is gradually compressed and the feature dimension is increased to obtain a time-frequency feature embedding token set.

[0166] The sound event feature extraction unit is used to embed the time-frequency features into the dimension of the token set and map it to the number of categories of device operating status through a two-dimensional convolutional layer, and then average pool the frequency dimension to obtain a device operating status sound event map per event and frame, i.e., the second feature set.

[0167] The latent feature extraction unit is used to obtain a multi-channel time-frequency map from the multi-channel audio data through short-time Fourier transform, and to extract the global low-frequency latent features of the multi-channel time-frequency map by downsampling of the residual encoder, as a third feature set.

[0168] The feature-guided embedding unit uses a linear layer with shared parameters to generate a set of time-varying dynamic scaling and offset parameters for the second feature set, so as to embed the second feature set into the third feature set, resulting in the third feature set after sound event embedding;

[0169] The feature fusion unit is used to perform weighted fusion on the first feature set through a learnable adaptive weighted neural network, and project the weighted first feature set onto the last dimension of the third feature set after the sound event embedding. The fusion is performed by adding elements one by one to obtain the third feature set after the sound event embedding and the feature embedding.

[0170] The result unit is used to output the third feature set after sound event embedding and feature embedding, and the fourth feature set after sound event embedding, which is the sound source separation result. The third feature set after sound event embedding and feature embedding is input into the residual encoder, and local high-frequency latent features are extracted by upsampling as the fourth feature set. The second feature set and the first feature set are fused with the fourth feature set in the same way as the third feature set to obtain the fourth feature set after sound event embedding and feature embedding, which is the sound source separation result.

[0171] Example 3

[0172] The third embodiment of the present invention also provides a multi-channel audio source separation device based on sound event guidance, which includes a memory and a processor. The memory stores a computer program, which can be executed by the processor to realize the multi-channel audio source separation method based on sound event guidance as described above.

[0173] Example 4

[0174] The fourth embodiment of the present invention also provides a computer-readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by the processor of the device where the computer-readable storage medium is located, they implement the multi-channel audio source separation method based on sound events as described above.

[0175] In the several embodiments provided in this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0176] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0177] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0178] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0179] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0180] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0181] The use of "first" and "second" in the embodiments is merely to distinguish similar objects and does not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0182] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-channel sound source separation method based on sound event guidance, characterized in that, include: S1, divide the multi-channel audio data of the collected industrial equipment into windows one by one, and perform Fourier transform and Mel filtering on the feature-aligned data in each window to obtain the Mel spectrum; S2, extract time-frequency feature embeddings from the Mel spectrum through a hierarchical self-attention mechanism as the first feature set; then gradually compress the length of the time-frequency feature embeddings and increase the feature dimension to obtain a time-frequency feature embedding token set; S3, the time-frequency features are embedded into the dimension of the token set and mapped to the number of categories of device operating status through a two-dimensional convolutional layer, and then averaged and pooled in the frequency dimension to obtain a device operating status sound event map per event and frame, i.e., the second feature set. S4, the multi-channel audio data is transformed by short-time Fourier transform to obtain a multi-channel time-frequency map, and the global low-frequency latent features of the multi-channel time-frequency map are extracted by downsampling of the residual encoder as a third feature set; S5, a set of time-varying dynamic scaling parameters and offset parameters are generated for the second feature set through a linear layer with shared parameters, so as to embed the second feature set into the third feature set to obtain the third feature set after sound event embedding; S6, the first feature set is weighted and fused by a learnable adaptive weighted neural network, and the weighted first feature set is projected onto the last dimension of the third feature set after the sound event is embedded. The fusion is performed by adding elements one by one to obtain the third feature set after the sound event is embedded and the feature is embedded. S7. Input the third feature set after sound event embedding and feature embedding into the residual decoder. Extract local high-frequency latent features through upsampling as the fourth feature set. Then, fuse the second feature set and the first feature set with the fourth feature set in the same way as the third feature set to obtain the fourth feature set after sound event embedding and feature embedding, which is the sound source separation result.

2. The method for multi-channel audio source separation based on sound events as described in claim 1, characterized in that... S1 specifically refers to: The data of each channel of the multi-channel audio data is divided into windows according to a preset time length; The data within each window is used to calculate the time difference of multi-channel audio using a cross-correlation function, and the data within each channel window is time-synchronized and aligned. Perform a Fourier transform on the aligned time-domain data in each window to obtain preliminary frequency-domain features; By using a set number of Mel filter banks, the linear frequencies of the initial frequency domain characteristics are converted into nonlinear Mel frequencies, generating a Mel spectrum to highlight key frequency components related to equipment failure.

3. The multi-channel sound source separation method based on sound event guidance according to claim 1, characterized in that... S2 specifically refers to: The Mel spectrum is divided into feature blocks, and a position embedding is added to each feature block to obtain blocks of size (T / P*F / P,D); where T is the time dimension, P is the window size, F is the number of frequencies, and D is the feature dimension. Each feature block is input into a multi-layer self-attention network, and time-frequency feature embeddings are extracted layer by layer as the first feature set. The time-frequency feature embedding of each layer gradually compresses the time series length through feature reconstruction, while using a linear layer to increase the feature dimension, resulting in a time-frequency feature embedding token set.

4. The multi-channel sound source separation method based on sound event guidance according to claim 1, characterized in that... S5 specifically refers to: Let the second feature set be Where R is a real number, In terms of time dimension, Classification of equipment operating status; Let the third feature set be ,in, The time dimension of the third feature set The frequency count of the third feature set. For feature dimensions; The resolution of the sound time map of the corresponding device operating status category in the second feature set is adjusted to align with the third feature set by bilinear interpolation; A time-varying dynamic scaling parameter is learned from a resolution-adjusted second feature set through a linear layer with shared parameters. With offset parameter These are used to control the intensity of the feature and adjust the baseline of the feature distribution, respectively; utilizing Adjust the amplitude of the third feature set, using The feature distribution of the third feature set is corrected to embed the state probability information of the second feature into the third feature set, resulting in the third feature set after sound event embedding. The formula is as follows: ； ； in, Let R be the third feature set after embedding the sound events, and R be a real number. This is the transpose operator.

5. The multi-channel sound source separation method based on sound event guidance according to claim 4, characterized in that... By using a linear layer with shared parameters, time-varying dynamic scaling parameters are learned from a resolution-adjusted second feature set. With offset parameter Specifically: ； in, Represents a linear layer with shared parameters; This is the second feature set after resolution adjustment.

6. The multi-channel sound source separation method based on sound event guidance according to claim 4, characterized in that... The learnable adaptive weighted neural network adopts a single-layer fully connected layer structure. It learns an importance score for each feature in the first feature set through a linear layer, and after obtaining weights based on softmax normalization, calculates a weighted sum of features. Finally, it interpolates the weighted feature dimensions two-dimensionally to the feature dimensions corresponding to the third feature set after the sound event embedding, and fuses them element-wise to obtain the third feature set after the sound event embedding and feature embedding. Specifically: The first feature set The input is a fully connected layer, from which the importance score of each feature is obtained. The expression is: ； in, For the set of importance scores, This represents the score of the j-th feature in the i-th sample; This is the weight matrix; For bias terms; The score vector of each sample is normalized using softmax to obtain the weight vector: ； in, This represents the weight of the j-th feature in the i-th sample; The feature dimension of the first feature set; This represents the importance score of the k-th feature dimension in the i-th sample; The first feature set is multiplied element-wise by the weight vector to obtain the weighted features. ; Weighted features The dimension is adjusted to match the dimension of the third feature set after the sound event is embedded through two-dimensional interpolation; The interpolated weighted features are added element-wise to the third feature set after the sound event embedding, resulting in the third feature set after the sound event embedding and feature embedding.

7. The multi-channel sound source separation method based on sound event guidance according to claim 1, characterized in that... It also includes: performing multiple iterative optimizations on the sound source separation results, and adding an additional channel for feedback during the iterative optimization training process; The additional channel is the (M+1)th channel, where M is the number of channels in the multi-channel audio data; The input signal of the additional channel changes dynamically with the number of iterations, specifically: During the initial iteration, the input signal of the additional channel is a mute signal, and the sound source is separated through the channels of the multi-channel audio data; In subsequent iterations, the input signal of the additional channel is the historical sound source separation result obtained from the previous iteration; based on the historical sound source separation result and the multi-channel audio data input in the current iteration, the weight of feature extraction is adaptively adjusted to gradually enhance the recognition and separation capability of multi-channel audio data.

8. A multi-channel audio source separation method based on sound event guidance according to any one of claims 1-7, characterized in that... It is also used for monitoring the operating status of industrial equipment, specifically: Calculate the signal-to-noise ratio based on the sound source separation results; When the signal-to-noise ratio reaches the set threshold, fault features are extracted from the sound source separation results to obtain the operation status monitoring results; wherein, the operation status monitoring results include normal or fault.

9. A multi-channel sound source separation device guided by sound events, characterized in that, include: The data acquisition and feature preprocessing unit is used to divide the multi-channel audio data of the acquired industrial equipment into windows one by one, and perform Fourier transform and Mel filtering on the feature-aligned data in each window to obtain the Mel spectrum. The time-frequency feature extraction unit is used to extract time-frequency feature embeddings from the Mel spectrum through a hierarchical self-attention mechanism, as a first feature set; Then, the length of the time-frequency feature embedding is gradually compressed and the feature dimension is increased to obtain a time-frequency feature embedding token set; The sound event feature extraction unit is used to embed the time-frequency features into the dimension of the token set and map it to the number of categories of device operating status through a two-dimensional convolutional layer, and then average pool the frequency dimension to obtain a device operating status sound event map per event and frame, i.e., the second feature set. The latent feature extraction unit is used to obtain a multi-channel time-frequency map from the multi-channel audio data through short-time Fourier transform, and to extract the global low-frequency latent features of the multi-channel time-frequency map by downsampling of the residual encoder, as a third feature set. The feature-guided embedding unit uses a linear layer with shared parameters to generate a set of time-varying dynamic scaling and offset parameters for the second feature set, so as to embed the second feature set into the third feature set, resulting in the third feature set after sound event embedding; The feature fusion unit is used to perform weighted fusion on the first feature set through a learnable adaptive weighted neural network, and project the weighted first feature set onto the last dimension of the third feature set after the sound event embedding. The fusion is performed by adding elements one by one to obtain the third feature set after the sound event embedding and the feature embedding. The result unit is used to output the third feature set after sound event embedding and feature embedding, and the fourth feature set after sound event embedding, which is the sound source separation result. The third feature set after sound event embedding and feature embedding is input into the residual encoder, and local high-frequency latent features are extracted by upsampling as the fourth feature set. The second feature set and the first feature set are fused with the fourth feature set in the same way as the third feature set to obtain the fourth feature set after sound event embedding and feature embedding, which is the sound source separation result.

10. A multi-channel audio source separation device based on sound event guidance, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that can be executed by the processor to implement a multi-channel audio source separation method based on sound events as described in any one of claims 1-7.

Images