A method and apparatus for processing a signal

Abstract

The application discloses a signal processing method and device, and relates to the technical field of data processing. A specific implementation of the method comprises the following steps: obtaining microphone information and collected sound signals of a vehicle in multiple sound areas, and screening a first sound area with a microphone quantity greater than a preset quantity threshold from the multiple sound areas; performing beamforming processing on the sound signals of the first sound area according to the spatial information of the microphone of the first sound area; calling a speech separation model to perform speech separation based on the processed sound signals of the first sound area and the sound signals of the sound areas other than the first sound area, and obtaining a speech instruction signal and a corresponding target sound area. The implementation can first perform dimension reduction processing on the collected signals in a manner of beamforming processing, reduce the complexity of subsequent sound signal separation, and improve the accuracy of the signal processing result.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a signal processing method and apparatus. Background Technology

[0002] Currently, with the development of intelligent voice interaction technology, users can trigger various business functions of the system through voice commands. For example, in the scenario of a smart car cockpit, users can trigger the car audio system or the air conditioning to start using voice commands. However, in this car interior environment, due to different seating positions, the space is divided into multiple sound zones corresponding to different seating locations. Users may trigger voice commands in each sound zone, but the signals collected by the microphones in different sound zones are different, requiring precise speech separation to accurately identify the content of the voice commands. A common approach is to use deep learning models in an end-to-end solution for speech separation. However, the input signal dimension of the model is usually high, leading to greater training difficulty and limited accuracy of the final output signal processing results. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a signal processing method and apparatus that can reduce the dimensionality of the signal input to the model and ensure the accuracy of the signal separation results of the model.

[0004] To achieve the above objectives, according to one aspect of the present invention, a signal processing method is provided, comprising: Acquire microphone information and collected sound signals from the vehicle to be processed in multiple audio zones; Based on the microphone information of the multiple sound zones, a first sound zone with a number of microphones greater than a preset threshold is selected from the multiple sound zones, and beamforming processing is performed on the sound signal of the first sound zone based on the spatial information of the microphones in the first sound zone. The speech separation model is invoked to perform speech separation based on the sound signal processed from the first voice region and the sound signals from the multiple voice regions other than the first voice region, to obtain a speech command signal and the target voice region corresponding to the speech command signal. The speech separation model is trained based on a preset mixed audio dataset.

[0005] Optionally, before invoking the speech separation model to perform speech separation based on the processed sound signal from the first voice region and the sound signals from the multiple voice regions excluding the first voice region, to obtain the speech command signal and the target voice region corresponding to the speech command signal, the method further includes: Obtain the size information of a first vehicle of the same model as the vehicle to be processed, and determine each tone zone of the first vehicle based on the size information; Based on the microphone location information and preset sound source location information of the first vehicle, a speech impulse response for each sound zone of the first vehicle is generated; Convolution operation is performed on multiple preset single-channel speech signals and the speech impulse response of each sound zone of the first vehicle to obtain the sound source signal of each sound zone of the first vehicle; The preset mixed audio dataset is constructed based on the sound source signals of each sound zone of the first vehicle.

[0006] Optionally, after constructing the preset mixed audio dataset based on the sound source signals of each sound zone of the first vehicle, the method further includes: Beamforming processing is performed on the sound source signals in the preset mixed audio dataset; The speech separation model is trained based on the processed sound source signal and the corresponding single-channel speech signal.

[0007] Optionally, the method further includes: The voice command signal is subjected to speech recognition to obtain the command content; A voice instruction is generated based on the instruction content and the target voice region corresponding to the voice instruction signal, and the voice instruction is sent to the corresponding functional unit so that the corresponding functional unit executes the voice instruction.

[0008] Optionally, before invoking the speech separation model to perform speech separation based on the processed sound signal from the first voice region and the sound signals from the multiple voice regions excluding the first voice region, to obtain the speech command signal and the target voice region corresponding to the speech command signal, the method further includes: The sound signal processed in the first sound region and the sound signals in the multiple sound regions other than the first sound region are preprocessed, and a separation matrix is ​​calculated based on the preprocessed sound signals. The preprocessed audio signal is initially separated based on the separation matrix.

[0009] Optionally, before selecting a first sound zone from the multiple sound zones based on the microphone information of the multiple sound zones, the method further includes: The acquired sound signals are subjected to echo cancellation processing.

[0010] According to another aspect of the present invention, a signal processing apparatus is provided, comprising: The first acquisition module is used to acquire microphone information and collected sound signals of the vehicle to be processed in multiple sound zones; The first processing module is used to filter out a first sound zone from the multiple sound zones based on the microphone information of the multiple sound zones, wherein the number of microphones is greater than a preset threshold, and to perform beamforming processing on the sound signal of the first sound zone based on the spatial information of the microphones in the first sound zone. The separation module is used to call the speech separation model to perform speech separation based on the sound signal processed by the first voice region and the sound signals of the voice regions other than the first voice region among the multiple voice regions, to obtain the speech command signal and the target voice region corresponding to the speech command signal. The speech separation model is trained based on a preset mixed audio dataset.

[0011] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the signal processing method provided in the embodiments of the present invention.

[0012] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the signal processing method provided in the embodiments of the present invention.

[0013] According to another aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the signal processing method provided in the embodiments of the present invention.

[0014] One embodiment of the above invention has the following advantages or beneficial effects: it can compress the acquired multi-channel signal into a single-channel signal through dimensionality reduction processing, reducing the complexity of the input signal in the subsequent speech separation process, thereby reducing the difficulty of the speech separation process and improving the accuracy of the speech separation results. This method of first reducing the dimensionality of the original signal and then using the speech separation model for fine separation can reduce the complexity of the model's learning task, enabling the model to achieve better separation results.

[0015] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0016] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram of the main steps of a signal processing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating a vocal register division method provided in an embodiment of the present invention; Figure 3This is a schematic diagram of the signal flow in the signal processing method based on the present invention; Figure 4 This is a schematic diagram of the main modules of a signal processing device according to an embodiment of the present invention; Figure 5 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0017] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0018] It should be noted that the technical solutions disclosed in this invention, regarding the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0019] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. When applicable, user personal information is subjected to de-identification and / or anonymization and / or encryption technical processing.

[0020] Currently, in the development of intelligent automotive cockpits, different locations within a car have different sound zones, and the voice signals collected by microphones at different locations from each seat are not entirely the same. To accurately identify sound signals from different sound zones, the collected signals need to undergo accurate voice separation before recognition to determine the precise voice commands. Specifically, identifying sound signals from different zones can be used to trigger command functions in different sound zones. Existing in-vehicle multi-zone voice separation solutions mainly include the following three categories: The first category is based on traditional signal processing solutions, mainly involving the cascaded combination of techniques such as echo cancellation (AEC), beamforming (BF), and blind source separation (BSS). These solutions rely on the sequential optimization of each module in signal processing, with each module designed and tuned independently. The second type is an end-to-end solution based on neural networks, which directly replaces all or part of the traditional signal processing steps with deep learning models to complete speech separation in an end-to-end manner. The third type is a hybrid approach that combines a neural network at the beginning and a traditional algorithm at the end. This approach uses a neural network model to predict beamforming weights at the beginning, and then uses the network output to guide the subsequent traditional blind source separation algorithm. The core idea of ​​this approach is to use a neural network to learn the beamforming weight parameters and then input them into a traditional signal processing chain.

[0021] However, the above methods have many problems. In purely traditional signal processing schemes, each module is designed independently, lacking global optimization, resulting in limited effectiveness in multi-region speech separation and a tendency to produce sound leakage. When there are many microphones in the vehicle, the computational complexity of blind source separation algorithms is high. Although end-to-end neural network schemes can directly model the acoustic environment, the model has a large number of parameters, high training difficulty, and the difference between simulation data and the real vehicle environment can lead to audio distortion, affecting the accuracy of subsequent speech recognition and wake-up. In the hybrid scheme of neural network pre-processing + traditional algorithm post-processing, the neural network directly processes the original multi-channel mixed signal, resulting in high input dimensionality and high training difficulty, and it does not utilize the pre-cleaning capabilities of traditional signal processing to reduce the learning burden of the neural network. At the same time, since the output of the neural network is only beamforming weights, its ability to finely compensate for separation residues is limited.

[0022] To address the aforementioned shortcomings, this invention provides a signal processing method that pre-processes multi-regional sound signals to reduce the dimensionality of the signals input to the speech separation model, thereby reducing the processing difficulty of the model and achieving accurate speech separation processing, ensuring the accuracy of speech command recognition results.

[0023] Figure 1 This is a schematic diagram illustrating the main steps of a signal processing method according to an embodiment of the present invention. Figure 1 As shown, the signal processing method mainly includes steps S101 to S103.

[0024] Step S101: Obtain microphone information and collected sound signals from the vehicle to be processed in multiple sound zones.

[0025] First, it's necessary to obtain microphone information for each audio zone in the vehicle being processed. This information includes the number and distribution of microphones in each zone. In addition, it's also necessary to acquire the collected sound signals, including those from each audio zone. Each audio zone includes four areas corresponding to different seats: the driver's area, the front passenger area, the area behind the driver's seat, and the area behind the front passenger seat. When a sound source produces sound, the sound signal is captured by the microphones in each of these different zones.

[0026] In one embodiment, because vehicle shapes may vary, the division of sound zones may also differ when the vehicle interior contains more than two rows of seats. For example, if the vehicle has three rows of seats with two seats in each row, the number of sound zones is six. The number of sound zones primarily depends on the number of seats, to ensure that users can trigger voice commands from any position.

[0027] In one embodiment, the microphones are typically positioned with identical brushes and locations within each audio zone, and each zone can have the same single microphone or microphone sequence. However, due to actual business needs, the number of microphones in different audio zones can vary, and the specific number can be set according to actual business requirements.

[0028] Step S102: Based on the microphone information of multiple sound zones, select the first sound zone from the multiple sound zones with a number of microphones greater than a preset threshold, and perform beamforming processing on the sound signal of the first sound zone based on the spatial information of the microphones in the first sound zone.

[0029] For each defined audio region, the number of microphones in each region needs to be determined first based on the microphone information. Then, the first audio region with a microphone count greater than a preset threshold is identified. This preset threshold is 1, primarily used to distinguish between single-microphone and microphone array signals. Because single-microphone signals are typically single-channel signals with low dimensionality, while microphone array signals are typically multi-channel signals with high dimensionality, further dimensionality reduction processing is needed before separation in the input model to reduce the difficulty of model inference and output accurate results. Dimensionality reduction can be achieved using beamforming. For example, if the microphone array is a 4-element array, acquiring 4 channels of signal, beamforming can be used to synthesize the 4 channels into a single enhanced signal using the spatial information of the microphone array, thus completing the signal dimensionality reduction. The signal in each audio region can undergo this dimensionality reduction process to obtain the single-channel signal for each region.

[0030] In one embodiment, the core operation process of beamforming is as follows: Array element signal acquisition: Multiple raw signals are acquired simultaneously through a multi-element array arranged in a specific geometric shape.

[0031] Amplitude and phase weighted adjustment: The amplitude and phase of each signal are independently compensated to offset the path difference between different array elements and the target position.

[0032] Signal synthesis output: The adjusted multi-channel signals are summed and superimposed to enhance the resonance of the signal in the target direction and cancel each other out of the signals in the non-target direction.

[0033] Beam directional output: ultimately forming a high-gain narrow beam pointing in a specific direction to achieve directional transmission or directional reception.

[0034] Step S103: Call the speech separation model to perform speech separation using the sound signal processed based on the first voice region and the sound signals of multiple voice regions other than the first voice region, to obtain the speech command signal and the target voice region corresponding to the speech command signal. The speech separation model is trained based on a preset mixed audio dataset.

[0035] After the dimensionality reduction of the sound signal is completed, the processed signal and the acquired single-channel sound signal can be input together into the speech separation model. The model then performs speech separation, identifies the accurate speech command signal, and determines the target pitch region from which the speech command signal originates. Based on the identified speech command signal and its corresponding target pitch region, the pitch region from which the sound signal originates can be determined, i.e., the source pitch region of the speech command signal can be identified.

[0036] In one embodiment, after determining the source voice command signal's audio region, in addition to identifying which user issued the command, business functions can be implemented according to the audio region, enabling differentiated functions for different locations inside the car. For example, if a user issues the command "turn on the air conditioning" from the left rear seat, the air conditioning vent can be turned on only at that location, leaving other locations untouched, thus achieving a more intelligent command function.

[0037] In one embodiment, echo cancellation can employ NLMS (Normalized Least Mean Squares), RLS (Recursive Least Squares), or frequency-domain-based adaptive algorithms; beamforming can utilize different algorithms such as MVDR (Minimum Variance Distortion-Free Response) and LCMV (Linearly Constrained Minimum Variance); blind source separation can employ FastICA (Fast Independent Component Analysis), JADE (Joint Approximate Diagonalization of Eigenmatrices), or natural gradient-based algorithms. Alternatives to training data sources: RIR (Room Impulse Response) data recorded in actual vehicles can replace the RIR data simulated using the mirror source method, or more accurate RIR data can be generated using acoustic simulation software based on the finite element method (FEM).

[0038] According to the signal processing method of the present invention, multi-channel signals can be compressed into single-channel signals through dimensionality reduction operations, thereby reducing the complexity of the input signals in the subsequent speech separation process, thus reducing the difficulty of the speech separation process and improving the accuracy of the speech separation results. This approach of first reducing the dimensionality of the original signal and then using a speech separation model for fine separation can reduce the complexity of the model's learning task and enable the model to achieve better separation results.

[0039] In one embodiment, before invoking a speech separation model to perform speech separation based on the processed sound signal from a first voice region and the sound signals from multiple voice regions excluding the first voice region, and obtaining the speech command signal and the target voice region corresponding to the speech command signal, the method further includes: Obtain the size information of a first vehicle of the same model as the vehicle to be processed, and determine each tone zone of the first vehicle based on the size information; Based on the microphone location information and preset sound source location information of the first vehicle, a speech impulse response for each sound zone of the first vehicle is generated; Convolution operation is performed on multiple preset single-channel speech signals and the speech impulse response of each sound zone of the first vehicle to obtain the sound source signal of each sound zone of the first vehicle; A pre-defined mixed audio dataset is constructed based on the sound source signals of each sound zone of the first vehicle.

[0040] To improve the accuracy of the speech separation model, a targeted training dataset needs to be constructed first. Specifically, the dimensions (length, width, and height) of a first vehicle with the same model as the vehicle to be processed are obtained. Based on these dimensions, the vehicle is divided into multiple sound zones, typically four: driver's area, passenger's area, driver's rear area, and passenger's rear area. The microphone position information for each sound zone is obtained to calculate the speech impulse response for each zone. Here, the speech impulse response refers to the room impulse response (RIR), which is the transfer function between the sound source and the microphone, recording the differences in sound propagation from the source to the microphone at different spatial locations. Simulated data for the corresponding sound zone can be obtained by convolving the RIR with single-channel high-fidelity speech, thus replacing large-scale actual data recording, reducing costs and improving efficiency. When calculating the speech impulse response, multiple preset sound source position information can be used to simulate the sound emission positions. These can be randomly selected, but must include all desired sound emission positions. The obtained speech impulse response is convolved with the single-channel speech signal corresponding to the preset sound source location to obtain the sound source signal of each single-channel speech signal relative to each microphone. The sound source signals of the microphones in each sound zone are then aggregated to obtain the sound source signal for each sound zone. In other words, if the number of microphones in a sound zone is not one, a multi-channel sound source signal of the microphone array for that sound zone is generated based on the single-channel sound source signal collected by each microphone. Finally, the sound source data and noise data are mixed according to a certain signal-to-noise ratio and signal-to-interference ratio to obtain mixed audio, thereby constructing a preset mixed audio dataset. The noise data can be generated based on vehicle internal and external noise, such as air conditioning noise, wind noise, tire noise, rain noise, turn signal noise, and wiper noise. The final preset mixed audio dataset should also include the single-channel speech signal corresponding to the preset sound source location information for each sound zone. This signal is a single-channel, high-fidelity signal that can be used as the sound source signal.

[0041] In one specific embodiment, firstly, the Image Source Method (ISM) is used to generate RIR data. This involves simulating and generating the corresponding RIR data given the actual vehicle dimensions, microphone location, and sound source location information. In one embodiment, audio from the sound source and microphone can be directly collected on the actual vehicle, and the actual RIR can be calculated using an algorithm. During RIR generation or acquisition, the RIR data for multiple locations within each sound region is obtained according to the desired region division method; that is, the sound source locations for the driver's seat, passenger seat, rear passenger seat, and rear passenger seat are traversed within their respective regions. Secondly, the single-channel high-fidelity audio is convolved with the RIR data for the driver's seat, passenger seat, rear passenger seat, and rear passenger seat regions, respectively, to obtain reverberant sound source data for each region. Finally, the sound source data and noise data are mixed according to a certain signal-to-noise ratio and signal-to-interference ratio to obtain mixed audio, thereby constructing a dataset.

[0042] In one embodiment, each audio zone contains M (M>=1) microphones, which can be a single microphone or a microphone array consisting of multiple microphones in a linear array, ring array, or other array configuration. The audio zone division method is as follows: Figure 2 As shown, it includes four sound zones: driver's seat, passenger seat, rear passenger seat, and rear passenger seat. Each sound zone includes a microphone array consisting of M microphones.

[0043] According to the signal processing method of the present invention, the speech impulse response of different sound zones in a vehicle can be calculated separately to learn the differences generated when the sound source signal is transmitted to the microphone. Furthermore, based on these differences, the signal collected by the microphone based on the sound source signal is calculated to generate the final mixed audio, resulting in an accurate, high-quality training dataset. Based on this training dataset, more accurate model training operations can be performed, resulting in a speech separation model with more accurate output.

[0044] In one embodiment, after constructing a preset mixed audio dataset based on the sound source signals of each sound zone of the first vehicle, the method further includes: performing beamforming processing on the sound source signals in the preset mixed audio dataset; and training a speech separation model based on the processed sound source signals and the single-channel speech signals corresponding to the sound source signals.

[0045] After constructing the dataset, the speech separation model can be trained using a pre-defined mixed audio dataset. The training primarily employs a joint training approach: before inputting the training data into the model, beamforming is performed on the multi-channel signals of each vocal region to obtain single-channel signals, achieving dimensionality reduction and decreasing the data complexity for the model. The processed signals and the unprocessed single-channel signals are then used to train the speech separation model. This training method results in a speech separation model that outputs more accurate results.

[0046] In one embodiment, before dimensionality reduction during training, echo cancellation can be performed to eliminate echoes generated in the environment, such as those from music played by car speakers or TTS voice recordings. The single-channel signals of each region of the same sound source after dimensionality reduction are arranged into multi-channel signals and blind source separation (BSS) is performed to obtain preliminary separation signals for each region. Finally, the preliminary separation signals are input into the speech separation model for training. Beamforming not only reduces the signal dimension of the input speech separation model but also reduces the signal dimension of the BSS operation, improving the accuracy of the preliminary separation results. Taking a 4-region speech model as an example, without pre-beamforming dimensionality reduction, the BSS needs to process 16 channels (16 microphones), with a computational complexity on the order of O(16²) = O(256); however, after beamforming dimensionality reduction, the BSS only needs to process 4 channels, reducing the computational complexity to the order of O(4²) = O(16), a reduction of approximately 16 times. Finally, the preliminary separation signals are used to train the speech separation model. The model takes 4 input channels (BSS output for each of the 4 registers) and outputs 4 channels (separated speech for each register). The model employs a lightweight U-Net architecture (encoder-decoder architecture), with a two-layer LSTM used for temporal modeling in the intermediate layers. Skip connections are used between the encoder and decoder to prevent information loss.

[0047] In one embodiment, this solution adopts a cascaded architecture of "signal processing before + neural network after", which has the following characteristics: First, pre-processing dimensionality reduction: For a sound zone containing a microphone array, beamforming (BF) is used to compress the multi-channel signal into a single-channel signal, so that the number of input channels for subsequent blind source separation (BSS) is equal to the number of sound zones rather than the total number of microphones, thereby reducing the computational complexity of BSS from O(N²) to O(K²), where N is the total number of microphones and K is the number of sound zones (K is much smaller than N).

[0048] Second, post-processing refinement: The initial separation signal output by the BSS is used as the input to the neural network. Since traditional processing has already completed most of the separation work, the neural network only needs to learn the elimination mode of residual interference, which greatly reduces the complexity of the learning task. Therefore, a lightweight model with fewer parameters can be used to achieve a separation effect comparable to or even better than that of a large-parameter end-to-end model.

[0049] Third, joint training: During the model training phase, the traditional signal processing chain (AEC→BF→BSS) is used as a fixed preprocessing pipeline and jointly trained with the subsequent neural network. The training data first passes through the complete signal processing chain to generate preliminary separated signals, and then the neural network is trained using clean speech from each vocal region as labels. This joint training method enables the neural network to learn specifically the residual error patterns of traditional processing, achieving global optimization.

[0050] The signal processing method according to embodiments of the present invention places the preliminary signal processing flow before the neural network. It first preprocesses, reduces dimensionality, and performs preliminary separation processing on the acquired multi-channel signals, and then uses speech separation to finely compensate for the preliminary separation results, improving the accuracy of the final separation result. This architecture allows the model input to be a signal that has undergone sufficient dimensionality reduction and preliminary separation, significantly reducing the complexity of the learning task. It enables the use of a model with far fewer parameters than end-to-end solutions to achieve better separation results, simplifying the complexity of the speech separation model while improving the accuracy of the separation results. Furthermore, by treating preprocessing, dimensionality reduction, and preliminary separation processing as pre-processing and the model as post-processing during training, end-to-end joint training further improves the accuracy of the speech separation model.

[0051] In one embodiment, the method further includes: performing speech recognition on the speech command signal to obtain the command content; generating a speech command based on the command content and the target voice region corresponding to the speech command signal; and sending the speech command to the functional unit corresponding to the speech command so that the corresponding functional unit executes the speech command.

[0052] The final voice command signal still needs to undergo speech recognition to determine the command content. That is, the voice signal is recognized as text content, and semantic commands are generated based on the recognized text content and the target audio region, which are then sent to the corresponding functional units to implement the corresponding functions. At this point, when executing the response function, the target audio region corresponding to the voice command signal can also be combined to implement the function at the audio region level. That is, the function can be implemented only in the target audio region without affecting other audio regions, such as turning on the air conditioner or the stereo only within the target audio region.

[0053] The signal processing method according to embodiments of the present invention can accurately determine the source voice region of a signal based on the recognition results of a speech separation model, and can perform differentiated functions based on the source voice region, thereby improving the accurate positioning of the corresponding functions of voice commands and performing intelligent differential processing. It can realize the function of the current voice region without affecting other voice regions, which helps to improve the user experience.

[0054] In one embodiment, before invoking a speech separation model to perform speech separation based on the processed sound signal of the first voice region and the sound signals of voice regions other than the first voice region in multiple voice regions to obtain a speech command signal and the target voice region corresponding to the speech command signal, the method further includes: preprocessing the processed sound signal of the first voice region and the sound signals of voice regions other than the first voice region in multiple voice regions, and calculating a separation matrix based on the preprocessed sound signal; and performing preliminary separation on the preprocessed sound signal based on the separation matrix.

[0055] In the signal processing flow, before inputting the dimensionality-reduced signal into the speech separation model, the BSS method can be used for preliminary signal separation. Specifically, the single-channel signals of all audio regions after beamforming are first preprocessed. Preprocessing operations include denoising, zero-mean transformation, whitening, and dimension matching. Denoising: Dimensionality reduction using PCA filters low-energy noise components and removes irrelevant interference from the observed signal; this is an optional sub-step in preprocessing. Zero-mean transformation: The mean of all observed signals is shifted to zero, satisfying the basic statistical assumptions of the blind source separation algorithm. Whitening: Linear transformation eliminates the linear correlation between observed components and normalizes the variance of all components to 1, significantly reducing the difficulty of solving the subsequent separation matrix. Dimension matching: The dimension of the observed signal is adjusted to match the number of source signals, avoiding computational errors due to dimension mismatch. Then, the separation matrix is ​​calculated based on the preprocessed audio signal, and a linear transformation is performed on the preprocessed audio signal based on the separation matrix to obtain the separated signal. When calculating the separation matrix, algorithms such as FastICA and JADE can be used.

[0056] In one embodiment, before selecting a first sound zone from multiple sound zones based on microphone information of multiple sound zones with a number of microphones greater than a preset threshold, the method further includes: performing echo cancellation processing on the acquired sound signal.

[0057] To ensure the quality of subsequent signal processing, after acquiring the sound signals from each audio region, echo cancellation processing is required to complete the initial noise reduction. This operation is performed not only during the signal processing flow but also during the training of the speech separation model.

[0058] In one embodiment, one application scenario of the present invention is as follows: In a car equipped with a 4-channel microphone array, each channel containing 4 microphones, the driver says "turn on the air conditioning" while the front passenger is on the phone, and two passengers in the back are talking. The system processes the 16 microphone signals sequentially using AEC→BF→BSS→NN, outputting a clear voice message "turn on the air conditioning" from the driver's channel. Simultaneously, the front passenger, the passenger behind the driver, and the passenger behind the front passenger each output their respective independent voice messages.

[0059] In one specific embodiment, after the separated speech corresponding to each voice region is sent to the voice interaction system, different interactive functions are implemented based on the wake-up result, recognition result, and partitioning result. For example: Scenario 1: The driver says "Hello XX, turn on the air conditioning". After the system recognizes the wake-up word in the driver's voice zone, it sends the driver's voice separately to ASR recognition, receives the instruction "turn on the air conditioning", and executes the air conditioning operation. At the same time, it does not respond to the dialogue content in other voice zones.

[0060] Scenario 2: The front passenger says, "Hello XX, play a song," while the driver is chatting with the rear passengers. The system accurately recognizes the front passenger's wake-up call and command through the separated front passenger voice area, playing music in the front passenger speaker area without interference from the driver's and rear passengers' voices.

[0061] Scenario 3: The rear passenger on the driver's side says, "Hello XX, navigate to the nearest gas station." The system initiates navigation on the driver's side navigation screen via a separate voice recognition command in the driver's side audio zone, while simultaneously playing a voice prompt in the corresponding audio zone.

[0062] According to the signal processing method of the present invention, a cascaded architecture of "preliminary signal processing + model separation" can be constructed based on signal dimensionality reduction processing and model recognition operations. The pre-processing signal can significantly reduce the dimensionality of the input signal to the model, reduce the difficulty of model separation, and improve the accuracy of the separation results. In the training phase, an end-to-end joint training process can be implemented based on the cascaded architecture, enabling more targeted signal separation training for model separation, improving training effectiveness and the accuracy of the trained speech separation model.

[0063] Figure 3 This is a schematic diagram of the signal flow in the signal processing method based on the present invention. Signals collected from four audio zones—driver's seat, passenger's seat, rear passenger's seat, and rear passenger's seat—are sequentially processed by echo cancellation and beamforming to obtain the processed signal for each audio zone. These processed signals are then combined into a 4-channel signal input to the blind source separation process. The blind source separation process performs initial separation based on the 4-channel signal and inputs the initial separation results into a separation model for further refined separation and recognition. The output signal for each audio zone is used to determine the speech command signal for each audio zone.

[0064] Figure 4 This is a schematic diagram of the main modules of the signal processing apparatus provided in an embodiment of the present invention. Figure 4 As shown, the signal processing device 400 mainly includes a first acquisition module 401, a first processing module 402, and a first separation module 403.

[0065] The first acquisition module 401 is used to acquire microphone information and collected sound signals of the vehicle to be processed in multiple sound zones; The first processing module 402 is used to select a first sound zone from multiple sound zones based on the microphone information of multiple sound zones, wherein the number of microphones is greater than a preset threshold, and to perform beamforming processing on the sound signal of the first sound zone based on the spatial information of the microphones in the first sound zone. The first separation module 403 is used to call the speech separation model to perform speech separation based on the sound signal processed based on the first voice region and the sound signals of multiple voice regions other than the first voice region, so as to obtain the speech command signal and the target voice region corresponding to the speech command signal. The speech separation model is trained based on a preset mixed audio dataset.

[0066] The signal processing apparatus according to embodiments of the present invention can compress acquired multi-channel signals into single-channel signals through dimensionality reduction processing, thereby reducing the complexity of the input signals in the subsequent speech separation process, thus reducing the difficulty of the speech separation process and improving the accuracy of the speech separation results. This method of first reducing the dimensionality of the original signal and then using the speech separation model for fine separation can reduce the complexity of the model's learning task and enable the model to achieve better separation results.

[0067] In one embodiment, the signal processing device 400 further includes: a second acquisition module 404 (not shown in the figure), used to acquire size information of a first vehicle of the same model as the vehicle to be processed, and to determine each sound zone of the first vehicle based on the size information; a generation module 405 (not shown in the figure), used to generate a speech impulse response of each sound zone of the first vehicle based on the microphone position information of the first vehicle and preset sound source position information; a calculation module 406 (not shown in the figure), used to perform convolution operation on preset multiple single-channel speech signals and the speech impulse response of each sound zone of the first vehicle to obtain the sound source signal of each sound zone of the first vehicle; and a construction module 407 (not shown in the figure), used to construct a preset mixed audio dataset based on the sound source signal of each sound zone of the first vehicle.

[0068] In one embodiment, the signal processing device 400 further includes: a second processing module 408 (not shown in the figure), used to perform beamforming processing on the sound source signals in the preset mixed audio dataset; and a training module 409 (not shown in the figure), used to train a speech separation model based on the processed sound source signals and the single-channel speech signals corresponding to the sound source signals.

[0069] In one embodiment, the signal processing device 400 further includes: a recognition module 410 (not shown in the figure), used to perform speech recognition on the speech command signal to obtain the command content; and a sending module 411 (not shown in the figure), used to generate a speech command based on the command content and the target voice region corresponding to the speech command signal, and send the speech command to the functional unit corresponding to the speech command so that the corresponding functional unit executes the speech command.

[0070] In one embodiment, the signal processing device 400 further includes: a preprocessing module 412 (not shown in the figure), used to preprocess the sound signal processed in the first tone region and the sound signals in the tone regions other than the first tone region in the plurality of tone regions, and to calculate a separation matrix based on the preprocessed sound signal; and a second separation module 413 (not shown in the figure), used to perform preliminary separation of the preprocessed sound signal based on the separation matrix.

[0071] In one embodiment, the signal processing device 400 further includes an echo cancellation module 414 (not shown) for performing echo cancellation processing on the acquired sound signal.

[0072] The signal processing apparatus according to embodiments of the present invention can construct a cascaded architecture of "preliminary signal processing + model separation" based on signal dimensionality reduction processing and model recognition operations. The pre-processing signal can significantly reduce the dimensionality of the input signal to the model, reduce the difficulty of model separation, and improve the accuracy of the separation results. Specifically, during the training phase, an end-to-end joint training process can be implemented based on the cascaded architecture, enabling more targeted signal separation training for model separation, improving training effectiveness and the accuracy of the trained speech separation model.

[0073] Figure 5 An exemplary system architecture 500 is shown, to which the signal processing method or signal processing apparatus of embodiments of the present invention can be applied.

[0074] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0075] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, and 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0076] Terminal devices 501, 502, and 503 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0077] Server 505 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 501, 502, and 503 (for example only). The backend management server can analyze and process data such as received signal processing requests, and feed back the processing results (such as target push information, product information voice command signals - for example only) to the terminal devices.

[0078] It should be noted that the signal processing method provided in the embodiments of the present invention is generally executed by the server 505, and correspondingly, the signal processing device is generally set in the server 505.

[0079] It should be understood that Figure 5 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0080] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing terminal devices or servers of the present invention. Figure 6 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0081] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0082] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0083] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.

[0084] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0086] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including a first acquisition module, a first processing module, and a separation module. The names of these units or modules do not necessarily limit the specific unit or module itself; for example, the first acquisition module can also be described as "a module for acquiring microphone information and collected sound signals from a vehicle in multiple audio zones."

[0087] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: Acquire microphone information and collected sound signals from the vehicle to be processed in multiple audio zones; Based on the microphone information from multiple sound zones, a first sound zone with a number of microphones greater than a preset threshold is selected from the multiple sound zones, and beamforming processing is performed on the sound signal of the first sound zone based on the spatial information of the microphones in the first sound zone. The speech separation model is invoked to separate the speech signal from the processed sound signal based on the first sound region and the sound signals from multiple sound regions other than the first sound region, thereby obtaining the speech command signal and the target sound region corresponding to the speech command signal. The speech separation model is trained based on a preset mixed audio dataset.

[0088] According to the technical solution of this invention, multi-channel signals can be compressed into single-channel signals through dimensionality reduction processing, reducing the complexity of the input signals in the subsequent speech separation process, thereby reducing the difficulty of the speech separation process and improving the accuracy of the speech separation results. This method of first reducing the dimensionality of the original signal and then using the speech separation model for fine separation can reduce the complexity of the model's learning task and enable the model to achieve better separation results.

[0089] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A signal processing method, characterized in that, include: Acquire microphone information and collected sound signals from the vehicle to be processed in multiple audio zones; Based on the microphone information of the multiple sound zones, a first sound zone with a number of microphones greater than a preset threshold is selected from the multiple sound zones, and beamforming processing is performed on the sound signal of the first sound zone based on the spatial information of the microphones in the first sound zone. The speech separation model is invoked to perform speech separation based on the sound signal processed from the first voice region and the sound signals from the multiple voice regions other than the first voice region, to obtain a speech command signal and the target voice region corresponding to the speech command signal. The speech separation model is trained based on a preset mixed audio dataset.

2. The method according to claim 1, characterized in that, Before invoking the speech separation model to perform speech separation based on the processed sound signal from the first voice region and the sound signals from the multiple voice regions excluding the first voice region, and obtaining the speech command signal and the target voice region corresponding to the speech command signal, the method further includes: Obtain the size information of a first vehicle of the same model as the vehicle to be processed, and determine each tone zone of the first vehicle based on the size information; Based on the microphone location information and preset sound source location information of the first vehicle, a speech impulse response for each sound zone of the first vehicle is generated; Convolution operation is performed on multiple preset single-channel speech signals and the speech impulse response of each sound zone of the first vehicle to obtain the sound source signal of each sound zone of the first vehicle; The preset mixed audio dataset is constructed based on the sound source signals of each sound zone of the first vehicle.

3. The method according to claim 2, characterized in that, After constructing the preset mixed audio dataset based on the sound source signals of each sound zone of the first vehicle, the method further includes: Beamforming processing is performed on the sound source signals in the preset mixed audio dataset; The speech separation model is trained based on the processed sound source signal and the corresponding single-channel speech signal.

4. The method according to claim 1, characterized in that, The method further includes: The voice command signal is subjected to speech recognition to obtain the command content; A voice instruction is generated based on the instruction content and the target voice region corresponding to the voice instruction signal, and the voice instruction is sent to the corresponding functional unit so that the corresponding functional unit executes the voice instruction.

5. The method according to any one of claims 1-4, characterized in that, Before invoking the speech separation model to perform speech separation based on the processed sound signal from the first voice region and the sound signals from the multiple voice regions excluding the first voice region, and obtaining the speech command signal and the target voice region corresponding to the speech command signal, the method further includes: The sound signal processed in the first sound region and the sound signals in the multiple sound regions other than the first sound region are preprocessed, and a separation matrix is ​​calculated based on the preprocessed sound signals. The preprocessed audio signal is initially separated based on the separation matrix.

6. The method according to any one of claims 1-4, characterized in that, Before selecting a first audio region from the multiple audio regions based on the microphone information of the multiple audio regions, where the number of microphones is greater than a preset threshold, the method further includes: The acquired sound signals are subjected to echo cancellation processing.

7. A signal processing apparatus, characterized in that, include: The first acquisition module is used to acquire microphone information and collected sound signals of the vehicle to be processed in multiple sound zones; The first processing module is used to filter out a first sound zone from the multiple sound zones based on the microphone information of the multiple sound zones, wherein the number of microphones is greater than a preset threshold, and to perform beamforming processing on the sound signal of the first sound zone based on the spatial information of the microphones in the first sound zone. The separation module is used to call the speech separation model to perform speech separation based on the sound signal processed by the first voice region and the sound signals of the voice regions other than the first voice region among the multiple voice regions, to obtain the speech command signal and the target voice region corresponding to the speech command signal. The speech separation model is trained based on a preset mixed audio dataset.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images