Method, device and equipment for training sound source separation model

Abstract

The embodiment of the invention discloses a method, device and equipment for training a sound source separation model. In the embodiments of the present invention, a plurality of sound signals of a wearer and / or a non-wearer are obtained, the plurality of sound signals are generated by recording a playing signal by an intelligent device in different scenes and different paths, and the playing signal is a sound signal emitted by the wearer and / or the non-wearer; determining a plurality of room impulse response (RIR) functions according to the plurality of sound signals; generating a plurality of basic training data according to the plurality of RIR functions; training a basic separation model according to the multiple basic training data; acquiring a plurality of real scene data of a wearer and a non-wearer; and performing fine tuning on the basic separation model according to the multiple pieces of real scene data to generate a target separation model. Through the method, the target separation model is generated, the voice signal separation effect is effectively improved, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a method, apparatus, and device for training a sound source separation model. Background Technology

[0002] With the rapid development of artificial intelligence (AI) and smart wearable device technologies, smart glasses, as a new generation of human-computer interaction carrier, are gradually moving from concept to practical application. Smart glasses integrate multiple cutting-edge technologies such as speech recognition, computer vision, and augmented reality, aiming to provide users with a more natural and convenient interactive experience. Among them, voice interaction, as the most intuitive human-computer interface, has become an important component of the core technology of smart glasses. In voice interaction technology, the accuracy of speech signal separation and recognition directly determines the user experience. Smart glasses face the challenge of complex and ever-changing acoustic environments in actual use. Unlike traditional fixed voice devices, smart glasses need to process multiple sound source signals from different directions and distances while in motion, including the wearer's own voice commands, conversations in the surrounding environment, background noise, etc. The aforementioned complex and ever-changing acoustic environment makes traditional voice processing methods difficult to meet the application requirements of smart glasses.

[0003] In existing technologies, speech signal separation is achieved using speech processing systems based on simple room impulse response (RIR) simulation, traditional beamforming speech enhancement systems, and traditional speech processing methods based on blind source separation. However, the aforementioned speech processing systems based on simple RIR simulation suffer from drawbacks such as low simulation accuracy, insufficient realism, and weak generalization ability. The aforementioned traditional beamforming speech enhancement systems suffer from drawbacks such as low spatial resolution, poor dynamic adaptability, weak ability to handle complex environments, and limited algorithm capabilities. The aforementioned traditional speech processing methods based on blind source separation suffer from drawbacks such as slow convergence speed, uncertainty in the arrangement of the separated signals, high computational complexity, and limited algorithm capabilities. These drawbacks lead to poor speech signal separation results.

[0004] In conclusion, improving the speech signal separation effect is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method, apparatus, and device for training a sound source separation model, generating a target separation model, effectively improving the speech signal separation effect, and enhancing the user experience.

[0006] In a first aspect, embodiments of the present invention provide a method for training a sound source separation model, the method comprising: acquiring multiple sound signals from a wearer and / or a non-wearer, wherein the multiple sound signals are generated by a smart device recording playback signals under different scenarios and paths, and the playback signals are sound signals emitted by the wearer and / or the non-wearer; determining multiple room impulse response (RIR) functions based on the multiple sound signals; generating multiple basic training data based on the multiple RIR functions; training a basic separation model based on the multiple basic training data; acquiring multiple real-world scene data of the wearer and non-wearer; and fine-tuning the basic separation model based on the multiple real-world scene data to generate a target separation model.

[0007] Optionally, determining multiple room impulse response (RIR) functions based on the multiple sound signals specifically includes: aligning the multiple sound signals with the playback signal; performing time correction on the multiple sound signals; performing frequency sweeping and signal quality inspection on the time-corrected multiple sound signals to determine multiple normal sound signals; and performing deconvolution calculation on the multiple normal generated sound signals to generate multiple RIR functions.

[0008] Optionally, the method further includes: naming and saving the RIR function according to a set data naming format.

[0009] Optionally, generating multiple basic training data based on the multiple RIR functions specifically includes: processing single-channel clean speech or multi-channel speech through the multiple RIR functions respectively to generate basic training data, wherein the basic training data includes the input data and output data of the basic separation model.

[0010] Optionally, the method further includes: superimposing real noise to generate the basic training data.

[0011] Optionally, the basic separation model is a deep neural network model.

[0012] Optionally, acquiring multiple real-world scene data of the wearer and non-wearer specifically includes: acquiring multiple quiet scene voice signals of the wearer and non-wearer; and superimposing the multiple quiet scene voice signals with multiple noisy scene signals according to a set signal-to-noise ratio and signal-to-interference ratio to generate the multiple real-world scene data.

[0013] Optionally, the method further includes: obtaining at least two RIR functions from the plurality of room impulse response RIR functions that match the business requirements, based on the scenario range of the wearer and / or non-wearer described in the business requirements.

[0014] Optionally, the different scenarios include anechoic chamber scenarios, conference room scenarios, office area scenarios, and outdoor scenarios; the different paths include the voice path of the wearer and the voice path of the non-wearer.

[0015] Secondly, embodiments of the present invention provide an apparatus for training a sound source separation model, the apparatus comprising: The system includes an acquisition unit for acquiring multiple sound signals from the wearer and / or non-wearer, wherein the multiple sound signals are generated by a smart device recording and playing signals under different scenarios and paths, and the playing signals are sound signals emitted by the wearer and / or non-wearer; a determination unit for determining multiple room impulse response (RIR) functions based on the multiple sound signals; a generation unit for generating multiple basic training data based on the multiple RIR functions; a training unit for training a basic separation model based on the multiple basic training data; the acquisition unit is further configured to: acquire multiple real-world scene data of the wearer and non-wearer; and a fine-tuning unit for fine-tuning the basic separation model based on the multiple real-world scene data to generate a target separation model.

[0016] Optionally, the determining unit is specifically used for: Align the multiple audio signals with the playback signal, and perform time correction on the multiple audio signals; perform frequency sweeping and signal quality inspection on the time-corrected multiple audio signals to determine multiple normal audio signals; perform deconvolution calculation on the multiple normal generated audio signals to generate multiple RIR functions.

[0017] Optionally, the device further includes a storage unit for naming and saving the RIR function according to a set data naming format.

[0018] Optionally, the generation unit is further configured to: process single-channel clean speech or multi-channel speech through the plurality of RIR functions respectively to generate basic training data, wherein the basic training data includes the input data and output data of the basic separation model.

[0019] Optionally, the generation unit is further configured to: superimpose real noise to generate the basic training data.

[0020] Optionally, the basic separation model is a deep neural network model.

[0021] Optionally, the acquisition unit is further configured to: acquire multiple quiet scene voice signals from the wearer and the non-wearer; and superimpose the multiple quiet scene voice signals with multiple noisy scene signals according to a set signal-to-noise ratio and signal-to-interference ratio to generate the multiple real scene data.

[0022] Optionally, the acquisition unit is further configured to: acquire at least two RIR functions from the plurality of room impulse response RIR functions that match the business requirements, based on the scenario range of the wearer and / or non-wearer in the business requirements.

[0023] Optionally, the different scenarios include anechoic chamber scenarios, conference room scenarios, office area scenarios, and outdoor scenarios; the different paths include the voice path of the wearer and the voice path of the non-wearer.

[0024] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect or any one of the possible methods of the first aspect.

[0025] Fourthly, embodiments of the present invention provide a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method as described in the first aspect or any one of the possibilities of the first aspect.

[0026] In this embodiment of the invention, multiple sound signals from the wearer and / or non-wearer are acquired. These multiple sound signals are generated by a smart device recording and playing signals under different scenarios and paths, and the playing signals are sound signals emitted by the wearer and / or non-wearer. Multiple room impulse response (RIR) functions are determined based on these multiple sound signals. Multiple basic training data are generated based on these RIR functions. A basic separation model is trained based on the multiple basic training data. Multiple real-world scene data of the wearer and non-wearer are acquired. The basic separation model is then fine-tuned based on the multiple real-world scene data to generate a target separation model. Through this method, the target separation model is generated using multi-dimensional training data, effectively improving the speech signal separation effect and enhancing the user experience. Attached Figure Description

[0027] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a flowchart of a method for training a sound source separation model according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the placement of a smart wearable device and a speaker in an embodiment of the present invention; Figure 3 This is a flowchart of a method for determining the RIR function according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the generation of basic training data in an embodiment of the present invention; Figure 5 This is another flowchart for generating basic training data in an embodiment of the present invention; Figure 6 This is a flowchart of another method for training a sound source separation model in an embodiment of the present invention; Figure 7 This is a schematic diagram of a device for training a sound source separation model according to an embodiment of the present invention; Figure 8 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0028] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0029] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0030] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0031] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0032] In existing technologies, smart glasses face several technical challenges in speech processing. One major challenge is the difficulty in effectively separating the wearer's speech from the surrounding environment. Specifically, current smart glasses typically employ simple energy thresholding or directional filtering methods for speech separation. While these methods perform well in quiet environments, they are less effective in complex real-world environments. The primary reasons for this problem are the lack of accurate modeling of the wearer's speech-specific acoustic propagation path, the inability to effectively distinguish the acoustic characteristics of the wearer's speech from near-field conversations, and the neglect of the impact of the glasses' wearing status on acoustic signal propagation. Another challenge is the insufficient quality and diversity of training data. Currently, mainstream speech separation algorithms rely on large amounts of high-quality training data, but existing datasets suffer from limitations. The following limitations exist: First, most datasets are collected using fixed microphone arrays, which cannot reflect the acoustic characteristics of smart glasses when worn; second, there is a lack of specialized data for near-field voice interaction scenarios; third, the data annotation accuracy is limited, making it difficult to support the requirements of high-precision separation algorithms; and finally, the scenario coverage is incomplete, failing to encompass the various usage environments that smart glasses may encounter. Third, the acoustic environment adaptability is poor. Specifically, smart glasses face drastically different acoustic environments in different usage scenarios. For example, in indoor environments, there are abundant early reflections and reverberation; in outdoor environments, the main challenges are wind noise and traffic noise interference; in meeting scenarios, the complex situation of multiple people speaking simultaneously needs to be handled; and in mobile scenarios, the Doppler effect and wind noise are introduced by device movement.

[0033] Currently, three main methods are commonly used for speech signal separation: Method 1, a speech processing system based on simple room impulse response (RIR) simulation; Method 2, a traditional beamforming speech enhancement system; and Method 3, a traditional speech processing method based on blind source separation. Method 1 specifically uses geometric acoustics to simulate room impulse response, trains and tests algorithms based on simulated RIR data, uses a simplified acoustic environment model, and employs basic signal processing algorithms. However, Method 1 suffers from simulation accuracy defects, primarily due to its simplified physical modeling (geoacoustic methods ignore complex physical phenomena such as sound wave diffraction and scattering), insufficient environmental complexity (the simulated environment is overly idealized and fails to reflect the complexity of the real environment), and lack of dynamic characteristics (it lacks the ability to model dynamic environmental changes). Method 1 also exhibits poor application performance, specifically due to insufficient realism (simulated RIR data differs significantly from actual measurement data), poor algorithm robustness (algorithms trained on simulated data show significant performance degradation in real-world environments), and limited generalization ability (the limitations of simulation methods). The algorithm's scene adaptability is improved; Method 2 specifically uses a microphone array for spatial filtering, and based on the traditional beamforming algorithm for speech enhancement, it uses a fixed directional pattern and a simple noise suppression method; wherein, the beamforming utilizes the spatial distribution characteristics of the microphone array and forms a directional receiving beam through digital signal processing; Method 2 has spatial resolution defects and complex environment processing defects, wherein the spatial resolution defects are specifically directional limitations, the spatial resolution of traditional beamforming methods is limited, making it difficult to accurately distinguish close sound sources; sidelobe interference, there is obvious sidelobe interference problem, affecting the separation effect; poor dynamic adaptability, unable to effectively track the position changes of moving sound sources; the complex environment processing defects are specifically insufficient reverberation suppression, performance significantly degrades in reverberant environments; difficulty in multi-source processing, it is difficult to effectively process multiple sound sources emitting sound simultaneously; poor noise robustness, poor performance in non-Gaussian noise environments; Method 3 specifically uses Independent Component Analysis (ICA). Blind source separation algorithms such as Component Analysis (ICA) are based on the statistical independence assumption for signal separation, using traditional feature extraction methods and fixed separation criteria. However, these methods suffer from theoretical limitations and practical shortcomings. Specifically, the theoretical limitations include: the independence assumption not holding true (in real-world dialogue scenarios, multi-person speech often exhibits correlation, violating the statistical independence assumption); the linear mixing assumption (while mixing processes in real acoustic environments are often non-linear); and the instantaneous mixing assumption (ignoring the time delay effect of sound wave propagation). The practical shortcomings include: convergence issues (the ICA algorithm sometimes converges slowly or not at all); and arrangement ambiguity (the separated signals suffer from arrangement uncertainty).Traditional blind source separation algorithms suffer from high computational complexity, making them unsuitable for real-time processing. Furthermore, existing technologies include deep learning solutions. In summary, traditional beamforming and blind source separation-based solutions, due to their algorithmic limitations, cannot meet the business requirements of smart glasses products. Deep learning solutions, constrained by data quality and quantity limitations, also cannot adequately meet the separation requirements for wearers and non-wearers in various scenarios.

[0034] Therefore, how to improve the speech signal separation effect is a problem that needs to be solved.

[0035] In this embodiment of the invention, to solve the above problems, a method for training a sound source separation model is proposed, specifically as follows: Figure 1 As shown, the method includes: Step S101: Acquire multiple sound signals from the wearer and / or non-wearer.

[0036] Specifically, the multiple sound signals are generated by the smart device recording and playing signals in different scenarios and paths, and the playing signals are sound signals emitted by the wearer and / or the non-wearer.

[0037] In one possible implementation, the different scenarios include an anechoic chamber scenario, a conference room scenario, an office area scenario, and an outdoor scenario; wherein, the anechoic chamber scenario is a pure, reflection-free, ideal acoustic environment; the conference room scenario represents a typical indoor office environment with a moderate reverberation time; the office area scenario simulates an open office environment, containing complex background noise and multiple sound sources; and the outdoor scenario represents an outdoor usage environment, mainly facing wind noise and traffic noise interference.

[0038] In one possible implementation, the different paths include the wearer's voice path and the non-wearer's voice path; wherein, the voice path refers to the complete path from the sound source to the sound source, the wearer's voice path uses a smart wearable device to play sound, and the same smart wearable device is used for sound pickup; assuming the smart wearable device is smart glasses, the wearing posture includes a standard wearing posture and a typical offset posture, wherein the typical offset posture includes turning the head to the left, turning the head to the right, and wearing it in combination with myopia glasses, etc., different head shapes are selected when wearing smart glasses, and different scenarios are selected for the wearer's voice path traversal under each path; the non-wearer's voice path uses a speaker to play sound, and the smart wearable device is used for sound pickup; assuming the smart wearable device is smart glasses, the smart glasses are placed in the center, the speaker is placed at a distance of 0.5 meters (m) from the smart glasses with a 4m dialogue range, covering the near field, mid field, and far field, and the speaker is placed at an angle range of 360 degrees, specifically as follows. Figure 2As shown, the speaker, when placed in each dialogue range and angle range, will form a new voice path with the smart wearable device placed in the center. Under each path, different scenarios are selected to traverse the non-wearer's voice path.

[0039] In this embodiment of the invention, by recording voices of the wearer and non-wearer in different scenarios and along different paths, the sound signals can be differentiated, which helps the smart wearable device to perform acoustic characteristic analysis in various practical use scenarios. The smart wearable device can be smart glasses, smart headphones, etc.

[0040] Step S102: Determine multiple room impulse response (RIR) functions based on the multiple sound signals.

[0041] In one possible implementation, determining multiple room impulse response (RIR) functions based on the multiple sound signals specifically includes, for example: Figure 3 As shown, it includes the following: Step S301: Align the plurality of sound signals with the playback signal and perform time correction on the plurality of sound signals.

[0042] Specifically, the signal alignment refers to the precise time synchronization of the plurality of audio signals with the playback signal.

[0043] In one possible implementation, the process of acquiring multiple audio signals before signal alignment can also be referred to as frequency sweep recording.

[0044] Step S302: Sweep and extract multiple time-corrected audio signals to obtain audio signals within the effective frequency response range.

[0045] In one possible implementation, the frequency sweep interception can also be referred to as signal extraction.

[0046] Step S303: Perform signal quality inspection on the sound signals in the effective frequency response range to identify multiple normal sound signals.

[0047] Specifically, the signal quality inspection is used to remove invalid data caused by abnormal operation during the recording process.

[0048] Step S304: Perform deconvolution calculations on the multiple normal generated sound signals to generate multiple RIR functions.

[0049] In this embodiment of the invention, sound signals are recorded differently in different scenarios. Through systematic environment selection, path traversal, and signal quality control, a high-quality and diverse acoustic data foundation is provided for subsequent speech separation model training. This effectively solves the technical problems of existing RIR data acquisition methods, such as simplicity and incomplete scenario coverage.

[0050] Step S103: Generate multiple basic training data based on the multiple RIR functions.

[0051] Specifically, single-channel clean speech or multi-channel speech is processed through the multiple RIR functions to generate basic training data, wherein the basic training data includes the input data and output data of the basic separation model.

[0052] During the process of generating basic training data, real noise can also be superimposed to generate the basic training data.

[0053] The following detailed explanations of single-speaker scenario simulation and multi-speaker scenario simulation are provided through two specific embodiments: Specific Implementation Example 1: Single-Speaker Scene Simulation, as follows: Figure 4 As shown, single-channel clean speech is processed through multiple RIR functions to generate basic training data. First, the clean speech, noise data, and RIR functions are determined, where the clean speech, noise data, and RIR functions are all forced to be single-channel during loading. Then, gain control is determined, including signal-to-noise ratio and volume. Before gain control, the clean speech is processed by the RIR functions. The clean speech processed by the RIR functions undergoes early RIR and gain control to generate the clean speech's direct speech tone (speech-drb). The clean speech's direct speech tone is determined as the target speech, which is the output data of the basic separation model. Then, based on the clean speech processed by the RIR functions and gain control, the clean speech's reverb (speech-rvb) is generated. The clean speech's reverb and mono noise (mono) are then combined... The noise is superimposed to generate mixed speech, which is the input data of the basic separation model. The mono noise can also be controlled by gain. After the above processing, a large amount of basic training data can be generated, covering various acoustic conditions.

[0054] In one possible implementation, label information, such as speaker location and scene, can also be set in each piece of basic training data.

[0055] Specific Implementation Example 2: Simulation of a Multi-Speaker Scene, as follows: Figure 5 As shown, the multi-channel speech is processed by the multiple RIR functions to generate basic training data.

[0056] First, the clean speech source noise, RIR function, recorded noise, and interference speech are identified. The clean speech, source noise, and interference speech are all forced to be single-channel during loading. The source noise is single-channel noise. The RIR function is multi-channel during loading, and the recorded noise is multi-channel and matched to the RIR function. Then, gain control is determined, including signal-to-interference ratio (SIR), signal-to-noise ratio (SNR), and volume. Before gain control, the clean speech is processed by the RIR function. The RIR-processed clean speech undergoes early RIR and gain control to generate the clean speech direct tone (speech-DRB). Based on the RIR-processed clean speech and gain control, clean speech reverb (speech-RVB) is generated. The recorded noise is bypassed, and the mono noise is up-mixed to generate multi-channel noise. The noise is processed by an RIR function. The noise is then subjected to early RIR processing and gain control to generate the noise's direct speech tone (disturb-drb). Based on the clean speech processed by the RIR function and gain control, the noise's reverberation tone (disturb-rvb) is generated. The clean speech's reverberation tone, multichannel noise, and the noise's reverberation tone are then superimposed to generate a mixture, which is the input data of the basic separation model. The multichannel noise can also be controlled by gain. The direct speech tone of the clean speech and the noise's direct speech tone are used to determine a signal combine, which is then used to determine the target speech, which is the output data of the basic separation model.

[0057] In one possible implementation, label information, such as speaker location and scene, can also be set in each piece of basic training data.

[0058] In this embodiment of the invention, a large amount of basic training data can be generated through the above method, and the balance and representativeness of the basic training data in data distribution can be determined. During simulation training, configuration can be performed, and the configuration includes regular training configuration and data path configuration. The regular training configuration includes batch size, number of processes, etc.; the data path configuration includes speech list files, noise list files, and room impulse response list files, etc.

[0059] Step S104: Train the basic separation model based on the multiple basic training data.

[0060] Specifically, the basic separation model is a deep neural network model. The deep neural network architecture of the model includes Conv-TasNet, Dual-Path Recurrent Neural Network (DPRNN), etc. The optimization objective function of the basic separation model includes signal distortion ratio and source interference ratio, etc. The basic separation model has basic speech separation capability, making the signal distortion ratio greater than or equal to 10dB. This is only an example for illustration.

[0061] In this embodiment of the invention, a general separation model that can work in a variety of acoustic environments is obtained through the above steps, and the ability to process speech signals at different distances and angles is established, providing a reliable initial model for subsequent fine processing.

[0062] Step S105: Obtain multiple real-world scenario data from both the wearer and non-wearers.

[0063] In one possible implementation, acquiring multiple real-world scene data of the wearer and non-wearer specifically includes: acquiring multiple quiet scene voice signals of the wearer and non-wearer; and superimposing the multiple quiet scene voice signals with multiple noisy scene signals according to a set signal-to-noise ratio and signal-to-interference ratio to generate the multiple real-world scene data.

[0064] For example, when a wearer is reading corpus in a quiet environment after wearing the smart wearable device, the voice signal received by the smart wearable device is identified as a quiet environment voice signal; when a non-wearer is reading corpus in a quiet environment after wearing the smart wearable device, the voice signal received by the smart wearable device is also identified as a quiet environment voice signal.

[0065] Step S106: Fine-tune the basic separation model based on the multiple real-world scenario data to generate the target separation model.

[0066] In this embodiment of the invention, the basic separation model is fine-tuned using multiple real-world scenario data from both wearers and non-wearers. This allows the speech separated by the target separation model to be closer to the quality of a real person's voice. During the fine-tuning process, the basic separation model is finely adjusted using a small learning rate. This reduces the sound quality loss and separation defects caused by inaccurate RIR function recording during the training of the basic separation model, thereby improving the speech quality in real-world wearing scenarios.

[0067] In one possible implementation, after step S102, the method further includes other steps, specifically as follows: Figure 6 As shown, it includes the following: Step S107: Name and save the RIR function according to the set data naming format.

[0068] In one possible implementation, after determining the recording scene and recording path of the smart device, the naming format of the RIR function can be pre-set. For example, the naming format of the RIR function corresponding to the wearer can be rir_mouth_scene_pos.wav, where rir_mouth identifies the wearer's voice-related RIR function; scene represents the scene identifier, assuming scene1-4 correspond to an anechoic chamber scene, a conference room scene, an office area scene, and an outdoor scene, respectively; pos represents the posture identifier, assuming pos0 is a standard posture and pos1 is a non-standard posture; the naming format of the RIR function corresponding to the wearer can be rir_talker_scene_dis*m_ang*.wav, where rir_ The talker identifier is the non-wearer's voice-related RIR function; the scene represents the scene identifier, assuming that scene1-4 correspond to the anechoic chamber scene, conference room scene, office area scene, and outdoor scene, respectively; the dis*m represents the distance parameter, covering key distances such as 0.5m, 1m, and 4m; the ang* represents the angle parameter, assuming that the front is 0°, the left is -90°, and the right is 90°; this is only an illustrative example and should be determined according to the actual situation.

[0069] In one possible implementation, business requirements can be analyzed based on different business scenarios, and the scenario ranges for the wearer and / or non-wearer can be set. For example, if the business requirement is to minimize the wearer's range, then a very small area of ​​the wearer's region is considered as the wearer, and the rest is considered as the non-wearer, for business requirements such as voice calls that require maximum suppression of external interference. If the business requirement requires a customized wearer's range, for example, extracting the wearer's area and the area directly in front, suppressing signals in other areas, this is suitable for specific business requirements, and the scenario ranges for the wearer and / or non-wearer can be determined according to actual needs. Based on the scenario ranges for the wearer and / or non-wearer, at least two RIR functions that match the business requirement from the multiple room impulse response RIR functions are obtained, i.e., the RIR function for the wearer and the RIR function for the non-wearer are determined, and then targeted adaptation training is performed.

[0070] In one possible implementation, after determining the RIR function of the wearer and the non-wearer according to business requirements, basic training data for simulation training is generated. The basic separation model is trained based on the basic training data, and then the basic separation model is fine-tuned several times based on real-world scenario data to generate a target separation model for different business requirements.

[0071] Through the above embodiments, a large amount of open source data can be fully utilized to train the separation model, obtain effective separation results between wearers and non-wearers, and take advantage of the controllability of the RIR function to obtain different separation results for different business needs.

[0072] In this embodiment of the invention, a target separation model is generated through a three-level progressive processing architecture of basic training, fine-tuning, and business adaptation. This model possesses both general capabilities and the ability to provide personalized services, balancing the dual requirements of algorithm performance and user experience. The differentiated RIR recording strategy for wearers and non-wearers provides a high-quality acoustic dataset specifically for smart wearable devices, significantly improving speech separation performance in complex environments. The configurable separation effect mechanism based on business needs enables personalized voice interaction experiences and supports productization requirements for different application scenarios.

[0073] In this embodiment of the invention, an apparatus for training a sound source separation model is provided, such as... Figure 7 As shown, it specifically includes: an acquisition unit 701, a determination unit 702, a generation unit 703, a training unit 704, and a fine-tuning unit 705; The acquisition unit 701 is used to acquire multiple sound signals from the wearer and / or non-wearer, wherein the multiple sound signals are generated by the smart device recording and playing signals under different scenarios and paths, and the playing signals are sound signals emitted by the wearer and / or non-wearer; the determination unit 702 is used to determine multiple room impulse response (RIR) functions based on the multiple sound signals; the generation unit 703 is used to generate multiple basic training data based on the multiple RIR functions; the training unit 704 is used to train a basic separation model based on the multiple basic training data; the acquisition unit 701 is also used to acquire multiple real-world scene data of the wearer and non-wearer; the fine-tuning unit 705 is used to fine-tune the basic separation model based on the multiple real-world scene data to generate a target separation model.

[0074] Furthermore, the determining unit is specifically used for: Align the multiple audio signals with the playback signal, and perform time correction on the multiple audio signals; perform frequency sweeping and signal quality inspection on the time-corrected multiple audio signals to determine multiple normal audio signals; perform deconvolution calculation on the multiple normal generated audio signals to generate multiple RIR functions.

[0075] Furthermore, the device also includes a storage unit for naming and saving the RIR function according to a set data naming format.

[0076] Furthermore, the generation unit is also used to: process single-channel clean speech or multi-channel speech through the plurality of RIR functions respectively to generate basic training data, wherein the basic training data includes the input data and output data of the basic separation model.

[0077] Furthermore, the generation unit is also used to: superimpose real noise to generate the basic training data.

[0078] Furthermore, the basic separation model is a deep neural network model.

[0079] Furthermore, the acquisition unit is also used to: acquire multiple quiet scene voice signals from the wearer and the non-wearer; and superimpose the multiple quiet scene voice signals with multiple noisy scene signals according to a set signal-to-noise ratio and signal-to-scratching ratio to generate the multiple real scene data.

[0080] Furthermore, the acquisition unit is also used to: acquire at least two RIR functions from the plurality of room impulse response RIR functions that match the business requirements, based on the scenario range of the wearer and / or non-wearer in the business requirements.

[0081] Furthermore, the different scenarios include anechoic chamber scenarios, conference room scenarios, office area scenarios, and outdoor scenarios; the different paths include the voice path of the wearer and the voice path of the non-wearer.

[0082] Figure 8 This is a schematic diagram of the structure of the electronic device described in an embodiment of the present invention. Figure 8 As shown, it includes a general computer hardware architecture, which includes at least a processor 801 and a memory 802. The processor 801 and the memory 802 are connected via a bus 803. The memory 802 is adapted to store instructions or programs executable by the processor 801. The processor 801 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 801 executes the instructions stored in the memory 802 to perform the method flow of the embodiments of the present invention as described above, thereby realizing data processing and control of other devices. The bus 803 connects the above-mentioned components together, and also connects the above-mentioned components to a display controller 804, a display device, and an input / output (I / O) device 805. The input / output (I / O) device 805 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 805 is connected to the system via an input / output (I / O) controller 806.

[0083] The instructions stored in memory 802 are executed by at least one processor 801 to: acquire multiple sound signals from the wearer and / or non-wearer; determine multiple room impulse response (RIR) functions based on the multiple sound signals; generate multiple basic training data based on the multiple RIR functions; train a basic separation model based on the multiple basic training data; acquire multiple real-world scene data from the wearer and non-wearer; and fine-tune the basic separation model based on the multiple real-world scene data to generate a target separation model.

[0084] Specifically, the electronic device includes: one or more processors 801 and a memory 802. Figure 8 Take a processor 801 as an example. The processor 801 and the memory 802 can be connected via a bus or other means. Figure 8 Taking a bus connection as an example, memory 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Processor 801 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 802, thereby implementing the aforementioned method for determining and training the sound source separation model.

[0085] Memory 802 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 802 may optionally include memory remotely located relative to processor 801, and these remote memories can be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0086] One or more modules are stored in memory 802, and when executed by one or more processors 801, they perform the method for training the sound source separation model in any of the above method embodiments.

[0087] As those skilled in the art will recognize, various aspects of the embodiments of the present invention can be implemented as a system, method, or computer program product. Therefore, various aspects of the embodiments of the present invention can take the form of a completely hardware implementation, a completely software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software and hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." Furthermore, various aspects of the embodiments of the present invention can take the form of a computer program product implemented in one or more computer-readable media having computer-readable program code implemented thereon.

[0088] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, (but not limited to) an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include: an electrical connection having one or more wires, a portable computer floppy disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the context of embodiments of the present invention, a computer-readable storage medium can be any tangible medium capable of containing or storing a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0089] Computer-readable signal media may include propagated digital signals having computer-readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and can communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0090] Program code implemented on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0091] Computer program code for performing operations relating to various aspects of embodiments of the present invention can be written in any combination of one or more programming languages, including: object-oriented programming languages ​​such as Java, Smalltalk, C++, etc.; and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code can be executed as a standalone software package entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet provided by an Internet service provider).

[0092] The flowchart illustrations and / or block diagrams of the methods, apparatus (systems), and computer program products according to embodiments of the present invention describe various aspects of the embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions (executed via the processor of the computer or other programmable data processing apparatus) create means for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0093] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus or other means to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of writing that includes instructions that implement the functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0094] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operable steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide for implementing the functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0095] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0096] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding access points are provided for users to choose to authorize or refuse processing. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

Claims

1. A method for training a sound source separation model, characterized in that, The method includes: Acquire multiple sound signals from the wearer and / or non-wearer, wherein the multiple sound signals are generated by the smart device recording and playing signals in different scenarios and paths, and the playing signals are sound signals emitted by the wearer and / or non-wearer; Determine multiple room impulse response RIR functions based on the multiple sound signals; Multiple basic training data are generated based on the multiple RIR functions; Train a basic separation model based on the aforementioned multiple basic training data; Acquire real-world scenario data from both wearers and non-wearers; The target separation model is generated by fine-tuning the basic separation model based on the data from multiple real-world scenarios.

2. The method according to claim 1, characterized in that, The step of determining multiple room impulse response (RIR) functions based on the multiple sound signals specifically includes: Align the plurality of audio signals with the playback signal, and perform time correction on the plurality of audio signals; Multiple time-corrected audio signals were subjected to frequency sweeping and signal quality inspection to identify multiple normal audio signals. Multiple normal generated sound signals are deconvolved to generate multiple RIR functions.

3. The method according to claim 1, characterized in that, The method further includes: Name and save the RIR function according to the set data naming format.

4. The method according to claim 1, characterized in that, The generation of multiple basic training data based on the multiple RIR functions specifically includes: Single-channel clean speech or multi-channel speech is processed through the multiple RIR functions to generate basic training data, wherein the basic training data includes the input data and output data of the basic separation model.

5. The method according to claim 4, characterized in that, The method further includes: The basic training data is generated by superimposing real noise.

6. The method according to claim 1, characterized in that, The basic separation model is a deep neural network model.

7. The method according to claim 1, characterized in that, The acquisition of multiple real-world scenario data from both wearers and non-wearers specifically includes: Acquire multiple quiet scene voice signals from the wearer and the non-wearer; The multiple quiet scene voice signals and multiple noisy scene signals are superimposed according to a set signal-to-noise ratio and signal-to-scratching ratio to generate the multiple real scene data.

8. The method according to claim 1, characterized in that, The method further includes: Based on the scenario range of wearers and / or non-wearers described in the business requirements, at least two RIR functions that match the business requirements are obtained from the plurality of room impulse response RIR functions.

9. The method according to claim 1, characterized in that, The different scenarios include anechoic chamber scenarios, conference room scenarios, office area scenarios, and outdoor scenarios; the different paths include the voice path of the wearer and the voice path of the non-wearer.

10. An apparatus for training a sound source separation model, characterized in that, The device includes: The acquisition unit is used to acquire multiple sound signals from the wearer and / or non-wearer, wherein the multiple sound signals are generated by the smart device recording and playing signals in different scenarios and different paths, and the playing signals are sound signals emitted by the wearer and / or non-wearer; The determining unit is used to determine multiple room impulse response RIR functions based on the multiple sound signals; A generation unit is used to generate multiple basic training data based on the multiple RIR functions; Training unit, used to train basic separation model based on the multiple basic training data; The acquisition unit is also used to: acquire multiple real-world scenario data of the wearer and non-wearers; The fine-tuning unit is used to fine-tune the basic separation model based on the multiple real-world scenario data to generate the target separation model.

11. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-9.

Images