Target speaker extraction and voice interaction method and device and related equipment

Abstract

The invention discloses a target speaker extraction method and device, a voice interaction method and device, related equipment and a computer program product. On the basis of performing voice separation on multi-channel original voice signals, a wake-up model is used for performing wake-up recognition on separated voice signals of each area; the method comprises the following steps: obtaining wake-up endpoint information in a separated voice signal of a target area according to a voice signal between wake-up endpoints, extracting attribute information, example voiceprint information, spatial position information and the like of a target speaker according to the voice signal between the wake-up endpoints, and extracting a target speaker based on the attribute information and the voice signal of a wake-up segment through a target speaker extraction model. And estimating to obtain a voice signal related to the target speaker. According to the method, the attribute information of the target speaker estimated in the wake-up stage is used as the reference information to be applied to the recognition stage, so that the distortion of the existing scheme can be effectively reduced, the quality of the extracted voice signal is improved, and the success rate of subsequent voice interaction is improved.

Description

Technical Field

[0001] The present application relates to the field of speech signal processing technology, and more specifically, to a target speaker extraction method, a speech interaction method, an apparatus, related equipment and a computer program product. Background Art

[0002] In the voice interaction scenario, the first thing to do is to extract the voice signal related to the target speaker, and then perform voice recognition, identify the wake-up word, and perform related interaction tasks. In some noisy scenarios, the process of extracting the voice signal related to the target speaker from the multi-channel original voice signals collected by the microphone is easily affected by environmental noise, resulting in low quality of the extracted voice signal, which in turn affects the accuracy of subsequent voice recognition results and the success rate of voice interaction.

[0003] Taking in-vehicle voice interaction as an example, in order to provide a complete vehicle interactive experience, people in different positions in the vehicle are generally allowed to interact freely. This requires that the target speaker should not be affected by other positions when interacting with the vehicle computer. For example, when other people are listening to music, watching movies, or talking, they can also interact effectively with the vehicle computer. In addition, there are noises such as tire noise and wind noise during vehicle driving, which leads to low signal-to-noise ratio and signal-to-interference ratio of the interactive voice. Traditional voice interaction methods generally perform voice separation on the multi-channel original voice signals collected by the microphone to obtain the estimated voice signal for each partition, and use the estimated voice signal to directly perform subsequent interactions. However, in harsh scenarios such as low signal-to-noise ratio and signal-to-interference ratio, the voice separation effect will be greatly reduced, and the separated voice signal will have serious distortion, which will affect the subsequent voice recognition effect and reduce the success rate of interaction. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a target speaker extraction method, a voice interaction method, an apparatus, related equipment and a computer program product to improve the quality of the extracted target speaker's voice signal and improve the success rate of voice interaction. The specific scheme is as follows:

[0005] In a first aspect, a target speaker extraction method is provided, comprising:

[0006] Performing speech separation on the received multi-channel original speech signal to obtain a separated speech signal for each zone, wherein the multi-channel original speech signal is obtained by collecting speech signals from a target space divided into a plurality of partitions through two or more microphones;

[0007] The separated speech signal of each area is sent to the wake-up model to obtain the wake-up endpoint information in the separated speech signal of the target area output by the wake-up model;

[0008] Extracting attribute information of a target speaker from a speech signal located between wake-up endpoints in the separated speech signal of the target area;

[0009] The target speaker extraction model is configured to refer to the attribute information of the target speaker and the speech signal of the wake-up segment to obtain the speech signal related to the target speaker. The speech signal of the wake-up segment is calculated based on the speech signal located between the wake-up endpoints in the separated speech signal of the target area.

[0010] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0011] The separated speech signal of each zone is used to guide the beam forming of each zone to obtain the beam signal of each zone;

[0012] The process of sending the separated speech signal of each area to the wake-up model to obtain the wake-up endpoint information in the separated speech signal of the target area output by the wake-up model includes:

[0013] The beam signal and separated speech signal of each zone are sent to the wake-up model to obtain the wake-up endpoint information in the separated speech signal of the target zone output by the wake-up model.

[0014] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of obtaining a speech signal related to the target speaker by referring to the attribute information of the target speaker and the speech signal of the wake-up segment through the configured target speaker extraction model includes:

[0015] Based on the wake-up endpoint information in the separated voice signal of the target area, extracting the beam signal located between the wake-up endpoints from the beam signal of the target area as the voice signal of the wake-up segment;

[0016] The attribute information of the target speaker and the speech signal of the wake-up segment are sent to a target speaker extraction model to obtain a speech signal related to the target speaker output by the model.

[0017] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of extracting the attribute information of the target speaker from the speech signal located between the wake-up endpoints in the separated speech signal of the target area includes:

[0018] The speech signal located between the wake-up endpoints in the separated speech signal of the target area is sent to the positioning model to obtain the direction information of the target speaker output by the model, and the direction information is used as the attribute information of the target speaker, wherein the positioning model is configured to process the input speech signal to obtain the direction information of the target speaker.

[0019] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of sending the speech signal located between the wake-up endpoints in the separated speech signal of the target area to the positioning model to obtain the direction information of the target speaker output by the model includes:

[0020] The speech signals located between the wake-up endpoints in the separated speech signals of the target area and the multi-channel original speech signals are sent to the positioning model together to obtain the direction information of the target speaker output by the model, and the direction information is used as the attribute information of the target speaker.

[0021] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of extracting the attribute information of the target speaker from the speech signal located between the wake-up endpoints in the separated speech signal of the target area includes:

[0022] Voiceprint extraction is performed on the voice signals located between the wake-up endpoints in the separated voice signals of the target area to obtain the voiceprint information of the target speaker, and the voiceprint information is used as the attribute information of the target speaker.

[0023] In one possible design, in another implementation of the first aspect of the embodiment of the present application, performing speech separation on the received multi-channel original speech signal to obtain a separated speech signal for each zone includes:

[0024] Performing short-time Fourier transform (STFT) on the multi-channel original speech signal to obtain time-frequency features, and sending the features to the separation model to obtain a time-frequency mask of the separated speech signal of each area output by the separation model;

[0025] The time-frequency mask of the separated speech signal of each zone is multiplied by the time-frequency feature respectively, and the separated speech signal of each zone is obtained by inverse short-time Fourier transform (ISTFT);

[0026] The process of guiding the beamforming of each zone by the separated speech signal of each zone to obtain the beam signal of each zone includes:

[0027] Taking the time-frequency mask of the separated speech signal of each zone as a reference, beamforming is performed on the time-frequency features to obtain the beam signal of each zone.

[0028] In a second aspect, a voice interaction method is provided, comprising:

[0029] Obtain multi-channel original speech signals;

[0030] Using any one of the target speaker extraction methods of the first aspect, extracting a speech signal related to the target speaker from the multi-channel original speech signal;

[0031] Sending the speech signal related to the target speaker into the recognition model to obtain a speech recognition result;

[0032] Execute corresponding interactive actions based on the speech recognition result.

[0033] In a third aspect, a target speaker extraction device is provided, comprising:

[0034] A speech separation unit is used to perform speech separation on the received multi-channel original speech signal to obtain a separated speech signal for each zone, wherein the multi-channel original speech signal is obtained by collecting speech signals from a target space divided into a plurality of partitions through two or more microphones;

[0035] A wake-up unit, used to send the separated speech signal of each zone to the wake-up model, and obtain the wake-up endpoint information in the separated speech signal of the target zone output by the wake-up model;

[0036] An attribute extraction unit, configured to extract attribute information of a target speaker from a speech signal located between wake-up endpoints in the separated speech signal of the target area;

[0037] The extraction unit is used to input the attribute information of the target speaker into the target speaker extraction model to obtain the speech signal related to the target speaker output by the model.

[0038] In a fourth aspect, a voice interaction device is provided, comprising:

[0039] A signal acquisition unit, used for acquiring multi-channel original speech signals;

[0040] A speaker extraction unit, configured to extract a speech signal related to a target speaker from the multi-channel original speech signal by using any target speaker extraction method of the first aspect;

[0041] A speech recognition unit, used to send the speech signal related to the target speaker into a recognition model to obtain a speech recognition result;

[0042] An interaction unit is used to perform corresponding interaction actions based on the speech recognition result.

[0043] In a fifth aspect, an electronic device is provided, comprising: a memory and a processor;

[0044] The memory is used to store programs;

[0045] The processor is used to execute the program to implement the target speaker extraction method described in any one of the first aspects of the present application, or to implement the voice interaction method described in the second aspect of the present application.

[0046] In a sixth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the target speaker extraction method described in any one of the first aspects of the present application is implemented, or the speech interaction method described in the second aspect of the present application is implemented.

[0047] In the seventh aspect, a computer program product is provided, including a computer program. When the computer program is executed by a processor, it implements the target speaker extraction method described in any one of the first aspects of the present application, or implements the speech interaction method described in the second aspect of the present application.

[0048] By means of the above technical scheme, the present application further uses the wake-up model to perform wake-up recognition on the separated voice signals of each zone on the basis of voice separation of the multi-channel original voice signals, and obtains the wake-up endpoint information in the separated voice signals of the target zone, that is, identifies the voice signals between the wake-up endpoints that meet the wake-up conditions. Since the voice signal of this segment can meet the wake-up conditions, the effective attribute information of the target speaker can be extracted from it, such as the voiceprint information and spatial position information of the target speaker. The attribute information of the target speaker is used as reference information, and the voice signal related to the target speaker is estimated based on the reference information and the voice signal of the wake-up segment through the configured target speaker extraction model. The present application scheme uses the attribute information of the target speaker estimated in the wake-up stage as reference information and applies it in the recognition stage, which can effectively reduce the distortion of the existing scheme, and can more accurately extract the voice signal related to the target speaker, improve the signal quality, and thus help to improve the success rate of subsequent voice interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0050] Figure 1 A schematic diagram of an implementation system architecture of the target speaker extraction method provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of a terminal structure provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of a server structure provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of a process flow of a target speaker extraction method provided in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of the network structure of a separation model is shown as an example;

[0055] Figure 6 An example of a target speaker extraction system structure diagram is shown;

[0056] Figure 7 Another schematic diagram of the target speaker extraction system is shown as an example;

[0057] Figure 8 An example of a positioning model network structure diagram is provided;

[0058] Fig. 9 A schematic diagram of the network structure of a voiceprint extraction module based on a speaker recognition model is shown as an example;

[0059] Fig.10 Another schematic diagram of the target speaker extraction system is illustrated;

[0060] Fig.11 A schematic diagram of the structure of a target speaker extraction device provided in an embodiment of the present application;

[0061] Fig.12 A schematic diagram of the structure of a voice interaction device provided in an embodiment of the present application;

[0062] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0064] This application provides a target speaker extraction method and a voice interaction method, which can be applied to Figure 1 The system architecture shown in FIG. 1 may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 A server is included as an example for explanation).

[0065] The terminal 100 or the server 200 can be used alone to execute the target speaker extraction method or the voice interaction method provided in the embodiment of the present application. In addition, the terminal 100 and the server 200 can also be used together to execute the target speaker extraction method or the voice interaction method provided in the embodiment of the present application.

[0066] Next describe Figure 1 The product form of the mid-terminal 100;

[0067] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a vehicle-mounted device, a teaching screen, a wearable device, a conference terminal, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0068] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0069] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160, a speaker 161, a microphone 162, an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0070] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0071] Among them, other input devices 132 can receive input data and so on.

[0072] The display unit 140 may be used to display information input by the user or information provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 may be used to display various interactive interfaces, processing results, etc. in the target speaker extraction method or the voice interaction method.

[0073] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0074] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0075] Among them, the memory 120 can be used to store software codes related to the target speaker extraction method or the voice interaction method, the processor 170 can execute the steps of the target speaker extraction method or the voice interaction method, and can also schedule other units (such as the above-mentioned input unit 130 and the display unit 140) to implement corresponding functions.

[0076] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0077] Among them, in the embodiment of the present application, the RF unit 110 can send data to the server 200 and receive the processing result sent by the server 200. Exemplarily, taking the implementation process of the voice interaction method as an example, the RF unit 110 sends the multi-channel original voice signal collected by the microphone to the server 200, and the server 200 extracts the voice signal related to the target speaker from the multi-channel original voice signal, performs voice recognition, and returns the voice recognition result to the terminal 100, and the terminal 100 performs the corresponding interaction action based on the voice recognition result.

[0078] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0079] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0080] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0081] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0082] Next describe Figure 1 The product form of the server 200;

[0083] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0084] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0085] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0086] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0087] The memory 204 may be used to store software codes related to the target speaker extraction method or the voice interaction method, and the processor 202 may execute the steps of the target speaker extraction method or the voice interaction method, and may also schedule other units to implement corresponding functions.

[0088] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0089] Voice interaction can be applied to a variety of different scenarios, such as user-home robot interaction scenarios, smart teaching scenarios, in-vehicle voice interaction scenarios, etc.

[0090] Taking the in-vehicle voice interaction scenario as an example, in order to provide a fully upgraded interactive experience, the in-vehicle intelligent voice interaction system first identifies the wake-up words of passengers in various positions, locates the sound source of the target speaker, and identifies its command words to operate the window opening and closing, air conditioning, navigation and other functions in the car. Usually when the vehicle is driving, due to the presence of large noises such as tire noise and wind noise, the interactive voice signal-to-noise ratio is relatively low, and the success rate is greatly reduced. At the same time, in order to achieve an upgrade in the interactive experience of the entire vehicle, each position is allowed to interact freely without being affected by other positions, so that effective interaction can be carried out when other speakers are listening to music, watching movies or talking. This requires the front-end signal processing module to effectively suppress noise and interference for interactive signals with low signal-to-noise ratio and signal-to-interference ratio.

[0091] Existing front-end noise reduction solutions usually use deep learning solutions for noise reduction and interference suppression, perform speech separation on the received multi-channel original signals, and perform corresponding speech interaction after each area obtains the estimated clean signal of the area. When in scenarios with extremely low signal-to-noise ratio and signal-to-interference ratio such as high-speed windowing and multi-point large interference, the effect of the speech separation network will be greatly reduced, and the separated speech will have serious distortion, thereby reducing the success rate of interaction.

[0092] To this end, the present application provides a target speaker extraction method, which adds reference information when extracting speech signals related to the target speaker, that is, the attribute information of the target speaker estimated in the wake-up phase is used as reference information and applied in the recognition phase. It can effectively reduce the signal distortion of the existing scheme and only extract speech signals related to the target speaker, thereby improving the quality of the extracted speech signal and thus helping to improve the success rate of voice interaction.

[0093] The present invention provides a method for extracting a target speaker. Figure 1 Taking the computer device in the example as an example, the computer device can be Figure 1 The terminal 100 or the system consisting of the terminal 100 and the server 200. Figure 4 , the target speaker extraction method specifically includes the following steps:

[0094] Step S100: performing speech separation on the received multi-channel original speech signal to obtain a separated speech signal for each zone.

[0095] The multi-channel original speech signal is obtained by collecting speech signals from a target space divided into several partitions through more than two microphones.

[0096] The target space is the space where the microphone is located. Different voice interaction scenarios may correspond to different target spaces. Taking the in-vehicle human-computer interaction scenario as an example, the target space may be the interior space of the vehicle. Generally, the interior space of the vehicle can be divided into four or six areas according to the distribution of seats. One or more microphones can be set in different partitions, and then the voice signal of the target space can be collected through the microphone array to obtain a multi-channel original voice signal.

[0097] In this step, the configured separation model can be used to perform speech separation on the multi-channel original speech signal to obtain the separated speech signal of each area. The separation model can adopt the existing or other improved network structure.

[0098] When using the separation model for speech separation, specifically, the input multi-channel original speech signal can be subjected to short-time Fourier transform STFT, the real part and the imaginary part of the frequency domain signal are spliced ​​together to obtain the time domain feature, and sent to the separation model to obtain the time-frequency mask mask of the separated speech signal in each area output by the separation model.

[0099] The time-frequency mask of each area is multiplied by the time-frequency feature respectively, and the separated speech signal of each area is obtained by inverse short-time Fourier transform (ISTFT).

[0100] Reference Figure 5As shown, this embodiment provides an optional network structure of a separation model.

[0101] The separation model may include a two-dimensional convolutional module, an encoder, a recurrent neural network GRU, and a decoder connected in series.

[0102] The input multi-channel original speech signal can be subjected to short-time Fourier transform STFT to concatenate the real and imaginary parts of the frequency domain signal to obtain the time domain features, and then sent to the two-dimensional convolution module to extract its spatial and spectral features. The features extracted by the two-dimensional convolution module are sent to the encoder. The encoder can use multiple cascaded two-dimensional convolutions to extract high-dimensional features by reducing the frequency domain dimension and increasing the channel dimension. The spatial and spectral features extracted by the encoder are sent to the GRU module of time series modeling to extract long-term information and further utilize the spatial information with low time variability. Finally, the decoder is used for decoding and output. The decoder can use a deconvolution module to obtain the mask of the separated speech signal in each area.

[0103] On this basis, the mask of each area output by the decoder can be multiplied by the time-frequency characteristics of the original input speech signal, and the separated speech signal of each area can be obtained after inverse Fourier transform (ISTFT).

[0104] Step S110: Send the separated speech signal of each zone to the wake-up model to obtain the wake-up endpoint information in the separated speech signal of the target zone output by the wake-up model.

[0105] Among them, the wake-up model can be pre-trained according to the wake-up task, and it can judge whether the wake-up conditions are met according to the input voice signal, that is, whether the wake-up can be triggered, and when the wake-up conditions are met, it can further determine the wake-up endpoint information in the input voice signal. The wake-up endpoint information may include the time coordinates of the two wake-up endpoints.

[0106] The voice segment between the two wake-up endpoints in the input voice signal can trigger wake-up, that is, it contains the set wake-up word.

[0107] Considering that in general scenarios, only one target speaker is interacting with the machine by voice. Therefore, among the separated voice signals of each partition obtained in the previous step, generally only one separated voice signal of the target zone can trigger wake-up. In this step, the separated voice signals of each zone are sent to the wake-up model for wake-up judgment. Usually, only one separated voice signal of the target zone can trigger wake-up, that is, the wake-up endpoint information in the separated voice signal of the target zone can be obtained through the wake-up model.

[0108] The separated speech signal in the target area can be understood as a speech signal containing the interaction between the target speaker and the machine.

[0109] Step S120: extracting attribute information of the target speaker from the speech signals located between the wake-up endpoints in the separated speech signals of the target area.

[0110] Specifically, in the previous step, by sending the separated speech signals of each area output by the separation model to the wake-up model, the separated speech signals of the target area that can meet the wake-up conditions and the wake-up endpoint information therein can be identified through the wake-up model. Then, the speech signals located between the wake-up endpoints in the separated speech signals of the target area can be extracted.

[0111] In order to better guide the subsequent target speaker extraction model to extract speech signals related to the target speaker, in this step, the attribute information of the target speaker can be extracted from the speech signals between the wake-up endpoints and sent to the subsequent target speaker extraction model as reference information.

[0112] Here, the attribute information of the target speaker can be understood as information that can characterize the characteristics of the target speaker, including but not limited to: direction information, voiceprint information, prosody information, etc. of the target speaker.

[0113] Step S130: obtaining a speech signal related to the target speaker by using the configured target speaker extraction model and referring to the attribute information of the target speaker and the speech signal of the wake-up segment.

[0114] The speech signal of the wake-up segment is calculated based on the speech signal located between the wake-up endpoints in the separated speech signal of the target area.

[0115] In a possible implementation, the speech signal located between the wake-up endpoints in the separated speech signal of the target area can be directly used as the speech signal of the wake-up segment.

[0116] In this step, in order to improve the performance of the target speaker extraction model and improve the quality of the extracted speech signal related to the target speaker (reduce distortion), the attribute information of the target speaker is sent to the target speaker extraction model as reference information to assist in extracting a single speech signal related to the target speaker from the speech signal of the wake-up segment.

[0117] The target speaker extraction model is configured to process the input target speaker attribute information and the speech signal of the wake-up segment, and output the speech signal related to the target speaker. It can be pre-trained using training data according to the target speaker extraction task.

[0118] The target speaker extraction method provided in the embodiment of the present application, on the basis of voice separation of multi-channel original voice signals, further uses the wake-up model to perform wake-up recognition on the separated voice signals of each zone, and obtains the wake-up endpoint information in the separated voice signals of the target zone, that is, identifies the voice signals between the wake-up endpoints that meet the wake-up conditions. Since this segment of voice signals can meet the wake-up conditions, effective attribute information of the target speaker can be extracted from it, such as the voiceprint information and spatial position information of the target speaker. The attribute information of the target speaker is used as reference information, and the voice signal related to the target speaker is estimated based on the reference information and the voice signal of the wake-up segment through the configured target speaker extraction model. The present application scheme uses the attribute information of the target speaker estimated in the wake-up stage as reference information and applies it in the recognition stage, which can effectively reduce the distortion of the existing scheme, can more accurately extract the voice signal related to the target speaker, improve the signal quality, and thus help to improve the success rate of subsequent voice interaction.

[0119] Reference Figure 6 , which illustrates a target speaker extraction system structure.

[0120] For the input multi-channel original speech signal, the time-frequency features can be obtained through short-time Fourier transform (STFT), and sent to the separation model to obtain the separated speech signal of each area. And sent to the wake-up model to identify the separated speech signal of the target area that meets the wake-up conditions, and the wake-up endpoint information therein. The speech signal between the wake-up endpoints is extracted from the separated speech signal of the target area and sent to the attribute extraction module.

[0121] In a possible implementation, the input multi-channel original speech signal can also be sent to the attribute extraction module to assist in extracting the attribute information of the target speaker.

[0122] The attribute extraction module can extract the attribute information of the target speaker and send it to the target speaker extraction model.

[0123] In addition, the speech signal of the wake-up segment can be obtained based on the speech signal located between the wake-up endpoints in the separated speech signal of the target area, and the speech signal of the wake-up segment is also sent to the target speaker extraction model.

[0124] The target speaker extraction model extracts the speech signal related to the target speaker from the speech signal of the wake-up segment with the help of the attribute information of the target speaker.

[0125] The extracted speech signal related to the target speaker can be sent to a subsequent speech recognition model for recognition processing.

[0126] In some embodiments of the present application, the target speaker extraction method of the present application may further include:

[0127] The separated speech signal of each zone is used to guide the beamforming of each zone to obtain the beam signal of each zone.

[0128] Combination Figure 7 As shown, the beamforming module can perform beamforming on the input multi-channel original speech signal under the guidance of the separated speech signal of each zone obtained by the separation model to obtain the beam signal of each zone.

[0129] On this basis, the beam signal and the separated speech signal of each zone can be sent to the wake-up model together to obtain the wake-up endpoint information in the separated speech signal of the target zone output by the wake-up model.

[0130] The beam signal is extracted from the target direction signal obtained by traditional signal processing. It is based on the assumption that the target direction is distortion-free. The target direction signal can be distortion-free in an ideal situation. And because the output of the filter is a linear operation with less distortion, while the output of the separation model is a nonlinear mask, there is more or less distortion for the speech signal. The beam signal complements the speech signal output by the separation model.

[0131] In addition, the beam signal has limited suppression of noise and other interference, especially for non-directional scattered noise, the beam signal suppression is insufficient, but the separation model can completely suppress it. Therefore, the beam signal and the separated speech signal output by the separation model are simultaneously input into the wake-up model to obtain a more robust wake-up result.

[0132] In this embodiment, the process of guiding the beamforming of each zone by the separated speech signal of each zone to obtain the beam signal of each zone may specifically include:

[0133] Taking the time-frequency mask of the separated speech signal of each area as a reference, beamforming is performed on the time-frequency features obtained by STFT transformation of the multi-channel original speech signal to obtain the beam signal of each area.

[0134] That is, the beamforming module can use the time-frequency mask of each area output by the separation model as a reference, perform beamforming on the time-frequency features obtained by STFT transformation of the input multi-channel original speech signal, and obtain the beam signal of each area.

[0135] Among them, the beamforming module can use the minimum variance distortion-free response beam algorithm (

[0136] Minimum Variance Distortionless Response, MVDR), Generalized Sidelobe Canceller, GSC and other beamforming algorithms. Taking the MVDR beamforming algorithm as an example, the calculation process of the algorithm is introduced:

[0137] The calculation formula is as follows:

[0138]

[0139] Where h is the coefficient of the MVDR filter and Y(t,f) is the frequency domain transform of the microphone signal.

[0140] MVDR is a filter designed based on the principle of no distortion in the target direction and minimum output power.

[0141]

[0142] where φ nn is the noise power spectral density, v(f) is the guidance vector in the target direction, and its formula is as follows:

[0143]

[0144] Where v(f) is obtained by eigenvalue decomposition of the power spectral density of clean speech:

[0145] v(f)=P{φ ss (f)}

[0146] The power spectral density of clean speech is φ ss (f) is estimated as follows:

[0147]

[0148] Where M(t, f) is the ideal time-frequency mask IRM (ideal ratio mask) of the target speech, which can be estimated by the model:

[0149]

[0150] Furthermore, based on the addition of the beamforming processing process in the above embodiment, the process of obtaining the speech signal related to the target speaker by using the configured target speaker extraction model and referring to the attribute information of the target speaker and the speech signal of the wake-up segment in step S130 in the above embodiment may include:

[0151] Based on the wake-up endpoint information in the separated speech signal of the target area, the beam signal located between the wake-up endpoints is extracted from the beam signal of the target area as the speech signal of the wake-up segment.

[0152] The attribute information of the target speaker and the speech signal of the wake-up segment are sent to the target speaker extraction model to obtain the speech signal related to the target speaker output by the model.

[0153] Specifically, in this embodiment, the beam signal located between the wake-up endpoints in the beam signal of the target area can be used as the speech signal of the wake-up segment and sent to the target speaker extraction model. Only the target speech in the direction of the target speaker is picked up, and the noise and speakers in other directions are suppressed. This can improve the quality of the extracted speech signal related to the target speaker.

[0154] In some embodiments of the present application, the aforementioned step S120 is described, in which the process of extracting the attribute information of the target speaker from the speech signal located between the wake-up endpoints in the separated speech signal of the target area is described.

[0155] In a possible implementation, the extracted attribute information of the target speaker may include the direction information of the target speaker, that is, the direction information of the target speaker in the target space, and the direction information may be three-dimensional coordinate information. Taking the vehicle-mounted scene as an example, the direction information may be the target position information under vehicle-mounted conditions.

[0156] In this embodiment, the process of extracting the direction information of the target speaker can be implemented by configuring the positioning model. Specifically, it may include:

[0157] The speech signals located between the wake-up endpoints in the separated speech signals of the target area are sent to the positioning model to obtain the direction information of the target speaker output by the model.

[0158] The localization model is configured to process the input speech signal to obtain the direction information of the target speaker.

[0159] Reference Figure 8 As shown, it provides an optional composition structure of the positioning model.

[0160] The positioning model may include a two-dimensional convolutional module, an encoder, and a recurrent neural network GRU connected in series.

[0161] The speech signal located between the wake-up endpoints in the separated speech signal of the target area is taken as input, and is processed by the two-dimensional convolution module, the encoder, and the recurrent neural network GRU in sequence, and finally the direction information of the target speaker is output, and the direction information can specifically be position information, such as position coordinates.

[0162] In some possible implementations, in order to improve the robustness of the target speaker's direction information output by the localization model, a multi-channel original speech signal may be further added to the input of the localization model. That is, the speech signal between the wake-up endpoints in the separated speech signal of the target area and the multi-channel original speech signal are sent to the localization model together to obtain the target speaker's direction information output by the model.

[0163] In the above embodiment, it is introduced that the direction information of the target speaker can be used as attribute information. In addition, the voiceprint information of the target speaker can also be used as attribute information.

[0164] This embodiment introduces the process of extracting voiceprint information, specifically:

[0165] Voiceprint extraction is performed on the speech signals located between the wake-up endpoints in the separated speech signals of the target area to obtain the voiceprint information of the target speaker.

[0166] The voiceprint extraction can be realized by a pre-configured voiceprint extraction module. The voiceprint extraction module is established on the basis of training a speaker recognition model, and uses the intermediate hidden layer features extracted by the speaker recognition model as the speaker's voiceprint information.

[0167] Reference Fig. 9 , which provides an optional structure of a voiceprint extraction module based on a speaker recognition model.

[0168] The speaker recognition model can include a two-dimensional convolutional module, an encoder, and a recurrent neural network GRU.

[0169] For the speech signals between the wake-up endpoints in the separated speech signals of the input target area, feature processing is performed through various modules, and speaker recognition is performed based on the output of the final GRU. In addition, the hidden layer features extracted from any hidden layer in the middle can be used as the voiceprint information of the target speaker.

[0170] Reference Fig.10 , Fig.10 An example of a target speaker extraction system structure diagram is shown.

[0171] For the input multi-channel original speech signal, the time-frequency feature can be obtained through short-time Fourier transform STFT, and then sent to the separation model to obtain the separated speech signal of each area.

[0172] The time-frequency mask of the separated speech signal of each area obtained by the separation model is used as a guidance signal to guide the beamforming of the beamforming module to obtain the beam signal of each partition.

[0173] The beam signal and separated speech signal of each zone are sent to the wake-up model to obtain the wake-up endpoint information in the separated speech signal of the target zone output by the wake-up model.

[0174] Furthermore, the speech signals between the wake-up endpoints in the separated speech signals of the target area and the multi-channel original speech signals can be sent to the positioning model together to obtain the direction information of the target speaker output by the positioning model.

[0175] The voiceprint extraction module is used to extract the voiceprint of the speech signal between the wake-up endpoints in the separated speech signal of the target area to obtain the voiceprint information of the target speaker.

[0176] The voiceprint information of the target speaker extracted by the voiceprint extraction module and the direction information obtained by the positioning model together constitute the attribute information of the target speaker and are sent to the target speaker extraction model.

[0177] In addition, based on the wake-up endpoint information in the separated speech signal of the target area, the beam signal located between the wake-up endpoints can be extracted from the beam signal of the target area as the speech signal of the wake-up segment and sent to the target speaker extraction model.

[0178] The target speaker extraction model is used to extract the speech signal related to the target speaker.

[0179] In this embodiment, the estimation of the space and voiceprint information in the wake-up phase is utilized to form complementary advantages. By utilizing them in the recognition phase, the distortion of the existing scheme can be effectively reduced, and only the speech signal related to the target speaker can be extracted.

[0180] In some embodiments of the present application, a voice interaction method is further provided, which specifically includes the following steps:

[0181] S1. Obtain multi-channel original speech signals.

[0182] Specifically, in a voice interaction scenario in a target space, a multi-channel original voice signal collected by multiple microphones arranged in the target space is obtained.

[0183] S2. Extracting speech signals related to the target speaker from the multi-channel original speech signals.

[0184] The process of extracting the speech signal related to the target speaker in this step can be implemented by using the target speaker extraction method introduced in any of the above embodiments, which can extract high-quality speech signals related to the target speaker and reduce signal distortion.

[0185] S3. Send the speech signal related to the target speaker to the recognition model to obtain the speech recognition result.

[0186] S4. Execute corresponding interactive actions based on the speech recognition results.

[0187] Specifically, there are many interactive actions that can be performed according to different voice interaction scenarios. Taking the in-vehicle voice interaction scenario as an example, the interactive actions that can be performed include but are not limited to: opening and closing doors and windows, turning on and off the air conditioner, navigation, playing music, etc.

[0188] The target speaker extraction device provided in the embodiment of the present application is described below. The target speaker extraction device described below and the target speaker extraction method described above can be referenced to each other.

[0189] See also Fig.11 , Fig.11 A schematic diagram of the structure of a target speaker extraction device disclosed in an embodiment of the present application.

[0190] like Fig.11 As shown, the device may include:

[0191] The speech separation unit 11 is used to perform speech separation on the received multi-channel original speech signal to obtain a separated speech signal for each zone, wherein the multi-channel original speech signal is obtained by collecting speech signals from a target space divided into a plurality of zones through two or more microphones;

[0192] A wake-up unit 12, used to send the separated speech signal of each zone to the wake-up model to obtain the wake-up endpoint information in the separated speech signal of the target zone output by the wake-up model;

[0193] An attribute extraction unit 13 is used to extract attribute information of a target speaker from a speech signal located between wake-up endpoints in the separated speech signal of the target area;

[0194] The extraction unit 14 is used to obtain a speech signal related to the target speaker by referring to the attribute information of the target speaker and the speech signal of the wake-up segment through the configured target speaker extraction model, and the speech signal of the wake-up segment is calculated based on the speech signal located between the wake-up endpoints in the separated speech signal of the target area.

[0195] In a possible implementation, the target speaker extraction device of the present application further includes:

[0196] The beamforming unit is used to guide the beamforming of each zone with the separated speech signal of each zone to obtain the beam signal of each zone. On this basis, the awakening unit sends the separated speech signal of each zone to the awakening model to obtain the awakening endpoint information in the separated speech signal of the target zone output by the awakening model, including:

[0197] The beam signal and separated speech signal of each zone are sent to the wake-up model to obtain the wake-up endpoint information in the separated speech signal of the target zone output by the wake-up model.

[0198] Further optionally, the extraction unit obtains the speech signal related to the target speaker by using the configured target speaker extraction model and referring to the attribute information of the target speaker and the speech signal of the wake-up segment, including:

[0199] Based on the wake-up endpoint information in the separated voice signal of the target area, extracting the beam signal located between the wake-up endpoints from the beam signal of the target area as the voice signal of the wake-up segment;

[0200] The attribute information of the target speaker and the speech signal of the wake-up segment are sent to a target speaker extraction model to obtain a speech signal related to the target speaker output by the model.

[0201] In a possible implementation, the speech separation unit performs speech separation on the received multi-channel original speech signal to obtain a separated speech signal for each zone, including:

[0202] Performing short-time Fourier transform (STFT) on the multi-channel original speech signal to obtain time-frequency features, and sending the features to the separation model to obtain a time-frequency mask of the separated speech signal of each area output by the separation model;

[0203] The time-frequency mask of the separated speech signal of each zone is multiplied by the time-frequency feature respectively, and the separated speech signal of each zone is obtained by inverse short-time Fourier transform (ISTFT). Furthermore, the beamforming unit guides the beamforming of each zone with the separated speech signal of each zone to obtain the beam signal of each zone, including:

[0204] Taking the time-frequency mask of the separated speech signal of each zone as a reference, beamforming is performed on the time-frequency features to obtain the beam signal of each zone.

[0205] In a possible implementation, the process of extracting the attribute information of the target speaker by the attribute extraction unit for the speech signal located between the wake-up endpoints in the separated speech signal of the target area includes:

[0206] The speech signal located between the wake-up endpoints in the separated speech signal of the target area is sent to the positioning model to obtain the direction information of the target speaker output by the model, and the direction information is used as the attribute information of the target speaker, wherein the positioning model is configured to process the input speech signal to obtain the direction information of the target speaker.

[0207] In a possible implementation, the attribute extraction unit sends the speech signal located between the wake-up endpoints in the separated speech signal of the target area to the positioning model to obtain the direction information of the target speaker output by the model, including:

[0208] The speech signals located between the wake-up endpoints in the separated speech signals of the target area and the multi-channel original speech signals are sent to the positioning model together to obtain the direction information of the target speaker output by the model, and the direction information is used as the attribute information of the target speaker.

[0209] In a possible implementation, the process of extracting the attribute information of the target speaker by the attribute extraction unit for the speech signal located between the wake-up endpoints in the separated speech signal of the target area includes:

[0210] Voiceprint extraction is performed on the voice signals located between the wake-up endpoints in the separated voice signals of the target area to obtain the voiceprint information of the target speaker, and the voiceprint information is used as the attribute information of the target speaker.

[0211] See also Fig.12 , Fig.12 A schematic diagram of the structure of a voice interaction device disclosed in an embodiment of the present application.

[0212] like Fig.12 As shown, the device may include:

[0213] A signal acquisition unit 21, used to acquire a multi-channel original speech signal;

[0214] A speaker extraction unit 22, configured to extract a speech signal related to a target speaker from the multi-channel original speech signal by using the target speaker extraction method of any of the above embodiments;

[0215] A speech recognition unit 23, used to send the speech signal related to the target speaker into a recognition model to obtain a speech recognition result;

[0216] The interaction unit 24 is used to perform corresponding interaction actions based on the speech recognition result.

[0217] The present application also provides an electronic device in an embodiment. Fig.13 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, tablet computers, vehicle-mounted terminals, teaching large screens, wearable devices, etc. Fig.13 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0218] like Fig.13As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603, so as to implement the target speaker extraction method or the voice interaction method of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0219] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig.13 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0220] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any target speaker extraction method or voice interaction method provided in the embodiment of the present application.

[0221] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any target speaker extraction method or voice interaction method provided in the embodiment of the present application.

[0222] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0223] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0224] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0225] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0226] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A target speaker extraction method, characterized in that: include: Performing speech separation on the received multi-channel original speech signal to obtain a separated speech signal for each zone, wherein the multi-channel original speech signal is obtained by collecting speech signals from a target space divided into a plurality of partitions through two or more microphones; The separated speech signal of each area is sent to the wake-up model to obtain the wake-up endpoint information in the separated speech signal of the target area output by the wake-up model; Extracting attribute information of a target speaker from a speech signal located between wake-up endpoints in the separated speech signal of the target area; The target speaker extraction model is configured to refer to the attribute information of the target speaker and the speech signal of the wake-up segment to obtain the speech signal related to the target speaker. The speech signal of the wake-up segment is calculated based on the speech signal located between the wake-up endpoints in the separated speech signal of the target area.

2. The method according to claim 1, characterized in that Also includes: The separated speech signal of each zone is used to guide the beam forming of each zone to obtain the beam signal of each zone; The process of sending the separated speech signal of each area to the wake-up model to obtain the wake-up endpoint information in the separated speech signal of the target area output by the wake-up model includes: The beam signal and separated speech signal of each zone are sent to the wake-up model to obtain the wake-up endpoint information in the separated speech signal of the target zone output by the wake-up model.

3. The method according to claim 2, characterized in that The process of obtaining the speech signal related to the target speaker by referring to the attribute information of the target speaker and the speech signal of the wake-up segment through the configured target speaker extraction model includes: Based on the wake-up endpoint information in the separated voice signal of the target area, extracting the beam signal located between the wake-up endpoints from the beam signal of the target area as the voice signal of the wake-up segment; The attribute information of the target speaker and the speech signal of the wake-up segment are sent to a target speaker extraction model to obtain a speech signal related to the target speaker output by the model.

4. The method according to claim 1, characterized in that: The process of extracting the attribute information of the target speaker from the speech signal located between the wake-up endpoints in the separated speech signal of the target area includes: The speech signal located between the wake-up endpoints in the separated speech signal of the target area is sent to the positioning model to obtain the direction information of the target speaker output by the model, and the direction information is used as the attribute information of the target speaker, wherein the positioning model is configured to process the input speech signal to obtain the direction information of the target speaker.

5. The method according to claim 4, characterized in that The process of sending the speech signals located between the wake-up endpoints in the separated speech signals of the target area to the positioning model to obtain the direction information of the target speaker output by the model includes: The speech signals located between the wake-up endpoints in the separated speech signals of the target area and the multi-channel original speech signals are sent to the positioning model together to obtain the direction information of the target speaker output by the model, and the direction information is used as the attribute information of the target speaker.

6. The method according to claim 1 or 4, characterized in that: The process of extracting the attribute information of the target speaker from the speech signal located between the wake-up endpoints in the separated speech signal of the target area includes: Voiceprint extraction is performed on the voice signals located between the wake-up endpoints in the separated voice signals of the target area to obtain the voiceprint information of the target speaker, and the voiceprint information is used as the attribute information of the target speaker.

7. The method according to claim 2, characterized in that The process of performing speech separation on the received multi-channel original speech signal to obtain the separated speech signal of each zone includes: Performing short-time Fourier transform (STFT) on the multi-channel original speech signal to obtain time-frequency features, and sending the features to the separation model to obtain a time-frequency mask of the separated speech signal of each area output by the separation model; The time-frequency mask of the separated speech signal of each zone is multiplied by the time-frequency feature respectively, and the separated speech signal of each zone is obtained by inverse short-time Fourier transform (ISTFT); The process of guiding the beamforming of each zone by the separated speech signal of each zone to obtain the beam signal of each zone includes: Taking the time-frequency mask of the separated speech signal of each zone as a reference, beamforming is performed on the time-frequency features to obtain the beam signal of each zone.

8. A voice interaction method, characterized in that: include: Obtain multi-channel original speech signals; Adopting the target speaker extraction method according to any one of claims 1 to 7, extracting a speech signal related to the target speaker from the multi-channel original speech signal; Sending the speech signal related to the target speaker into the recognition model to obtain a speech recognition result; Execute corresponding interactive actions based on the speech recognition result.

9. A target speaker extraction device, characterized in that: include: A speech separation unit is used to perform speech separation on the received multi-channel original speech signal to obtain a separated speech signal for each zone, wherein the multi-channel original speech signal is obtained by collecting speech signals from a target space divided into a plurality of partitions through two or more microphones; A wake-up unit, used to send the separated speech signal of each zone to the wake-up model, and obtain the wake-up endpoint information in the separated speech signal of the target zone output by the wake-up model; An attribute extraction unit, configured to extract attribute information of a target speaker from a speech signal located between wake-up endpoints in the separated speech signal of the target area; The extraction unit is used to input the attribute information of the target speaker into the target speaker extraction model to obtain the speech signal related to the target speaker output by the model.

10. A voice interaction device, characterized in that: include: A signal acquisition unit, used for acquiring multi-channel original speech signals; A speaker extraction unit, configured to extract a speech signal related to a target speaker from the multi-channel original speech signal by using the target speaker extraction method according to any one of claims 1 to 7; A speech recognition unit, used to send the speech signal related to the target speaker into a recognition model to obtain a speech recognition result; An interaction unit is used to perform corresponding interaction actions based on the speech recognition result.

11. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the target speaker extraction method according to any one of claims 1 to 7, or to implement the various steps of the voice interaction method according to claim 8.

12. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the target speaker extraction method according to any one of claims 1 to 7 is implemented, or each step of the speech interaction method according to claim 8 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the target speaker extraction method according to any one of claims 1 to 7 are implemented, or the steps of the speech interaction method according to claim 8 are implemented.

Images