A sound source positioning method and related device

Abstract

The application provides a sound source positioning method and related equipment, the method comprising: determining a target voice channel according to mixed voice collected by a voice collection module, the target voice channel being a voice transmission channel with a signal-to-noise ratio greater than a signal-to-noise ratio threshold; obtaining processed voice collected by a target microphone array, the target microphone array being used to indicate a microphone array corresponding to the target voice channel in the voice collection module; and positioning a user issuing a voice instruction according to the processed voice. In the application, the accuracy of sound source positioning is improved.

Description

Technical Field

[0001] This application relates to the field of sound source localization technology, and in particular to a sound source localization method and related equipment. Background Technology

[0002] With the acceleration of intelligentization, voice wake-up control has become a basic function of robots, in-vehicle systems, smart homes, and various portable terminals.

[0003] In this exemplary technology, multiple microphone arrays form a distributed voice acquisition module. The voice acquired by this module is used to locate the sound source of the user uttering the voice. This method estimates the time delay difference of the signal arriving at the microphone by utilizing the phase difference between the signals received by different microphones in the microphone array, thereby estimating the location of the sound source.

[0004] However, the conversion from phase difference to delay difference needs to follow the "spatial sampling law", that is, the distance between two microphones in the microphone array needs to be less than or equal to half of the minimum wavelength of the signal. Otherwise, multiple "grating lobes" will appear, that is, several grating lobes will appear on the localization spectrum, and it will be impossible to determine which grating lobe is located in the direction of the peak of the sound source.

[0005] In a distributed voice acquisition module, although the distance between microphone arrays is relatively far, all signals collected by the microphones are required for sound source localization. This method can also produce grating lobes, leading to inaccurate sound source localization, which means there is a problem of low sound source localization accuracy. Summary of the Invention

[0006] Based on the above-mentioned technological status, this application provides a sound source localization method and related equipment to solve the problem of low accuracy in sound source localization.

[0007] To achieve the above-mentioned technical objectives, this application proposes the following technical solution: Firstly, this application provides a sound source localization method, including: The target speech channel is determined based on the mixed speech acquired by the speech acquisition module. The target speech channel is a speech transmission channel with a signal-to-noise ratio greater than the signal-to-noise ratio threshold. Acquire the voice to be processed from the target microphone array, wherein the target microphone array is used to indicate the microphone array corresponding to the target voice channel in the voice acquisition module; Based on the voice to be processed, the user who issued the voice command is located.

[0008] In some implementations, locating the user issuing the voice command based on the voice to be processed includes: Determine the time-frequency masking code based on the mixed speech; Based on the speech to be processed and the time-frequency masking code, a first covariance matrix of the speech to be processed is determined, and the output power of the target microphone array in each scanning direction is determined based on the first covariance matrix. The location of the user issuing the voice command is determined based on the scanning direction corresponding to the maximum output power.

[0009] In some implementations, determining the output power of the target microphone array in each scanning direction based on the first covariance matrix includes: Based on the time-frequency masking code, determine the second covariance matrix of the non-noise in the speech to be processed and the third covariance matrix of the noise in the speech to be processed; The beamforming vector is determined based on the second covariance matrix and the third covariance matrix; The output power of the target microphone array in each scanning direction is determined based on the beamforming vector, the first covariance matrix, and the steering vector in each scanning direction.

[0010] In some implementations, determining the first covariance matrix of the speech to be processed based on the speech to be processed and the time-frequency masking code includes: The location requirement of the event to be located is determined based on the voice to be processed; Based on the positioning requirements, the speech to be processed, and the time-frequency masking code, the first covariance matrix of the speech to be processed is determined.

[0011] In some implementations, determining the first covariance matrix of the speech to be processed based on the positioning requirements, the speech to be processed, and the time-frequency masking code includes: When the positioning requirement indicates that positioning should be performed after the event to be located has ended, the first covariance matrix of the speech to be processed is determined based on the speech to be processed and the time-frequency masking code.

[0012] In some implementations, determining the first covariance matrix of the speech to be processed based on the positioning requirements, the speech to be processed, and the time-frequency masking code includes: When the positioning requirement indicates that the event to be located needs to be located in real time, the fourth covariance matrix of the speech to be processed in the previous frame is obtained. The first covariance matrix of the speech to be processed is determined based on the fourth covariance matrix, the speech to be processed, and the time-frequency masking code.

[0013] In some implementations, determining the first covariance matrix of the speech to be processed based on the speech to be processed and the time-frequency masking code includes: Based on the speech to be processed and the time-frequency masking code, determine the initial covariance matrix of the speech to be processed; A phase transformation is performed on each element of the initial covariance matrix to obtain the first covariance matrix.

[0014] Secondly, this application provides an electronic device, including a memory and a processor, wherein, The memory is connected to the processor and is used to store programs; The processor is used to implement the sound source localization method as described in the first aspect or any implementation thereof by running a program in the memory.

[0015] Thirdly, this application provides a computer program product, including computer instructions, which, when executed by a processor, implement the sound source localization method as described in the first aspect or any implementation thereof.

[0016] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the sound source localization method as described in the first aspect or any implementation thereof.

[0017] This application provides a sound source localization method and related equipment. The method determines a target speech channel by collecting mixed speech from a speech acquisition module. Within each microphone array of the speech acquisition module, the microphone array corresponding to the target speech channel is selected as the target microphone array. The speech to be processed is then acquired from the target microphone array, thereby determining the location of the user issuing the voice command based on the processed speech. In this application, the target speech channel is a speech transmission channel with a signal-to-noise ratio (SNR) greater than a threshold. That is, among the distributed microphone arrays, the speech acquired by the microphone array corresponding to the speech transmission channel with better signal quality is selected for sound source localization. This eliminates the need for all microphone arrays to participate in sound source localization, avoiding the occurrence of grating lobes due to excessive distance between microphone arrays, thus improving the accuracy of sound source localization. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of a grating lobe involved in an exemplary technology; Figure 2 A flowchart of a sound source localization method provided in this application embodiment Figure 1 .

[0020] Figure 3 This is a schematic diagram illustrating the deployment of a distributed microphone array according to an embodiment of this application.

[0021] Figure 4 This is a schematic diagram illustrating the working process of the recognition model involved in this application.

[0022] Figure 5 A flowchart of a sound source localization method provided in this application embodiment Figure 2 .

[0023] Figure 6 A flowchart of a sound source localization method provided in this application embodiment Figure 3 .

[0024] Figure 7 A flowchart of a sound source localization method provided in this application embodiment Figure 4 .

[0025] Figure 8 This is a simplified schematic diagram of the sound source localization method involved in the embodiments of this application.

[0026] Figure 9 This is a schematic diagram of the functional modules of the sound source localization device involved in the embodiments of this application.

[0027] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0030] With the acceleration of intelligentization, voice wake-up control has become a basic function of robots, in-vehicle systems, smart homes, and various portable terminals.

[0031] In this exemplary technology, multiple microphone arrays form a distributed voice acquisition module. The voice acquired by this module is used to locate the sound source of the user uttering the voice. This method estimates the time delay difference of the signal arriving at the microphone by utilizing the phase difference between the signals received by different microphones in the microphone array, thereby estimating the location of the sound source.

[0032] However, the conversion from phase difference to delay difference must follow the "spatial sampling law," which states that the distance between two microphones in a microphone array must be less than or equal to half the minimum wavelength of the signal; otherwise, multiple "grating lobes" will appear. Figure 1 Several grating lobes will appear on the localization spectrum, making it impossible to determine which grating lobe's peak the sound source is located in.

[0033] In a distributed voice acquisition module, although the distance between microphone arrays is relatively far, all signals collected by the microphones are required for sound source localization. This method can also produce grating lobes, leading to inaccurate sound source localization, which means there is a problem of low sound source localization accuracy.

[0034] To address the above problems, this application proposes a sound source localization method. The sound source localization method proposed in this application will be described in detail below with reference to various embodiments.

[0035] Reference Figure 2 , Figure 2 A flowchart of a sound source localization method provided in this application embodiment Figure 1 Sound source localization methods include: Step S201: Determine the target speech channel based on the mixed speech acquired by the speech acquisition module. The target speech channel is a speech transmission channel with a signal-to-noise ratio greater than the signal-to-noise ratio threshold.

[0036] In this embodiment, the executing entity is a sound source localization device. For ease of description, the device is referred to as the sound source localization device below. The device can be a terminal device with sound source localization function or a component in a system with sound source localization function.

[0037] The voice acquisition module includes multiple microphone arrays, each positioned at a different location, and each microphone array consists of multiple microphones. For example, refer to... Figure 3 The vehicle is equipped with three microphone arrays on its exterior: one on the left and one on the right, and one at the rear of the vehicle. Each microphone array consists of two microphones.

[0038] The device acquires mixed speech collected by the voice acquisition module. This mixed speech includes the voice of the user issuing the command, the voice of the user emitting interference (voices without the command), and environmental noise (such as sounds from televisions, speakers, and other home appliances). The device needs to extract the target speech from the mixed speech; the target speech is the voice of the user issuing the command. For example, this can be achieved using a feedforward time-frequency masking code. Extracting the target speech from the mixed speech, where, Frequency band number, The frame number is used here. The time-frequency masking code can be obtained from the previous moment and can be defined as a historical time-frequency masking code.

[0039] The device determines the covariance matrix of the target speech using formula (1) and the covariance matrix of the interference noise component using formula (2). Formulas (1) and (2) are as follows: ,(1); in, For the first in mixed speech The covariance matrix of the target speech in the frame. Forgetting factor, For mixed speech, for The signal after conjugate transpose For the first in mixed speech ; (2); in, For the first in mixed speech The covariance matrix of the frame interference noise component, Forgetting factor, For the first in mixed speech The covariance matrix of the interference noise components of the frame.

[0040] In obtaining as well as Then, the beamforming vector is determined based on formula (3). The details are as follows: (3); among which, Let M be an M-dimensional vector, where M is the number of input channels of the microphone array used to acquire mixed speech.

[0041] After obtaining the beamforming vector This allows the target speech to be separated from the mixed speech. .

[0042] The voice acquisition module contains multiple microphone arrays, each capturing a mixed speech, thus allowing the extraction of the corresponding target speech from each mixed speech. Each target speech is based on... After the speech enhancement, the device inputs each target speech into the recognition model, and the recognition model can output the target speech channel. The target speech channel is a speech transmission channel with a signal-to-noise ratio greater than the signal-to-noise ratio threshold. The target speech channel is the speech channel with a higher signal-to-noise ratio among the speech channels corresponding to each microphone array. Preferably, the target speech channel can be the speech channel with the highest signal-to-noise ratio among these speech channels.

[0043] The backbone architecture of the recognition model is not required; any mainstream network structure can be used as its backbone, such as RNN (Recurrent Neural Network), CNN (Convolutional Neural Network), FSMN (Finite State Machine Network), Conformer (a combination of Convolution and Transformer), etc. (See reference...) Figure 4 , Figure 4 The process of determining the target speech channel for the recognition model: The backbone network of the recognition model first independently processes the information of each speech channel (i.e., the target speech input from each microphone array). Each backbone network processes one target speech, and the parameters of each backbone network are the same, but the data processed are different. Before reaching the final decision layer, the multi-channel information is first fused into a single channel through ReLU->maxpooling, and then the fused single-channel data is sent to the decision layer. The ReLU activation function first removes the negative values ​​in the input data, so the usable information is only contained in the remaining positive values, which will help enhance the sparsity of the data. Afterwards, due to the effect of maxpooling, only the feature with the largest value can be retained, so the network behavior will be trained to increase the feature encoding of useful information, while the feature values ​​corresponding to useless information such as noise and interference will decrease. Therefore, the optimal channel number can be selected by the method in formula (3). (3), of which For channel Features before being fed into ReLU. Furthermore, the recognition model can take these multi-channel signals as input and detect the probability of predefined events occurring from them. Depending on the task, The meaning of "can" can also vary; for example, it can refer to keyword modeling units, various sound events, target speech events, etc.

[0044] Step S202: Acquire the speech to be processed collected by the target microphone array. The target microphone array is used to indicate the microphone array corresponding to the target speech channel in the speech acquisition module.

[0045] After determining the target voice channel, the target microphone array is determined among the various microphone arrays in the voice acquisition module. That is, the microphone array corresponding to the target voice channel in the voice acquisition module is taken as the target microphone array, and the voice acquired by the target microphone array is acquired. This voice is defined as the voice to be processed.

[0046] Step S203: Based on the voice to be processed, locate the user who issued the voice command.

[0047] Once the voice to be processed is obtained, the user issuing the voice command can be located based on the voice. For example, based on the phase difference between the voice to be processed and the voice received by each microphone in the target microphone array, the delay difference between the voice to be processed and the microphone is estimated. Then, the location of the sound source is estimated based on the delay difference, and the location of the sound source is the user who issued the voice command.

[0048] In this embodiment, the target voice channel is determined by the mixed voice collected by the voice acquisition module. Among the various microphone arrays in the voice acquisition module, the microphone array corresponding to the target voice channel is selected as the target microphone array. The voice to be processed collected by the target microphone array is then acquired, thereby determining the location of the user issuing the voice command based on the voice to be processed. In this embodiment, the target voice channel is a voice transmission channel with a signal-to-noise ratio greater than the signal-to-noise ratio threshold. That is, among the distributed microphone arrays, the voice collected by the microphone array corresponding to the voice transmission channel with better signal quality is selected for sound source localization. This avoids the need for all microphone arrays to participate in sound source localization, preventing the formation of grating lobes due to excessive distance between microphone arrays, thus improving the accuracy of sound source localization.

[0049] Reference Figure 5 , Figure 5 A flowchart of a sound source localization method provided in this application embodiment Figure 2 .based on Figure 2 In the embodiment shown, step S203 includes: Step S501: Determine the time-frequency masking code based on the mixed speech.

[0050] In this embodiment, the recognition model can also be used to output a time-frequency masking code. That is, the recognition model outputs a time-frequency masking code while outputting the target speech channel. In other words, the target speech is separated based on each mixed speech, and the target speech is input into the recognition model to obtain the time-frequency masking code.

[0051] Step S502: Based on the speech to be processed and the time-frequency masking code, determine the first covariance matrix of the speech to be processed, and determine the output power of the target microphone array in each scanning direction based on the first covariance matrix.

[0052] After obtaining the time-frequency masking code, the covariance matrix of the speech to be processed is determined based on the speech to be processed and the time-frequency masking code. This covariance matrix is ​​defined as the first covariance matrix, which can be determined in the following way: ;in, For the frequency band range used for positioning, , This refers to the beginning and end of an event, such as... It is the starting point of the event. It is the point in time when the event ends; To identify the time-frequency masking code of the model's current output, it is defined as the current time-frequency masking code; The speech to be processed is the mixed speech collected by the target microphone array.

[0053] In this embodiment, the localization spectrum has multiple scanning directions of sound sources, meaning the scanning angle of the target microphone array is divided into multiple scanning directions. The device needs to calculate the output power in each scanning direction. Therefore, the device determines the output power of the target microphone array in each scanning direction based on the first covariance matrix.

[0054] In another example, if the error between the first covariance matrix and the covariance matrix calculated by the historical time-frequency masking code (referring to the covariance matrix calculated by formula (1), which can be used as the historical covariance matrix) is less than a preset threshold, then the historical power in the scanning direction is obtained as the output power in that scanning direction.

[0055] In one example, the device acquires the steering vector of the target microphone array in each scanning direction. The steering vector refers to the complex vector formed by the phase difference between the signals received by each microphone in the target microphone array when a plane wave is incident from the scanning direction. The output power of the target microphone array in each scanning direction can be determined by the device's first covariance matrix and the steering vector in each scanning direction. For example, the output power is: ;in, Let be the first covariance matrix. , As the guiding vector, The conjugate transpose parameter of the guiding vector.

[0056] Step S503: Determine the location of the user who issued the voice command based on the scanning direction corresponding to the maximum output power.

[0057] After obtaining the output power in each scanning direction, the location of the user who issued the voice command is determined according to the scanning direction corresponding to the maximum output power. In other words, the scanning direction corresponding to the maximum output power contains the user who issued the command.

[0058] In this embodiment, time-frequency masking codes are used to extract target components from the original noisy signal for sound source localization, which improves the localization accuracy under noise and human voice interference conditions; the deep neural network receives the enhanced speech signal, which is more conducive to event detection and mask estimation.

[0059] Reference Figure 6 , Figure 6 A flowchart of a sound source localization method provided in this application embodiment Figure 3 .based on Figure 5 In the embodiment shown, step S502 includes: Step S601: Determine the location requirements of the event to be located based on the voice to be processed.

[0060] In this embodiment, the device determines the location requirement of the event to be located by processing the speech. For example, target speech can be separated from the speech to be processed, and the location requirement of the event to be located is determined by the instructions contained in the target speech, such as the need to locate during the duration of the event or after the event ends.

[0061] Step S602: Determine the first covariance matrix of the speech to be processed based on the positioning requirements, the speech to be processed, and the time-frequency masking code.

[0062] After determining the positioning requirements, the first covariance matrix of the speech to be processed is determined based on the positioning requirements, the speech to be processed, and the time-frequency masking code.

[0063] In one example, if the location request indicates that location should be performed after the location event has ended, then the first covariance matrix of the speech to be processed is determined based on the speech to be processed and the time-frequency masking code, such as... .

[0064] In another example, if the location requirement indicates that the event to be located needs real-time location, then the fourth covariance matrix of the speech to be processed in the previous frame is obtained, and then the first covariance matrix of the speech to be processed is determined based on the fourth covariance matrix, the speech to be processed, and the time-frequency masking code. For example, the first covariance matrix is: , This is the fourth covariance matrix.

[0065] In this embodiment, the device accurately determines the first covariance matrix of the speech to be processed based on the positioning requirements of the event to be located.

[0066] Reference Figure 7 , Figure 7 A flowchart of a sound source localization method provided in this application embodiment Figure 4 .based on Figure 5 or Figure 6 In the embodiment shown, step S502 includes: Step S701: Determine the initial covariance matrix of the speech to be processed based on the speech to be processed and the time-frequency masking code.

[0067] In this embodiment, the device determines the initial covariance matrix of the speech to be processed based on the speech to be processed and the time-frequency masking code.

[0068] The initial covariance matrix is, for example, .

[0069] Step S702: Perform a phase transformation on each element in the initial covariance matrix to obtain the first covariance matrix.

[0070] After obtaining the initial covariance matrix, a phase transformation is performed on each element of the initial covariance matrix to obtain the first covariance matrix. For example, the phase transformation of the elements in the first covariance matrix can be characterized as follows: ;in, for The first in One element, It is a small positive number used to maintain computational stability.

[0071] In this embodiment, a first covariance matrix with good stability can be obtained by phase transformation of the elements in the initial covariance matrix.

[0072] Based on the above embodiments, and in combination with Figure 8 The sound source localization method of this application will be briefly described below.

[0073] The voice acquisition module includes a microphone array consisting of two microphones and a microphone array consisting of four microphones; both microphone arrays acquire the voice signal, which consists of the target sound source, interfering sound sources, ambient noise, and room reverberation. and voice signals ; Voice signal and voice signals After speech enhancement, the enhanced signal is obtained. and enhanced signal ; Enhanced signal and enhanced signal The input is fed into the task model (i.e., the recognition model mentioned above), and the task model outputs the time-frequency masking code g and the optimal speech channel. ,based on Perform subarray selection, i.e., determine The corresponding microphone array is used as the target microphone array, and the sound source is located based on the speech x to be processed collected by the target microphone array and the time-frequency masking code.

[0074] Corresponding to the sound source localization method described above, this application also provides a sound source localization device. Figure 9 This is a schematic diagram of a sound source localization device provided in an embodiment of this application. The sound source localization device 900 provided in this embodiment includes: The determination module 910 is used to determine the target speech channel based on the mixed speech acquired by the speech acquisition module. The target speech channel is a speech transmission channel with a signal-to-noise ratio greater than the signal-to-noise ratio threshold. The acquisition module 920 is used to acquire the voice to be processed collected by the target microphone array, and the target microphone array is used to indicate the microphone array corresponding to the target voice channel in the voice acquisition module; The positioning module 930 is used to locate the user who issued the voice command based on the voice to be processed.

[0075] In some implementations, the sound source localization device 900 is also used for: Determine the time-frequency masking code based on the mixed speech; Based on the speech to be processed and the time-frequency masking code, the first covariance matrix of the speech to be processed is determined, and the output power of the target microphone array in each scanning direction is determined based on the first covariance matrix. The location of the user issuing the voice command is determined based on the scanning direction corresponding to the maximum output power.

[0076] In some implementations, the sound source localization device 900 is also used for: Obtain the steering vector of the target microphone array in each scanning direction; The output power of the target microphone array in each scanning direction is determined based on the first covariance matrix and the steering vector in each scanning direction.

[0077] In some implementations, the sound source localization device 900 is also used for: The location requirements of the event to be located are determined based on the voice to be processed; Based on the positioning requirements, the speech to be processed, and the time-frequency masking code, the first covariance matrix of the speech to be processed is determined.

[0078] In some implementations, the sound source localization device 900 is also used for: When the positioning requirement indicates that positioning will be delayed until the positioning event has ended, the first covariance matrix of the speech to be processed is determined based on the speech to be processed and the time-frequency masking code.

[0079] In some implementations, the sound source localization device 900 is also used for: When the location requirement indicates that the event to be located needs to be located in real time, obtain the fourth covariance matrix of the speech to be processed in the previous frame. Based on the fourth covariance matrix, the speech to be processed, and the time-frequency masking code, the first covariance matrix of the speech to be processed is determined.

[0080] In some implementations, the sound source localization device 900 is also used for: Based on the speech to be processed and the time-frequency masking code, determine the initial covariance matrix of the speech to be processed; A phase transformation is performed on each element of the initial covariance matrix to obtain the first covariance matrix.

[0081] The sound source localization device and the sound source localization method provided in the above embodiments of this application belong to the same application concept and can execute the sound source localization method provided in any of the above embodiments of this application. They have the corresponding functional modules and beneficial effects for executing the parking space allocation method. Technical details not described in detail in this embodiment can be found in the specific processing content of the sound source localization method provided in the above embodiments of this application, and will not be repeated here.

[0082] The functions of each module in the sound source localization device can be implemented by the same or different processors, and this application embodiment does not limit this.

[0083] It should be understood that the modules in the above sound source localization device can be implemented by a processor calling firmware. For example, the system includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each module of the device. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal to the device or external to the system. Alternatively, the modules in the system can be implemented as hardware circuits. By designing the hardware circuits, some or all of the module functions can be implemented. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above modules are implemented by designing the logical relationships of the components within the circuit. In another implementation, the hardware circuit can be implemented by a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through a configuration file, thereby implementing the functions of some or all of the above modules. All modules of the above sound source localization device can be implemented entirely by a processor calling firmware, or entirely by hardware circuits, or partially by a processor calling firmware with the remaining parts implemented by hardware circuits.

[0084] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above modules. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0085] As can be seen, each module in the above sound source localization device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor types.

[0086] Furthermore, the modules in the above-mentioned sound source localization device can be integrated in whole or in part, or they can be implemented independently. In one implementation, these modules are integrated together and implemented in the form of a System-on-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the modules of the device. The at least one processor can be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0087] This application provides a schematic diagram of the structure of an electronic device, see [link]. Figure 10 As shown, the electronic device includes a memory 1000 and a processor 1010; wherein the memory 1000 is connected to the processor 1010 and is used to store programs; the processor 1010 is used to implement the sound source localization method disclosed in any of the above embodiments by running the programs stored in the memory 1000.

[0088] Specifically, the aforementioned electronic device may further include: a bus, a communication interface 1020, an input device 1030, and an output device 1040. The electronic device may also include a data transceiver module, an image monitoring module, and a signal monitoring module.

[0089] The processor 1010, memory 1000, communication interface 1020, input device 1030, and output device 1040 are interconnected via a bus. Among them: A bus can include a pathway for transmitting information between various components in an electronic device.

[0090] The processor 1010 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0091] The processor 1010 may include a main processor, as well as a baseband chip, modem, etc.

[0092] The memory 1000 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 1000 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0093] Input device 1030 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0094] Output device 1040 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0095] The communication interface 1020 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0096] The processor 1010 executes the program stored in the memory 1000 and calls other devices, which can be used to implement the various steps of any of the sound source localization methods provided in the above embodiments of this application.

[0097] It should be noted that the electronic device can be an in-vehicle terminal, a mobile phone, a wearable device, or a server, etc.; or it can be a vehicle that includes an in-vehicle terminal, etc.

[0098] This application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in the memory through the data interface to execute the sound source localization method described in any of the above embodiments. For the specific processing procedure and its beneficial effects, please refer to the embodiments of the sound source localization method described above.

[0099] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the sound source localization method described in any of the above embodiments of this specification.

[0100] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone firmware package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0101] Furthermore, embodiments of this application may also be storage media storing computer programs, which are executed by a processor to perform the steps of the sound source localization method according to various embodiments of this application described in any of the above embodiments of this specification, specifically implementing the steps of the sound source localization method as described above.

[0102] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0103] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0104] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0105] The units of the apparatus in the various embodiments of this application can be merged, divided, and deleted according to actual needs.

[0106] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0107] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0108] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or as firmware functional modules or sub-modules.

[0109] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer firmware, or a combination of both. To clearly illustrate the interchangeability of hardware and firmware, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or firmware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0110] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, firmware units executed by a processor, or a combination of both. The firmware unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0111] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0112] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for locating a sound source, characterized in that, include: The target speech channel is determined based on the mixed speech acquired by the speech acquisition module. The target speech channel is a speech transmission channel with a signal-to-noise ratio greater than the signal-to-noise ratio threshold. Acquire the voice to be processed from the target microphone array, wherein the target microphone array is used to indicate the microphone array corresponding to the target voice channel in the voice acquisition module; Based on the voice to be processed, the user who issued the voice command is located.

2. The sound source localization method according to claim 1, characterized in that, The step of locating the user who issued the voice command based on the voice to be processed includes: Determine the time-frequency masking code based on the mixed speech; Based on the speech to be processed and the time-frequency masking code, a first covariance matrix of the speech to be processed is determined, and the output power of the target microphone array in each scanning direction is determined based on the first covariance matrix. The location of the user issuing the voice command is determined based on the scanning direction corresponding to the maximum output power.

3. The sound source localization method according to claim 2, characterized in that, Determining the output power of the target microphone array in each scanning direction based on the first covariance matrix includes: Obtain the steering vector of the target microphone array in each of the scanning directions; The output power of the target microphone array in each scanning direction is determined based on the first covariance matrix and the steering vector in each scanning direction.

4. The sound source localization method according to claim 2, characterized in that, The step of determining the first covariance matrix of the speech to be processed based on the speech to be processed and the time-frequency masking code includes: The location requirement of the event to be located is determined based on the voice to be processed; Based on the positioning requirements, the speech to be processed, and the time-frequency masking code, the first covariance matrix of the speech to be processed is determined.

5. The sound source localization method according to claim 4, characterized in that, The step of determining the first covariance matrix of the speech to be processed based on the positioning requirements, the speech to be processed, and the time-frequency masking code includes: When the positioning requirement indicates that positioning should be performed after the event to be located has ended, the first covariance matrix of the speech to be processed is determined based on the speech to be processed and the time-frequency masking code.

6. The sound source localization method according to claim 4, characterized in that, The step of determining the first covariance matrix of the speech to be processed based on the positioning requirements, the speech to be processed, and the time-frequency masking code includes: When the positioning requirement indicates that the event to be located needs to be located in real time, the fourth covariance matrix of the speech to be processed in the previous frame is obtained. The first covariance matrix of the speech to be processed is determined based on the fourth covariance matrix, the speech to be processed, and the time-frequency masking code.

7. The sound source localization method according to claim 2, characterized in that, The step of determining the first covariance matrix of the speech to be processed based on the speech to be processed and the time-frequency masking code includes: Based on the speech to be processed and the time-frequency masking code, determine the initial covariance matrix of the speech to be processed; A phase transformation is performed on each element of the initial covariance matrix to obtain the first covariance matrix.

8. An electronic device, characterized in that, Including memory and processor, among which, The memory is connected to the processor and is used to store programs; The processor is used to implement the sound source localization method as described in any one of claims 1-7 by running the program in the memory.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the sound source localization method as described in any one of claims 1-7.

10. A computer program product, characterized in that, It includes computer instructions, which, when executed by a processor, implement the sound source localization method as described in any one of claims 1-7.

Images