Sound source direction estimation method, head-mounted device, and storage medium

Abstract

The application discloses a sound source direction estimation method, a head-mounted device and a computer readable storage medium. The method comprises the following steps: acquiring sound signals collected by each microphone channel in the head-mounted device; calculating band amplitude values corresponding to each first sub-band according to the amplitude values of each frequency point, wherein each first sub-band is a sub-band obtained by dividing a preset frequency range according to a mel scale; calculating band phase values corresponding to each second sub-band according to the phase values of each frequency point, wherein each second sub-band is a sub-band obtained by dividing the preset frequency range according to the mel scale; splicing the band amplitude values and the band phase values corresponding to each channel sound signal to obtain a first signal feature; and estimating a sound source direction estimation result based on the first signal feature by using a preset sound source direction estimation model. The application can realize a high-precision and low-power voice sound source direction estimation task at the chip end of the head-mounted device.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of acoustics, in particular to a sound source direction estimation method, a head-mounted device and a computer readable storage medium. BACKGROUND

[0002] Deep learning algorithm is difficult to be deployed on chip end with low computing resources due to its high complexity of calculation, so only small-scale algorithm model can be run, but the size of algorithm model will directly affect the accuracy of algorithm, that is, the larger the size of algorithm model, the stronger the fitting ability of complex data in theory, and the better the reasoning calculation effect of algorithm model. In order to deploy the deep learning algorithm model into the chip end of the head-mounted device, such as AR glasses chip end, and ensure that the microphone data can be processed in real time, the calculation amount of the algorithm model is usually simplified at the cost of the effect of the algorithm model. In addition, the deep learning algorithm has a very large demand for data quantity, which brings difficulties to the development of high-precision speech sound source direction estimation algorithm for head-mounted device chip end. SUMMARY

[0003] The main purpose of the present application is to provide a sound source direction estimation method, a head-mounted device and a computer readable storage medium, which aims to provide a sound source direction estimation scheme applied to a head-mounted device, so as to realize high-precision, low-power speech sound source direction estimation task on the chip end of the head-mounted device.

[0004] To achieve the above purpose, the present application provides a sound source direction estimation method, which is applied to a head-mounted device, and the sound source direction estimation method comprises:

[0005] Obtaining sound signals collected by each microphone channel in the head-mounted device, performing time-frequency conversion on the sound signals of each channel to obtain complex values of each frequency point in a preset frequency range;

[0006] Calculating the amplitude of each frequency point according to the complex value, and calculating the frequency band amplitude corresponding to each first sub-band respectively according to the amplitude of each frequency point, wherein each first sub-band is a first preset number of sub-bands obtained by dividing the preset frequency range according to the mel scale;

[0007] Calculating the phase value of each frequency point according to the complex value, and calculating the frequency band phase value corresponding to each second sub-band respectively according to the phase value of each frequency point, wherein each second sub-band is a second preset number of sub-bands obtained by dividing the preset frequency range according to the mel scale;

[0008] Splicing the frequency band amplitude and the frequency band phase value corresponding to each channel of the sound signal to obtain a first signal feature;

[0009] adopting a preset sound source direction estimation model to estimate based on the first signal feature to obtain a sound source direction estimation result.

[0010] Optionally, the step of calculating a frequency band amplitude corresponding to each first sub-band according to the amplitude of each frequency point comprises:

[0011] For any first target sub-band in each of the first sub-bands, a first initial result is calculated by adopting a preset Mel filter corresponding to the first target sub-band to filter the power of the frequency points in the first target sub-band after calculating the power of the frequency points in the first target sub-band according to the amplitude of the frequency points in the first target sub-band.

[0012] The first initial result is constrained within a first preset threshold range and normalized to obtain a frequency band amplitude corresponding to the first target sub-band.

[0013] Optionally, the step of calculating a frequency band phase value corresponding to each second sub-band according to the phase value of each frequency point comprises:

[0014] For any second target sub-band in each of the second sub-bands, a second initial result is obtained by adopting a preset weight value corresponding to the frequency point to weight and sum the phase values of the frequency points in the second target sub-band.

[0015] The second initial result is constrained within a second preset threshold range and normalized to obtain a frequency band phase value corresponding to the second target sub-band.

[0016] Optionally, after the steps of calculating the amplitude of each frequency point according to the complex value and calculating a frequency band amplitude corresponding to each first sub-band according to the amplitude of each frequency point, the method further comprises:

[0017] The frequency band amplitudes corresponding to the sound signals of each channel are spliced to obtain a second signal feature.

[0018] A preset voice endpoint detection model is adopted to detect based on the second signal feature to obtain a voice endpoint detection result.

[0019] If the voice endpoint detection result indicates that there is human voice in the sound signals of each channel, the step of calculating the phase value of each frequency point according to the complex value is executed.

[0020] If the voice endpoint detection result indicates that there is no human voice in the sound signals of each channel, the step of obtaining the sound signals collected by each microphone channel in the head-mounted device is returned.

[0021] Optionally, after the step of detecting the voice endpoint detection result based on the second signal feature by using the preset voice endpoint detection model, the method further comprises:

[0022] If the voice endpoint detection result indicates that there is no human voice in the sound signal of each channel, the upload flag is updated to a first preset value.

[0023] After the step of estimating the sound source direction estimation result based on the first signal feature by using the preset sound source direction estimation model, the method further comprises:

[0024] After obtaining the sound source direction estimation result, the upload flag is updated to a second preset value, and the step of obtaining the sound signal collected by each microphone channel in the head-mounted device is performed after a first preset time period.

[0025] The sound source direction estimation method further comprises:

[0026] When it is detected that the current upload flag is the second preset value, a sound source direction signal is extracted from the sound signal collected by each microphone channel in a second preset time period according to the latest obtained sound source direction estimation result, and the sound source direction signal is uploaded to a server, so that the server performs a preset voice task based on the sound source direction signal.

[0027] Optionally, the voice endpoint detection model comprises a time convolution layer, a max-pooling layer, a full connection layer and an activation layer connected in series.

[0028] Optionally, the sound source direction estimation model comprises a preset number of convolution blocks connected in series, and comprises a voice sound judgment network and a sound source angle estimation network connected in parallel after the last convolution block; each convolution block comprises a convolution layer, a batch normalization layer, an activation layer and a max-pooling layer, and the convolution kernel size in the first convolution block is 1*1.

[0029] Optionally, the sound source direction estimation result comprises a first result representing the number of sound sources in the sound signal of each channel and a second result representing the angle value of each sound source; the voice sound judgment network comprises a time convolution layer, a full connection layer and an activation layer connected in series, and the voice sound judgment network is used to output the first result; the sound source angle estimation network comprises a time convolution layer, a full connection layer and an activation layer connected in series, and the sound source angle estimation network is used to output the second result.

[0030] To achieve the above object, the application further provides a head-mounted device, comprising a memory, a processor, and a sound source direction estimation program stored in the memory and executable on the processor, which, when executed by the processor, implements the steps of the sound source direction estimation method.

[0031] In addition, to achieve the above object, the application further provides a computer readable storage medium, which stores a sound source direction estimation program, which, when executed by a processor, implements the steps of the sound source direction estimation method.

[0032] The embodiment of the application provides a sound source direction estimation scheme applied to a head-mounted device, which comprises the following steps: acquiring sound signals collected by each microphone channel in the head-mounted device, performing time-frequency conversion on the sound signals of each channel to obtain complex values of each frequency point in a preset frequency range; calculating the amplitude of each frequency point according to the complex values, and calculating the frequency band amplitude corresponding to each first sub-band respectively according to the amplitude of each frequency point, wherein each first sub-band is a first preset number of sub-bands obtained by dividing the preset frequency range according to a mel scale; calculating the phase value of each frequency point according to the complex values, and calculating the frequency band phase value corresponding to each second sub-band respectively according to the phase value of each frequency point, wherein each second sub-band is a second preset number of sub-bands obtained by dividing the preset frequency range according to a mel scale; splicing the frequency band amplitude and the frequency band phase value corresponding to each channel to obtain a first signal feature; and estimating a sound source direction estimation result based on the first signal feature by using a preset sound source direction estimation model.

[0033] In the above sound source direction estimation scheme, since the input data of the sound source direction estimation model is the signal feature spliced by the compressed frequency band amplitude and frequency band phase value, the amount of data processed by the sound source direction estimation model is greatly compressed, so that the sound source direction estimation model can be implemented by using a structure with low complexity, and accurate sound source direction estimation can also be achieved, so that the sound source direction estimation model can be deployed in the head-mounted device with limited computing power to implement a low-power voice sound source direction estimation task; and the sound source direction estimation is performed in combination with the amplitude and phase of the sound signal, and the mel scale close to human auditory perception is used for frequency band division, so that the low-dimensional first signal feature obtained by compression can also well represent the voice feature in the sound signal, so that the sound source direction estimation model can accurately estimate the sound source direction in the sound signal. Therefore, in the embodiment, a sound source direction estimation scheme applied to a head-mounted device is implemented to enable the head-mounted device chip to complete a high-precision, low-power voice sound source direction estimation task. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A flowchart of a sound source direction estimation method according to an embodiment of the present application;

[0035] Figure 2 A flowchart of a sound source direction estimation method according to an embodiment of the present application;

[0036] Figure 3 A flowchart of a data uploading method according to an embodiment of the present application.

[0037] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0038] It should be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application.

[0039] Reference Figure 1 , Figure 1 A flowchart of a sound source direction estimation method according to an embodiment of the present application;

[0040] The embodiments of the sound source direction estimation method provided by the present application need to be explained that although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be performed in an order different from that shown herein. The sound source direction estimation method of the embodiments of the present application is applied to a head-mounted device, which is an electronic device worn on the head of a user, equipped with a microphone for collecting sound signals to realize corresponding functions. In the present embodiment, it is not limited to which type of head-mounted device is applied, for example, it can be AR glasses, VR glasses, etc. It can be understood that the head-mounted device is limited in size due to being worn on the head of the user, so the computing power of the configured chip is also limited. How to realize high-precision and low-power sound source direction estimation task on the chip end with low computing power is the problem to be solved by the present embodiment. In the present embodiment, the sound source direction estimation method comprises:

[0041] In step S10, the sound signals collected by each microphone channel in the head-mounted device are obtained, and the time-frequency conversion of the sound signals of each channel is performed to obtain the complex values of each frequency point in the preset frequency range.

[0042] The head-mounted device is provided with at least one microphone, and the following is described by way of example of multiple microphones. In the process of sound source direction estimation, the head-mounted device can obtain sound signals collected by each microphone channel, that is, obtain sound signals of multiple channels. The manner of time-frequency conversion of each channel sound signal is not limited in the embodiment, for example, the fast Fourier transform method can be used for time-frequency conversion, that is, the sound signal in the time domain is converted into a signal in the frequency domain, and a complex value of each frequency point in a preset frequency range is obtained. The preset frequency range is a frequency range set in advance according to needs, and the embodiment is not limited, for example, the minimum frequency of the preset frequency range can be set to 0, and the maximum frequency can be set to the Nyquist frequency.

[0043] In a feasible implementation, the head-mounted device can collect sound signals in real time through each microphone in the process of sound source direction estimation, the head-mounted device frames and buffers the collected sound signals of each channel, obtains one or more frames of data from the buffer at a certain frequency for time-frequency conversion, and then performs subsequent processing steps.

[0044] In step S20, the amplitude of each frequency point is calculated according to the complex value, and the frequency band amplitude corresponding to each first sub-band is calculated according to the amplitude of each frequency point, wherein each first sub-band is a first preset number of sub-bands obtained by dividing the preset frequency range according to the mel scale.

[0045] According to the complex value of each frequency point obtained by time-frequency conversion, the amplitude of each frequency point can be calculated, and the calculation manner is not limited in the embodiment. In the embodiment, the preset frequency range is divided to obtain multiple sub-bands (hereinafter referred to as first sub-bands for distinction), and the first sub-bands are divided according to the mel scale, and the number of the first sub-bands is a first preset number. The first preset number can be set in advance according to needs, and is not limited herein, for example, it can be set to 6. In a feasible implementation, the manner of dividing the preset frequency range according to the mel scale can be: converting the maximum frequency f1 and the minimum frequency f2 of the preset frequency range into corresponding mel scales m1 and m2, uniformly taking a first preset number of points in the interval formed by the two mel scales, obtaining a mel scale sequence including the first and last two mel scales, and converting each mel scale in the mel scale sequence into a frequency (that is, hertz) respectively, obtaining a frequency sequence, and each adjacent two frequencies (frequency points) in the frequency sequence define a first sub-band.

[0046] The division result of the first sub-band can be pre-configured in the head-mounted device, or can be obtained by the head-mounted device according to the first preset number of the preset frequency range, and the embodiment is not limited.

[0047] The head-mounted device can calculate the frequency band amplitudes corresponding to each of the first sub-bands according to the amplitudes of the frequency points in the preset frequency range. That is, for each first sub-band, the frequency band amplitude corresponding to the first sub-band can be calculated according to the amplitudes of the frequency points in the first sub-band. The specific calculation method can be various, and is not limited in the embodiment.

[0048] It should be noted that, since the number of frequency points in the preset frequency range is greater than the first preset number (i.e., the number of first sub-bands), the data quantity is compressed when calculating the frequency band amplitudes corresponding to each of the first sub-bands, compared to the amplitudes of the frequency points in the preset frequency range. Moreover, since the first sub-bands are obtained by dividing the preset frequency range according to the mel scale, the frequency band division adopts the mel scale closer to human auditory perception, so that the low-dimensional frequency band amplitudes obtained by compression can also well represent the speech features in the sound signal, thereby helping the sound source direction estimation model to accurately estimate the sound source direction in the sound signal.

[0049] In the embodiment, a feasible implementation of calculating the frequency band amplitudes is provided. In the implementation, the step of calculating the frequency band amplitudes corresponding to each of the first sub-bands according to the amplitudes of the frequency points in the step S20 includes:

[0050] In step S201, for any one first target sub-band in the first sub-bands, the power of the frequency points in the first target sub-band is calculated according to the amplitudes of the frequency points in the first target sub-band, then the first target sub-band is filtered by using the preset mel filter corresponding to the first target sub-band, and the logarithm of the energy of the filtered signal is calculated to obtain a first initial result.

[0051] The method of calculating the frequency band amplitudes of each first sub-band is the same, and an example of any one of the first sub-bands is taken for description, and the first sub-band is referred to as a first target sub-band for distinction. An example of a sound signal of one channel is taken for description. After the amplitudes of the frequency points in the preset frequency range are calculated, the amplitudes of the frequency points in the first target sub-band can be extracted. According to the amplitudes of the frequency points in the first target sub-band, the power of the frequency points in the first target sub-band is calculated, and then the first target sub-band is filtered by using the preset mel filter corresponding to the first target sub-band. The signal energy in the first target sub-band is obtained after filtering, and the logarithm of the signal energy is calculated to obtain a first initial result for distinction. The mel filter corresponding to each first sub-band is set, and the filter coefficients corresponding to each frequency point are included in the mel filter, which can be calculated according to the calculation method of the mel filter coefficients when the first sub-bands are divided.

[0052] In step S202, the first initial result is constrained in a first preset threshold range and normalized to obtain a frequency band amplitude corresponding to the first target sub-band.

[0053] The first preset threshold range can be set as needed, for example, set as (-8, 8). After the first initial result is calculated, the first initial result can be constrained in the first preset threshold range to remove data that is obviously not a speech feature, and then normalized to constrain the frequency band amplitude and the frequency band phase value in a uniform order of magnitude, which is beneficial to the subsequent sound source direction estimation model to make an accurate estimation result.

[0054] In an implementable embodiment, the frequency band amplitude can be calculated in the following manner:

[0055]

[0056] wherein, amp n (i) represents the frequency band amplitude of the i th first sub-band in the n th channel, M represents the total number of frequency points in the i th first sub-band, is the amplitude of the m th frequency point in the n th channel, thra is a threshold value (which can be 8) for constraint, and w i (m) represents a coefficient in the triangular band-pass filter (Mel filter) corresponding to the m th frequency point.

[0057] In step S30, a phase value of each frequency point is calculated according to the complex value, and a frequency band phase value corresponding to each second sub-band is calculated according to the phase value of each frequency point, wherein each second sub-band is a second preset number of sub-bands obtained by dividing the preset frequency range according to a Mel scale.

[0058] According to the complex value of each frequency point obtained by time-frequency conversion, the phase value of each frequency point can be calculated, and the calculation manner is not limited in the present embodiment. In the present embodiment, the preset frequency range is divided to obtain a plurality of sub-bands (hereinafter referred to as second sub-bands for distinction), and the second sub-bands are divided according to a Mel scale, and the number of the divided second sub-bands is a second preset number. The second preset number can be set in advance as needed, which is not limited herein, for example, can be set as 13. In an implementable embodiment, the manner of dividing the preset frequency range according to a Mel scale can be: converting the maximum frequency f1 and the minimum frequency f2 of the preset frequency range into corresponding Mel scales m1 and m2, respectively, taking a second preset number of points uniformly in the interval formed by the two Mel scales to obtain a Mel scale sequence including the first and last two Mel scales, and converting each Mel scale in the Mel scale sequence into a frequency (i.e., hertz) corresponding thereto to obtain a frequency sequence, wherein each adjacent two frequencies (frequency points) in the frequency sequence define a second sub-band.

[0059] The division result of the second sub-band can be pre-configured in the head-mounted device, or can be obtained by the head-mounted device by dividing the preset frequency range according to the second preset number, which is not limited in the embodiment.

[0060] The head-mounted device can calculate the frequency band phase value corresponding to each second sub-band according to the phase values of the frequency points in the preset frequency range. That is, for each second sub-band, the frequency band phase value corresponding to the second sub-band can be calculated according to the phase values of the frequency points in the second sub-band, and the specific calculation method is various, which is not limited in the embodiment.

[0061] It should be noted that, since the number of frequency points in the preset frequency range is greater than the second preset number (that is, the number of second sub-bands), the frequency band phase value corresponding to each second sub-band is calculated, and the data amount is compressed compared with the phase values of the frequency points in the preset frequency range; and since the preset frequency range is divided into each second sub-band according to the mel scale, the frequency band division adopts the mel scale closer to the human ear hearing, so that the low-dimensional frequency band phase value obtained by compression can also well represent the speech features in the sound signal, thereby enabling the sound source direction estimation model to accurately estimate the sound source direction in the sound signal.

[0062] In the embodiment, a feasible implementation manner of calculating the frequency band phase value is proposed. In the embodiment, the step S30 of calculating the frequency band phase value corresponding to each second sub-band according to the phase values of the frequency points includes:

[0063] In step S301, for any one second target sub-band in each second sub-band, the phase values of the frequency points in the second target sub-band are weighted and summed by using the preset weight values corresponding to the frequency points, to obtain a second initial result.

[0064] The method of calculating the frequency band phase value of each second sub-band is the same, and an arbitrary second sub-band in each second sub-band is taken as an example for description, and the second sub-band is referred to as a second target sub-band for distinction. And taking a sound signal of one channel as an example for description. After calculating the phase values of the frequency points in the preset frequency range, the phase values of the frequency points in the second target sub-band can be extracted therefrom. The phase values of the frequency points in the second target sub-band are weighted and summed by using the preset weight values corresponding to the frequency points, and the obtained result is referred to as a second initial result for distinction. Each frequency point in each second sub-band is provided with a corresponding weight, which can be pre-set according to experience, and each frequency point in the frequency band of human speech can be provided with a larger weight.

[0065] Step S302, the second initial result is constrained in a second preset threshold range and normalized to obtain a frequency band phase value corresponding to the second target sub-band.

[0066] The second preset threshold range can be set as needed, for example, set as (-180, 180). After the second initial result is calculated, the second initial result can be constrained in the second preset threshold range to remove data that is obviously not a speech feature, and then normalized to constrain the frequency band amplitude and the frequency band phase value in a uniform order of magnitude, which is beneficial to subsequent sound source direction estimation model to make an accurate estimation result.

[0067] In an implementable embodiment, the phase amplitude can be calculated in the following manner:

[0068]

[0069] Pha n (j) represents a frequency band phase value of the jth second sub-band of the nth channel, D represents the total number of frequency points in the jth second sub-band, is a phase value of the dth frequency point in the nth channel, thrp is a constraint threshold (which can be 180); w j (d) represents an importance degree (weight) of the dth frequency point phase value in the jth second sub-band, is a hyperparameter set in advance according to experience, and w j (d).

[0070] Step S40, the frequency band amplitudes and the frequency band phase values corresponding to the sound signals of each channel are spliced to obtain a first signal feature.

[0071] The sound signals of each channel are respectively calculated for the frequency band amplitude and the frequency band phase value, and the frequency band amplitudes and the frequency band phase values corresponding to the sound signals of each channel are obtained, which are spliced to obtain a result called a first signal feature for distinction.

[0072] For example, assuming that there are N channels, one frame of sound signal in each channel has 480 sampling points, time-frequency conversion is performed to obtain the amplitudes and phase values corresponding to 241 frequency points in a preset frequency range, and the data amount is N*241*2; assuming that the first preset number is 6 and the second preset number is 13, the frequency band amplitudes and the frequency band phase values corresponding to the sound signals of each channel are calculated, and the data amount is N*(6+13), which is greatly compressed compared with N*241*2.

[0073] Step S50, a preset sound source direction estimation model is used to estimate based on the first signal feature to obtain a sound source direction estimation result.

[0074] The sound source direction estimation model can be a model for estimating the sound source direction set in advance as needed, and in this embodiment, the specific model structure adopted to implement the sound source direction estimation model is not limited, and a deep learning model can be adopted to implement the sound source direction estimation model. The result output by the sound source direction estimation model can be preconfigured as needed, for example, it can be configured to output a result representing whether a sound source exists and representing the sound source direction, that is, the sound source direction estimation result obtained by the preset sound source direction estimation model based on the first signal feature can include a result representing whether a sound source exists and representing the sound source direction. The sound source direction estimation model can be pre-trained using sample data. When training the sound source direction estimation model, the data input to the model can be the signal feature calculated according to steps S10-S40 from the sound signal in the sample data; the training method of the sound source direction estimation model is not limited; the training process can be completed in the head-mounted device, or the model can be deployed in the head-mounted device after being trained in other devices.

[0075] In this embodiment, the use of the estimated sound source direction estimation result is not limited, for example, it can be used for voice recognition, voice translation, and other voice tasks.

[0076] The amount of data of the time sequence sampling points captured by the multiple channel microphones is very large. If it is directly input into the deep learning model for calculation, it will bring great challenges to the memory capacity, low power consumption, etc. of the chip platform end. Therefore, in this embodiment, multiple channel frequency band information is used instead of directly captured time sequence sampling points as the input of the deep learning model.

[0077] Since the input data of the sound source direction estimation model is the signal feature spliced by the compressed frequency band amplitude and frequency band phase values, the amount of data processed by the sound source direction estimation model is greatly compressed, so that the sound source direction estimation model can be implemented by using a structure with low complexity, and accurate sound source direction estimation can also be achieved, so that the sound source direction estimation model can be deployed in the head-mounted device with limited computing power to implement a low-power voice sound source direction estimation task. Moreover, the amplitude and phase features of the sound signal are combined for sound source direction estimation, and the frequency band division adopts a mel scale closer to human auditory perception, so that the low-dimensional first signal feature obtained by compression can also well represent the voice features in the sound signal, thereby enabling the sound source direction estimation model to accurately estimate the sound source direction in the sound signal. Therefore, in this embodiment, a sound source direction estimation scheme applied to a head-mounted device is implemented to enable the head-mounted device chip end to complete a high-precision, low-power voice sound source direction estimation task.

[0078] Based on the first embodiment, a second embodiment of the sound source direction estimation method is provided. In the embodiment, after the step S20, the method further comprises:

[0079] In step S60, the frequency band amplitudes corresponding to the sound signals of each channel are spliced to obtain a second signal feature.

[0080] In the embodiment, it is proposed to first detect whether there is a human voice in the sound signal, and only in the case where there is a human voice, the sound source direction estimation is performed, so as to avoid the sound source direction estimation algorithm running all the time, thereby further reducing the power consumption of the head-mounted device.

[0081] After the frequency band amplitudes corresponding to each first sub-band are calculated, the frequency band amplitudes corresponding to the sound signals of each channel can be spliced, and the result is called a second signal feature for distinction. For example, assuming that there are N channels, each channel has 480 sampling points in a frame of sound signal, time-frequency conversion is performed, and the amplitudes corresponding to 241 frequency points in a predetermined frequency range are obtained, and the data amount is N*241*2; assuming that the first predetermined number is 6, the frequency band amplitudes corresponding to the sound signals of each channel are calculated, and the second signal feature is obtained by splicing, and the data amount is N*6.

[0082] In step S70, a predetermined voice endpoint detection model is used to detect the second signal feature to obtain a voice endpoint detection result.

[0083] The voice endpoint detection model can be a model for performing voice endpoint detection set in advance according to needs, that is, to detect whether there is a human voice in the sound signal. In the embodiment, the specific model structure used to implement the voice endpoint detection model is not limited; the result output by the voice endpoint detection model can be preconfigured as needed, and is not limited in the embodiment. The voice endpoint detection model can be trained in advance using sample data. When training the voice endpoint detection model, the data input to the model can be the signal features calculated according to steps S10, S20 and S60 from the sound signals in the sample data; the training method of the sound source direction estimation model is not limited; the training process can be completed in the head-mounted device, or the model can be trained in other devices and then deployed in the head-mounted device.

[0084] If the voice endpoint detection result indicates that there is a human voice in the sound signals of each channel, the step S30 is performed.

[0085] The voice endpoint detection result is a detection result representing whether there is human voice in the sound signal of each channel. If it is determined according to the voice endpoint detection result that there is human voice in the sound signal of each channel, it indicates that sound source direction estimation needs to be performed, at this time, step S30 and the following steps can be executed, that is, the frequency band phase value of each second sub-band is calculated based on the complex value of each frequency point, and then the frequency band amplitude and the frequency band phase value are spliced to obtain the first signal feature, and the sound source direction estimation is performed based on the first signal feature.

[0086] If the voice endpoint detection result represents that there is no human voice in the sound signal of each channel, the step S10 is returned to be executed.

[0087] If it is determined according to the voice endpoint detection result that there is no human voice in the sound signal of each channel, at this time, sound source direction estimation is not needed, and the step S10 can be returned to be executed, that is, the newly collected sound signal of each microphone channel is acquired again, and time-frequency conversion, frequency band amplitude calculation, human voice detection and the like are performed based on the new sound signal. For example, in a feasible implementation manner, the head-mounted device can collect sound signals through each microphone in real time when performing sound source direction estimation, and the head-mounted device frames and buffers the collected sound signals of each channel; when it is detected that there is no human voice in the current processing one or more frames of sound signals, the head-mounted device deletes the current processing one or more frames of sound signals from the buffer, acquires the latest one or more frames of data from the buffer for time-frequency conversion, and performs the subsequent processing steps.

[0088] It should be noted that the sound source direction estimation is performed only when the voice endpoint detection model detects that there is human voice in the sound signal, which avoids unnecessary calculation and reduces the power consumption of the head-mounted device. Moreover, the frequency band amplitude used for human voice detection by the voice endpoint detection model is spliced with the frequency band phase value to provide the sound source direction estimation model, which avoids the secondary calculation of the frequency band amplitude information of all microphone channel data frames when performing sound source direction estimation, thereby further reducing the power consumption of the head-mounted device; at the same time, the amplitude information is also provided to the sound source direction estimation model, so that the amplitude attenuation at different microphone positions due to head shielding for different sound source directions is analyzed in the sound source direction estimation model, thereby obtaining a more accurate sound source direction estimation result.

[0089] In the embodiment, a feasible implementation manner for further reducing the power consumption of the head-mounted device is proposed. In the implementation manner, after the step S70, the following steps are further included:

[0090] In step A10, if the voice endpoint detection result represents that there is no human voice in the sound signal of each channel, the upload flag is updated to a first preset value.

[0091] When the head-mounted device determines that there is no human voice in the sound signals of each channel according to the voice endpoint detection result, on one hand, it can return to step S10, and on the other hand, it can update the upload flag to a first preset value. The upload flag is a flag that is set in advance to indicate whether to upload data to the server. The first preset value is used to represent that the server does not need to upload data, and the specific value of the first preset value can be set in advance as needed, for example, it can be set to false. The server is a server that is configured in advance to perform a voice task based on the sound signal data uploaded by the head-mounted device, and can feed back the execution result to the head-mounted device; complex algorithms can be configured in the server to perform the voice task to achieve a more optimal task execution result; since the head-mounted device only needs to upload sound signal data, it does not need to configure complex algorithms for performing voice tasks, so the power consumption of the head-mounted device can be reduced, so that a head-mounted device with low power consumption or low computing power can also provide voice task corresponding functions and services for users. The voice task can be configured in advance as needed, for example, it can be voice recognition and voice translation, etc., which is not limited in this embodiment.

[0092] When the voice endpoint detection result represents that there is no human voice in the sound signals of each channel, sound source direction estimation does not need to be performed, and the sound signal data does not need to be uploaded to the server to perform a voice task. By updating the upload flag to the first preset value, unnecessary data upload is avoided, thereby further reducing the power consumption of the head-mounted device.

[0093] After the step S50, it further includes:

[0094] Step A20, after obtaining the sound source direction estimation result, updating the upload flag to a second preset value, and returning to execute the step S10 after a first preset time length.

[0095] After the head-mounted device obtains the sound source direction estimation result, it can update the upload flag to the second preset value. The second preset value is used to represent that the server needs to upload data, and the specific value of the second preset value can be set in advance as needed, for example, it can be set to true. After obtaining the sound direction estimation result, by setting the upload flag to the second preset value, the sound signal data can be uploaded to the server in time to perform a voice task, thereby ensuring the normal execution of the voice task.

[0096] In this embodiment, the head-mounted device can start timing after obtaining the sound source direction estimation result, and after a first preset time length, it returns to execute step S10, that is, it reacquires new sound signals to perform time-frequency conversion, amplitude calculation, human voice detection, etc. The first preset time length can be set in advance as needed, for example, it can be set to 3 seconds, indicating that the sound source is not moving fast by default within the first preset time length, so as to ensure the stability of the execution of the voice task.

[0097] The sound source direction estimation method further comprises:

[0098] In step A30, when it is detected that the current upload flag is the second preset value, a sound source direction signal is extracted from sound signals collected by each microphone channel in a latest second preset time period according to a latest obtained sound source direction estimation result, and the sound source direction signal is uploaded to a server, so that the server performs a preset voice task based on the sound source direction signal.

[0099] The head-mounted device can detect the value of the upload flag at a certain frequency. If it is detected that the current upload flag is the second preset value, a sound source direction signal can be extracted from sound signals collected by each microphone channel in a latest second preset time period according to a latest obtained sound source direction estimation result. The second preset time period can be set in advance as needed, and can be the time period of the sound signal corresponding to the sound source direction estimation result. Extracting the sound source direction signal from the sound signals collected by each microphone channel according to the sound source direction estimation result means extracting the signal of the voice sound source direction, that is, the microphone channel collects signals from all directions. For the purpose of performing a voice task, important data is a voice signal. Therefore, the voice signal from the voice sound source direction can be extracted from the sound signal collected by the microphone, the important data is uploaded to the server, the amount of data uploaded is reduced, and the power consumption of the head-mounted device is reduced. In the embodiment, the specific method of extracting the sound source direction signal from the sound signal collected by the microphone channel according to the sound source direction estimation result is not limited, and can be set in advance as needed, for example, a beamforming method can be used.

[0100] In a feasible embodiment, the voice detection and sound source direction estimation can be performed according to the flow of Figure 2 , and the flow of the dashed line part is performed only when the voice detection result is that there is voice. Figure 3 In the flow, the DOA (Direction Of Arrival, direction of arrival) network is the sound source direction estimation model in the embodiment of the application, and the VAD (Voice activity detection, voice activity detection) network is the voice endpoint detection model in the embodiment of the application. The processing and uploading of the sound signal can be performed according to the flow of Figure 3 . Figure 2 and Figure 3 can be executed in two threads respectively. When the upload flag is true, the data in the cache is processed based on the estimated angle value and then uploaded to the server, otherwise the upload flag is waited for again.

[0101] Based on the second embodiment, a third embodiment of the sound source estimation method is provided. In this embodiment, a feasible implementation structure of the speech endpoint detection model is provided. The speech endpoint detection model can include a time convolution layer, a max pooling layer, a full connection layer, and an activation layer connected in series. The activation layer can be, but is not limited to, a hard-sigmod activation layer. Compared with the sound source direction estimation model, the speech endpoint detection model has a higher frequency of use and a simpler task, so only the amplitudes of a small number of frequency bands can be selected as the input of the speech endpoint detection model, that is, the first preset number can be less than the second preset number, so that the size of the speech endpoint detection model can be reduced to a certain extent (the calculation of part of the down-sampling can be reduced). The result output by the speech endpoint detection model can be pre-configured as needed, for example, it can be configured to output a probability value in the range of 0-1, greater than 0.5 indicating the presence of human voice, and less than or equal to 0.5 indicating the absence of human voice. Before training the speech endpoint detection model, sound signals collected by the head-mounted device under different types of noise and sound signals collected by the head-mounted device under a quiet environment with a speech sound source can be collected, and the noise data of all channels is added to the corresponding quiet speech data based on a random signal-to-noise ratio as training data for the speech endpoint detection model. The training data is used to train the speech endpoint detection model. When training the speech endpoint detection model, the label value can be 0 or 1: when the input frequency band amplitude feature (the signal feature obtained by processing the training data according to the methods of steps S10, S20, and S60) saves the environment sound, the label value of the speech endpoint detection network is 0; when the input frequency band amplitude feature saves the noisy speech or pure speech, the label value of the speech endpoint detection network is 1. The loss function used is as follows:

[0102]

[0103] where N represents the number of training sample batches; a is a weight factor for solving the data imbalance problem between the speech category and the environment sound category, a [0, 1], when the input frequency band amplitude feature saves the noisy speech or pure speech, a n = a, when the input frequency band amplitude feature saves the environment sound, a n = 1-a; p n is the output value of the forward calculation of the speech endpoint detection model.

[0104] When the voice endpoint detection model makes a prediction, if the output probability value is greater than 0.5, it is determined that there is human voice in the sound signal collected by each channel microphone; otherwise, it is determined that there is no human voice in the sound signal collected by each channel microphone. The result predicted by the voice endpoint detection model determines whether to continue the calculation of the voice sound source direction estimation. Since the real application scenario contains a large amount of background noise unrelated to human voice, a large amount of background noise in high noise or low noise can be filtered through the calculation of the voice endpoint detection model, and the size of the voice endpoint detection model is extremely small, thereby reducing the power consumption of the head-mounted device to a certain extent (that is, the voice endpoint detection model only needs to be run in most cases).

[0105] Based on the second and / or third embodiments described above, a fourth embodiment of the sound source estimation method of the present application is proposed. In this embodiment, a feasible implementation structure of the sound source direction estimation model is proposed. The sound source direction estimation model can include a preset number of convolutional blocks connected in series, and include a voice sound source judgment network and a sound source angle estimation network connected in parallel after the last convolutional block, wherein the specific structure of the voice sound source judgment network and the sound source angle estimation network is not limited in this embodiment and can be set as needed; the voice sound source judgment network can be used to output a result representing whether there is a sound source, and the sound source angle estimation network can be used to output a result representing the direction of the sound source. Each convolutional block can include a convolutional layer, a batch normalization layer, an activation layer, and a max pooling layer, wherein the activation layer can but is not limited to a PreLU activation layer. The size of the convolution kernel in the first convolutional block can be 1*1. For each convolutional block other than the first convolutional block, the size of the convolution kernel can be 3*3 to facilitate the calling of related library functions for chip platform end hardware calculation acceleration. A batch normalization layer is connected after the convolutional layer in each convolutional block, which can reduce the difficulty of training the sound source direction estimation model without increasing the chip end calculation cost, that is, a batch normalization layer is added after each convolution calculation to constrain the value range of the output features. In specific scenarios, the microphone array arranged in the head-mounted device is usually nonlinear, and there is a case of head occlusion of the sound source. The structure of the sound source direction estimation model provided in this embodiment realizes a method for positioning the voice sound source direction in the occlusion case of the nonlinear distributed microphone array, so as to meet the demand for accurate estimation of the voice sound source direction of the head-mounted device under low power consumption and low signal-to-noise ratio.

[0106] In an implementation, the sound source direction estimation result can include a first result representing the number of sound sources in each channel of the sound signal and a second result representing the angle value (i.e., direction) of each sound source. The speech sound source judgment network can include a time convolution layer, a full connection layer, and an activation layer connected in series. The activation layer can be, but is not limited to, a hard-sigmod activation layer. The speech sound source judgment network is configured to output the first result, and the specific numerical form of the first result is not limited in the implementation. For example, the maximum number of identifiable sound sources can be pre-set, for example, C, and the first result can include C probability values, each probability value corresponding to a speech sound source. If the probability value is greater than a set threshold (e.g., 0.5), it indicates that the speech sound source exists, otherwise it indicates that the speech sound source does not exist. For example, C is set to 3, and the output first result is 0.6, 0.8, and 0.1, indicating that there are two sound sources. The training label of the speech sound source judgment network is set corresponding to the output result, i.e., the training label has C numerical values in total. When there is only one speech sound source, only the first numerical value is 1 and the rest of the numerical values are 0. When there are only two speech sound sources, only the first and second numerical values are 1 and the rest of the numerical values are 0. And so on.

[0107] The sound source angle estimation network can include a time convolution layer, a full connection layer and an activation layer connected in series, wherein the activation layer can adopt, but is not limited to, a softmax activation layer. The sound source angle estimation network is used to output a second result, and the specific numerical form of the second result is not limited in the embodiment. For example, the possible angle values of the sound source can be divided into a preset number of angle categories in advance, for example, from 0 degrees to 360 degrees, and an angle category is set every 15 degrees, a total of 24 categories, representing 0 degrees, 15 degrees, 30 degrees, …, 345 degrees respectively; the second result can include C groups of probability values, and each group of probability values includes a preset number of probability values corresponding to a preset number of angle categories; when the i-th (i = 1, 2, …, C) value in the first result is greater than a set threshold, it indicates that there is a speech sound source, and the i-th group of probability values in the second result is further obtained, and the maximum probability value is selected therefrom, and if it is greater than a set threshold (for example, 0.5), the angle category corresponding to the maximum probability value is taken as the angle value of the speech sound source; when the i-th value in the first result is less than or equal to a set threshold, it indicates that there is no speech sound source, and at this time, the i-th group of probability values in the second result can not be obtained to analyze the angle value. The training label of the sound source angle estimation network is correspondingly set with the output result, that is, the training label has a total of C groups of values, and each group of values includes a preset number of values corresponding to a preset number of categories; when there are N speech sound sources, according to the category in which the angle value of the speech sound source is located, the values of the corresponding categories in the first N groups of values in the C groups of values are set to 1, and the rest are 0, and the values of each category in the remaining groups of values are all 0, that is, the label values of each category are represented in a one-hot manner.

[0108] During the training of the sound source direction estimation model, the error between the output result and the true label value can be calculated based on the training data, and the error is propagated back to each layer in the model to update the weight parameters of the model, so as to realize the training of the sound source direction estimation model. The following loss function can be used:

[0109]

[0110]

[0111]

[0112] wherein L DOA is the loss of the sound source direction estimation model, which is the loss of the speech sound source judgment network and the loss of the sound source angle estimation network , wherein N represents the number of batch samples, C is the maximum number of preset identifiable sound sources, p is a decimal number in the range of 0 to 1 output by the speech sound source judgment network, y is a corresponding label value (0 or 1), p k is a probability value of the kth angle category, and is a decimal number in the range of 0 to 1 (K is the number of divided angle categories), y k is a corresponding label value (one-hot vector).

[0113] In the embodiment, the time convolution layer is set in the sound source direction estimation model, the relationship between the phase and amplitude in the frequency band between different microphone channels is analyzed by using the time convolution layer, compared with the traditional sound source positioning algorithm, the deep learning model can easily construct the mathematical model of the time delay when the direction of different sound sources propagates to different microphones, the amplitude attenuation of different sound sources due to head shielding, and other complex problems, and finally output the correct angle value of the direction estimation of different speech sound sources.

[0114] The head-mounted device in the embodiment of the application can include a structural shell, a communication module, a master control module (such as a micro control unit MCU), a speaker, a microphone, a memory, and the like. The master control module can include a microprocessor, an audio decoding unit, a power supply and a power management unit, sensors and other active or passive devices required by the system (which can be replaced, reduced or increased according to actual functions), to realize the functions of receiving and playing wireless audio. The memory of the earphone can store a sound source direction estimation program, and the microprocessor can be used to call the sound source direction estimation program stored in the memory and perform the following operations:

[0115] Obtain sound signals collected by each microphone channel in the head-mounted device, and perform time-frequency conversion on the sound signals of each channel to obtain complex values of each frequency point in a preset frequency range;

[0116] Calculate the amplitude of each frequency point according to the complex values, and calculate the frequency band amplitude corresponding to each first sub-band respectively according to the amplitude of each frequency point, wherein each first sub-band is a first preset number of sub-bands obtained by dividing the preset frequency range according to a mel scale;

[0117] Calculate the phase value of each frequency point according to the complex values, and calculate the frequency band phase value corresponding to each second sub-band respectively according to the phase value of each frequency point, wherein each second sub-band is a second preset number of sub-bands obtained by dividing the preset frequency range according to a mel scale;

[0118] Splice the frequency band amplitude and the frequency band phase value corresponding to each channel of the sound signals to obtain a first signal feature;

[0119] The preset sound source direction estimation model is used to estimate the sound source direction estimation result based on the first signal feature.

[0120] In an implementation, the operation of calculating the frequency band amplitude corresponding to each first sub-band based on the amplitude of each frequency point comprises:

[0121] For any first target sub-band in each first sub-band, the power of the frequency point in the first target sub-band is calculated based on the amplitude of the frequency point in the first target sub-band, then the power is filtered by using the preset mel filter corresponding to the first target sub-band, and the logarithm of the energy of the filtered signal is calculated to obtain a first initial result.

[0122] The first initial result is constrained in a first preset threshold range and normalized to obtain the frequency band amplitude corresponding to the first target sub-band.

[0123] In an implementation, the operation of calculating the frequency band phase value corresponding to each second sub-band based on the phase value of each frequency point comprises:

[0124] For any second target sub-band in each second sub-band, the phase values of the frequency points in the second target sub-band are weighted and summed by using the preset weight value corresponding to the frequency point to obtain a second initial result.

[0125] The second initial result is constrained in a second preset threshold range and normalized to obtain the frequency band phase value corresponding to the second target sub-band.

[0126] In an implementation, after the operation of calculating the amplitude of each frequency point based on the complex value and calculating the frequency band amplitude corresponding to each first sub-band based on the amplitude of each frequency point, the microprocessor can further be configured to invoke the sound source direction estimation program stored in the memory to perform the following operations:

[0127] The frequency band amplitudes corresponding to the sound signals of each channel are spliced to obtain a second signal feature.

[0128] A preset voice endpoint detection model is used to detect the voice endpoint detection result based on the second signal feature.

[0129] If the voice endpoint detection result indicates that there is human voice in the sound signals of each channel, the operation of calculating the phase value of each frequency point based on the complex value is performed.

[0130] If the voice endpoint detection result represents that there is no human voice in the sound signal of each channel, the operation of obtaining the sound signal collected by each microphone channel in the head-mounted device is performed.

[0131] In an implementation, after the operation of detecting the voice endpoint detection result based on the second signal feature by using the preset voice endpoint detection model, the microprocessor can further be configured to call a sound source direction estimation program stored in the memory and perform the following operation:

[0132] If the voice endpoint detection result represents that there is no human voice in the sound signal of each channel, the upload flag is updated to a first preset value.

[0133] After the operation of estimating the sound source direction estimation result based on the first signal feature by using the preset sound source direction estimation model, the microprocessor can further be configured to call a sound source direction estimation program stored in the memory and perform the following operation:

[0134] After obtaining the sound source direction estimation result, the upload flag is updated to a second preset value, and the operation of obtaining the sound signal collected by each microphone channel in the head-mounted device is performed after a first preset time period.

[0135] The microprocessor can further be configured to call a sound source direction estimation program stored in the memory and perform the following operation:

[0136] When it is detected that the current upload flag is the second preset value, a sound source direction signal is extracted from the sound signal collected by each microphone channel in a second preset time period according to the latest obtained sound source direction estimation result, and the sound source direction signal is uploaded to a server, so that the server performs a preset voice task based on the sound source direction signal.

[0137] In an implementation, the voice endpoint detection model comprises a time convolution layer, a max-pooling layer, a full connection layer and an activation layer connected in series.

[0138] In an implementation, the sound source direction estimation model comprises a preset number of convolution blocks connected in series, and comprises a voice sound judgment network and a sound source angle estimation network connected in parallel after the last convolution block; each convolution block comprises a convolution layer, a batch normalization layer, an activation layer and a max-pooling layer, and the convolution kernel size in the first convolution block is 1*1.

[0139] In an implementation, the sound source direction estimation result includes a first result representing a number of sound sources in the sound signal of each channel and a second result representing an angle value of each sound source; the speech sound source judgment network includes a time convolution layer, a full connection layer and an activation layer connected in series, and is configured to output the first result; and the sound source angle estimation network includes a time convolution layer, a full connection layer and an activation layer connected in series, and is configured to output the second result.

[0140] Embodiments of the head-mounted device and the computer readable storage medium can refer to the embodiments of the sound source direction estimation method, which will not be described herein.

[0141] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that processes, methods, articles, or devices that comprise a list of elements not only include those elements, but also include other elements that are not expressly listed, or other elements inherent in such processes, methods, articles, or devices. Without more limitations, the element defined by the sentence "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0142] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0143] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the methods of various embodiments of the present application.

[0144] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method of sound source direction estimation, characterized by, The sound source direction estimation method is applied to a head-mounted device, and the sound source direction estimation method comprises: obtaining sound signals collected by each microphone channel in the head-mounted device, performing time-frequency conversion on the sound signals of each channel to obtain complex values of each frequency point in a preset frequency range; calculating the amplitude values of the frequency points according to the complex values, and calculating the frequency band amplitude values corresponding to each first sub-band respectively according to the amplitude values of the frequency points, wherein each first sub-band is a first preset number of sub-bands obtained by dividing the preset frequency range according to a mel scale; calculating the phase values of the frequency points according to the complex values, and calculating the frequency band phase values corresponding to each second sub-band respectively according to the phase values of the frequency points, wherein each second sub-band is a second preset number of sub-bands obtained by dividing the preset frequency range according to a mel scale; splicing the frequency band amplitude values and the frequency band phase values corresponding to each channel to obtain a first signal feature; using a preset sound source direction estimation model to estimate based on the first signal feature to obtain a sound source direction estimation result.

2. The acoustic source direction estimation method of claim 1, wherein, The step of calculating the frequency band amplitude values corresponding to each first sub-band respectively according to the amplitude values of the frequency points comprises: for any one first target sub-band in each first sub-band, calculating the power of the frequency points in the first target sub-band according to the amplitude values of the frequency points in the first target sub-band, then filtering using a preset mel filter corresponding to the first target sub-band, and calculating the logarithm of the energy of the filtered signal to obtain a first initial result; constraining the first initial result within a first preset threshold range and performing normalization processing to obtain the frequency band amplitude value corresponding to the first target sub-band.

3. The acoustic source direction estimation method of claim 1, wherein, The step of calculating the frequency band phase values corresponding to each second sub-band respectively according to the phase values of the frequency points comprises: for any one second target sub-band in each second sub-band, performing weighted summation on the phase values of the frequency points in the second target sub-band using preset weight values corresponding to the frequency points to obtain a second initial result; constraining the second initial result within a second preset threshold range and performing normalization processing to obtain the frequency band phase value corresponding to the second target sub-band.

4. The acoustic source direction estimation method of claim 1, wherein, After the steps of calculating the amplitude values of the frequency points according to the complex values, and calculating the frequency band amplitude values corresponding to each first sub-band respectively according to the amplitude values of the frequency points, the method further comprises: splicing the frequency band amplitude values corresponding to each channel to obtain a second signal feature; using a preset speech endpoint detection model to detect based on the second signal feature to obtain a speech endpoint detection result; if the speech endpoint detection result indicates that there is human voice in the sound signals of each channel, performing the step of calculating the phase values of the frequency points according to the complex values; if the speech endpoint detection result indicates that there is no human voice in the sound signals of each channel, returning to perform the step of obtaining the sound signals collected by each microphone channel in the head-mounted device.

5. The acoustic source direction estimation method of claim 4, wherein, The step of detecting the voice endpoint based on the second signal feature by using the preset voice endpoint detection model further comprises: If the voice endpoint detection result indicates that there is no human voice in the sound signal of each channel, the upload flag is updated to a first preset value; The step of estimating the sound source direction based on the first signal feature by using the preset sound source direction estimation model further comprises: After obtaining the sound source direction estimation result, the upload flag is updated to a second preset value, and the step of obtaining the sound signal collected by each microphone channel in the head-mounted device is performed after a first preset time period; The sound source direction estimation method further comprises: When the current upload flag is detected to be the second preset value, a sound source direction signal is extracted from the sound signal collected by each microphone channel in the last second preset time period according to the latest obtained sound source direction estimation result, and the sound source direction signal is uploaded to a server, so that the server performs a preset voice task based on the sound source direction signal.

6. The acoustic source direction estimation method of claim 4, wherein, The voice endpoint detection model comprises a time convolution layer, a max-pooling layer, a full connection layer and an activation layer connected in series.

7. The acoustic source direction estimation method of any one of claims 1 to 6, wherein, The sound source direction estimation model comprises a preset number of convolution blocks connected in series, and comprises a voice source judgment network and a sound source angle estimation network connected in parallel after the last convolution block; each convolution block comprises a convolution layer, a batch normalization layer, an activation layer and a max-pooling layer, and the convolution kernel size in the first convolution block is 1*1.

8. The acoustic source direction estimation method of claim 7, wherein, The sound source direction estimation result comprises a first result representing the number of sound sources in the sound signal of each channel and a second result representing the angle value of each sound source; the voice source judgment network comprises a time convolution layer, a full connection layer and an activation layer connected in series, and the voice source judgment network is used to output the first result; the sound source angle estimation network comprises a time convolution layer, a full connection layer and an activation layer connected in series, and the sound source angle estimation network is used to output the second result.

9. A head-mounted device, comprising: The head-mounted device comprises a memory, a processor and a sound source direction estimation program stored on the memory and executable on the processor, and the sound source direction estimation program implements the steps of the sound source direction estimation method according to any one of claims 1 to 8 when executed by the processor.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a sound source direction estimation program, and the sound source direction estimation program implements the steps of the sound source direction estimation method according to any one of claims 1 to 8 when executed by the processor.

Images