Voice real-time tracking positioning method and system, terminal and storage medium

Abstract

The embodiment of the present application relates to the technical field of communication, and discloses a voice real-time tracking positioning method, system, terminal and storage medium.In the present application, a voice direction estimation value is calculated in real time based on each frame of audio data collected by a device, and it is judged whether the audio data is voice data or noise data; a relative energy ratio of voice data and noise data is calculated according to the judgment result; the voice direction estimation value is corrected according to the relative energy ratio to obtain a voice direction; and a voice real-time tracking positioning result is output, including the voice direction.After the voice direction is initially calculated, the voice direction is corrected by voice data and noise data information in the present application, so that a more accurate voice tracking positioning result is obtained, and meanwhile, detection and distinction of voice signals in audio signals are not needed separately, so that the complexity of the audio data processing process is reduced, and the real-time performance of voice tracking positioning is ensured.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of communication, in particular to a voice real-time tracking positioning method and system, a terminal and a storage medium. BACKGROUND

[0002] With the development and maturity of signal processing technology and embedded chips, and the increase of user demand, voice real-time tracking positioning technology is gradually moving from pure research field to product application, such as intelligent conference system, which involves voice tracking and speaker positioning technology, and is one of the typical application scenarios and devices of audio and video signal processing technology together with face recognition, automatic focusing and other technologies.

[0003] Voice tracking and positioning technology is mainly based on multi-microphone array, and the technology mainly includes TDOA (Time Destination of Arrival) and DOA (Destination of Arrival) two estimation directions, wherein the GCC (Generalized Cross Correlation) method is the core in TDOA, including PHAT (Phase Transformation) method, etc. The SRP (Steered-response Power) method, the beamforming method and the subspace method are included in the DOA method. Since the subspace method involves matrix eigenvalue decomposition and covariance matrix inversion processes, the calculation amount is large, and the beamforming method and the SRP method are mainly used in practical application.

[0004] However, at least the following problems exist in the current voice tracking and positioning technology:

[0005] On the one hand, in the mainstream GCC method, the delay is obtained by the cross-correlation of two microphones, and the accuracy is affected by the microphone spacing, sampling rate, distance, etc. The quantization error of the calculated delay leads to insufficient azimuth estimation accuracy and positioning accuracy. On the other hand, most of the current voice real-time tracking and positioning technology has large calculation amount, and the mainstream embedded chip is difficult to bear the large calculation amount, which leads to that the real-time performance of voice tracking and positioning is difficult to meet, and most of the current technology needs the cooperation of VAD (Voice Activity Detection) algorithm, which separately detects whether there is a voice signal in the current audio signal and distinguishes it from the background noise, leading to more complex audio processing process, so that the position of the speaker cannot be responded in time, and the voice tracking and positioning hysteresis is larger. SUMMARY

[0006] The embodiment of the present application aims to provide a speech real-time tracking positioning method, system, terminal and storage medium, so as to improve the speech tracking positioning precision under the premise of meeting the real-time requirement.

[0007] To solve the above technical problems, the embodiment of the present application provides a speech real-time tracking positioning method, comprising:

[0008] Real-time calculation of a speech direction estimation value based on each frame of audio data collected by a device, and judgment of whether the audio data is speech data or noise data;

[0009] Calculation of a relative energy ratio of the speech data and the noise data according to the judgment result;

[0010] Correction of the speech direction estimation value according to the relative energy ratio to obtain a speech direction;

[0011] Output of a speech real-time tracking positioning result, including the speech direction.

[0012] The embodiment of the present application also provides a speech real-time tracking positioning system, comprising:

[0013] A real-time processing module, configured to real-time calculate a speech direction estimation value based on each frame of audio data collected by a device, and judge whether the audio data is speech data or noise data; and further configured to calculate a relative energy ratio of the speech data and the noise data according to the judgment result, correct the speech direction estimation value according to the relative energy ratio, and obtain a speech direction;

[0014] A result output module, configured to output a speech real-time tracking positioning result, including the speech direction.

[0015] The embodiment of the present application also provides a terminal, comprising:

[0016] At least one processor; and

[0017] A memory in communication connection with the at least one processor; wherein

[0018] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned speech real-time tracking positioning method.

[0019] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned speech real-time tracking positioning method.

[0020] Compared with the prior art, the embodiment of the present application realizes more accurate voice tracking positioning results according to actual voice data information in audio data except noise, corrects the voice direction after preliminary calculation of the voice direction, realizes the voice tracking positioning, and reduces the complexity of the audio data processing process without separately detecting and distinguishing the voice signal in the audio signal, thereby ensuring the real-time performance of the voice tracking positioning. BRIEF DESCRIPTION OF DRAWINGS

[0021] One or more embodiments are illustrated by way of example in the figures that form a part of this patent document, these example are not to be construed as limiting the embodiments, elements having the same reference numbers in different figures indicate like elements, the figures in the drawings are not to scale unless otherwise indicated.

[0022] Figure 1 is a voice tracking positioning method flowchart provided according to a general voice tracking positioning technology;

[0023] Figure 2 is a voice real-time tracking positioning method flowchart provided according to the embodiment of the present application;

[0024] Figure 3 is a microphone linear array distribution schematic diagram provided according to the embodiment of the present application;

[0025] Figure 4 is a microphone circular ring array distribution schematic diagram provided according to the embodiment of the present application;

[0026] Figure 5 is a voice and noise second-order moment statistical distribution difference simulation schematic diagram provided according to the embodiment of the present application;

[0027] Figure 6 is a sub-flowchart of the voice real-time tracking positioning method provided according to the embodiment of the present application;

[0028] Figure 7 is another sub-flowchart of the voice real-time tracking positioning method provided according to the embodiment of the present application;

[0029] Figure 8 is an output result simulation schematic diagram of the voice real-time tracking positioning method provided according to the embodiment of the present application;

[0030] Figure 9 is a voice tracking positioning system module schematic diagram provided according to the embodiment of the present application;

[0031] Figure 10 is another voice tracking positioning system module schematic diagram provided according to the embodiment of the present application;

[0032] Figure 11 is a structural diagram of a network side service end provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. However, those skilled in the art can understand that in the embodiments of the present application, many technical details are presented in order to make the readers better understand the present application. However, the technical solutions claimed by the present application can be implemented even without these technical details and based on various changes and modifications of the following embodiments. The division of the following embodiments is for the convenience of description and should not constitute any limitation on the specific embodiments of the present application, and the embodiments can be combined and referenced with each other without contradiction.

[0034] The intelligent conference system is one of typical applications of audio and video signal processing technology. There are many difficulties in applying the voice tracking and positioning technology in the intelligent conference system, for example:

[0035] Firstly, the intelligent conference system is usually located in a limited three-dimensional space, and the consistency of signals collected by each channel of the microphone array is poor. Room reverberation, reflection and absorption cause obvious spatial correlation loss of each microphone, and the array signal model is seriously mismatched. Secondly, the current commonly used geometric method and grid point scanning method have the problems of low precision and large calculation amount, and the mainstream embedded chip is difficult to bear such a large calculation amount, which makes it difficult to meet the real-time requirement. Thirdly, as a typical short-time stationary signal, the limitation caused by the short-time characteristic of the voice signal is that the signal processing algorithm cannot obtain a long time gain. Secondly, the voice signal has clear and voiced sounds, and the voiced sound has obvious periodicity, mainly concentrated in the low frequency. The clear sound is similar to white noise spectrum, mainly concentrated in the high frequency, which leads to different positioning accuracy in different frequency bands, and it is difficult to use the same processing method. Fourthly, the lag problem is serious. Most voice tracking and positioning technologies need the cooperation of VAD algorithm, and the use of different lengths of historical information causes the algorithm to be unable to respond to the position of the speaker in time. For example, in a situation similar to two people talking, the position of the speaker cannot be reflected in time.

[0036] In terms of system and hardware, the voice tracking module for voice tracking and positioning mainly needs a multi-channel collection module, an audio processing module and a related data stream communication module. The general system structure is shown in Figure 1 .

[0037] To solve the problems of low accuracy and poor real-time performance in current voice tracking and positioning, the embodiment of the present application relates to a voice real-time tracking and positioning method, which is characterized in that a voice direction estimation value is calculated in real time based on each frame of audio data collected by a device, and it is determined whether the audio data is voice data or noise data; a relative energy ratio is calculated according to the determination result; the voice direction estimation value is corrected according to the relative energy ratio to obtain a voice direction; and a voice real-time tracking and positioning result is output, including the voice direction. After the voice direction estimation value is calculated, the voice direction is corrected by determining whether the audio data is voice data or noise data and calculating the relative energy ratio, so that a more accurate voice tracking and positioning result is obtained according to the actual voice data information in the audio data except noise, and compared with the current voice tracking and positioning method, the voice signal detection and distinction in the audio signal are not needed, the complexity of the audio data processing process is reduced, the problem of excessive calculation amount is avoided, and the real-time performance of voice tracking and positioning is ensured. The implementation details of the embodiment are described below, and the following content is only provided for the implementation details for easy understanding, and is not necessary for implementing the scheme.

[0038] The voice real-time tracking and positioning method flow of the embodiment of the present application is shown in Figure 2 , and specifically includes:

[0039] S101, a voice direction estimation value is calculated in real time based on each frame of audio data collected by a device, and it is determined whether the audio data is voice data or noise data.

[0040] In the embodiment, the audio data in a space is collected by a device, each frame of audio data is processed in real time, a voice direction estimation value is calculated, and voice and noise are determined.

[0041] In one example, each frame of audio data collected by a device is each frame of microphone signal data collected in multiple channels, and real-time signal data collected by multiple microphones in a conference room is obtained by a device. A microphone array can be used for signal data collection, and the microphone array can be a linear array or a circular array. The linear array is very long, and the circular array is usually small. The linear array and the circular array each have advantages and disadvantages. Generally, the linear array can only locate one side of the space, and can be conveniently connected to a television video in a wired manner. The circular array can locate the entire space, and is usually placed separately from the television video in a wireless transmission manner. The typical array distribution of the linear array and the circular array is shown in Figure 3 and Figure 4 .

[0042] In one example, the sampling parameter is configured in advance before the audio data is collected, and the current spatial information is acquired. The current space is sampled based on the pre-configured sampling parameter and the pre-acquired spatial information to obtain the audio data of each sampling point. The pre-configured sampling parameter includes a horizontal scanning interval, a sampling rate, a shot integration time, a processing frequency band, and a frame length. The horizontal scanning interval refers to the interval of the scanning line in the horizontal direction, which is used to reflect the horizontal resolution. The sampling rate, also known as the sampling speed or the sampling frequency, defines the number of samples extracted from the continuous signal and composed of the discrete signal per unit time, which is expressed in hertz (Hz), and the reciprocal is the sampling period or the sampling time. In the display signal processing, the shot refers to a sampling in the time domain, and the shot integration time is the adjustment time for eliminating the deviation when the controller performs the shot. The processing frequency band refers to the frequency interval for which the data is currently processed. The frame length refers to the length of the data frame, which refers to the length of each frame of audio data in the embodiment, and the audio data includes a plurality of sampling points. The pre-acquired spatial information includes the number of microphones, the microphone serial number, and the relative coordinates of the microphones, which are acquired based on different microphone array distributions.

[0043] In one example, the SRP-PHAT (Steered Response Power-Phase Transform) algorithm can be used to calculate the speech direction estimation value in real time based on each frame of audio data collected by the device. The SRP (steered-response power) method is a method of using beamforming to enhance the sound in different directions of the space, and the direction with the strongest sound signal is considered to be the direction of the sound source. The PHAT (Phase Transformation) is used to weight the calculation result, which is the SRP-PHAT algorithm. SRP-PHAT is a sound source positioning algorithm based on the phase transformation weighted controllable response power. The basic principle is to calculate the sum of the phase transformation weighted generalized cross-correlation GCC-PHAT functions of all microphone pairs receiving signals at the assumed sound source position, and to find the point with the maximum SRP value in the entire sound source space, which is the sound source position estimation. SRP-PHAT is an effective method that can be applied to both narrowband and wideband signals, and has strong robustness to reverberation, which can realize sound source positioning in real environment and has been widely studied and applied. However, the positioning performance is poor in low SNR (Signal-to-Noise Ratio) environment, and the operation amount of the global search process is large. In the embodiment, the SRP-PHAT algorithm can be used to obtain the preliminary estimation of the speech direction.

[0044] In one example, based on the calculation requirement of the SRP-PHAT algorithm, a direction calculation table of the SRP-PHAT algorithm is constructed before collecting the audio data, and the direction of the grid point scanned by the steering vector is stored. It can be set as a three-dimensional complex array, and the three dimensions are frequency, channel and direction grid point respectively. The direction calculation table can be recorded as:

[0045]

[0046] wherein w is the channel, M is the number of microphones, m is the microphone channel index, f is the frequency, k is the frequency point number after Fourier transform of each frame of audio data, θ is the direction grid point, N is the number of sampling points of each frame of audio data, and n is the direction grid point index.

[0047] After scanning the space based on the direction calculation table and obtaining the audio sampling data of each sampling point, each frame of audio data is processed in real time. One frame contains multiple sampling points, that is, each frame of audio data contains multiple audio sampling data. For example, if the sampling rate in the preset parameter is 48000 Hz and the length of one frame is 20 ms, then the length of one frame of data stream is 960 sampling points, that is, N is 960. If the sampling rate in the preset parameter is 48000 Hz and the length of one frame is 10 ms, then the length of one frame of data stream is 480 sampling points, that is, N is 480.

[0048] In one example, based on the multi-channel microphone signal data collected by the device for each frame of audio data, N-point short-time Fourier transform is performed on the multi-channel microphone signal data, and the calculation formula is:

[0049]

[0050] wherein f is the frequency, M is the number of microphones, m is the microphone channel index, L is the total number of frames of all audio data, and l is the frame number.

[0051] In one example, the SRP-PHAT algorithm is used to calculate the spatial spectrum of the audio data, and the calculation result is P(θ n ), which is specifically an audio data covariance matrix based on microphone channel summation. According to the spatial spectrum information, the speech direction estimation value can be obtained, and the calculation formula is:

[0052] θ estimate =arg θ max P(θ n )

[0053]

[0054] wherein θ estimateθ is the azimuth grid point, N is the number of sampling points of each frame of audio data, and n is the index of the azimuth grid point.

[0055] That is, the azimuth grid point at which the audio sampling data in the spatial spectrum has a peak is the speech azimuth estimation value.

[0056] In one example, because the SRP-PHAT algorithm has poor positioning performance in a low SNR (Signal-to-Noise Ratio) environment, the audio data is determined to be speech data or noise data, and the calculation result of the SRP-PHAT algorithm is modified according to the determination result, which can effectively improve the accuracy of speech tracking and positioning.

[0057] In one example, the speech data or noise data can be determined by calculating the second moment. The second moment is the variance. For stationary noise and speech signals, the statistical distribution of the second moment has obvious differences, as shown in FIG. 1. The likelihood ratio detection method can be used to determine the speech and noise, or a fixed threshold value can be used for determination. Because the second moment has the advantage of not being easily affected by the signal-to-noise ratio, the determination method using the fixed threshold value has high generalization ability. Figure 5

[0058] In one example, the second moment of the energy of the pre-stored noise template data is used as a threshold value. The energy of the current frame of audio data is calculated according to the current frame of audio data, and the second moment of the energy is calculated.

[0059] The short-time multi-channel microphone weighted average energy of the current frame of audio data can be calculated, and the calculation formula is as follows:

[0060]

[0061] where w is the channel, M is the number of microphones, m is the microphone channel index, L is the total number of frames of all audio data, and l is the frame number.

[0062] Based on the calculated energy of multiple frames of audio data, the energy queue can be constructed using the queue accumulation method. The queue has the characteristics that one frame of data is entered and one frame of data is exited, and the first-in last-out. The accumulation method is as follows:

[0063]

[0064] where l is the frame number, D is the queue length, and E is the short-time multi-channel microphone weighted average energy.

[0065] The constructed energy queue is as follows:

[0066] [e1, e2, …, eT], t = 0, 1, …, T T

[0067] ​​Wherein, e is the short-time multi-channel microphone weighted average energy of each frame of audio data in the queue, and T is the queue length.

[0068] Meanwhile, the second moment of the energy queue is calculated, and the calculation formula is:

[0069]

[0070] Wherein, E is the short-time multi-channel microphone weighted average energy of each frame of audio data, and var is the second moment of the energy.

[0071] According to the statistical distribution of the second moment of the energy and the pre-stored threshold, it can be judged whether the audio data is mainly voice data or noise data. Specifically, if the second moment of the energy is greater than the pre-set threshold, it means that the current is mainly voice, and the audio data is judged as voice data; if the second moment of the energy is less than the pre-set threshold, it means that the current is mainly noise, and the audio data is judged as noise data, and the threshold is updated to the second moment of the energy of the current frame of audio data.

[0072] S102, calculating the relative energy ratio according to the judgment result.

[0073] Specifically, after judging that the audio data is voice data or noise data, the relative energy ratio of voice data and noise data is calculated, that is, the ratio of the energy of voice data to the energy of noise data in the audio data.

[0074] It is worth mentioning that, as mentioned above, the difference between the statistical distribution of the second moment of the voice and the noise can be used to distinguish the voice data and the noise data, and theoretically, the absolute value of the two can be used to distinguish. Considering that in actual application scenarios, the voice signal and the noise signal may be large or small, using the ratio as a parameter to represent the energy relationship can improve the generalization ability of the calculated parameter. For example, in a conference room scenario, the voice signals generated by different speakers are different in size, and the noise signals are also different in size when the air conditioner and fan are turned on. Calculating the relative energy ratio can improve the adaptability of the embodiment to various application scenarios.

[0075] In one example, the same way as constructing the energy queue is used to construct the orientation queue, which contains the voice orientation estimation value of multiple frames of audio data. The constructed orientation queue is:

[0076] [s1, s2, …, s T ], t = 0, 1, …, T

[0077] Wherein, s is the voice orientation estimation value of each frame of audio data in the queue, and T is the queue length.

[0078] In one example, after calculating the relative energy ratio, the orientation queue and energy queue are updated based on the relative energy ratio. Specifically, when the relative energy ratio is large, the audio data is more likely to be speech data, and a more accurate speech orientation can ultimately be calculated. However, when the calculated relative energy ratio is small, the audio data is likely to be noise data. Based on this, the speech orientation is calculated, resulting in a noise orientation that deviates significantly from the orientation of the actual sound source. In this case, the orientation queue is not updated, and the original orientation is maintained, thereby reducing inaccurate target orientations in the orientation queue. By updating the orientation queue and energy queue based on the relative energy ratio, the accuracy of the ultimately calculated speech orientation can be effectively improved.

[0079] S103: Correct the speech direction estimation value according to the relative energy ratio to obtain the speech direction.

[0080] Specifically, the relative energy ratio reflects the relative proportion of the energy of speech and noise in the current audio. Correcting the speech direction estimation value based on the relative energy ratio can avoid excessive impact of noise on the accuracy of speech tracking and positioning.

[0081] In one example, the updated energy queue is weighted with the updated direction queue to obtain the voice direction, which is calculated as follows:

[0082]

[0083] Among them, θ frame is the voice direction, T is the queue length of the energy queue and direction queue, e t is the short-time multi-channel microphone weighted average energy of the t-th audio data in the energy queue, s t is the estimated value of the voice position of the t-th audio data in the position queue.

[0084] S104: Output the real-time voice tracking and positioning result, including the voice direction.

[0085] Specifically, the speech direction estimation value is corrected according to the relative energy ratio to obtain the speech direction, which is output as the final real-time tracking and positioning result.

[0086] In one example, the output real-time voice tracking and positioning result also includes the energy of the voice data, etc.

[0087] In one example, the audio information to be sent may be organized according to a network interface protocol and output through the interface.

[0088] In one example, further interaction with the video information is possible. Subsequent processing such as speaker continuous tracking, localization and identification and other subsequent processing can be performed according to the estimated orientation information.

[0089] In an embodiment of the present application, the judgment of the audio data as speech data or noise data in S101, as shown in Figure 6 includes mainly the following sub-steps:

[0090] S201, obtaining the second moment of the energy of the pre-stored noise template data, i.e. the threshold value;

[0091] S202, judging whether the calculated second moment of the energy of the speech data is greater than the threshold value;

[0092] Specifically, if the second moment of the energy is greater than the threshold value, step S203 is executed; if the second moment of the energy is less than the threshold value, the audio data is judged as noise data, step S203 is skipped and step S204 is entered.

[0093] S203, updating the threshold value;

[0094] Specifically, the threshold value is updated as the second moment of the energy of the audio data of the current frame, and the audio data is judged as speech data.

[0095] S204, obtaining the judgment result of the audio data as speech data or noise data.

[0096] In an embodiment of the present application, the relative energy ratio is calculated according to the judgment result in S102. In one example, the probability of the existence of speech data is further calculated for each frame of audio data before the orientation queue and the energy queue are updated. As shown in Figure 7 S102 includes the following sub-steps:

[0097] S301, obtaining the judgment result of the audio data and calculating the relative energy ratio;

[0098] Specifically, the judgment result of the audio data as speech data or noise data in step S101 is obtained, and the relative energy ratio of the speech data and the noise data is calculated according to the result.

[0099] S302, judging whether the probability of the existence of speech data needs to be updated;

[0100] Specifically, if the audio data of the current frame is judged as speech data, the probability of the existence of speech data needs to be updated, and step S303 is executed; if the audio data of the current frame is judged as noise data, the probability of the existence of speech data does not need to be updated, and step S304 is executed.

[0101] S303, updating the probability of the existence of speech data;

[0102] Specifically, the history voice data existing probability calculated by the previous frame of audio data is updated as the voice data existing probability calculation formula is:

[0103] p(l) = 0.9p(l-1) + 0.1, l = 0, 1, …, L

[0104] Wherein, p is the voice existing probability, L is the total frame number of all audio data, and l is the frame serial number.

[0105] S304, attenuating the history voice data existing probability;

[0106] Specifically, the history voice data existing probability calculated by the previous frame of audio data is attenuated to obtain the voice data existing probability, and the calculation formula is:

[0107] p(l) = 0.9p(l-1), l = 0, 1, …, L

[0108] Wherein, p is the voice existing probability, L is the total frame number of all audio data, and l is the frame serial number.

[0109] S305, obtaining the voice data existing probability;

[0110] S306, constructing the orientation queue and updating the orientation queue and the energy queue based on the relative energy ratio.

[0111] In the embodiment of the application, if the voice data existing probability is also calculated in S102, the voice real-time tracking positioning result output in S104 also includes the voice data existing probability. Moreover, based on the actual use needs, after the voice real-time tracking positioning result is output, the corresponding judgment can also be made according to the voice existing probability, and the relevant target can be locked according to the target number and the human voice flag bit and other information.

[0112] In one example, the voice real-time tracking positioning test is carried out based on the real environment. Specifically, four omnidirectional microphones in linear arrangement are used, two people are located at a position of 3 meters and speak respectively, and the voice real-time tracking positioning is carried out in the foregoing steps and in the conventional manner respectively, and the obtained voice real-time tracking positioning result is as shown in Figure 8 The result obtained by using the conventional technology for voice tracking positioning is indicated by the original red line, and the orientation jitter is relatively obvious, while using the calculation method in the embodiment of the application for processing, there is basically no jitter, and it can be seen that compared with the traditional smoothing mechanism, the result obtained by the embodiment of the application also basically does not have the problem of estimation lag, and the judgment result of noise is also better. Since a separate VAD algorithm is not used for voice data detection and distinction, the storage occupation and time consumption are relatively small, and the use demand of the general embedded platform can be met.

[0113] The step division of the above various methods is only for the purpose of clear description, and can be combined into one step or split into multiple steps in implementation, as long as the same logical relationship is included, and all are within the protection scope of the patent; adding insignificant modifications or introducing insignificant designs in the algorithm or flow, but not changing the core design of the algorithm and flow, are within the protection scope of the patent.

[0114] Embodiments of the present application also relate to a speech real-time tracking positioning system. As shown in Figure 9 , comprising:

[0115] The real-time processing module 901 is configured to calculate a speech direction estimation value in real time based on each frame of audio data collected by the device, and determine whether the audio data is speech data or noise data; and calculate a relative energy ratio according to the determination result, correct the speech direction estimation value according to the relative energy ratio, and obtain a speech direction.

[0116] The result output module 902 is configured to output a speech real-time tracking positioning result, including the speech direction.

[0117] In one example, as shown in Figure 10 , based on the needs of actual applications, the speech real-time tracking positioning system comprises:

[0118] The real-time processing module 1001 is the same as the real-time processing module 901, and will not be described here.

[0119] The result output module 1002 is the same as the result output module 902, and will not be described here.

[0120] The algorithm parameter configuration module 1003 is configured to set parameters required by the algorithm. For example, when the direction calculation table of the SRP-PHAT algorithm is constructed for data collection and the SRP-PHAT algorithm is used to calculate the speech direction estimation value, the parameters can include the cumulative time, the selection on the frequency band, the selection of the number of short-time Fourier transform points, the selection of horizontal and vertical direction grid scanning points, the setting of the sampling rate, the selection of the frame length, and also include the setting of the sound speed in the air (343 m / s) and the like.

[0121] The array topology information module 1004 is configured to select a three-dimensional or two-dimensional space coordinate origin, and select the array distribution of the reference microphone.

[0122] The data preprocessing and input module 1005 is configured to perform data preprocessing, algorithm parameter configuration, and input of array topology information. The data preprocessing and input module 1005 includes pre-emphasis processing, noise reduction processing, direct current removal processing, and the like, and can selectively enable the corresponding modules according to the needs.

[0123] The front-end interface module 1006 is configured to serve as an interaction module for communication with video and upper-layer software.

[0124] It can be found that the voice real-time tracking positioning system described above is a system embodiment corresponding to the voice real-time tracking positioning method, and can be implemented in cooperation with the voice real-time tracking positioning method. The related technical details mentioned in the voice real-time tracking positioning method are still valid in the present embodiment, and in order to reduce repetition, they will not be described here. Correspondingly, the related technical details mentioned in the above content can also be applied in the voice real-time tracking positioning method.

[0125] It is worth mentioning that each module involved in the present embodiment is a logical module. In actual application, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present application, units not closely related to solving the technical problems proposed by the present application are not introduced in the present embodiment, but this does not mean that there are no other units in the present embodiment.

[0126] The embodiment of the present application also relates to a terminal, such as Figure 11 As shown in the figure, the terminal comprises at least one processor 1101; and a memory 1102 connected with the at least one processor 1101; wherein the memory 1102 stores instructions executable by the at least one processor 1101, and the instructions are executed by the at least one processor 1101 to enable the at least one processor 1101 to execute the above-mentioned sensitive data protection method.

[0127] The memory 1102 and the processor 1101 are connected in a bus manner, the bus can include any number of interconnected buses and bridges, and the bus connects one or more processors 1101 and various circuits of the memory 1102 together. The bus can also connect various other circuits such as peripheral devices, voltage stabilizers, and power management circuits together, which are well known in the art, and therefore, they will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, which provide a unit for communicating with various other devices on the transmission medium. The data processed by the processor 1101 is transmitted on the wireless medium through the antenna, and further, the antenna also receives data and transmits the data to the processor 1101.

[0128] The processor 1101 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. And the memory 1102 can be used to store the data used by the processor 1101 in the execution operation.

[0129] Embodiments of the present application also relate to a computer readable storage medium storing a computer program. The computer program, when executed by a processor, implements the aforementioned real-time tracking and positioning method of voice.

[0130] That is, those skilled in the art can understand that all or part of the steps in the methods of the above-mentioned embodiments can be completed by a program instructing relevant hardware, the program being stored in a storage medium and including a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0131] Those skilled in the art can understand that the above-mentioned embodiments are specific embodiments for implementing the present application, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application.

Claims

1. A method for real-time tracking and locating of speech, characterized by, The method comprises: calculating a speech direction estimation value in real time based on each frame of audio data collected by a device, and determining whether the audio data is speech data or noise data; calculating a relative energy ratio of the speech data and the noise data according to a determination result; correcting the speech direction estimation value according to the relative energy ratio to obtain a speech direction; outputting a real-time tracking and positioning result of speech, including the speech direction; wherein the relative energy ratio is a ratio of energy of the speech data to energy of the noise data; wherein the determination of whether the audio data is the speech data or the noise data comprises: calculating energy of current frame of audio data according to the current frame of audio data; and constructing an energy queue based on the energy; wherein the energy queue contains the energy of multiple frames of audio data; wherein, before the calculation of the relative energy ratio of the speech data and the noise data according to the determination result, the method comprises: constructing a direction queue based on the speech direction estimation value; wherein the direction queue contains the speech direction estimation value of multiple frames of audio data; wherein, the correction of the speech direction estimation value according to the relative energy ratio to obtain a speech direction comprises: weighting the direction queue based on the energy queue to obtain the speech direction.

2. The voice real-time tracking and locating method of claim 1, wherein, The determination of whether the audio data is the speech data or the noise data further comprises: calculating a second-order moment of energy of the energy queue; if the second-order moment of energy is greater than a preset threshold, determining that the audio data is speech data; if the second-order moment of energy is less than a preset threshold, determining that the audio data is noise data, and updating the threshold to the second-order moment of energy; wherein the threshold is a second-order moment of energy of pre-stored noise template data.

3. The voice real-time tracking and locating method of claim 2, wherein, After the calculation of the relative energy ratio of the speech data and the noise data according to the determination result, the method comprises: updating the energy queue and the direction queue based on the relative energy ratio.

4. The voice real-time tracking and locating method according to any one of claims 2 or 3, characterized in that, The device collects each frame of audio data in multiple channels; each frame of audio data contains audio sampling data obtained at multiple sampling points; The real-time calculation of the speech direction estimation value based on each frame of audio data collected by the device comprises: performing N-point short-time Fourier transform on the current frame of audio data; wherein N is the number of sampling points contained in the current frame of audio data; calculating a spatial spectrum of the current frame of audio data according to the N-point short-time Fourier transform result, and the direction grid point where the audio sampling data appears a peak in the spatial spectrum is the speech direction estimation value.

5. The voice real-time tracking and locating method of claim 4, wherein, The calculation of the energy of the current frame of audio data according to the current frame of audio data comprises: calculating a short-time multi-channel microphone weighted average energy of the current frame of audio data; The weighting of the direction queue based on the energy queue to obtain the speech direction comprises: weighting the updated direction queue with the updated energy queue to obtain the speech direction; the calculation formula is: where θ frame is the direction of the speech, T is the length of the energy queue and the direction queue, e t is the tth energy in the energy queue, s t is the tth speech direction estimate in the direction queue.

6. The voice real-time tracking and locating method of claim 5, wherein, Before the updating of the energy queue and the direction queue based on the relative energy ratio, the method further comprises: If the audio data of the current frame is determined as speech data, update the historical speech data existence probability calculated from the audio data of the previous frame as the speech data existence probability; If the audio data of the current frame is determined as noise data, perform decay processing on the historical speech data existence probability calculated from the audio data of the previous frame to obtain the speech data existence probability; The output speech real-time tracking positioning result further includes: Output the speech data existence probability.

7. The voice real-time tracking and locating method of claim 1, wherein, Before the speech direction estimation value is calculated in real time based on each frame of audio data collected by the device, the method further includes: Sample the current space based on pre-configured sampling parameters and pre-acquired spatial information to obtain audio data of each sampling point; The pre-configured sampling parameters include at least one of the following parameters or any combination thereof: horizontal scanning interval, sampling rate, integration time of a snapshot, processing frequency band, frame length; The pre-acquired spatial information includes at least one of the following parameters or any combination thereof: number of microphones, microphone serial number, relative coordinates of microphones; The sampling of the current space based on the pre-configured sampling parameters and the pre-acquired spatial information to obtain the audio data of each sampling point includes: Construct a direction calculation table based on the sampling parameters; the direction calculation table is a three-dimensional complex array including a frequency index, a channel index, and a direction grid point; Sample the current space according to the direction calculation table to obtain the audio data of each sampling point.

8. A voice real-time tracking and locating system, characterized by It includes: A real-time processing module configured to calculate a speech direction estimation value in real time based on each frame of audio data collected by the device, and determine whether the audio data is speech data or noise data; Further configured to calculate a relative energy ratio of the speech data and the noise data according to the determination result, correct the speech direction estimation value according to the relative energy ratio, and obtain a speech direction. A result output module configured to output a speech real-time tracking positioning result including the speech direction.

9. A terminal, characterized by comprising: It includes: At least one processor; And A memory in communication connection with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the speech real-time tracking positioning method according to any one of claims 1 to 7.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the speech real-time tracking positioning method according to any one of claims 1 to 7. The computer program is executed by the processor to implement the speech real-time tracking positioning method according to any one of claims 1 to 7.

Images