Speaker counting method, device, equipment and storage medium based on deep learning

Abstract

The present invention relates to the field of deep learning technology, and discloses a speaker counting method, device, equipment and storage medium based on deep learning. The method comprises: obtaining corresponding amplitude spectrum information and phase spectrum information according to multi-channel time-domain speech signals in a target area; generating corresponding feature dimension information according to the amplitude spectrum information, the phase spectrum information and preset frame sequence length information; predicting the feature dimension information according to a preset convolutional recurrent neural network model; and determining the number of speakers in the target area based on the predicted speech signal probability distribution information. The present invention obtains speech signal probability distribution information through a preset convolutional recurrent neural network model and feature dimension information, and determines the number of speakers in the target area according to the speech signal probability distribution information, so as to realize the counting of speakers in the target area. Compared with the prior art that realizes the counting of speakers through traditional audio algorithms, the accuracy of speaker counting can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a speaker counting method, device, equipment and storage medium based on deep learning. Background Art

[0002] Speaker number detection refers to detecting the number of speakers in a speech signal. It is usually located in the preprocessing stage of speech-related systems and affects the performance of subsequent tasks to a certain extent, such as speech separation tasks, speaker differentiation tasks, sound source localization tasks, etc. The realization of speech tasks such as speaker differentiation tasks and sound source localization tasks requires determining the number of speakers contained in the speech signal within a period of time. Therefore, how to accurately and efficiently count the number of speakers is extremely important for speech-related systems. The currently commonly used technical solution for counting the number of speakers is achieved through intelligent speech separation products. However, during the use of intelligent speech separation products, users are required to input the number of speakers in advance. However, if a speaker leaves or a new speaker is added during the meeting, and once the number of speakers is different from the number initially notified to the system, the accuracy of counting the number of speakers through the speech separation function of the intelligent speech separation product is low.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present invention is to provide a speaker counting method, device, equipment and storage medium based on deep learning, aiming to solve the technical problem that the existing technology cannot effectively improve the accuracy of speaker counting.

[0005] To achieve the above object, the present invention provides a speaker counting method based on deep learning, and the speaker counting method based on deep learning comprises the following steps:

[0006] Collecting multi-channel time-domain speech signals in the target area, and obtaining corresponding amplitude spectrum information and phase spectrum information according to the time-domain speech signals;

[0007] Generate corresponding feature dimension information according to the amplitude spectrum information, the phase spectrum information and the preset frame sequence length information;

[0008] Predicting the feature dimension information according to a preset convolutional recursive neural network model to obtain speech signal probability distribution information;

[0009] The number of speakers in the target area is determined based on the speech signal probability distribution information, so as to count the speakers in the target area.

[0010] Optionally, the collecting of multi-channel time-domain speech signals in the target area and obtaining corresponding amplitude spectrum information and phase spectrum information according to the time-domain speech signals include:

[0011] Collect multi-channel time-domain speech signals in the target area;

[0012] Framing the time-domain speech signal;

[0013] Performing Fourier transform on the framed time-domain speech signal;

[0014] Obtain target frequency domain information according to a preset sampling rule and a Fourier transformed time domain speech signal;

[0015] Corresponding amplitude spectrum information and phase spectrum information are determined according to the target frequency domain information.

[0016] Optionally, generating corresponding feature dimension information according to the amplitude spectrum information, the phase spectrum information and the preset frame sequence length information includes:

[0017] Obtain the quantity information of the target acquisition devices;

[0018] Determine channel occupancy information of the amplitude spectrum information according to the quantity information;

[0019] Determine channel occupancy information of the phase spectrum information according to the quantity information;

[0020] Acquire preset continuous frame sequence information, and obtain preset frame sequence length information according to the preset continuous frame sequence information;

[0021] Corresponding feature dimension information is generated based on the channel occupancy information of the amplitude spectrum information, the channel occupancy information of the phase spectrum information, and the preset frame sequence length information.

[0022] Optionally, the acquiring preset continuous frame sequence information and obtaining preset frame sequence length information according to the preset continuous frame sequence information includes:

[0023] Get the number of target speakers supported by the preset convolutional recurrent neural network model;

[0024] Obtaining corresponding category information according to the number of target speakers;

[0025] Determining the format information of the label vector according to the preset vector label information and the category information;

[0026] Detecting the time domain speech signal according to a preset speech detection strategy and the format information of the label vector to obtain speech frame information and non-speech frame information;

[0027] Preset continuous frame sequence information is acquired, and preset frame sequence length information is obtained according to the preset continuous frame sequence information, voice frame information and non-voice frame information.

[0028] Optionally, predicting the feature dimension information according to a preset convolutional recursive neural network model to obtain speech signal probability distribution information includes:

[0029] Extracting convolutional neural network information and recurrent recurrent neural network information from the preset convolutional recurrent neural network model;

[0030] Convolving the feature dimension information according to the convolutional neural network information;

[0031] The convolved feature dimension information is predicted according to the cyclic recursive neural network information to obtain the speech signal probability distribution information.

[0032] Optionally, predicting the convolved feature dimension information according to the recurrent neural network information to obtain speech signal probability distribution information includes:

[0033] Extracting long short-term memory recurrent neural network information from the recurrent recurrent neural network information;

[0034] Determining a target prediction strategy based on the long short-term memory recurrent neural network information;

[0035] Determine a corresponding feature dimension time series sequence according to the feature dimension information after the convolution;

[0036] The feature dimension time series sequence is predicted based on the target prediction strategy to obtain speech signal probability distribution information.

[0037] Optionally, determining the number of speakers in the target area based on the speech signal probability distribution information includes:

[0038] Sorting the speech signal probabilities corresponding to the speech information probability distribution information;

[0039] Obtain a target probability value according to the sorted speech signal probability;

[0040] The number of speakers in the target area is determined based on the subscript of the target probability value.

[0041] In addition, to achieve the above-mentioned purpose, the present invention further proposes a speaker counting device based on deep learning, and the speaker counting device based on deep learning comprises:

[0042] An acquisition module is used to acquire multi-channel time-domain speech signals in a target area, and obtain corresponding amplitude spectrum information and phase spectrum information according to the time-domain speech signals;

[0043] A generating module, used to generate corresponding feature dimension information according to the amplitude spectrum information, the phase spectrum information and the preset frame sequence length information;

[0044] A prediction module, used to predict the feature dimension information according to a preset convolutional recursive neural network model to obtain speech signal probability distribution information;

[0045] The counting module is used to determine the number of speakers in the target area based on the speech signal probability distribution information, so as to count the speakers in the target area.

[0046] In addition, to achieve the above-mentioned purpose, the present invention also proposes a deep learning-based speaker counting device, the deep learning-based speaker counting device comprising: a memory, a processor, and a deep learning-based speaker counting program stored in the memory and executable on the processor, the deep learning-based speaker counting program being configured to implement the deep learning-based speaker counting method as described above.

[0047] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a deep learning-based speaker counting program is stored. When the deep learning-based speaker counting program is executed by a processor, the deep learning-based speaker counting method as described above is implemented.

[0048] The deep learning-based speaker counting method proposed in the present invention collects multi-channel time-domain speech signals in a target area, obtains corresponding amplitude spectrum information and phase spectrum information according to the time-domain speech signals; generates corresponding feature dimension information according to the amplitude spectrum information, phase spectrum information and preset frame sequence length information; predicts the feature dimension information according to a preset convolutional recurrent neural network model to obtain speech signal probability distribution information; determines the number of speakers in the target area based on the speech signal probability distribution information; the present invention obtains speech signal probability distribution information through a preset convolutional recurrent neural network model and feature dimension information, determines the number of speakers in the target area according to the speech signal probability distribution information, so as to realize the counting of speakers in the target area. Compared with the prior art that realizes speaker counting through traditional audio algorithms, the accuracy of speaker counting can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a structural diagram of a speaker counting device based on deep learning in a hardware operating environment involved in an embodiment of the present invention;

[0050] Figure 2 It is a flowchart of a first embodiment of a method for counting speakers based on deep learning according to the present invention;

[0051] Figure 3 It is a flowchart of a second embodiment of a method for counting speakers based on deep learning according to the present invention;

[0052] Figure 4 It is a flowchart of a third embodiment of a method for counting speakers based on deep learning according to the present invention;

[0053] Figure 5 This is a schematic diagram of functional modules of the first embodiment of a speaker counting device based on deep learning according to the present invention.

[0054] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0056] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a speaker counting device based on deep learning in the hardware operating environment involved in an embodiment of the present invention.

[0057] like Figure 1 As shown, the speaker counting device based on deep learning may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0058] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the deep learning-based speaker counting device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0059] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a speaker counting program based on deep learning.

[0060] exist Figure 1 In the deep learning-based speaker counting device shown, the network interface 1004 is mainly used for data communication with the network speaker counting program; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the deep learning-based speaker counting device of the present invention can be set in the deep learning-based speaker counting device, and the deep learning-based speaker counting device calls the deep learning-based speaker counting program stored in the memory 1005 through the processor 1001, and executes the deep learning-based speaker counting method provided by the embodiment of the present invention.

[0061] Based on the above hardware structure, an embodiment of a speaker counting method based on deep learning of the present invention is proposed.

[0062] Reference Figure 2 , Figure 2 4 is a flow chart of a first embodiment of a method for counting speakers based on deep learning according to the present invention.

[0063] In a first embodiment, the speaker counting method based on deep learning comprises the following steps:

[0064] Step S10, collecting multi-channel time-domain speech signals in the target area, and obtaining corresponding amplitude spectrum information and phase spectrum information according to the time-domain speech signals.

[0065] It should be noted that the executor of this embodiment is a speaker counting device based on deep learning, and may also be other devices that can achieve the same or similar functions, such as a speaker counting program, etc. This embodiment does not limit this. In this embodiment, the speaker counting program is taken as an example for explanation.

[0066] It should be understood that the time domain speech signal refers to the short-term unchanged speech signal in the target area and within the preset time. The multi-channel time domain speech signal is collected by a microphone array arranged in the target area, and the frequency of each microphone in the microphone array is 16kHZ.

[0067] It can be understood that the amplitude spectrum information refers to the information presented by the characteristics of the size of the speech energy in the time domain speech signal. In judging the target area, judging the number of speakers through the time domain speech signal needs to rely on the speech energy in the time domain speech signal. The phase spectrum information refers to the information presented by the characteristics of the spatial information of each sound source obtained from the arrival delay and sampling offset.

[0068] Furthermore, step S10 includes: collecting multi-channel time domain speech signals in the target area; framing the time domain speech signals; performing Fourier transform on the time domain speech signals after framing; obtaining target frequency domain information according to preset sampling rules and the time domain speech signals after Fourier transform; and determining corresponding amplitude spectrum information and phase spectrum information according to the target frequency domain information.

[0069] It can be understood that after obtaining the multi-channel time domain speech signal in the target area, the time domain speech signal of each channel is framed respectively. Framing refers to dividing the speech frame corresponding to the time domain speech signal into unit frames, and then Fourier transforming the time domain speech signal after framing. Fourier transforming refers to converting the form of the time domain speech signal. Fourier transforming includes continuous Fourier transforming and discrete Fourier transforming. After the conversion is completed, the target frequency domain information is obtained by the preset sampling rules and the time domain speech signal after Fourier transforming. The preset sampling rules refer to the rules for sampling the target samples. The preset sampling rules are used to sample the time domain speech signal after Fourier transforming to obtain the target frequency domain information, and the amplitude spectrum information and phase spectrum information in the target frequency information are extracted. For example, 512 points are taken in the time domain speech signal after Fourier transforming, and the overlap rate is 50%. At this time, the frequency domain size in the target frequency domain information is 257.

[0070] It should be understood that the amplitude spectrum information is obtained by calculating the target spectrum information through formula one, specifically:

[0071]

[0072] Among them, |A m | is the amplitude spectrum information, F m re is the real part of the frequency domain information corresponding to the signal collected by the mth microphone, F m im The imaginary part of the frequency domain information corresponding to the signal collected by the mth microphone.

[0073] It can be understood that the phase spectrum information is obtained by calculating the target spectrum information through Formula 2, which is specifically:

[0074]

[0075] in, <P m is the phase spectrum information, F m re is the real part of the frequency domain information corresponding to the signal collected by the mth microphone, F m im The imaginary part of the frequency domain information corresponding to the signal collected by the mth microphone.

[0076] In a specific implementation, the speaker counting program collects multi-channel time-domain speech signals in a target area, and obtains corresponding amplitude spectrum information and phase spectrum information according to the time-domain speech signals.

[0077] Step S20, generating corresponding feature dimension information according to the amplitude spectrum information, the phase spectrum information and the preset frame sequence length information.

[0078] It should be understood that after obtaining the amplitude spectrum information and the phase spectrum information, it is necessary to determine the number of devices for collecting the time domain speech signal, and determine the number of channels occupied by the amplitude spectrum information and the phase spectrum information through the number of devices. The channel information occupied by the amplitude spectrum information and the phase spectrum information is consistent. For example, if the number of microphones is 4, then the number of channels occupied by the amplitude spectrum information and the phase spectrum information is also 4. The time domain speech signal collected by a single microphone is Y m , specifically calculated by formula three:

[0079]

[0080] Among them, Ym is the time domain speech signal collected by the mth microphone in the microphone array, Xi is the sound signal of the i-th speaker, and N refers to the noise.

[0081] It can be understood that the preset frame sequence length information refers to the continuous frame sequence length information in the time domain speech signal after framing. The feature dimension information input into the preset convolutional recursive neural network model is generated through the channel occupancy information of the amplitude spectrum information, the channel occupancy information of the phase spectrum information and the preset frame sequence length information. For example, the length corresponding to the preset frame sequence length information is 10, the number of channels of the channel occupancy information of the amplitude spectrum information and the channel occupancy information of the phase spectrum information is 4, and the feature dimension information at this time is (8, 10, 257).

[0082] In a specific implementation, the speaker counting program generates corresponding feature dimension information according to the amplitude spectrum information, the phase spectrum information and the preset frame sequence length information.

[0083] Step S30, predicting the feature dimension information according to a preset convolutional recursive neural network model to obtain speech signal probability distribution information.

[0084] It should be understood that after obtaining the feature dimension information, it is necessary to set the loss function to optimize the preset convolutional recurrent neural network model. Only when the preset convolutional recurrent neural network model reaches the optimal value, the prediction of the preset convolutional recurrent neural network model is effective and reliable. The optimal measurement standard for the preset convolutional recurrent neural network model is that the loss function converges. The loss function can only be the cross entropy loss function. The specific expression of the cross entropy loss function is:

[0085]

[0086] Among them, C is the category information of the number of speakers, yc is the label vector information corresponding to category c, and Pc is the category probability information predicted by the preset convolutional recurrent neural network model.

[0087] It can be understood that after the preset convolutional recurrent neural network model reaches the optimal state, the feature dimension information is input into the preset convolutional recurrent neural network model so that the preset convolutional recurrent neural network model predicts the feature dimension information to obtain the corresponding speech probability distribution information. The preset convolutional recurrent neural network model is a neural network model composed of a convolutional neural network (CNN) and a recurrent neural network (RNN). The speech signal probability distribution information refers to the probability distribution information in the sequence output by the preset convolutional recurrent neural network model. For example, the probability distribution information is [0.2, 0.2, 0.4, 0.05, 0.05, 0.1].

[0088] In a specific implementation, the speaker counting program predicts the feature dimension information according to a preset convolutional recursive neural network model to obtain the probability distribution information of the speech signal.

[0089] Step S40: determining the number of speakers in the target area based on the speech signal probability distribution information, so as to count the speakers in the target area.

[0090] It should be understood that after obtaining the speech signal probability distribution information, a value with the maximum speech signal probability is selected from the speech signal probability distribution information, and the subscript of the value is used as the number of speakers in the target area.

[0091] Further, step S40 includes: sorting the speech signal probabilities corresponding to the speech information probability distribution information; obtaining a target probability value according to the sorted speech signal probabilities; and determining the number of speakers in the target area based on the subscript of the target probability value.

[0092] It can be understood that the target probability value refers to the probability value corresponding to the probability distribution information of the speech information. For example, the probability values ​​are 0.2, 0.2, 0.4, 0.05, 0.05, 0.1. The probability values ​​are sorted in ascending order to obtain the sorted probability values, specifically 0.05, 0.05, 0.1, 0.2, 0.2, 0.4. At this time, the target probability value is 0.4, and the subscript corresponding to the target probability value 0.4 is 2, then the number of speakers in the target area is 2.

[0093] In a specific implementation, the speaker counting program determines the number of speakers in the target area based on the speech signal probability distribution information, so as to count the speakers in the target area.

[0094] This embodiment collects multi-channel time-domain speech signals in the target area, obtains corresponding amplitude spectrum information and phase spectrum information according to the time-domain speech signals; generates corresponding feature dimension information according to the amplitude spectrum information, phase spectrum information and preset frame sequence length information; predicts the feature dimension information according to a preset convolutional recurrent neural network model to obtain speech signal probability distribution information; determines the number of speakers in the target area based on the speech signal probability distribution information; obtains speech signal probability distribution information through a preset convolutional recurrent neural network model and feature dimension information, and determines the number of speakers in the target area according to the speech signal probability distribution information, so as to count the speakers in the target area. Compared with the prior art that counts speakers through traditional audio algorithms, this embodiment can effectively improve the accuracy of speaker counting.

[0095] In one embodiment, if Figure 3 Based on the first embodiment, a second embodiment of the speaker counting method based on deep learning of the present invention is proposed, and step S20 includes:

[0096] Step S201, obtaining the quantity information of the target acquisition devices.

[0097] It should be understood that the quantity information refers to the total number of target acquisition devices constituting the acquisition device array. The target acquisition device may be a microphone or other acquisition device. This implementation does not impose any limitation on this and takes the microphone as an example for explanation.

[0098] In a specific implementation, the speaker counting program obtains the quantity information of the target collection devices.

[0099] Step S202: determining channel occupancy information of the amplitude spectrum information according to the quantity information.

[0100] It can be understood that after obtaining the number information of the target acquisition device, the channel information occupied by the amplitude spectrum information is determined according to the number information. Since the amplitude spectrum information is obtained based on the time domain speech signal, and the time domain speech signal is collected by a microphone, the channel information occupied by the amplitude spectrum information is consistent with the number information of the microphone. For example, if the number of microphones is 4, the number of channels occupied by the amplitude spectrum information is also 4.

[0101] In a specific implementation, the speaker counting program determines the channel occupancy information of the amplitude spectrum information according to the quantity information.

[0102] Step S203: determining channel occupancy information of the phase spectrum information according to the quantity information.

[0103] It should be understood that after obtaining the number information of the target acquisition device, the channel information occupied by the phase spectrum information is determined according to the number information. Since the phase spectrum information is obtained based on the time domain speech signal, and the time domain speech signal is collected through a microphone, the channel information occupied by the phase spectrum information is consistent with the number information of the microphone. For example, if the number of microphones is 4, the number of channels occupied by the phase spectrum information is also 4.

[0104] In a specific implementation, the speaker counting program determines the channel occupancy information of the phase spectrum information according to the quantity information.

[0105] Step S204: obtaining preset continuous frame sequence information, and obtaining preset frame sequence length information according to the preset continuous frame sequence information.

[0106] It can be understood that the preset continuous frame sequence information refers to the continuous frame sequence information in the time domain speech signal after framing. The corresponding preset frame sequence length information is obtained according to the preset continuous frame sequence information. For example, if the continuous frame sequence corresponding to the preset continuous frame sequence information is 10, then the frame sequence length corresponding to the frame sequence length information is also set to 10.

[0107] Further, step S204 includes: obtaining the number of target speakers supported in the preset convolutional recurrent neural network model; obtaining corresponding category information according to the number of target speakers; determining the format information of the label vector according to the preset vector label information and the category information; detecting the time domain speech signal according to the preset speech detection strategy and the format information of the label vector to obtain speech frame information and non-speech frame information; obtaining preset continuous frame sequence information, and obtaining preset frame sequence length information according to the preset continuous frame sequence information, speech frame information and non-speech frame information.

[0108] It should be understood that the target number of speakers refers to the maximum number of speakers supported by the preset convolutional recurrent neural network model, the category information refers to the category information representing the number of speakers, for example, category 0 represents no speaker, category 1 represents one speaker... and so on, category 5 represents 5 speakers, and the preset label format information refers to a one-hot vector label, which indicates a vector in which only one element is 1 and the rest are 0, for example, [0,1,0,0,0,0].

[0109] It can be understood that the preset voice detection strategy refers to a strategy for detecting voice information. The preset voice detection strategy can detect the voice frame information and non-voice frame information in the voice information. The preset voice detection strategy can be a voice activation detection (Voice Activation Detection, VAD) strategy, or other detection strategies. This embodiment does not limit this. The VAD strategy is used as an example for explanation. After obtaining the voice frame information and non-voice frame information, the preset frame sequence length information is obtained according to the voice frame information, the non-voice frame information and the preset continuous frame sequence information.

[0110] In a specific implementation, the speaker counting program obtains preset continuous frame sequence information, and obtains preset frame sequence length information according to the preset continuous frame sequence information.

[0111] Step S205 , generating corresponding feature dimension information based on the channel occupancy information of the amplitude spectrum information, the channel occupancy information of the phase spectrum information, and the preset frame sequence length information.

[0112] It should be understood that after obtaining the channel occupancy information of the amplitude spectrum information, the channel occupancy information of the phase spectrum information and the preset frame sequence length information, the characteristic dimension information is generated according to the channel occupancy information of the amplitude spectrum information, the channel occupancy information of the phase spectrum information and the preset frame sequence length information. For example, the preset frame sequence length is 10, the channel occupancy information of the amplitude spectrum information is 4 channels, and the channel occupancy information of the phase spectrum information is 4 channels. By splicing them, the characteristic dimension information obtained is (8, 10, 257).

[0113] In a specific implementation, the speaker counting program generates corresponding feature dimension information based on the channel occupancy information of the amplitude spectrum information, the channel occupancy information of the phase spectrum information, and the preset frame sequence length information.

[0114] This embodiment obtains the number information of the target acquisition device; determines the channel occupancy information of the amplitude spectrum information according to the number information; determines the channel occupancy information of the phase spectrum information according to the number information; obtains the preset continuous frame sequence information, and obtains the preset frame sequence length information according to the preset continuous frame sequence information; generates corresponding feature dimension information based on the channel occupancy information of the amplitude spectrum information, the channel occupancy information of the phase spectrum information, and the preset frame sequence length information; obtains the channel occupancy information of the amplitude spectrum information and the phase spectrum information through the number information of the target acquisition device, obtains the preset frame sequence length information according to the preset continuous frame sequence information, and generates corresponding feature dimension information based on the channel occupancy information of the amplitude spectrum information, the channel occupancy information of the phase spectrum information, and the preset frame sequence length information, thereby effectively improving the accuracy of obtaining the feature dimension information.

[0115] In one embodiment, if Figure 4 Based on the first embodiment, a third embodiment of the method for counting speakers based on deep learning of the present invention is proposed. Step S30 includes:

[0116] Step S301, extracting convolutional neural network information and recurrent recurrent neural network information from the preset convolutional recurrent neural network model.

[0117] It can be understood that the convolutional neural network information includes 4 convolutional layers and 2 maximum pooling layers (max-pooling). The convolutional layers all use 2D convolution, the convolution kernel size is 3x3, and the number of convolution channels from top to bottom are 64 channels, 32 channels, 128 channels, and 64 channels. After each convolution, zero padding is used to maintain the features at the size specified on the right side of the figure. The activation function after convolution uses the linear rectification function (Rectified Linear Unit, ReLU), and the maximum pooling layer uses a kernel size of 1x3.

[0118] It should be understood that the recurrent neural network information consists of a long short-term memory recurrent neural network (LSTM RNN) with 40 units for modeling the correlation between time series, and its activation function is the hyperbolic tangent function (Tanh).

[0119] In a specific implementation, the speaker counting program extracts convolutional neural network information and recurrent recurrent neural network information in the preset convolutional recurrent neural network model.

[0120] Step S302: convolve the feature dimension information according to the convolutional neural network information.

[0121] It can be understood that after obtaining the feature dimension information, the feature dimension information is convolved through the convolution layer in the convolutional neural network information, so that the feature dimension information becomes feature dimension information of smaller units.

[0122] In a specific implementation, the speaker counting program convolves the feature dimension information according to the convolutional neural network information.

[0123] Step S303, predicting the convolved feature dimension information according to the cyclic recursive neural network information to obtain speech signal probability distribution information.

[0124] It should be understood that there are four different gating units in the LSTM RNN in the recurrent neural network information. Since the middle and front sequences of the feature dimension time series sequence corresponding to the feature dimension information after convolution will affect the prediction results of the latter sequence, the LSTM RNN only needs to predict the final time series sequence when training the feature dimension information after convolution. After the prediction is completed, the probability distribution information of the speech signal is obtained.

[0125] Furthermore, step S303 includes: extracting long short-term memory recurrent neural network information from the recurrent neural network information; determining a target prediction strategy based on the long short-term memory recurrent neural network information; determining a corresponding feature dimension time series sequence based on the convolved feature dimension information; and predicting the feature dimension time series sequence based on the target prediction strategy to obtain speech signal probability distribution information.

[0126] It can be understood that the prediction strategy refers to the prediction strategy determined according to the gating unit of the LSTM RNN, that is, predicting the final time series sequence in the feature dimension information after convolution. After obtaining the feature dimension information after convolution, the corresponding feature dimension time series sequence is determined according to the feature dimension information after convolution. By predicting the feature dimension time series sequence through the prediction strategy, the probability distribution information of the speech signal can be obtained.

[0127] In a specific implementation, the speaker counting program predicts the convolved feature dimension information according to the recurrent neural network information to obtain the speech signal probability distribution information.

[0128] This embodiment extracts convolutional neural network information and recurrent recurrent neural network information from the preset convolutional recurrent neural network model; convolves the feature dimension information according to the convolutional neural network information; predicts the convolved feature dimension information according to the recurrent recurrent neural network information to obtain speech signal probability distribution information; convolves the feature dimension information with the convolutional neural network information, and predicts the convolved feature dimension information according to the recurrent recurrent neural network information to obtain speech signal probability distribution information, thereby effectively improving the accuracy of predicting speech signal probability distribution information.

[0129] In addition, an embodiment of the present invention further proposes a storage medium, on which a deep learning-based speaker counting program is stored. When the deep learning-based speaker counting program is executed by a processor, the steps of the deep learning-based speaker counting method described above are implemented.

[0130] Since the storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here.

[0131] In addition, refer to Figure 5 The embodiment of the present invention further provides a speaker counting device based on deep learning, and the speaker counting device based on deep learning includes:

[0132] The acquisition module 10 is used to acquire multi-channel time-domain speech signals in the target area, and obtain corresponding amplitude spectrum information and phase spectrum information according to the time-domain speech signals.

[0133] It should be understood that the time domain speech signal refers to the short-term unchanged speech signal in the target area and within the preset time. The multi-channel time domain speech signal is collected by a microphone array arranged in the target area, and the frequency of each microphone in the microphone array is 16kHZ.

[0134] It can be understood that the amplitude spectrum information refers to the information presented by the characteristics of the size of the speech energy in the time domain speech signal. In judging the target area, judging the number of speakers through the time domain speech signal needs to rely on the speech energy in the time domain speech signal. The phase spectrum information refers to the information presented by the characteristics of the spatial information of each sound source obtained from the arrival delay and sampling offset.

[0135] Furthermore, the acquisition module 10 is also used to acquire multi-channel time domain speech signals in the target area; to frame the time domain speech signals; to perform Fourier transform on the framed time domain speech signals; to obtain target frequency domain information according to preset sampling rules and the time domain speech signals after Fourier transform; and to determine corresponding amplitude spectrum information and phase spectrum information according to the target frequency domain information.

[0136] It can be understood that after obtaining the multi-channel time domain speech signal in the target area, the time domain speech signal of each channel is framed respectively. Framing refers to dividing the speech frame corresponding to the time domain speech signal into unit frames, and then Fourier transforming the time domain speech signal after framing. Fourier transforming refers to converting the form of the time domain speech signal. Fourier transforming includes continuous Fourier transforming and discrete Fourier transforming. After the conversion is completed, the target frequency domain information is obtained by the preset sampling rules and the time domain speech signal after Fourier transforming. The preset sampling rules refer to the rules for sampling the target samples. The preset sampling rules are used to sample the time domain speech signal after Fourier transforming to obtain the target frequency domain information, and the amplitude spectrum information and phase spectrum information in the target frequency information are extracted. For example, 512 points are taken in the time domain speech signal after Fourier transforming, and the overlap rate is 50%. At this time, the frequency domain size in the target frequency domain information is 257.

[0137] It should be understood that the amplitude spectrum information is obtained by calculating the target spectrum information through formula one, specifically:

[0138]

[0139] Among them, |A m | is the amplitude spectrum information, F m re is the real part of the frequency domain information corresponding to the signal collected by the mth microphone, F m im The imaginary part of the frequency domain information corresponding to the signal collected by the mth microphone.

[0140] It can be understood that the phase spectrum information is obtained by calculating the target spectrum information through Formula 2, which is specifically:

[0141]

[0142] in, <P m is the phase spectrum information, F m re is the real part of the frequency domain information corresponding to the signal collected by the mth microphone, F m im The imaginary part of the frequency domain information corresponding to the signal collected by the mth microphone.

[0143] In a specific implementation, the speaker counting program collects multi-channel time-domain speech signals in a target area, and obtains corresponding amplitude spectrum information and phase spectrum information according to the time-domain speech signals.

[0144] The generating module 20 is used to generate corresponding feature dimension information according to the amplitude spectrum information, the phase spectrum information and the preset frame sequence length information.

[0145] It should be understood that after obtaining the amplitude spectrum information and the phase spectrum information, it is necessary to determine the number of devices for collecting the time domain speech signal, and determine the number of channels occupied by the amplitude spectrum information and the phase spectrum information through the number of devices. The channel information occupied by the amplitude spectrum information and the phase spectrum information is consistent. For example, if the number of microphones is 4, then the number of channels occupied by the amplitude spectrum information and the phase spectrum information is also 4. The time domain speech signal collected by a single microphone is Y m , specifically calculated by formula three:

[0146]

[0147] Among them, Ym is the time domain speech signal collected by the mth microphone in the microphone array, Xi is the sound signal of the i-th speaker, and N refers to the noise.

[0148] It can be understood that the preset frame sequence length information refers to the continuous frame sequence length information in the time domain speech signal after framing. The feature dimension information input into the preset convolutional recursive neural network model is generated through the channel occupancy information of the amplitude spectrum information, the channel occupancy information of the phase spectrum information and the preset frame sequence length information. For example, the length corresponding to the preset frame sequence length information is 10, the number of channels of the channel occupancy information of the amplitude spectrum information and the channel occupancy information of the phase spectrum information is 4, and the feature dimension information at this time is (8, 10, 257).

[0149] In a specific implementation, the speaker counting program generates corresponding feature dimension information according to the amplitude spectrum information, the phase spectrum information and the preset frame sequence length information.

[0150] The prediction module 30 is used to predict the feature dimension information according to a preset convolutional recursive neural network model to obtain the speech signal probability distribution information.

[0151] It should be understood that after obtaining the feature dimension information, it is necessary to set the loss function to optimize the preset convolutional recurrent neural network model. Only when the preset convolutional recurrent neural network model reaches the optimal value, the prediction of the preset convolutional recurrent neural network model is effective and reliable. The optimal measurement standard for the preset convolutional recurrent neural network model is that the loss function converges. The loss function can only be the cross entropy loss function. The specific expression of the cross entropy loss function is:

[0152]

[0153] Among them, C is the category information of the number of speakers, yc is the label vector information corresponding to category c, and Pc is the category probability information predicted by the preset convolutional recurrent neural network model.

[0154] It can be understood that after the preset convolutional recurrent neural network model reaches the optimal state, the feature dimension information is input into the preset convolutional recurrent neural network model so that the preset convolutional recurrent neural network model predicts the feature dimension information to obtain the corresponding speech probability distribution information. The preset convolutional recurrent neural network model is a neural network model composed of a convolutional neural network (CNN) and a recurrent neural network (RNN). The speech signal probability distribution information refers to the probability distribution information in the sequence output by the preset convolutional recurrent neural network model. For example, the probability distribution information is [0.2, 0.2, 0.4, 0.05, 0.05, 0.1].

[0155] In a specific implementation, the speaker counting program predicts the feature dimension information according to a preset convolutional recursive neural network model to obtain the probability distribution information of the speech signal.

[0156] The counting module 40 is used to determine the number of speakers in the target area based on the speech signal probability distribution information, so as to count the speakers in the target area.

[0157] It should be understood that after obtaining the speech signal probability distribution information, a value with the maximum speech signal probability is selected from the speech signal probability distribution information, and the subscript of the value is used as the number of speakers in the target area.

[0158] Furthermore, the counting module 40 is further configured to sort the speech signal probabilities corresponding to the speech information probability distribution information; obtain a target probability value according to the sorted speech signal probabilities; and determine the number of speakers in the target area based on the subscript of the target probability value.

[0159] It can be understood that the target probability value refers to the probability value corresponding to the probability distribution information of the speech information. For example, the probability values ​​are 0.2, 0.2, 0.4, 0.05, 0.05, 0.1. The probability values ​​are sorted in ascending order to obtain the sorted probability values, specifically 0.05, 0.05, 0.1, 0.2, 0.2, 0.4. At this time, the target probability value is 0.4, and the subscript corresponding to the target probability value 0.4 is 2, then the number of speakers in the target area is 2.

[0160] In a specific implementation, the speaker counting program determines the number of speakers in the target area based on the speech signal probability distribution information, so as to count the speakers in the target area.

[0161] This embodiment collects multi-channel time-domain speech signals in the target area, obtains corresponding amplitude spectrum information and phase spectrum information according to the time-domain speech signals; generates corresponding feature dimension information according to the amplitude spectrum information, phase spectrum information and preset frame sequence length information; predicts the feature dimension information according to a preset convolutional recurrent neural network model to obtain speech signal probability distribution information; determines the number of speakers in the target area based on the speech signal probability distribution information; obtains speech signal probability distribution information through a preset convolutional recurrent neural network model and feature dimension information, and determines the number of speakers in the target area according to the speech signal probability distribution information, so as to count the speakers in the target area. Compared with the prior art that counts speakers through traditional audio algorithms, this embodiment can effectively improve the accuracy of speaker counting.

[0162] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0163] In addition, for technical details not described in detail in this embodiment, reference can be made to the deep learning-based speaker counting method provided in any embodiment of the present invention, which will not be described again here.

[0164] In one embodiment, the acquisition module 10 is also used to acquire multi-channel time domain speech signals in the target area; to frame the time domain speech signals; to perform Fourier transform on the time domain speech signals after framing; to obtain target frequency domain information according to preset sampling rules and the time domain speech signals after Fourier transform; and to determine corresponding amplitude spectrum information and phase spectrum information according to the target frequency domain information.

[0165] In one embodiment, the generation module 20 is also used to obtain the quantity information of the target acquisition device; determine the channel occupancy information of the amplitude spectrum information based on the quantity information; determine the channel occupancy information of the phase spectrum information based on the quantity information; obtain preset continuous frame sequence information, and obtain preset frame sequence length information based on the preset continuous frame sequence information; generate corresponding feature dimension information based on the channel occupancy information of the amplitude spectrum information, the channel occupancy information of the phase spectrum information and the preset frame sequence length information.

[0166] In one embodiment, the generation module 20 is also used to obtain the number of target speakers supported in a preset convolutional recurrent neural network model; obtain corresponding category information based on the number of target speakers; determine the format information of the label vector based on the preset vector label information and the category information; detect the time domain speech signal based on the preset speech detection strategy and the format information of the label vector to obtain speech frame information and non-speech frame information; obtain preset continuous frame sequence information, and obtain preset frame sequence length information based on the preset continuous frame sequence information, speech frame information and non-speech frame information.

[0167] In one embodiment, the prediction module 30 is also used to extract convolutional neural network information and recurrent recurrent neural network information in the preset convolutional recurrent neural network model; convolve the feature dimension information according to the convolutional neural network information; predict the convolved feature dimension information according to the recurrent recurrent neural network information to obtain speech signal probability distribution information.

[0168] In one embodiment, the prediction module 30 is also used to extract long short-term memory recurrent neural network information from the recurrent neural network information; determine a target prediction strategy based on the long short-term memory recurrent neural network information; determine a corresponding feature dimension time series sequence based on the feature dimension information after convolution; and predict the feature dimension time series sequence based on the target prediction strategy to obtain speech signal probability distribution information.

[0169] In one embodiment, the counting module 40 is further used to sort the speech signal probabilities corresponding to the speech information probability distribution information; obtain a target probability value according to the sorted speech signal probabilities; and determine the number of speakers in the target area based on the subscript of the target probability value.

[0170] Other embodiments or implementation methods of the speaker counting device based on deep learning of the present invention can refer to the above-mentioned method embodiments, which are not repeated here.

[0171] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0172] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a speaker counting program, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0174] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A speaker counting method based on deep learning, characterized in that: The speaker counting method based on deep learning comprises the following steps: Collecting multi-channel time-domain speech signals in the target area, and obtaining corresponding amplitude spectrum information and phase spectrum information according to the time-domain speech signals; Generate corresponding feature dimension information according to the amplitude spectrum information, the phase spectrum information and the preset frame sequence length information, wherein the preset frame sequence length is the continuous frame sequence length information in the time domain speech signal after framing; Predicting the feature dimension information according to a preset convolutional recursive neural network model to obtain speech signal probability distribution information; The number of speakers in the target area is determined based on the speech signal probability distribution information, so as to count the speakers in the target area.

2. The method for counting speakers based on deep learning as claimed in claim 1, characterized in that: The collecting of multi-channel time-domain speech signals in the target area and obtaining corresponding amplitude spectrum information and phase spectrum information according to the time-domain speech signals include: Collect multi-channel time-domain speech signals in the target area; Framing the time-domain speech signal; Performing Fourier transform on the framed time-domain speech signal; Obtain target frequency domain information according to a preset sampling rule and a Fourier transformed time domain speech signal; Corresponding amplitude spectrum information and phase spectrum information are determined according to the target frequency domain information.

3. The method for counting speakers based on deep learning according to claim 1, wherein: The generating corresponding feature dimension information according to the amplitude spectrum information, the phase spectrum information and the preset frame sequence length information includes: Obtain the quantity information of the target acquisition devices; Determine channel occupancy information of the amplitude spectrum information according to the quantity information; Determine channel occupancy information of the phase spectrum information according to the quantity information; Acquire preset continuous frame sequence information, and obtain preset frame sequence length information according to the preset continuous frame sequence information; Corresponding feature dimension information is generated based on the channel occupancy information of the amplitude spectrum information, the channel occupancy information of the phase spectrum information, and the preset frame sequence length information.

4. The method for counting speakers based on deep learning as claimed in claim 3, characterized in that: The obtaining of preset continuous frame sequence information and obtaining preset frame sequence length information according to the preset continuous frame sequence information includes: Get the number of target speakers supported by the preset convolutional recurrent neural network model; Obtaining corresponding category information according to the number of target speakers; Determining the format information of the label vector according to the preset vector label information and the category information; Detecting the time domain speech signal according to a preset speech detection strategy and the format information of the label vector to obtain speech frame information and non-speech frame information; Preset continuous frame sequence information is acquired, and preset frame sequence length information is obtained according to the preset continuous frame sequence information, voice frame information and non-voice frame information.

5. The method for counting speakers based on deep learning according to claim 1, wherein: The predicting of the feature dimension information according to the preset convolutional recursive neural network model to obtain the speech signal probability distribution information includes: Extracting convolutional neural network information and recurrent recurrent neural network information from the preset convolutional recurrent neural network model; Convolving the feature dimension information according to the convolutional neural network information; The convolved feature dimension information is predicted according to the cyclic recursive neural network information to obtain the speech signal probability distribution information.

6. The method for counting speakers based on deep learning as claimed in claim 5, characterized in that: The step of predicting the convolved feature dimension information according to the cyclic recursive neural network information to obtain the speech signal probability distribution information includes: Extracting long short-term memory recurrent neural network information from the recurrent recurrent neural network information; Determining a target prediction strategy based on the long short-term memory recurrent neural network information; Determine a corresponding feature dimension time series sequence according to the feature dimension information after the convolution; The feature dimension time series sequence is predicted based on the target prediction strategy to obtain speech signal probability distribution information.

7. The method for counting speakers based on deep learning according to any one of claims 1 to 6, characterized in that: Determining the number of speakers in the target area based on the speech signal probability distribution information includes: sorting the speech signal probabilities corresponding to the speech signal probability distribution information; Obtain a target probability value according to the sorted speech signal probability; The number of speakers in the target area is determined based on the subscript of the target probability value.

8. A speaker counting device based on deep learning, characterized in that: The speaker counting device based on deep learning comprises: An acquisition module is used to acquire multi-channel time-domain speech signals in a target area, and obtain corresponding amplitude spectrum information and phase spectrum information according to the time-domain speech signals; A generating module, used to generate corresponding feature dimension information according to the amplitude spectrum information, the phase spectrum information and the preset frame sequence length information, wherein the preset frame sequence length is the continuous frame sequence length information in the time domain speech signal after framing; A prediction module, used to predict the feature dimension information according to a preset convolutional recursive neural network model to obtain speech signal probability distribution information; The counting module is used to determine the number of speakers in the target area based on the speech signal probability distribution information, so as to count the speakers in the target area.

9. A speaker counting device based on deep learning, characterized in that: The deep learning-based speaker counting device comprises: a memory, a processor, and a deep learning-based speaker counting program stored in the memory and executable on the processor, wherein the deep learning-based speaker counting program is configured to implement the deep learning-based speaker counting method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a deep learning-based speaker counting program, and when the deep learning-based speaker counting program is executed by the processor, the deep learning-based speaker counting method according to any one of claims 1 to 7 is implemented.

Images