Method and system for voice separation during interrogation

Abstract

The invention provides a voice separation method and system during interrogation, and belongs to the technical field of audio processing, and the method comprises the steps: constructing an audio training signal set, and determining a dynamic alternating time period; obtaining an initial audio signal during interrogation, and importing the initial audio signal according to a time sequence; respectively performing framing windowing processing on the initial audio signal and the audio training signal set to obtain windowed signals, and respectively extracting dynamic alternating features; constructing a long-short-term memory network recognition model, and training by taking the audio training signal set as input and the dynamic alternating features as output to obtain a trained long-short-term memory network recognition model; identifying the initial audio signal to obtain all dynamic alternating features; and splitting the initial audio signal according to the identified time sequence information of the dynamic alternating characteristics, and alternately splicing to obtain two groups of interrogation voice separation information. For single mixed voice information, interrogation voice information and interrogated voice information can be efficiently separated.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to a method and system for voice separation during interrogation. Background Art

[0002] Currently, manual recording is commonly used in judicial settings, such as criminal interrogations. This method is inefficient and consumes significant manpower and resources. Due to the unique constraints of interrogation environments, microphone recording often captures the voices of multiple speakers, making it difficult to effectively distinguish between different speakers by directly identifying the recorded signal. Furthermore, the voice of the interrogator is often much quieter than that of the interrogator, sometimes even significantly so. Therefore, most interrogation recordings still rely on manual methods.

[0003] This situation not only leads to inefficient interrogation recording but also compromises its accuracy and fairness. During manual recording, recorders may inadvertently omit or misinterpret key statements from the interrogator, potentially affecting the judgment and outcome of the case. Furthermore, long hours of manual recording are extremely arduous and stressful for recorders, leading to fatigue and further compromising the quality of the recording.

[0004] To improve this situation, existing technologies use speech recognition technology to automatically transcribe interrogation sessions, optimize algorithms to improve the speaker's voice distinction, or enhance microphone sensitivity, particularly its ability to capture quieter sounds. However, these existing technologies are still unable to accurately separate the interrogator's audio from the interrogated's, and can only achieve separation by enhancing a single sound source. Therefore, an efficient sound source separation method is urgently needed to address this issue. Summary of the Invention

[0005] To solve the above problems, the present invention provides a method and system for voice separation during interrogation. The method is capable of efficiently separating the interrogating voice information and the voice information of the person being interrogated, targeting a single mixed voice information.

[0006] To achieve the above objectives, the present invention provides the following technical solutions.

[0007] A method for speech separation during interrogation, comprising the following steps:

[0008] Constructing an audio training signal set, determining multiple audio dynamic alternation time periods of each audio training signal in the audio training signal set; obtaining an initial audio signal during the interrogation, and importing the initial audio signal according to the time sequence;

[0009] Performing frame-by-frame windowing processing on the initial audio signal and the audio training signal set to obtain an interrogation windowed speech signal and a training windowed speech signal;

[0010] Extracting dynamic alternation features of interrogation windowed speech signals and training windowed speech signals respectively;

[0011] Construct a long short-term memory network recognition model, take the audio training signal set as input, and train the dynamic alternation features with time series information as output to obtain the trained long short-term memory network recognition model;

[0012] The initial audio signal is recognized through the trained long short-term memory network recognition model to obtain all dynamic alternation features;

[0013] According to the time series information of the identified dynamic alternating features, the initial audio signal is split and spliced alternately to obtain two sets of interrogation speech separation information.

[0014] Preferably, the method further includes performing silence removal on the initial audio signal before the initial audio signal is framed and windowed, comprising the following steps:

[0015] Determine the threshold for sound intensity;

[0016] The initial audio signal is segmented into fixed-length time segments, the initial audio signal is scanned in sequence, and the volume of each time segment in the audio signal is analyzed. When the signal volume is lower than this threshold, the initial audio signal in this time segment is considered to be a blank signal or silence;

[0017] A plurality of time periods below a threshold are determined, removed from the initial audio signal, and the initial audio signal is rejoined to obtain an initial audio signal after silence removal.

[0018] Preferably, the extracting of dynamic alternating features of the interrogation windowed speech signal and the training windowed speech signal respectively comprises the following steps:

[0019] Each window of the interrogation windowed speech signal and the training windowed speech signal is subjected to time-frequency decomposition using short-time Fourier transform to obtain spectrum information; the spectrum information of each window is synthesized into a two-dimensional time-frequency signal of the interrogation speech and a two-dimensional time-frequency signal of the training speech in chronological order;

[0020] According to multiple audio dynamic alternation time periods, continuous spectrum envelopes of the training speech time-frequency two-dimensional signal in the time period are extracted to form long vector features as dynamic alternation features.

[0021] Preferably, the performing frame-by-frame windowing processing on the initial audio signal and the audio training signal set respectively comprises the following steps:

[0022] Split the continuous initial audio signal and the audio training signal in the audio training signal set into fixed-length frames, with an overlap of 50% between adjacent frames;

[0023] The initial audio signal and audio training signal after framing are windowed, as shown in the following formula:

[0024] x0(n)=x(n)×w(n)

[0025] Among them, x(n) is the initial speech signal; w(n) is the window function, which is a Hamming window function; x0(n) represents the windowed speech signal; the window function is a Hamming window function, and the frame length is 20ms-40ms.

[0026] Preferably, the time-frequency decomposition is performed on each window of the interrogation windowed speech signal and the training windowed speech signal using short-time Fourier transform, as shown in the following formula:

[0027]

[0028] Where X(t,f) is the time-frequency two-dimensional signal; t is the time frame; f is the frequency, w(·) is the Hamming window function; and x(k) is the window signal.

[0029] Preferably, the construction of the dynamic alternation feature comprises the following steps:

[0030] Extract the spectrum of each frame within the target time period of the training speech time-frequency two-dimensional signal, take the logarithm of the spectrum amplitude, and then smooth the logarithm through a low-pass filter to obtain the smoothed spectrum;

[0031] Detect the significant peaks of the smoothed spectrum and determine the energy distribution of the spectrum;

[0032] Through the spline interpolation method, interpolation is performed between the significant peaks of the spectrum to form a continuous envelope, which constitutes the spectrum envelope of each frame;

[0033] Based on the spectrum envelope of each frame, the continuous spectrum envelope of the time period is obtained by spline interpolation;

[0034] The continuous spectrum envelope is converted into a long vector form to form a dynamic alternating feature, and all the dynamic alternating features are normalized.

[0035] Preferably, the long short-term memory network recognition model includes an input convolutional layer, two LSTM layers, an intermediate fully connected layer, a BN layer and an output fully connected layer connected in sequence; the intermediate fully connected layer adopts a ReLU activation function, and the output fully connected layer adopts a Softmax activation function.

[0036] Preferably, the method of splitting the initial audio signal according to the time series information of the identified dynamic alternating features and alternately splicing the signals to obtain two sets of interrogation speech separation information comprises the following steps:

[0037] Determining multiple time series information of the identified dynamic alternating features;

[0038] Split the initial audio signal according to multiple time series information, and remove the audio signal with dynamic alternating features;

[0039] The segmented initial audio signals after dynamic alternating features are removed are re-alternated and spliced to obtain two sets of interrogation speech separation information, namely interrogation speech information and interrogated speech information.

[0040] A system for speech separation during interrogation, comprising:

[0041] processor;

[0042] a memory having stored thereon a computer program executable on the processor;

[0043] Wherein, when the computer program is executed by the processor, the steps of the method for speech separation during interrogation are implemented.

[0044] A computer-readable storage medium stores a data processing program, which, when executed by a processor, implements the steps of the method for voice separation during interrogation.

[0045] Beneficial effects of the present invention:

[0046] The present invention proposes a method and system for voice separation during interrogation. The method targets the particularity of interrogation audio information, namely, that it is composed of alternating interrogation and interrogated sound sources. By identifying the dynamic alternation characteristics of the initial audio signal during interrogation, the audio signal is disassembled and eliminated, and then re-spliced to obtain interrogation voice information and interrogated voice information, thereby realizing automatic separation of the audio signal, effectively improving the automatic voice separation capability and efficiency, quickly collecting interrogation information and interrogated information, and subsequently converting them into text to record the interrogation content, completely retaining the key statements of the interrogated person, and improving the quality of interrogation records. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is an overall flow chart of an embodiment of the present invention;

[0048] Figure 2 is a flow chart of a silent elimination step according to an embodiment of the present invention;

[0049] Figure 3 is a flow chart of the steps for obtaining dynamic alternating features according to an embodiment of the present invention;

[0050] Figure 4 This is a flow chart of constructing a dynamic alternating feature based on the spectrum envelope according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] Example 1

[0053] In the traditional manual recording of interrogations, the recorder may inadvertently omit or misunderstand certain key statements of the person being interrogated, thus affecting the judgment and outcome of the case. In addition, long periods of manual recording are an extremely heavy and stressful task for the recorder, which can easily lead to fatigue, further affecting the quality of the recording. To improve this situation, the existing technology uses speech recognition technology to automatically transcribe the interrogation process, and optimizes the algorithm to improve the distinction of the speaker's voice, or enhance the sensitivity of the microphone, especially the ability to capture smaller sounds. However, the above-mentioned existing technology is still unable to accurately separate the audio information of the interrogator and the audio information of the person being interrogated, and can only enhance a single sound source to achieve the purpose of separation. There is an urgent need for an efficient sound source separation method to solve the above problem. In view of the above content, this embodiment proposes a method for speech separation during interrogation, such as Figure 1 As shown, Figure 1 The flowchart of the method for speech separation during interrogation specifically includes the following steps:

[0054] S1: Construct an audio training signal set, determine multiple audio dynamic alternation time periods of each audio training signal in the audio training signal set; obtain the initial audio signal during the interrogation, and import the initial audio signal according to the time sequence.

[0055] S2: performing frame-by-frame windowing processing on the initial audio signal and the audio training signal set respectively to obtain an interrogation windowed speech signal and a training windowed speech signal.

[0056] S3: Extract the dynamic alternation features of the interrogation windowed speech signal and the training windowed speech signal respectively.

[0057] S4: Construct a long short-term memory network recognition model, take the audio training signal set as input, and train the dynamic alternating features with time series information as output to obtain the trained long short-term memory network recognition model.

[0058] S5: The initial audio signal is recognized through the trained long short-term memory network recognition model to obtain all dynamic alternation features.

[0059] S6: Based on the time series information of the identified dynamic alternating features, the initial audio signal is split and alternately spliced to obtain two sets of interrogation speech separation information.

[0060] Among them, the separation step specifically includes: determining multiple time series information of the identified dynamic alternating features; splitting the audio from the initial audio signal according to the multiple time series information, and eliminating the audio signal where the dynamic alternating features are located; re-alternatingly splicing the segmented initial audio signal after the dynamic alternating features are eliminated to obtain two sets of interrogation voice separation information, namely, interrogation voice information and interrogated voice information.

[0061] Furthermore, before performing the initial audio signal frame segmentation and windowing processing, the signal is silenced (the training set is preset to have no silence part, so there is no need to perform silence removal), such as Figure 2 As shown, the specific steps include:

[0062] S1.1: Determine the threshold value for sound intensity.

[0063] S1.2: The initial audio signal is segmented into fixed-length time segments, and the initial audio signal is scanned sequentially. The volume of each time segment in the audio signal is analyzed. When the signal volume is lower than a threshold, the initial audio signal in that time segment is considered to be a blank signal or silence.

[0064] S1.3: Determine multiple time periods below a threshold, remove them from the initial audio signal, and reconstruct the initial audio signal to obtain an initial audio signal after silence removal.

[0065] The aforementioned silence removal method optimizes the entire audio stream by removing empty signals and silent sections. This technique effectively reduces the risk of misinterpretation of dynamic alternating features caused by background silence during speech separation. Dynamic alternating features refer to the sudden changes in intensity and pitch within a continuous speech stream. These changes are more likely to be misinterpreted in speech signals containing intervals of silence, leading to reduced performance of the speech recognition system. By precisely removing these information-free silences, the speech recognition system not only reduces the amount of data processed but also improves its efficiency and responsiveness. Furthermore, this processing method helps reduce potential decoding errors, such as misinterpreting silence as the start or end of speech, which is crucial for improving speech recognition accuracy. Furthermore, the application of silence removal improves the signal-to-noise ratio (SNR) of the speech signal—that is, the ratio between the useful signal and background noise—making speech clearer and easier to understand. This technique is particularly important in multi-speaker environments, helping to distinguish and recognize speech from different speakers, especially during brief pauses between speakers.

[0066] Dynamic alternating features refer to the dynamic changes of signals in the two-dimensional space of time and frequency, that is, the joint changes of different signal sources or events in time and frequency. In this case, different signal sources may appear with different frequency distributions in different time periods, and thus can be used as key features for signal source separation. The acquisition of dynamic alternating features in S3 of this embodiment is as follows: Figure 3 As shown, the specific steps include:

[0067] S3.1: Perform time-frequency decomposition on each window of the interrogation windowed speech signal and the training windowed speech signal using short-time Fourier transform to obtain spectrum information:

[0068] S3.1.1: Split the continuous initial audio signal and the audio training signal in the audio training signal set into fixed-length frames, with an overlap of 50% between adjacent frames.

[0069] S3.1.2: Window the initial audio signal and the audio training signal after framing, as shown in the following formula:

[0070] x0(n)=x(n)×w(n)

[0071] Among them, x(n) is the initial speech signal; w(n) is the window function, which is a Hamming window function; x0(n) represents the windowed speech signal; the window function is a Hamming window function, and the frame length is 20ms-40ms.

[0072] S3.1.3: Use short-time Fourier transform to perform time-frequency decomposition on each window of the interrogation windowed speech signal and the training windowed speech signal, as shown in the following formula:

[0073]

[0074] Where X(t,f) is the time-frequency two-dimensional signal; t is the time frame; f is the frequency, w(·) is the Hamming window function; and x(k) is the window signal.

[0075] The short-time Fourier transform (STFT) generates a two-dimensional time-frequency signal. The horizontal axis represents time, the vertical axis represents frequency, and the color or brightness represents the intensity or energy of the spectrum. This representation helps analyze signal variations in time and frequency and is important for feature extraction and spectral analysis in speech signal processing.

[0076] S3.2: Synthesize the spectrum information of each window into a two-dimensional time-frequency signal of the interrogation speech and a two-dimensional time-frequency signal of the training speech in time sequence.

[0077] S3.3: According to multiple audio dynamic alternation time periods, extract the continuous spectrum envelope of the training speech time-frequency two-dimensional signal in the time period to form a long vector feature as a dynamic alternation feature.

[0078] The spectrum envelope is a curve showing the maximum amplitude of each frequency component in the signal spectrum, and is used to describe the energy distribution characteristics of the signal in the frequency domain. Specifically, the spectrum envelope describes the general picture of how the amplitude of each frequency component in the signal changes with frequency. Therefore, in this embodiment, the long vector feature of the spectrum envelope is used as a dynamic alternating feature, such as Figure 4 As shown, the specific steps include:

[0079] S3.3.1: Extract the spectrum of each frame within the target time period of the training speech time-frequency two-dimensional signal, take the logarithm of the spectrum amplitude, and then smooth the logarithm through a low-pass filter to obtain the smoothed spectrum.

[0080] S3.3.2: Detect significant peaks in the smoothed spectrum and determine the energy distribution of the spectrum.

[0081] S3.3.3: Use the spline interpolation method to interpolate between significant peaks of the spectrum to form a continuous envelope, which constitutes the spectrum envelope of each frame.

[0082] S3.3.4: Based on the spectrum envelope of each frame, obtain the continuous spectrum envelope of the time period by spline interpolation.

[0083] S3.3.5: Convert the continuous spectrum envelope into a long vector form to form a dynamic alternating feature, and normalize all the dynamic alternating features.

[0084] In addition, the long short-term memory network recognition model adopted in this embodiment adopts a conventional network architecture, including an input convolutional layer, two LSTM layers, an intermediate fully connected layer, a BN layer and an output fully connected layer connected in sequence; the intermediate fully connected layer adopts a ReLU activation function, and the output fully connected layer adopts a Softmax activation function.

[0085] The method for separating speech during interrogation proposed in this embodiment addresses the particularity of interrogation audio information, namely, that it is composed of alternating interrogating and interrogated sound sources. By identifying the dynamic alternating characteristics of the initial audio signal during interrogation, the audio signal is disassembled and removed, and then reassembled to obtain the interrogating speech information and the interrogated speech information, thereby achieving automatic separation of the audio signal, effectively improving the automatic speech separation capability and efficiency, and rapidly collecting interrogation information and interrogated information, which is subsequently converted into text to record the interrogation content, completely preserving the key statements of the interrogated person, and improving the quality of interrogation records.

[0086] The above is a method for speech separation during interrogation provided by one embodiment of this embodiment. Based on the same idea, this embodiment also provides a corresponding system for speech separation during interrogation. For the specific definition of the system for speech separation during interrogation, please refer to the definition of the method for speech separation during interrogation above, which will not be repeated here. The various modules in the above-mentioned system for speech separation during interrogation can be fully or partially implemented by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of the above-mentioned modules.

[0087] This embodiment also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 A method for speech separation during interrogation is provided.

[0088] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0089] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for speech separation during interrogation, characterized in that: The following steps are involved: Constructing an audio training signal set, determining multiple audio dynamic alternation time periods of each audio training signal in the audio training signal set; obtaining an initial audio signal during the interrogation, and importing the initial audio signal according to the time sequence; Performing frame-by-frame windowing processing on the initial audio signal and the audio training signal set to obtain an interrogation windowed speech signal and a training windowed speech signal; Extracting dynamic alternation features of interrogation windowed speech signals and training windowed speech signals respectively; Construct a long short-term memory network recognition model, take the audio training signal set as input, and train the dynamic alternation features with time series information as output to obtain the trained long short-term memory network recognition model; The initial audio signal is recognized through the trained long short-term memory network recognition model to obtain all dynamic alternation features; According to the time series information of the identified dynamic alternating features, the initial audio signal is split and spliced alternately to obtain two sets of interrogation speech separation information.

2. The method for speech separation during interrogation according to claim 1, characterized in that: The method further includes performing silence removal on the initial audio signal before the initial audio signal is framed and windowed, including the following steps: Determine the threshold for sound intensity; The initial audio signal is segmented into fixed-length time segments, the initial audio signal is scanned in sequence, and the volume of each time segment in the audio signal is analyzed. When the signal volume is lower than this threshold, the initial audio signal in this time segment is considered to be a blank signal or silence; A plurality of time periods below a threshold are determined, removed from the initial audio signal, and the initial audio signal is rejoined to obtain an initial audio signal after silence removal.

3. The method for speech separation during interrogation according to claim 1, characterized in that: The method of extracting the dynamic alternation features of the interrogation windowed speech signal and the training windowed speech signal respectively comprises the following steps: Each window of the interrogation windowed speech signal and the training windowed speech signal is subjected to time-frequency decomposition using short-time Fourier transform to obtain spectrum information; the spectrum information of each window is synthesized into a two-dimensional time-frequency signal of the interrogation speech and a two-dimensional time-frequency signal of the training speech in chronological order; According to multiple audio dynamic alternation time periods, continuous spectrum envelopes of the training speech time-frequency two-dimensional signal in the time period are extracted to form long vector features as dynamic alternation features.

4. The method for speech separation during interrogation according to claim 1, characterized in that: The step of performing frame dividing and windowing processing on the initial audio signal and the audio training signal set respectively comprises the following steps: Split the continuous initial audio signal and the audio training signal in the audio training signal set into fixed-length frames, with an overlap of 50% between adjacent frames; The initial audio signal and audio training signal after framing are windowed, as shown in the following formula: x0(n)=x(n)×w(n) Among them, x(n) is the initial speech signal; w(n) is the window function, which is a Hamming window function; x0(n) represents the windowed speech signal; the window function is a Hamming window function, and the frame length is 20ms-40ms.

5. The method for speech separation during interrogation according to claim 3, characterized in that: The time-frequency decomposition is performed on each window of the interrogation windowed speech signal and the training windowed speech signal using short-time Fourier transform, as shown in the following formula: Where X(t,f) is the time-frequency two-dimensional signal; t is the time frame; f is the frequency, w(·) is the Hamming window function; and x(k) is the window signal.

6. The method for speech separation during interrogation according to claim 3, characterized in that: The construction of the dynamic alternation feature includes the following steps: Extract the spectrum of each frame within the target time period of the training speech time-frequency two-dimensional signal, take the logarithm of the spectrum amplitude, and then smooth the logarithm through a low-pass filter to obtain the smoothed spectrum; Detect the significant peaks of the smoothed spectrum and determine the energy distribution of the spectrum; Through the spline interpolation method, interpolation is performed between the significant peaks of the spectrum to form a continuous envelope, which constitutes the spectrum envelope of each frame; Based on the spectrum envelope of each frame, the continuous spectrum envelope of the time period is obtained by spline interpolation; The continuous spectrum envelope is converted into a long vector form to form a dynamic alternating feature, and all the dynamic alternating features are normalized.

7. The method for speech separation during interrogation according to claim 1, characterized in that: The long short-term memory network recognition model includes an input convolutional layer, two LSTM layers, an intermediate fully connected layer, a BN layer and an output fully connected layer connected in sequence; the intermediate fully connected layer adopts a ReLU activation function, and the output fully connected layer adopts a Softmax activation function.

8. The method for speech separation during interrogation according to claim 1, characterized in that: The method of splitting the initial audio signal according to the time series information of the identified dynamic alternating features and alternately splicing the signals to obtain two sets of interrogation speech separation information includes the following steps: Determining multiple time series information of the identified dynamic alternating features; Split the initial audio signal according to multiple time series information, and remove the audio signal with dynamic alternating features; The segmented initial audio signals after dynamic alternating features are removed are re-alternated and spliced to obtain two sets of interrogation speech separation information, namely interrogation speech information and interrogated speech information.

9. A system for speech separation during interrogation, characterized in that: include: processor; a memory having stored thereon a computer program executable on the processor; When the computer program is executed by the processor, the steps of the method for speech separation during interrogation as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a data processing program, which, when executed by a processor, implements the steps of the method for speech separation during interrogation according to any one of claims 1 to 8.

Images