Interactive speech segmentation and clustering method, device and equipment

Abstract

The present invention discloses an interactive speech segmentation and clustering method, apparatus, device, and storage medium, which includes: preprocessing audio data to be processed to obtain N types of speech; auditing the N types of speech and merging speech belonging to the same person to obtain M types of speech, wherein the M types of speech correspond to the number of people in the audio conversation; calculating the center vector of each type of speech and the similarity of each speech segment contained in each type of speech based on the M types of speech, and marking speech segments whose similarity is lower than a preset value; auditing the marked speech segments and reallocating the marked speech segments to obtain audio classification results. This method can improve the accuracy of speech segmentation and clustering results.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to an interactive speech segmentation and clustering method, device and equipment. Background Art

[0002] Speech segmentation and clustering aims to identify who is speaking and when in audio. In a voice file containing multiple people speaking, the start and end times of each person's speech are annotated. This technology can be used for voiceprint extraction, automatic speech recognition, and target speaker detection.

[0003] Currently, the main implementation method is to segment the speech, cluster it using a voiceprint algorithm, and then calculate the start and end times based on the clustering results. Because the algorithm cannot directly determine the number of people speaking, a similarity threshold is generally used for clustering. However, this method, which uses clustering based on similarity thresholds, suffers from inaccurate clustering results. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to propose an interactive speech segmentation and clustering method, apparatus and device, aiming to solve the problem of inaccurate speech segmentation and clustering results in the existing speech segmentation and clustering.

[0005] To achieve the above object, the present invention provides an interactive speech segmentation and clustering method, the method comprising:

[0006] Preprocess the audio data to be processed to obtain N types of speech;

[0007] Auditing the N types of speech and merging speech belonging to the same person to obtain M types of speech, wherein the M types of speech correspond to the number of people in the audio conversation;

[0008] Calculating the central vector of each type of speech and the similarity of each speech segment contained in each type of speech based on the M types of speech, and marking the speech segments whose similarity is lower than a preset value;

[0009] The annotated speech segments are auditioned and reallocated to obtain audio classification results.

[0010] Preferably, after auditing the annotated voice segments and reallocating the annotated voice segments to obtain audio classification results, the method further includes:

[0011] Speaker transition points are marked and audited based on the audio classification results to determine time segmentation points.

[0012] Preferably, the preprocessing of the audio data to be processed to obtain N types of speech comprises:

[0013] De-noising the audio data to be processed using a VAD algorithm to obtain a first audio;

[0014] Slicing the first audio to obtain multiple audio segments;

[0015] The audio segments are clustered based on a preset similarity threshold to obtain the N types of speech.

[0016] Preferably, clustering the audio segments based on a preset similarity threshold to obtain the N types of speech includes:

[0017] Extracting a voiceprint feature vector of each of the audio clips using a voiceprint extraction algorithm;

[0018] The similarity between the audio segments is calculated based on the voiceprint feature vector, and clustering is performed based on the similarity threshold to obtain the N types of speech.

[0019] Preferably, the center vector is obtained by calculating the mean of the vectors of each of the speech segments in the M types of speech.

[0020] To achieve the above object, the present invention further provides an interactive speech segmentation and clustering device, comprising:

[0021] A preprocessing unit, used for preprocessing the audio data to be processed to obtain N types of speech;

[0022] a merging unit, configured to audit the N types of speech and merge speech belonging to the same person to obtain M types of speech, wherein the M types of speech correspond to the number of people in the audio conversation;

[0023] a calculation unit, configured to calculate, based on the M types of speech, a central vector of each type of speech and a similarity of each speech segment contained in each type of speech, and mark the speech segments whose similarity is lower than a preset value;

[0024] The classification unit is used to audit the annotated speech segments and reallocate the annotated speech segments to obtain audio classification results.

[0025] Preferably, the device further comprises:

[0026] A determination unit is used to mark and audit speaker transition points based on the audio classification results to determine time segmentation points.

[0027] To achieve the above objectives, the present invention also proposes a device comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of an interactive speech segmentation and clustering method as described in the above embodiment.

[0028] In order to achieve the above objectives, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement the steps of an interactive speech segmentation and clustering method as described in the above embodiment.

[0029] Beneficial effects:

[0030] The above scheme pre-processes the audio data, audits the resulting N types of speech, merges speech belonging to the same person, and calculates the center vector of each type of speech and the similarity of each speech segment within each type of speech for the M types of speech corresponding to the number of people in the conversation. Segments with a score below a preset value are annotated, and the annotated speech segments are audited and redistributed to obtain audio classification results. This can improve the accuracy of speech segmentation and clustering results. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 A flowchart of an interactive speech segmentation and clustering method provided by one embodiment of the present invention.

[0033] Figure 2 A flowchart of an interactive speech segmentation and clustering method provided by another embodiment of the present invention.

[0034] Figure 3 A schematic structural diagram of an interactive speech segmentation and clustering device provided by one embodiment of the present invention.

[0035] Figure 4 A schematic structural diagram of an interactive speech segmentation and clustering device provided in another embodiment of the present invention.

[0036] The realization of the objectives of the invention, the functional features and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0039] The present invention is described in detail below with reference to the embodiments.

[0040] Reference Figure 1 FIG2 is a flow chart of an interactive speech segmentation and clustering method provided by an embodiment of the present invention.

[0041] In this embodiment, the method includes:

[0042] S11, pre-processing the audio data to be processed to obtain N types of speech.

[0043] The preprocessing of the audio data to be processed to obtain N types of speech includes:

[0044] S11-1, denoising the audio data to be processed using a VAD algorithm to obtain a first audio;

[0045] S11-2, slicing the first audio to obtain multiple audio segments;

[0046] S11-3, clustering the audio segments based on a preset similarity threshold to obtain the N types of speech.

[0047] Furthermore, clustering the audio segments based on a preset similarity threshold to obtain the N types of speech includes:

[0048] S11-3-1, extracting the voiceprint feature vector of each audio clip using a voiceprint extraction algorithm;

[0049] S11-3-2, calculating the similarity between the audio segments based on the voiceprint feature vector, and clustering them based on the similarity threshold to obtain the N types of speech.

[0050] In this embodiment, the processed audio data is processed using a VAD (Voice Activity Detection) algorithm to remove invalid speech sounds, primarily silence and noise. After denoising, the resulting first audio is sliced to produce multiple audio segments. A voiceprint extraction algorithm is then used to extract voiceprint feature vectors for each audio segment. The similarity between the audio segments is calculated based on the voiceprint feature vectors, and clustering is performed based on a preset similarity threshold. The preset similarity threshold is typically set relatively high to ensure the purity of each cluster and avoid lumping different clusters into the same category. The first audio can be sliced according to a preset length. For example, if the preset segment length is 1.28 seconds and the offset is 0.32 seconds, then a 10-second segment of speech will have a number of segments: (10 - 1.28) / 0.32 + 1. This is rounded to the nearest integer, resulting in 28 audio segments. The remaining seconds are then grouped together in the last segment. In specific implementation, the similarity threshold can obtain the best value through test data. Usually, in order to ensure the purity of classification, a slightly higher threshold value is used.

[0051] S12: Auditing the N types of voices and merging voices belonging to the same person to obtain M types of voices, wherein the M types of voices correspond to the number of people in the audio conversation.

[0052] In this embodiment, N types of speech can be audited and two misclassified types can be merged. The final number of types after the merger is the subsequent number threshold, that is, the number threshold is the number of people in the audio conversation.

[0053] S13, calculating the central vector of each type of speech and the similarity of each speech segment contained in each type of speech based on the M types of speech, and marking the speech segments whose similarity is lower than a preset value.

[0054] The center vector is obtained by calculating the mean of the vectors of each speech segment in the M types of speech.

[0055] This embodiment further includes clustering based on the number of people. This involves segmenting and clustering the resulting clusters using a number threshold. Simultaneously, the center vector of each cluster and the similarity between each segment within that cluster are calculated, with segments with low similarity being labeled as difficult to classify. Because the model evaluates the required thresholds based on collected test data before use, the evaluated thresholds can be directly used during final implementation. The criterion for determining low similarity is to determine the appropriate threshold for misclassification based on different thresholds evaluated on the test data. The center vector is the mean of the vectors for each segment within each cluster. The similarity between segments within a cluster can be calculated using different methods depending on specific needs, such as cosine. Clustering based on a number threshold is generally superior to a similarity threshold. Therefore, clustering based on a specific number of people can be performed again to improve algorithm accuracy. Furthermore, clustering based on the number of people can be performed using algorithms such as AHC and VBx. In the AHC algorithm, the total number of clusters decreases by one with each iteration, so the number of iterations is determined by the number of people threshold. Specifically, when AHC is used for calculation, the calculation process includes: (1) each fragment is initially a class, and the pairwise similarity is calculated; (2) the two classes A and B with the largest similarity are merged, and the new class after the merger is C, and the similarity with other classes is updated, and the total number of classes is reduced by one; there are many ways to update the similarity, such as taking the average of the similarities between A and B and other classes; (3) repeat step (2) until the number of classes reaches the requirement.

[0056] S14: auditing the annotated speech segments and reallocating the annotated speech segments to obtain audio classification results.

[0057] In this embodiment, the annotated speech segments are auditioned and the difficult-to-classify segments annotated in the previous step are redistributed to obtain segmentation results, thereby improving the accuracy of segmentation and clustering.

[0058] In another embodiment, referring to Figure 2 The method further includes:

[0059] S15: marking and auditioning speaker transition points based on the audio classification results to determine time segmentation points.

[0060] Since the audio time range of each person can be marked in the audio, the starting and ending points are the segmentation points. Based on the final segmentation results, the segmentation points with shorter speaker transition intervals are marked out, and further segmentation is performed at this point through audition. Since the algorithm uses a speech fragmentation processing method, when the speaker transition interval is short, the segmentation point is not easy to accurately determine. Therefore, by marking the speaker transition points and performing audition again, a more accurate segmentation result is obtained. In other words, due to problems with segmentation accuracy and character dialogue transitions, the position of the segmentation point may be inaccurate. The point can be manually moved back and forth to a more appropriate position to improve the segmentation accuracy. For example, 0 to 10 seconds is speaker A, 11 seconds to 20 seconds is speaker B, and the split point is at 9 seconds. After adjustment, it is corrected to 10 seconds.

[0061] Here is an example: For example, there is a complete audio clip with a duration of 2 hours and a conversation between 4 people.

[0062] (1) First, the speech is preprocessed, including denoising, segmentation, and clustering. For example, a segment length of 1.28 seconds is used, with an offset of 0.32 seconds. Then, the AHC algorithm is used to cluster based on a pre-set threshold. Thus, 6 segments of speech are obtained.

[0063] (2) By auditioning the six speech segments obtained in (1), the speech segments of the same person are merged to obtain four speech segments.

[0064] (3) The speech is processed again, and the number of segments merged in the previous step is used as the clustering termination condition. Four segments of speech are re-output, and the parts of each segment with low similarity to the segment are marked.

[0065] (4) Review the marked part and reallocate the speech in that part.

[0066] (5) Based on the speech segmentation results, the speaker transition points (the time periods when the previous and next speakers are different) are marked, and then more accurate time point segmentation is performed (at the speaker transition point, due to algorithmic problems, it is difficult to avoid misclassification, so further processing of the segmentation points is performed to obtain the final segmentation results). In other words, each category is composed of many segments (such as 1.28s), and then the segments are merged into continuous large segments. Each category can have multiple continuous large segments. From the audio point of view, the audio is divided into many large segments, and each segment is assigned to a certain category. The segmentation point of two adjacent segments is the transition point between different people. The point position can be manually adjusted to obtain a more accurate segmentation result.

[0067] Reference Figure 3 FIG2 is a schematic structural diagram of an interactive speech segmentation and clustering device provided by an embodiment of the present invention.

[0068] In this embodiment, the device 30 includes:

[0069] A preprocessing unit 31 is used to preprocess the audio data to be processed to obtain N types of speech;

[0070] a merging unit 32 for auditing the N types of speech and merging speech belonging to the same person to obtain M types of speech, wherein the M types of speech correspond to the number of people in the audio conversation;

[0071] A calculation unit 33 is configured to calculate the center vector of each type of speech and the similarity of each speech segment contained in each type of speech based on the M types of speech, and mark the speech segments whose similarity is lower than a preset value;

[0072] The classification unit 34 is configured to audit the annotated speech segments and reallocate the annotated speech segments to obtain audio classification results.

[0073] The center vector is obtained by calculating the mean of the vectors of each speech segment in the M types of speech.

[0074] Furthermore, the pre-processing unit 31 includes:

[0075] a denoising unit, configured to denoise the audio data to be processed using a VAD algorithm to obtain a first audio;

[0076] a slicing processing unit, configured to perform slicing processing on the first audio to obtain a plurality of audio segments;

[0077] The clustering unit is configured to cluster the audio segments based on a preset similarity threshold to obtain the N types of speech.

[0078] Furthermore, the clustering unit includes:

[0079] a voiceprint extraction unit, configured to extract a voiceprint feature vector of each of the audio segments using a voiceprint extraction algorithm;

[0080] A calculation unit is configured to calculate the similarity between the audio segments according to the voiceprint feature vector, and perform clustering according to the similarity threshold to obtain the N types of speech.

[0081] In another embodiment, referring to Figure 4 The device 40 further includes:

[0082] The determining unit 41 is configured to mark and audit speaker transition points based on the audio classification result to determine time segmentation points.

[0083] Each unit module of the device 30 / 40 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the corresponding steps above for details.

[0084] The embodiment of the present invention further provides a device, which includes the interactive speech segmentation and clustering device as described above, wherein the interactive speech segmentation and clustering device can adopt Figure 3 The structure of the embodiment can be executed accordingly. Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments and will not be repeated here.

[0085] The device includes: a mobile phone, digital camera, tablet computer, or other device with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0086] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0087] The input unit can be used to receive input digital, character, or image information, and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment can include not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0088] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel. Optionally, the display panel can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), etc. Furthermore, the touch-sensitive surface can cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event. The processor then provides a corresponding visual output on the display panel based on the type of touch event.

[0089] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The interactive speech segmentation and clustering method shown in FIG. The computer readable storage medium may be a read-only memory, a magnetic disk or an optical disk.

[0090] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. For similar or identical parts between the various embodiments, reference can be made to each other. For the apparatus embodiments, device embodiments, and storage medium embodiments, since they are generally similar to the method embodiments, their descriptions are relatively simple. For relevant parts, reference can be made to the descriptions of the method embodiments.

[0091] Furthermore, in this document, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0092] While the foregoing description shows and describes preferred embodiments of the present invention, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments, and can be modified within the scope of the present invention by the teachings herein or by techniques or knowledge in the relevant art. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the present invention are intended to be within the scope of the appended claims.

Claims

1. An interactive speech segmentation and clustering method, characterized in that: The method comprises: Preprocess the audio data to be processed to obtain N types of speech; The preprocessing of the audio data to be processed to obtain N types of speech includes: De-noising the audio data to be processed using a VAD algorithm to obtain a first audio; Slicing the first audio to obtain multiple audio segments; Clustering the audio segments based on a preset similarity threshold to obtain the N types of speech; Clustering the audio segments based on a preset similarity threshold to obtain the N types of speech includes: Extracting a voiceprint feature vector of each of the audio clips using a voiceprint extraction algorithm; Calculating the similarity between the audio segments according to the voiceprint feature vectors, and clustering them according to the similarity threshold to obtain the N types of speech; Auditing the N types of speech and merging speech belonging to the same person to obtain M types of speech, wherein the M types of speech correspond to the number of people in the audio conversation; Calculating the central vector of each type of speech and the similarity of each speech segment contained in each type of speech based on the M types of speech, and marking the speech segments whose similarity is lower than a preset value; The center vector is obtained by calculating the mean of the vectors of each of the speech segments in the M types of speech; The annotated speech segments are auditioned and reallocated to obtain audio classification results.

2. The interactive speech segmentation and clustering method according to claim 1, wherein: After auditing the annotated voice segments and reallocating the annotated voice segments to obtain audio classification results, the method further includes: Speaker transition points are marked and audited based on the audio classification results to determine time segmentation points.

3. An interactive speech segmentation and clustering device, characterized in that: The device comprises: A preprocessing unit, used for preprocessing the audio data to be processed to obtain N types of speech; The pre-processing unit comprises: a denoising unit, configured to denoise the audio data to be processed using a VAD algorithm to obtain a first audio; a slicing processing unit, configured to perform slicing processing on the first audio to obtain a plurality of audio segments; A clustering unit, configured to cluster the audio segments based on a preset similarity threshold to obtain the N types of speech; The clustering unit comprises: a voiceprint extraction unit, configured to extract a voiceprint feature vector of each of the audio segments using a voiceprint extraction algorithm; a calculation unit, configured to calculate the similarity between the audio segments according to the voiceprint feature vector, and perform clustering according to the similarity threshold to obtain the N types of speech; a merging unit, configured to audit the N types of speech and merge speech belonging to the same person to obtain M types of speech, wherein the M types of speech correspond to the number of people in the audio conversation; a calculation unit, configured to calculate, based on the M types of speech, a central vector of each type of speech and a similarity of each speech segment contained in each type of speech, and mark the speech segments whose similarity is lower than a preset value; The center vector is obtained by calculating the mean of the vectors of each of the speech segments in the M types of speech; The classification unit is used to audit the annotated speech segments and reallocate the annotated speech segments to obtain audio classification results.

4. The interactive speech segmentation and clustering device according to claim 3, characterized in that: The device further comprises: A determination unit is used to mark and audit speaker transition points based on the audio classification results to determine time segmentation points.

5. An interactive speech segmentation and clustering device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of the interactive speech segmentation and clustering method according to any one of claims 1 to 2.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is executed by a processor to implement the steps of an interactive speech segmentation and clustering method as described in any one of claims 1 to 2.

Images