Audio segment clustering method, device, electronic device and medium

Abstract

The present disclosure provides a method, apparatus, electronic device, and medium for clustering audio clips. The clustering method includes obtaining a first clustering result corresponding to a first audio clip, obtaining a second audio clip collected during the current sampling period, and parsing the second audio clip based on the first clustering result to obtain a second clustering result. The second audio clip and the first audio clip collected during the previous sampling period are both extracted from the same real-time audio stream, and the second clustering result includes the identity information of at least one speaker within the second audio clip and a timestamp corresponding to the identity information. This method implements segmentation and clustering of streaming speaker audio streams and updates speaker information online in real time, thereby reflecting the speaker's speech status in real time and improving speaker recognition accuracy and user experience.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device, and medium for clustering audio segments. Background Art

[0002] Speaker diarization (SD) is a technology that distinguishes the voices of different speakers according to their identities. It solves the problem of "who spoke and when".

[0003] In the process of parsing audio files, it is generally supported that different speakers use different microphones. In other words, different speakers and speech content can only be identified through different microphones or sound recording devices. If two or more speakers use the same microphone, for example, in a conference room, two or more users speak successively, it is impossible to detect who is currently speaking, including whether the current speaker is a new speaker or a previous speaker. As a result, for online conference listeners, they cannot understand the current speaker situation in real time, affecting user experience and conference quality. Summary of the Invention

[0004] To solve the above technical problems and improve the accuracy of speaker recognition, this disclosure discloses the following technical solutions:

[0005] In a first aspect, an embodiment of the present disclosure provides a method for clustering audio clips, the method comprising the following steps: obtaining a first clustering result corresponding to a first audio clip, the first clustering result including identity information of at least one speaker in the first audio clip, and timestamp information corresponding to the identity information; obtaining a second audio clip collected in a current sampling period, and parsing the second audio clip according to the first clustering result to obtain a second clustering result.

[0006] The second audio segment and the first audio segment collected in the previous sampling period are both captured from the same real-time audio stream, and the second clustering result includes identity information of at least one speaker in the second audio segment and timestamp information corresponding to the identity information.

[0007] The method provided in this aspect obtains the current audio segment in real time according to a preset sampling period during the audio stream generation process, and aligns it with the first clustering result of the audio segment recorded at the previous moment to obtain a second clustering result. This method realizes the segmentation and clustering of the streaming speaker audio stream and updates the speaker information online in real time, thereby reflecting the speaking situation of different speakers in real time, improving the accuracy of speaker recognition and user experience.

[0008] With reference to the first aspect, in a possible implementation of the first aspect, the current sampling period is smaller than the second audio segment, and parsing the second audio segment according to the first clustering result to obtain a second clustering result includes:

[0009] Based on the current sampling period being less than the second audio segment, a first sub-segment and a second sub-segment are obtained, respectively. Based on the first clustering result, the first sub-segment is clustered according to the principle of longest speaking time of the speaker to obtain a first sub-result. Based on the first clustering result, the second sub-segment is clustered according to the principle of voiceprint feature matching to obtain a second sub-result. The first sub-segment is the audio stream of the temporal overlap between the first and second audio segments, and the second sub-segment is the remaining audio segment of the second audio segment after removing the first sub-segment.

[0010] In combination with the first aspect, in another possible implementation of the first aspect, based on the first clustering result, the first sub-segments are clustered according to the principle of the longest speaking time of the speaker to obtain the first sub-result, including: using the Hungarian algorithm to find the optimal mapping relationship based on the speaking time of each speaker in the first clustering result and the identity ID corresponding to the speaker to obtain the first sub-result; wherein, under the optimal mapping relationship, the overlapping time of the clustered audio parts is the longest.

[0011] In combination with the first aspect, in another possible implementation of the first aspect, based on the first clustering result, clustering the second sub-segments according to a voiceprint feature matching principle to obtain a second sub-result includes: obtaining a first voiceprint feature corresponding to the second sub-segment and at least one second voiceprint feature in the first clustering result, each second voiceprint feature corresponding to a speaker of the first audio segment; performing a similarity comparison between the first voiceprint feature and the at least one second voiceprint feature; if a target second voiceprint feature is present in the at least one second voiceprint feature, marking the first voiceprint feature and the target second voiceprint feature as being from the same speaker, thereby obtaining the second sub-result. The similarity between the target second voiceprint feature and the first voiceprint feature is greater than or equal to a threshold value.

[0012] In combination with the first aspect, in another possible implementation of the first aspect, the above method also includes: if the target second voiceprint feature does not exist, marking the first voiceprint feature as corresponding to a new speaker to obtain a second sub-result, and the new speaker is different from any speaker in the first audio segment cluster.

[0013] In combination with the first aspect, in another possible implementation of the first aspect, the method further includes: obtaining a second voiceprint feature based on an average value of voiceprint vectors for the same speaker in each sampling period of the real-time audio stream.

[0014] In combination with the first aspect, in another possible implementation of the first aspect, the current sampling period is greater than or equal to the second audio segment, and the second audio segment is analyzed according to the first clustering result to obtain a second clustering result, including: according to the first clustering result, the second audio segment is clustered according to the voiceprint feature matching principle to obtain a second clustering result.

[0015] In combination with the first aspect, in another possible implementation of the first aspect, obtaining a second audio segment collected in the current sampling period includes: when the end of the human voice ASR final is detected by voice activity detection VAD, obtaining the second audio segment according to the sampling period.

[0016] In combination with the first aspect, in another possible implementation of the first aspect, the above method also includes: updating the first clustering result based on the second clustering result, obtaining a third clustering result of the current audio stream, and displaying the third clustering result; the current audio stream includes a first audio segment and a second audio segment, and the third clustering result includes at least one speaker identity information in the current audio stream, and timestamp information corresponding to the identity identification information.

[0017] In a second aspect, an embodiment of the present disclosure further provides a device for clustering audio clips, the device comprising:

[0018] an acquiring unit, configured to acquire a first clustering result corresponding to the first audio segment, the first clustering result including identity information of at least one speaker in the first audio segment and a timestamp corresponding to the identity information;

[0019] An acquisition unit, configured to acquire a second audio segment acquired during a current sampling period;

[0020] The processing unit is configured to parse the second audio segment according to the first clustering result to obtain a second clustering result.

[0021] The second audio segment and the first audio segment collected in the previous sampling period are both captured from the same real-time audio stream, and the second clustering result includes identity information of at least one speaker in the second audio segment and timestamp information corresponding to the identity information.

[0022] In addition, the device also includes other functional units or modules for implementing the methods of various embodiments in the aforementioned first aspect.

[0023] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores computer program instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor executes the audio segment clustering method as in the first aspect or any implementation of the first aspect.

[0024] In addition, the electronic device may further include at least one functional module or unit such as an interface and a transceiver.

[0025] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the first aspect or any implementation of the first aspect can be implemented.

[0026] It should be noted that the beneficial effects corresponding to the technical solutions of the various implementation methods of the second to fourth aspects can be referred to the beneficial effects of the first aspect and the various implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 A schematic diagram of a scenario of an audio streaming application provided by an embodiment of the present disclosure;

[0029] Figure 2 A flowchart of a method for clustering audio clips provided in an embodiment of the present disclosure;

[0030] Figure 3 A schematic diagram of obtaining a first audio clip and a second audio clip provided in an embodiment of the present disclosure;

[0031] Figure 4 A schematic diagram of updating a clustering result of a current audio stream provided by an embodiment of the present disclosure;

[0032] Figure 5 A flowchart of parsing a second audio clip to obtain a second clustering result provided by an embodiment of the present disclosure;

[0033] Figure 6 A flowchart for obtaining a second sub-result provided in an embodiment of the present disclosure;

[0034] Figure 7 A schematic diagram of determining a second sub-result in a second clustering result provided by an embodiment of the present disclosure;

[0035] Figure 8 A structural block diagram of an audio segment clustering device provided by an embodiment of the present disclosure;

[0036] Figure 9 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The technical solutions of the present disclosure are described clearly and completely below in conjunction with the accompanying drawings. Obviously, the embodiments described are only a portion of the embodiments of the present disclosure, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present disclosure without creative effort are within the scope of protection of the present disclosure.

[0038] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0039] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, machine learning, and deep learning.

[0040] Biometrics refers to the use of computers to identify people using their own physiological or behavioral characteristics. Based on unique, reliable, and stable physiological characteristics (such as fingerprints, irises, faces, and palm prints) or behavioral characteristics (such as voice, keystrokes, gait, and signatures), it leverages the power of computers and network technologies to perform image processing and pattern recognition to verify a person's identity. This technology offers excellent security, reliability, and effectiveness.

[0041] Voiceprint recognition (VPR), also known as speaker recognition (SR), is a type of biometric identification technology that determines a speaker's identity based on their voice. Due to its security, reliability, and convenience, voiceprint recognition is widely used in situations where identity verification is required.

[0042] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the user information involved in this disclosure, such as user identity information, voiceprint information in the user audio stream, etc., should be authorized in an appropriate manner in accordance with relevant laws and regulations.

[0043] For example, before attending a meeting, the system will send a prompt message to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the technical solution of the present disclosure based on the prompt message, and to identify the speaker's identity based on the acquired voiceprint information. In addition, during the user personal information authorization process, a prompt message can be sent to the user, such as a pop-up window, in which the prompt message can be presented in text form.

[0044] It should be understood that the above notification and authorization process for obtaining user personal information are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0045] The present invention provides a clustering method based on audio clips recorded in real time by a speaker, which is suitable for Figure 1 In the application environment. Figure 1 The application environment 100 shown includes at least one terminal device, such as a mobile phone 10 , a notebook 20 , and a server 30 , and the mobile phone 10 , the notebook 20 , and the server 30 can be connected via a network 40 .

[0046] The terminal devices include, but are not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. The server 30 can be implemented as a standalone server or a server cluster consisting of multiple servers. Furthermore, the server 30 can be a standalone server or a server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0047] In addition, the network connecting the terminal device and the server may include various connection types, such as wired, wireless communication links or fiber optic cables, etc. In one example, a user can use a terminal device to communicate with the server through the network to receive and / or send information between the terminal device and the server.

[0048] It should be noted that the method provided in the embodiment of the present disclosure can be executed by a server, for example, the server obtains the audio stream reported by each terminal device and then processes the audio stream, or it can also be executed by a terminal device, such as a laptop computer collecting the speaker's audio stream in real time and processing it in real time. This embodiment does not limit the execution subject of the method.

[0049] It should be understood that Figure 1 The number of terminal devices, networks and servers is only illustrative. According to implementation requirements, there can be any number of terminal devices, networks and servers. The terminal devices in the embodiments of the present disclosure can specifically correspond to application systems in actual production.

[0050] The disclosed embodiments provide a method for clustering audio clips. Clustering involves dividing a dataset into distinct classes or clusters based on a specific criterion, such that data objects within the same cluster are as similar as possible, while data objects in different clusters are as different as possible. In other words, after clustering, data within the same class is grouped together as closely as possible, while data from different classes is separated as much as possible. In this embodiment, clustering is performed on audio streams of real-time speech from different speakers.

[0051] This embodiment also involves speech recognition technology, also known as automatic speech recognition (ASR), which aims to convert the vocabulary content in human speech into computer-readable input, such as keystrokes, binary codes, or character sequences. The process includes acoustic feature extraction, acoustic modeling, language modeling, and language processing.

[0052] The embodiments of the present disclosure are further described below with reference to the accompanying drawings.

[0053] like Figure 2 FIG. 1 is a flow chart of a method for clustering audio segments provided by an embodiment of the present disclosure. The method can be applied to the aforementioned server or terminal device. The method specifically includes:

[0054] Step 101: Obtain a first clustering result corresponding to a first audio segment, where the first clustering result includes identity information of at least one speaker in the first audio segment and timestamp information corresponding to the identity information.

[0055] The first audio segment is a portion of a speaker's audio stream recorded in real time, and can be obtained by sampling according to a preset sampling period. The audio stream is an audio file generated by real-time sampling of the speech of at least one speaker.

[0056] Specifically, in the process of generating an audio stream, the microphone can convert the speaker's acoustic signal into an electrical signal, and then capture the speech to be processed through the microphone. This speech to be processed is then input into a speech detection model. All the results output by the speech detection model after processing are then spliced ​​together to generate an audio stream. The speech frames in this audio stream contain the speech of at least one speaker. For example, in a conference audio stream in a meeting room, the audio stream or audio clip captured by the same microphone includes the speech of at least one speaker. In this embodiment, the specific implementation method for obtaining the audio stream is not limited.

[0057] The identity identification information of at least one speaker in the first clustering result may include, for example: the identity ID of each speaker, also known as "speaker ID". Furthermore, the identity ID may include but is not limited to a user name, a customized identity code / number. The timestamp information corresponding to the identity identification information may, for example, refer to the length of time each speaker speaks in the first audio segment. The timestamp information is actually a time period, recording the time interval from the start moment to the end moment of each speaker's speech in the first audio segment. And there is a mapping relationship between the identity ID of each speaker and the timestamp. For example, it is recorded in the first clustering result: there is a mapping relationship between speaker 1 and the first time interval, there is a mapping relationship between speaker 2 and the second time interval, and so on. In addition, the first clustering result can be obtained by parsing the first audio segment.

[0058] It can be understood that if the first audio segment contains the speech of multiple speakers, the clustering result in step 101 may include the identity information of at least one speaker in the first audio segment and its corresponding timestamp information; or, the clustering result may also include the identity information of all speakers in the first audio segment and their corresponding timestamp information.

[0059] Step 102: Obtain a second audio segment collected in the current sampling period, and analyze the second audio segment according to the first clustering result to obtain a second clustering result.

[0060] The second audio segment and the first audio segment collected in the previous sampling period are both captured from the same real-time audio stream, and the second clustering result includes identity information of at least one speaker in the second audio segment and timestamp information corresponding to the identity information.

[0061] It should be noted that the speakers in the first audio segment and the second audio segment may be the same or different.

[0062] In this embodiment, the duration of an audio segment is represented by "Ni". For example, if the duration of the first audio segment is N1 and the duration of the second audio segment is N2, where N1 = N2, two possible relationships exist between the first and second audio segments, depending on the sampling period. The first relationship is when the sampling period is less than the duration of the second audio segment, N2, in which case N1 and N2 overlap in time. The second relationship is when the sampling period is greater than or equal to the duration of the second audio segment, N2, in which case N1 and N2 do not overlap in time. This embodiment primarily discusses the processing of audio streams with overlapping durations.

[0063] like Figure 3 As shown, the sampling period is less than the duration N of the audio segment, and there are audio streams with overlapping time between N1 and N2, and of course, there are also audio streams with non-overlapping time. Optionally, N is 300 seconds.

[0064] Optionally, the first audio segment in step 101 and the second audio segment in step 102 may be obtained by clustering using a Voice Activity Detection (VAD) algorithm. The purpose of VAD detection is to judge the input signal and distinguish the voice signal from various background noise signals.

[0065] Specifically, when the human voice is detected by VAD, that is, at each ASR final, the current time is recorded, clustering is triggered, and an audio segment is obtained. The current time is also recorded as t. In this example, the timestamp recorded at the ASR final of the first audio segment is t1, and the timestamp recorded at the ASR final of the second audio segment is t2. There is some temporal overlap between the audio streams of these two audio segments.

[0066] It should be understood that during the audio stream segmentation process, the duration of the audio stream segmented at the current time t2 and the previous time t1 may be different, that is, N1≠N2. In this embodiment, speech segmentation is performed according to a fixed segmentation duration to obtain audio segments of the same duration.

[0067] In step 102, the second audio segment is parsed based on the first clustering result to obtain a second clustering result. The specific process may be the same as the method for parsing the first audio segment to obtain the first clustering result in step 101, or it may be different. For example, based on the speakers identified in the first clustering result and the timestamps corresponding to each speaker, the second audio segment is parsed using a preset algorithm to identify the speakers in the second audio segment and the timestamps corresponding to each speaker. The preset algorithm may be one or more deep learning algorithms based on a neural network model, such as the Hungarian algorithm. Moreover, in the two audio segments, the processing algorithms for the audio streams of the time-overlapping part and the time-non-overlapping part may be the same or different. It should be understood that the above-mentioned preset algorithm may also be other clustering algorithms, and this embodiment does not limit this.

[0068] In addition, the method of this embodiment further includes: updating the first clustering result based on the second clustering result to obtain a third clustering result of the current audio stream, and displaying the third clustering result.

[0069] Among them, the current audio stream has a different duration from the first audio segment N1 and the second audio segment N2, that is, the current audio stream includes the first audio segment and the second audio segment, and the third clustering result includes at least one speaker identity information in the current audio stream and timestamp information corresponding to the identity identification information.

[0070] In a specific example, Figure 4 As shown in FIG, a schematic diagram of updating the clustering result of the current audio stream is obtained. In the first audio segment N1, the first clustering result is obtained, and four speakers are clustered, namely speaker 1, speaker 2, speaker 3 and speaker 4, which are represented by speaker ID numbers as "1, 2, 3, 4" respectively. The timestamp corresponding to each speaker is as follows: Figure 4 As shown, the speech duration of each speaker is represented. Each speech duration is composed of one or more feature vectors (embeddings). The embedding duration can be set to 1.5 seconds, and the embedding sliding window step length is set to 0.75 seconds. It should be understood that other larger or smaller window lengths and step lengths can also be set in this embodiment.

[0071] Based on the first clustering result, the second audio segment N2 is parsed to obtain a second clustering result. The second clustering result includes speakers 4 and 5, as well as the timestamps (durations) of the speeches of speakers 4 and 5, respectively. The second clustering result is aligned with the labels of the first clustering result. Since the speech duration of "speaker 4" in the corresponding clustering result for the audio streams in the temporally overlapping portion of N1 and N2 is longer than that of "speaker 3," after label alignment, the audio segments in the temporally overlapping portion of the second audio segment N2 are labeled as "speaker 4" or speaker ID 4. For the audio streams in the temporally non-overlapping portion of N2, the second clustering result obtained includes speaker 5, and the identity information of speaker 5 is different from that of the four speakers in the first clustering result. Therefore, the audio segments in the temporally non-overlapping portion of the second audio segment N2 are labeled as "speaker 5" or speaker ID 5, thereby obtaining a second clustering result including "speaker 4" and "speaker 5," as well as the timestamp information corresponding to each of the two speakers.

[0072] A third clustering result is obtained based on the above-mentioned first clustering result and second clustering result. The third clustering result includes the first clustering result and the second clustering result, and the third clustering result is based on the second clustering result. In this example, the current audio stream corresponding to the third clustering result includes 5 speakers, and the speaker identity information is speaker ID1 to 5, respectively, and also includes the timestamp information corresponding to each speaker.

[0073] It should be noted that if two different results are displayed in two adjacent clustering results, the most recent clustering result, that is, the second clustering result obtained by clustering at the current time t2, is used to update the first clustering result to obtain the third clustering result.

[0074] The method provided in this embodiment obtains the current audio segment in real time according to a preset sampling period during the audio stream generation process, and aligns it with the first clustering result of the audio segment recorded at the previous moment to obtain a second clustering result. This method realizes the segmentation and clustering of the streaming speaker audio stream and updates the speaker information online in real time, thereby reflecting the speaking situation of different speakers in real time, improving the accuracy of speaker recognition and user experience.

[0075] In addition, in this method, audio clips are acquired in real time based on the detection of the ASR final time point, and the clustering results of the audio clips are updated, thereby realizing audio clustering and processing of streaming speakers. Compared with offline audio stream processing, this method has stronger timeliness and higher feedback quality.

[0076] The clustering process of step 102 of the above method is described in detail below.

[0077] See also Figure 5 , which is a flow chart for parsing a second audio clip to obtain a second clustering result. The second clustering result obtained in step 102 includes a first sub-result and a second sub-result. The first sub-result is the clustering result corresponding to the first sub-segment in the second audio clip, and the second sub-result is the clustering result corresponding to the second sub-segment in the second audio clip. The first sub-segment is the audio stream containing the temporal overlap between the first and second audio clips, and the second sub-segment is the remaining audio clip in the second audio clip after removing the first sub-segment.

[0078] In step 102, the second audio clip is parsed according to the first clustering result to obtain a second clustering result. The specific process includes:

[0079] Step 1021: Based on the first clustering result, cluster the first sub-segments according to the principle of the longest speaking time of the speaker to obtain the first sub-result, and,

[0080] Step 1022: Based on the first clustering result, cluster the second sub-segments according to the voiceprint feature matching principle to obtain a second sub-result.

[0081] Specifically, step 1021 includes: using the Hungarian algorithm to find an optimal mapping relationship based on the speaking duration of each speaker and the speaker's corresponding ID in the first clustering result to obtain the first sub-result. Under the optimal mapping relationship, the overlapping time of the clustered audio portions is the longest.

[0082] The Hungarian algorithm is a combinatorial optimization algorithm for solving task allocation problems in polynomial time and is used to find the maximum matching algorithm. In this embodiment, the Hungarian algorithm is used to find the optimal mapping relationship, thereby achieving label alignment of the clustering results of different audio clips.

[0083] The mapping relationship is the mapping relationship between each speaker ID and embedding in the previous sampling period (history). Optionally, the mapping relationship is marked as "M0", and M0 records all the embeddings corresponding to each speaker ID in the previous audio segment. For ease of explanation, in this embodiment, the sum of all embedding times contained in each speaker ID is called a "chunk", such as Figure 4In the example shown, the speaker clustering algorithm SD is used to parse the first audio segment N1, and the first clustering result S1 obtained is: speaker ID1-chunk 1, speaker ID2-chunk 2, speaker ID3-chunk 3, speaker ID4-chunk 4, a total of 4 mappings, corresponding to the identity information and timestamp information of the 4 speakers.

[0084] Similarly, the SD algorithm is used to parse the second audio segment N2, resulting in the second clustering result S2: speaker ID4-chunk 5 and speaker ID5-chunk 6, two mappings corresponding to the two speakers. Then, based on the temporally overlapping audio streams, the speaker IDs and timestamps of S1 and S2 are label-aligned. The Hungarian algorithm is then used to find the optimal mapping relationship, ensuring that the overlapping audio streams of S1 and S2 are the longest. The resulting mapping is: the overlap duration of speaker ID4 in S2 is greater than the overlap duration of speaker ID3 in S1. Therefore, based on the optimal duration match, the mapping for the first sub-result is: speaker ID4-chunk 5.

[0085] In this embodiment, for audio clips with temporal overlap, an algorithm with optimal duration is used for matching and mapping to obtain clustering results of the temporal overlap audio streams. The clustering results obtained by this method have high accuracy and are time-efficient.

[0086] Furthermore, in step 1022, based on the first clustering result, the second sub-segments are clustered according to the voiceprint feature matching principle to obtain a second sub-result, such as Figure 6 As shown, specifically including:

[0087] Step 301: Obtain a first voiceprint feature corresponding to the second sub-segment and at least one second voiceprint feature in the first clustering result, where each second voiceprint feature corresponds to a speaker of the first audio segment.

[0088] Specifically, one possible implementation for extracting voiceprint features is to use a feature model to extract acoustic features from the second sub-segment, where the acoustic features include, for example, voiceprint features, to obtain a first voiceprint feature. Optionally, the first voiceprint feature can be represented by a feature vector. Similarly, acoustic feature extraction is performed on the first audio segment to obtain a second voiceprint feature, where the second voiceprint feature includes at least one feature vector.

[0089] It should be noted that the method of voiceprint feature extraction includes but is not limited to the method of extracting features based on Mel-Frequency Cepstral Coefficients (MFCC) to extract the features of the speech segment. Since MFCC is based on the cepstrum extraction method, MFCC features are more in line with the human auditory principles and have better effect in speech feature extraction. In addition, MFCC feature extraction technology is well known to those skilled in the art and will not be described in detail here.

[0090] Step 302: perform a similarity comparison between the first voiceprint feature and the at least one second voiceprint feature.

[0091] Specifically, it is determined whether a target second voiceprint feature exists in the at least one second voiceprint feature, and whether the similarity between the target second voiceprint feature and the first voiceprint feature is greater than or equal to a threshold value.

[0092] Step 303: If the target second voiceprint feature exists, the first voiceprint feature and the target second voiceprint feature are marked as being from the same speaker, and the second sub-result is obtained.

[0093] Specifically, if there is a second voiceprint feature in at least one second voiceprint feature, and the similarity between it and the first voiceprint feature is greater than or equal to the threshold value, then the second voiceprint feature is used as the target second voiceprint feature, and the speaker ID corresponding to the target second voiceprint feature is the speaker ID of the first voiceprint feature.

[0094] For example, Figure 7 As shown, in the second audio segment N2, the first voiceprint feature of the non-overlapping audio stream (the second sub-segment) is determined to be most similar to the second voiceprint feature of speaker ID3 in the first audio segment N1. Therefore, this second sub-segment is considered to be the same person as speaker ID3 in chunk 3. The second sub-segment is then labeled as speaker ID3-chunk 5, resulting in the second sub-result. The voiceprint feature in chunk 3 is the average of all feature vectors (embeddings) contained in chunk 3, where each embedding can be calculated as a voiceprint feature. Similarly, the voiceprint features of other chunks are obtained by calculating the average of the embeddings.

[0095] In addition, the above method also includes: if the target second voiceprint feature does not exist, marking the first voiceprint feature as corresponding to a new speaker to obtain the second sub-result, where the new speaker is different from any speaker in the first audio segment cluster.

[0096] For example, the speaker ID5 and the corresponding timestamp information are obtained in the second clustering result S2, that is, the number of speakers clustered in S2 is greater than the number of speakers in S1, and the speaker ID5 is not among the speakers speaker ID1 to 4 clustered in the first clustering result S1, that is, all the second voiceprint features are not similar to the first voiceprint features, or the similarity is less than the above threshold value. At this time, the speaker ID5 is marked as the new speaker ID, that is, the second sub-result is speaker ID5-chunk6.

[0097] In another possible implementation, in step 301, extracting at least one second voiceprint feature may be accomplished by calculating the second voiceprint feature based on the average of voiceprint vectors (e.g., embeddings) for the same speaker within each sampling period of the real-time audio stream; and then performing a similarity comparison between each second voiceprint feature and the first voiceprint feature. The first audio segment includes M embeddings, each of which slides periodically according to a preset duration, where M is ≥ 1 and is a positive integer.

[0098] In this embodiment, for the audio streams with non-overlapping time, voiceprint feature extraction and similarity comparison are used to determine whether the clustering result of the current audio stream is the same as the speaker clustered in the previous sampling period, or whether they are different and belong to a new speaker. This allows the audio streams with non-overlapping time to be accurately aligned, achieving label alignment of the two audio clips.

[0099] Optionally, in another embodiment, if the current sampling period is greater than or equal to the second audio segment, there is no time-overlapping audio between the first audio segment and the second audio segment, and the above step 102, parsing the second audio segment according to the first clustering result to obtain the second clustering result, includes: according to the first clustering result, clustering the second audio segment according to the voiceprint feature matching principle to obtain the second clustering result.

[0100] Furthermore, the voiceprint matching principle is the same as the aforementioned method of parsing the second sub-segment to obtain the clustering result. The specific process can be found in the description of steps 301 to 303 of the above embodiment, which will not be repeated here.

[0101] Optionally, in the above embodiment, the third clustering result corresponding to the current audio stream is displayed, specifically including displaying the speaker ID. Figure 4In the example shown, after updating the second clustering result and obtaining the third clustering result, the speaker ID5 corresponding to the current speaker is displayed in the text description of the conference window. Furthermore, in the same conference scene, when speakers 1 to 5 use the same microphone to speak, the method of this embodiment can display the speaker IDs of different speakers after clustering in real time, thereby achieving the beneficial effect of identifying different speakers under the same microphone and improving the audio recording quality and user experience.

[0102] Optionally, in another embodiment, the method further includes: obtaining clustering results of historical audio segments, wherein the historical audio segments include the first audio segment, or further include more historical audio segments. Figure 3 In the example shown, audio is recorded starting from t=0, and clustering is triggered every time an ASR is detected by VAD to obtain a clustering result.

[0103] If a clustering operation is triggered at a certain time t, but the duration is less than N, assuming N is 300s, the current clustering time period is 240s, and 240s < 300s, then all embeddings collected in all historical time periods (240s) are clustered to obtain a clustering result, such as S0. When the historical time period is greater than or equal to 300s, the above-mentioned Figure 2 or Figure 5 For the method flow shown, please refer to the description of the aforementioned method embodiment for the specific process, which will not be repeated here.

[0104] In the embodiment of the present disclosure, a device for clustering audio segments is also provided. The device corresponds to the method for clustering audio segments in the above embodiment and is used to implement the above method for clustering audio segments in the process of real-time voice recording. Specifically, Figure 8 As shown, the device includes the following modules:

[0105] The acquiring unit 801 is configured to acquire a first clustering result corresponding to a first audio segment, where the first clustering result includes identity information of at least one speaker in the first audio segment and a timestamp corresponding to the identity information.

[0106] The collecting unit 802 is configured to obtain a second audio segment collected in a current sampling period.

[0107] The processing unit 803 is configured to parse the second audio segment according to the first clustering result to obtain a second clustering result.

[0108] The second audio segment and the first audio segment collected in the previous sampling period are both captured from the same real-time audio stream, and the second clustering result includes identity information of at least one speaker in the second audio segment and timestamp information corresponding to the identity information.

[0109] It should be understood that the above-mentioned device may also include other more or fewer units / modules, such as a sending unit, a computing unit, a storage unit, etc. This embodiment does not limit the structure of the device.

[0110] Optionally, in a specific implementation, the current sampling period is smaller than the second audio segment.

[0111] The collection unit 802 is further configured to obtain, based on the current sampling period being less than the second audio segment, a first sub-segment and a second sub-segment, respectively, where the first sub-segment is an audio stream of a temporally overlapping portion of the first audio segment and the second audio segment, and the second sub-segment is a remaining audio segment in the second audio segment after removing the first sub-segment.

[0112] The processing unit 803 is specifically configured to cluster the first sub-segments according to the first clustering result and the principle of the longest speaking time of the speaker to obtain a first sub-result; and, based on the first clustering result, cluster the second sub-segments according to the voiceprint feature matching principle to obtain a second sub-result.

[0113] Furthermore, processing unit 803 is further configured to use the Hungarian algorithm to search for an optimal mapping relationship in the parsed results based on the speaking duration of each speaker in the first clustering result and the speaker's corresponding ID, thereby obtaining the first sub-result. Under the optimal mapping relationship, the overlapping time of the clustered audio portions is the longest.

[0114] Optionally, in another specific embodiment, the processing unit 803 is further configured to obtain the first voiceprint feature corresponding to the second sub-segment and at least one second voiceprint feature in the first clustering result, each second voiceprint feature corresponding to a speaker of the first audio segment; perform a similarity comparison between the first voiceprint feature and the at least one second voiceprint feature; and if a target second voiceprint feature exists in the at least one second voiceprint feature, mark the first voiceprint feature and the target second voiceprint feature as being from the same speaker, thereby obtaining a second sub-result. The similarity between the target second voiceprint feature and the first voiceprint feature is greater than or equal to a threshold value.

[0115] Optionally, in another specific embodiment, the processing unit 803 is further used to detect that if the target second voiceprint feature does not exist, the first voiceprint feature is marked as corresponding to a new speaker to obtain a second sub-result, and the new speaker is different from all speakers clustered with the first audio segment.

[0116] Optionally, the above device further includes a computing unit, which Figure 8Not shown in the figure, the calculation unit is used to calculate the average value of the voiceprint vectors of the same speaker in each sampling period of the real-time audio stream before performing the similarity comparison to obtain the second voiceprint feature.

[0117] Optionally, in another specific implementation, the collecting unit 802 is further configured to obtain the second audio segment according to a sampling period when the voice activity detection (VAD) detects that the human voice ends.

[0118] Optionally, in another specific embodiment, the above device further includes a display unit.

[0119] The processing unit 803 is further configured to update the first clustering result based on the second clustering result to obtain a third clustering result for the current audio stream, and to display the third clustering result via the display unit. The current audio stream includes the first audio segment and the second audio segment, and the third clustering result includes at least one speaker identity information in the current audio stream and timestamp information corresponding to the speaker identity information.

[0120] It should be noted that, in the application, relational terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. In addition, the terms "including" and "having" in the above modules / units and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of modules in this application is merely a logical division, and there may be other division methods when implemented in actual applications.

[0121] The specific definition of the audio segment clustering device can be found in the definition of the audio segment clustering method described above and will not be repeated here. The various modules in the aforementioned audio segment clustering device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the aforementioned modules can be embedded in or independent of a processor in an electronic device in hardware form, or can be stored in a memory in the electronic device in software form, so that the processor can call and execute the corresponding operations of each of the aforementioned modules.

[0122] In another embodiment of the present application, an electronic device is provided. Figure 9 As shown, the electronic device includes a processor 901, a memory 902 and at least one interface 903, and the processor 901, the memory 902 and the at least one interface 903 can be connected via a bus.

[0123] The memory 902 stores a computer program that can be run on the processor 901. When the processor 901 executes the computer program, it can implement the steps of the audio segment clustering method in the above embodiment, for example: Figure 2 Alternatively, when the processor 901 executes the computer program, the functions of the modules / units of the audio segment clustering device in the above embodiment can be realized, such as Figure 8 All or part of the functions of the acquisition unit 801, the collection unit 802 and the processing unit 803 are not described here in detail to avoid repetition.

[0124] Furthermore, the processor 901 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 901 is the control center of the electronic device, connecting various parts of the entire electronic device using various interfaces and lines.

[0125] The memory 902 can be used to store the computer programs and / or modules, and the processor 901 implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory 902 and calling the data stored in the memory. In addition, the memory 902 can mainly include a program storage area and a data storage area. The program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as audio data, video data), etc.

[0126] The memory 902 may be integrated into the processor 901 or may be provided separately from the processor 901 .

[0127] The at least one interface 903 includes a communication interface and an input / output interface, such as a USB interface. The communication interface is used to enable communication between the electronic device and other devices, such as information transmission between a server or terminal device. The input / output interface is used to connect to external devices, such as a monitor / display, mouse, keyboard, microphone, and receiver.

[0128] It should be understood that the electronic device in this embodiment may further include more or fewer other components, such as at least one sensor.

[0129] In one embodiment, the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the audio segment clustering method in the above embodiment can be implemented, such as the above Figure 2 、 Figure 5 or Figure 6 Alternatively, when the computer program is executed by a processor, the functions of the modules / units of the apparatus for clustering audio segments in the above embodiment are realized.

[0130] In addition, it should be noted that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0131] Furthermore, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAM bus dynamic RAM (RDRAM).

[0132] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0133] The embodiments described above are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure, and should all be included in the scope of protection of the present disclosure.

Claims

1. A method for clustering audio clips, characterized in that: The method comprises: Obtaining a first clustering result corresponding to a first audio segment, the first clustering result including identity information of at least one speaker in the first audio segment and timestamp information corresponding to the identity information, wherein the first audio segment is an audio segment collected in a previous sampling period; Obtaining a second audio segment collected during the current sampling period, parsing the second audio segment to obtain a second clustering result, the second clustering result including identity information of at least one speaker in the second audio segment and timestamp information corresponding to the identity information; wherein there is an audio stream with a temporally overlapping portion between the second audio segment and the first audio segment; Compare the duration of the speaker's occupation in the temporal overlap between the first audio segment and the second audio segment in the first clustering result and the second clustering result, take the speaker with the longest duration as the speaker clustering result of the audio segment in the temporal overlap to obtain a first sub-result, and take the remaining portion of the second clustering result after removing the first sub-result as a second sub-result.

2. The method according to claim 1, characterized in that Comparing the duration of speakers occupying the time overlap portion of the first audio segment and the second audio segment in the first clustering result and the second clustering result, and using the speaker occupying the longest duration as the speaker clustering result of the audio segment in the time overlap portion, to obtain a first sub-result, including: Using the Hungarian algorithm, an optimal mapping relationship is found based on the speaking duration of each speaker in the first clustering result and the second clustering result and the speaker's corresponding identity ID. The time overlapping portion of the first audio clip and the second audio clip is aligned, and the identity ID of the speaker with the longest duration in the audio clip of the time overlapping portion is determined to obtain the first sub-result.

3. The method according to claim 1, characterized in that The second audio segment includes a first sub-segment and a second sub-segment, and the remaining portion of the second clustering result after removing the first sub-result is used as the second sub-result, including: Obtaining a first voiceprint feature corresponding to the second sub-segment and at least one second voiceprint feature in the first clustering result, where each second voiceprint feature corresponds to a speaker of the first audio segment; performing a similarity comparison between the first voiceprint feature and the at least one second voiceprint feature; If a target second voiceprint feature exists in the at least one second voiceprint feature, and the similarity between the target second voiceprint feature and the first voiceprint feature is greater than or equal to a threshold value, the first voiceprint feature and the target second voiceprint feature are marked as being from the same speaker, and the second sub-result is obtained.

4. The method according to claim 3, characterized in that The method further comprises: If the target second voiceprint feature does not exist, the first voiceprint feature is marked as corresponding to a new speaker to obtain the second sub-result, where the new speaker is different from any speaker in the first audio segment cluster.

5. The method according to claim 3, characterized in that The method further comprises: The second voiceprint feature is obtained by calculating the average value of the voiceprint vectors of the same speaker in each sampling period of the audio stream.

6. The method according to any one of claims 1 to 5, characterized in that The obtaining of the second audio segment collected in the current sampling period includes: When the human voice is detected by the voice activity detection (VAD), the second audio segment is acquired according to the sampling period.

7. The method according to any one of claims 1 to 5, characterized in that After comparing the speaker's duration in the time overlap portion of the first audio segment and the second audio segment in the first clustering result and the second clustering result to obtain the first sub-result and the second sub-result, the method further includes: A third clustering result is obtained based on the first clustering result and the second clustering result, wherein the third clustering result includes the second clustering result and at least part of the first clustering result, and the at least part of the first clustering result is the remaining clustering result after removing the audio segments with time overlapping parts.

8. An audio segment clustering device, characterized in that: The device comprises: an acquiring unit, configured to acquire a first clustering result corresponding to a first audio segment, the first clustering result including identity information of at least one speaker in the first audio segment and a timestamp corresponding to the identity information, the first audio segment being an audio segment collected in a previous sampling period; The acquisition unit is further configured to acquire a second audio segment acquired during the current sampling period; a processing unit configured to parse the second audio segment to obtain a second clustering result, compare the duration of speaker occupancy in a temporally overlapping portion of the first audio segment and the second audio segment between the first clustering result and the second clustering result, select the speaker with the longest duration as the speaker clustering result for the temporally overlapping portion of the audio segment to obtain a first sub-result, and select the remainder of the second clustering result after removing the first sub-result as a second sub-result; The second clustering result includes identity information of at least one speaker in the second audio segment and timestamp information corresponding to the identity information; and there is an audio stream with a time overlapping portion between the second audio segment and the first audio segment.

9. An electronic device, characterized in that: comprising a memory and a processor, wherein the memory and the processor are coupled; The memory is used to store computer program instructions; When the computer program instructions are read and executed by the processor, the method for clustering audio segments according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program; When the computer program is executed by a computer, the audio segment clustering method according to any one of claims 1 to 7 is implemented.

Images