Speaker identification methods, devices, electronic equipment and storage media

Abstract

The speaker determination method, apparatus, electronic device, and storage medium provided in this invention include: separating and obtaining audio files and image files from a video file to be identified; segmenting the audio files according to the start and end times corresponding to each audio segment to obtain multiple audio segment information; performing face recognition on the image files to obtain face information corresponding to each start and end time; extracting voiceprint features from each audio segment information to obtain voiceprint features corresponding to each audio segment information; and determining the speaker corresponding to each audio segment based on the face information and voiceprint features. This invention can accurately determine the speaker corresponding to each audio segment using only face information and voiceprint features, eliminating the need to establish a voiceprint feature database for retrieval and comparison, greatly reducing workload, difficulty, and limitations, and effectively improving the accuracy and efficiency of speaker determination.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a method, apparatus, electronic device and storage medium for determining the speaker. Background Technology

[0002] Voiceprint recognition, also known as speaker recognition, uses computer speech processing technology to analyze and process speech signals to determine the speaker's identity.

[0003] Current methods determine the speaker based on a pre-established voiceprint feature database through voiceprint retrieval and comparison. Specifically, the voiceprint features of each character in the video are extracted in advance to build a voiceprint feature database. Then, the voiceprint features corresponding to each audio segment (e.g., each sentence) in the video are extracted. Subsequently, each extracted voiceprint feature is compared one by one with the voiceprint features of each character in the voiceprint feature database. The most similar voiceprint feature is found from the database, and the person corresponding to that voiceprint feature is identified as the speaker of the audio segment. This method requires the speaker's voiceprint features to be collected and stored in the database in advance, which is tedious and complex. Secondly, the same person may dub multiple different roles, and the voiceprint features of the same person vary greatly under different emotions, volumes, and scenes. Considering these factors, the establishment of the voiceprint feature database is difficult and limited, which reduces the accuracy and efficiency of speaker identification. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a speaker determination method, apparatus, electronic device, and storage medium, which can effectively improve the accuracy and efficiency of speaker determination.

[0005] In a first aspect, embodiments of the present invention provide a speaker determination method, wherein the method includes:

[0006] Separate and extract audio and image files from the video file to be identified;

[0007] The audio file is segmented according to the start and end times of each audio segment to obtain multiple audio segment information, and the image file is subjected to face recognition to obtain face information corresponding to each start and end time.

[0008] Voiceprint features are extracted from each audio segment to obtain the voiceprint features corresponding to each audio segment.

[0009] The speaker corresponding to each audio segment is determined based on facial information and voiceprint features.

[0010] In one possible implementation, voiceprint features are extracted for each audio segment to obtain the voiceprint features corresponding to each audio segment, including:

[0011] Each audio segment is input into a pre-trained voiceprint recognition model, which then outputs the voiceprint features corresponding to each audio segment.

[0012] In one possible implementation, the speaker corresponding to each audio segment is determined based on facial information and voiceprint features, including:

[0013] Multiple voiceprint features are clustered to obtain multiple cluster categories;

[0014] For each cluster category, the duration of appearance of the target face information within the corresponding target start and end time period is calculated.

[0015] The target object corresponding to the target face information that appears for a duration exceeding a preset duration threshold is associated with the cluster category.

[0016] In one possible implementation, the method further includes:

[0017] Query whether each cluster category is associated with multiple target objects;

[0018] Clusters that are associated with multiple target objects are designated as the first cluster category, and clusters that are associated with one target object are designated as the second cluster category.

[0019] In the second cluster category, find the target second cluster category that is associated with the same specific target object as the first cluster category;

[0020] If the target second cluster category is found, calculate the cluster similarity between the first cluster center features of the first cluster category and the second cluster center features of the target second cluster category;

[0021] If the cluster similarity is greater than a preset cluster similarity threshold, a specific target object is associated with the first cluster category.

[0022] In one possible implementation, the method further includes:

[0023] If the cluster similarity is less than or equal to the preset cluster similarity threshold, and the target objects associated with the first cluster category include specific target objects and non-specific target objects, then the non-specific target objects will be associated with the first cluster category.

[0024] In one possible implementation, after determining the cluster category associated with multiple target objects as the first cluster category and the cluster category associated with one target object as the second cluster category, the method further includes:

[0025] The mean feature similarity is calculated based on the voiceprint features included in the first cluster category;

[0026] If the mean feature similarity is greater than the preset mean similarity, the step of searching in the second cluster category whether it is associated with the same specific target object as the first cluster category is executed.

[0027] In one possible implementation, the mean feature similarity is calculated based on the voiceprint features included in the first cluster category, including:

[0028] Calculate the feature similarity between each pair of voiceprint features included in the first cluster category;

[0029] The mean of the similarity of multiple features is calculated to obtain the mean of feature similarity.

[0030] In one possible implementation, the method further includes:

[0031] Each voiceprint feature is used as the first voiceprint feature, and the following operations are performed on each first voiceprint feature:

[0032] Calculate the first feature similarity between the first voiceprint feature and each second voiceprint feature in the third cluster category to which the first voiceprint feature belongs; wherein, the second voiceprint feature is not the first voiceprint feature;

[0033] Count the number of first feature similarities that exceed a preset similarity threshold;

[0034] If the number exceeds a preset threshold, the cluster category of the first voiceprint feature is determined to be the third cluster category;

[0035] If the number is less than or equal to the preset number threshold, the fourth cluster category of the first voiceprint feature is determined based on the first start and end time corresponding to the first voiceprint feature.

[0036] In one possible implementation, determining the fourth cluster category of the first voiceprint feature based on the first start and end time corresponding to the first voiceprint feature includes:

[0037] Obtain the start time and / or end time of the first start and end time;

[0038] The search duration range is determined based on the start time and / or end time.

[0039] Search for the third voiceprint feature within the search duration; wherein the start and end times corresponding to the third voiceprint feature are within the search duration, and the third voiceprint feature is not the first voiceprint feature;

[0040] Calculate the similarity between the second feature of the first voiceprint feature and each of the third voiceprint features;

[0041] The cluster category of the third voiceprint feature corresponding to the highest second feature similarity is determined as the fourth cluster category of the first voiceprint feature.

[0042] In one possible implementation, the search duration range is determined based on the start time and / or end time, including:

[0043] Starting from the start time, a first time range of a first preset duration is extracted in the direction of time less than the start time, and / or, starting from the end time, a second time range of a second preset duration is extracted in the direction of time greater than the end time.

[0044] The first time range and / or the second time range are determined as the search time range.

[0045] Secondly, embodiments of the present invention provide a speaker determination device, wherein the device includes:

[0046] The separation module is used to separate and obtain audio and image files from the video file to be identified;

[0047] The segmentation and recognition module is used to segment audio files according to the start and end times corresponding to each audio segment to obtain multiple audio segment information, and to perform face recognition on image files to obtain face information corresponding to each start and end time.

[0048] The extraction module is used to extract voiceprint features from each audio segment to obtain the voiceprint features corresponding to each audio segment.

[0049] The determination module is used to identify the speaker corresponding to each audio segment based on facial information and voiceprint features.

[0050] Thirdly, embodiments of the present invention provide an electronic device, comprising: a processor and a memory, wherein the processor is configured to execute a speaker determination program stored in the memory to implement the above-described speaker determination method.

[0051] Fourthly, embodiments of the present invention provide a storage medium, wherein the storage medium stores one or more programs, which can be executed by one or more processors to implement the speaker determination method described above.

[0052] The speaker determination method, apparatus, electronic device, and storage medium provided in this invention include: separating and obtaining audio files and image files from a video file to be identified; segmenting the audio files according to the start and end times corresponding to each audio segment to obtain multiple audio segment information; performing face recognition on the image files to obtain face information corresponding to each start and end time; extracting voiceprint features from each audio segment information to obtain voiceprint features corresponding to each audio segment information; and determining the speaker corresponding to each audio segment based on the face information and voiceprint features. This invention can accurately determine the speaker corresponding to each audio segment using only face information and voiceprint features, eliminating the need to establish a voiceprint feature database for retrieval and comparison, greatly reducing workload, difficulty, and limitations, and effectively improving the accuracy and efficiency of speaker determination. Attached Figure Description

[0053] Figure 1 A flowchart illustrating an embodiment of a speaker determination method provided by this invention;

[0054] Figure 2 A flowchart illustrating another embodiment of the speaker determination method provided by this invention;

[0055] Figure 3 A flowchart illustrating another embodiment of the speaker determination method provided by this invention;

[0056] Figure 4 A schematic diagram of a timeline provided for an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of a speaker determination device provided in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0061] This invention provides a method for determining the speaker, see [link to relevant documentation]. Figure 1 This is a flowchart illustrating an embodiment of a speaker determination method provided by an embodiment of the present invention. Figure 1 As shown, the process may include the following steps:

[0062] Step 101: Separate and obtain the audio file and image file from the video file to be identified;

[0063] Audio and image files can be extracted from video files using the multimedia video processing tool FFmpeg (Fast Forward Mpeg).

[0064] Typically, if the extracted audio file is a compressed file, it needs to be decompressed to obtain an uncompressed WAV (waveform audio file) or uncompressed PCM (Pulse Code Modulation) audio file. Uncompressed audio files can better extract the speaker's voiceprint features, which helps with subsequent voiceprint feature clustering and thus improves the speaker identification effect.

[0065] In this embodiment, the video file is a timeline video file such as a movie or TV drama video file or a speech video file. The separate audio file and the image file are time-corresponding, that is, the audio and the image are aligned one-to-one in time.

[0066] Step 102: Perform audio segmentation on the audio file according to the start and end times corresponding to each audio segment to obtain multiple audio segment information, and perform face recognition on the image file to obtain face information corresponding to each start and end time;

[0067] An audio segment is the audio generated after a speaker finishes speaking a sentence. The start and end times of an audio segment are the start and end times of the speaker finishing speaking a sentence. For audio files, the entire audio file can be segmented according to the start and end times of each audio segment using voiceprint tracking technology to obtain multiple audio segment information.

[0068] For image files, facial recognition can be performed on the image frames they contain to determine which speakers are included in the image file based on facial features. As an example, image frames can be extracted from the image file according to the start and end times corresponding to each audio segment, and facial recognition technology can be used to identify the facial information appearing within each start and end time period.

[0069] Step 103: Extract voiceprint features from each audio segment to obtain the voiceprint features corresponding to each audio segment.

[0070] In practical use, each audio segment can be input into a pre-trained voiceprint recognition model, which then outputs the voiceprint features corresponding to each audio segment.

[0071] Typically, the aforementioned voiceprint recognition model includes a classification layer and a multi-layered cascaded feature extraction network layer. This voiceprint recognition model can be trained through the following steps:

[0072] S1, Obtain the sample set, which includes multiple sample audios and the corresponding voiceprint tags for each sample audio;

[0073] S2, For each sample audio, the feature extraction network layer in the voiceprint recognition model extracts features layer by layer based on the sample audio, and obtains the feature information output by each feature extraction network layer.

[0074] S3, the classification layer performs voiceprint classification based on the feature information output by each feature extraction network layer, and obtains the sample voiceprint classification result corresponding to the feature information output by each feature extraction network layer;

[0075] S4. Based on the sample voiceprint classification results corresponding to the feature information output by each feature extraction network layer, the voiceprint label corresponding to the sample audio, the target computational amount corresponding to each feature extraction network layer, and the preset loss function, calculate the loss value of each feature extraction network layer for the sample audio.

[0076] The target computational cost corresponding to a feature extraction network layer is equal to the sum of the computational cost of the feature extraction network layer and the computational cost of the feature extraction network layers preceding it.

[0077] S5. Calculate the target loss value based on the loss values ​​of each feature extraction network layer for the sample audio.

[0078] S6. Adjust the parameters of the voiceprint recognition model in reverse according to the target loss value until the model training ends.

[0079] Step 104: Determine the speaker corresponding to each audio segment based on facial information and voiceprint features.

[0080] Since facial information and voiceprint features are unique characteristics of the speaker, they can be considered as specific identifying features. Therefore, the speaker determination method provided in this embodiment of the invention can accurately determine the speaker corresponding to each audio segment through facial information and voiceprint features, without the need to establish a voiceprint feature database for retrieval and comparison to determine the speaker. This greatly reduces the workload, difficulty and limitations, and effectively improves the accuracy and efficiency of speaker determination.

[0081] See Figure 2 This is a flowchart illustrating another embodiment of a speaker determination method provided by the present invention. Figure 2 The process shown above Figure 1Based on the illustrated process, describe how to determine the speaker corresponding to each audio segment based on facial information and voiceprint features. For example... Figure 2 As shown, the process may include the following steps:

[0082] Step 201: Separate and obtain the audio file and image file from the video file to be identified;

[0083] Step 202: Perform audio segmentation on the audio file according to the start and end times corresponding to each audio segment to obtain multiple audio segment information, and perform face recognition on the image file to obtain face information corresponding to each start and end time;

[0084] Step 203: Extract voiceprint features from each audio segment to obtain the voiceprint features corresponding to each audio segment.

[0085] The process of steps 201-203 in this embodiment can be referred to the process of steps 101-103 above, and will not be repeated here.

[0086] Step 204: Cluster the multiple voiceprint features to obtain multiple cluster categories;

[0087] In this embodiment, clustering methods such as spectral clustering, mean-shift clustering, and agglomerative hierarchical clustering can be used to cluster voiceprint features so that voiceprint features with similar characteristics are grouped into one cluster category. Thus, each cluster category can be treated as a whole unit to determine the speaker of that whole unit. This allows for the simultaneous determination of the speaker for multiple audio segments with similar voiceprint features, greatly improving the processing efficiency of determining the speaker for multiple audio segments.

[0088] Step 205: For each cluster category, calculate the duration of occurrence of the target face information within the corresponding target start and end time period in the cluster category;

[0089] Since each cluster category includes multiple voiceprint features, each cluster category corresponds to multiple target start and end times. These target start and end times are the start and end times corresponding to the audio segments of each voiceprint feature. For example, the voiceprint features in cluster category 1 include voiceprint feature 1 and voiceprint feature 2. The target start and end times corresponding to the audio segments of voiceprint feature 1 are from 5 minutes 18 seconds to 5 minutes 30 seconds, and the target start and end times corresponding to the audio segments of voiceprint feature 2 are from 8 minutes 20 seconds to 8 minutes 40 seconds. Therefore, the target start and end times corresponding to the cluster category are from 5 minutes 18 seconds to 5 minutes 30 seconds and from 8 minutes 20 seconds to 8 minutes 40 seconds.

[0090] Continuing from the previous example, for instance, target facial information appearing between 5 minutes 18 seconds and 5 minutes 30 seconds and between 8 minutes 20 seconds and 8 minutes 40 seconds includes target facial information a, target facial information b, and target facial information c. Calculate the total duration of these three target facial information appearing between 5 minutes 18 seconds and 5 minutes 30 seconds and between 8 minutes 20 seconds and 8 minutes 40 seconds.

[0091] Step 206: Associate the target object corresponding to the target face information whose appearance duration exceeds the preset duration threshold with the cluster category;

[0092] In this embodiment, the target face information that appears for a duration exceeding a preset duration threshold is considered to be the speaker corresponding to the voiceprint feature in the current cluster category.

[0093] Continuing from the previous example, assuming that the appearance duration of target face information a is 30 seconds, the appearance duration of target face information b is 20 seconds, and the appearance duration of target face information c is 10 seconds, and the preset duration threshold is set to 25 seconds, then the target object A corresponding to target face information a is associated with the cluster category, that is, the speaker of the audio segment corresponding to each voiceprint feature included in the cluster category is determined to be the above target object.

[0094] The preset duration threshold can be set according to actual needs and is not limited here. It can be understood that the larger the preset duration threshold is, the more accurate the target object associated with the cluster category will be, that is, the higher the accuracy of identifying the speaker of the audio segment.

[0095] In actual use, there may be multiple target face information that exceeds the preset duration threshold. The target objects corresponding to the multiple target face information are all associated with the cluster category. That is, an audio segment may identify multiple speakers. In order to further optimize the multiple associated speakers and exclude irrelevant speakers, after performing step 206 to determine the target objects associated with each cluster category, it is necessary to perform step 207 to determine whether each cluster category is associated with multiple target objects.

[0096] Step 207: Query whether each cluster category is associated with multiple target objects;

[0097] If each cluster category is found to be associated with only one target object, then steps 208 to 211 do not need to be executed, indicating that the speaker of each cluster category, including the audio segment corresponding to the voiceprint feature, is uniquely determined. If one or more cluster categories are found to be associated with multiple target objects, then steps 208 to 211 need to be executed to remove the incorrectly associated target objects.

[0098] Step 208: Determine the cluster category associated with multiple target objects as the first cluster category, and determine the cluster category associated with one target object as the second cluster category;

[0099] Typically, the first cluster category that associates multiple target objects may have poor clustering quality. Therefore, it is necessary to first check the clustering quality of the first cluster category. If the clustering quality is good, the associated target objects will be removed. If the clustering quality is poor, the first cluster category will be abandoned, that is, the target objects will not be associated with the first cluster category.

[0100] The quality of clustering can be measured by the mean of feature similarity. Specifically, the process of determining the quality of the first cluster category is as follows: calculate the mean of feature similarity based on the voiceprint features included in the first cluster category; if the mean of feature similarity is greater than the preset mean of similarity, perform the step of searching in the second cluster category whether it is a target second cluster category associated with the same specific target object as the first cluster category.

[0101] The process of calculating the mean feature similarity is as follows: calculate the feature similarity of each pair of voiceprint features included in the first cluster category; calculate the mean of multiple feature similarities to obtain the mean feature similarity.

[0102] In this embodiment, if the average feature similarity is greater than the preset average similarity, it indicates that the clustering quality of the first cluster category is acceptable. If the average feature similarity is less than or equal to the preset average similarity, it indicates that the clustering quality of the first cluster category is poor, and there is no need to associate the target object with this first cluster category. The preset average similarity can be set according to actual needs and is not limited here.

[0103] Step 209: In the second cluster category, find whether the target second cluster category is associated with the first cluster category for the same specific target object;

[0104] The purpose of this step is to determine whether multiple target objects associated with the first cluster category have a reference target object, that is, whether the same specific target object exists among the target objects associated with the second cluster category.

[0105] For example, the correspondence between the second cluster category and the associated target object is: cluster category 1 - target object A, cluster category 2 - target object B;

[0106] The first cluster category and its associated target object correspondence are as follows: Cluster category 4 - {Target object A, Target object D}, Cluster category 5 - {Target object E, Target object F}, Cluster category 6 - {Target object B, Target object D, Target object E}.

[0107] As shown above, cluster category 4 and cluster category 1 are both associated with the same specific target object A. Therefore, cluster category 1 is determined as the target second cluster category of cluster category 4. Cluster category 6 and cluster category 2 are both associated with the same specific target object B. Therefore, cluster category 2 is determined as the target second cluster category of cluster category 6. Since no reference target object was found for target objects E and F associated with cluster category 5, i.e., they are not the same target objects as target objects A and B, cluster category 5 is not associated with target objects in this embodiment. In other words, cluster category 5 is removed.

[0108] Step 210: If the target second cluster category is found, calculate the cluster similarity between the first cluster center feature of the first cluster category and the second cluster center feature of the target second cluster category;

[0109] Cluster center features refer to the average value of the voiceprint features included in the current cluster category. For example, if a cluster category includes 100 voiceprint features, and each voiceprint feature is an N-dimensional feature vector, such as 256-dimensional, then the cluster center feature is the average value of these 100 256-dimensional feature vectors.

[0110] In this embodiment, cosine similarity can be calculated based on the first cluster center features and the second cluster center features. This cosine similarity is used as the cluster similarity between the first cluster category and the second cluster category. In actual use, in addition to calculating cosine similarity, it is not limited to calculating Pearson correlation coefficient, Euclidean distance, etc. as cluster similarity to characterize the degree of similarity between the first cluster category and the second cluster category.

[0111] Step 211: If the cluster similarity is greater than the preset cluster similarity threshold, associate the specific target object with the first cluster category.

[0112] If the cluster similarity calculated in step 210 is greater than the preset cluster similarity threshold, it means that the category features of the first cluster category are similar to those of the second cluster category. In this case, the specific target object can be regarded as the speaker of the audio segment corresponding to the voiceprint feature in the first cluster category. For example, if the cluster similarity calculated between cluster category 4 and cluster category 1 is greater than the preset cluster similarity threshold, it means that the category features of cluster category 4 are similar to those of cluster category 1. Target object D can be removed, and target object A can be retained. Target object A can then be associated with cluster category 4 as the only associated object. Thus, it can be known that the speaker of the audio segment corresponding to the voiceprint feature in cluster category 4 is the target object A.

[0113] If the cluster similarity is less than or equal to the preset cluster similarity threshold, and the target objects associated with the first cluster category include specific target objects and non-specific target objects, then the non-specific target objects will be associated with the first cluster category.

[0114] In this embodiment, the number of non-specific target objects is unique, that is, at least one specific target object and one non-specific target object are associated in the first cluster category. After calculating the cluster similarity between the first cluster category and at least one target second cluster category, by comparing it with the preset cluster similarity threshold, it is found that at least one cluster similarity is less than or equal to the preset cluster similarity threshold. Then, the unique non-specific target object is determined as the speaker of the audio segment corresponding to the voiceprint feature in the first cluster category.

[0115] For example, if the cluster similarity calculated between cluster category 4 and cluster category 1 is less than or equal to the preset cluster similarity threshold, it means that the category features of cluster category 4 are not similar to those of cluster category 1. Target object A can be removed, and target object D can be retained. Target object D can then be used as the only associated object to be associated with cluster category 4. Thus, it can be determined that the speaker in cluster category 4 who is the speaker of the audio segment corresponding to the voiceprint feature is the target object D.

[0116] However, there is also a situation where cluster category 6 is associated. If the cluster similarity calculated between cluster category 6 and cluster category 2 is greater than the preset cluster similarity threshold, then target object B is associated with cluster category 6 as the only associated object. If the calculated cluster similarity is less than or equal to the preset cluster similarity threshold, then target object B is removed. After removal, since the associated objects also include target object D and target object E, there is no specific target object that matches them in either cluster category 1 or cluster category 2, in this embodiment, cluster category 6 is discarded and no target object is associated with cluster category 6, that is, cluster category 6 is removed.

[0117] The preset clustering similarity threshold can be set according to actual needs, and is not limited here.

[0118] The speaker determination method provided in this application embodiment can eliminate incorrectly associated target objects when multiple target objects are associated with a cluster category. This is achieved by comparing the cluster similarity calculated based on the cluster center feature of the cluster category with a preset cluster similarity threshold, thereby ensuring the uniqueness of the associated target objects and thus ensuring the accuracy of speaker determination.

[0119] To ensure the accuracy of speaker identification based on audio segments, after associating the target object, it is also necessary to check the correctness of each voiceprint feature clustering. See [link to relevant documentation]. Figure 3The following is a flowchart of another embodiment of the speaker determination method provided by the present invention, as shown in the figure. Figure 3 As shown, the process may include the following steps:

[0120] Step 301: Separate and obtain the audio file and image file from the video file to be identified;

[0121] Step 302: Perform audio segmentation on the audio file according to the start and end times corresponding to each audio segment to obtain multiple audio segment information, and perform face recognition on the image file to obtain face information corresponding to each start and end time;

[0122] Step 303: Extract voiceprint features from each audio segment to obtain the voiceprint features corresponding to each audio segment.

[0123] Step 304: Cluster the multiple voiceprint features to obtain multiple cluster categories;

[0124] Step 305: For each cluster category, calculate the duration of occurrence of the target face information within the corresponding target start and end time period in the cluster category;

[0125] Step 306: Associate the target object corresponding to the target face information whose appearance duration exceeds the preset duration threshold with the cluster category;

[0126] The process of steps 301-306 in this embodiment can be referred to the process of steps 201-206 above, and will not be repeated here.

[0127] Step 307: Take each voiceprint feature as the first voiceprint feature, and perform the operations from step 308 to step 311 for each first voiceprint feature:

[0128] Step 308: Calculate the first feature similarity between the first voiceprint feature and each second voiceprint feature in the third cluster category to which the first voiceprint feature belongs;

[0129] Among them, the second voiceprint feature is not the first voiceprint feature, that is, the second voiceprint feature is a voiceprint feature other than the first voiceprint feature.

[0130] The calculated first feature similarity can be cosine similarity, Pearson correlation coefficient, Euclidean distance, etc., and no specific limitation is made here.

[0131] Step 309: Count the number of first feature similarities that exceed the preset similarity threshold;

[0132] The preset similarity threshold can be set according to actual needs and is not limited here.

[0133] Step 310: If the number is greater than the preset number threshold, determine the cluster category of the first voiceprint feature as the third cluster category;

[0134] If the number is greater than the preset threshold, it means that the first voiceprint feature is similar to most of the second voiceprint features. This first voiceprint feature is likely to belong to the current third cluster category. Therefore, it can be concluded that the first voiceprint feature is correctly clustered, which further indicates that the speaker of the audio segment of the first voiceprint feature is the target object associated with the third cluster category.

[0135] Step 311: If the number is less than or equal to a preset number threshold, determine the fourth cluster category of the first voiceprint feature based on the first start and end time corresponding to the first voiceprint feature.

[0136] If the number exceeds the preset threshold, it indicates that the first voiceprint feature is not similar to most of the second voiceprint features. This first voiceprint feature is likely not in the current third cluster category. Therefore, the first voiceprint feature is incorrectly clustered and needs to be re-clustered to determine the cluster category to which the first voiceprint feature should belong.

[0137] The process of determining the fourth cluster category to which the first voiceprint feature belongs can be achieved through steps A1 to A5:

[0138] Step A1: Obtain the start time and / or end time of the first start and end time.

[0139] Step A2: Determine the search duration range based on the start time and / or end time;

[0140] Typically, the speaker with the first voiceprint feature has a higher probability of speaking within the search time range. Therefore, within this search time range, there is a high probability of finding voiceprint features that are similar to the first voiceprint feature, so as to further determine the fourth cluster category to which the first voiceprint feature should belong.

[0141] Specifically, the process of determining the search duration range is as follows: taking the starting time as the starting point, a first duration range of a first preset duration is extracted in the direction of time less than the starting time, and / or, taking the ending time as the starting point, a second duration range of a second preset duration is extracted in the direction of time greater than the ending time; the first duration range and / or the second duration range are determined as the search duration range.

[0142] For ease of understanding, Figure 4 A schematic diagram of a timeline is shown, such as... Figure 4 As shown, the total duration of this timeline is 10 minutes, indicating that the total duration of the video file is 10 minutes. Figure 4As can be seen, the value of the time increases as you move to the right. In this embodiment, we take the starting time of the first start and end time of the audio segment corresponding to the first voiceprint feature as 4 minutes and the ending time as 5 minutes as an example. If the first preset duration and the second preset duration are both 1 minute, then the range of the first duration is 3 minutes to 4 minutes and the range of the second duration is 5 minutes to 6 minutes. Therefore, the range of the search duration is 3 minutes to 4 minutes and 5 minutes to 6 minutes.

[0143] In practical use, the search duration range can also be determined based solely on the start time. Continuing the previous example, the determined search duration range is 3-4 minutes. If the search duration range is determined solely based on the end time, it is 5-6 minutes.

[0144] The first preset duration and the second preset duration can be set to the same duration or different durations, and this is not limited here.

[0145] Step A3: Search for the third voiceprint feature within the search duration range;

[0146] Among them, the start and end times corresponding to the third voiceprint feature are within the search duration range, and the third voiceprint feature is not the first voiceprint feature.

[0147] If no third voiceprint feature is found within the search time range, the first voiceprint feature can be discarded.

[0148] Step A4: Calculate the similarity between the second feature of the first voiceprint feature and each of the third voiceprint features;

[0149] Step A5: Determine the cluster category of the third voiceprint feature corresponding to the largest second feature similarity as the fourth cluster category of the first voiceprint feature.

[0150] As can be seen from the above, if the first voiceprint feature determines the fourth cluster category, then the speaker of the audio segment corresponding to the first voiceprint feature is the target object associated with the cluster category where the third voiceprint feature is located.

[0151] The speaker determination method provided in this application requires further detection of the correctness of each voiceprint feature cluster after the clustering category completes the association of the target object, so as to ensure the correctness of the target object associated with each voiceprint feature, and thus ensure the accuracy of speaker determination.

[0152] Corresponding to the above method embodiments, this embodiment provides a speaker determination device, see [link to previous document]. Figure 5 The diagram shows a speaker determination device, which includes:

[0153] Separation module 51 is used to separate and obtain audio files and image files from the video file to be identified;

[0154] The segmentation and recognition module 52 is used to segment the audio file according to the start and end times corresponding to each audio segment to obtain multiple audio segment information, and to perform face recognition on the image file to obtain face information corresponding to each start and end time.

[0155] Extraction module 53 is used to extract voiceprint features from each audio segment information to obtain the voiceprint features corresponding to each audio segment information;

[0156] The determination module 54 is used to determine the speaker corresponding to each audio segment based on facial information and voiceprint features.

[0157] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 6 The illustrated electronic device 500 includes at least one processor 501, a memory 502, at least one network interface 504, and other user interfaces 503. The various components in the electronic device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to implement communication between these components. In addition to a data bus, the bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 6 The general designated all buses as Bus System 505.

[0158] The user interface 503 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0159] It is understood that the memory 502 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchlink Dynamic Random Access Memory (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0160] In some implementations, memory 502 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 5021 and application program 5022.

[0161] The operating system 5021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 5022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 5022.

[0162] In this embodiment of the invention, the processor 501 executes the method steps provided in each method embodiment by calling the program or instructions stored in the memory 502, specifically the program or instructions stored in the application program 5022.

[0163] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in the form of software. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 502. Processor 501 reads the information in memory 502 and, in conjunction with its hardware, completes the steps of the above method.

[0164] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0165] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0166] The electronic device provided in this embodiment may be as follows: Figure 6The electronic device shown can perform the following: Figure 1-3 The speaker determines all the steps of the method, thereby achieving Figure 1-3 For details on the technical effects of the speaker determination method shown, please refer to [link / reference]. Figure 1-3 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0167] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; the memory may also include combinations of the above types of memory.

[0168] When one or more programs in the storage medium can be executed by one or more processors to implement the speaker determination method described above.

[0169] The processor is used to execute the speaker determination program stored in the memory to implement the steps of the speaker determination method.

[0170] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0171] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0172] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for determining the speaker, characterized in that, The method includes: Separate and extract audio and image files from the video file to be identified; The audio file is segmented according to the start and end times of each audio segment to obtain multiple audio segment information, and the image file is subjected to face recognition to obtain face information corresponding to each start and end time; Voiceprint features are extracted from each audio segment information to obtain the voiceprint features corresponding to each audio segment information; The speaker corresponding to each audio segment is determined based on the facial information and the voiceprint features; The step of determining the speaker corresponding to each audio segment based on the facial information and the voiceprint features includes: Multiple voiceprint features are clustered to obtain multiple cluster categories; For each cluster category, the duration of occurrence of the target face information within the corresponding target start and end time period in the cluster category is calculated. Associate the target object corresponding to the target face information whose appearance duration exceeds a preset duration threshold with the clustering category; The method further includes: Query whether each cluster category is associated with multiple target objects; The cluster category associated with multiple target objects is determined as the first cluster category, and the cluster category associated with one target object is determined as the second cluster category; In the second cluster category, find whether the target second cluster category is associated with the same specific target object as the first cluster category; If the target second cluster category is found, the cluster similarity between the first cluster center feature of the first cluster category and the second cluster center feature of the target second cluster category is calculated; If the cluster similarity is greater than a preset cluster similarity threshold, the specific target object is associated with the first cluster category.

2. The method according to claim 1, characterized in that, The step of extracting voiceprint features for each audio segment to obtain the voiceprint features corresponding to each audio segment includes: Each audio segment information is input into a pre-trained voiceprint recognition model, and the voiceprint recognition model outputs the voiceprint features corresponding to each audio segment information.

3. The method according to claim 1, characterized in that, The method further includes: If the cluster similarity is less than or equal to the preset cluster similarity threshold, and the target objects associated with the first cluster category include the specific target object and non-specific target objects, then the non-specific target objects are associated with the first cluster category.

4. The method according to claim 1, characterized in that, After determining the cluster category associated with multiple target objects as the first cluster category and the cluster category associated with one target object as the second cluster category, the method further includes: The mean feature similarity is calculated based on the voiceprint features included in the first cluster category; If the average feature similarity is greater than a preset average similarity, the step of searching in the second cluster category whether it is associated with the same specific target object as the first cluster category is executed.

5. The method according to claim 4, characterized in that, The step of calculating the mean feature similarity based on the voiceprint features included in the first cluster category includes: Calculate the feature similarity of each pair of voiceprint features included in the first cluster category; The mean of the multiple feature similarities is calculated to obtain the feature similarity mean.

6. The method according to claim 1, characterized in that, The method further includes: Each voiceprint feature is used as the first voiceprint feature, and the following operations are performed on each of the first voiceprint features: Calculate the first feature similarity between the first voiceprint feature and each second voiceprint feature in the third cluster category to which the first voiceprint feature belongs; wherein, the second voiceprint feature is not the first voiceprint feature; Count the number of first feature similarities that exceed a preset similarity threshold; If the number is greater than a preset number threshold, the cluster category of the first voiceprint feature is determined to be the third cluster category; If the quantity is less than or equal to the preset quantity threshold, the fourth cluster category to which the first voiceprint feature belongs is determined based on the first start and end time corresponding to the first voiceprint feature.

7. The method according to claim 6, characterized in that, The step of determining the fourth cluster category of the first voiceprint feature based on the first start and end time corresponding to the first voiceprint feature includes: Obtain the start time and / or end time of the first start and end time; The search duration range is determined based on the start time and / or the end time; Search for a third voiceprint feature within the search duration; wherein the start and end times corresponding to the third voiceprint feature are within the search duration, and the third voiceprint feature is not the first voiceprint feature; Calculate the similarity between the second feature of the first voiceprint feature and each of the third voiceprint features; The cluster category of the third voiceprint feature corresponding to the largest second feature similarity is determined as the fourth cluster category of the first voiceprint feature.

8. The method according to claim 7, characterized in that, Determining the search duration range based on the start time and / or the end time includes: Starting from the starting time, a first time range of a first preset duration is extracted in the direction of time less than the starting time, and / or, starting from the ending time, a second time range of a second preset duration is extracted in the direction of time greater than the ending time. The first duration range and / or the second duration range are determined as the search duration range.

9. A speaker identification device, characterized in that, The device includes: The separation module is used to separate and obtain audio and image files from the video file to be identified; The segmentation and recognition module is used to segment the audio file according to the start and end times corresponding to each audio segment to obtain multiple audio segment information, and to perform face recognition on the image file to obtain face information corresponding to each start and end time; The extraction module is used to extract voiceprint features from each of the audio segments to obtain the voiceprint features corresponding to each of the audio segments. The determination module is used to determine the speaker corresponding to each audio segment based on the facial information and the voiceprint features; The determining module is further configured to: perform clustering processing on the multiple voiceprint features to obtain multiple cluster categories; For each cluster category, the duration of occurrence of the target face information within the corresponding target start and end time period in the cluster category is calculated. Associate the target object corresponding to the target face information whose appearance duration exceeds a preset duration threshold with the clustering category; The device further includes: The query module is used to query whether each cluster category is associated with multiple target objects; The category determination module is used to determine the cluster category associated with multiple target objects as the first cluster category, and to determine the cluster category associated with one target object as the second cluster category; The search module is used to search within the second cluster category for a target second cluster category that is associated with the first cluster category for the same specific target object; The calculation module is used to calculate the cluster similarity between the first cluster center feature of the first cluster category and the second cluster center feature of the target second cluster category when the target second cluster category is found; The association module is used to associate the specific target object with the first cluster category when the cluster similarity is greater than a preset cluster similarity threshold.

10. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a speaker determination program stored in the memory to implement the speaker determination method according to any one of claims 1 to 8.

11. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the speaker determination method according to any one of claims 1 to 8.

Images