Line speaker recognition method and related device

Abstract

The invention discloses a line speaker recognition method and a related device, and the method comprises the steps: firstly obtaining a video file and a line list, then inputting a target text fragment in the line list and a context text fragment into a large language model, determining N target texts with complete semantics, and carrying out the recognition of a line speaker through the N target texts; and finally, P speakers used for expressing the N target texts are determined in the video text, identities of the P speakers are determined, speaker identities corresponding to the N target texts are obtained, the target texts with complete semantics serve as a whole, speaker identity recognition corresponding to each target text is achieved in the video file, and the recognition efficiency is improved. Compared with a mode of recognizing the line speaker by taking each text segment as an independent recognition unit in the related technology, the method and the device have the advantages that too short duration or wrong recognition of the line corresponding to the same speaker can be avoided, and the accuracy of line speaker recognition is effectively improved.

Description

Technical Field

[0001] This application relates to the field of multimedia content processing and analysis technology, specifically to a method and related apparatus for identifying the speaker in a dialogue. Background Technology

[0002] Speaker recognition technology analyzes speech signals and facial features to identify the speaker delivering lines. It can be applied to automated processing of video content such as films, TV dramas, variety shows, and short videos. Specific applications include: intelligent subtitle creation: automatically labeling each line of dialogue with its corresponding speaker during subtitle generation, improving the user viewing experience; content analysis and retrieval: accurately calculating the amount of dialogue and appearance time for each character, providing data support for script analysis, character importance assessment, and content trend discovery; automated editing and indexing: quickly locating dialogue segments of specific characters based on speaker information, providing technical convenience for video editing, generating highlight clips, and indexing video content; and overseas translation: automatically translating the original video into multiple languages ​​based on speaker information, etc.

[0003] In related technologies, for the list of lines corresponding to the video content, the voiceprint features of each line can be extracted, and then clustering can be performed based on the voiceprint features. Audios with the same or similar voiceprint features are divided into the same cluster. Then, based on the face recognition results obtained from the video frame corresponding to the audio at a certain time point in the cluster, the speaker's identity information is determined by combining the identity information in the face database.

[0004] However, the aforementioned speech recognition technology lacks sufficient accuracy in practical application scenarios. Summary of the Invention

[0005] In view of this, this application provides a method and related apparatus for identifying the speaker of a dialogue. By using a large language model to determine the complete semantic information of a target text segment, the audio corresponding to the target text with complete semantics is treated as a whole, and the speaker identity information corresponding to the target text segment is identified, which effectively improves the accuracy of the identification results.

[0006] To solve the above problems, the technical solution provided in this application is as follows:

[0007] On the one hand, this application provides a method for identifying the speaker in a dialogue, the method comprising:

[0008] Obtain a video file and a list of dialogues, wherein the list of dialogues includes M text segments spoken by the speaker in the video file;

[0009] The target text fragments in the dialogue list and the context text fragments of the target text fragments are input into the large language model to determine N target texts with complete semantics. The target text fragments are one or more of the M text fragments. Each of the N target texts includes multiple consecutive text fragments in the dialogue list, and N>0.

[0010] In the video file, P speakers are identified to represent the N target texts, and the speaker identities of the P speakers are determined to obtain the speaker identities corresponding to the N target texts respectively, where N≥P.

[0011] In one possible implementation, determining P speakers in the video file to represent the N target texts, and determining the speaker identities of the P speakers to obtain the speaker identities corresponding to the N target texts respectively includes:

[0012] Identify N audio files in the video file that represent the N target texts, and extract audio features from each of the N audio files to obtain N audio features;

[0013] Based on the N audio features, the N audio files are clustered to obtain P clusters;

[0014] The faces of each speaker in the video file are extracted and compared with a pre-built face database to determine the identity of each speaker;

[0015] Align the first audio within the first cluster of the P clusters with the time each speaker appears in the video file to determine the identity of the first speaker corresponding to the first audio, and use the identity of the first speaker as the identity of the speaker corresponding to the first cluster to obtain the speaker identities corresponding to the N target texts respectively.

[0016] In one possible implementation, after obtaining the video file and the list of lines, the method further includes:

[0017] Based on the video file and the dialogue list, the M text segments are identified to obtain Q clusters. Each of the Q clusters is labeled with a speaker identity. Each cluster includes audio corresponding to multiple text segments from the M text segments. The speaker identity includes unknown roles, which are speakers whose identities cannot be confirmed. The target text segment is the text segment corresponding to the cluster labeled with the unknown role.

[0018] The step of determining P speakers in the video file to represent the N target texts, and determining the speaker identities of the P speakers to obtain the speaker identities corresponding to the N target texts respectively includes:

[0019] Identify N audio files in the video file that represent the N target texts, and extract audio features from each of the N audio files to obtain N audio features;

[0020] Based on the N audio features, the N audio files are clustered to obtain P target clusters;

[0021] The P target clusters and the Q clusters are subjected to secondary clustering to obtain multiple new clusters. If the first target cluster and the second cluster form a new cluster after the secondary clustering, the speaker identity corresponding to the new cluster is marked as the speaker identity of the second cluster. The first target cluster is one of the P target clusters, the second cluster is one of the Q clusters, and the new cluster is one of the multiple new clusters.

[0022] In one possible implementation, determining N audio clips in the video file used to represent the N target texts, and extracting audio features from each of the N audio clips to obtain N audio features includes:

[0023] Extract the audio information from the video file;

[0024] Identify the N audio clips in the audio information used to represent the N target texts;

[0025] For the first audio among the N audios, if the duration of the first audio is less than a preset duration threshold, then the audio at the beginning or end of the first audio in the audio information with a preset duration is added to the first audio to obtain N second audios;

[0026] The audio features of the N second audio frequencies are extracted respectively to obtain N audio features.

[0027] In one possible implementation, the target text fragment also includes text fragments corresponding to lines in the dialogue list whose duration is less than a preset threshold.

[0028] In one possible implementation, the method further includes:

[0029] Multiple audio features within the new clusters labeled as unknown roles are identified to obtain multiple sound attributes, which are used to identify different speakers;

[0030] Based on the multiple sound attributes, the speaker identities of the multiple audio features are re-labeled.

[0031] In another aspect, this application provides a speech recognition device, which includes an acquisition unit and a determination unit:

[0032] The acquisition unit is used to acquire a video file and a list of lines, wherein the list of lines includes M text segments expressed by the speaker in the video file;

[0033] The determining unit is used to input the target text fragments in the dialogue list and the context text fragments of the target text fragments into the large language model to determine N target texts with complete semantics. The target text fragments are one or more of the M text fragments. Each target text in the N target texts includes multiple consecutive text fragments in the dialogue list, and N>0.

[0034] The determining unit is further configured to determine P speakers in the video file that represent the N target texts, and to determine the speaker identities of the P speakers, thereby obtaining the speaker identities corresponding to the N target texts respectively, wherein N≥P.

[0035] In one possible implementation, the determining unit is configured to:

[0036] Identify N audio files in the video file that represent the N target texts, and extract audio features from each of the N audio files to obtain N audio features;

[0037] Based on the N audio features, the N audio files are clustered to obtain P clusters;

[0038] The faces of each speaker in the video file are extracted and compared with a pre-built face database to determine the identity of each speaker;

[0039] Align the first audio within the first cluster of the P clusters with the time each speaker appears in the video file to determine the identity of the first speaker corresponding to the first audio, and use the identity of the first speaker as the identity of the speaker corresponding to the first cluster to obtain the speaker identities corresponding to the N target texts respectively.

[0040] In one possible implementation, the device further includes an identification unit for:

[0041] Based on the video file and the dialogue list, the M text segments are identified to obtain Q clusters. Each of the Q clusters is labeled with a speaker identity. Each cluster includes audio corresponding to multiple text segments from the M text segments. The speaker identity includes unknown roles, which are speakers whose identities cannot be confirmed. The target text segment is the text segment corresponding to the cluster labeled with the unknown role.

[0042] The determining unit is used for:

[0043] Identify N audio files in the video file that represent the N target texts, and extract audio features from each of the N audio files to obtain N audio features;

[0044] Based on the N audio features, the N audio files are clustered to obtain P target clusters;

[0045] The P target clusters and the Q clusters are subjected to secondary clustering to obtain multiple new clusters. If the first target cluster and the second cluster form a new cluster after the secondary clustering, the speaker identity corresponding to the new cluster is marked as the speaker identity of the second cluster. The first target cluster is one of the P target clusters, the second cluster is one of the Q clusters, and the new cluster is one of the multiple new clusters.

[0046] In one possible implementation, the determining unit is configured to:

[0047] Extract the audio information from the video file;

[0048] Identify the N audio clips in the audio information used to represent the N target texts;

[0049] For the first audio among the N audios, if the duration of the first audio is less than a preset duration threshold, then the audio at the beginning or end of the first audio in the audio information with a preset duration is added to the first audio to obtain N second audios;

[0050] The audio features of the N second audio frequencies are extracted respectively to obtain N audio features.

[0051] In one possible implementation, the target text fragment also includes text fragments corresponding to lines in the dialogue list whose duration is less than a preset threshold.

[0052] In one possible implementation, the method further includes a labeling unit;

[0053] The identification unit is used to identify multiple audio features within the new clusters labeled as unknown roles in the multiple new clusters, and obtain multiple sound attributes, which are used to identify different speakers;

[0054] The annotation unit is used to re-annotate the speaker identity of the multiple audio features based on the multiple sound attributes.

[0055] In another aspect, this application provides a computer device, which includes a processor and a memory:

[0056] The memory is used to store computer programs;

[0057] The processor is configured to execute the method described in any of the above-described embodiments according to the computer program.

[0058] In another aspect, this application provides a computer-readable storage medium for storing a computer program that, when executed by a computer device, implements the method described in any of the above-mentioned embodiments.

[0059] In another aspect, this application provides a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform any of the methods described above.

[0060] As can be seen from the above technical solution, this solution acquires a video file and a list of lines, where the list of lines includes M text segments spoken by speakers in the video file. Then, the target text segments in the list of lines and their context text segments are input into a large language model to determine N target texts with complete semantics. Finally, P speakers used to express the N target texts are identified in the video text, and the identities of the P speakers are determined, thus obtaining the speaker identities corresponding to the N target texts. Since the large language model can consider the semantic information of the context, it obtains target texts with complete semantics. Then, the target texts with complete semantics are treated as a whole, and the speaker identity corresponding to each target text is identified in the video file. Compared with the related technology that treats each text segment as an independent recognition unit for dialogue speaker identification, this method can avoid misidentification of lines corresponding to the same speaker due to excessively short duration, thereby effectively improving the accuracy of dialogue speaker identification. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 One of the flowcharts for a speech recognition method provided in this application embodiment;

[0063] Figure 2 A second schematic flowchart illustrating a speech recognition method for dialogue provided in an embodiment of this application;

[0064] Figure 3 This is a schematic diagram of a speech recognition device provided in an embodiment of this application. Detailed Implementation

[0065] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0066] As described in the background section, related technologies treat the start and end times of each line as the start and end times of the corresponding audio, extract the voiceprint features of each line separately, and then perform clustering based on the voiceprint features to determine the speaker's identity information. However, this ignores the semantic information of the line's context and cannot identify adjacent lines with complete semantics as a whole, resulting in a low accuracy rate in identifying the speaker's identity.

[0067] The applicant argues that lines with complete semantics are delivered by the same speaker, and that punctuation marks (such as periods, question marks, and exclamation marks) serve as natural pause markers in language, implying the speaker's complete intention and the end of vocal units. Furthermore, large language models can accurately understand the semantic boundary information represented by these punctuation marks, providing audio segmentation with a much more reliable recognition boundary than the start and end points of lines.

[0068] This application provides a method and related apparatus for identifying the speaker in a video script. By treating the target text with complete semantics as a whole, the speaker identity corresponding to each target text is identified in the video file. Compared with the related technology that identifies the speaker in a video script by treating each text segment as an independent identification unit, this method can avoid misidentification of lines corresponding to the same speaker due to excessively short duration, thereby effectively improving the accuracy of the speaker identification.

[0069] The solutions provided in this application relate to the field of multimedia content processing and analysis technology, and are specifically illustrated through the following embodiments.

[0070] See Figure 1 The diagram shown is a flowchart illustrating a speech recognition method for dialogue provided in an embodiment of this application. In this embodiment, it can be executed by a computer device.

[0071] S101: Obtain the video file and the list of lines.

[0072] Video files refer to multimedia files that include audio and video content, and are produced by multiple speakers.

[0073] A dialogue list refers to the text content that matches the audio in video files such as movies, TV dramas, variety shows, and short videos. It is displayed as visual subtitles in the video playback interface and does not include punctuation. The dialogue list consists of M text segments spoken by multiple speakers in the video file.

[0074] Computer devices acquire a list of dialogues and a video file to determine the speaker's identity corresponding to a text segment in the dialogue list within the video file.

[0075] S102: Input the target text fragments from the dialogue list and the context text fragments of the target text fragments into the large language model to determine N target texts with complete semantics.

[0076] The target text fragment is one or more of the M text fragments, and each of the N target texts includes multiple consecutive text fragments from the dialogue list, where N>0.

[0077] The contextual dialogue of the target text fragment refers to the dialogue in the dialogue list that is closely related to the target text fragment and can provide background information or semantic connection. It is similar to the target text fragment in the time domain, such as the sentences before and after the target text fragment.

[0078] The large language model is a pre-trained model. By fine-tuning an existing large model, it can be made to focus on recognizing the semantic boundary information of text, thereby improving the accuracy of judgment. This application does not impose restrictions on the architecture or specific training method of the large language model.

[0079] The following is an example illustrating fine-tuning of a large language model:

[0080] First, training data is constructed, including a list of lines and labels. The labels refer to the target text with complete semantics corresponding to multiple text fragments in the list of lines, which can be obtained through manual annotation.

[0081] The training data is then input into the large language model. The large language model is fine-tuned by generating prompts (such as identifying semantic boundaries based on the input list of lines and outputting the target text corresponding to the text fragment based on the semantic boundaries). The difference between the output and the label is continuously reduced. When the difference is reduced to a certain threshold, the fine-tuned large language model is obtained.

[0082] The semantic boundary is used to identify the boundary where semantic meaning ends. This application does not impose any restrictions on this; for example, it can be represented by the type of punctuation mark. By inputting the target text fragment and the context dialogue into a large language model, the type of punctuation mark (such as period, question mark, exclamation mark, comma, no punctuation, etc.) at the beginning and end of each text fragment can be determined. The semantic boundary information of the target text fragment can be determined by the type of punctuation mark. For example, the types of some punctuation marks, such as period, question mark, and exclamation mark, can be preset as the boundary where semantic meaning ends.

[0083] Based on the semantic boundary recognition of the large language model, the boundary marking the end of semantics can be found forward and backward respectively, with the target text segment as the reference. The lines within the boundary are the target text with complete semantics corresponding to the target text segment.

[0084] As an example, the target text fragment is "you you", and the context includes the preceding sentence "sorry" and the following sentence "you're making me so angry". Inputting the target text fragment and the context into the large language model, the output "sorry. you, you, you're making me so angry." can be obtained as the target text with complete semantic boundaries, which is "you, you, you're making me so angry".

[0085] Therefore, by taking the target text fragment as the dialogue to be identified, and obtaining N corresponding target texts with complete semantics through a large language model, the target text with complete semantics can be used as the smallest identification unit for dialogue speaker identification.

[0086] S103: Determine P speakers in the video file to represent N target texts, and determine the speaker identities of the P speakers to obtain the speaker identities corresponding to the N target texts respectively.

[0087] Where, N P. In other words, a speaker can express multiple target texts with complete semantics.

[0088] Specifically, based on the audio corresponding to N target texts, the speaker's face is determined in the video file, and then compared with the face database to obtain the speaker's identity corresponding to the target text.

[0089] Therefore, by using a large language model, the semantic information of the context can be taken into account, and the semantic boundary information of the dialogue list can be provided. Since the target text with complete semantics corresponds to the same speaker, the target text with complete semantics is treated as a whole to realize the identification of the speaker of the dialogue. This can avoid misidentification of dialogues that are too short or dialogues corresponding to the same speaker, and effectively improve the accuracy of speaker identification.

[0090] In one possible implementation, N>1, then S103 includes:

[0091] A1: Identify the N audio files in the video file that are used to represent N target texts, and extract audio features from each of the N audio files to obtain N audio features.

[0092] A2: Cluster N audio files based on N audio features to obtain P clusters.

[0093] A3: Extract the face of each speaker from the video file and compare it with a pre-built face database to determine the identity of each speaker.

[0094] A4: Align the first audio within the first cluster of P clusters with the time each speaker appears in the video file to determine the identity of the first speaker corresponding to the first audio, and use the identity of the first speaker as the identity of the speaker corresponding to the first cluster to obtain the speaker identities corresponding to N target texts.

[0095] This application does not impose specific limitations on the method of audio feature extraction. For example, audio feature extraction can be performed by constructing a machine learning model using a convolutional neural network or a recurrent neural network.

[0096] Specifically, N audio clips are identified in the video file to represent N target texts. Audio features are extracted from each of the N audio clips to obtain N audio features. Then, a clustering algorithm (such as K-nearest neighbors algorithm, SVM algorithm, etc.) is used to divide the audio features with similarity greater than a preset threshold into the same clusters to obtain P clusters. Then, the face of each speaker in the video file is extracted and compared with a pre-built face database to determine the identity of each speaker (for example, the identity of a speaker who fails to match the face database is considered an unknown role). Then, for the first cluster, the speaker whose identity appears for the longest time in the video progress corresponding to the first audio clip, or the speaker whose identity corresponds to a certain time point in the video progress corresponding to the first audio clip, is used as the speaker identity of the target text corresponding to that cluster.

[0097] Therefore, by clustering audio features, multiple clusters are obtained. Then, the speaker identity of the longest duration appearing in the video progress corresponding to the audio in the cluster, or the speaker identity corresponding to a certain time point in the video progress corresponding to the first audio, is used as the speaker identity of the target text corresponding to that cluster. This eliminates the need to determine the speaker identity for each target text individually, effectively improving the effect and accuracy of speaker identity recognition.

[0098] The relevant technology uses text fragments from a script list as recognition units. It combines information from two modalities—facial recognition and voiceprint clustering—to identify the speaker. The basic process is as follows: First, facial recognition technology detects faces in the video file and compares them with a pre-established facial database to identify the speaker appearing in the scene. Second, the audio corresponding to each text fragment in the video file is extracted, the audio information is processed to obtain audio features, and the audio is clustered using a clustering algorithm to obtain multiple clusters, each cluster representing a speaker. Finally, the facial recognition results and clustering results are aligned and associated in the temporal domain, that is, the facial identity detected at a certain time point is assigned to the same cluster at the same time point, thereby completing the speaker identification.

[0099] In one possible implementation, after S101, the method further includes:

[0100] B1: Based on the video file and the list of lines, identify M text segments and obtain Q clusters.

[0101] Each of the Q clusters is labeled with a speaker identity. Each cluster includes audio corresponding to multiple text segments from M text segments. The speaker identities include unknown roles, which are speaker identities that cannot be confirmed.

[0102] First, relevant technologies are used to identify the speaker in the dialogue. Existing speech recognition technologies combine information from two modalities: facial recognition and voiceprint clustering. The basic process is as follows: First, facial recognition technology detects faces in the video frame and compares them with a pre-established face database to identify the speaker in the frame. Second, the audio corresponding to each text segment in the video file is extracted, the voiceprint features of the audio are extracted, and the audio is clustered using a clustering algorithm to obtain multiple clusters, each cluster representing a speaker. Finally, the facial recognition results and clustering results are aligned and associated in the temporal domain, that is, the facial identity detected at a certain time point is assigned to the same cluster at the same time point, thus completing the speaker identification.

[0103] Specifically, firstly, the speaker recognition technology is used to extract the audio corresponding to M text segments from the script list. Based on the extracted audio features, the audio is clustered to obtain Q clusters. Then, combined with the face recognition results and identity information from the face database, the speaker identity of each cluster is labeled. For example, the speaker identity of cluster 1 is actor 1, the speaker identity of cluster 2 is actor 2, the speaker identity of cluster 3 is an unknown role (such as "extra"), and so on. The speaker identity corresponding to each text segment can be determined through the audio within the cluster. Among them, the unknown role is the speaker whose identity information is not stored in the face database.

[0104] Thus, the initial identification of the speaker's identity was achieved through dialogue speaker recognition technology, and clustering results were obtained using text fragments as recognition units. However, the accuracy of the clustering algorithm depends on audio of sufficient duration to extract stable and distinguishable acoustic features. When the dialogue is too short (e.g., a single word or interjection), the extracted acoustic features are insufficient, causing these audio clips to be unable to accurately participate in clustering. This results in some dialogues corresponding to speakers with identity information in the face database being incorrectly labeled as unknown roles.

[0105] Based on this, the embodiments of this application use the text fragments corresponding to the clusters marked as unknown roles as target text fragments, identify N target texts with complete semantics corresponding to the target text fragments through a large language model, and then use the target texts as new recognition units to re-identify the speaker identity of the text fragments marked as unknown roles, specifically including:

[0106] B2: Identify the N audio files in the video file that are used to represent N target texts, and extract audio features from each of the N audio files to obtain N audio features.

[0107] B3: Cluster N audio files based on N audio features to obtain P target clusters.

[0108] B4: Perform secondary clustering on the P target clusters and Q clusters to obtain multiple new clusters.

[0109] If the first target cluster and the second cluster form a new cluster after secondary clustering, the speaker identity corresponding to the new cluster is marked as the speaker identity of the second cluster. The first target cluster is one of P target clusters, the second cluster is one of Q clusters, and the new cluster is one of multiple new clusters.

[0110] Specifically, N audio clips are identified in the video file to represent N target texts. Audio features are then extracted from these N audio clips to obtain N corresponding audio features. These N audio features are then further clustered using a preset threshold to obtain P target clusters. Finally, these P target clusters are used as the next clusters to be clustered and subjected to secondary clustering with Q clusters obtained through related techniques.

[0111] First, the similarity between the center vector of each target cluster and the center vector of each of the multiple clusters is calculated. For example, cosine similarity can be used. If the similarity between the first target cluster and the second cluster is greater than a preset threshold, the first target cluster and the second cluster are grouped into the same cluster to form a new cluster. The speaker identity corresponding to the first target cluster is then labeled as the speaker identity corresponding to the first cluster, thus successfully associating the text fragment that was originally misjudged as an unknown character with the correct speaker. If the similarity between the target cluster and the multiple clusters does not reach the preset threshold, the speaker corresponding to the target cluster is considered to be a true unknown character, and their speaker identity remains unknown.

[0112] Therefore, by using dialogue speaker technology to initially identify the dialogue list, and then expanding the target text fragments identified as positional roles based on a large language model to obtain target texts with complete semantics, and using them as identification units for re-clustering, text fragments that were misjudged as unknown roles can be corrected.

[0113] In one possible implementation, the target text fragment also includes text fragments in the dialogue list whose audio duration is less than a preset threshold.

[0114] Specifically, text segments with audio durations shorter than a preset threshold (such as single words, interjections, etc.) are also included as target text segments for speaker identification correction. This can further avoid misidentification caused by excessively short text segments in related technologies.

[0115] In one possible implementation, N audio clips in the video file are determined to represent the N target texts, and audio features are extracted from each of the N audio clips to obtain N audio features including:

[0116] C1: Extract audio information from a video file.

[0117] C2: Determine the N audio segments in the audio information used to represent N target texts.

[0118] C3: For the first audio among N audios, if the duration of the first audio is less than a preset duration threshold, then add the audio at the beginning or end of the first audio in the audio information with a preset duration to the first audio, to obtain N second audios.

[0119] C4: Extract the audio features of N second audio segments respectively to obtain N audio features.

[0120] The preset duration can be a fixed duration, such as 0.5 seconds, or it can be adjusted according to the actual scene. For example, in intense scenes such as fighting, the speaker's breath sound is relatively long after speaking, so the preset duration is longer.

[0121] If the duration of the first audio is less than a preset duration threshold, the confidence level of its recognition result is considered low. The first audio is then expanded by merging audio within a preset duration before or after it to obtain the second audio.

[0122] Furthermore, the merged second audio should not include audio corresponding to other text segments besides the complete semantics corresponding to the target text segment, in order to avoid interfering with the information. For example, if the expanded audio is found to contain audio corresponding to other text segments, the preset duration can be reduced to obtain the second audio again, or the audio can be directly extracted from the end (start) time point of the audio of the text segment before (after) the target text segment to obtain the second audio.

[0123] Therefore, audio that was previously judged to be too short can be expanded to capture the natural pauses and breath sounds when the speaker begins or finishes speaking, generating a longer and more complete generation unit, thereby improving the stability of audio features and further enhancing the accuracy of recognition.

[0124] Furthermore, the face database is a lagging and closed collection. For speakers whose specific identities cannot be identified by the facial recognition system, their corresponding lines are grouped into a separate cluster. Because no clear identity information can be associated with them, they are ultimately labeled as unknown characters. This results in some well-known actors and background voices being mixed under the same label, greatly reducing the usability of the recognition results.

[0125] In one possible implementation, the method further includes:

[0126] D1: Identify multiple audio features within new clusters labeled as unknown roles to obtain multiple sound attributes, which are used to identify different speakers;

[0127] D2: Based on multiple sound attributes, the speaker identity of multiple audio features is re-labeled.

[0128] Voice attributes are information used to distinguish the speaker's identity and may include age, gender, language type, and accent, etc. This application does not impose any restrictions on them.

[0129] For clusters that are still labeled as unknown roles after re-clustering, they are considered to be the true unknown roles after correct classification. There can be one or more such clusters. The audio in each cluster can be identified by an audio attribute classification model to obtain the corresponding attribute features. Then, based on the attribute features, the speaker identity of each cluster is re-labeled. For example, clusters 1, 2 and 3, which were labeled as unknown roles, are re-labeled as "male-young", "male-middle-aged" and "female-child voice".

[0130] Among them, the audio attribute classification model is used to identify the attribute features of audio. It can be trained based on a large amount of audio information to improve its recognition accuracy.

[0131] As a result, the clusters labeled as unknown roles were finely divided into distinct and traceable independent individuals, which improved the granularity of information and enhanced the usability of the identification results.

[0132] refer to Figure 2 The diagram shown is a second flowchart of a speech recognition method provided in this application embodiment. To more clearly describe this speech recognition method, the following description is based on a specific implementation scenario.

[0133] Input: A video and a list of lines from the video.

[0134] Step 1: Based on the speech recognition technology, perform preliminary identification of the speakers in the speech list.

[0135] Step 2: Semantic boundary analysis of the large model.

[0136] Take short lines of dialogue and lines identified as belonging to unknown characters as target text segments, along with their context (such as the sentences before and after), and input them into a large language model (such as the Qwen2 general model) to output semantic boundary information, such as the type of punctuation mark (such as a period, question mark, exclamation mark, comma, no punctuation, etc.) at the end of each line of dialogue. This model can be fine-tuned to focus on this task to improve the accuracy of judgment. Output the semantic boundary information for each text segment. For example, for "You, you, you're气死 me了", the model outputs "You, you, you're气死 me了.", and thus obtains the target text with complete semantics.

[0137] Step 3: Audio duration extension.

[0138] According to the semantic boundary information output by the large model, extend the audio corresponding to the target text. Specifically, for the target text, find the positions of full-stop, question mark, exclamation mark, etc. indicating the end of a complete sentence both forward and backward. If the target text is followed by a symbol indicating the end of a complete sentence, extend the original audio backward by a preset duration (such as 0.5 seconds). The purpose of this extension operation is to capture the natural pauses and breaths after the speaker finishes speaking, and these information are crucial for generating stable voiceprint features.

[0139] Step 4: Feature extraction and re-clustering.

[0140] Feature extraction: Re-extract the voiceprint features of the extended audio.

[0141] Secondary clustering: Cluster the new voiceprint features to form new clusters (denoted as "待聚类簇").

[0142] Identity association: Calculate the similarity (for example, using cosine similarity) between the center vector of each "待聚类簇" and the center vector of the clusters with confirmed identities in Step 1. Set a similarity threshold T (such as 0.9). If the similarity is greater than T, label the identity of this "待聚类簇" as the corresponding speaker identity. After this step, some lines of dialogue that were previously misjudged as belonging to unknown characters are successfully associated back to the correct speaker identity. The remaining "待聚类簇" that have not been successfully associated enter the next step.

[0143] Step 5: Refined annotation of unknown characters.

[0144] Attribute features are identified using an audio attribute classification model, and attribute labels are generated for each cluster. For the remaining unidentified "clusters to be clustered" from step 4, we consider them to be genuine extras, minor characters who cannot be identified by facial recognition. For each cluster to be clustered, the audio attribute classification model is used to identify attributes such as age and gender in the audio within that cluster. The combined identification results of each audio segment are used to obtain refined speaker identity labels for the cluster, such as "male-young," "male-middle-aged," and "female-child voice."

[0145] Step 6: Integrate all recognition results.

[0146] Output: A final, high-precision list of dialogue speakers. Previously misidentified known characters have been corrected, and all unknown characters have been broken down into distinct, individual entities.

[0147] Therefore, the innovative points and specific advantages of this application are listed below:

[0148] (1) Introduce a large language model to perform semantic boundary analysis of dialogue text, which is used to guide audio segmentation.

[0149] This approach completely revolutionizes the traditional audio segmentation model, which relies on pure signal processing (such as silence detection). By leveraging prior linguistic knowledge, audio segmentation more closely resembles the natural units of human speech, fundamentally solving the problem of unstable acoustic features caused by short dialogue. This directly improves the accuracy and robustness of subsequent clustering based on voiceprint features.

[0150] (2) A hierarchical association strategy is proposed, which has a high “error correction” capability.

[0151] This provides an opportunity to correct dialogue that was initially misjudged due to technical limitations. By comparing it with speakers whose identities have been confirmed, it can accurately "rescue" a large number of dialogues that were misjudged as belonging to unknown characters, significantly improving the overall recall and accuracy of the identification.

[0152] (3) Perform refined labeling of speakers whose identities cannot be identified based on audio attribute features (gender, age).

[0153] Transforming previously useless and vague character tags into distinctive, descriptive, independent entities provides application value for downstream content analysis tasks (such as counting the number of lines spoken by different characters) and video translation.

[0154] It has the following advantages in application scenarios:

[0155] (1) Reduce content production costs and improve efficiency: Automated and high-precision speaker recognition can significantly reduce the workload and time of manual subtitle proofreading, and reduce the cost of post-production.

[0156] (2) Empowering in-depth content analysis and commercialization: The dialogue data of each character is a valuable data asset that can be used to analyze video plots and assess the importance of characters; to discover memorable lines and scenes of characters for publicity, marketing and secondary creation; and to provide advertisers with more accurate character-related advertising placement solutions (for example, placing relevant ads in the dialogue scenes of a popular supporting character).

[0157] Based on the above embodiments, this application provides a speech recognition device for dialogue, with reference to... Figure 3 The diagram shown is a schematic of a speech recognition device according to an embodiment of this application. The device 300 includes an acquisition unit 301 and a determination unit 302.

[0158] The acquisition unit is used to acquire a video file and a list of lines, wherein the list of lines includes M text segments expressed by the speaker in the video file;

[0159] The determining unit is used to input the target text fragments in the dialogue list and the context text fragments of the target text fragments into the large language model to determine N target texts with complete semantics. The target text fragments are one or more of the M text fragments. Each target text in the N target texts includes multiple consecutive text fragments in the dialogue list, and N>0.

[0160] The determining unit is further configured to determine P speakers in the video file that represent the N target texts, and to determine the speaker identities of the P speakers, thereby obtaining the speaker identities corresponding to the N target texts respectively, wherein N≥P.

[0161] Therefore, by using a large language model, the semantic information of the context can be taken into account, and the semantic boundary information of the dialogue list can be provided. Since the target text with complete semantics corresponds to the same speaker, the target text with complete semantics is treated as a whole to realize the identification of the speaker of the dialogue. This can avoid misidentification of dialogues that are too short or dialogues corresponding to the same speaker, and effectively improve the accuracy of speaker identification.

[0162] In one possible implementation, the determining unit is configured to:

[0163] Identify N audio files in the video file that represent the N target texts, and extract audio features from each of the N audio files to obtain N audio features;

[0164] Based on the N audio features, the N audio files are clustered to obtain P clusters;

[0165] The faces of each speaker in the video file are extracted and compared with a pre-built face database to determine the identity of each speaker;

[0166] Align the first audio within the first cluster of the P clusters with the time each speaker appears in the video file to determine the identity of the first speaker corresponding to the first audio, and use the identity of the first speaker as the identity of the speaker corresponding to the first cluster to obtain the speaker identities corresponding to the N target texts respectively.

[0167] Therefore, by clustering audio features, multiple clusters are obtained. Then, the speaker identity of the longest duration appearing in the video progress corresponding to the audio in the cluster, or the speaker identity corresponding to a certain time point in the video progress corresponding to the first audio, is used as the speaker identity of the target text corresponding to that cluster. This eliminates the need to determine the speaker identity for each target text individually, effectively improving the effect and accuracy of speaker identity recognition.

[0168] In one possible implementation, the device further includes an identification unit for:

[0169] Based on the video file and the dialogue list, the M text segments are identified to obtain Q clusters. Each of the Q clusters is labeled with a speaker identity. Each cluster includes audio corresponding to multiple text segments from the M text segments. The speaker identity includes unknown roles, which are speakers whose identities cannot be confirmed. The target text segment is the text segment corresponding to the cluster labeled with the unknown role.

[0170] The determining unit is used for:

[0171] Identify N audio files in the video file that represent the N target texts, and extract audio features from each of the N audio files to obtain N audio features;

[0172] Based on the N audio features, the N audio files are clustered to obtain P target clusters;

[0173] The P target clusters and the Q clusters are subjected to secondary clustering to obtain multiple new clusters. If the first target cluster and the second cluster form a new cluster after the secondary clustering, the speaker identity corresponding to the new cluster is marked as the speaker identity of the second cluster. The first target cluster is one of the P target clusters, the second cluster is one of the Q clusters, and the new cluster is one of the multiple new clusters.

[0174] Therefore, by using dialogue speaker technology to initially identify the dialogue list, and then expanding the target text fragments identified as positional roles based on a large language model to obtain target texts with complete semantics, and using them as identification units for re-clustering, text fragments that were misjudged as unknown roles can be corrected.

[0175] In one possible implementation, the determining unit is configured to:

[0176] Extract the audio information from the video file;

[0177] Identify the N audio clips in the audio information used to represent the N target texts;

[0178] For the first audio among the N audios, if the duration of the first audio is less than a preset duration threshold, then the audio at the beginning or end of the first audio in the audio information with a preset duration is added to the first audio to obtain N second audios;

[0179] The audio features of the N second audio frequencies are extracted respectively to obtain N audio features.

[0180] Therefore, audio that was previously judged to be too short can be expanded to capture the natural pauses and breath sounds when the speaker begins or finishes speaking, generating a longer and more complete generation unit, thereby improving the stability of audio features and further enhancing the accuracy of recognition.

[0181] In one possible implementation, the target text fragment also includes text fragments corresponding to lines in the dialogue list whose duration is less than a preset threshold.

[0182] This can further avoid misidentification caused by excessively short text fragments in related technologies.

[0183] In one possible implementation, the method further includes a labeling unit;

[0184] The identification unit is used to identify multiple audio features within the new clusters labeled as unknown roles in the multiple new clusters, and obtain multiple sound attributes, which are used to identify different speakers;

[0185] The annotation unit is used to re-annotate the speaker identity of the multiple audio features based on the multiple sound attributes.

[0186] As a result, the clusters labeled as unknown roles were finely divided into distinct and traceable independent individuals, which improved the granularity of information and enhanced the usability of the identification results.

[0187] Based on the above embodiments, this application provides a computer device, which includes a processor and a memory:

[0188] The memory is used to store computer programs;

[0189] The processor is used to execute the above-described speech recognition method according to the computer program.

[0190] Based on the above embodiments, this application provides a computer-readable storage medium for storing a computer program, which, when executed by a computer device, implements the above-described speech recognition method.

[0191] Based on the above embodiments, this application provides a computer program product including a computer program, which, when run on a computer device, causes the computer device to execute the above-described speech recognition method.

[0192] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0193] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying the speaker in a dialogue, characterized in that, The method includes: Obtain a video file and a list of dialogues, wherein the list of dialogues includes M text segments spoken by the speaker in the video file; The target text fragments in the dialogue list and the context text fragments of the target text fragments are input into the large language model to determine N target texts with complete semantics. The target text fragments are one or more of the M text fragments. Each of the N target texts includes multiple consecutive text fragments in the dialogue list, and N>0. In the video file, P speakers are identified to represent the N target texts, and the speaker identities of the P speakers are determined to obtain the speaker identities corresponding to the N target texts respectively. P.

2. The method according to claim 1, characterized in that, Where N>1, the step of determining P speakers in the video file to represent the N target texts, and determining the speaker identities of the P speakers to obtain the speaker identities corresponding to the N target texts respectively includes: Identify N audio files in the video file that represent the N target texts, and extract audio features from each of the N audio files to obtain N audio features; Based on the N audio features, the N audio files are clustered to obtain P clusters; The faces of each speaker in the video file are extracted and compared with a pre-built face database to determine the identity of each speaker; Align the first audio within the first cluster of the P clusters with the time each speaker appears in the video file to determine the identity of the first speaker corresponding to the first audio, and use the identity of the first speaker as the identity of the speaker corresponding to the first cluster to obtain the speaker identities corresponding to the N target texts respectively.

3. The method according to claim 1, characterized in that, After obtaining the video file and the list of lines, the method further includes: Based on the video file and the dialogue list, the M text segments are identified to obtain Q clusters. Each of the Q clusters is labeled with a speaker identity. Each cluster includes audio corresponding to multiple text segments from the M text segments. The speaker identity includes unknown roles, which are speakers whose identities cannot be confirmed. The target text segment is the text segment corresponding to the cluster labeled with the unknown role. The step of determining P speakers in the video file to represent the N target texts, and determining the speaker identities of the P speakers to obtain the speaker identities corresponding to the N target texts respectively includes: Identify N audio files in the video file that represent the N target texts, and extract audio features from each of the N audio files to obtain N audio features; Based on the N audio features, the N audio files are clustered to obtain P target clusters; The P target clusters and the Q clusters are subjected to secondary clustering to obtain multiple new clusters. If the first target cluster and the second cluster form a new cluster after the secondary clustering, the speaker identity corresponding to the new cluster is marked as the speaker identity of the second cluster. The first target cluster is one of the P target clusters, the second cluster is one of the Q clusters, and the new cluster is one of the multiple new clusters.

4. The method according to any one of claim 2 or 3, characterized in that, The step involves determining N audio clips in the video file used to represent the N target texts, and extracting audio features from each of the N audio clips to obtain N audio features, including: Extract the audio information from the video file; Identify the N audio clips in the audio information used to represent the N target texts; For the first audio among the N audios, if the duration of the first audio is less than a preset duration threshold, then the audio at the beginning or end of the first audio in the audio information with a preset duration is added to the first audio to obtain N second audios; The audio features of the N second audio frequencies are extracted respectively to obtain N audio features.

5. The method according to claim 3, characterized in that, The target text fragments also include text fragments in the dialogue list whose audio duration is less than a preset threshold.

6. The method according to claim 3, characterized in that, The method further includes: Multiple audio features within the new clusters labeled as unknown roles are identified to obtain multiple sound attributes, which are used to identify different speakers; Based on the multiple sound attributes, the speaker identities of the multiple audio features are re-labeled.

7. A speech recognition device, characterized in that, The device includes an acquisition unit and a determination unit: The acquisition unit is used to acquire a video file and a list of lines, wherein the list of lines includes M text segments expressed by the speaker in the video file; The determining unit is used to input the target text fragments in the dialogue list and the context text fragments of the target text fragments into the large language model to determine N target texts with complete semantics. The target text fragments are one or more of the M text fragments. Each target text in the N target texts includes multiple consecutive text fragments in the dialogue list, and N>0. The determining unit is further configured to determine P speakers in the video file that represent the N target texts, and to determine the speaker identities of the P speakers, thereby obtaining the speaker identities corresponding to the N target texts respectively, wherein N≥P.

8. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs; The processor is configured to perform the method according to any one of claims 1-6 according to the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when executed by a computer device, performs the method described in any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, When it is run on a computer device, it causes the computer device to perform the method described in any one of claims 1-6.

Images