A method and device for determining a character corresponding to a dialogue, and an electronic device

Abstract

Embodiments of the present application provide a method and device for determining a character corresponding to a dialogue and an electronic device. The electronic device can obtain to-be-processed voiceprint features, cluster the voiceprint features, determine categories to which the voiceprint features belong, for each category, determine a proportion of an appearance duration of a character in a speaking state corresponding to the voiceprint features of the category based on video frame information of a video segment in a target video corresponding to the category, and determine a character corresponding to a dialogue corresponding to the voiceprint features of the category based on the proportion of the appearance duration of the character in the speaking state corresponding to the voiceprint features of the category. Since a character with a high proportion of the appearance duration of the character in the speaking state is usually a character corresponding to a dialogue, the proportion of the appearance duration of the character in the speaking state corresponding to the voiceprint features of the category is determined based on the video frame information corresponding to the category, and the character corresponding to the dialogue corresponding to the voiceprint features of the category is determined based on the proportion of the appearance duration of the character in the speaking state, thereby improving the accuracy of determining the character corresponding to the dialogue.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a method, apparatus, and electronic device for determining the corresponding character in a dialogue. Background Technology

[0002] In video editing and narration, it's crucial to identify the person speaking each line of dialogue in the video. This allows for effective editing and narration. To determine the person speaking each line, it's necessary to extract the voiceprint features of each character in the video, creating a voiceprint feature database. Then, each extracted voiceprint feature is compared with the voiceprint features of each character in the database. The most similar voiceprint feature is then identified, and the person corresponding to that feature is determined as the speaker of the dialogue.

[0003] However, in the above methods of determining the characters corresponding to the lines, the background sounds may be different in different scenes, and the characters' emotions may be different. Therefore, the voiceprint features of the same character extracted in different scenes may vary greatly. Also, because the voice actors are the same, the voice timbres of different characters are very similar. Therefore, the accuracy of determining the characters corresponding to the lines by using a voiceprint feature database is not high. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, and electronic device for determining the corresponding character in a line of dialogue, thereby improving the accuracy of determining the corresponding character. The specific technical solution is as follows:

[0005] Firstly, a method for determining the character corresponding to a line of dialogue, the method comprising:

[0006] Obtain the voiceprint features to be processed, wherein the voiceprint features to be processed are the voiceprint features corresponding to the lines in the target video;

[0007] The voiceprint features are clustered to determine the category to which each voiceprint feature belongs;

[0008] For each category, based on the image information of the video segments in the target video corresponding to that category, the proportion of the appearance time of the person in the speaking state corresponding to the voiceprint features of that category is determined;

[0009] Based on the proportion of time a person in a speaking state appears corresponding to the voiceprint feature of that category, the person corresponding to the dialogue of that category is determined.

[0010] Optionally, the step of determining the proportion of appearance time of a person in a speaking state corresponding to the voiceprint features of that category based on the image information of the video segments in the target video corresponding to that category includes:

[0011] Speaker detection is performed on the video clips in the target video corresponding to the category to determine the area where the person in the speaking state is located, and face recognition is performed on the area where the person is located to determine the person in the speaking state in the area where the person is located.

[0012] Based on the number of times the person in a speaking state appears in the video clip, the proportion of the speaking state corresponding to the voiceprint feature of that category in the video clip is determined.

[0013] Optionally, the step of performing speaker detection on the video clips in the target video corresponding to the category, determining the area where a person is speaking, and performing face recognition on the area where the person is speaking to determine the person in the area where the person is speaking includes:

[0014] The target video is divided into video segments corresponding to each line of dialogue according to the start and end times of the dialogue.

[0015] A speaker detection model is used to detect speakers in the video clips corresponding to this category, determine the area where the person in the speaking state is located, and perform face recognition on the area where the person is located to determine the person in the speaking state in the area.

[0016] Optionally, the step of determining the character corresponding to the lines of a given voiceprint feature based on the proportion of time a character in a speaking state appears according to that voiceprint feature category includes:

[0017] Among the characters who are speaking and whose speaking time exceeds a first preset threshold, the character with the highest speaking time is identified as the character whose voiceprint features correspond to the lines of that category.

[0018] Optionally, the step of obtaining the voiceprint features to be processed includes:

[0019] The audio corresponding to the target video is divided into audio segments corresponding to each line of dialogue according to the start and end times of the dialogue.

[0020] Extract the voiceprint features of each audio segment to obtain the voiceprint features to be processed.

[0021] Optionally, the step of clustering the voiceprint features to determine the category to which each voiceprint feature belongs includes:

[0022] Based on the scene information of the target video, determine the time period corresponding to each scene included in the target video;

[0023] Cluster the voiceprint features corresponding to each time period to determine the category to which the voiceprint features corresponding to each time period belong.

[0024] Optionally, the method further includes:

[0025] If, based on the proportion of time a person in a speaking state appears corresponding to a certain category of voiceprint features, it is determined that there is no corresponding person for the lines corresponding to that category of voiceprint features, then, based on the image information of the target video corresponding to that category, a candidate person corresponding to that category of voiceprint features is determined.

[0026] Based on the number of candidate characters corresponding to the voiceprint features of this category, the character corresponding to the lines of this category is determined.

[0027] Optionally, the step of determining the candidate individuals corresponding to the voiceprint features of the category based on the image information of the target video corresponding to the category includes:

[0028] Obtain video frames of the target video within the time period corresponding to the voiceprint features included in this category;

[0029] Perform facial recognition on the video frames to determine the proportion of time each person appears in the video frames included in that time period;

[0030] Individuals whose corresponding proportions reach the preset proportions are identified as candidate individuals for that category of voiceprint characteristics.

[0031] Optionally, the step of determining the character corresponding to the lines of a given category of voiceprint features based on the number of candidate characters corresponding to that category of voiceprint features includes:

[0032] If there is only one candidate person corresponding to the voiceprint feature of this category, then the candidate person is determined to be the person corresponding to the lines of the voiceprint feature of this category.

[0033] If there are multiple candidate characters corresponding to the voiceprint features of this category, the character corresponding to the lines of this category is determined based on the similarity between the cluster center of this category and the cluster center of the first target category. The first target category includes categories that are in the same scene as this category and have one candidate character, as well as categories that are in the same scene as this category and have had their corresponding characters determined based on the proportion of time characters appearing in a speaking state.

[0034] Optionally, the step of determining the character corresponding to the dialogue of a given category based on the similarity between the cluster centers of the current category and the cluster centers of the first target category includes:

[0035] Calculate the similarity between the cluster centers of this category and the cluster centers of each first target category;

[0036] The person corresponding to the first target category whose similarity reaches the first preset similarity is identified as the person corresponding to the voiceprint feature of that category.

[0037] Optionally, the method further includes:

[0038] If the similarity between the cluster center of this category and the cluster center of each first target category does not reach the first preset similarity, calculate the similarity between the cluster center of this category and the cluster center of each second target category respectively. The second target category is a category that is different from the scene corresponding to this category and has one corresponding candidate person, and a category that is different from the scene corresponding to this category and has been determined based on the appearance time ratio of the person in the speaking state.

[0039] The characters corresponding to the second target category whose similarity reaches the second preset similarity are identified as the characters whose voiceprint features correspond to the lines of that category.

[0040] Optionally, the method further includes:

[0041] If the similarity between the cluster center of this category and the cluster center of each second target category does not reach the second preset similarity, for each voiceprint feature of this category, calculate the similarity between the voiceprint feature and other voiceprint features in this category;

[0042] If the proportion of similarity reaching the second preset threshold is less than the preset proportion, obtain each voiceprint feature to be compared within a preset duration, including the video time point corresponding to the voiceprint feature.

[0043] Calculate the similarity between the voiceprint feature and each voiceprint feature to be compared;

[0044] The person whose voiceprint feature has the highest similarity to the one to be compared is identified as the person whose lines correspond to that voiceprint feature.

[0045] Secondly, a device for determining the character corresponding to a line of dialogue, the device comprising:

[0046] The voiceprint feature acquisition module is used to acquire voiceprint features to be processed, wherein the voiceprint features to be processed are the voiceprint features corresponding to the lines in the target video;

[0047] The voiceprint feature clustering module is used to cluster the voiceprint features and determine the category to which each voiceprint feature belongs;

[0048] The proportion determination module is used to determine the proportion of the appearance time of the person in the speaking state corresponding to the voiceprint features of each category, based on the picture information of the video segment in the target video corresponding to that category.

[0049] The first character identification module is used to determine the character corresponding to the lines of a given voiceprint feature based on the proportion of time a character in a speaking state appears in the voiceprint feature of that category.

[0050] Thirdly, an electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0051] Memory, used to store computer programs;

[0052] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect above.

[0053] Fourthly, a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0054] Beneficial effects of the embodiments of the present invention:

[0055] In the solution provided by this embodiment of the invention, an electronic device can acquire voiceprint features to be processed, wherein the voiceprint features to be processed are voiceprint features corresponding to lines in a target video; the voiceprint features are clustered to determine the category to which each voiceprint feature belongs; for each category, based on the image information of the video segment in the target video corresponding to that category, the appearance duration ratio of the person in the speaking state corresponding to the voiceprint features of that category is determined; based on the appearance duration ratio of the person in the speaking state corresponding to the voiceprint features of that category, the person corresponding to the lines in the dialogue corresponding to the voiceprint features of that category is determined. In this solution, since the voiceprint features corresponding to the lines spoken by the same person in the target video are similar, the voiceprint features corresponding to the lines in the target video can be clustered to determine the category to which each voiceprint feature belongs. Since the person appearing in a speaking state for a higher proportion in the video footage is usually the person corresponding to the lines, the proportion of the speaking state corresponding to the voiceprint features of that category can be determined based on the video footage of the target video corresponding to that category. Then, based on the proportion of the speaking state corresponding to the voiceprint features of that category, the person corresponding to the lines corresponding to the lines corresponding to that category can be determined, without the need to search and compare voiceprint features based on a voiceprint feature database, thus improving the accuracy of determining the person corresponding to the lines. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0057] Figure 1 A flowchart illustrating the method for determining the corresponding character for dialogue provided in an embodiment of the present invention;

[0058] Figure 2 Based on Figure 1 A flowchart illustrating how the duration of a character in a speaking state is determined in the illustrated embodiment.

[0059] Figure 3 Based on Figure 2 A flowchart illustrating the method for determining the location of a person in a speaking state in the illustrated embodiment.

[0060] Figure 4 Based on Figure 1 A flowchart illustrating the method for obtaining the voiceprint features to be processed in the embodiment shown.

[0061] Figure 5 This is a schematic flowchart of a voiceprint feature extraction method provided in an embodiment of the present invention;

[0062] Figure 6 Based on Figure 1A flowchart illustrating the method for determining the category of voiceprint features in the illustrated embodiment;

[0063] Figure 7 A flowchart illustrating the method for determining the character corresponding to a line based on the number of candidate characters provided in an embodiment of the present invention;

[0064] Figure 8 Based on Figure 7 A flowchart illustrating the method for determining candidate individuals corresponding to voiceprint features in the illustrated embodiment.

[0065] Figure 9 This is a flowchart illustrating the method for determining the character corresponding to a line based on a first target category, as provided in an embodiment of the present invention.

[0066] Figure 10 A flowchart illustrating the method for determining the character corresponding to a line based on a second target category, as provided in an embodiment of the present invention;

[0067] Figure 11 A flowchart illustrating the method for calculating voiceprint feature similarity provided in an embodiment of the present invention;

[0068] Figure 12 This is a flowchart illustrating the method for determining the corresponding character for dialogue provided in an embodiment of the present invention.

[0069] Figure 13 A schematic diagram of the structure of the device for determining the corresponding character for dialogue provided in an embodiment of the present invention;

[0070] Figure 14 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0071] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0072] In video editing and narration, it's necessary to identify the person who speaks each line of dialogue in the video to facilitate better editing and narration. To do this, it's essential to extract the voiceprint features of each character in the video. Based on the extracted voiceprint features and their corresponding relationships, a voiceprint feature database is established.

[0073] Then, the voiceprint features corresponding to each line of dialogue in the video are extracted. Each extracted voiceprint feature is then compared with the voiceprint features of each person in the voiceprint feature database. The most similar voiceprint feature is found in the voiceprint feature database, and the person corresponding to that voiceprint feature is identified as the speaker of the dialogue.

[0074] However, in the above methods of determining the characters corresponding to the lines, the background sounds may be different in different scenes, and the characters' emotions may be different. Therefore, the voiceprint features of the same character extracted in different scenes may vary greatly. Also, because the voice actors are the same, the voice timbres of different characters are very similar. Therefore, the accuracy of determining the characters corresponding to the lines by using a voiceprint feature database is not high.

[0075] To improve the accuracy of determining the corresponding character for a line of dialogue, embodiments of the present invention provide a method, apparatus, and electronic device for determining the corresponding character for a line of dialogue. The method for determining the corresponding character for a line of dialogue provided by embodiments of the present invention will be described first below.

[0076] like Figure 1 As shown, a method for determining the character corresponding to a line of dialogue, the method includes:

[0077] S101, Obtain the voiceprint features to be processed;

[0078] The voiceprint features to be processed are the voiceprint features corresponding to the lines in the target video;

[0079] S102, cluster the voiceprint features to determine the category to which each voiceprint feature belongs;

[0080] S103, for each category, based on the image information of the video segment in the target video corresponding to that category, determine the proportion of the appearance time of the person in the speaking state corresponding to the voiceprint feature of that category;

[0081] S104. Based on the proportion of time that the voiceprint features of this category correspond to the characters in the speaking state, determine the characters corresponding to the lines of this category.

[0082] As can be seen, in the solution provided by the embodiments of the present invention, the electronic device can acquire voiceprint features to be processed, wherein the voiceprint features to be processed are voiceprint features corresponding to lines in the target video; the voiceprint features are clustered to determine the category to which each voiceprint feature belongs; for each category, based on the image information of the video segment in the target video corresponding to the category, the appearance duration ratio of the person in the speaking state corresponding to the voiceprint features of that category is determined; based on the appearance duration ratio of the person in the speaking state corresponding to the voiceprint features of that category, the person corresponding to the lines corresponding to the voiceprint features of that category is determined. In this solution, since the voiceprint features corresponding to the lines spoken by the same person in the target video are similar, the voiceprint features corresponding to the lines in the target video can be clustered to determine the category to which each voiceprint feature belongs. Since the person appearing in a speaking state for a higher proportion in the video footage is usually the person corresponding to the lines, the proportion of the speaking state corresponding to the voiceprint features of that category can be determined based on the video footage of the target video corresponding to that category. Then, based on the proportion of the speaking state corresponding to the voiceprint features of that category, the person corresponding to the lines corresponding to the lines corresponding to that category can be determined, without the need to search and compare voiceprint features based on a voiceprint feature database, thus improving the accuracy of determining the person corresponding to the lines.

[0083] The method for determining the corresponding character for dialogue disclosed in this invention is applied to the field of speech processing technology. Specifically, it can be applied to scenarios such as video editing and video narration, without being specifically limited here.

[0084] Since voiceprint features can represent the voice of the person corresponding to the lines in the target video, in order to determine the person corresponding to the lines, electronic devices can acquire voiceprint features to be processed, where the voiceprint features to be processed are the voiceprint features corresponding to the lines in the target video.

[0085] Since the voiceprint features corresponding to the lines spoken by the same person in the target video are similar, these voiceprint features can be clustered to determine the category to which each voiceprint feature belongs. Clustering can be implemented using algorithms such as K-means clustering and hierarchical clustering, and no specific limitation is made here.

[0086] For example, the target video contains five lines of dialogue: Line 1, Line 2, Line 3, Line 4, and Line 5. Lines 1 and 3 are spoken by the same person, while Lines 2, 4, and 5 are spoken by a different person. If we cluster the voiceprint features corresponding to the lines, we can obtain two categories: the voiceprint features corresponding to Lines 1 and 3 belong to one category, and the voiceprint features corresponding to Lines 2, 4, and 5 belong to another category.

[0087] Since the speaker in the dialogue is usually in a speaking state within the visual information of the target video, the proportion of time a person in a speaking state appears corresponding to the voiceprint features of that category can be determined based on the visual information of the video clips in the target video corresponding to that category. The proportion of time a person in a speaking state appears can be the ratio between the appearance duration of a person in a speaking state in that video clip and the duration of that video clip.

[0088] For example, as shown in the table below, the target video includes lines 1-10. The voiceprint features 1-10 corresponding to lines 1-10 are clustered to obtain voiceprint feature categories 1-4. Based on the video clips in the target video corresponding to each voiceprint feature category, the characters in speaking mode corresponding to categories 1-4 and the proportion of time these characters are in speaking mode are determined.

[0089]

[0090] At this point, the proportion of time a person appears in a speaking state has been determined for each voiceprint feature category. Electronic devices can determine the person corresponding to the lines spoken by a given voiceprint feature category based on this proportion of time a person appears in a speaking state.

[0091] Continuing from the previous example, as shown in the table above, for category 2, the person in a speaking state is person A, and person A speaks for 95% of the video clips corresponding to category 2. If the electronic device determines the person corresponding to the dialogue based on the voiceprint features of this category by identifying the person whose speaking time exceeds 85%, then the electronic device can determine that the person corresponding to dialogues 2 and 4 for voiceprint features 2 and 4 in category 2 is person A.

[0092] As can be seen, since the voiceprint features corresponding to the lines spoken by the same person in the target video are similar, the voiceprint features corresponding to the lines in the target video can be clustered to determine the category to which each voiceprint feature belongs. Since the person appearing in a speaking state for a higher proportion in the video footage is usually the person corresponding to the lines, the proportion of the speaking state corresponding to the voiceprint features of that category can be determined based on the video footage of the target video corresponding to that category. Then, based on the proportion of the speaking state corresponding to the voiceprint features of that category, the person corresponding to the lines corresponding to the lines corresponding to that category can be determined, without the need to search and compare voiceprint features based on a voiceprint feature database, thus improving the accuracy of determining the person corresponding to the lines.

[0093] As one embodiment of the present invention, such as Figure 2 As shown, the step of determining the proportion of appearance time of a person in a speaking state corresponding to the voiceprint features of that category based on the image information of the video segments in the target video corresponding to that category may include:

[0094] S201, perform speaker detection on the video clip in the target video corresponding to the category, determine the area where the person in the speaking state is located, and perform face recognition on the area where the person is located to determine the person in the speaking state in the area where the person is located.

[0095] Since the person corresponding to the dialogue is usually the person speaking in the video clip corresponding to the dialogue, speaker detection (Activate SpeakerDetect) can be performed on the video clip in the target video of this category to determine the area where the person speaking is located, and then face recognition can be performed on the area where the person is located to determine the person speaking in the area.

[0096] S202, based on the number of times the person in the speaking state appears in the video clip, determine the proportion of the appearance time of the person in the speaking state corresponding to the voiceprint feature of this category in the video clip.

[0097] Step S201 can identify the person speaking in the video clip. Therefore, the electronic device can determine the proportion of the duration of the person speaking in the video clip corresponding to the voiceprint feature of that category based on the number of times the person speaking appears in the video clip.

[0098] Since video clips are composed of video frames, the number of times a person speaking appears in a video clip can be reflected by the number of video frames in the video clip that include the person speaking. For the video frames included in a video clip within a target video corresponding to this category, the proportion of time a person speaking appears in the video clip corresponding to the voiceprint features of this category is determined based on the number of times the video frames including the speaking person appear in all the video frames included in that video clip.

[0099] For example, the video segment in the target video corresponding to the dialogue is 5 seconds long, and the number of frames per second is 24. After determining the person in the video segment who is speaking in step S201, the proportion of the appearance time of the person in the video segment corresponding to the voiceprint feature of that category can be determined based on the video frames in the video segment that include the person in the speaking state.

[0100] If the person speaking in the video clip is person A, then the number of frames in the video clip that include person A speaking is 93. Since the total number of frames is 120, it can be determined that the proportion of the speaking person corresponding to this category of voiceprint features in the video clip is 93 / 120.

[0101] As can be seen, in this embodiment of the invention, the electronic device can perform speaker detection on the video clips in the target video corresponding to the category, determine the area where the person in a speaking state is located, and perform face recognition on the area where the person is located to determine the person in a speaking state in the area where the person is located; based on the number of times the person in a speaking state appears in the video clip, the device can determine the proportion of the appearance time of the person in a speaking state corresponding to the voiceprint features of that category in the video clip. Since the person corresponding to the dialogue is usually the person in a speaking state in the video clip corresponding to the dialogue, the electronic device can determine the proportion of the appearance time of the person in a speaking state corresponding to the voiceprint features of that category in the video clip based on the number of times the person in a speaking state appears in the video clip. In this way, the accuracy of determining the proportion of the appearance time of the person in a speaking state in the video clip can be improved, thereby improving the accuracy of determining the person corresponding to the dialogue.

[0102] As one embodiment of the present invention, such as Figure 3 As shown, the steps described above, including speaker detection in the video clips of the target video corresponding to the category, determining the area where a person is speaking, and performing face recognition in the area where the person is speaking, can include:

[0103] S301, the target video is divided into video segments corresponding to each line of dialogue according to the start and end times of the dialogue;

[0104] Since a line of dialogue is usually spoken by a single character, the video clips in the target video corresponding to that line usually contain that character. Therefore, electronic devices can divide the target video into video clips corresponding to each line of dialogue according to the start and end times of the dialogue.

[0105] For example, if the dialogue in the target video starts at 5 minutes and 20 seconds and ends at 5 minutes and 23 seconds, then the video segment corresponding to the above time period in the target video can be divided. This video segment is the video segment corresponding to the dialogue in the target video that starts at 5 minutes and 20 seconds and ends at 5 minutes and 23 seconds.

[0106] S302, using a speaker detection model, perform speaker detection on the video clips corresponding to this category, determine the area where the person in the speaking state is located, and perform face recognition on the area where the person is located to determine the person in the speaking state in the area where the person is located.

[0107] Electronic devices can employ an Activate Speaker Detection (ASD) model to detect speakers in video clips corresponding to a specific category, thus identifying the area where a person is speaking. Since the person speaking in a video is usually the speaker of the dialogue, the electronic device can perform facial recognition on the area where the person is located, thereby identifying the person speaking in the task's area.

[0108] Following the example of step S301, the electronic device can use a speaker detection model to identify video segments with start and end times of 5 minutes and 20 seconds and 5 minutes and 23 seconds respectively, determine the area where the person in the speaking state is located, and perform face recognition on the area where the person is located to determine the person in the speaking state in the area.

[0109] As can be seen, in this embodiment of the invention, the electronic device can divide the target video into video segments corresponding to each line of dialogue according to the start and end times of the dialogue; it then uses a speaker detection model to detect speakers in the video segments corresponding to each category, determines the area where the person in the speaking state is located, and performs facial recognition on the area where the person is located to determine the person in the speaking state within that area. Since a line of dialogue is usually spoken by one person, in order to determine the speaker corresponding to the dialogue, the electronic device can divide the target video into video segments corresponding to each line of dialogue according to the start and end times of the dialogue, and then use a speaker detection model to detect speakers in the video segments, determine the area where the person in the speaking state is located, and perform facial recognition on the area where the person is located to determine the person in the speaking state within that area. This improves the accuracy of determining the area where the person in the speaking state is located, thereby improving the accuracy of determining the person corresponding to the dialogue.

[0110] As one embodiment of the present invention, the step of determining the character corresponding to the lines of a given category of voiceprint features based on the proportion of appearance time of characters in a speaking state corresponding to that category of voiceprint features may include:

[0111] Among the characters who are speaking and whose speaking time exceeds a first preset threshold, the character with the highest speaking time is identified as the character whose voiceprint features correspond to the lines of that category.

[0112] Since the appearance duration of the character corresponding to the voiceprint feature of this category should be greater than that of other characters in the same category, and the appearance duration of this character should also be greater than the first preset threshold, the electronic device can identify the character with the highest appearance duration among the characters in the speaking state whose appearance duration is greater than the first preset threshold as the character corresponding to the voiceprint feature of this category.

[0113] For example, as shown in the table below, the target video includes lines 1-10. The voiceprint features 1-10 corresponding to lines 1-10 are clustered to obtain voiceprint feature categories 1-4. Based on the video clips in the target video corresponding to each voiceprint feature category, the characters in speaking mode corresponding to categories 1-4 and the proportion of time these characters are in speaking mode are determined.

[0114] If the first preset threshold is 85%, for category 1, the character whose appearance duration exceeds the first preset threshold is character B, and the characters corresponding to lines 1, 3, and 5 corresponding to voiceprint features 1, 3, and 5 for this category can be identified as character B. For category 2, the character whose appearance duration exceeds the first preset threshold is character A, and the characters corresponding to lines 2 and 4 corresponding to voiceprint features 2 and 4 for this category can be identified as character A.

[0115] For category 3, characters whose appearance duration exceeds the first preset threshold are identified as character C. The characters corresponding to lines 6, 7, and 8 of voiceprint features 6, 7, and 8 for this category can be identified as character C. For category 4, characters whose appearance duration exceeds the first preset threshold are identified as character D. The characters corresponding to lines 9 and 10 of voiceprint features 9 and 10 for this category can be identified as character D.

[0116]

[0117]

[0118] As can be seen, in this embodiment of the invention, the electronic device can identify the person with the highest appearance duration among those speaking (whose appearance duration is greater than a first preset threshold) as the person corresponding to the dialogue of that category of voiceprint features. Since the appearance duration of the person corresponding to the dialogue of that category of voiceprint features should be greater than the appearance duration of other characters in the same category, and this person's appearance duration should also be greater than the first preset threshold, the electronic device can identify the person with the highest appearance duration among those speaking (whose appearance duration is greater than the first preset threshold) as the person corresponding to the dialogue of that category of voiceprint features. This improves the accuracy of identifying the person corresponding to the dialogue.

[0119] As one embodiment of the present invention, such as Figure 4 As shown, the steps for obtaining the voiceprint features to be processed described above may include:

[0120] S401, the audio corresponding to the target video is divided into audio segments corresponding to each line of dialogue according to the start and end times of the dialogue;

[0121] Since a line of dialogue is usually spoken by one character, in order to determine the character corresponding to the line, electronic devices can divide the audio of the target video according to the start and end times of the line. For example, electronic devices can use OCR (Optical Character Recognition) to determine the start time information of each line of dialogue to divide the audio, thereby obtaining the audio segment corresponding to each line of dialogue. The audio segment can be used for subsequent operations to determine the character corresponding to the line of dialogue.

[0122] Prior to step S401, the electronic device can separate the audio file from the target video, extract the uncompressed WAV audio file, and segment the WAV audio file into speaker audio segments. Each audio segment is a mono audio segment with a sampling rate of 16kHz.

[0123] Continuing from the previous example, as shown in the table below, taking voiceprint features included in category 1 as an example, electronic devices can determine the time segment of the dialogue based on OCR (Optical Character Recognition), that is, the start and end time of the dialogue in the target video. Among them, the time segment of dialogue 1 is 1 second to 4 seconds in the video, the time segment of dialogue 3 is 8 seconds to 10 seconds in the video, and the time segment of dialogue 5 is 13 seconds to 15 seconds in the video, thus dividing it into audio segments corresponding to each line of dialogue.

[0124] Dialogue Dialogue time period Dialogue 1 1 second - 4 seconds Dialogue 3 8 seconds - 10 seconds Dialogue 5 13-15 seconds

[0125] S402, extract the voiceprint features of each audio segment to obtain the voiceprint features to be processed.

[0126] Since voiceprint features can represent the voice of a person in an audio segment, electronic devices can extract the voiceprint features of each audio segment, and then obtain the voiceprint features to be processed, that is, the voiceprint features corresponding to the lines in the target video.

[0127] It should be noted that when obtaining the voiceprint features to be processed, it is not necessary to obtain the voiceprint features corresponding to all the lines in the target video. It is sufficient to obtain only a few lines or the lines corresponding to a video segment.

[0128] Following the example of step S401, as shown in the table below, voiceprint features are extracted from lines 1, 3, and 5 obtained in step S401, resulting in voiceprint features 1, 3, and 5 for lines 1, 3, and 5 respectively.

[0129] Voiceprint characteristics Dialogue Dialogue time period Voiceprint Features 1 Dialogue 1 1 second - 4 seconds Voiceprint Features 3 Dialogue 3 8 seconds - 10 seconds Voiceprint feature 5 Dialogue 5 13-15 seconds

[0130] For example, models that can be used to extract voiceprint features from audio segments include the Ecapa-tdnn model and the WavLM model. In voiceprint feature extraction, one model can be used for extraction, or multiple models can be used for extraction followed by normalization, splicing, and PCA (principal component analysis) dimensionality reduction. Any voiceprint extraction method that can obtain voiceprint features is acceptable, and no specific limitation is made here.

[0131] like Figure 5 As shown, Figure 5 This is a flowchart illustrating a voiceprint feature extraction method, which includes the following steps:

[0132] S501, Ecapa-tdnn model;

[0133] Electronic devices can extract voiceprint features from audio segments based on the Ecapa-tdnn model;

[0134] S502, WavLM model;

[0135] Electronic devices can extract the voiceprint features of audio segments based on the WavLM model;

[0136] S503, Normalization;

[0137] The voiceprint features extracted in step S501 are normalized so that the voiceprint features obtained in steps S501 and S502 can be spliced ​​together.

[0138] S504, Normalization;

[0139] The voiceprint features extracted in step S502 are normalized so that the voiceprint features obtained in steps S501 and S502 can be spliced ​​together.

[0140] S505, spliced ​​and PCA dimensionality reduction.

[0141] The electronic device can splice the normalized voiceprint features obtained in steps S503 and S504 and perform PCA dimensionality reduction to obtain the voiceprint features corresponding to the audio segment.

[0142] As can be seen, in this embodiment of the invention, the electronic device can divide the audio corresponding to the target video into audio segments corresponding to each line of dialogue according to the start and end times of the dialogue; and extract the voiceprint features of each audio segment to obtain the voiceprint features to be processed. Since voiceprint features can be used to represent the voice of the person corresponding to the audio segment, the voiceprint feature extraction method provided in this embodiment of the invention can accurately extract the voiceprint features corresponding to the audio segment, which can improve the accuracy of determining the person corresponding to the dialogue.

[0143] As one embodiment of the present invention, such as Figure 6 As shown, the step of clustering the voiceprint features and determining the category to which each voiceprint feature belongs may include:

[0144] S601, Based on the scene information of the target video, determine the time period corresponding to each scene included in the target video;

[0145] Since the emotions of characters and the background noise of the scene are usually similar under the same target video scene information, the voiceprint features obtained under the same target video scene information are quite similar in terms of character emotions and scene noise. Clustering the voiceprint features obtained under the same target video scene information can approximately ignore the influence of changes in character emotions and background noise on the voiceprints. Clustering the voiceprint features corresponding to videos under the same scene yields voiceprint feature categories. Then, based on the number of candidate characters, the character corresponding to the dialogue of the voiceprint feature category is determined, which improves the accuracy of determining the character corresponding to the dialogue.

[0146] This allows for quick and efficient identification of the corresponding characters in dialogue, without the need to first extract the character's voiceprint features and then compare voiceprints one by one in the speaker's voiceprint feature database.

[0147] In order to divide the target video according to different scenes, electronic devices can use transition point detection technology based on the scene information of the target video to determine the time period corresponding to each scene included in the target video. In this way, scene information can be represented by time periods.

[0148] For example, if the target video is 30 seconds long, the first 15 seconds of the target video are outdoor scenes, called scene A, and the remaining 15 seconds are indoor scenes, called scene B, then the electronic device can determine that the first 15 seconds are one scene of the target video and the remaining 15 seconds are another scene of the target video based on the outdoor scene information such as lawns and trees, and the indoor scene information such as wallpapers and furniture.

[0149] S602, cluster the voiceprint features corresponding to each time period to determine the category to which the voiceprint features corresponding to each time period belong.

[0150] Since the emotions of people and background sounds are similar in the same scene of the target video, the voiceprint features in the same scene can be clustered. Because different scenes in the target video have been divided into different time periods, the voiceprint features corresponding to the time periods of different scenes can be clustered, thus minimizing the impact of changes in people's emotions and background sounds during the clustering process.

[0151] Following the example in step S601, the voiceprint features corresponding to the first 15 seconds of the target video are voiceprint features 1-5, which correspond to lines 1-5 respectively. The voiceprint features corresponding to the last 15 seconds are voiceprint features 6-10, which correspond to lines 6-10 respectively. The voiceprint features corresponding to the first 15 seconds and the last 15 seconds are clustered separately, and the results are shown in the table below:

[0152]

[0153]

[0154] Clustering the voiceprint features corresponding to scene A yields two categories: Category 1 and Category 2. Category 1 includes voiceprint features 1, 3, and 5, while Category 2 includes voiceprint features 2 and 4. Similarly, clustering the voiceprint features corresponding to scene B yields two categories: Category 3 and Category 4. Category 3 includes voiceprint features 6, 7, and 8, while Category 4 includes voiceprint features 9 and 10. Based on the clustering results shown in the table above, since the voiceprint features in both scene A and scene B are clustered into two categories, the characters corresponding to the dialogue in both scenes A and B can be two separate groups.

[0155] As can be seen, in this embodiment of the invention, the electronic device can determine the time period corresponding to each scene in the target video based on the scene information of the target video; and cluster the voiceprint features corresponding to each time period to determine the category to which the voiceprint features corresponding to each time period belong. Since the emotions of characters and background sounds are similar in the same scene, the time period corresponding to each scene in the target video can be determined, and then the voiceprint features corresponding to each time period can be clustered. In this way, the accuracy of identifying the characters corresponding to the lines can be improved, while reducing the resource consumption of a single analysis.

[0156] As one embodiment of the present invention, such as Figure 7 As shown, the above method may further include:

[0157] S701, if it is determined that there is no corresponding person for the lines corresponding to the voiceprint features of this category based on the appearance time ratio of the person in the speaking state corresponding to the voiceprint features of this category, then the candidate person corresponding to the voiceprint features of this category is determined based on the picture information of the target video corresponding to this category.

[0158] If, based on the proportion of time a person in a speaking state appears corresponding to the voiceprint features of a certain category, it is determined that there is no corresponding person for the lines in that category, that is, there is no person in a speaking state whose appearance time proportion is greater than the first preset threshold.

[0159] Possible reasons for this situation include: the video clip in the target video corresponding to this type of voiceprint feature is an empty shot, that is, there are no people in the video, but someone is talking.

[0160] Since the characters corresponding to the dialogue usually appear in the target video, the video information of the target video corresponding to each category can be identified, thereby determining the candidate characters corresponding to the voiceprint features.

[0161] For example, as shown in the table below, the target video includes lines 1-10. The voiceprint features 1-10 corresponding to lines 1-10 are clustered to obtain voiceprint feature categories 1-4. The target video corresponding to each voiceprint feature category is then identified. Based on the video's visual information, candidate characters A and B are determined for voiceprint features of category 1, character A for category 2, character B and C for category 3, and character D for category 4.

[0162]

[0163] At this point, each voiceprint feature has been assigned a corresponding candidate character. The electronic device can determine the character corresponding to the lines of each category based on the number of candidate characters corresponding to the voiceprint features of that category.

[0164] S702, Based on the number of candidate characters corresponding to the voiceprint features of this category, determine the character corresponding to the lines of this category.

[0165] Electronic devices can determine the character corresponding to the lines of a given category of voiceprint features based on the number of candidate characters for that category.

[0166] Following the example in step S701, as shown in the table above, for category 2, the number of candidate characters is one. Then the electronic device can determine that the character corresponding to the lines 2 and 4 of voiceprint features 2 and 4 in category 2 is character A.

[0167] As can be seen, in this embodiment of the invention, the electronic device can, when determining that there is no corresponding person for a line of dialogue corresponding to a certain category of voiceprint features based on the appearance duration ratio of the speaking person corresponding to that category of voiceprint features, determine the candidate person corresponding to that category of voiceprint features based on the image information of the target video corresponding to that category; and determine the person corresponding to the line of dialogue corresponding to that category of voiceprint features based on the number of candidate persons corresponding to that category of voiceprint features. Since the person appearing most frequently in the video image information is usually the person corresponding to the line of dialogue, the candidate person corresponding to that category of voiceprint features can be determined based on the image information of the target video corresponding to each category, and then the person corresponding to the line of dialogue corresponding to that category of voiceprint features can be determined based on the number of candidate persons, without needing to search and compare voiceprint features based on a voiceprint feature database, thus improving the accuracy of determining the person corresponding to the line of dialogue.

[0168] As one embodiment of the present invention, such as Figure 8 As shown, the step of determining the candidate individuals corresponding to the voiceprint features of the category based on the image information of the target video corresponding to that category may include:

[0169] S801, Obtain video frames of the target video within the time period corresponding to the voiceprint features included in this category;

[0170] Since people usually appear in the frame when they speak in a video, video frames of the target video within the time period corresponding to the voiceprint features of that category can be obtained for each category, so as to calculate the proportion of each person appearing in the time period corresponding to the dialogue.

[0171] Following the example from step S401, as shown in the table below, taking the voiceprint features included in category 1 as an example, the dialogue time period is determined in step S401. The total dialogue time period corresponding to voiceprint features 1, 3, and 5 included in category 1 is 7 seconds. Assuming that the target video includes 24 frames per second, the electronic device can acquire a total of 168 frames of dialogue time periods corresponding to voiceprint features 1, 3, and 5.

[0172]

[0173] S802, perform face recognition on the video frames to determine the proportion of time each person appears in the video frames included in the time period;

[0174] Since the characters corresponding to the dialogue usually appear in the video footage, the proportion of each character's appearance time in the video frames of the target video within the time period corresponding to the same category of voiceprint features can be calculated. Based on the appearance proportion, the character corresponding to the voiceprint features can be determined.

[0175] Following the example of step S801, as shown in the table below, face recognition is performed frame-by-frame on the target video frames within the time period corresponding to the voiceprint features of this category, thereby determining the proportion of each person appearing. There are a total of 168 frames. Person A appears in 117 frames, so the appearance ratio of Person A is 117 / 168. Person B and Person C appear in 121 and 21 frames respectively; similarly, their appearance ratios are 121 / 168 and 21 / 168 respectively. Accordingly, the appearance duration ratios of Person A, Person B, and Person C within the video frames included in this time period are 117 / 168, 121 / 168, and 21 / 168 respectively.

[0176]

[0177] S803, identify individuals whose corresponding proportions reach the preset proportions as candidate individuals for the voiceprint characteristics of that category.

[0178] Since the person appears for a relatively long period of time, it is highly likely that the person is the one corresponding to the voiceprint feature. Therefore, a ratio can be preset. If the appearance ratio of the task obtained in step S402 reaches the preset ratio, the person can be identified as a candidate person corresponding to the voiceprint feature of this category.

[0179] Following the example of step S802, as shown in the table below, the preset ratio is 0.65. Since the appearance time ratio of person A and person B reaches 0.65, person A and person B are identified as candidate persons corresponding to the voiceprint features of this category.

[0180]

[0181]

[0182] As can be seen, in this embodiment of the invention, the electronic device can acquire video frames of the target video within the time period corresponding to the voiceprint features included in the category; perform face recognition on the video frames to determine the proportion of each person's appearance time in the video frames included in the time period; and determine the persons whose corresponding proportion reaches a preset proportion as candidate persons corresponding to the voiceprint features of that category. Since the person corresponding to the lines in the target video usually appears in the video frame, face recognition can be performed on the video frames of the target video within the time period corresponding to the voiceprint features of the same category, and the persons whose appearance proportion reaches a preset proportion can be determined as candidate persons corresponding to that voiceprint feature. Based on the above method, the proportion of each person appearing in the video frames of the target video within the time period corresponding to the voiceprint features can be accurately obtained, thereby improving the accuracy of the person corresponding to the lines.

[0183] As one embodiment of the present invention, the step of determining the character corresponding to the lines of a certain category of voiceprint features based on the number of candidate characters corresponding to that category of voiceprint features may include:

[0184] If there is only one candidate person corresponding to the voiceprint feature of this category, then the candidate person is determined to be the person corresponding to the lines of the voiceprint feature of this category.

[0185] If there are multiple candidate characters corresponding to the voiceprint features of this category, the character corresponding to the lines of this category is determined based on the similarity between the cluster center of this category and the cluster center of the first target category.

[0186] The first target category includes a category with the same scene as the category and a corresponding candidate character of one, and a category with the same scene as the category and a corresponding character whose appearance time has been determined based on the proportion of characters in a speaking state.

[0187] Since the candidate characters are those whose appearance rate in the video frames of the target video reaches a preset proportion within the time period corresponding to the voiceprint features of that category, and the characters corresponding to the lines usually appear in the video screen with a high frequency, if there is only one candidate character corresponding to the voiceprint features of that category, the electronic device can determine that the candidate character is the character corresponding to the lines of that category of voiceprint features.

[0188] Following the example in step S702, as shown in the table below, since the number of candidate characters corresponding to the voiceprint features of category 2 and category 4 is only one, it can be determined that character A is the character corresponding to the lines of the voiceprint features of category 2, and character D is the character corresponding to the lines of the voiceprint features of category 4.

[0189]

[0190] Since each category obtained after clustering voiceprint features has a cluster center, and these cluster centers represent the characteristics of that category's voiceprint features, the cluster centers of two categories obtained from clustering the voiceprint features of the same person have a high degree of similarity. Because a person's emotions and scene noise are similar in the same scene, comparing the cluster centers of the same category in two scenes can make the results more accurate.

[0191] When there are multiple candidate individuals corresponding to the voiceprint features of a certain category—that is, when multiple individuals appear in video frames of the target video within the time period corresponding to the voiceprint features of that category at a predetermined proportion—the similarity between the cluster centers of this category and the cluster centers of the first target category can be compared. The first target category includes categories with the same scene as the category and only one candidate individual, as well as categories with the same scene as the category and whose corresponding individuals have been determined based on the proportion of time they are speaking. Since the first target category corresponds to only one individual, its cluster centers have significant reference value.

[0192] Continuing from the previous example, as shown in the table below, there are multiple candidate characters corresponding to the voiceprint features of categories 1 and 3. Since category 2 corresponds to the same scene as category 1 and has only one candidate character, category 2 can be used as the first target category for category 1. Similarly, category 4 can be used as the first target category for category 3. Electronic devices can determine the character corresponding to the dialogue of category 1 based on the similarity between the cluster centers of category 1 and category 2. Similarly, electronic devices can also determine the character corresponding to the dialogue of category 3 based on the similarity between the cluster centers of category 3 and category 4.

[0193]

[0194] As can be seen, in this embodiment of the invention, if the number of candidate characters corresponding to the voiceprint features of a certain category is one, the electronic device can determine that the candidate character is the character corresponding to the lines corresponding to the voiceprint features of that category; if the number of candidate characters corresponding to the voiceprint features of a certain category is multiple, the electronic device can determine the character corresponding to the lines corresponding to the voiceprint features of that category based on the similarity between the cluster center of that category and the cluster center of the first target category. The first target category includes categories that are in the same scene as the category and have one candidate character, and categories that are in the same scene as the category and have already determined the corresponding character based on the proportion of the appearance time of characters in the speaking state. Since the characters corresponding to the dialogue usually appear frequently in the video footage, if there is only one candidate character for a particular voiceprint feature, then that candidate character can be identified as the character corresponding to the dialogue for that voiceprint feature. Since the cluster center represents the category obtained by clustering voiceprint features, and comparing cluster centers in the same scene can reduce interference from background noise and character emotions, for categories with multiple candidate characters corresponding to voiceprint features, the cluster center similarity can be compared with the first target category to determine the character corresponding to the dialogue for that category. Based on the first target category, the character corresponding to the dialogue can be determined more quickly, improving the computational speed for determining the character corresponding to the dialogue.

[0195] As one embodiment of the present invention, such as Figure 9 As shown, the step of determining the character corresponding to the dialogue of a given category based on the similarity between the cluster centers of the current category and the cluster centers of the first target category may include:

[0196] S901, calculate the similarity between the cluster center of this category and the cluster center of each first target category;

[0197] Since the similarity between cluster centers can represent the degree of similarity between voiceprint features of categories, if the similarity between cluster centers is high, it can be determined that the person corresponding to the first target category is the person corresponding to the voiceprint features of that category. Therefore, electronic devices can calculate the similarity between the cluster center of that category and the cluster center of each first target category.

[0198] For example, the voiceprint features corresponding to the time period in the video corresponding to scene C are clustered into categories 5-9. The candidate persons corresponding to categories 5-9 are person A and person B, person C, person A, person D, and person B, respectively. For category 5, since the number of candidate persons corresponding to categories 6-9 is only one, and the scene corresponding to category 5 is the same, categories 6-9 are the first target categories. Category 5 can be compared with the cluster centers of categories 6-9 respectively.

[0199]

[0200] Continuing from the previous example, if the voiceprint feature category corresponding to scenario C also includes category 10, where category 10 is the category of the person whose appearance time is determined based on the proportion of the person in the speaking state, then category 10 is the first target category, and category 5 can be compared with the cluster center of category 10 in terms of similarity.

[0201] S902, the person corresponding to the first target category whose similarity reaches the first preset similarity is identified as the person corresponding to the voiceprint feature of that category.

[0202] Since a high similarity between cluster centers indicates that the voiceprint features of the two are relatively similar, the category can identify the person corresponding to the first target category whose similarity reaches the first preset similarity as the person corresponding to the lines of the voiceprint features of that category.

[0203] Following the first example in step S901, if the first preset similarity is 0.8, and the similarities between the cluster center of category 5 and the cluster centers of categories 6, 7, 8, and 9 are 0.1, 0.2, 0.1, and 0.9 respectively, since the similarity between the cluster centers of category 5 and category 9 reaches the first preset similarity, it can be determined that the character B corresponding to category 9 is the character corresponding to the lines of the voiceprint feature of category 5.

[0204] As can be seen, in this embodiment of the invention, the electronic device can calculate the similarity between the cluster center of the current category and the cluster center of each first target category; the person corresponding to the first target category whose similarity reaches a first preset similarity is determined as the person corresponding to the line corresponding to the voiceprint feature of that category. Since a high similarity of the cluster centers indicates that the voiceprint features corresponding to the cluster centers are relatively close, if the similarity between the current category and the cluster center of the first target category reaches a first preset similarity, then the person corresponding to the first target category whose similarity reaches the first preset similarity can be determined as the person corresponding to the line corresponding to the voiceprint feature of that category; by comparing the similarity between the cluster centers of categories with multiple candidate persons and the first target category, the person corresponding to the category with multiple candidate persons can be quickly determined, improving the computational efficiency of the line corresponding to the person.

[0205] As one embodiment of the present invention, such as Figure 10 As shown, the above method may further include:

[0206] S1001, If ​​the similarity between the cluster center of this category and the cluster center of each first target category does not reach the first preset similarity, calculate the similarity between the cluster center of this category and the cluster center of each second target category respectively;

[0207] Among them, the second target category is a category that is different from the scene corresponding to the category and has one corresponding candidate character, and a category that is different from the scene corresponding to the category and has been determined based on the proportion of the appearance time of the character in the speaking state.

[0208] Since the similarity between the cluster center of this category and the cluster center of each first target category is less than the first preset similarity, and the cluster center of the second target category can be compared with the cluster center of this category, if the similarity between the cluster center of this category and the cluster center of each first target category does not reach the first preset similarity, the similarity between the cluster center of this category and the cluster center of each second target category is calculated respectively.

[0209] For example, as shown in the table below, since there is only one candidate character for category 4, and the scenarios corresponding to category 4 and category 3 are the same, the similarity between the cluster centers obtained for category 3 and category 4 is less than the first preset similarity. Therefore, the similarity between category 3 and the cluster centers of each first target category has not reached the first preset similarity. Since there is only one candidate character for category 2, and the scenarios corresponding to category 2 and category 3 are different, the similarity between the cluster centers of category 3 and category 4 can be calculated.

[0210]

[0211] Continuing from the previous example, if there is another scenario C, which includes category 5, where the category of a person has been determined based on the proportion of time a person is in a speaking state, and the person corresponding to category 5 is person B. Since the similarity between the cluster centers of category 3 and category 4 does not reach the first preset similarity, and since category 5 corresponds to a different scenario than category 3, and the category of the person has already been determined based on the proportion of time a person is in a speaking state, the electronic device can calculate the similarity between the cluster centers of category 3 and category 5.

[0212] S1002, the person corresponding to the second target category whose similarity reaches the second preset similarity is identified as the person corresponding to the voiceprint feature of that category.

[0213] Since the second target category is different from the scene corresponding to that category, the background sounds and characters' emotions in the two scenes may be different. Therefore, the second preset similarity can be less than the first preset similarity.

[0214] If the similarity between the cluster centers of the current category and the second target category reaches the second preset similarity, then the person corresponding to the second target category whose similarity reaches the second preset similarity can be determined as the person corresponding to the lines corresponding to the voiceprint features of the current category.

[0215] As can be seen, in this embodiment of the invention, if the similarity between the cluster center of the current category and the cluster center of each first target category does not reach the first preset similarity, the electronic device can calculate the similarity between the cluster center of the current category and the cluster center of each second target category. The second target category includes categories with different scenes and one corresponding candidate character, as well as categories with different scenes and whose corresponding characters are determined based on the proportion of characters appearing in a speaking state. The character corresponding to the second target category whose similarity reaches the second preset similarity is determined as the character corresponding to the dialogue in that category. Since the similarity between the cluster center of the first target category and the cluster center of the current category does not reach the first preset similarity, the current category can calculate the similarity between its cluster centers and the second target category. If the similarity between the cluster centers of the current category and the second target category reaches the second preset similarity, then the character corresponding to the second target category whose similarity reaches the second preset similarity can be determined as the character corresponding to the dialogue in that category. This improves the success rate of matching dialogue to characters.

[0216] As one embodiment of the present invention, such as Figure 11 As shown, the above method may further include:

[0217] S1101, if the similarity between the cluster center of this category and the cluster center of each second target category does not reach the second preset similarity, for each voiceprint feature of this category, calculate the similarity between the voiceprint feature and other voiceprint features in this category;

[0218] Since the similarity between the cluster center of this category and the cluster center of each second target category does not reach the second preset similarity, the electronic device can calculate the similarity between the voiceprint feature and other voiceprint features in this category to check whether the voiceprint feature was incorrectly assigned to a category in the clustering operation.

[0219] S1102, if the proportion of similarity reaching the second preset threshold is less than the preset proportion, obtain each voiceprint feature to be compared within a preset duration including the video time point corresponding to the voiceprint feature.

[0220] Since the similarity ratio between the voiceprint feature and other voiceprint features in the same category that reaches the second preset threshold is less than the preset threshold, meaning that the similarity ratio between the voiceprint feature and other voiceprint features in the same category that reaches the second preset threshold is low, the voiceprint feature may be misclassified into the category. The electronic device can acquire each voiceprint feature to be compared within a preset duration, including the video time point corresponding to the voiceprint feature.

[0221] For example, if the preset duration is 45 seconds, if the similarity between voiceprint feature A and other voiceprint features of the same category is calculated, and the proportion of similarity reaching the threshold is less than the preset proportion, then the electronic device can obtain other voiceprint features to be compared within a 45-second time period before and after the time point of the dialogue corresponding to voiceprint feature A in the target video.

[0222] S1103, calculate the similarity between the voiceprint feature and each voiceprint feature to be compared;

[0223] Similarity can represent the degree of similarity between a voiceprint feature and a voiceprint feature to be compared. In order to determine whether the voiceprint feature is clustered incorrectly, the similarity between the voiceprint feature and each voiceprint feature to be compared can be calculated separately.

[0224] Following the example in step S1102, the electronic device can calculate the similarity between voiceprint feature A and the lines corresponding to voiceprint feature A in the target video within a 45-second time period before and after the time point.

[0225] S1104, the person corresponding to the voiceprint feature with the highest similarity is identified as the person corresponding to the line of the voiceprint feature.

[0226] Because the original clustering of the voiceprint feature was incorrect, when calculating the similarity between the voiceprint feature and each voiceprint feature to be compared, the person corresponding to the voiceprint feature with the highest similarity can be identified as the person corresponding to the line of the voiceprint feature, and the voiceprint feature is added to the category to which the voiceprint feature with the highest similarity belongs, so that the voiceprint feature can be used in subsequent applications.

[0227] As can be seen, in this embodiment of the invention, if the similarity between the cluster center of the current category and the cluster center of each second target category does not reach the second preset similarity, for each voiceprint feature of the current category, the similarity between the voiceprint feature and other voiceprint features in the current category is calculated; if the proportion of similarity reaching the second preset threshold is less than the preset proportion, each voiceprint feature to be compared within a preset duration including the video time point corresponding to the voiceprint feature is obtained; the similarity between the voiceprint feature and each voiceprint feature to be compared is calculated respectively; the person corresponding to the voiceprint feature with the highest similarity is determined as the person corresponding to the line corresponding to the voiceprint feature; since the voiceprint feature was misclassified in the clustering process, the similarity between the voiceprint feature and each voiceprint feature to be compared within the preset duration of the video time point corresponding to the voiceprint feature can be calculated, and the person corresponding to the voiceprint feature with the highest similarity can be determined as the person corresponding to the line corresponding to the voiceprint feature; the voiceprint features with incorrect clustering can be corrected, further improving the success rate of the line corresponding to the person.

[0228] like Figure 12 As shown, Figure 12 A flowchart illustrating a method for mapping dialogue to characters provided in an embodiment of the present invention is shown below:

[0229] S1201, Activate Speaker Detection;

[0230] Electronic devices can divide a target video into video segments corresponding to each line of dialogue, based on the start and end times of lines 1-N, thus obtaining video segments 1-N corresponding to lines 1-N. A speaker detection model is then used to identify the speaker within the video segments corresponding to this category, determining the area where the person speaking is located.

[0231] S1202, transition point segmentation and scene clustering;

[0232] Electronic devices can divide the audio corresponding to the target video into audio segments 1-N according to the start and end times of lines 1-N. Based on the Ecapa-tdnn model and the WavLM model, speaker voiceprint features are extracted from audio segments 1, 2, ..., N, resulting in voiceprint features 1, 2, ..., N corresponding to each audio segment.

[0233] Furthermore, electronic devices can segment the target video into time periods according to the transition points of the target video, cluster the voiceprint features within the time periods corresponding to the same scene, divide the scene into scene a, scene b... scene m, and cluster the voiceprint features corresponding to the time periods of each scene to obtain several clusters under each scene, i.e., classification.

[0234] S1203, perform facial recognition on video clips within clusters, and statistically analyze the proportion of associated characters;

[0235] Electronic devices can perform facial recognition on video clips corresponding to voiceprint features included in each cluster. Based on statistical proportions and related roles, i.e. candidate characters, the association between clusters and character roles in each scene can be obtained, i.e., the correspondence between categories and candidate characters.

[0236] S1204, Cross-Scenario: Clustering - Character Association Role Purification;

[0237] In this scenario, if the category of a character cannot be determined based on the number of candidate characters (only one) or the proportion of time a character is in a speaking state, the category of the character corresponding to the dialogue can be obtained by comparing the similarity of the cluster centers. This can be done by comparing the cluster center similarity of the candidate characters across scenarios with categories of one candidate character or categories of characters already determined based on the proportion of time a character is in a speaking state. This will purify the candidate characters, that is, eliminate candidate characters with low similarity.

[0238] S1205 verifies and compares each line of dialogue with the speaker, and refines the connection between abrupt lines and speakers.

[0239] For lines that still do not have a corresponding character, their corresponding voiceprint features can be compared with the similarity of other voiceprint features in the same category. If the similarity threshold is less than the preset percentage, it means that the line has been misclassified into this category. Therefore, the similarity can be calculated between the line and the voiceprint features corresponding to the line sequence 1-N within a preset time period before and after the corresponding time point in the video. The character corresponding to the line with the voiceprint feature with the highest similarity is taken as the corresponding character.

[0240] Corresponding to the above method for determining the character corresponding to a line, this embodiment of the invention also provides a device for determining the character corresponding to a line. The following is a description of the device for determining the character corresponding to a line provided by this embodiment of the invention.

[0241] like Figure 13 As shown, a device for determining the character corresponding to a line of dialogue, the device comprising:

[0242] The voiceprint feature acquisition module 1301 is used to acquire voiceprint features to be processed, wherein the voiceprint features to be processed are voiceprint features corresponding to lines in the target video;

[0243] The voiceprint feature clustering module 1302 is used to cluster the voiceprint features and determine the category to which each voiceprint feature belongs;

[0244] The proportion determination module 1303 is used to determine the proportion of the appearance time of the person in the speaking state corresponding to the voiceprint feature of each category, based on the picture information of the video segment in the target video corresponding to that category.

[0245] The first character identification module 1304 is used to determine the character corresponding to the lines of a certain type of voiceprint feature based on the proportion of time the characters in the speaking state appear corresponding to the voiceprint features of that type.

[0246] As can be seen, in the solution provided by this invention, the electronic device can acquire voiceprint features to be processed, wherein the voiceprint features to be processed are voiceprint features corresponding to lines in a target video; the voiceprint features are clustered to determine the category to which each voiceprint feature belongs; for each category, based on the image information of the video segment in the target video corresponding to that category, the appearance duration ratio of the person in the speaking state corresponding to the voiceprint features of that category is determined; based on the appearance duration ratio of the person in the speaking state corresponding to the voiceprint features of that category, the person corresponding to the lines corresponding to the voiceprint features of that category is determined. In this solution, since the voiceprint features corresponding to the lines spoken by the same person in the target video are similar, the voiceprint features corresponding to the lines in the target video can be clustered to determine the category to which each voiceprint feature belongs. Since the person appearing in a speaking state for a higher proportion in the video footage is usually the person corresponding to the lines, the proportion of the speaking state corresponding to the voiceprint features of that category can be determined based on the video footage of the target video corresponding to that category. Then, based on the proportion of the speaking state corresponding to the voiceprint features of that category, the person corresponding to the lines corresponding to the lines corresponding to that category can be determined, without the need to search and compare voiceprint features based on a voiceprint feature database, thus improving the accuracy of determining the person corresponding to the lines.

[0247] As one embodiment of the present invention, the ratio determination module 1303 may include:

[0248] The first person determination submodule is used to perform speaker detection on the video clips in the target video corresponding to the category, determine the area where the person in the speaking state is located, and perform face recognition on the area where the person is located to determine the person in the speaking state in the area where the person is located.

[0249] The second proportion determination submodule is used to determine the proportion of the appearance time of the person in the video segment corresponding to the voiceprint feature of that category, based on the number of times the person in the speaking state appears in the video segment.

[0250] As one embodiment of the present invention, the first person determination submodule may include:

[0251] The video segment division unit is used to divide the target video into video segments corresponding to each line of dialogue according to the start and end times of the dialogue.

[0252] The first person determination unit is used to use a speaker detection model to detect speakers in the video clips corresponding to the category, determine the area where the person in the speaking state is located, and perform face recognition on the area where the person is located to determine the person in the speaking state in the area where the person is located.

[0253] As one embodiment of the present invention, the first person determination module 1304 may include:

[0254] The second character determination submodule is used to identify the character with the highest appearance duration among those who are speaking and whose appearance duration is greater than the first preset threshold, as the character corresponding to the lines corresponding to the voiceprint features of that category.

[0255] As one embodiment of the present invention, the voiceprint feature acquisition module 1301 may include:

[0256] The audio segment division submodule is used to divide the audio corresponding to the target video into audio segments corresponding to each line of dialogue according to the start and end times of the dialogue.

[0257] The voiceprint feature extraction submodule is used to extract the voiceprint features of each audio segment to obtain the voiceprint features to be processed.

[0258] As one embodiment of the present invention, the voiceprint feature clustering module 1302 may include:

[0259] The time period determination submodule is used to determine the time period corresponding to each scene included in the target video based on the scene information of the target video;

[0260] The category determination submodule is used to cluster the voiceprint features corresponding to each time period and determine the category to which the voiceprint features corresponding to each time period belong.

[0261] As one embodiment of the present invention, the apparatus may further include:

[0262] The candidate character determination module is used to determine the candidate character corresponding to the voiceprint feature of a certain category based on the image information of the target video corresponding to that category, when it is determined that there is no corresponding character for the lines corresponding to the voiceprint feature of that category based on the appearance time ratio of the characters in the speaking state corresponding to the voiceprint feature of that category.

[0263] The second character identification module is used to determine the character corresponding to the lines of a given category of voiceprint features based on the number of candidate characters corresponding to that category of voiceprint features.

[0264] As one embodiment of the present invention, the candidate candidate determination module may include:

[0265] The video frame acquisition submodule is used to acquire video frames of the target video within the time period corresponding to the voiceprint features included in the category.

[0266] The proportion determination submodule is used to perform face recognition on the video frames and determine the proportion of the appearance time of each person in the video frames included in the time period;

[0267] The candidate person determination submodule is used to determine the candidates whose voiceprint features correspond to the preset proportion of the corresponding person.

[0268] As one embodiment of the present invention, the second person determination module may include:

[0269] The third-person identification submodule is used to determine the candidate person as the person corresponding to the lines corresponding to the voiceprint features of that category if there is only one candidate person for that category.

[0270] The fourth character determination submodule is used to determine the character corresponding to the lines of a certain category based on the similarity between the cluster center of the category and the cluster center of the first target category if there are multiple candidate characters corresponding to the voiceprint features of that category. The first target category includes categories that are in the same scene as the category and have one candidate character, as well as categories that are in the same scene as the category and have already determined the corresponding character based on the proportion of the appearance time of characters in the speaking state.

[0271] As one embodiment of the present invention, the fourth person determination submodule may include:

[0272] The similarity calculation unit is used to calculate the similarity between the cluster center of this category and the cluster center of each first target category.

[0273] The second character identification unit is used to identify the characters corresponding to the first target category whose similarity reaches the first preset similarity as the characters corresponding to the lines of the voiceprint features of that category.

[0274] As one embodiment of the present invention, the apparatus may further include:

[0275] The first similarity calculation module is used to calculate the similarity between the cluster center of the category and the cluster center of each first target category if the similarity between the cluster center of the category and the cluster center of each second target category does not reach the first preset similarity. The second target category is a category that is different from the scene corresponding to the category and has one corresponding candidate person, and a category that is different from the scene corresponding to the category and has been determined based on the appearance time ratio of the person in the speaking state.

[0276] The third-person identification module is used to identify the person corresponding to the second target category whose similarity reaches the second preset similarity as the person corresponding to the voiceprint feature of that category.

[0277] As one embodiment of the present invention, the apparatus may further include:

[0278] The second similarity calculation module is used to calculate the similarity between the voiceprint feature and other voiceprint features in the category for each voiceprint feature of the category if the similarity between the cluster center of the category and the cluster center of each second target category does not reach the second preset similarity.

[0279] The voiceprint feature acquisition module is used to acquire each voiceprint feature to be compared within a preset duration, including the video time point corresponding to the voiceprint feature, if the proportion of similarity reaching the second preset threshold is less than the preset proportion.

[0280] The third similarity calculation module is used to calculate the similarity between the voiceprint feature and each voiceprint feature to be compared.

[0281] The fourth character identification module identifies the character whose voiceprint feature has the highest similarity to the character being compared, and determines the character whose lines correspond to that voiceprint feature.

[0282] This invention also provides an electronic device, such as... Figure 14 As shown, it includes a processor 1401, a communication interface 1402, a memory 1403, and a communication bus 1404, wherein the processor 1401, the communication interface 1402, and the memory 1403 communicate with each other through the communication bus 1404.

[0283] Memory 1403 is used to store computer programs;

[0284] When the processor 1401 executes the program stored in the memory 1403, it implements the steps of the method for determining the character corresponding to the lines described in any of the above embodiments.

[0285] As can be seen, in the solution provided by this invention, the electronic device can acquire voiceprint features to be processed, wherein the voiceprint features to be processed are voiceprint features corresponding to lines in a target video; the voiceprint features are clustered to determine the category to which each voiceprint feature belongs; for each category, based on the image information of the video segment in the target video corresponding to that category, the appearance duration ratio of the person in the speaking state corresponding to the voiceprint features of that category is determined; based on the appearance duration ratio of the person in the speaking state corresponding to the voiceprint features of that category, the person corresponding to the lines corresponding to the voiceprint features of that category is determined. In this solution, since the voiceprint features corresponding to the lines spoken by the same person in the target video are similar, the voiceprint features corresponding to the lines in the target video can be clustered to determine the category to which each voiceprint feature belongs. Since the person appearing in a speaking state for a higher proportion in the video footage is usually the person corresponding to the lines, the proportion of the speaking state corresponding to the voiceprint features of that category can be determined based on the video footage of the target video corresponding to that category. Then, based on the proportion of the speaking state corresponding to the voiceprint features of that category, the person corresponding to the lines corresponding to the lines corresponding to that category can be determined, without the need to search and compare voiceprint features based on a voiceprint feature database, thus improving the accuracy of determining the person corresponding to the lines.

[0286] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0287] The communication interface is used for communication between the aforementioned terminal and other devices.

[0288] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0289] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0290] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the method for determining the character corresponding to the lines described in any of the above embodiments.

[0291] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the method for determining the character corresponding to the lines described in any of the above embodiments.

[0292] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0293] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0294] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0295] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for determining the character corresponding to a line of dialogue, characterized in that, The method includes: Obtain the voiceprint features to be processed, wherein the voiceprint features to be processed are the voiceprint features corresponding to the lines in the target video; Based on the scene information of the target video, the time period corresponding to each scene in the target video is determined using transition point detection technology; Cluster the voiceprint features corresponding to each time period to determine the category to which the voiceprint features corresponding to each time period belong; For each category, based on the image information of the video segments in the target video corresponding to that category, the proportion of the appearance time of the person in the speaking state corresponding to the voiceprint features of that category is determined; Based on the proportion of time a person in a speaking state appears corresponding to the voiceprint feature of that category, the person corresponding to the line of dialogue corresponding to that category of voiceprint feature is determined. The method further includes: If, based on the proportion of time a person in a speaking state appears corresponding to a certain category of voiceprint features, it is determined that there is no corresponding person for the lines corresponding to that category of voiceprint features, then, based on the image information of the target video corresponding to that category, a candidate person corresponding to that category of voiceprint features is determined. If there is only one candidate person corresponding to the voiceprint feature of this category, then the candidate person is determined to be the person corresponding to the lines of the voiceprint feature of this category. If there are multiple candidate characters corresponding to the voiceprint features of this category, the character corresponding to the lines of this category is determined based on the similarity between the cluster center of this category and the cluster center of the first target category. The first target category includes categories that are in the same scene as this category and have one candidate character, as well as categories that are in the same scene as this category and have had their corresponding characters determined based on the proportion of time characters appearing in a speaking state.

2. The method according to claim 1, characterized in that, The step of determining the proportion of the appearance duration of a person in a speaking state corresponding to the voiceprint features of that category based on the image information of the video segments in the target video corresponding to that category includes: Speaker detection is performed on the video clips in the target video corresponding to the category to determine the area where the person in the speaking state is located, and face recognition is performed on the area where the person is located to determine the person in the speaking state in the area where the person is located. Based on the number of times the person in a speaking state appears in the video clip, the proportion of the speaking state corresponding to the voiceprint feature of that category in the video clip is determined.

3. The method according to claim 2, characterized in that, The steps of performing speaker detection on video segments in the target video corresponding to the category, determining the area where a person is speaking, and performing facial recognition on the area where the person is speaking to determine the person in the area who is speaking include: The target video is divided into video segments corresponding to each line of dialogue according to the start and end times of the dialogue. A speaker detection model is used to detect speakers in the video clips corresponding to this category, determine the area where the person in the speaking state is located, and perform face recognition on the area where the person is located to determine the person in the speaking state in the area.

4. The method according to claim 1, characterized in that, The step of determining the character corresponding to the lines based on the proportion of time a character in a speaking state appears according to the voiceprint features of that category includes: Among the characters who are speaking and whose speaking time exceeds a first preset threshold, the character with the highest speaking time is identified as the character whose voiceprint features correspond to the lines of that category.

5. The method according to claim 1, characterized in that, The step of obtaining the voiceprint features to be processed includes: The audio corresponding to the target video is divided into audio segments corresponding to each line of dialogue according to the start and end times of the dialogue. Extract the voiceprint features of each audio segment to obtain the voiceprint features to be processed.

6. The method according to claim 1, characterized in that, The step of determining the candidate individuals corresponding to the voiceprint features of the category based on the image information of the target video corresponding to that category includes: Obtain video frames of the target video within the time period corresponding to the voiceprint features included in this category; Perform facial recognition on the video frames to determine the proportion of time each person appears in the video frames included in that time period; Individuals whose corresponding proportions reach the preset proportions are identified as candidate individuals for that category of voiceprint characteristics.

7. The method according to claim 1, characterized in that, The step of determining the character corresponding to the voiceprint features of a certain category based on the similarity between the cluster centers of the current category and the cluster centers of the first target category includes: Calculate the similarity between the cluster centers of this category and the cluster centers of each first target category; The person corresponding to the first target category whose similarity reaches the first preset similarity is identified as the person corresponding to the voiceprint feature of that category.

8. The method according to claim 7, characterized in that, The method further includes: If the similarity between the cluster center of this category and the cluster center of each first target category does not reach the first preset similarity, calculate the similarity between the cluster center of this category and the cluster center of each second target category respectively. The second target category is a category that is different from the scene corresponding to this category and has one corresponding candidate person, and a category that is different from the scene corresponding to this category and has been determined based on the appearance time ratio of the person in the speaking state. The characters corresponding to the second target category whose similarity reaches the second preset similarity are identified as the characters whose voiceprint features correspond to the lines of that category.

9. The method according to claim 8, characterized in that, The method further includes: If the similarity between the cluster center of this category and the cluster center of each second target category does not reach the second preset similarity, for each voiceprint feature of this category, calculate the similarity between the voiceprint feature and other voiceprint features in this category; If the proportion of similarity reaching the second preset threshold is less than the preset proportion, obtain each voiceprint feature to be compared within a preset duration, including the video time point corresponding to the voiceprint feature. Calculate the similarity between the voiceprint feature and each voiceprint feature to be compared; The person whose voiceprint feature has the highest similarity to the one to be compared is identified as the person whose lines correspond to that voiceprint feature.

10. A device for determining the character corresponding to a line of dialogue, characterized in that, The device includes: The voiceprint feature acquisition module is used to acquire voiceprint features to be processed, wherein the voiceprint features to be processed are the voiceprint features corresponding to the lines in the target video; The voiceprint feature clustering module is used to cluster the voiceprint features and determine the category to which each voiceprint feature belongs; The proportion determination module is used to determine the proportion of the appearance time of the person in the speaking state corresponding to the voiceprint features of each category, based on the picture information of the video segment in the target video corresponding to that category. The first character identification module is used to determine the character corresponding to the lines of a certain type of voiceprint feature based on the proportion of time the characters in the speaking state appear corresponding to the voiceprint features of that type. The voiceprint feature clustering module includes: The time period determination submodule is used to determine the time period corresponding to each scene included in the target video based on the scene information of the target video and using transition point detection technology; The category determination submodule is used to cluster the voiceprint features corresponding to each time period and determine the category to which the voiceprint features corresponding to each time period belong. The device further includes: The candidate character determination module is used to determine the candidate character corresponding to the voiceprint feature of a certain category based on the image information of the target video corresponding to that category, when it is determined that there is no corresponding character for the lines corresponding to the voiceprint feature of that category based on the appearance time ratio of the characters in the speaking state corresponding to the voiceprint feature of that category. The second character determination module includes: a third character determination submodule, used to determine the candidate character as the character corresponding to the line corresponding to the voiceprint feature of the category if the number of candidate characters corresponding to the voiceprint feature of the category is one; and a fourth character determination submodule, used to determine the character corresponding to the line corresponding to the voiceprint feature of the category if the number of candidate characters corresponding to the voiceprint feature of the category is multiple, based on the similarity between the cluster center of the category and the cluster center of the first target category, wherein the first target category includes categories with the same scene as the category and the number of candidate characters corresponding to it is one, and categories with the same scene as the category and whose corresponding characters have been determined based on the proportion of appearance time of characters in the speaking state.

11. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-9.

Images