Method, device and electronic equipment for determining characters corresponding to lines

Abstract

Embodiments of the present invention provide a method, device, and electronic device for determining the character corresponding to a line. The electronic device can obtain voiceprint features to be processed; cluster the voiceprint features to determine the category to which each voiceprint feature belongs; determine the candidate characters corresponding to the voiceprint features of that category based on the target video screen information corresponding to each category; and for each category, determine the character corresponding to the lines corresponding to the voiceprint features of that category based on the number of candidate characters corresponding to the voiceprint features of that category. Since the voiceprint features corresponding to the lines spoken by the same character in the target video are similar, the voiceprint features corresponding to the lines are clustered to determine the category to which the voiceprint features belong. Since characters that appear frequently in the video screen are usually the characters corresponding to the lines, the candidate characters corresponding to the voiceprint features of that category are determined based on the screen information corresponding to the category, and the character corresponding to the lines corresponding to the voiceprint features of that category is determined based on the number of candidate characters, thereby improving the accuracy of determining the character corresponding to the lines.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a method, device and electronic equipment for determining characters corresponding to lines. Background Art

[0002] In video editing and video commentary, it is necessary to identify the corresponding character for each line in the video, that is, the speaker of the line, so that the video can be edited and commented on better. To determine the corresponding character for each line in the video, it is necessary to extract the voiceprint features of each character in the video, thereby establishing a voiceprint feature library. Then, the voiceprint features corresponding to each line in the video are extracted. Furthermore, each extracted voiceprint feature is compared one by one with the voiceprint features of each character in the voiceprint feature library. The most similar voiceprint feature is found in the voiceprint feature library, and the character corresponding to the voiceprint feature is determined as the speaker of the line.

[0003] However, in the above method of determining the characters corresponding to the lines, since the background sounds in different scenes may be different and the emotions of the characters may be different, the voiceprint features of the same character extracted in different scenes may be quite different. In addition, the timbre of different characters is very similar due to the same voice actor. Therefore, the accuracy of determining the characters corresponding to the lines through the voiceprint feature library is not high. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a method, device, and electronic device for determining the corresponding characters of lines, so as to improve the accuracy of determining the corresponding characters of lines. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a method for determining a character corresponding to a line, the method comprising:

[0006] Acquire a voiceprint feature to be processed, wherein the voiceprint feature to be processed is a voiceprint feature corresponding to a line in a target video;

[0007] Clustering the voiceprint features to determine the category to which each voiceprint feature belongs;

[0008] Based on the image information of the target video corresponding to each category, determining the candidate person corresponding to the voiceprint feature of the category;

[0009] For each category, the character corresponding to the lines corresponding to the voiceprint features of the category is determined based on the number of candidate characters corresponding to the voiceprint features of the category.

[0010] Optionally, the step of obtaining the voiceprint features to be processed includes:

[0011] Divide the audio corresponding to the target video into audio segments corresponding to each line according to the start time and end time of the line;

[0012] The voiceprint features of each audio clip are extracted to obtain the voiceprint features to be processed.

[0013] Optionally, the step of clustering the voiceprint features and determining the category to which each voiceprint feature belongs includes:

[0014] Determining, based on the scene information of the target video, a time period corresponding to each scene included in the target video;

[0015] The voiceprint features corresponding to each time period are clustered to determine the category to which the voiceprint features corresponding to each time period belong.

[0016] Optionally, the step of determining a candidate person corresponding to the voiceprint feature of each category based on the picture information of the target video corresponding to the category includes:

[0017] For each category, obtaining video frames of the target video within a time period corresponding to the voiceprint features included in the category;

[0018] Performing face recognition on the video frames to determine the proportion of the appearance time of each character in the video frames included in the time period;

[0019] The person whose corresponding ratio reaches the preset ratio is determined as the candidate person corresponding to the voiceprint feature of this category.

[0020] Optionally, the step of determining the character corresponding to the lines corresponding to the voiceprint features of this category according to the number of candidate characters corresponding to the voiceprint features of this category includes:

[0021] If the number of candidate characters corresponding to the voiceprint feature of the category is one, determine that the candidate character is the character corresponding to the lines corresponding to the voiceprint feature of the category;

[0022] If there are multiple alternative characters corresponding to the voiceprint features of this category, the character corresponding to the lines corresponding to the voiceprint features of this category is determined based on the similarity between the cluster center of this category and the cluster center of the first target category, wherein the first target category is the same as the scene corresponding to this category, and the number of corresponding alternative characters is one.

[0023] Optionally, the step of determining the character corresponding to the lines corresponding to the voiceprint features of the category based on the similarity between the cluster center of the category and the cluster center of the first target category includes:

[0024] Calculate the similarity between the cluster center of the category and the cluster center of each first target category respectively;

[0025] The candidate characters corresponding to the first target category whose corresponding similarity reaches the first preset similarity are determined as the characters corresponding to the lines corresponding to the voiceprint features of the category.

[0026] Optionally, the method further includes:

[0027] If the similarity between the cluster center of the category and the cluster center of each first target category does not reach the first preset similarity, respectively calculate the similarity between the cluster center of the category and the cluster center of each second target category, wherein the second target category is a category that has a different scene from the category and has only one candidate character.

[0028] The candidate characters corresponding to the second target category whose corresponding similarity reaches the second preset similarity are determined as the characters corresponding to the lines corresponding to the voiceprint features of the category.

[0029] Optionally, the method further includes:

[0030] If the similarity between the cluster center of the category and the cluster center of each second target category does not reach the second preset similarity, for each voiceprint feature of the category, calculate the similarity between the voiceprint feature and other voiceprint features in the category;

[0031] If the proportion of similarities reaching the preset threshold is less than the preset proportion, obtaining each voiceprint feature to be compared within a preset time period including the video time point corresponding to the voiceprint feature;

[0032] Calculate the similarity between the voiceprint feature and each voiceprint feature to be compared respectively;

[0033] The character corresponding to the voiceprint feature to be compared with the highest similarity is determined as the character corresponding to the line corresponding to the voiceprint feature.

[0034] In a second aspect, an embodiment of the present invention provides a device for determining a character corresponding to a line, the device comprising:

[0035] A first voiceprint feature acquisition module is configured to acquire a voiceprint feature to be processed, wherein the voiceprint feature to be processed is a voiceprint feature corresponding to a line in a target video;

[0036] A voiceprint feature clustering module, configured to cluster the voiceprint features and determine the category to which each voiceprint feature belongs;

[0037] A candidate person determination module is used to determine the candidate person corresponding to the voiceprint feature of each category based on the image information of the target video corresponding to each category;

[0038] The first character determination module is used to determine, for each category, the character corresponding to the lines corresponding to the voiceprint features of the category according to the number of candidate characters corresponding to the voiceprint features of the category.

[0039] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0040] Memory for storing computer programs;

[0041] The processor is configured to implement any of the method steps described in the first aspect when executing a program stored in the memory.

[0042] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of the first aspects are implemented.

[0043] Embodiments of the present invention provide a method, device, and electronic device for determining the character corresponding to a line. The method comprises obtaining voiceprint features to be processed, wherein the audio voiceprint features to be processed are voiceprint features corresponding to lines in a target video; clustering the voiceprint features to determine the category to which each voiceprint feature belongs; determining the candidate characters corresponding to the voiceprint features of each category based on the picture information of the target video corresponding to the category; and for each category, determining the character corresponding to the lines corresponding to the voiceprint features of the category based on the number of candidate characters corresponding to the voiceprint features of the category. In this scheme, since the voiceprint features corresponding to the lines spoken by the same character in the target video are similar, the voiceprint features corresponding to the lines in the target video can be clustered to determine the category to which each voiceprint feature belongs. Since the characters that appear more frequently in the video screen information are usually the characters corresponding to the lines, the alternative characters corresponding to the voiceprint features of each category can be determined based on the screen information of the target video corresponding to each category, and then the characters corresponding to the lines corresponding to the voiceprint features of this category can be determined based on the number of alternative characters. There is no need to search and compare the voiceprint features based on the voiceprint feature library, which improves the accuracy of determining the characters corresponding to the lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.

[0045] Figure 1 A flowchart of a method for determining characters corresponding to lines provided by an embodiment of the present invention;

[0046] Figure 2 Based on Figure 1 Flowchart of the voiceprint feature extraction method of the illustrated embodiment;

[0047] Figure 3 A flow chart of a voiceprint feature extraction method provided by an embodiment of the present invention;

[0048] Figure 4 A flow chart of a method for determining the category to which a voiceprint feature belongs provided by an embodiment of the present invention;

[0049] Figure 5 A flow chart of a method for determining candidate characters provided by an embodiment of the present invention;

[0050] Figure 6 A flow chart of a cluster center comparison method provided by an embodiment of the present invention;

[0051] Figure 7 A flowchart of another cluster center comparison method provided by an embodiment of the present invention;

[0052] Figure 8 Flowchart of a method for calculating voiceprint feature similarity provided by an embodiment of the present invention;

[0053] Figure 9 A schematic flow chart of a method for determining the characters corresponding to lines provided in an embodiment of the present invention;

[0054] Figure 10 A schematic diagram of the structure of a device for determining the character corresponding to a line provided by an embodiment of the present invention;

[0055] Figure 11 This is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present invention will be described below with reference to the accompanying drawings in the embodiments of the present invention.

[0057] In order to improve the accuracy of determining the characters corresponding to lines, embodiments of the present invention provide a method, device, and electronic device for determining the characters corresponding to lines.

[0058] like Figure 1 As shown, a method for determining the character corresponding to a line includes:

[0059] S101, obtaining the voiceprint features to be processed;

[0060] The audio voiceprint features to be processed are voiceprint features corresponding to the lines in the target video;

[0061] S102, clustering the voiceprint features to determine the category to which each voiceprint feature belongs;

[0062] S103, based on the image information of the target video corresponding to each category, determining a candidate person corresponding to the voiceprint feature of the category;

[0063] S104 , for each category, determining the character corresponding to the lines corresponding to the voiceprint features of the category according to the number of candidate characters corresponding to the voiceprint features of the category.

[0064] As can be seen, in the solution provided by the embodiment of the present invention, the electronic device can obtain voiceprint features to be processed, wherein the audio voiceprint features to be processed are voiceprint features corresponding to lines in a target video; cluster the voiceprint features to determine the category to which each voiceprint feature belongs; determine the candidate characters corresponding to the voiceprint features of each category based on the screen information of the target video corresponding to each category; and determine, for each category, the character corresponding to the lines corresponding to the voiceprint features of that category based on the number of candidate characters corresponding to the voiceprint features of that category. In this solution, since the voiceprint features corresponding to the lines spoken by the same character in the target video are similar, the voiceprint features corresponding to the lines in the target video can be clustered to determine the category to which each voiceprint feature belongs. Since the characters that appear more frequently in the video screen information are usually the characters corresponding to the lines, the candidate characters corresponding to the voiceprint features of each category can be determined based on the screen information of the target video corresponding to each category. Furthermore, the character corresponding to the lines corresponding to the voiceprint features of that category can be determined based on the number of candidate characters, without the need to search and compare the voiceprint features based on the voiceprint feature library, thereby improving the accuracy of determining the characters corresponding to the lines.

[0065] The method for determining the characters corresponding to lines disclosed in the present invention is applied to the field of speech processing technology. Specifically, it can be applied to scenes such as video editing and video commentary, and is not specifically limited here.

[0066] Since the voiceprint feature can represent the voice of the character corresponding to the lines in the target video, in order to determine the character corresponding to the lines, the electronic device can obtain the voiceprint feature to be processed, where the voiceprint feature to be processed is the voiceprint feature corresponding to the lines in the target video.

[0067] Since the voiceprint features corresponding to the lines spoken by the same person in the target video are similar, the voiceprint features can be clustered to determine the category to which each voiceprint feature belongs. Clustering can be achieved based on a K-means clustering algorithm, a hierarchical clustering algorithm, or other methods, which are not specifically limited here.

[0068] For example, the target video contains five lines of dialogue: Line 1, Line 2, Line 3, Line 4, and Line 5. Line 1 and Line 3 are spoken by the same character, while Line 2, Line 4, and Line 5 are spoken by another character. If the voiceprint features corresponding to the lines are clustered, two categories can be obtained: the voiceprint features corresponding to Line 1 and Line 3 belong to one category, and the voiceprint features corresponding to Line 2, Line 4, and Line 5 belong to the other category.

[0069] Since the characters corresponding to the lines usually appear in the screen of the target video, the screen information of the target video corresponding to each category can be identified to determine the candidate characters corresponding to the voiceprint features.

[0070] For example, as shown in the table below, the target video includes lines 1-10. Voiceprint features 1-10 corresponding to lines 1-10 are clustered to obtain voiceprint feature categories 1-4. The target video corresponding to each voiceprint feature category is identified separately. Based on the image information of the target video, the candidate characters corresponding to the voiceprint features of category 1 are determined to be characters A and B, the candidate character corresponding to the voiceprint features of category 2 is determined to be character A, the candidate characters corresponding to the voiceprint features of category 3 are determined to be characters B and C, and the candidate character corresponding to the voiceprint features of category 4 is determined to be character D.

[0071]

[0072] At this point, each voiceprint feature has determined a corresponding candidate character. The electronic device can determine, for each category, the character corresponding to the lines corresponding to the voiceprint features of that category based on the number of candidate characters corresponding to the voiceprint features of that category.

[0073] Continuing with the above example, see the table above. For category 2, the number of candidate characters is one. Then the electronic device can determine that the character corresponding to lines 2 and 4 corresponding to voiceprint features 2 and 4 of category 2 is character A.

[0074] It can be seen that since the voiceprint features corresponding to the lines spoken by the same character in the target video are similar, the voiceprint features corresponding to the lines in the target video can be clustered to determine the category to which each voiceprint feature belongs. Since the characters that appear more frequently in the video screen information are usually the characters corresponding to the lines, the alternative characters corresponding to the voiceprint features of each category can be determined based on the screen information of the target video corresponding to each category, and then the characters corresponding to the lines corresponding to the voiceprint features of this category can be determined according to the number of alternative characters. There is no need to search and compare the voiceprint features based on the voiceprint feature library, which improves the accuracy of determining the characters corresponding to the lines.

[0075] like Figure 2As shown, as an implementation of an embodiment of the present invention, the step of obtaining the voiceprint features to be processed may include:

[0076] S201, dividing the audio corresponding to the target video into audio segments corresponding to each line according to the start time and end time of the line;

[0077] Since a line of dialogue is usually spoken by a character, in order to determine the character corresponding to the line, the electronic device can divide the audio corresponding to the target video according to the start time and end time of the line. For example, the electronic device can use OCR (character recognition) to determine the starting time point information of each line to divide the audio, and then obtain the audio segment corresponding to each line. The audio segment can be used for subsequent operations to determine the character corresponding to the line.

[0078] Before step S201, the electronic device can separate the audio file from the target video, extract the uncompressed WAV audio file, and divide the WAV audio file into speaker audio segments. The audio segments are mono audio segments with a sampling rate of 16k.

[0079] Continuing with the example of step S104, as shown in the table below, taking the voiceprint features included in category 1 as an example, the electronic device can determine the line time period based on OCR (text recognition), that is, the start time and end time of the line in the target video, where the time period of line 1 is 1 second to 4 seconds in the video, the time period of line 3 is 8 seconds to 10 seconds in the video, and the time period of line 5 is 13 seconds to 15 seconds in the video, and then divide the audio into corresponding segments for each line.

[0080] Lines Line time period Line 1 1 second to 4 seconds Line 3 8-10 seconds Line 5 13 seconds - 15 seconds

[0081] S202: Extract the voiceprint features of each audio clip to obtain the voiceprint features to be processed.

[0082] Since the voiceprint feature can represent the voice of the person in the audio clip, the electronic device can extract the voiceprint feature of each audio clip, and then obtain the voiceprint feature to be processed, that is, the voiceprint feature corresponding to the lines in the target video.

[0083] It should be noted that when obtaining the voiceprint features to be processed, it is not necessary to obtain the voiceprint features to be processed corresponding to all the lines in the target video. It is sufficient to obtain only a few lines or lines corresponding to video clips.

[0084] Continuing with the example of step S201, as shown in the following table, voiceprint features are extracted for lines 1, 3, and 5 obtained in step S201, respectively, to obtain voiceprint features 1, 3, and 5 of lines 1, 3, and 5, respectively.

[0085] Voiceprint characteristics Lines Line time period Voiceprint feature 1 Line 1 1 second to 4 seconds Voiceprint feature 3 Line 3 8-10 seconds Voiceprint feature 5 Line 5 13 seconds - 15 seconds

[0086] For example, models that can be used to extract voiceprint features of audio clips include the Ecapa-tdnn model and the WavLM model. In voiceprint feature extraction, one model can be used for extraction, or multiple models can be used for extraction and then normalized, spliced, and PCA (principal component analysis) dimensionality reduction can be performed. Any voiceprint extraction method that can obtain voiceprint features is acceptable and is not specifically limited here.

[0087] like Figure 3 As shown, Figure 3 The figure is a flow chart of a voiceprint feature extraction method, which includes the following steps:

[0088] S301, Ecapa-tdnn model;

[0089] Electronic devices can extract voiceprint features of audio clips based on the Ecapa-tdnn model;

[0090] S302, WavLM model;

[0091] Electronic devices can extract voiceprint features of audio clips based on the WavLM model;

[0092] S303, normalization;

[0093] Normalizing the voiceprint features extracted in step S301 so that the voiceprint features obtained in step S301 and step S302 can be spliced;

[0094] S304, normalization;

[0095] Normalizing the voiceprint features extracted in step S302 so that the voiceprint features obtained in step S301 and step S302 can be spliced;

[0096] S305: concatenate and perform PCA dimensionality reduction.

[0097] The electronic device may concatenate the normalized voiceprint features obtained in step S303 and step 304, and perform PCA dimensionality reduction to obtain the voiceprint features corresponding to the audio clip.

[0098] As can be seen, in this embodiment of the present invention, the electronic device can divide the audio corresponding to the target video into audio segments corresponding to each line of dialogue according to the start and end times of the lines; extract the voiceprint features of each audio segment to obtain the voiceprint features to be processed. Because voiceprint features can be used to represent the voice of the person corresponding to the audio segment, the voiceprint feature extraction method provided in this embodiment of the present invention can accurately extract the voiceprint features corresponding to the audio segment, thereby improving the accuracy of determining the person corresponding to the line of dialogue.

[0099] As an implementation method of the embodiment of the present invention, Figure 4 As shown, the step of clustering the voiceprint features and determining the category to which each voiceprint feature belongs may include:

[0100] S401, determining a time period corresponding to each scene included in the target video based on scene information of the target video;

[0101] Since the emotions of the characters and the background sounds of the scenes are usually similar under the same target video scene information, the emotions of the characters and the sounds of the scenes are relatively similar between the voiceprint features obtained under the same target video scene information. By clustering the voiceprint features obtained under the same target video scene information, we can approximately ignore the changes in the emotions of the characters and the influence of the background sounds of the scenes on the voiceprints. By clustering the voiceprint features corresponding to the videos under the same scene, we can obtain the voiceprint feature category, and then determine the character corresponding to the lines of the voiceprint features of this category according to the number of alternative characters, thereby improving the accuracy of determining the character corresponding to the lines.

[0102] In this way, the character corresponding to the lines can be determined quickly and effectively without having to first extract the character's voiceprint features and then compare the voiceprints one by one according to the speaker's voiceprint feature retrieval library.

[0103] In order to divide the target video into different scenes, the electronic device can use transition point detection technology based on the scene information of the target video to determine the time period corresponding to each scene included in the target video. In this way, the scene information can be represented by the time period.

[0104] For example, the target video is thirty seconds long, the first fifteen seconds of the target video are outdoors, called scene A, and the remaining fifteen seconds are indoors, called scene B. Then the electronic device can determine that the first fifteen seconds are one scene of the target video and the remaining fifteen seconds are another scene of the target video based on the outdoor scene information such as lawns and trees and the indoor scene information such as wallpaper and furniture in the target video.

[0105] S402: Cluster the voiceprint features corresponding to each time period to determine the category to which the voiceprint features corresponding to each time period belong.

[0106] Because the emotions and background sounds of characters in the same scene in the target video are similar, the voiceprint features in the same scene can be clustered. Since the different scenes in the target video are divided into different time periods, the voiceprint features corresponding to the time periods corresponding to different scenes can be clustered. This way, the clustering process is less affected by changes in character emotions and background sounds.

[0107] Continuing with the above example, the voiceprint features corresponding to the first fifteen seconds of the target video are voiceprint features 1-5, which correspond to lines 1-5 respectively. The voiceprint features corresponding to the last fifteen seconds are voiceprint features 6-10, which correspond to lines 6-10 respectively. The voiceprint features corresponding to the first fifteen seconds and the last fifteen seconds are clustered separately, and the results are shown in the following table:

[0108]

[0109] The voiceprint features corresponding to scene A are clustered into categories 1 and 2, where category 1 includes voiceprint features 1, 3, and 5, and category 2 includes voiceprint features 2 and 4. The voiceprint features corresponding to scene B are clustered into categories 3 and 4, where category 3 includes voiceprint features 6, 7, and 8, and category 4 includes voiceprint features 9 and 10. The clustering results shown in the table above show that since the voiceprint features in scenes A and B are clustered into two categories, the lines in scenes A and B can each correspond to two characters.

[0110] As can be seen, in this embodiment of the present invention, the electronic device can determine the time period corresponding to each scene in the target video based on the scene information of the target video; cluster the voiceprint features corresponding to each time period, and determine the category to which the voiceprint features corresponding to each time period belong. Because the emotions and background sounds of characters in the same scene are similar, the time period corresponding to each scene in the target video can be determined, and the voiceprint features corresponding to each time period can be clustered. This improves the accuracy of identifying the characters corresponding to the lines, while reducing the resource consumption of a single analysis.

[0111] like Figure 5 As shown, as an implementation of an embodiment of the present invention, the step of determining the candidate person corresponding to the voiceprint feature of each category based on the picture information of the target video corresponding to each category may include:

[0112] S501, for each category, obtaining video frames of the target video within a time period corresponding to the voiceprint features included in the category;

[0113] Since characters usually appear on screen when speaking in a video, for each category, we can obtain the video frames of the target video in the time period corresponding to the voiceprint features of that category, so as to facilitate the subsequent calculation of the proportion of each character appearing in the time period corresponding to the lines.

[0114] Continuing with the example of step S201, as shown in the table below, taking the voiceprint features included in category 1 as an example, the speech time period is determined in step S201. The speech time period corresponding to voiceprint features 1, 3, and 5 included in category 1 totals 7 seconds. Assuming the target video has 24 frames per second, the electronic device can obtain a total of 168 frames of speech time period corresponding to voiceprint features 1, 3, and 5.

[0115]

[0116] S502, performing face recognition on the video frames to determine the proportion of the appearance time of each character in the video frames included in the time period;

[0117] Since the characters corresponding to the lines usually appear in the video screen, the proportion of the appearance time of each character in the video frame of the target video within the time period corresponding to the voiceprint features of the same category can be calculated, and the character corresponding to the voiceprint features can be determined based on the appearance ratio.

[0118] Continuing with the example of step S501, as shown in the table below, face recognition is performed frame by frame on the target video frames within the time period corresponding to the voiceprint features of this category, and the proportion of each character's appearance is determined. There are a total of 168 frames, and character A appears in 117 frames, so the appearance ratio of character A is 117 / 168. Character B and character C appear in 121 frames and 21 frames respectively. Similarly, the appearance ratios of character B and character C are 121 / 168 and 21 / 168 respectively. Correspondingly, in the video frames included in this time period, the appearance duration ratios of character A, character B, and character C are 117 / 168, 121 / 168, and 21 / 168 respectively.

[0119]

[0120] S503: Determine the person whose corresponding ratio reaches a preset ratio as the candidate person corresponding to the voiceprint feature of this category.

[0121] Since the character's appearance time ratio is high, it is highly likely that the character is the character corresponding to the voiceprint feature. Therefore, a ratio can be set in advance. If the task appearance ratio obtained in step S402 reaches the preset ratio, the character can be determined as the candidate character corresponding to the voiceprint feature of this category.

[0122] Continuing with the example of step S502, as shown in the table below, the preset ratio is 0.65. Since the appearance time ratio of character A and character B reaches 0.65, character A and character B are determined as candidate characters corresponding to the voiceprint features of this category.

[0123]

[0124]

[0125] It can be seen that in the embodiment of the present invention, the electronic device can, for each of the categories, obtain the video frames of the target video within the time period corresponding to the voiceprint features included in the category; perform face recognition on the video frames to determine the proportion of the appearance time of each character in the video frames included in the time period; and determine the characters whose corresponding proportion reaches a preset proportion as the candidate characters corresponding to the voiceprint features of the category. Since the characters corresponding to the lines in the target video usually appear in the video screen, face recognition can be performed on the video frames of the target video within the time period corresponding to the voiceprint features included in the same category, and the characters whose appearance proportion reaches a preset proportion can be determined as the candidate characters corresponding to the voiceprint features. Based on the above method, the proportion of each character appearing in the video frames of the target video within the time period corresponding to the voiceprint features can be accurately obtained, thereby improving the accuracy of the characters corresponding to the lines.

[0126] As an implementation of an embodiment of the present invention, the step of determining the character corresponding to the lines corresponding to the voiceprint features of the category based on the number of candidate characters corresponding to the voiceprint features of the category may include:

[0127] If the number of candidate characters corresponding to the voiceprint feature of the category is one, determine that the candidate character is the character corresponding to the lines corresponding to the voiceprint feature of the category;

[0128] If there are multiple candidate characters corresponding to the voiceprint features of this category, the character corresponding to the lines corresponding to the voiceprint features of this category is determined based on the similarity between the cluster center of this category and the cluster center of the first target category. The first target category is a category with the same scene as the corresponding category and has only one candidate character.

[0129] Since the alternative characters are characters whose appearance ratio in the video frames of the target video reaches a preset ratio within the time period corresponding to the voiceprint features of this category, and the characters corresponding to the lines usually appear in the video screen with a higher frequency, if the number of alternative characters corresponding to the voiceprint features of this category is one, then the electronic device can determine that the alternative character is the character corresponding to the lines corresponding to the voiceprint features of this category.

[0130] Continuing with the example in step S104, as shown in the table below, since the number of candidate characters corresponding to the voiceprint features of category 2 and category 4 is one, it can be determined that character A is the character corresponding to the lines corresponding to the voiceprint features of category 2, and character D is the character corresponding to the lines corresponding to the voiceprint features of category 4.

[0131]

[0132]

[0133] Since each clustered voiceprint feature has a cluster center, the cluster center represents the characteristics of the voiceprint features within that category. Therefore, the similarity between the cluster centers of two clusters derived from clustering the voiceprint features of the same person is high. Because emotions and noise levels are similar in the same scene, comparing cluster centers of two categories in the same scene can lead to more accurate results.

[0134] When there are multiple candidate characters corresponding to the voiceprint features of this category, that is, there are multiple characters whose appearance ratio reaches a preset ratio in the video frames of the target video within the time period corresponding to the voiceprint features included in this category, the similarity between the cluster center of this category and the cluster center of the first target category can be compared, wherein the first target category is the same as the scene corresponding to this category, and the number of candidate characters is one. Since the number of candidate characters of the first target category is one, the cluster center of the first target category has a greater reference value.

[0135] Continuing with the previous example, as shown in the table below, the number of candidate characters corresponding to the voiceprint features of category 1 and category 3 is multiple. Since category 2 corresponds to the same scene as category 1 and the number of corresponding candidate characters is one, category 2 can be used as the first target category for category 1. Similarly, category 4 can be used as the first target category for category 3. The electronic device can determine the character corresponding to the lines corresponding to the voiceprint features of category 1 based on the similarity between the cluster centers of category 1 and category 2. Similarly, the electronic device can also determine the character corresponding to the lines corresponding to the voiceprint features of category 3 based on the similarity between the cluster centers of category 3 and category 4.

[0136]

[0137] It can be seen that in the embodiment of the present invention, if the number of alternative characters corresponding to the voiceprint features of this category is one, the alternative character is determined to be the character corresponding to the lines corresponding to the voiceprint features of this category; if the number of alternative characters corresponding to the voiceprint features of this category is multiple, based on the similarity between the cluster center of this category and the cluster center of the first target category, the character corresponding to the lines corresponding to the voiceprint features of this category is determined, wherein the first target category is the same as the scene corresponding to this category, and the number of corresponding alternative characters is one. Since the characters corresponding to the lines usually appear frequently in the video, if the number of candidate characters corresponding to the voiceprint features of this category is one, then it can be determined that the candidate character is the character corresponding to the lines corresponding to the voiceprint features of this category; since the cluster center represents the category obtained by clustering the voiceprint features, and comparing the cluster centers in the same scene can reduce the interference caused by background noise and character emotions, for categories with multiple candidate characters corresponding to the voiceprint features, the cluster center similarity can be compared with the first target category, and then the character corresponding to the lines corresponding to the voiceprint features of this category can be determined. The character corresponding to the lines can be determined more quickly based on the first target category, thereby improving the calculation speed of determining the character corresponding to the lines.

[0138] like Figure 6 As shown, as an implementation of an embodiment of the present invention, the step of determining the character corresponding to the lines corresponding to the voiceprint features of the category based on the similarity between the cluster center of the category and the cluster center of the first target category includes:

[0139] S601, respectively calculating the similarity between the cluster center of the category and the cluster center of each first target category;

[0140] Since the similarity between cluster centers can represent the degree of similarity between the voiceprint features of categories, if the similarity between cluster centers is high, it can be determined that the candidate person of the first target category is the person corresponding to the voiceprint feature of that category. Therefore, the electronic device can calculate the similarity between the cluster center of that category and the cluster center of each first target category.

[0141] For example, the voiceprint features corresponding to the time period in the video corresponding to scene C are clustered into categories 5-9. The candidate characters corresponding to categories 5-9 are person A and person B, person C, person A, person D, and person B. For category 5, since the number of candidate characters corresponding to categories 6-9 is only one and the scene is the same as that corresponding to category 5, categories 6-9 are the first target categories, and category 5 can be compared with the similarity of the cluster centers of categories 6-9 respectively.

[0142]

[0143] S602: Determine the candidate character corresponding to the first target category whose similarity reaches a first preset similarity as the character corresponding to the line corresponding to the voiceprint feature of the category.

[0144] Since the high similarity of the cluster centers indicates that the voiceprint features between the two are relatively close, the category can determine the candidate character corresponding to the first target category whose corresponding similarity reaches the first preset similarity as the character corresponding to the lines corresponding to the voiceprint features of the category.

[0145] Continuing with the example in step S601, if the first preset similarity is 0.8, and the similarities between the cluster center of category 5 and the cluster centers of categories 6, 7, 8, and 9 are 0.1, 0.2, 0.1, and 0.9, respectively, since the similarity between the cluster centers of category 5 and category 9 reaches the first preset similarity, it can be determined that character B corresponding to category 9 is the character corresponding to the lines corresponding to the voiceprint features of category 5.

[0146] It can be seen that in the embodiment of the present invention, the electronic device respectively calculates the similarity between the cluster center of the category and the cluster center of each first target category; the alternative character corresponding to the first target category whose corresponding similarity reaches the first preset similarity is determined as the character corresponding to the lines corresponding to the voiceprint characteristics of the category; since the high similarity of the cluster centers can indicate that the voiceprint characteristics corresponding to the cluster centers are relatively close, if the similarity between the category and the cluster center of the first target category reaches the first preset similarity, then the alternative character corresponding to the first target category whose corresponding similarity reaches the first preset similarity can be determined as the character corresponding to the lines corresponding to the voiceprint characteristics of the category; by comparing the similarity between the category with multiple alternative characters and the cluster center of the first target category, the character corresponding to the category with multiple alternative characters can be quickly determined, thereby improving the calculation efficiency of the character corresponding to the lines.

[0147] like Figure 7 As an implementation of an embodiment of the present invention, the above method may further include:

[0148] S701: If the similarity between the cluster center of the category and the cluster center of each first target category does not reach the first preset similarity, respectively calculate the similarity between the cluster center of the category and the cluster center of each second target category;

[0149] The second target category is a category that has a different scene from the first category and has only one corresponding candidate character.

[0150] Since the similarity between the cluster center of the category and the cluster center of each first target category is less than the first preset similarity, and the cluster center of the second target category can be compared with the similarity of the cluster center of the category, if the similarity between the cluster center of the category and the cluster center of each first target category does not reach the first preset similarity, the similarity between the cluster center of the category and the cluster center of each second target category is calculated separately.

[0151] For example, as shown in the table below, Category 4 has only one candidate character, and Category 4 and Category 3 have the same corresponding scenes, but the similarity between the cluster centers of Category 3 and Category 4 is less than the first preset similarity. Thus, the similarity between Category 3 and the cluster centers of each first target category has not reached the first preset similarity. Category 2 has only one candidate character, and Category 2 and Category 3 have different corresponding scenes, so the similarity between the cluster centers of Category 3 and Category 3 can be calculated.

[0152]

[0153] S702: Determine the candidate character corresponding to the second target category whose similarity reaches a second preset similarity as the character corresponding to the line corresponding to the voiceprint feature of the category.

[0154] Since the second target category and the scene corresponding to the category are different, the background sounds and character emotions of the two scenes may be different, so the second preset similarity may be smaller than the first preset similarity.

[0155] If the similarity between the cluster center of the category and the second target category reaches a second preset similarity, then it can be determined that the candidate character corresponding to the second target category with the corresponding similarity reaching the second preset similarity is the character corresponding to the lines corresponding to the voiceprint features of the category.

[0156] It can be seen that in the embodiment of the present invention, if the similarity between the cluster center of the category and the cluster center of each first target category does not reach the first preset similarity, the similarity between the cluster center of the category and the cluster center of each second target category is calculated respectively, wherein the second target category is a category that is different from the scene corresponding to the category and has one corresponding candidate character; the candidate character corresponding to the second target category whose corresponding similarity reaches the second preset similarity is determined as the character corresponding to the lines corresponding to the voiceprint feature of the category; since the similarity between the cluster center of the first target category and the cluster center of the category does not reach the first preset similarity, the similarity between the cluster centers of the category and the second target category can be calculated; if the similarity between the cluster center of the category and the second target category reaches the second preset similarity, the candidate character corresponding to the second target category whose corresponding similarity reaches the second preset similarity can be determined as the character corresponding to the lines corresponding to the voiceprint feature of the category; in this way, the success rate of corresponding lines to characters can be improved.

[0157] like Figure 8 As shown, as an implementation of an embodiment of the present invention, the method may further include:

[0158] S801: If the similarity between the cluster center of the category and the cluster center of each second target category does not reach the second preset similarity, for each voiceprint feature of the category, calculate the similarity between the voiceprint feature and other voiceprint features in the category;

[0159] Since the similarity between the cluster center of this category and the cluster center of each second target category does not reach the second preset similarity, the electronic device can calculate the similarity between the voiceprint feature and other voiceprint features in this category to verify whether the voiceprint feature is classified into the wrong category in the clustering operation.

[0160] S802: If the proportion of similarities reaching a preset threshold is less than a preset proportion, obtain each voiceprint feature to be compared within a preset duration of the video time point corresponding to the voiceprint feature;

[0161] Since the proportion of similarities between the voiceprint feature and other voiceprint features in the category that reach the preset threshold is less than the preset proportion, that is, the proportion of similarities between the voiceprint feature and other voiceprint features in the same category that reach the second preset threshold is relatively low, the voiceprint feature may be misclassified into the category. The electronic device can obtain each voiceprint feature to be compared within a preset duration including the video time point corresponding to the voiceprint feature.

[0162] For example, if the preset duration is 45 seconds, if the similarity between voiceprint feature A and other voiceprint features of the same category is calculated, and the proportion of similarities reaching the threshold is less than the preset proportion, then the electronic device can obtain other voiceprint features to be compared in the 45-second time period before and after the time point of the lines corresponding to voiceprint feature A in the target video.

[0163] S803, respectively calculating the similarity between the voiceprint feature and each voiceprint feature to be compared;

[0164] The similarity can represent the degree of similarity between the voiceprint feature and the voiceprint feature to be compared. In order to determine whether the voiceprint feature is clustered incorrectly, the similarity between the voiceprint feature and each voiceprint feature to be compared can be calculated separately.

[0165] Following the example in step S802, the electronic device may respectively calculate the similarity between the voiceprint feature A and other to-be-compared voiceprint features in a time period of 45 seconds before and after the time point of the speech corresponding to the voiceprint feature A in the target video.

[0166] S804: Determine the character corresponding to the voiceprint feature to be compared with the one with the highest similarity as the character corresponding to the line corresponding to the voiceprint feature.

[0167] Since the original clustering of the voiceprint feature was wrong, when calculating the similarity between the voiceprint feature and each voiceprint feature to be compared, the character corresponding to the voiceprint feature to be compared with the highest similarity can be determined as the character corresponding to the lines corresponding to the voiceprint feature, and the voiceprint feature is added to the category to which the voiceprint feature to be compared with the highest similarity belongs, so that the voiceprint feature can be used subsequently.

[0168] It can be seen that in the embodiment of the present invention, if the similarity between the cluster center of the category and the cluster center of each second target category does not reach the second preset similarity, for each voiceprint feature of the category, the similarity between the voiceprint feature and other voiceprint features in the category is calculated; if the proportion of the similarity reaching the preset threshold is less than the preset proportion, each voiceprint feature to be compared within the preset duration of the video time point corresponding to the voiceprint feature is obtained; the similarity between the voiceprint feature and each voiceprint feature to be compared is calculated respectively; the character corresponding to the voiceprint feature to be compared with the highest similarity is determined as the character corresponding to the lines corresponding to the voiceprint feature; since the category of the voiceprint feature is wrongly classified in the clustering process, the similarity between the voiceprint feature and each voiceprint feature to be compared within the preset duration of the video time point corresponding to the voiceprint feature can be calculated, and the character corresponding to the voiceprint feature to be compared with the highest similarity is determined as the character corresponding to the lines corresponding to the voiceprint feature; the voiceprint features with clustering errors can be corrected, and the success rate of matching lines to characters can be further improved.

[0169] like Figure 9 As shown, Figure 9 A flow chart of a method for assigning lines to characters provided by an embodiment of the present invention:

[0170] S901, speaker voiceprint feature extraction, Ecapa-tdnn model, WavLM model;

[0171] The electronic device can extract the speaker's voiceprint features from audio clip 1, audio clip 2...audio clip N based on the Ecapa-tdnn model and the WavLM model, and obtain voiceprint feature 1, voiceprint feature 2...voiceprint feature N that correspond one to one with the audio clips.

[0172] S902, segmentation of transition points and clustering by scene;

[0173] The electronic device can divide the target video into time periods according to the transition points of the target video, cluster the voiceprint features within the time periods corresponding to the same scene, divide the scenes into scene a, scene b...scene m, and cluster the voiceprint features corresponding to the time periods corresponding to each scene to obtain several clusters under each scene, that is, classification.

[0174] S903, clustering the video clips in the cluster, performing face recognition, and calculating the percentage of related characters;

[0175] The electronic device can perform face recognition on the video clips corresponding to the voiceprint features included in each cluster. Based on the statistical proportion of related roles, that is, candidate characters, the association between clusters and characters in each scene, that is, the correspondence between classifications and candidate characters, can be obtained.

[0176] S904, cross-scene: cluster-character association role purification;

[0177] For this scenario, it is impossible to obtain the category of the character corresponding to the lines based on the similarity comparison of the cluster center with the category with one alternative character. The similarity of the cluster center can be compared across scenes with the category with one alternative character in other scenes, and then the alternative characters can be purified, that is, alternative characters with lower similarity can be excluded.

[0178] S905: Verify and compare the speaker of each line and purify any abrupt line-speaker associations.

[0179] For lines that still have no corresponding characters, their corresponding voiceprint features can be compared with other voiceprint features of the same category for similarity. If the similarity threshold is less than the preset ratio, it means that the line has been misclassified into this category. Therefore, the similarity can be calculated with the voiceprint features corresponding to the line sequence 1-N in the preset time period before and after the corresponding time point in the video. The character corresponding to the line corresponding to the voiceprint feature with the greatest similarity is taken as the corresponding character.

[0180] Corresponding to the above-mentioned method for determining the characters corresponding to lines, an embodiment of the present invention further provides a device for determining the characters corresponding to lines. The following introduces a device for determining the characters corresponding to lines provided by an embodiment of the present invention.

[0181] like Figure 10 As shown, a device for determining the character corresponding to a line includes:

[0182] A first voiceprint feature acquisition module 1001 is configured to acquire a voiceprint feature to be processed, wherein the voiceprint feature to be processed is a voiceprint feature corresponding to a line in a target video;

[0183] The voiceprint feature clustering module 1002 is used to cluster the voiceprint features and determine the category to which each voiceprint feature belongs;

[0184] The candidate person determination module 1003 is configured to determine the candidate person corresponding to the voiceprint feature of each category based on the image information of the target video corresponding to the category;

[0185] The first character determination module 1004 is configured to determine, for each category, the character corresponding to the lines corresponding to the voiceprint features of the category according to the number of candidate characters corresponding to the voiceprint features of the category.

[0186] As can be seen, in the solution provided by the embodiment of the present invention, the electronic device can obtain voiceprint features to be processed, wherein the audio voiceprint features to be processed are voiceprint features corresponding to lines in a target video; cluster the voiceprint features to determine the category to which each voiceprint feature belongs; determine the candidate characters corresponding to the voiceprint features of each category based on the screen information of the target video corresponding to each category; and determine, for each category, the character corresponding to the lines corresponding to the voiceprint features of that category based on the number of candidate characters corresponding to the voiceprint features of that category. In this solution, since the voiceprint features corresponding to the lines spoken by the same character in the target video are similar, the voiceprint features corresponding to the lines in the target video can be clustered to determine the category to which each voiceprint feature belongs. Since the characters that appear more frequently in the video screen information are usually the characters corresponding to the lines, the candidate characters corresponding to the voiceprint features of each category can be determined based on the screen information of the target video corresponding to each category. Furthermore, the character corresponding to the lines corresponding to the voiceprint features of that category can be determined based on the number of candidate characters, without the need to search and compare the voiceprint features based on the voiceprint feature library, thereby improving the accuracy of determining the characters corresponding to the lines.

[0187] As an implementation of an embodiment of the present invention, the first voiceprint feature acquisition module 1001 may include:

[0188] An audio segmentation unit, configured to segment the audio corresponding to the target video into audio segments corresponding to each line according to the start time and end time of the lines;

[0189] The voiceprint feature extraction unit is used to extract the voiceprint features of each audio clip to obtain the voiceprint features to be processed.

[0190] As an implementation of an embodiment of the present invention, the voiceprint feature clustering module 1002 may include:

[0191] a time period determining unit, configured to determine a time period corresponding to each scene included in the target video based on scene information of the target video;

[0192] The voiceprint feature clustering unit is used to cluster the voiceprint features corresponding to each time period and determine the category to which the voiceprint features corresponding to each time period belong.

[0193] As an implementation of an embodiment of the present invention, the candidate character determination module 1003 may include:

[0194] a video frame acquisition unit, configured to acquire, for each category, a video frame of the target video within a time period corresponding to the voiceprint features included in the category;

[0195] a character appearance duration ratio determination unit, configured to perform face recognition on the video frames and determine the ratio of the appearance duration of each character in the video frames included in the time period;

[0196] The candidate person determination unit is used to determine the person whose corresponding ratio reaches a preset ratio as the candidate person corresponding to the voiceprint feature of this category.

[0197] As an implementation manner of an embodiment of the present invention, the first person determination module 1004 may include:

[0198] A first character determination unit is configured to, if the number of candidate characters corresponding to the voiceprint feature of the category is one, determine that the candidate character is the character corresponding to the lines corresponding to the voiceprint feature of the category;

[0199] The second character determination unit is used to determine the character corresponding to the lines corresponding to the voiceprint features of this category based on the similarity between the cluster center of this category and the cluster center of the first target category if there are multiple alternative characters corresponding to the voiceprint features of this category, wherein the first target category is the same as the scene corresponding to this category and the number of corresponding alternative characters is one.

[0200] As an implementation manner of this embodiment of the present invention, the second person determination unit may include:

[0201] a similarity calculation subunit, configured to respectively calculate the similarity between the cluster center of the category and the cluster center of each first target category;

[0202] The character determination subunit is configured to determine the candidate character corresponding to the first target category whose corresponding similarity reaches a first preset similarity as the character corresponding to the line corresponding to the voiceprint feature of the category.

[0203] As an implementation manner of the embodiment of the present invention, the device may further include:

[0204] a first similarity calculation module configured to, if the similarity between the cluster center of the category and the cluster center of each first target category does not reach the first preset similarity, respectively calculate the similarity between the cluster center of the category and the cluster center of each second target category, wherein the second target category is a category having a different scene from that of the category and corresponding to one candidate character;

[0205] The second character determination module is configured to determine the candidate characters corresponding to the second target category whose corresponding similarity reaches a second preset similarity as the characters corresponding to the lines corresponding to the voiceprint features of the category.

[0206] As an implementation manner of the embodiment of the present invention, the device may further include:

[0207] a second similarity calculation module configured to calculate, for each voiceprint feature of the category, a similarity between the voiceprint feature and other voiceprint features in the category if the similarity between the cluster center of the category and the cluster center of each second target category does not reach the second preset similarity;

[0208] The second voiceprint feature acquisition module is configured to acquire, if the ratio of similarities reaching a preset threshold is less than a preset ratio, each voiceprint feature to be compared within a preset duration including the video time point corresponding to the voiceprint feature;

[0209] A third similarity calculation module is used to calculate the similarity between the voiceprint feature and each voiceprint feature to be compared;

[0210] The third character determination module is used to determine the character corresponding to the voiceprint feature to be compared with the highest similarity as the character corresponding to the line corresponding to the voiceprint feature.

[0211] The embodiment of the present invention further provides an electronic device, such as Figure 11 As shown, it includes a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104.

[0212] Memory 1103, used for storing computer programs;

[0213] The processor 1101 is configured to implement the steps of the method for determining the character corresponding to the lines described in any of the above embodiments when executing the program stored in the memory 1103 .

[0214] As can be seen, in the solution provided by the embodiment of the present invention, the electronic device can obtain voiceprint features to be processed, wherein the audio voiceprint features to be processed are voiceprint features corresponding to lines in a target video; cluster the voiceprint features to determine the category to which each voiceprint feature belongs; determine the candidate characters corresponding to the voiceprint features of each category based on the screen information of the target video corresponding to each category; and determine, for each category, the character corresponding to the lines corresponding to the voiceprint features of that category based on the number of candidate characters corresponding to the voiceprint features of that category. In this solution, since the voiceprint features corresponding to the lines spoken by the same character in the target video are similar, the voiceprint features corresponding to the lines in the target video can be clustered to determine the category to which each voiceprint feature belongs. Since the characters that appear more frequently in the video screen information are usually the characters corresponding to the lines, the candidate characters corresponding to the voiceprint features of each category can be determined based on the screen information of the target video corresponding to each category. Furthermore, the character corresponding to the lines corresponding to the voiceprint features of that category can be determined based on the number of candidate characters, without the need to search and compare the voiceprint features based on the voiceprint feature library, thereby improving the accuracy of determining the characters corresponding to the lines.

[0215] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0216] The communication interface is used for communication between the above terminal and other devices.

[0217] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0218] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0219] In another embodiment provided by the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for determining the character corresponding to the lines described in any of the above embodiments is implemented.

[0220] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is executed, the computer executes the method for determining the character corresponding to the lines described in any one of the above embodiments.

[0221] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0222] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0223] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0224] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A method for determining the character corresponding to a line, characterized in that: The method comprises: Acquire a voiceprint feature to be processed, wherein the voiceprint feature to be processed is a voiceprint feature corresponding to a line in a target video; Clustering the voiceprint features to determine the category to which each voiceprint feature belongs; Based on the image information of the target video corresponding to each category, determining the candidate person corresponding to the voiceprint feature of the category; For each category, the character corresponding to the lines corresponding to the voiceprint features of the category is determined based on the number of candidate characters corresponding to the voiceprint features of the category.

2. The method according to claim 1, characterized in that The step of obtaining the voiceprint features to be processed includes: Divide the audio corresponding to the target video into audio segments corresponding to each line according to the start time and end time of the line; The voiceprint features of each audio clip are extracted to obtain the voiceprint features to be processed.

3. The method according to claim 1, characterized in that The step of clustering the voiceprint features and determining the category to which each voiceprint feature belongs includes: Determining, based on the scene information of the target video, a time period corresponding to each scene included in the target video; The voiceprint features corresponding to each time period are clustered to determine the category to which the voiceprint features corresponding to each time period belong.

4. The method according to claim 1, wherein The step of determining the candidate person corresponding to the voiceprint feature of each category based on the picture information of the target video corresponding to each category includes: For each category, obtaining video frames of the target video within a time period corresponding to the voiceprint features included in the category; Performing face recognition on the video frames to determine the proportion of the appearance time of each character in the video frames included in the time period; The person whose corresponding ratio reaches the preset ratio is determined as the candidate person corresponding to the voiceprint feature of this category.

5. The method according to any one of claims 1 to 4, characterized in that The step of determining the character corresponding to the lines corresponding to the voiceprint features of the category according to the number of candidate characters corresponding to the voiceprint features of the category includes: If the number of candidate characters corresponding to the voiceprint feature of the category is one, determine that the candidate character is the character corresponding to the lines corresponding to the voiceprint feature of the category; If there are multiple alternative characters corresponding to the voiceprint features of this category, the character corresponding to the lines corresponding to the voiceprint features of this category is determined based on the similarity between the cluster center of this category and the cluster center of the first target category, wherein the first target category is the same as the scene corresponding to this category, and the number of corresponding alternative characters is one.

6. The method according to claim 5, characterized in that The step of determining the character corresponding to the lines corresponding to the voiceprint features of the category based on the similarity between the cluster center of the category and the cluster center of the first target category includes: Calculate the similarity between the cluster center of the category and the cluster center of each first target category respectively; The candidate characters corresponding to the first target category whose corresponding similarity reaches the first preset similarity are determined as the characters corresponding to the lines corresponding to the voiceprint features of the category.

7. The method according to claim 6, characterized in that The method further comprises: If the similarity between the cluster center of the category and the cluster center of each first target category does not reach the first preset similarity, respectively calculate the similarity between the cluster center of the category and the cluster center of each second target category, wherein the second target category is a category that has a different scene from the category and has only one candidate character. The candidate characters corresponding to the second target category whose corresponding similarity reaches the second preset similarity are determined as the characters corresponding to the lines corresponding to the voiceprint features of the category.

8. The method according to claim 7, characterized in that The method further comprises: If the similarity between the cluster center of the category and the cluster center of each second target category does not reach the second preset similarity, for each voiceprint feature of the category, calculate the similarity between the voiceprint feature and other voiceprint features in the category; If the proportion of similarities reaching the preset threshold is less than the preset proportion, obtaining each voiceprint feature to be compared within a preset time period including the video time point corresponding to the voiceprint feature; Calculate the similarity between the voiceprint feature and each voiceprint feature to be compared respectively; The character corresponding to the voiceprint feature to be compared with the highest similarity is determined as the character corresponding to the line corresponding to the voiceprint feature.

9. A device for determining the character corresponding to a line, characterized in that: The device comprises: A first voiceprint feature acquisition module is configured to acquire a voiceprint feature to be processed, wherein the voiceprint feature to be processed is a voiceprint feature corresponding to a line in a target video; A voiceprint feature clustering module, configured to cluster the voiceprint features and determine the category to which each voiceprint feature belongs; A candidate person determination module is used to determine the candidate person corresponding to the voiceprint feature of each category based on the image information of the target video corresponding to each category; The first character determination module is used to determine, for each category, the character corresponding to the lines corresponding to the voiceprint features of the category according to the number of candidate characters corresponding to the voiceprint features of the category.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 8 when executing a program stored in a memory.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 8 are implemented.

Images