Role audio matching method and device, electronic equipment and storage medium

Abstract

The invention provides a role audio matching method and device, electronic equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: obtaining the audio-visual work data of a to-be-processed audio-visual work, and according to the target role audio information of each role of the to-be-processed audio-visual work in the audio-visual work data, matching the target role audio information with the to-be-processed audio-visual work; obtaining a first voiceprint representation of each role; performing single-role audio extraction on the audio-visual work data to obtain multiple pieces of single-role audio information; based on the second voiceprint representation corresponding to each piece of single-role audio information, performing audio clustering on the multiple pieces of single-role audio information to obtain at least one single-role audio information set; determining a single-role audio information set matched with each role based on the target role audio information of each role and voiceprint representation similarity information of each single-role audio information set; wherein the voiceprint representation similarity information is determined according to the first voiceprint representation and the second voiceprint representation.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a character audio matching method, device, electronic device and storage medium. Background Art

[0002] In audiovisual works, the voice characteristics of characters are key elements in shaping character traits and conveying emotions. Character audio not only enhances the audience's viewing experience, but is also an important data source for voice model training. The data for voice model training usually needs to clearly mark the characters corresponding to each audio information to ensure the accuracy of training.

[0003] However, in the related art, the method of extracting character audio mainly relies on manual editing, which is time-consuming and labor-intensive, and it is difficult to efficiently match a large amount of character audio with the character.

[0004] Therefore, how to more efficiently match characters and their audio in audiovisual works has become a problem that needs to be solved urgently in the industry. Summary of the invention

[0005] The present invention provides a character audio matching method, device, electronic device and storage medium, which are used to solve the problem of how to more efficiently match characters and character audio in audiovisual works in the prior art.

[0006] The present invention provides a character audio matching method, comprising the following steps: Acquire audiovisual work data of the audiovisual work to be processed, and first voiceprint representations corresponding to each character in the audiovisual work to be processed; Extracting single-character audio from the audiovisual work data to obtain multiple single-character audio information; Based on the second voiceprint representation corresponding to each of the single-character audio information, audio clustering is performed on the multiple single-character audio information to obtain at least one single-character audio information set; Based on the voiceprint representation similarity information between the target character audio information of each of the characters and each of the single character audio information sets, determine the single character audio information set that matches each of the characters; wherein the voiceprint representation similarity information is determined based on the first voiceprint representation and the second voiceprint representation.

[0007] According to a role audio matching method provided by the present invention, single role audio is extracted from the audio-visual work data to obtain multiple single role audio information, including: Converting the audiovisual work data into audiovisual work audio data, and dividing the audiovisual work audio data into a plurality of sub-audio data segments by voice activity detection; After removing the background sound from each of the sub-audio data, performing speech quality evaluation on each of the sub-audio data, and determining target sub-audio data from each of the sub-audio data; wherein the target sub-audio data is the sub-audio data whose speech quality evaluation value exceeds a first preset threshold; The target sub-audio data including only the single-character audio is acquired to obtain a plurality of single-character audio information.

[0008] According to a character audio matching method provided by the present invention, the method for calculating the voiceprint representation similarity information specifically includes: Determining the class center of the single-character audio information set according to the mean of the second voiceprint representation of the single-character audio information in the single-character audio information set; The voiceprint representation similarity information is obtained according to the class center of the single-character audio information set and the first voiceprint representation corresponding to the target character audio information of the character.

[0009] According to a role audio matching method provided by the present invention, based on the voiceprint representation similarity information of the target role audio information of each role and each single role audio information set, a single role audio information set matching each role is determined, comprising: When the voiceprint representation similarity information between the class center of any single-character audio information set and the first voiceprint representation of any character is less than a second preset threshold, the single-character audio information set whose voiceprint representation similarity information is less than the second preset threshold is matched with the character to obtain a single-character audio information set that matches each character in the audiovisual work to be processed.

[0010] According to a character audio matching method provided by the present invention, matching a single character audio information set whose voiceprint representation similarity information is less than a second preset threshold with the character includes: In each of the single-character audio information sets, if the voiceprint representation similarity information between the class center of any target single-character audio information set and the first voiceprint representations of multiple characters is less than a second preset threshold, the target character is matched with the target single-character audio information set; The target character is a character among the multiple characters whose voiceprint representation similarity information with the class center of the target single character audio information set is the smallest.

[0011] According to a character audio matching method provided by the present invention, the method for obtaining the first voiceprint representation of each character specifically includes: In the audiovisual work data, obtaining character audio information corresponding to each character marked by the user; After removing the background sound of each of the character audio information, the character audio information corresponding to each character is spliced ​​to obtain the target character audio information corresponding to each character; Voiceprint representation is extracted from the target character audio information corresponding to each character to obtain a first voiceprint representation corresponding to each character.

[0012] According to a character audio matching method provided by the present invention, after the step of determining a single character audio information set matching each character based on the target character audio information of each character and the voiceprint representation similarity information of each single character audio information set, the method further comprises: Based on the single-role audio information sets matched with the roles, an audio library of the audiovisual work roles of the audiovisual work to be processed is constructed.

[0013] The present invention also provides a character audio matching device, comprising: an acquisition module, configured to acquire audiovisual work data of the audiovisual work to be processed, and acquire a first voiceprint representation of each character according to target character audio information of each character in the audiovisual work to be processed in the audiovisual work data; An extraction module, used for extracting single-character audio from the audio-visual work data to obtain multiple single-character audio information; A clustering module, configured to perform audio clustering on the plurality of single-character audio information based on the second voiceprint representation corresponding to each of the single-character audio information, to obtain at least one single-character audio information set; A matching module is used to determine the single-character audio information set that matches each of the characters based on the voiceprint representation similarity information of the target character audio information of each of the characters and each of the single-character audio information sets; wherein the voiceprint representation similarity information is determined based on the first voiceprint representation and the second voiceprint representation.

[0014] According to the character audio matching device provided by the present invention, the device is also used for: Converting the audiovisual work data into audiovisual work audio data, and dividing the audiovisual work audio data into a plurality of sub-audio data segments by voice activity detection; After removing the background sound from each of the sub-audio data, performing speech quality evaluation on each of the sub-audio data, and determining target sub-audio data from each of the sub-audio data; wherein the target sub-audio data is the sub-audio data whose speech quality evaluation value exceeds a first preset threshold; The target sub-audio data including only the single-character audio is acquired to obtain a plurality of single-character audio information.

[0015] According to the character audio matching device provided by the present invention, the device is also used for: In the audiovisual work data, obtaining character audio information corresponding to each character marked by the user; After removing the background sound of each of the character audio information, the character audio information corresponding to each character is spliced ​​to obtain the target character audio information corresponding to each character; Voiceprint representation is extracted from the target character audio information corresponding to each character to obtain a first voiceprint representation corresponding to each character.

[0016] According to the character audio matching device provided by the present invention, the device is also used for: Determining the class center of the single-character audio information set according to the mean of the second voiceprint representation of the single-character audio information in the single-character audio information set; The voiceprint representation similarity information is obtained according to the class center of the single-character audio information set and the first voiceprint representation corresponding to the target character audio information of the character.

[0017] According to the character audio matching device provided by the present invention, the device is also used for: When the voiceprint representation similarity information between the class center of any single-character audio information set and the first voiceprint representation of any character is less than a second preset threshold, the single-character audio information set whose voiceprint representation similarity information is less than the second preset threshold is matched with the character to obtain a single-character audio information set that matches each character in the audiovisual work to be processed.

[0018] According to the character audio matching device provided by the present invention, the device is also used for: In each of the single-character audio information sets, if the voiceprint representation similarity information between the class center of any target single-character audio information set and the first voiceprint representations of multiple characters is less than a second preset threshold, the target character is matched with the target single-character audio information set; The target character is a character among the multiple characters whose voiceprint representation similarity information with the class center of the target single character audio information set is the smallest.

[0019] According to the character audio matching device provided by the present invention, the device is also used for: Based on the single-role audio information sets matched with the roles, an audio library of the roles of the audiovisual work to be processed is constructed.

[0020] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned character audio matching methods is implemented.

[0021] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the character audio matching method as described in any one of the above is implemented.

[0022] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the character audio matching method as described above is implemented.

[0023] The character audio matching method, device, electronic device and storage medium provided by the present invention ensure that clear and independent single-character audio information is obtained from complex audio-visual work data by realizing accurate extraction of single-character audio. Subsequently, these single-character audio information are automatically classified through audio clustering technology to form an audio information set that is convenient for subsequent processing. In addition, using voiceprint representation and similarity measurement technology, the class center of each audio information set is matched with the preset first voiceprint representation, thereby achieving accurate correspondence between each single-character audio information set and the most similar character. This automated matching process not only reduces the workload of manual editing and matching, reduces post-production costs, but also improves the accuracy of matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a flow chart of the character audio matching method provided by the present invention; Figure 2 The character audio library construction process provided by the present invention; Figure 3 A schematic diagram of the structure of the character audio matching device provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Figure 1is a flow chart of the character audio matching method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 110, obtaining audiovisual work data of the audiovisual work to be processed, and obtaining a first voiceprint representation of each character according to target character audio information of each character in the audiovisual work to be processed in the audiovisual work data; In the present invention, the audio-visual works to be processed may specifically include: movies, TV series, animations, and game audio-visual contents, etc.

[0028] In the present invention, the audiovisual work data of the audiovisual work to be processed may refer to the audio data or video data of the audiovisual work.

[0029] In the present invention, voiceprint representation is a kind of data that can represent the characteristics of an individual's voice, which is usually obtained by analyzing parameters such as the spectrum, rhythm, and pitch of the voice.

[0030] In the present invention, the first voiceprint representation corresponding to each character in the audiovisual work to be processed may specifically be the first voiceprint representation obtained by extracting the voiceprint representation of target character audio information corresponding to each character manually screened by the user.

[0031] Step 120, extracting single-character audio from the audiovisual work data to obtain multiple single-character audio information; In the present invention, single character audio extraction may specifically refer to a process of separating a single character audio signal from an audiovisual work.

[0032] In the present invention, a multi-person conversation speech recognition tool can be used to screen out situations where multiple people are talking in the audio, ensuring that basically only one character is speaking in each sub-audio. The multi-person conversation speech recognition tool can specifically refer to tools such as speech recognition models with corresponding functions.

[0033] In the present invention, the single-character audio information is separated from the audio-visual work data to include only the audio of the single character.

[0034] Step 130, based on the second voiceprint representation corresponding to each of the single-role audio information, performing audio clustering on the plurality of single-role audio information to obtain at least one single-role audio information set; In the present invention, the audio information of each single character can be clustered according to a clustering algorithm, and the single character audio information with similar voiceprint representations can be clustered together. The single character audio information with similar voiceprint representations is usually audio information from the same user character. Therefore, the audio information of each user character can be clustered and divided through a clustering process.

[0035] In the present invention, each single-role audio information set may specifically refer to a set consisting of individual single-role audio information of a user role.

[0036] In the present invention, the audio clustering process can be achieved through clustering algorithms such as k-means clustering algorithm and spectral clustering algorithm.

[0037] Step 140, based on the voiceprint representation similarity information of the target character audio information of each character and each of the single character audio information sets, determine the single character audio information set that matches each of the characters; wherein the voiceprint representation similarity information is determined based on the first voiceprint representation and the second voiceprint representation.

[0038] In the present invention, In the present invention, a pre-trained voiceprint recognition model can be used, the voiceprint representation extraction module in the voiceprint recognition model can be taken out separately, and then the audio clips can be processed to extract the voiceprint vector information (such as i-vector, x-vector, etc.) of each audio clip.

[0039] In the present invention, the voiceprint representation similarity information may specifically refer to Euclidean distance, cosine distance or Manhattan distance. The similarity between the class center of the single-character audio information set and the first voiceprint representation of each character can be quantified through the voiceprint representation similarity information. Thus, the single-character audio information set matching each character is determined.

[0040] The class center is determined based on the average of the second voiceprint representation of each single-role audio information in the single-role audio information set. Specifically, the average of the voiceprint representation of all audio in each audio information set can be calculated to obtain the class center of the set.

[0041] In the present invention, by realizing accurate extraction of single-character audio, it is ensured that clear and independent single-character audio information is obtained from complex audiovisual work data. Subsequently, through audio clustering technology, these single-character audio information are automatically classified to form an audio information set that is convenient for subsequent processing. In addition, using voiceprint representation and similarity measurement technology, the class center of each audio information set is matched with the preset first voiceprint representation, so as to achieve accurate correspondence between each single-character audio information set and the most similar character. This automated matching process not only reduces the workload of manual editing and matching, reduces post-production costs, but also improves the accuracy of matching.

[0042] Optionally, single-role audio is extracted from the audiovisual work data to obtain multiple single-role audio information, including: Converting the audiovisual work data into audiovisual work audio data, and dividing the audiovisual work audio data into a plurality of sub-audio data segments by voice activity detection; After removing the background sound from each of the sub-audio data, performing speech quality evaluation on each of the sub-audio data, and determining target sub-audio data from each of the sub-audio data; wherein the target sub-audio data is the sub-audio data whose speech quality evaluation value exceeds a first preset threshold; The target sub-audio data including only the single-character audio is acquired to obtain the audio information of each single character.

[0043] In the present invention, first, the audiovisual work data is converted into audio data, and voice activity detection is performed to identify the time period containing voice. In this way, the audiovisual work audio data can be divided into multiple sub-audio data.

[0044] In the segmented sub-audio data, background sound is removed to improve the clarity of speech. This step can be achieved through audio signal processing technology, such as using a noise reduction algorithm to reduce or eliminate background noise.

[0045] More specifically, the denoised sub-audio data is subjected to speech quality evaluation to obtain speech quality evaluation values ​​of each sub-audio data, and the sub-audio data whose speech quality evaluation value exceeds a first preset threshold is determined as the target sub-audio data, thereby ensuring that the extracted audio information is a valid speech signal.

[0046] In the present invention, a portion containing only a single character audio is further extracted from the target sub-audio data to obtain each single character audio information. This step can be achieved by voiceprint recognition technology, which can avoid introducing multiple audios into one audio segment, resulting in the inability to determine the user role to which the audio belongs.

[0047] In the present invention, the audio information of each single character can be accurately extracted from the audiovisual work, which provides a solid foundation for subsequent audio processing and analysis. It not only improves the accuracy and efficiency of the extraction process, but also enhances the robustness and practicality of the system.

[0048] Optionally, the performing audio clustering on the plurality of single-role audio information based on the second voiceprint representation corresponding to each of the single-role audio information to obtain at least one single-role audio information set includes: Performing voiceprint extraction on each of the single-character audio information to obtain a second voiceprint representation corresponding to each of the single-character audio information; Audio clustering is performed according to the second voiceprint representation corresponding to each of the single-character audio information to obtain at least one single-character audio information set; wherein each of the single-character audio information sets includes at least one single-character audio information.

[0049] In the invention, voiceprint extraction is performed on each single-character audio information to obtain a corresponding second voiceprint representation.

[0050] In the present invention, specifically, the second voiceprint representation may be extracted through a voiceprint extraction model, such as xvector, etc., and the character audio may be input to obtain the corresponding voiceprint representation.

[0051] In the present invention, audio clustering is performed based on the extracted second voiceprint representation to obtain at least one single-character audio information set. This process can be implemented by a clustering algorithm, such as K-means, Gaussian mixture model or a method based on deep learning. Thus, a single-character audio information set with similar voiceprint representation is obtained.

[0052] In the present invention, single-character audio information with similar voiceprint representations can be effectively clustered together, and a set of single-character audio information matching each character can be determined, providing an accurate audio data set for subsequent character audio matching and analysis.

[0053] Optionally, the method for calculating the voiceprint representation similarity information specifically includes: Determining the class center of the single-character audio information set according to the mean of the second voiceprint representation of the single-character audio information in the single-character audio information set; The voiceprint representation similarity information is obtained according to the class center of the single-character audio information set and the first voiceprint representation corresponding to the target character audio information of the character.

[0054] In the present invention, in the audio clustering process, each single-character audio information set forms a cluster. The cluster center is the statistical average of the voiceprint representations of all single-character audio information in this cluster. Specifically, for each cluster, the mean of the second voiceprint representations of all audio information within it is calculated as the cluster center of the cluster.

[0055] For the first voiceprint representation of each character, it is necessary to match it with the class center of all single-character audio information sets. This step is to find the audio information set that matches the voiceprint representation of each character.

[0056] A similarity measurement algorithm is used to calculate the similarity between the first voiceprint representation and each class center. Commonly used similarity measurement methods include cosine similarity, Euclidean distance, Manhattan distance, etc. These metrics can help determine which class center is closest to the voiceprint representation of a specific character.

[0057] In the present invention, each calculated voiceprint representation similarity information is compared with a second preset threshold value. If the voiceprint representation similarity information is less than the threshold value, then the single character audio information set is considered to be sufficiently similar to the character's voiceprint representation.

[0058] For single-role audio information sets and roles whose voiceprint representation similarity information is less than the second preset threshold, matching confirmation is performed, which means that these audio information sets are considered to match the corresponding roles.

[0059] For example, when the voiceprint representation similarity information is Euclidean distance, in the context of voiceprint recognition and audio clustering, Euclidean distance is used to measure the similarity between two voiceprint representations (such as MFCC feature vectors). The smaller the Euclidean distance, the closer the positions of the two voiceprint representations in the feature space are, that is, their similarity is higher.

[0060] Specifically, if the Euclidean distance between the class center of two single-character audio information sets and the first voiceprint representation of a certain character is small, this indicates that the voiceprint features in these audio information sets are very similar to the voiceprint features of the character, and therefore these audio information sets can be considered to belong to the same character.

[0061] In the present invention, by using the similarity information of the class center of the single-character audio information set and the voiceprint representation of the first angular print representation of each character, the single-character audio information set in the audiovisual work can be accurately matched with the corresponding character, ensuring the accuracy and efficiency of audio processing.

[0062] Optionally, matching the single-role audio information set whose voiceprint representation similarity information is less than a second preset threshold with the role includes: In each of the single-character audio information sets, if the voiceprint representation similarity information between the class center of any target single-character audio information set and the first voiceprint representations of multiple characters is less than a second preset threshold, the target character is matched with the target single-character audio information set; The target character is a character among the multiple characters whose voiceprint representation similarity information with the class center of the target single character audio information set is the smallest.

[0063] In the present invention, if the voiceprint representation similarity information between the class center of any target single-character audio information set and the first voiceprint representations of multiple characters is less than the second preset threshold, it means that it is impossible to determine which character the target single-character audio information set should match.

[0064] Therefore, the size of the voiceprint representation similarity information between the class centers of the target single-character audio information set can be further compared, and the character with the smallest voiceprint representation similarity information can be matched with the target single-character audio information set.

[0065] In the present invention, when the voiceprint representation similarity information between the class center of any target single-character audio information set and the first voiceprint representations of multiple characters is less than a second preset threshold, the target character can also be matched with the target single-character audio information set, thereby effectively ensuring the accuracy of the character audio matching.

[0066] Optionally, the method for acquiring the first voiceprint representation of each character specifically includes: In the audiovisual work data, obtaining character audio information corresponding to each character marked by the user; After removing the background sound of each of the character audio information, the character audio information corresponding to each character is spliced ​​to obtain the target character audio information corresponding to each character; Voiceprint representation is extracted from the target character audio information corresponding to each character to obtain a first voiceprint representation corresponding to each character.

[0067] In the present invention, character audio information corresponding to each character is selected from the audio-visual work data according to user annotations.

[0068] For example, from all episodes of a TV series, select about 5 episodes containing videos of the main characters (if it is a movie type, only the episode of the movie needs to be provided), and provide them to the annotators for viewing and manual labeling. During the labeling process, the appearance and entry timestamps of the characters will be recorded. Each character only needs to be labeled for about 15 seconds of appearance time in each episode.

[0069] Then, you can use video processing software or command line tools to convert the audiovisual work data into audio formats. Common audio formats include WAV, MP3, etc.

[0070] Specifically, each sub-audio clip is also processed using a background music removal tool or algorithm to remove the background music. This can be achieved through spectral subtraction, sound separation algorithms or deep learning methods.

[0071] In the present invention, all the role audio information of the same role can be spliced ​​together to form a continuous audio of about 1 minute in length. This can be done manually by audio editing software, or by using programming library automation processing, and finally the target role audio information corresponding to each role is obtained.

[0072] In the present invention, voiceprint features are extracted from the spliced ​​audio, such as MFCC, i-vector, etc., to obtain the first voiceprint representation corresponding to each role. These features will be used for voiceprint recognition and matching. Data storage.

[0073] In the present invention, by using the target character audio information corresponding to each character annotated by the user in small amounts, the accurate first voiceprint representation corresponding to each character can be effectively obtained, thereby achieving automatic matching of the character audio of the entire audiovisual work through a small amount of manual annotation.

[0074] Optionally, after the step of determining a single-character audio information set matching each character based on the class center of each of the single-character audio information sets and the voiceprint representation similarity information of the first voiceprint representation of each character, the method further comprises: Based on the single-role audio information sets matched with the roles, an audio library of the roles of the audiovisual work to be processed is constructed.

[0075] In the present invention, a database management system can be used to construct a role audio library to store single-role audio information sets of various roles.

[0076] In the present invention, an audio library of film and television drama characters can be established, and massive audio materials can be allocated to specific characters. This not only can obtain a rich library of audio materials with different character characteristics, but also can facilitate multi-style and multi-character voice models to quickly identify and manage large-scale character audio during training.

[0077] Alternatively, taking film and television dramas as an example, Figure 2 The character audio library construction process provided by the present invention is as follows: Figure 2 As shown in the figure, obtain film and television resources; in the voiceprint extraction process of film and television characters, select videos containing each character through manual labeling, record the time when the character appears and capture about 15 seconds of sub-audio; then, remove the background music of non-character audio, splice it into about 1 minute of audio, and use a voiceprint extraction model such as xvector to obtain voiceprint representation.

[0078] At the same time, in the process of preprocessing the film and television drama data, high-quality, single-character audio data is extracted from the original film and television drama resources. First, the video is converted into long audio, and then the dialogue is segmented through voice activity detection, background music is removed, voice quality is evaluated, and sub-audio of single-person dialogue is screened out. At the same time, the speech transcription tool is used to obtain text information, and finally multiple high-quality sub-audio of single-person speaking is obtained.

[0079] In the process of audio clustering, the voiceprint extraction model is used to obtain the voiceprint representation of each sub-audio; then, clustering tools such as the k-means clustering algorithm are used to input the voiceprint representation of all sub-audio to obtain the voiceprint category corresponding to each sub-audio. Different voiceprint categories generally represent different roles.

[0080] Finally, the class and the character are matched. If the Euclidean distance is lower than a certain threshold, it means that the audio voiceprint of this class is close to the character. After this step is completed, all the single-person sub-audios corresponding to each character in each film and television drama can be obtained, and the construction of the film and television drama character audio library is completed.

[0081] The character audio matching device provided by the present invention is described below. The character audio matching device described below and the character audio matching method described above can be referenced to each other.

[0082] Figure 3 A schematic diagram of the structure of the character audio matching device provided by the present invention, such as Figure 3 As shown, including: The acquisition module 310 is used to acquire audiovisual work data of the audiovisual work to be processed, and acquire the first voiceprint representation of each character according to the target character audio information of each character in the audiovisual work to be processed in the audiovisual work data; The extraction module 320 is used to extract single-character audio from the audio-visual work data to obtain multiple single-character audio information; The clustering module 330 is used to perform audio clustering on the plurality of single-role audio information based on the second voiceprint representation corresponding to each of the single-role audio information to obtain at least one single-role audio information set; The matching module 340 is used to determine the single-character audio information set that matches each of the characters based on the voiceprint representation similarity information between the target character audio information of each of the characters and each of the single-character audio information sets; wherein the voiceprint representation similarity information is determined based on the first voiceprint representation and the second voiceprint representation.

[0083] According to the character audio matching device provided by the present invention, the device is also used for: Converting the audiovisual work data into audiovisual work audio data, and dividing the audiovisual work audio data into a plurality of sub-audio data segments by voice activity detection; After removing the background sound from each of the sub-audio data, performing speech quality evaluation on each of the sub-audio data, and determining target sub-audio data from each of the sub-audio data; wherein the target sub-audio data is the sub-audio data whose speech quality evaluation value exceeds a first preset threshold; The target sub-audio data including only the single-character audio is acquired to obtain a plurality of single-character audio information.

[0084] According to the character audio matching device provided by the present invention, the device is also used for: In the audiovisual work data, obtaining character audio information corresponding to each character marked by the user; After removing the background sound of each of the character audio information, the character audio information corresponding to each character is spliced ​​to obtain the target character audio information corresponding to each character; Voiceprint representation is extracted from the target character audio information corresponding to each character to obtain a first voiceprint representation corresponding to each character.

[0085] According to the character audio matching device provided by the present invention, the device is also used for: Determining the class center of the single-character audio information set according to the mean of the second voiceprint representation of the single-character audio information in the single-character audio information set; The voiceprint representation similarity information is obtained according to the class center of the single-character audio information set and the first voiceprint representation corresponding to the target character audio information of the character.

[0086] According to the character audio matching device provided by the present invention, the device is also used for: When the voiceprint representation similarity information between the class center of any single-character audio information set and the first voiceprint representation of any character is less than a second preset threshold, the single-character audio information set whose voiceprint representation similarity information is less than the second preset threshold is matched with the character to obtain a single-character audio information set that matches each character in the audiovisual work to be processed.

[0087] According to the character audio matching device provided by the present invention, the device is also used for: In each of the single-character audio information sets, if the voiceprint representation similarity information between the class center of any target single-character audio information set and the first voiceprint representations of multiple characters is less than a second preset threshold, the target character is matched with the target single-character audio information set; The target character is a character among the multiple characters whose voiceprint representation similarity information with the class center of the target single character audio information set is the smallest.

[0088] According to the character audio matching device provided by the present invention, the device is also used for: Based on the single-role audio information sets matched with the roles, an audio library of the roles of the audiovisual work to be processed is constructed.

[0089] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the role audio matching method, which includes: obtaining audiovisual work data of the audiovisual work to be processed, and obtaining the first voiceprint representation of each role according to the target role audio information of each role of the audiovisual work to be processed in the audiovisual work data; Extracting single-character audio from the audiovisual work data to obtain multiple single-character audio information; Based on the second voiceprint representation corresponding to each of the single-character audio information, audio clustering is performed on the multiple single-character audio information to obtain at least one single-character audio information set; Based on the voiceprint representation similarity information between the target character audio information of each of the characters and each of the single character audio information sets, determine the single character audio information set that matches each of the characters; wherein the voiceprint representation similarity information is determined based on the first voiceprint representation and the second voiceprint representation.

[0090] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0091] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the character audio matching method provided by the above methods, the method comprising: obtaining audiovisual work data of an audiovisual work to be processed, and obtaining a first voiceprint representation of each character according to target character audio information of each character of the audiovisual work to be processed in the audiovisual work data; Extracting single-character audio from the audiovisual work data to obtain multiple single-character audio information; Based on the second voiceprint representation corresponding to each of the single-character audio information, audio clustering is performed on the multiple single-character audio information to obtain at least one single-character audio information set; Based on the voiceprint representation similarity information between the target character audio information of each of the characters and each of the single character audio information sets, determine the single character audio information set that matches each of the characters; wherein the voiceprint representation similarity information is determined based on the first voiceprint representation and the second voiceprint representation.

[0092] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the character audio matching method provided by the above methods, the method comprising: obtaining audiovisual work data of an audiovisual work to be processed, and obtaining a first voiceprint representation of each character according to target character audio information of each character of the audiovisual work to be processed in the audiovisual work data; Extracting single-character audio from the audiovisual work data to obtain multiple single-character audio information; Based on the second voiceprint representation corresponding to each of the single-character audio information, audio clustering is performed on the multiple single-character audio information to obtain at least one single-character audio information set; Based on the voiceprint representation similarity information between the target character audio information of each of the characters and each of the single character audio information sets, determine the single character audio information set that matches each of the characters; wherein the voiceprint representation similarity information is determined based on the first voiceprint representation and the second voiceprint representation.

[0093] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0094] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A character audio matching method, characterized in that: include: Acquire audiovisual work data of the audiovisual work to be processed, and acquire a first voiceprint representation of each character according to target character audio information of each character in the audiovisual work to be processed in the audiovisual work data; Extracting single-character audio from the audiovisual work data to obtain multiple single-character audio information; Based on the second voiceprint representation corresponding to each of the single-character audio information, audio clustering is performed on the multiple single-character audio information to obtain at least one single-character audio information set; Based on the voiceprint representation similarity information between the target character audio information of each of the characters and each of the single character audio information sets, determine the single character audio information set that matches each of the characters; wherein the voiceprint representation similarity information is determined based on the first voiceprint representation and the second voiceprint representation.

2. The character audio matching method according to claim 1, characterized in that: Single-character audio is extracted from the audiovisual work data to obtain multiple single-character audio information, including: Converting the audiovisual work data into audiovisual work audio data, and dividing the audiovisual work audio data into a plurality of sub-audio data segments by voice activity detection; After removing the background sound from each of the sub-audio data, performing speech quality evaluation on each of the sub-audio data, and determining target sub-audio data from each of the sub-audio data; wherein the target sub-audio data is the sub-audio data whose speech quality evaluation value exceeds a first preset threshold; The target sub-audio data including only the single-character audio is acquired to obtain a plurality of single-character audio information.

3. The character audio matching method according to claim 1, characterized in that: The method for calculating the voiceprint representation similarity information specifically includes: Determining the class center of the single-character audio information set according to the mean of the second voiceprint representation of the single-character audio information in the single-character audio information set; The voiceprint representation similarity information is obtained according to the class center of the single-character audio information set and the first voiceprint representation corresponding to the target character audio information of the character.

4. The character audio matching method according to claim 3, characterized in that: Based on the voiceprint representation similarity information between the target character audio information of each character and each of the single character audio information sets, determining a single character audio information set matching each character, including: When the voiceprint representation similarity information between the class center of any single-character audio information set and the first voiceprint representation of any character is less than a second preset threshold, the single-character audio information set whose voiceprint representation similarity information is less than the second preset threshold is matched with the character to obtain a single-character audio information set that matches each character in the audiovisual work to be processed.

5. The character audio matching method according to claim 4, characterized in that: Matching the single-role audio information set whose voiceprint representation similarity information is less than a second preset threshold with the role includes: In each of the single-character audio information sets, if the voiceprint representation similarity information between the class center of any target single-character audio information set and the first voiceprint representations of multiple characters is less than a second preset threshold, the target character is matched with the target single-character audio information set; The target character is a character among the multiple characters whose voiceprint representation similarity information with the class center of the target single character audio information set is the smallest.

6. The character audio matching method according to claim 1, characterized in that: The method for obtaining the first voiceprint representation of each role specifically includes: In the audiovisual work data, obtaining character audio information corresponding to each character marked by the user; After removing the background sound of each of the character audio information, the character audio information corresponding to each character is spliced ​​to obtain the target character audio information corresponding to each character; Voiceprint representation is extracted from the target character audio information corresponding to each character to obtain a first voiceprint representation corresponding to each character.

7. The character audio matching method according to claim 1, characterized in that: After the step of determining a single-role audio information set matching each of the characters based on the voiceprint representation similarity information of the target character audio information of each of the characters and each of the single-role audio information sets, the method further comprises: Based on the single-role audio information sets matched with the roles, an audio library of the roles of the audiovisual work to be processed is constructed.

8. A character audio matching device, characterized in that: include: an acquisition module, configured to acquire audiovisual work data of the audiovisual work to be processed, and acquire a first voiceprint representation of each character according to target character audio information of each character in the audiovisual work to be processed in the audiovisual work data; An extraction module, used for extracting single-character audio from the audiovisual work data to obtain multiple single-character audio information; A clustering module, configured to perform audio clustering on the plurality of single-character audio information based on the second voiceprint representation corresponding to each of the single-character audio information, to obtain at least one single-character audio information set; A matching module is used to determine the single-character audio information set that matches each of the characters based on the voiceprint representation similarity information of the target character audio information of each of the characters and each of the single-character audio information sets; wherein the voiceprint representation similarity information is determined based on the first voiceprint representation and the second voiceprint representation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the character audio matching method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the character audio matching method according to any one of claims 1 to 7 is implemented.

Images