A speech feature extraction method and system for dubbing

Abstract

The application relates to the technical field of speech signal processing, and relates to a speech feature extraction method and system for animation dubbing, which comprises the following steps: confirming an animation to be dubbed, performing audio stream extraction on the animation to be dubbed to obtain an original audio stream, performing speech activity detection on the original audio stream to obtain a pure human voice segment, performing speaker separation on the pure human voice segment to obtain a plurality of speech segment sets, confirming an oral type video frame sequence based on the speech segment sets and the animation to be dubbed, performing acoustic feature extraction on the speech segment sets to obtain an acoustic feature sequence, performing emotion analysis on the acoustic feature sequence to obtain an emotion category, confirming visual timing features based on a plurality of oral type video frames in the oral type video frame sequence, summarizing character speech feature portraits to obtain a plurality of character speech feature portraits, and completing speech feature extraction. The application can overcome the strong subjectivity and poor consistency of traditional artificial listening and distinguishing imitation, and further realizes accurate and stable extraction of speech features for animation dubbing.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and in particular to a method and system for speech feature extraction in animation translation. Background Technology

[0002] With the increasing frequency of cultural exchanges and the rapid development of the animation industry, a large number of excellent animated works need to be translated and disseminated in different language markets. In the process of translating animation, dubbing not only needs to accurately convey the content of the original work semantically, but also needs to maintain a high degree of consistency with the original dubbing and the lip movements and performances of the animated characters in terms of timbre, intonation, emotion, and rhythm, so as to ensure the viewing experience and artistic expression of the translated works.

[0003] Currently, voice processing in animation dubbing mainly relies on a combination of voice actors' manual listening and experience-based imitation. During dubbing, dubbing directors and voice actors usually listen to the original dubbing repeatedly, grasping the timbre, intonation, and emotional characteristics of the original voice based on their subjective feelings. Then, the voice actors imitate and reproduce it. Through multiple trial recordings, comparisons, and adjustments, the dubbing is made to be as close as possible to the original voice in terms of speech performance, and finally the dubbing recording and synthesis are completed.

[0004] However, the existing methods mentioned above have many shortcomings. Manual listening and imitation heavily rely on the voice actors' subjective judgment and experience, making it difficult to objectively and accurately quantify and extract the timbre, intonation, and emotional characteristics of the original voice acting. This easily leads to significant deviations between the translated voice acting and the original voice. The process of repeated recordings and manual comparison and adjustment is cumbersome, time-consuming, and inefficient, and the consistency of results among different voice actors is difficult to guarantee, failing to meet the demands of animation dubbing for efficient, accurate, and stable voice feature extraction. Therefore, overcoming the shortcomings of traditional manual listening and imitation—high subjectivity and poor consistency—and achieving accurate and stable extraction of voice features in animation dubbing has become an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a speech feature extraction method and a computer-readable storage medium for animation translation. Its main purpose is to overcome the shortcomings of traditional manual listening and imitation, which are subject to strong subjectivity and poor consistency, and thus achieve accurate and stable extraction of speech features for animation translation.

[0006] To achieve the above objectives, the present invention provides a speech feature extraction method for animation translation, comprising: Once the animation to be translated is identified, its audio stream is extracted to obtain the original audio stream. Speech activity detection is performed on the original audio stream to obtain a clean human voice segment; Speaker separation is performed on the pure human voice segment to obtain multiple speech fragment sets; For each of the multiple speech segment sets, perform the following operation: Based on the set of speech segments and the animation to be translated, the lip-sync video frame sequence was identified, which includes: multiple lip-sync video frames; Acoustic features are extracted from the speech segment set to obtain an acoustic feature sequence, which includes multiple sets of Mel frequency cepstral coefficients. Sentiment analysis is performed on acoustic feature sequences to obtain sentiment categories; Prosodic feature parameters were identified based on multiple Mel frequency cepstral coefficient sets in the acoustic feature sequence; By jointly encoding prosodic feature parameters and emotion categories, a character's voice style vector is obtained; Visual temporal features were identified based on multiple lip-sync video frames in the lip-sync video frame sequence. The sound-image coupling features were identified based on acoustic feature sequences and visual temporal features. A character voice feature profile was identified based on the character's voice style vector and audio-visual coupling features. By summarizing the voice feature profiles of the characters, multiple voice feature profiles are obtained, and voice feature extraction is completed.

[0007] Optionally, the step of performing speech activity detection on the original audio stream to obtain a clean human voice segment includes: The original audio stream is resampled to obtain a resampled audio stream; The resampled audio stream is then emphasized to obtain the target audio stream; The target audio stream is segmented into frames according to a preset first duration window to obtain a long-time frame sequence, which includes multiple long-time frames. For each of the multiple long-time frames in a long-time frame sequence, perform the following operation: Energy decomposition is performed on long time frames to obtain the speech main frequency energy and non-speech energy; Speech saliency is calculated based on the dominant speech energy and non-speech energy, where speech saliency is equal to the dominant speech energy divided by the non-speech energy. Summarize the speech saliency to obtain multiple speech saliency values; The target audio stream is segmented into frames according to a preset second duration window to obtain a short frame sequence, which includes multiple short frames. For each of the multiple short-time frames in a short-time frame sequence, perform the following operation: Speech activity scores are determined based on short time frames; Summarize the voice activity scores to obtain multiple voice activity scores; Multiple initial speech segments were identified based on the original audio stream, long-time frame sequence, short-time frame sequence, multiple speech saliency, and multiple speech activity scores. For each of the multiple initial speech segments, perform the following operation: The initial speech segment is enhanced with human voice to obtain the enhanced speech segment; The enhanced speech segment is smoothed to obtain a clean speech segment; By summing up the clean speech segments, multiple clean speech segments are obtained; A clean human voice segment was identified based on multiple clean speech segments.

[0008] Optionally, the energy decomposition of the long-time frame to obtain the speech frequency energy and non-speech energy includes: The long-time frame is subjected to time-frequency variation to obtain the long-time frame spectrum, which includes multiple frequency points; The power spectrum sequence was determined based on multiple frequency points, wherein the power spectrum sequence includes multiple power spectral densities; The power spectrum sequence is divided into sub-bands according to the preset human voice dominant frequency range to obtain the dominant frequency sub-band power sequence and the non-dominant frequency sub-band power sequence. The fundamental frequency is detected by performing fundamental frequency detection on the power sequence of the main frequency sub-band to obtain the long-time frame fundamental frequency estimate and fundamental frequency confidence. The fundamental frequency confidence level is compared with a preset confidence threshold. If the fundamental frequency confidence level is greater than the confidence threshold, multiple harmonic frequency points are identified based on the long-time frame fundamental frequency estimate. Perform the following operation for each of the multiple harmonic frequency points: The peak harmonic power was determined based on the harmonic frequency, the preset harmonic bandwidth, and the main frequency sub-band power sequence. By summing the peak harmonic powers, multiple peak harmonic powers are obtained; The harmonic structure energy is determined based on the peak power of multiple harmonics, where the harmonic structure energy is the sum of the peak powers of multiple harmonics; Multiple harmonic peak powers are removed from the main frequency subband power sequence to obtain the removal sequence, which includes multiple background powers. The dominant frequency background energy is determined based on multiple background powers in the elimination sequence, where the dominant frequency background energy is the sum of multiple background powers in the elimination sequence; The speech frequency energy is calculated based on the harmonic structure energy, the main frequency background energy, and the preset harmonic gain coefficient. If the fundamental frequency confidence is less than or equal to the confidence threshold, the speech fundamental frequency energy is determined based on the fundamental frequency subband power sequence, where the speech fundamental frequency energy is the sum of the power of multiple fundamental frequency subbands in the fundamental frequency subband power sequence; The non-speech energy of the subband was identified based on the power sequence of the non-dominant frequency subband. The noise proportion factor is calculated based on the power spectrum sequence, using the following formula:

[0009] in, Indicates the noise proportion coefficient. This indicates the number of power spectral densities among multiple power spectral densities in a power spectrum sequence. Represents the first power spectral density among multiple power spectral densities in a power spectral sequence. A power spectral density, Represents the logarithmic function. Indicated by An exponential function with base π, where π is the base of the function. It is a natural constant; The converted non-speech energy is calculated based on the sub-band non-speech energy and the noise ratio coefficient, where the converted non-speech energy is the product of the sub-band non-speech energy and the noise ratio coefficient. The residual energy of the main frequency was determined based on the main frequency sub-band power sequence and the preset power lower limit. The non-speech energy is determined based on the calculated non-speech energy and the residual energy of the main frequency. The non-speech energy is the sum of the calculated non-speech energy and the residual energy of the main frequency.

[0010] Optionally, the step of determining the voice activity score based on short time frames includes: Zero-crossing rate detection is performed on short frames to obtain the zero-crossing rate value; Energy is calculated for short-time frames to obtain short-time energy values; The short-time spectrum is obtained by performing time-frequency transformation on the short-time frame; The spectral flatness value was determined based on the short-time spectrum. The zero-crossing rate, short-time energy, and spectral flatness values ​​are normalized to obtain normalized zero-crossing rate, normalized short-time energy, and normalized spectral flatness. Speech activity scores are calculated based on normalized zero-crossing rate, normalized short-time energy, and normalized spectral flatness.

[0011] Optionally, the step of identifying multiple initial speech segments based on the original audio stream, long-time frame sequence, short-time frame sequence, multiple speech saliency scores, and multiple speech activity scores includes: For each of the multiple short-time frames in a short-time frame sequence, perform the following operation: The center time and target speech activity score were determined based on short time frames and multiple speech activity scores. The corresponding long time frame was identified based on the center time and the long time frame sequence. The target speech saliency was determined based on the corresponding long time frame and multiple speech saliency values. The saliency of the target speech is compared with a preset saliency threshold. If the saliency of the target speech is less than the saliency threshold, the short frame is treated as a non-speech frame. If the saliency of the target speech is greater than or equal to the saliency threshold, then compare the target speech activity score with the preset first threshold. If the target speech activity score is greater than or equal to the first threshold, then the short frame is taken as the speech frame; If the target speech activity score is less than the first threshold, then compare the target speech activity score with the preset second threshold; If the target speech activity score is less than the second threshold, the short frame is treated as a non-speech frame. If the target speech activity score is greater than or equal to the second threshold, then the short frame is designated as a pending frame. Speech frames, non-speech frames, and frames to be determined are summarized separately to obtain multiple speech frames, multiple non-speech frames, and multiple frames to be determined. For each of the multiple undetermined frames, perform the following operation: The annotation results of the undetermined frames are confirmed based on the undetermined frames, multiple speech frames, and multiple non-speech frames. The standard result of the undetermined frames is either a speech frame or a non-speech frame. The annotation results of the undetermined frames are summarized to obtain multiple annotation results of the undetermined frames; Based on the original audio stream, the annotation results of multiple undetermined frames, multiple speech frames, multiple non-speech frames, and multiple undetermined frames, multiple initial speech segments were identified.

[0012] Optionally, the speaker separation of the pure human voice segment yields multiple speech fragment sets, including: The pure human voice segment is overlapped and segmented according to the preset segment length and preset sliding step size to obtain multiple speech analysis segments; For each of the multiple speech analysis segments, perform the following operation: Acoustic features are extracted from the speech analysis segment to obtain the segment's acoustic features; The acoustic features of the segments are input into a pre-constructed speaker embedding network to obtain speaker embedding vectors; Summarize the speaker embedding vectors to obtain multiple speaker embedding vectors; Extracting the first from multiple speaker embedding vectors Each speaker embedding vector, where... The initial value is 1; Extracting the first from multiple speaker embedding vectors Each speaker embedding vector, where... The initial value is 1; Based on the The speaker embedding vector and the first speaker embedding vector The speaker embedding vector confirms the first speaker embedding vector. The similarity of n targets, where the nth target is... The similarity of the nth target is... The speaker embedding vector and the first speaker embedding vector Cosine similarity of speaker embedding vectors; The first Each target similarity is placed into a pre-constructed similarity set to obtain an updated similarity set; make ,Will As The updated similarity set is used as the new similarity set, and the result extracted from the embedding vectors of multiple speakers is returned. The steps of embedding speaker vectors until... The updated similarity set is used as the first target set, where... The number of speaker embedding vectors in a set of multiple speaker embedding vectors; make ,Will As Using the first target set as the similarity set, the method returns the first similarity set extracted from multiple speaker embedding vectors. The steps of embedding speaker vectors until... The first target set is used as the target similarity set; Multiple groups of speaking humans were identified based on the target similarity set; Perform the following operation for each of the multiple speaking human clusters: A set of speech segments was identified based on the speaking human cluster and multiple speech analysis segments; By aggregating the audio segments, multiple audio segment sets are obtained.

[0013] Optionally, the identification of multiple speaking human clusters based on the target similarity set includes: The similarity matrix is ​​determined based on the target similarity set; The similarity matrix is ​​symmetrically normalized to obtain the normalized similarity matrix; Construct the Laplacian matrix based on the normalized similarity matrix; The Laplacian matrix is ​​decomposed to obtain multiple eigenvalues ​​and multiple eigenvectors; The eigenvector matrix is ​​determined based on multiple eigenvalues ​​and multiple eigenvectors; The eigenvector matrix is ​​normalized row by row to obtain the normalized eigenvector matrix; After identifying multiple candidate cluster numbers, perform the following operation for each of these candidate cluster numbers: Extract the first eigenvector from the normalized eigenvector matrix. The columns are used to obtain the candidate eigenvalue submatrix, where, Equal to the number of candidate clusters; Cluster the candidate feature submatrices to obtain the clustering results; Calculate the clustering coefficient based on the clustering results; Summarize the clustering coefficients to obtain multiple clustering coefficients; The largest clustering coefficient was determined based on multiple clustering coefficients, where the largest clustering coefficient is the largest among the multiple clustering coefficients; The optimal number of speaker categories was determined based on the maximum clustering coefficient and the number of multiple candidate clusters. Multiple speaker clusters were identified based on the optimal number of speaker categories and the normalized feature vector matrix.

[0014] Optionally, the prosodic feature parameters are identified based on multiple Mel frequency cepstral coefficient sets in the acoustic feature sequence, including: For each Mel frequency cepstral coefficient set in multiple Mel frequency cepstral coefficient sets in the acoustic feature sequence, the following operation is performed: Extract the zeroth-order Mel frequency cepstral coefficients from the Mel frequency cepstral coefficient set; The zeroth-order Mel frequency cepstral coefficients are used as the frame energy value; Summarize the frame energy values ​​to obtain multiple frame energy values; The average energy value is determined based on the energy values ​​of multiple frames, where the average energy value is the average of the energy values ​​of multiple frames. Prosodic feature parameters are calculated based on the energy values ​​of multiple frames and the average energy value. The calculation formula is as follows:

[0015] in, Represents prosodic feature parameters, Represents the energy value of multiple frames. Energy value per frame Represents the energy value of multiple frames. Energy value per frame Indicates the average energy. This indicates the number of frame energy values ​​among multiple frame energy values. and These represent the preset first prosodic weight coefficient and the preset second prosodic weight coefficient, respectively. This indicates taking the absolute value.

[0016] Optionally, the step of identifying visual temporal features based on multiple lip-sync video frames in a lip-sync video frame sequence includes: For each lip-sync video frame in a sequence of multiple lip-sync video frames, the following operation is performed: Face region detection is performed on lip-sync video frames to obtain face region images; Lip key point detection was performed on the face region image to obtain multiple inner lip key points; The vertical opening distance and horizontal width of the inner lip were determined based on multiple key points of the inner lip. The mouth opening degree is determined based on the vertical opening distance of the inner lip and the horizontal width of the inner lip. The mouth opening degree is equal to the vertical opening distance of the inner lip divided by the horizontal width of the inner lip. By connecting multiple key points of the inner lip in sequence, an inner lip contour polygon is obtained. The inner lip contour polygon includes multiple vertices, where each vertex has a vertex x-coordinate and a vertex y-coordinate. Calculate the area of ​​the inner lip contour based on the polygonal shape of the inner lip; The mouth opening degree is combined with the inner lip contour area to obtain the mouth shape geometric features; By summarizing the lip shape geometric features, multiple lip shape geometric features are obtained; Multiple lip shape geometric features are temporally encoded to obtain visual temporal features.

[0017] To achieve the above objectives, the present invention also provides a speech feature extraction system for animation translation, comprising: The speech segment acquisition module is used to identify the animation to be translated, extract the audio stream of the animation to be translated to obtain the original audio stream, perform speech activity detection on the original audio stream to obtain the clean human voice segment, and perform speaker separation on the clean human voice segment to obtain multiple speech segment sets; The emotion type recognition module performs the following operations on each of the multiple speech segment sets: based on the speech segment set and the animation to be translated, it identifies the lip-sync video frame sequence, wherein the lip-sync video frame sequence includes: multiple lip-sync video frames; it extracts acoustic features from the speech segment set to obtain an acoustic feature sequence, wherein the acoustic feature sequence includes: multiple Mel frequency cepstral coefficient sets; and it performs emotion analysis on the acoustic feature sequence to obtain the emotion category. The visual feature confirmation module is used to confirm prosodic feature parameters based on multiple Mel frequency cepstral coefficient sets in the acoustic feature sequence, and to jointly encode the prosodic feature parameters with the emotion category to obtain the character voice style vector. It also confirms visual temporal features based on multiple lip-sync video frames in the lip-sync video frame sequence. The speech feature extraction module is used to identify audio-visual coupling features based on acoustic feature sequences and visual temporal features, identify character speech feature profiles based on character speech style vectors and audio-visual coupling features, summarize character speech feature profiles, obtain multiple character speech feature profiles, and complete speech feature extraction.

[0018] To address the above problems, the present invention also provides an electronic device, the electronic device comprising: Memory, storing at least one instruction; and The processor executes the instructions stored in the memory to implement the above-described speech feature extraction method for animation translation.

[0019] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the aforementioned speech feature extraction method for animation translation.

[0020] To address the problems described in the background art, this invention identifies the animation to be translated, extracts its audio stream, and obtains the original audio stream. This invention separates independent audio from the animation to be translated, obtaining complete original sound data. This provides basic audio material for subsequent voice detection and feature analysis. Furthermore, it performs speech activity detection on the original audio stream to obtain pure human voice segments. This invention detects the human voice activity intervals in the original audio, distinguishing human voices from non-human voice parts such as silence, background music, and ambient sounds, extracting pure human voice segments containing only dialogue, reducing interference from irrelevant signals. Speaker separation is then performed on the pure human voice segments to obtain multiple speech segment sets. This invention clusters the pure human voice segments according to different speakers based on features such as timbre and frequency. All speech segments of each speaker are grouped into one set, facilitating subsequent independent processing by role. For each of the multiple speech segment sets, the following operations are performed: Based on the speech segment set and the animation to be translated, an lip-sync video frame sequence is identified, wherein the lip-sync video frame sequence includes: multiple lip-sync video frames. It can be seen that in this embodiment of the invention, the speech segment corresponding to the speaker is aligned with the animation screen on the time axis, and the lip-sync screen frames corresponding to the speech segment are extracted from the video to form a lip-sync sequence for lip movement analysis. Acoustic features are extracted from the speech segment set to obtain an acoustic feature sequence, wherein the acoustic feature sequence includes: multiple Mel frequency cepstral coefficient sets. It can be seen that in this embodiment of the invention, acoustic parameters are extracted from the speaker's speech segment to form an acoustic feature sequence that changes over time. Sentiment analysis is performed on the acoustic feature sequence to obtain the sentiment category. It can be seen that in this embodiment of the invention, by analyzing the energy changes, intonation fluctuations, etc. in the acoustic features, the emotional attributes expressed by the speech segment are determined, providing a sentiment dimension for the style vector.Based on multiple Mel-frequency cepstral coefficient sets in the acoustic feature sequence, prosodic feature parameters are identified. This embodiment of the invention further extracts prosodic-related parameters such as speech rate, pitch change rate, and pause distribution from the acoustic features to reflect speaking rhythm and intonation patterns. The prosodic feature parameters are jointly encoded with emotion categories to obtain a character speech style vector. This embodiment of the invention combines prosodic information and emotion categories into a fixed-dimensional vector to express the speaker's unique speech expression style. Based on multiple lip-sync video frames in the lip-sync video frame sequence, visual temporal features are identified. This embodiment of the invention analyzes visual information such as lip opening and closing, and shape changes in the lip-sync frame sequence to extract contour features that change over time. Based on acoustic feature sequences and visual temporal features, the audio-visual coupling features are identified. This invention demonstrates that by performing temporal alignment and correlation analysis between acoustic features and lip-sync visual features, the synchronization relationship and matching degree between sound and image are found. Based on the character's voice style vector and audio-visual coupling features, a character's voice feature profile is identified. This invention demonstrates that by merging the voice style vector and audio-visual coupling features, a complete profile of the character's voice quality, rhythm, emotion, and lip-sync is constructed. By summarizing the character's voice feature profiles, multiple character voice feature profiles are obtained, completing the voice feature extraction. This invention demonstrates that by aggregating the feature profiles of all characters, a complete set of character voice features for the entire animation is formed. Therefore, this invention overcomes the shortcomings of traditional manual listening and imitation, which suffers from strong subjectivity and poor consistency, thereby achieving accurate and stable extraction of voice features in animated translation. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a speech feature extraction method for animation translation provided in an embodiment of the present invention. Figure 2 A functional block diagram of a speech feature extraction system for animation translation provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device that implements the speech feature extraction method for animation translation according to an embodiment of the present invention.

[0022] Explanation of reference numerals in the attached figures: 10. Electronic device; 11. Processor; 12. Memory; 13. Bus.

[0023] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0024] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0025] This application provides a method for extracting speech features for animation translation. The execution entity of this speech feature extraction method for animation translation includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the speech feature extraction method for animation translation can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0026] Reference Figure 1 The diagram shown is a schematic flowchart of a speech feature extraction method for animation translation provided in an embodiment of the present invention. In this embodiment, the speech feature extraction method for animation translation includes: S1. Identify the animation to be translated, extract the audio stream from the animation to obtain the original audio stream.

[0027] For example, Xiao Zhang is an employee of an animation production company and needs to extract speech features from the completed animation. So Xiao Zhang identifies the animation to be translated, which will facilitate the subsequent extraction of speech features from the animation to be translated.

[0028] It should be explained that the completed animation is the animation to be translated, and the audio stream extraction of the animation to be translated refers to the process of extracting the audio stream of the animation to be translated, which is the original audio stream.

[0029] S2. Perform speech activity detection on the original audio stream to obtain a clean human voice segment.

[0030] In detail, the process of performing speech activity detection on the original audio stream to obtain a clean human voice segment includes: The original audio stream is resampled to obtain a resampled audio stream; The resampled audio stream is then emphasized to obtain the target audio stream; The target audio stream is segmented into frames according to a preset first duration window to obtain a long-time frame sequence, which includes multiple long-time frames. For each of the multiple long-time frames in a long-time frame sequence, perform the following operation: Energy decomposition is performed on long time frames to obtain the speech main frequency energy and non-speech energy; Speech saliency is calculated based on the dominant speech energy and non-speech energy, where speech saliency is equal to the dominant speech energy divided by the non-speech energy. Summarize the speech saliency to obtain multiple speech saliency values; The target audio stream is segmented into frames according to a preset second duration window to obtain a short frame sequence, which includes multiple short frames. For each of the multiple short-time frames in a short-time frame sequence, perform the following operation: Speech activity scores are determined based on short time frames; Summarize the voice activity scores to obtain multiple voice activity scores; Multiple initial speech segments were identified based on the original audio stream, long-time frame sequence, short-time frame sequence, multiple speech saliency, and multiple speech activity scores. For each of the multiple initial speech segments, perform the following operation: The initial speech segment is enhanced with human voice to obtain the enhanced speech segment; The enhanced speech segment is smoothed to obtain a clean speech segment; By summing up the clean speech segments, multiple clean speech segments are obtained; A clean human voice segment was identified based on multiple clean speech segments.

[0031] It should be explained that the resampling process of the original audio stream refers to resampling the original audio stream, and the method for resampling the original audio stream is existing technology, which will not be described in detail here. The resampled audio stream refers to the original audio stream after resampling. The emphasis process of the resampled audio stream refers to pre-emphasizing the resampled audio stream, and the method for pre-emphasizing the resampled audio stream is existing technology, which will not be described in detail here. The target audio stream refers to the resampled audio stream after pre-emphasizing. Optionally, the first duration window is 500ms, and the second duration window is 100ms.

[0032] For example, if the first duration window is 500ms, the second duration window is 100ms, and the duration of the target audio stream is 1000ms, then after framing the target audio stream according to the first duration window, the resulting long frames are: 0-500ms, 500-1000ms, and the resulting long frame sequence is: 0-500ms, 500-1000ms. After framing the target audio stream according to the second duration window, the resulting short frames are: 0-100ms, 100-200ms, 200-300ms, ..., 800-900ms, 900-1000ms, and the resulting short frame sequence is: 0-100ms, 100-200ms, 200-300ms, ..., 800-900ms, 900-1000ms.

[0033] It is understood that the "voice enhancement" of the initial speech segment refers to the automatic noise suppression of the initial speech segment, and the method for automatic noise suppression of the initial speech segment is existing technology and will not be elaborated here. The "enhanced speech segment" refers to the initial speech segment after voice enhancement. The "smoothing" of the enhanced speech segment refers to the fade-in / fade-out processing of the enhanced speech segment, and the method for fade-in / fade-out processing of the enhanced speech segment is existing technology and will not be elaborated here. The "clean speech segment" refers to the enhanced speech segment after smoothing. The "clean voice segment" refers to the audio obtained by splicing multiple clean speech segments in the original audio stream according to their appearance time in the original audio stream from earliest to latest.

[0034] Specifically, the energy decomposition of long-time frames to obtain the dominant speech frequency energy and non-speech energy includes: The long-time frame is subjected to time-frequency variation to obtain the long-time frame spectrum, which includes multiple frequency points; The power spectrum sequence was determined based on multiple frequency points, wherein the power spectrum sequence includes multiple power spectral densities; The power spectrum sequence is divided into sub-bands according to the preset human voice dominant frequency range to obtain the dominant frequency sub-band power sequence and the non-dominant frequency sub-band power sequence. The fundamental frequency is detected by performing fundamental frequency detection on the power sequence of the main frequency sub-band to obtain the long-time frame fundamental frequency estimate and fundamental frequency confidence. The fundamental frequency confidence level is compared with a preset confidence threshold. If the fundamental frequency confidence level is greater than the confidence threshold, multiple harmonic frequency points are identified based on the long-time frame fundamental frequency estimate. Perform the following operation for each of the multiple harmonic frequency points: The peak harmonic power was determined based on the harmonic frequency, the preset harmonic bandwidth, and the main frequency sub-band power sequence. By summing the peak harmonic powers, multiple peak harmonic powers are obtained; The harmonic structure energy is determined based on the peak power of multiple harmonics, where the harmonic structure energy is the sum of the peak powers of multiple harmonics; Multiple harmonic peak powers are removed from the main frequency subband power sequence to obtain the removal sequence, which includes multiple background powers. The dominant frequency background energy is determined based on multiple background powers in the elimination sequence, where the dominant frequency background energy is the sum of multiple background powers in the elimination sequence; The speech frequency energy is calculated based on the harmonic structure energy, the dominant frequency background energy, and the preset harmonic gain coefficient. The calculation formula is as follows:

[0035] in, Indicates the main frequency energy of the voice signal. Represents the harmonic structure energy. Indicates the background energy of the main frequency. Indicates the harmonic gain coefficient; If the fundamental frequency confidence is less than or equal to the confidence threshold, the speech fundamental frequency energy is determined based on the fundamental frequency subband power sequence, where the speech fundamental frequency energy is the sum of the power of multiple fundamental frequency subbands in the fundamental frequency subband power sequence; The non-speech energy of the subband was identified based on the power sequence of the non-dominant frequency subband. The noise proportion factor is calculated based on the power spectrum sequence, using the following formula:

[0036] in, Indicates the noise proportion coefficient. This indicates the number of power spectral densities among multiple power spectral densities in a power spectrum sequence. Represents the first power spectral density among multiple power spectral densities in a power spectral sequence. A power spectral density, Represents the logarithmic function. Indicated by An exponential function with base π, where π is the base of the function. It is a natural constant; The converted non-speech energy is calculated based on the sub-band non-speech energy and the noise ratio coefficient, where the converted non-speech energy is the product of the sub-band non-speech energy and the noise ratio coefficient. The residual energy of the main frequency was determined based on the main frequency sub-band power sequence and the preset power lower limit. The non-speech energy is determined based on the calculated non-speech energy and the residual energy of the main frequency. The non-speech energy is the sum of the calculated non-speech energy and the residual energy of the main frequency.

[0037] It should be explained that the time-frequency transformation of the long-time frame refers to performing a standard time-frequency transformation on the long-time frame, and the method for performing the standard time-frequency transformation on the long-time frame is existing technology, which will not be elaborated here. The long-time frame spectrum refers to the spectrum obtained after performing the time-frequency transformation on the long-time frame, and the frequency point refers to multiple points in the long-time frame spectrum that represent specific frequency values. The process of determining the power spectrum sequence based on multiple frequency points means: first, for each of the multiple frequency points, calculate its power spectral density to obtain multiple power spectral densities corresponding to multiple frequency points; then, arrange the multiple power spectral densities in ascending order according to the magnitude of the corresponding frequency points to finally obtain the power spectrum sequence. The sub-band division of the power spectrum sequence according to the preset dominant frequency range of human voice refers to: comparing each power spectral density in the power spectrum sequence to see if it falls within the dominant frequency range of human voice. If the power spectral density falls within the dominant frequency range of human voice, it is taken as the dominant frequency sub-band power; otherwise, it is taken as the non-dominant frequency sub-band power. This results in multiple dominant frequency sub-band powers and multiple non-dominant frequency sub-band powers. The dominant frequency sub-band power sequence is obtained by arranging these multiple dominant frequency sub-band powers in ascending order of their corresponding frequencies, and the non-dominant frequency sub-band power sequence is obtained by arranging these multiple non-dominant frequency sub-band powers in ascending order of their corresponding frequencies. Based on common knowledge, the dominant frequency range of human voice is 1.2kHz–3.5kHz.

[0038] It is understood that the fundamental frequency detection of the main frequency sub-band power sequence refers to the process of extracting the fundamental frequency of the main frequency sub-band power sequence to obtain the fundamental frequency of the long-time frame and the confidence level of the fundamental frequency of the long-time frame. The fundamental frequency of the long-time frame is the estimated value of the fundamental frequency of the long-time frame, and the confidence level of the fundamental frequency of the long-time frame is the fundamental frequency confidence level. The method of extracting the fundamental frequency of the main frequency sub-band power sequence is existing technology and will not be described in detail here.

[0039] It should be understood that the confidence threshold is a value manually set by the animation production company's staff based on the minimum confidence value of the fundamental frequency of long-time frames containing audio content in history. For example, if the minimum confidence value of the fundamental frequency of long-time frames containing audio content in history is 0.7, then the confidence threshold is 0.7. A harmonic frequency point refers to a frequency point whose frequency is an integer multiple of the estimated fundamental frequency of the long-time frame. The determination of the harmonic peak power based on the harmonic frequency point, the preset harmonic bandwidth, and the power sequence of the main frequency sub-bands means: in the power sequence of the main frequency sub-bands, searching for the maximum main frequency sub-band power within the harmonic bandwidth, centered on the harmonic frequency point, and using the maximum main frequency sub-band power as the harmonic peak power of the harmonic frequency point. The harmonic bandwidth is a value manually set by the animation production company's staff based on 0.4 times the estimated fundamental frequency of the long-time frame. For example, if the estimated fundamental frequency of the long-time frame is 100Hz, then the harmonic bandwidth is 40Hz.

[0040] For example, if the main frequency sub-band power sequence is: 100, 150, 140, 230, 330, 220, 410, 110, and the multiple harmonic peak powers are: 150, 140, 230, 330, 220, 410, then after removing the multiple harmonic peak powers from the main frequency sub-band power sequence, the resulting removal sequence is: 100, 110.

[0041] It should be explained that background power is the power spectral density in the removed sequence. The harmonic gain coefficient is a coefficient used to adjust the contribution weight of harmonic structure energy in the speech dominant frequency energy. The harmonic gain coefficient ranges from 0 to 1 and is a monotonically increasing function of the fundamental frequency confidence. Speech dominant frequency energy is used to characterize the intensity of voiced components produced by periodic vibrations of the vocal cords within the dominant frequency range of the human voice in a long-time frame. The greater the speech dominant frequency energy, the greater the intensity of voiced components produced by periodic vibrations of the vocal cords within the dominant frequency range of the human voice in a long-time frame. The determination of sub-band non-speech energy based on the non-dominant frequency sub-band power sequence refers to summing multiple power spectral densities in the non-dominant frequency sub-band power sequence to obtain the sub-band non-speech energy. The noise proportion coefficient is a parameter used to quantify whether the spectral distribution of a frame of signal is flat like noise or has a clear peak-valley structure like speech. The closer the noise proportion coefficient is to 1, the flatter the spectrum, and the more like noise; the closer the noise proportion coefficient is to 0, the more concentrated the energy is in a few spectral peaks, and the more like speech. The determination of the residual power of the main frequency based on the main frequency sub-band power sequence and the preset power lower limit refers to summing the power spectral densities below the power lower limit in the main frequency sub-band power sequence to obtain the residual power of the main frequency. The power lower limit is a value set manually by the animation production company staff based on the mean of the lowest 10% to 20% quantile of the main frequency sub-band power sequence. For example, if the mean of the lowest 10% to 20% quantile of the main frequency sub-band power sequence is 20dB, then the power lower limit is 20dB.

[0042] Specifically, the process of determining the speech activity score based on short time frames includes: Zero-crossing rate detection is performed on short frames to obtain the zero-crossing rate value; Energy is calculated for short-time frames to obtain short-time energy values; The short-time spectrum is obtained by performing time-frequency transformation on the short-time frame; The spectral flatness value was determined based on the short-time spectrum. The zero-crossing rate, short-time energy, and spectral flatness values ​​are normalized to obtain normalized zero-crossing rate, normalized short-time energy, and normalized spectral flatness. The speech activity score is calculated based on the normalized zero-crossing rate, normalized short-time energy, and normalized spectral flatness, using the following formula:

[0043] in, Indicates the score for voice activity. Represents the normalized zero-crossing rate. Represents the normalized short-time energy. Indicates the normalized spectral flatness. , and These represent the preset first weighting coefficient, the preset second weighting coefficient, and the preset third weighting coefficient, respectively. Represents the natural constant.

[0044] It should be explained that the zero-crossing rate detection of the short-time frame refers to analyzing multiple sampling points in the short-time frame to obtain multiple sampling points where the signal amplitude changes from positive to negative or from negative to positive, and counting the number of such sampling points to obtain the zero-crossing rate value. The sampling point refers to the point corresponding to different times in the short-time frame, and the signal amplitude refers to the amplitude corresponding to the sampling point. The energy calculation of the short-time frame refers to squaring the signal amplitude of multiple sampling points in the short-time frame, and then calculating the arithmetic mean of the squares of the signal amplitudes of multiple sampling points. The arithmetic mean of the squares of the signal amplitudes of multiple sampling points is the short-time energy value. The time-frequency transformation of the short-time frame refers to performing a standard time-frequency transformation on the short-time frame. The method of performing the standard time-frequency transformation on the short-time frame is existing technology and will not be elaborated here. The short-time spectrum refers to the spectrum obtained after performing the time-frequency transformation on the short-time frame. The determination of spectral flatness based on short-time spectrum means: firstly, calculating the geometric mean and arithmetic mean of multiple short-time power values ​​from the short-time spectrum; the spectral flatness value is equal to the geometric mean of the multiple short-time power values ​​divided by the arithmetic mean of the multiple short-time power values. The short-time power value refers to the power value corresponding to a frequency point in the short-time spectrum.

[0045] It is understood that the normalization of the zero-crossing rate, short-time energy, and spectral flatness values ​​refers to normalizing these values ​​using a normalization method (e.g., extreme value normalization). This method is existing technology and will not be elaborated further here. The normalized zero-crossing rate, normalized short-time energy, and normalized spectral flatness are the normalized values.

[0046] It should be understood that the speech activity score reflects the likelihood that a short frame contains a speech signal. The higher the speech activity score, the higher the likelihood that the short frame contains a speech signal, and the more likely the short frame is to be a speech frame. The first, second, and third weighting coefficients are all values ​​manually set by the animation production company's staff. Optionally, the first weighting coefficient is 0.3, the second weighting coefficient is 0.3, and the third weighting coefficient is 0.4.

[0047] Specifically, the process of identifying multiple initial speech segments based on the original audio stream, long-time frame sequences, short-time frame sequences, multiple speech saliencies, and multiple speech activity scores includes: For each of the multiple short-time frames in a short-time frame sequence, perform the following operation: The center time and target speech activity score were determined based on short time frames and multiple speech activity scores. The corresponding long time frame was identified based on the center time and the long time frame sequence. The target speech saliency was determined based on the corresponding long time frame and multiple speech saliency values. The saliency of the target speech is compared with a preset saliency threshold. If the saliency of the target speech is less than the saliency threshold, the short frame is treated as a non-speech frame. If the saliency of the target speech is greater than or equal to the saliency threshold, then compare the target speech activity score with the preset first threshold. If the target speech activity score is greater than or equal to the first threshold, then the short frame is taken as the speech frame; If the target speech activity score is less than the first threshold, then compare the target speech activity score with the preset second threshold; If the target speech activity score is less than the second threshold, the short frame is treated as a non-speech frame. If the target speech activity score is greater than or equal to the second threshold, then the short frame is designated as a pending frame. Speech frames, non-speech frames, and frames to be determined are summarized separately to obtain multiple speech frames, multiple non-speech frames, and multiple frames to be determined. For each of the multiple undetermined frames, perform the following operation: The annotation results of the undetermined frames are confirmed based on the undetermined frames, multiple speech frames, and multiple non-speech frames. The standard result of the undetermined frames is either a speech frame or a non-speech frame. The annotation results of the undetermined frames are summarized to obtain multiple annotation results of the undetermined frames; Based on the original audio stream, the annotation results of multiple undetermined frames, multiple speech frames, multiple non-speech frames, and multiple undetermined frames, multiple initial speech segments were identified.

[0048] It should be explained that the center moment refers to the moment corresponding to the center point of the time interval occupied by the short-time frame. The time interval occupied by the short-time frame refers to the time span from the beginning to the end of the short-time frame. For example, if the time interval occupied by the short-time frame is 300-400ms, then the center moment is 350ms. The target speech activity score refers to the speech activity score corresponding to the short-time frame among multiple speech activity scores. The corresponding long-time frame refers to the long-time frame in the long-time frame sequence whose time interval includes the center moment. The time interval occupied by the long-time frame refers to the time span from the beginning to the end of the long-time frame. The target speech saliency refers to the speech saliency that corresponds to the corresponding long-time frame among multiple speech saliencies.

[0049] It should be understood that the saliency threshold is manually set by the animation production company's staff based on the minimum saliency value of multiple historical speech frames. For example, if the minimum saliency value of multiple historical speech frames is 5, then the saliency threshold is 5. Non-speech frames refer to short-time frames where the target speech saliency is less than the saliency threshold, or short-time frames where the target speech saliency is greater than or equal to the saliency threshold and the target speech activity score is less than the second threshold. Speech frames refer to short-time frames where the target speech saliency is greater than or equal to the saliency threshold and the target speech activity score is greater than or equal to the first threshold. The first threshold is a value manually set by the animation production company's staff based on the minimum target speech activity score of multiple historical speech frames. For example, if the minimum target speech activity score of multiple historical speech frames is 85, then the first threshold is 85. The second threshold is a value manually set by the animation production company's staff based on the maximum target speech activity score of multiple historical non-speech frames. For example, if the maximum target speech activity score of multiple historical non-speech frames is 30, then the second threshold is 30. A pending frame is a short frame in which the saliency of the target speech is greater than or equal to the saliency threshold, the activity score of the target speech is greater than or equal to the second threshold, and the activity score of the target speech is less than the first threshold.

[0050] Understandably, the process of identifying the annotation results of the undetermined frame based on the undetermined frame, multiple speech frames, and multiple non-speech frames refers to the following: First, from multiple speech frames and multiple non-speech frames, the left and right adjacent frames adjacent to the undetermined frame are selected. The left adjacent frame refers to a speech frame or non-speech frame located to the left of the undetermined frame, and the right adjacent frame refers to a speech frame or non-speech frame located to the right of the undetermined frame. If both the left and right adjacent frames are non-speech frames, then the non-speech frame is used as the annotation result for the undetermined frame; otherwise, the speech frame is used as the annotation result for the undetermined frame. The process of identifying multiple initial speech segments based on the original audio stream, the annotation results of multiple undetermined frames, multiple speech frames, multiple non-speech frames, and multiple undetermined frames refers to the following: First, multiple undetermined frames are annotated according to the annotation results of multiple undetermined frames. Then, the annotated multiple undetermined frames are merged with multiple speech frames and multiple non-speech frames respectively, resulting in multiple final speech frames and multiple final non-speech frames. Finally, the original audio stream is segmented according to the multiple final speech frames and multiple final non-speech frames, and adjacent final speech frames are merged, resulting in multiple initial speech segments. For example, if the multiple final speech frames in the original audio stream are: V1, V2, V3, V4, V5, V6, V7, V8, and the multiple final non-speech frames are: B1, B2, B3, B4, and the order of the multiple final speech and multiple final non-speech frames in the original audio stream is: V1, V2, B1, V3, V4, V5, V6, B2, B3, V7, B4, V8, then the multiple initial speech segments are: (V1, V2), (V3, V4, V5, V6), (V7), (V8).

[0051] S3. Speaker separation is performed on the pure human voice segment to obtain multiple speech segment sets. For each speech segment set, the following operation is performed: Based on the speech segment set and the animation to be translated, the lip-sync video frame sequence is confirmed, wherein the lip-sync video frame sequence includes: multiple lip-sync video frames.

[0052] It should be explained that the lip-sync video frame sequence confirmed based on the audio segment set and the animation to be translated means: extracting multiple characters' lip-sync video frames from multiple audio segments in the audio segment set of the animation to be translated, and then arranging the multiple characters' lip-sync video frames in order from front to back according to the extraction time to finally obtain the lip-sync video frame sequence.

[0053] In detail, the speaker separation of the pure human voice segment yields multiple speech fragment sets, including: The pure human voice segment is overlapped and segmented according to the preset segment length and preset sliding step size to obtain multiple speech analysis segments; For each of the multiple speech analysis segments, perform the following operation: Acoustic features are extracted from the speech analysis segment to obtain the segment's acoustic features; The acoustic features of the segments are input into a pre-constructed speaker embedding network to obtain speaker embedding vectors; Summarize the speaker embedding vectors to obtain multiple speaker embedding vectors; Extracting the first from multiple speaker embedding vectors Each speaker embedding vector, where... The initial value is 1; Extracting the first from multiple speaker embedding vectors Each speaker embedding vector, where... The initial value is 1; Based on the The speaker embedding vector and the first speaker embedding vector The speaker embedding vector confirms the first speaker embedding vector. The similarity of n targets, where the nth target is... The similarity of the nth target is... The speaker embedding vector and the first speaker embedding vector Cosine similarity of speaker embedding vectors; The first Each target similarity is placed into a pre-constructed similarity set to obtain an updated similarity set; make ,Will As The updated similarity set is used as the new similarity set, and the result extracted from the embedding vectors of multiple speakers is returned. The steps of embedding speaker vectors until... The updated similarity set is used as the first target set, where... The number of speaker embedding vectors in a set of multiple speaker embedding vectors; make ,Will As Using the first target set as the similarity set, the method returns the first similarity set extracted from multiple speaker embedding vectors. The steps of embedding speaker vectors until... The first target set is used as the target similarity set; Multiple groups of speaking humans were identified based on the target similarity set; Perform the following operation for each of the multiple speaking human clusters: A set of speech segments was identified based on the speaking human cluster and multiple speech analysis segments; By aggregating the audio segments, multiple audio segment sets are obtained.

[0054] For example, if the duration of a pure human voice segment is 15 seconds, the preset segment duration is 5 seconds, and the preset sliding step size is 2 seconds, then after overlapping and segmenting the pure human voice segment according to the segment duration and sliding step size, the resulting multiple speech analysis segments are: 0-5 seconds, 2-7 seconds, 4-9 seconds, 6-11 seconds, 8-13 seconds, and 10-15 seconds.

[0055] It should be explained that the speaker embedding network is a deep neural network based on convolutional neural networks. The speaker embedding vector is the vector obtained by the speaker embedding network after analyzing the acoustic features of the segment. The method of inputting the acoustic features of the segment into the pre-constructed speaker embedding network to obtain the speaker embedding vector is existing technology and will not be elaborated here. The acoustic feature extraction of the speech analysis segment refers to extracting multiple Mel-frequency cepstral coefficients of the speech analysis segment through an acoustic feature extraction method (e.g., Discrete Fourier Transform). The method of extracting multiple Mel-frequency cepstral coefficients of the speech analysis segment through an acoustic feature extraction method (e.g., Discrete Fourier Transform) is existing technology and will not be elaborated here. The multiple Mel-frequency cepstral coefficients of the speech analysis segment are the acoustic features of the segment.

[0056] For example, if multiple speaker embedding vectors are p1, p2, p3, and p4, then the second speaker embedding vector extracted from the multiple speaker embedding vectors is p2.

[0057] It should be understood that the extraction of the first speaker from multiple speaker embedding vectors... The method for extracting the first speaker embedding vector from multiple speaker embedding vectors is similar to the method for extracting the first speaker embedding vector from multiple speaker embedding vectors. The method for embedding vectors for each speaker is the same, and will not be repeated here.

[0058] It should be explained that the initial state of the similarity set is an empty set. Updating the similarity set means adding the first empty set to the set. The similarity set of each target. The first target set refers to... The updated similarity set. The target similarity set refers to... The first target set at that time. The determination of the speech segment set based on the human speaker cluster and multiple speech analysis segments means: firstly, extracting multiple speech analysis segments corresponding to the human speaker cluster from multiple speech analysis segments; then, merging adjacent speech analysis segments in the multiple speech analysis segments to finally obtain multiple speech segments, and the speech segment set is the set composed of the multiple speech segments.

[0059] Specifically, the identification of multiple speaking human clusters based on the target similarity set includes: The similarity matrix is ​​determined based on the target similarity set; The similarity matrix is ​​symmetrically normalized to obtain the normalized similarity matrix; Construct the Laplacian matrix based on the normalized similarity matrix; The Laplacian matrix is ​​decomposed to obtain multiple eigenvalues ​​and multiple eigenvectors; The eigenvector matrix is ​​determined based on multiple eigenvalues ​​and multiple eigenvectors; The eigenvector matrix is ​​normalized row by row to obtain the normalized eigenvector matrix; After identifying multiple candidate cluster numbers, perform the following operation for each of these candidate cluster numbers: Extract the first eigenvector from the normalized eigenvector matrix. The columns are used to obtain the candidate eigenvalue submatrix, where, Equal to the number of candidate clusters; Cluster the candidate feature submatrices to obtain the clustering results; Calculate the clustering coefficient based on the clustering results; Summarize the clustering coefficients to obtain multiple clustering coefficients; The largest clustering coefficient was determined based on multiple clustering coefficients, where the largest clustering coefficient is the largest among the multiple clustering coefficients; The optimal number of speaker categories was determined based on the maximum clustering coefficient and the number of multiple candidate clusters. Multiple speaker clusters were identified based on the optimal number of speaker categories and the normalized feature vector matrix.

[0060] It should be explained that the similarity matrix is ​​a... OK A matrix of columns, where, The number of speaker embedding vectors in a plurality of speaker embedding vectors, and the number of the similarity vectors in the matrix. Line 1 The elements of the column correspond to the first element in the target similarity set. The similarity of the targets. The symmetric normalization of the similarity matrix refers to the process of normalizing the similarity matrix, and the method of normalizing the similarity matrix is ​​existing technology, which will not be described in detail here. The normalized similarity matrix refers to the similarity matrix after symmetric normalization.

[0061] Understandably, constructing a Laplacian matrix based on a normalized similarity matrix means: first, summing each row of the normalized similarity matrix to obtain the degree value corresponding to each row; then, constructing a degree matrix based on the degree values ​​corresponding to each row, where the degree matrix is ​​a diagonal matrix, and the first element on the diagonal of the degree matrix... The element is the first The degree values ​​corresponding to each row are all zero for off-diagonal elements. Next, the degree matrix is ​​inverted to obtain the inverse degree matrix. The method for inverting the degree matrix is ​​existing technology and will not be elaborated here. Then, the square root of the inverse degree matrix is ​​performed to obtain the inverse square root of the degree matrix. Finally, the Laplacian matrix is ​​calculated based on the degree matrix, the inverse square root of the degree matrix, and the normalized similarity matrix. The calculation formula is shown below:

[0062] in, Represents the Laplace matrix, Represents the identity matrix. Degree matrix, Represents the inverse square root of the degree matrix. Represents the normalized similarity matrix. This represents matrix multiplication.

[0063] It should be understood that the decomposition of the Laplace matrix refers to the eigenvalue decomposition of the Laplace matrix, and the method for eigenvalue decomposition is existing technology and will not be elaborated here. Eigenvalues ​​are the eigenvalues ​​obtained after eigenvalue decomposition of the Laplace matrix, and eigenvectors are the eigenvectors obtained after eigenvalue decomposition of the Laplace matrix. The eigenvector matrix is ​​a matrix composed of multiple eigenvalues ​​and multiple eigenvectors. The row-by-row normalization of the eigenvector matrix refers to normalizing each row of the eigenvector matrix. The normalized eigenvector matrix is ​​the eigenvector matrix after row-by-row normalization. The number of candidate clusters is a set value manually determined by the animation production company staff by adding or subtracting 3 from the number of characters in the animation to be translated. For example, if the number of characters in the animation to be translated is 5, then the number of candidate clusters is: 2, 3, 4, 5, 6, 7, 8.

[0064] It should be explained that the candidate eigenvector matrix refers to the matrix of top-ranking eigenvectors extracted from the normalized eigenvector matrix. The matrix formed by the columns. Clustering the candidate feature submatrix refers to using a clustering method (e.g., K-means clustering algorithm) to perform cluster analysis on the candidate feature submatrix, thereby obtaining multiple clusters. This method of using a clustering method (e.g., K-means clustering algorithm) to perform cluster analysis on the candidate feature submatrix to obtain multiple clusters is existing technology and will not be elaborated here. The multiple clusters are the clustering results. The clustering coefficient refers to the silhouette coefficient of multiple clusters in the clustering results. The optimal number of speaker categories refers to the number of candidate clusters corresponding to the number of multiple candidate clusters and the maximum clustering coefficient.

[0065] It is understood that the identification of multiple speaker clusters based on the optimal number of speaker categories and the normalized feature vector matrix refers to: using a clustering method (e.g., K-means clustering algorithm) to perform cluster analysis on the normalized feature vector matrix according to the optimal number of speaker categories, thereby obtaining multiple clusters. The method of using a clustering method (e.g., K-means clustering algorithm) to perform cluster analysis on the normalized feature vector matrix according to the optimal number of speaker categories to obtain multiple clusters is existing technology and will not be elaborated here. The multiple clusters are the multiple speaker clusters.

[0066] S4. Perform acoustic feature extraction on the speech segment set to obtain an acoustic feature sequence, which includes multiple Mel frequency cepstral coefficient sets. Perform sentiment analysis on the acoustic feature sequence to obtain the sentiment category. Based on the multiple Mel frequency cepstral coefficient sets in the acoustic feature sequence, identify the prosodic feature parameters.

[0067] It should be understood that the method for extracting acoustic features from a set of speech segments to obtain an acoustic feature sequence, wherein the acoustic feature sequence includes multiple sets of Mel-frequency cepstral coefficients, is the same as the method for extracting acoustic features from a speech segment to obtain segment acoustic features, and will not be repeated here. The acoustic feature sequence refers to the sequence obtained by arranging multiple sets of Mel-frequency cepstral coefficients extracted from the set of speech segments in order of extraction time from front to back. The set of Mel-frequency cepstral coefficients is a collection of multiple Mel-frequency cepstral coefficients extracted from a speech segment in the set of speech segments.

[0068] It should be explained that the emotion analysis of the acoustic feature sequence refers to inputting the acoustic feature sequence into a pre-built emotion recognition model to obtain the emotion category corresponding to the acoustic feature sequence. The emotion category corresponding to the acoustic feature sequence is the emotion category. Optionally, a convolutional neural network can be used as the emotion recognition model.

[0069] Specifically, the prosodic feature parameters identified based on multiple Mel-frequency cepstral coefficient sets in the acoustic feature sequence include: For each Mel frequency cepstral coefficient set in multiple Mel frequency cepstral coefficient sets in the acoustic feature sequence, the following operation is performed: Extract the zeroth-order Mel frequency cepstral coefficients from the Mel frequency cepstral coefficient set; The zeroth-order Mel frequency cepstral coefficients are used as the frame energy value; Summarize the frame energy values ​​to obtain multiple frame energy values; The average energy value is determined based on the energy values ​​of multiple frames, where the average energy value is the average of the energy values ​​of multiple frames. Prosodic feature parameters are calculated based on the energy values ​​of multiple frames and the average energy value. The calculation formula is as follows:

[0070] in, Represents prosodic feature parameters, Represents the energy value of multiple frames. Energy value per frame Represents the energy value of multiple frames. Energy value per frame Indicates the average energy. This indicates the number of frame energy values ​​among multiple frame energy values. and These represent the preset first prosodic weight coefficient and the preset second prosodic weight coefficient, respectively. This indicates taking the absolute value.

[0071] It should be explained that the zero-order Mel frequency cepstral coefficient refers to the coefficient with an index of 0 in the Mel frequency cepstral coefficient set. The index refers to the sequential number assigned to each Mel frequency cepstral coefficient after sorting them according to their corresponding cepstral order from smallest to largest. Prosodic feature parameters reflect the degree of energy fluctuation and the rate of energy change of a speech segment in the time dimension. The larger the prosodic feature parameter, the greater the degree of energy fluctuation and the rate of energy change of the speech segment in the time dimension. The first prosodic weight coefficient and the second prosodic weight coefficient are values ​​manually set by the animation production company staff; optionally, the first prosodic weight coefficient is 0.6 and the second prosodic weight coefficient is 0.4.

[0072] S5. The prosodic feature parameters and the emotion category are jointly encoded to obtain the character's voice style vector. The visual temporal features are identified based on multiple lip-sync video frames in the lip-sync video frame sequence.

[0073] It should be explained that the joint encoding of prosodic feature parameters and emotion categories to obtain a character voice style vector means that the prosodic feature parameters and emotion categories are combined to obtain a vector that represents the character voice style. The vector that represents the character voice style is the character voice style vector.

[0074] Specifically, the visual temporal features identified based on multiple lip-sync video frames in the lip-sync video frame sequence include: For each lip-sync video frame in a sequence of multiple lip-sync video frames, the following operation is performed: Face region detection is performed on lip-sync video frames to obtain face region images; Lip key point detection was performed on the face region image to obtain multiple inner lip key points; The vertical opening distance and horizontal width of the inner lip were determined based on multiple key points of the inner lip. The mouth opening degree is determined based on the vertical opening distance of the inner lip and the horizontal width of the inner lip. The mouth opening degree is equal to the vertical opening distance of the inner lip divided by the horizontal width of the inner lip. By connecting multiple key points of the inner lip in sequence, an inner lip contour polygon is obtained. The inner lip contour polygon includes multiple vertices, where each vertex has a vertex x-coordinate and a vertex y-coordinate. The area of ​​the inner lip contour is calculated based on the polygonal shape of the inner lip, using the following formula:

[0075] in, Indicates the area of ​​the inner lip contour. This indicates the number of vertices in the polygon representing the inner lip contour. Represents the first vertex among multiple vertices of the polygon representing the inner lip contour. The x-coordinate of each vertex. Represents the first vertex among multiple vertices of the polygon representing the inner lip contour. The y-coordinate of each vertex. Represents the first vertex among multiple vertices of the polygon representing the inner lip contour. The x-coordinate of each vertex. Represents the first vertex among multiple vertices of the polygon representing the inner lip contour. The y-coordinate of each vertex. Indicates taking the absolute value; The mouth opening degree is combined with the inner lip contour area to obtain the mouth shape geometric features; By summarizing the lip shape geometric features, multiple lip shape geometric features are obtained; Multiple lip shape geometric features are temporally encoded to obtain visual temporal features.

[0076] It should be explained that the face region detection in the lip-sync video frame refers to using a face detection algorithm (e.g., a face recognition algorithm) to identify the image of the region containing the face in the lip-sync video frame. The method for using a face detection algorithm (e.g., a face recognition algorithm) to identify the image of the region containing the face in the lip-sync video frame is existing technology and will not be elaborated here. The image of the region containing the face is the face region image. Lip keypoint detection in the face region image refers to using a keypoint detection model to detect the key parts of the lip shape in the face region image. The method for using a keypoint detection model to detect the key parts of the lip shape in the face region image is existing technology and will not be elaborated here. Optionally, the OpenPose model can be used as the keypoint detection model. Inner lip keypoints refer to the points inside the lips obtained after performing lip keypoint detection on the face region image.

[0077] Understandably, determining the vertical opening distance and horizontal width of the inner lip based on multiple key points of the inner lip means: first, identifying the center point of the upper edge and the center point of the lower edge of the inner lip from multiple key points; then, calculating the vertical coordinate difference between the center points of the upper and lower edges of the inner lip, and using the absolute value of this vertical coordinate difference as the vertical opening distance of the inner lip; second, identifying the center point of the left and right edges of the inner lip from multiple key points; then, calculating the horizontal coordinate difference between the center points of the left and right edges of the inner lip, and using the absolute value of this horizontal coordinate difference as the horizontal width of the inner lip. The center point of the upper edge of the inner lip refers to the point located at the center of the upper edge of the inner lip among multiple key points; the center point of the lower edge of the inner lip refers to the point located at the center of the lower edge of the inner lip among multiple key points; the center point of the left edge of the inner lip refers to the point located at the center of the left edge of the inner lip among multiple key points; and the center point of the right edge of the inner lip refers to the point located at the center of the right edge of the inner lip among multiple key points.

[0078] It should be understood that the inner lip contour polygon refers to the polygon obtained by connecting multiple inner lip key points sequentially. A vertex refers to an inner lip key point, and the vertex includes: the x-coordinate (the horizontal coordinate of the inner lip key point) and the y-coordinate (the vertical coordinate of the inner lip key point). The inner lip contour area refers to the area of ​​the inner lip contour; the larger the inner lip contour area, the larger the overall area of ​​the inner lip contour. The temporal encoding of multiple lip shape geometric features refers to encoding multiple lip shape geometric features using a visual encoding method (e.g., a visual encoder). The method of encoding multiple lip shape geometric features using a visual encoding method (e.g., a visual encoder) is existing technology and will not be elaborated here. Visual temporal features refer to the result obtained after temporally encoding multiple lip shape geometric features.

[0079] S6. Based on acoustic feature sequences and visual temporal features, the audio-visual coupling features are identified, and based on the character's voice style vector and audio-visual coupling features, the character's voice feature profile is identified.

[0080] It should be explained that the identification of audio-visual coupling features based on acoustic feature sequences and visual temporal features means aligning the acoustic feature sequences and visual temporal features according to time to obtain the audio-visual coupling features. A character's voice feature profile refers to a user profile that represents the character's voice features, including the character's voice style vector and audio-visual coupling features.

[0081] S7. Summarize the voice feature profiles of the characters to obtain multiple voice feature profiles and complete the voice feature extraction.

[0082] For example, once multiple character voice feature profiles are obtained, Xiao Zhang can arrange for professionals to translate the animation based on the voice features in the multiple character voice feature profiles, thereby completing the voice feature extraction.

[0083] To address the problems described in the background art, this invention identifies the animation to be translated, extracts its audio stream, and obtains the original audio stream. This invention separates independent audio from the animation to be translated, obtaining complete original sound data. This provides basic audio material for subsequent voice detection and feature analysis. Furthermore, it performs speech activity detection on the original audio stream to obtain pure human voice segments. This invention detects the human voice activity intervals in the original audio, distinguishing human voices from non-human voice parts such as silence, background music, and ambient sounds, extracting pure human voice segments containing only dialogue, reducing interference from irrelevant signals. Speaker separation is then performed on the pure human voice segments to obtain multiple speech segment sets. This invention clusters the pure human voice segments according to different speakers based on features such as timbre and frequency. All speech segments of each speaker are grouped into one set, facilitating subsequent independent processing by role. For each of the multiple speech segment sets, the following operations are performed: Based on the speech segment set and the animation to be translated, an lip-sync video frame sequence is identified, wherein the lip-sync video frame sequence includes: multiple lip-sync video frames. It can be seen that in this embodiment of the invention, the speech segment corresponding to the speaker is aligned with the animation screen on the time axis, and the lip-sync screen frames corresponding to the speech segment are extracted from the video to form a lip-sync sequence for lip movement analysis. Acoustic features are extracted from the speech segment set to obtain an acoustic feature sequence, wherein the acoustic feature sequence includes: multiple Mel frequency cepstral coefficient sets. It can be seen that in this embodiment of the invention, acoustic parameters are extracted from the speaker's speech segment to form an acoustic feature sequence that changes over time. Sentiment analysis is performed on the acoustic feature sequence to obtain the sentiment category. It can be seen that in this embodiment of the invention, by analyzing the energy changes, intonation fluctuations, etc. in the acoustic features, the emotional attributes expressed by the speech segment are determined, providing a sentiment dimension for the style vector.Based on multiple Mel-frequency cepstral coefficient sets in the acoustic feature sequence, prosodic feature parameters are identified. This embodiment of the invention further extracts prosodic-related parameters such as speech rate, pitch change rate, and pause distribution from the acoustic features to reflect speaking rhythm and intonation patterns. The prosodic feature parameters are jointly encoded with emotion categories to obtain a character speech style vector. This embodiment of the invention combines prosodic information and emotion categories into a fixed-dimensional vector to express the speaker's unique speech expression style. Based on multiple lip-sync video frames in the lip-sync video frame sequence, visual temporal features are identified. This embodiment of the invention analyzes visual information such as lip opening and closing, and shape changes in the lip-sync frame sequence to extract contour features that change over time. Based on acoustic feature sequences and visual temporal features, the audio-visual coupling features are identified. This invention demonstrates that by performing temporal alignment and correlation analysis between acoustic features and lip-sync visual features, the synchronization relationship and matching degree between sound and image are found. Based on the character's voice style vector and audio-visual coupling features, a character's voice feature profile is identified. This invention demonstrates that by merging the voice style vector and audio-visual coupling features, a complete profile of the character's voice quality, rhythm, emotion, and lip-sync is constructed. By summarizing the character's voice feature profiles, multiple character voice feature profiles are obtained, completing the voice feature extraction. This invention demonstrates that by aggregating the feature profiles of all characters, a complete set of character voice features for the entire animation is formed. Therefore, this invention overcomes the shortcomings of traditional manual listening and imitation, which suffers from strong subjectivity and poor consistency, thereby achieving accurate and stable extraction of voice features in animated translation.

[0084] like Figure 2 The diagram shown is a functional block diagram of a speech feature extraction system for animation translation provided in an embodiment of the present invention.

[0085] The speech feature extraction system 100 for animation translation described in this invention can be installed in an electronic device. Depending on the functions implemented, the speech feature extraction system 100 for animation translation may include a speech segment acquisition module 101, an emotion type recognition module 102, a visual feature confirmation module 103, and a speech feature extraction module 104. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0086] The speech segment acquisition module 101 is used to identify the animation to be translated, extract the audio stream of the animation to be translated to obtain the original audio stream, perform speech activity detection on the original audio stream to obtain a clean human voice segment, and perform speaker separation on the clean human voice segment to obtain multiple speech segment sets. The emotion type recognition module 102 is used to perform the following operations on each of the multiple speech segment sets: based on the speech segment set and the animation to be translated, confirm the lip-sync video frame sequence, wherein the lip-sync video frame sequence includes: multiple lip-sync video frames; extract acoustic features from the speech segment set to obtain an acoustic feature sequence, wherein the acoustic feature sequence includes: multiple Mel frequency cepstral coefficient sets; and perform emotion analysis on the acoustic feature sequence to obtain the emotion category. The visual feature confirmation module 103 is used to confirm prosodic feature parameters based on multiple Mel frequency cepstral coefficient sets in the acoustic feature sequence, jointly encode the prosodic feature parameters with the emotion category to obtain the character voice style vector, and confirm visual temporal features based on multiple lip-sync video frames in the lip-sync video frame sequence. The speech feature extraction module 104 is used to identify audio-visual coupling features based on acoustic feature sequences and visual temporal features, identify character speech feature profiles based on character speech style vectors and audio-visual coupling features, summarize character speech feature profiles, obtain multiple character speech feature profiles, and complete speech feature extraction.

[0087] In detail, the modules in the speech feature extraction system 100 for animation translation described in this embodiment of the invention employ the same methods as described above. Figure 1 The technique used is the same as the speech feature extraction method for animation translation described in the previous section, and it can produce the same technical effect, so it will not be repeated here.

[0088] like Figure 3 The diagram shown is a schematic representation of an electronic device for implementing a speech feature extraction method for animation translation, according to an embodiment of the present invention.

[0089] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a speech feature extraction method program for animation translation.

[0090] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as the portable hard drive of the electronic device 1. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 includes both internal storage units and external storage devices of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a speech feature extraction method program for animation translation, but also to temporarily store data that has been output or will be output.

[0091] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., a speech feature extraction method program for animation translation) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0092] The bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 can be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0093] Figure 3 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 3The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0094] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0095] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0096] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0097] The speech feature extraction method program for animation translation stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can achieve the following: Once the animation to be translated is identified, its audio stream is extracted to obtain the original audio stream. Speech activity detection is performed on the original audio stream to obtain a clean human voice segment; Speaker separation is performed on the pure human voice segment to obtain multiple speech fragment sets; For each of the multiple speech segment sets, perform the following operation: Based on the set of speech segments and the animation to be translated, the lip-sync video frame sequence was identified, which includes: multiple lip-sync video frames; Acoustic features are extracted from the speech segment set to obtain an acoustic feature sequence, which includes multiple sets of Mel frequency cepstral coefficients. Sentiment analysis is performed on acoustic feature sequences to obtain sentiment categories; Prosodic feature parameters were identified based on multiple Mel frequency cepstral coefficient sets in the acoustic feature sequence; By jointly encoding prosodic feature parameters and emotion categories, a character's voice style vector is obtained; Visual temporal features were identified based on multiple lip-sync video frames in the lip-sync video frame sequence. The sound-image coupling features were identified based on acoustic feature sequences and visual temporal features. A character voice feature profile was identified based on the character's voice style vector and audio-visual coupling features. By summarizing the voice feature profiles of the characters, multiple voice feature profiles are obtained, and voice feature extraction is completed.

[0098] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figures 1 to 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0099] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0100] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following: Once the animation to be translated is identified, its audio stream is extracted to obtain the original audio stream. Speech activity detection is performed on the original audio stream to obtain a clean human voice segment; Speaker separation is performed on the pure human voice segment to obtain multiple speech fragment sets; For each of the multiple speech segment sets, perform the following operation: Based on the set of speech segments and the animation to be translated, the lip-sync video frame sequence was identified, which includes: multiple lip-sync video frames; Acoustic features are extracted from the speech segment set to obtain an acoustic feature sequence, which includes multiple sets of Mel frequency cepstral coefficients. Sentiment analysis is performed on acoustic feature sequences to obtain sentiment categories; Prosodic feature parameters were identified based on multiple Mel frequency cepstral coefficient sets in the acoustic feature sequence; By jointly encoding prosodic feature parameters and emotion categories, a character's voice style vector is obtained; Visual temporal features were identified based on multiple lip-sync video frames in the lip-sync video frame sequence. The sound-image coupling features were identified based on acoustic feature sequences and visual temporal features. A character voice feature profile was identified based on the character's voice style vector and audio-visual coupling features. By summarizing the voice feature profiles of the characters, multiple voice feature profiles are obtained, and voice feature extraction is completed.

[0101] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and actual implementations may have other classification methods.

[0102] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0104] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for extracting speech features for animation translation, characterized in that, The method includes: Once the animation to be translated is identified, its audio stream is extracted to obtain the original audio stream. Speech activity detection is performed on the original audio stream to obtain a clean human voice segment; Speaker separation is performed on the pure human voice segment to obtain multiple speech fragment sets; For each of the multiple speech segment sets, perform the following operation: Based on the set of speech segments and the animation to be translated, the lip-sync video frame sequence was identified, which includes: multiple lip-sync video frames; Acoustic features are extracted from the speech segment set to obtain an acoustic feature sequence, which includes multiple sets of Mel frequency cepstral coefficients. Sentiment analysis is performed on acoustic feature sequences to obtain sentiment categories; Prosodic feature parameters were identified based on multiple Mel frequency cepstral coefficient sets in the acoustic feature sequence; By jointly encoding prosodic feature parameters and emotion categories, a character's voice style vector is obtained; Visual temporal features were identified based on multiple lip-sync video frames in the lip-sync video frame sequence. The sound-image coupling features were identified based on acoustic feature sequences and visual temporal features. A character voice feature profile was identified based on the character's voice style vector and audio-visual coupling features. By summarizing the voice feature profiles of the characters, multiple voice feature profiles are obtained, and voice feature extraction is completed.

2. The speech feature extraction method for animation translation as described in claim 1, characterized in that, The process of detecting speech activity in the original audio stream to obtain a clean human voice segment includes: The original audio stream is resampled to obtain a resampled audio stream; The resampled audio stream is then emphasized to obtain the target audio stream; The target audio stream is segmented into frames according to a preset first duration window to obtain a long-time frame sequence, which includes multiple long-time frames. For each of the multiple long-time frames in a long-time frame sequence, perform the following operation: Energy decomposition is performed on long time frames to obtain the speech main frequency energy and non-speech energy; Speech saliency is calculated based on the dominant speech energy and non-speech energy, where speech saliency is equal to the dominant speech energy divided by the non-speech energy. Summarize the speech saliency to obtain multiple speech saliency values; The target audio stream is segmented into frames according to a preset second duration window to obtain a short frame sequence, which includes multiple short frames. For each of the multiple short-time frames in a short-time frame sequence, perform the following operation: Speech activity scores are determined based on short time frames; Summarize the voice activity scores to obtain multiple voice activity scores; Multiple initial speech segments were identified based on the original audio stream, long-time frame sequence, short-time frame sequence, multiple speech saliency, and multiple speech activity scores. For each of the multiple initial speech segments, perform the following operation: The initial speech segment is enhanced with human voice to obtain the enhanced speech segment; The enhanced speech segment is smoothed to obtain a clean speech segment; By summing up the clean speech segments, multiple clean speech segments are obtained; A clean human voice segment was identified based on multiple clean speech segments.

3. The speech feature extraction method for animation translation as described in claim 2, characterized in that, The energy decomposition of long-time frames to obtain speech frequency energy and non-speech energy includes: The long-time frame is subjected to time-frequency variation to obtain the long-time frame spectrum, which includes multiple frequency points; The power spectrum sequence was determined based on multiple frequency points, wherein the power spectrum sequence includes multiple power spectral densities; The power spectrum sequence is divided into sub-bands according to the preset human voice dominant frequency range to obtain the dominant frequency sub-band power sequence and the non-dominant frequency sub-band power sequence. The fundamental frequency is detected by performing fundamental frequency detection on the power sequence of the main frequency sub-band to obtain the long-time frame fundamental frequency estimate and fundamental frequency confidence. The fundamental frequency confidence level is compared with a preset confidence threshold. If the fundamental frequency confidence level is greater than the confidence threshold, multiple harmonic frequency points are identified based on the long-time frame fundamental frequency estimate. Perform the following operation for each of the multiple harmonic frequency points: The peak harmonic power was determined based on the harmonic frequency, the preset harmonic bandwidth, and the main frequency sub-band power sequence. By summing the peak harmonic powers, multiple peak harmonic powers are obtained; The harmonic structure energy is determined based on the peak power of multiple harmonics, where the harmonic structure energy is the sum of the peak powers of multiple harmonics; Multiple harmonic peak powers are removed from the main frequency subband power sequence to obtain the removal sequence, which includes multiple background powers. The dominant frequency background energy is determined based on multiple background powers in the elimination sequence, where the dominant frequency background energy is the sum of multiple background powers in the elimination sequence; The speech frequency energy is calculated based on the harmonic structure energy, the dominant frequency background energy, and the preset harmonic gain coefficient. If the fundamental frequency confidence is less than or equal to the confidence threshold, the speech fundamental frequency energy is determined based on the fundamental frequency subband power sequence, where the speech fundamental frequency energy is the sum of the power of multiple fundamental frequency subbands in the fundamental frequency subband power sequence; The non-speech energy of the subband was identified based on the power sequence of the non-dominant frequency subband. The noise proportion factor is calculated based on the power spectrum sequence, using the following formula: ； in, Indicates the noise proportion coefficient. This indicates the number of power spectral densities among multiple power spectral densities in a power spectrum sequence. Represents the first power spectral density among multiple power spectral densities in a power spectral sequence. A power spectral density, Represents the logarithmic function. Indicates An exponential function with base , where It is a natural constant; The converted non-speech energy is calculated based on the sub-band non-speech energy and the noise ratio coefficient, where the converted non-speech energy is the product of the sub-band non-speech energy and the noise ratio coefficient. The residual energy of the main frequency was determined based on the main frequency sub-band power sequence and the preset power lower limit. The non-speech energy is determined based on the calculated non-speech energy and the residual energy of the main frequency. The non-speech energy is the sum of the calculated non-speech energy and the residual energy of the main frequency.

4. The speech feature extraction method for animation translation as described in claim 3, characterized in that, The process of determining the speech activity score based on short time frames includes: Zero-crossing rate detection is performed on short frames to obtain the zero-crossing rate value; Energy is calculated for short-time frames to obtain short-time energy values; The short-time spectrum is obtained by performing time-frequency transformation on the short-time frame; The spectral flatness value was determined based on the short-time spectrum. The zero-crossing rate, short-time energy, and spectral flatness values ​​are normalized to obtain normalized zero-crossing rate, normalized short-time energy, and normalized spectral flatness. Speech activity scores are calculated based on normalized zero-crossing rate, normalized short-time energy, and normalized spectral flatness.

5. The speech feature extraction method for animation translation as described in claim 4, characterized in that, The process involves identifying multiple initial speech segments based on the original audio stream, long-time frame sequences, short-time frame sequences, multiple speech saliency scores, and multiple speech activity scores, including: For each of the multiple short-time frames in a short-time frame sequence, perform the following operation: The center time and target speech activity score were determined based on short time frames and multiple speech activity scores. The corresponding long-time frame was identified based on the center time and the long-time frame sequence. The target speech saliency was determined based on the corresponding long time frame and multiple speech saliency values. The saliency of the target speech is compared with a preset saliency threshold. If the saliency of the target speech is less than the saliency threshold, the short frame is treated as a non-speech frame. If the saliency of the target speech is greater than or equal to the saliency threshold, then compare the target speech activity score with the preset first threshold. If the target speech activity score is greater than or equal to the first threshold, then the short frame is taken as the speech frame; If the target speech activity score is less than the first threshold, then compare the target speech activity score with the preset second threshold; If the target speech activity score is less than the second threshold, the short frame is treated as a non-speech frame. If the target speech activity score is greater than or equal to the second threshold, then the short frame is designated as a pending frame. Speech frames, non-speech frames, and frames to be determined are summarized separately to obtain multiple speech frames, multiple non-speech frames, and multiple frames to be determined. For each of the multiple undetermined frames, perform the following operation: The annotation results of the undetermined frames are confirmed based on the undetermined frames, multiple speech frames, and multiple non-speech frames. The standard result of the undetermined frames is either a speech frame or a non-speech frame. The annotation results of the undetermined frames are summarized to obtain multiple annotation results of the undetermined frames; Based on the original audio stream, the annotation results of multiple undetermined frames, multiple speech frames, multiple non-speech frames, and multiple undetermined frames, multiple initial speech segments were identified.

6. The speech feature extraction method for animation translation as described in claim 5, characterized in that, The speaker separation of the pure human voice segment yields multiple speech fragment sets, including: The pure human voice segment is overlapped and segmented according to the preset segment length and preset sliding step size to obtain multiple speech analysis segments; For each of the multiple speech analysis segments, perform the following operation: Acoustic features are extracted from the speech analysis segment to obtain the segment's acoustic features; The acoustic features of the segments are input into a pre-constructed speaker embedding network to obtain speaker embedding vectors; Summarize the speaker embedding vectors to obtain multiple speaker embedding vectors; Extracting the first from multiple speaker embedding vectors Each speaker embedding vector, where... The initial value is 1; Extracting the first from multiple speaker embedding vectors Each speaker embedding vector, where... The initial value is 1; Based on the The speaker embedding vector and the first speaker embedding vector The speaker embedding vector confirms the first speaker embedding vector. The similarity of n targets, where the nth target is... The similarity of the target is the embedding vector of the i-th speaker and the i-th speaker. Cosine similarity of speaker embedding vectors; The first Each target similarity is placed into a pre-constructed similarity set to obtain an updated similarity set; make ,Will As The updated similarity set is used as the new similarity set, and the result extracted from the embedding vectors of multiple speakers is returned. The steps of embedding speaker vectors until... The updated similarity set is used as the first target set, where... The number of speaker embedding vectors in a set of multiple speaker embedding vectors; make ,Will As Using the first target set as the similarity set, the method returns the first similarity set extracted from multiple speaker embedding vectors. The steps of embedding speaker vectors until... The first target set is used as the target similarity set; Multiple groups of speaking humans were identified based on the target similarity set; Perform the following operation for each of the multiple speaking human clusters: A set of speech segments was identified based on the speaking human cluster and multiple speech analysis segments; By aggregating the audio segments, multiple audio segment sets are obtained.

7. The speech feature extraction method for animation translation as described in claim 6, characterized in that, The identification of multiple human speaking clusters based on the target similarity set includes: The similarity matrix is ​​determined based on the target similarity set; The similarity matrix is ​​symmetrically normalized to obtain the normalized similarity matrix; Construct the Laplacian matrix based on the normalized similarity matrix; The Laplacian matrix is ​​decomposed to obtain multiple eigenvalues ​​and multiple eigenvectors; The eigenvector matrix is ​​determined based on multiple eigenvalues ​​and multiple eigenvectors; The eigenvector matrix is ​​normalized row by row to obtain the normalized eigenvector matrix; After identifying multiple candidate cluster numbers, perform the following operation for each of these candidate cluster numbers: Extract the first eigenvector from the normalized eigenvector matrix. The columns are used to obtain the candidate eigenvalue submatrix, where, Equal to the number of candidate clusters; Cluster the candidate feature submatrices to obtain the clustering results; Calculate the clustering coefficient based on the clustering results; Summarize the clustering coefficients to obtain multiple clustering coefficients; The largest clustering coefficient was determined based on multiple clustering coefficients, where the largest clustering coefficient is the largest among the multiple clustering coefficients; The optimal number of speaker categories was determined based on the maximum clustering coefficient and the number of multiple candidate clusters. Multiple speaker clusters were identified based on the optimal number of speaker categories and the normalized feature vector matrix.

8. The speech feature extraction method for animation translation as described in claim 7, characterized in that, The prosodic feature parameters identified based on multiple Mel-frequency cepstral coefficient sets in the acoustic feature sequence include: For each Mel frequency cepstral coefficient set in multiple Mel frequency cepstral coefficient sets in the acoustic feature sequence, the following operation is performed: Extract the zeroth-order Mel frequency cepstral coefficients from the Mel frequency cepstral coefficient set; The zeroth-order Mel frequency cepstral coefficients are used as the frame energy value; Summarize the frame energy values ​​to obtain multiple frame energy values; The average energy value is determined based on the energy values ​​of multiple frames, where the average energy value is the average of the energy values ​​of multiple frames. Prosodic feature parameters are calculated based on the energy values ​​of multiple frames and the average energy value. The calculation formula is as follows: ； in, Represents prosodic feature parameters, Represents the energy value of multiple frames. Energy value per frame Represents the energy value of multiple frames. Energy value per frame Indicates the average energy. This indicates the number of frame energy values ​​among multiple frame energy values. and These represent the preset first prosodic weight coefficient and the preset second prosodic weight coefficient, respectively. This indicates taking the absolute value.

9. The speech feature extraction method for animation translation as described in claim 8, characterized in that, The visual temporal features identified based on multiple lip-sync video frames in the lip-sync video frame sequence include: For each lip-sync video frame in a sequence of multiple lip-sync video frames, the following operation is performed: Face region detection is performed on lip-sync video frames to obtain face region images; Lip key point detection was performed on the face region image to obtain multiple inner lip key points; The vertical opening distance and horizontal width of the inner lip were determined based on multiple key points of the inner lip. The mouth opening degree is determined based on the vertical opening distance of the inner lip and the horizontal width of the inner lip. The mouth opening degree is equal to the vertical opening distance of the inner lip divided by the horizontal width of the inner lip. By connecting multiple key points of the inner lip in sequence, an inner lip contour polygon is obtained. The inner lip contour polygon includes multiple vertices, where each vertex has a vertex x-coordinate and a vertex y-coordinate. Calculate the area of ​​the inner lip contour based on the polygonal shape of the inner lip; The mouth opening degree is combined with the inner lip contour area to obtain the mouth shape geometric features; By summarizing the lip shape geometric features, multiple lip shape geometric features are obtained; Multiple lip shape geometric features are temporally encoded to obtain visual temporal features.

10. A speech feature extraction system for animation translation, characterized in that, The system includes: The speech segment acquisition module is used to identify the animation to be translated, extract the audio stream of the animation to be translated to obtain the original audio stream, perform speech activity detection on the original audio stream to obtain the clean human voice segment, and perform speaker separation on the clean human voice segment to obtain multiple speech segment sets; The emotion type recognition module performs the following operations on each of the multiple speech segment sets: based on the speech segment set and the animation to be translated, it identifies the lip-sync video frame sequence, wherein the lip-sync video frame sequence includes: multiple lip-sync video frames; it extracts acoustic features from the speech segment set to obtain an acoustic feature sequence, wherein the acoustic feature sequence includes: multiple Mel frequency cepstral coefficient sets; and it performs emotion analysis on the acoustic feature sequence to obtain the emotion category. The visual feature confirmation module is used to confirm prosodic feature parameters based on multiple Mel frequency cepstral coefficient sets in the acoustic feature sequence, and to jointly encode the prosodic feature parameters with the emotion category to obtain the character voice style vector. It also confirms visual temporal features based on multiple lip-sync video frames in the lip-sync video frame sequence. The speech feature extraction module is used to identify audio-visual coupling features based on acoustic feature sequences and visual temporal features, identify character speech feature profiles based on character speech style vectors and audio-visual coupling features, summarize character speech feature profiles, obtain multiple character speech feature profiles, and complete speech feature extraction.

Images