Sound duplicating method and system

Abstract

The invention discloses a sound duplicating method and system, and the method comprises the steps: obtaining a long audio from a network, and carrying out the audio processing, and obtaining a preprocessed audio; performing noise reduction processing on the preprocessed audio based on a deep learning noise reduction method to obtain a noise-reduced voice; performing long audio recognition and alignment on the noise-reduced voice by adopting a large multi-language voice recognition model to obtain an alignment result; segmenting the long audio and the long text in the alignment result to obtain a segmentation result; performing multi-modal feature extraction on the segmentation result to obtain a multi-modal feature; performing sound repetition training and reasoning according to the multi-modal features; and during sound duplicate reasoning, data generation is carried out. The long audio alignment accuracy is improved, and the sound duplicating effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of speech signal processing, and in particular to a sound reproduction method and system. Background Art

[0002] With the rapid development of artificial intelligence technology, related technologies in the field of speech are becoming more and more mature, and speech technology applications are becoming more and more extensive. For example, when bank customer service staff have business conversations with customers, they often need to communicate with synthetic voices. However, if the synthetic voice is different from the real voice, it is easy for customers to notice it, which will affect the business experience and make customers feel unfriendly. Therefore, under the condition that the original data of the target speaker is limited, the more natural the synthetic voice is, the better. The speech synthesis technology models that have been implemented in the industry are all trained based on massive, high-quality speech alignment data, and the acquisition of massive high-quality annotated data requires a large manual annotation team. Since there are a large number of cheap and massive audio resources on the Internet, most of which are long audio, however, the subtitle timestamps of these audios may be wrong, and the subtitle text may also be missing or wrong. Therefore, in order to effectively use such corpus, it is necessary to solve the problem of long audio recognition and alignment. As an effective data annotation method, long audio alignment has begun to be applied to massive data annotation due to its low cost and high efficiency.

[0003] In terms of data processing, the commonly used method is to first extract long audio data from long audio and video data, then recognize the long audio data to obtain long text data, and finally align the long text data and the long audio data to obtain aligned data. In terms of speech synthesis models, the commonly used method is to first train a basic speech synthesis model using a large amount of aligned data, and then use the target speaker data to fine-tune the basic model to obtain the target speaker synthesis model, that is, to achieve the purpose of sound replication. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to provide a sound replication method and system that overcomes the above problems or at least partially solves the above problems.

[0005] According to one aspect of the present invention, a sound replication method is provided, the sound replication method comprising:

[0006] Obtain long audio from the Internet, perform audio processing, and obtain pre-processed audio;

[0007] Performing noise reduction processing on the preprocessed audio using a deep learning-based noise reduction method to obtain noise-reduced speech;

[0008] The noise reduction speech is subjected to long audio recognition and alignment using a large multilingual speech recognition model to obtain an alignment result;

[0009] Segment the long audio and long text in the alignment result to obtain a segmentation result;

[0010] Performing multimodal feature extraction on the segmentation result to obtain multimodal features;

[0011] Performing sound reproduction training and reasoning according to the multimodal features;

[0012] During the sound replication inference, data generation is performed.

[0013] Optionally, the acquiring of the long audio from the network and performing audio processing to obtain the pre-processed audio includes:

[0014] Get long audio from the Internet, including languages ​​​​including Chinese, English, Japanese, and Korean;

[0015] Preprocessing the long audio into a unified format includes downsampling and equalization;

[0016] Audio formats include wav and mp3.

[0017] Optionally, performing noise reduction processing on the preprocessed audio based on a deep learning-based noise reduction method to obtain noise-reduced speech specifically includes:

[0018] Use a deep learning-based noise reduction method to remove noise and retain speech. The specific steps are:

[0019] Extract features from long audio. Long audio features include time domain features and frequency domain features. Taking frequency domain feature STFT extraction as an example, pre-emphasize the audio data and perform frame operations. The 10ms to 30ms speech signal segment is a quasi-steady-state process. The frame length is 16m or 25ms, the frame shift is 8ms or 10ms, and the overlap between two frames is 50% to 60%. Perform Hamming window processing on each frame, and obtain the frequency domain amplitude spectrum through fast Fourier transform and modulus.

[0020] The extracted long audio features are input into the denoising network, and the denoised frequency domain features are obtained by forward calculation through the parameters learned by the denoising network, wherein the encoder or decoder of the denoising network includes a convolutional layer, a pooling layer, a recurrent neural network, and a generative adversarial network, and the denoising network is a model parameter learned by minimizing the difference between the predicted audio and the real clean audio;

[0021] The denoised frequency domain signal is converted into a denoised time domain signal, wherein the vocoder includes short-time inverse Fourier transform, Griffin-Lim, and HiFiGAN to complete the reconstruction of the audio signal from the frequency domain to the time domain.

[0022] Optionally, the step of using a large multilingual speech recognition model to perform long audio recognition and alignment on the noise reduction speech to obtain an alignment result specifically includes:

[0023] Input the denoised long audio into a large multilingual speech recognition model, which includes an encoder and multiple decoders, wherein the encoder or decoder of the denoising network includes a convolutional layer, a pooling layer, a recurrent neural network, a generative adversarial network, and an attention mechanism;

[0024] The output of the multilingual speech recognition model is the text corresponding to the long audio. Each sentence of text includes the language, speaker, text timestamp, and punctuation information, including:

[0025] Use Mel filter banks on the extracted long audio features to map the amplitude spectrum to the Mel frequency scale;

[0026] Take the logarithm of the output of the Mel filter bank to obtain the Mel spectrum feature;

[0027] The Mel spectrum features are mapped to high-dimensional hidden vectors by the encoder of the multi-language speech recognition model. The vectors are input to the language recognition decoder to obtain the language information corresponding to each segment of the current audio. The output categories include Chinese, English, Japanese, and Korean.

[0028] The high-dimensional hidden layer vector and language information obtained from the Mel-spectrogram feature map are input into the speaker recognition decoder to obtain the speaker information corresponding to each segment of the current audio. It is assumed that the speakers in different long audios are independent and the speaker IDs are not repeated.

[0029] The high-dimensional hidden layer vector, language and speaker information obtained from the Mel-spectrogram feature map are input into the speech recognition decoder to obtain the text content corresponding to each segment of the current audio and the timestamp information of each word.

[0030] Optionally, segmenting the long audio and long text in the alignment result to obtain the segmentation result includes:

[0031] The alignment results are segmented, where the long audio is the audio after noise reduction, and the long text is the content with timestamp after alignment;

[0032] Set the silence duration thresholds θ1 and θ2, and the audio clip duration threshold θ max ,θ min , in seconds, where θ max >θ1>θ2>θ min , and the text quantity threshold is S1, and the segmentation rules from the left end to the right end of the audio include:

[0033] If the silence duration is greater than or equal to θ1, the audio clip is greater than or equal to θmin and less than or equal to θ max , then the segmentation is done based on the center time of silence;

[0034] If the silence duration is greater than or equal to θ2, the number of characters is greater than or equal to S1, and the audio clip is greater than or equal to θ min and less than or equal to θ max , then the segmentation is done based on the center time of silence;

[0035] Other data that do not meet the conditions are automatically filtered until the current long audio is segmented. The final segmentation result is that the duration of each audio segment is θ min to θ max Not waiting.

[0036] Optionally, extracting multimodal features from the segmentation results to obtain multimodal features includes:

[0037] The segmented audio and text segments with language and speaker information are input into the pre-trained multi-language, multi-modal encoding model, and the text context is linked to its corresponding context in the joint multi-modal space to obtain multi-modal alignment information. The specific steps are as follows:

[0038] When a text is used as the input of a pre-trained multi-language, multi-modal encoding model, a sentence of text uses the semantic capabilities of the large model to integrate contextual information and obtain the text representation vector of the target token. The components of the text encoder of the large model include convolutional layers, pooling layers, recurrent neural networks, generative adversarial networks, and attention mechanisms.

[0039] When using audio as input to a pre-trained multi-language, multi-modal large model, the speech corresponding to the target token is input into the network to extract the prosodic representation;

[0040] The audio encoder of the multimodal coding model includes convolutional layers, pooling layers, recurrent neural networks, generative adversarial networks and attention mechanisms.

[0041] Optionally, the performing sound replication training and reasoning according to the multimodal features specifically includes:

[0042] The multimodal features are used for sound replication training and reasoning to obtain the replication effect. The implementation process specifically includes:

[0043] Use massive data to train a multi-speaker synthesis model and use it as a base model. Use multimodal features to fine-tune the replica model. When the loss function value of the replica model no longer decreases, stop fine-tuning the training.

[0044] When replicating model reasoning, the multimodal feature extraction model used during fine-tuning training is introduced to assist in the fine-tuning and control of emotions, pauses, and rhythm during reasoning, so that the model can accurately fit the speaker's pronunciation characteristics.

[0045] Optionally, the data generation during the sound replication reasoning specifically includes: during the sound replication reasoning, the speech speed, intonation, and volume gain of the audio are realized by a speech signal processing method.

[0046] The present invention also provides a sound reproduction system, which uses the above-mentioned sound reproduction method, and the sound reproduction system comprises:

[0047] The data acquisition module is used to obtain long audio from the network and perform audio processing to obtain pre-processed audio;

[0048] A long audio noise reduction module, used to perform noise reduction processing on the pre-processed audio based on a deep learning noise reduction method to obtain noise-reduced speech;

[0049] A long audio recognition and alignment module is used to perform long audio recognition and alignment on the noise reduction speech using a large multilingual speech recognition model to obtain an alignment result;

[0050] A long audio and long text segmentation module, used for segmenting the long audio and long text in the alignment result to obtain a segmentation result;

[0051] A multimodal feature extraction module, used for performing multimodal feature extraction on the segmentation result to obtain multimodal features;

[0052] A sound replication module, used for performing sound replication training and reasoning according to the multimodal features;

[0053] The data generation module is used to generate data during sound replication reasoning.

[0054] The present invention provides a sound replication method and system, the sound replication method comprising: obtaining long audio from the network, and performing audio processing to obtain pre-processed audio; performing noise reduction processing on the pre-processed audio based on a deep learning noise reduction method to obtain noise-reduced speech; using a multilingual speech recognition large model to perform long audio recognition and alignment on the noise-reduced speech to obtain an alignment result; segmenting the long audio and long text in the alignment result to obtain a segmentation result; performing multimodal feature extraction on the segmentation result to obtain multimodal features; performing sound replication training and reasoning based on the multimodal features; and generating data during sound replication reasoning. The alignment accuracy of long audio and the sound replication effect are improved.

[0055] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0057] Figure 1 A flow chart of a sound reproduction method provided by an embodiment of the present invention;

[0058] Figure 2 The Chinese audio and its corresponding text and timestamp provided by the embodiment of the present invention;

[0059] Figure 3 A flowchart for implementing sound replication provided by an embodiment of the present invention;

[0060] Figure 4 A block diagram of a sound reproduction system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0062] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0063] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0064] like Figure 1 As shown, a sound reproduction method includes:

[0065] Step 1) Data acquisition module, the specific steps include:

[0066] 1.1) Data collection: A large amount of long audio (such as movies, TV series, short videos, etc.) can be obtained from the Internet, and the languages ​​include but are not limited to Chinese, English, Japanese, Korean, etc.

[0067] 1.2) Audio processing: Since the data formats of massive audio obtained from the Internet are not unified, they need to be preprocessed into a unified format, including but not limited to downsampling, equalization, etc. The audio formats are but not limited to wav, mp3, etc. Taking wav format as an example, its detailed parameter settings are as follows: the number of channels is 1, the sampling rate is 16KHz, and the bit depth is 16bit.

[0068] Step 2) Long audio noise reduction module, the specific steps include:

[0069] Since the long audio in step 1) comes from thousands of scenes, each audio may contain one or more noises, such as background music, environmental noise, special effects sound, etc., and due to the post-production of the video, including but not limited to soundtracks, sound effects, etc., the audio signal-to-noise ratio is low, which directly affects the accuracy of subsequent long audio recognition and alignment. By effectively removing noise and retaining speech through a deep learning-based denoising method, not only can the quality of long audio be improved, but also the success rate and accuracy of long audio alignment can be improved. The denoising method can be implemented in the frequency domain or the time domain. Here, a patent for a denoising algorithm is referenced [An end-to-end speech enhancement method for low-resource Tujia language: CN201910966022.6[P].2019-12-27.] The specific steps of long audio denoising are as follows:

[0070] 2.1) Extract features of the long audio in step 1.3). The long audio features include time domain features, frequency domain features, etc. Taking the frequency domain feature STFT extraction step as an example, first pre-emphasize the audio data and perform frame operations. The speech signal segment of 10ms to 30ms is a quasi-steady-state process. Therefore, the general frame length is 16m or 25ms, and the frame shift is 8ms or 10ms, that is, the overlap of the two frames is 50% to 60%. Then, each frame is processed by Hamming window, and finally the frequency domain amplitude spectrum is obtained by fast Fourier transform and modulus.

[0071] 2.2) Inputting the long audio features extracted in step 2.1) into the denoising network, and performing forward calculations through the parameters learned by the denoising network to obtain the denoised frequency domain features, wherein the components of the encoder or decoder of the denoising network include but are not limited to convolutional layers, pooling layers, recurrent neural networks, generative adversarial networks, etc., and the denoising network is a model parameter learned by minimizing the difference between the predicted audio and the real clean audio.

[0072] 2.3) The frequency domain signal after the noise reduction processing in step 2.2) is passed through a vocoder module to generate a denoised time domain signal, where the vocoder module includes short-time inverse Fourier transform, Griffin-Lim, HiFiGAN, etc. Here, short-time inverse Fourier transform is taken as an example to complete the reconstruction of the audio signal from frequency domain to time domain.

[0073] Step 3) Long audio recognition and alignment module, the specific steps include:

[0074] The long audio after denoising in step 2) is input into the multilingual speech recognition model. The multilingual speech recognition model consists of an encoder and multiple decoders, where the components of the encoder or decoder of the denoising network include but are not limited to convolutional layers, pooling layers, recurrent neural networks, generative adversarial networks, attention mechanisms, etc. The output of the multilingual speech recognition model is the text corresponding to the long audio, where each sentence of text includes information such as language, speaker, text timestamp, punctuation, etc. The specific implementation steps are as follows:

[0075] 3.1) First, the long audio features extracted in step 2.1) are mapped to the Mel frequency scale using a Mel filter bank, and then the output of the Mel filter bank is logarithmized to better simulate the human auditory system, that is, to obtain the Mel spectrum feature. The Mel spectrum feature is mapped to a high-dimensional hidden layer vector by the encoder of the multilingual speech recognition model, and the vector is input to the language recognition decoder to obtain the language information corresponding to each segment of the current audio. The output categories include but are not limited to Chinese, English, Japanese, Korean, etc.

[0076] 3.2) Input the high-dimensional hidden layer vector and language information obtained by the Mel-spectrogram feature mapping in step 3.1) into the speaker recognition decoder to obtain the speaker information corresponding to each segment of the current audio. Here, it is assumed that the speakers in different long audios are independent, that is, the speaker IDs are not repeated.

[0077] 3.3) Input the high-dimensional hidden layer vector obtained by Mel-spectrogram feature mapping in step 3.1), the language, and the speaker information in step 3.2) into the speech recognition decoder to obtain the text content corresponding to each segment of the current audio and the timestamp information of each character. Taking Chinese as an example, the recognition and alignment effect is as follows Figure 2 As shown:

[0078] Specifically, in Figure 2 In the figure, the first layer is the audio waveform, and the second layer is the recognition result, that is, the text content corresponding to the audio is "Would you like to know if you are interested?", where the blank space of the label in the figure indicates silence or pause in tone.

[0079] Step 4) Long audio and long text segmentation module, the specific steps include:

[0080] The alignment result in step 3) is segmented, where the long audio is the audio after noise reduction, and the long text is the content with timestamp after alignment. Set the silence duration thresholds θ1 and θ2, and the audio segment duration threshold θ max ,θ min , in seconds, where θ max >θ1>θ2>θ min , and the text quantity threshold is S1, the segmentation rules and steps from the left end to the right end of the audio are as follows:

[0081] 4.1) If the silence duration is greater than or equal to θ1, the audio segment is greater than or equal to θ min and less than or equal to θ max , the segmentation point is the center time of silence.

[0082] 4.2) If the silence duration is greater than or equal to θ2, the number of words is greater than or equal to S1, and the audio clip is greater than or equal to θ min and less than or equal to θ max , the segmentation point is the center time of silence.

[0083] 4.3) Other data that do not meet the conditions are automatically filtered until the current long audio is segmented. The final segmentation result is that the duration of each audio segment is θ min to θ max Not waiting.

[0084] Step 5) Multimodal feature extraction module, the specific steps include:

[0085] The audio and text segments with language and speaker information after segmentation in step 4) are input into the pre-trained multi-language, multi-modal encoding model, and the text context is linked to its corresponding context in the joint multi-modal space to obtain multi-modal alignment information. The specific implementation steps are as follows:

[0086] 5.1) When text is used as the input of a pre-trained multilingual, multimodal large model, a sentence of text uses the semantic capabilities of the large model to integrate contextual information and obtain the text representation vector of the target token. The components of the text encoder of the large model include but are not limited to convolutional layers, pooling layers, recurrent neural networks, generative adversarial networks, attention mechanisms, etc.

[0087] 5.2) When the audio is used as the input of the pre-trained multi-language, multi-modal large model, the speech corresponding to the target token is input into the network to extract the prosodic representation. The components of the audio encoder of the large model include but are not limited to convolutional layers, pooling layers, recurrent neural networks, generative adversarial networks, attention mechanisms, etc.

[0088] Step 6) Sound replication module, the specific steps include:

[0089] The multimodal features in step 5) are used for sound reproduction training and reasoning respectively to obtain richer and more robust reproduction effects. The implementation process is as follows: Figure 3 As shown, the specific implementation steps are as follows:

[0090] 6.1) Train a multi-speaker synthesis model with massive data, use it as the base model, and use the multimodal features in step 5) to fine-tune the replica model. When the loss function value of the replica model no longer decreases, for example, the average loss function value of the first 100 steps and the average loss function value of the last 100 steps differ by 0.01, stop the fine-tuning training.

[0091] 6.2) When replicating model reasoning, the multimodal feature extraction model used in fine-tuning training is introduced to assist in fine-tuning and controlling emotions, pauses, and rhythm during reasoning, so that the model can further finely fit the speaker’s pronunciation characteristics and improve the replication effect.

[0092] Step 7) Data generation module, the specific steps include:

[0093] During sound replication reasoning, the speech speed, intonation, volume gain and other adjustments of the audio are achieved through voice signal processing methods to meet the needs of different scenarios and applications.

[0094] like Figure 4 As shown, a sound reproduction system comprises:

[0095] Data acquisition module, long audio noise reduction module, long audio recognition and alignment module, long audio and long text segmentation module, multimodal feature extraction module, sound reproduction module and data generation module.

[0096] Among them, the data acquisition module is connected to the long audio noise reduction module and transmits the long audio to be denoised, the long audio noise reduction module is connected to the long audio recognition and alignment module and transmits the long audio after noise reduction, the long audio recognition and alignment module is connected to the long audio and long text segmentation module and transmits the alignment result of the long audio and its text, the long audio and long text segmentation module is connected to the multimodal feature extraction module and transmits the short audio and its short text, the multimodal feature extraction module is connected to the sound reproduction module and transmits the feature information of the audio and text, the sound reproduction module is connected to the data generation module and transmits the speech time domain feature information, and finally generates the sound data with the required attributes.

[0097] Beneficial effects: Improve the accuracy of long audio alignment and enhance the sound reproduction effect.

[0098] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A sound reproduction method, characterized in that: The sound reproduction method comprises: Obtain long audio from the Internet, perform audio processing, and obtain pre-processed audio; Performing noise reduction processing on the preprocessed audio using a deep learning-based noise reduction method to obtain noise-reduced speech; The noise reduction speech is subjected to long audio recognition and alignment using a large multilingual speech recognition model to obtain an alignment result; Segment the long audio and long text in the alignment result to obtain a segmentation result; Performing multimodal feature extraction on the segmentation result to obtain multimodal features; Performing sound reproduction training and reasoning according to the multimodal features; During the sound replication inference, data generation is performed.

2. A sound reproduction method according to claim 1, characterized in that: The step of obtaining the long audio from the network and performing audio processing to obtain the pre-processed audio includes: Get long audio from the Internet, including languages ​​​​including Chinese, English, Japanese, and Korean; Preprocessing the long audio into a unified format includes downsampling and equalization; Audio formats include wav and mp3.

3. A sound reproduction method according to claim 1, characterized in that: The performing noise reduction processing on the pre-processed audio based on the deep learning noise reduction method to obtain the noise-reduced speech specifically includes: Use a deep learning-based noise reduction method to remove noise and retain speech. The specific steps are: Extract features from long audio. Long audio features include time domain features and frequency domain features. Taking frequency domain feature STFT extraction as an example, pre-emphasize the audio data and perform frame operations. The 10ms to 30ms speech signal segment is a quasi-steady-state process. The frame length is 16m or 25ms, the frame shift is 8ms or 10ms, and the overlap between two frames is 50% to 60%. Perform Hamming window processing on each frame, and obtain the frequency domain amplitude spectrum through fast Fourier transform and modulus. The extracted long audio features are input into the denoising network, and the denoised frequency domain features are obtained by forward calculation through the parameters learned by the denoising network, wherein the encoder or decoder of the denoising network includes a convolutional layer, a pooling layer, a recurrent neural network, and a generative adversarial network, and the denoising network is a model parameter learned by minimizing the difference between the predicted audio and the real clean audio; The denoised frequency domain signal is converted into a denoised time domain signal, wherein the vocoder includes short-time inverse Fourier transform, Griffin-Lim, and HiFiGAN to complete the reconstruction of the audio signal from the frequency domain to the time domain.

4. A sound reproduction method according to claim 1, characterized in that: The noise reduction speech is subjected to long audio recognition and alignment using a large multilingual speech recognition model to obtain an alignment result, specifically comprising: Input the denoised long audio into a large multilingual speech recognition model, which includes an encoder and multiple decoders, wherein the encoder or decoder of the denoising network includes a convolutional layer, a pooling layer, a recurrent neural network, a generative adversarial network, and an attention mechanism; The output of the multilingual speech recognition model is the text corresponding to the long audio. Each sentence of text includes the language, speaker, text timestamp, and punctuation information, including: Use Mel filter banks on the extracted long audio features to map the amplitude spectrum to the Mel frequency scale; Take the logarithm of the output of the Mel filter bank to obtain the Mel spectrum feature; The Mel spectrum features are mapped to high-dimensional hidden vectors by the encoder of the multi-language speech recognition model. The vectors are input to the language recognition decoder to obtain the language information corresponding to each segment of the current audio. The output categories include Chinese, English, Japanese, and Korean. The high-dimensional hidden layer vector and language information obtained from the Mel-spectrogram feature map are input into the speaker recognition decoder to obtain the speaker information corresponding to each segment of the current audio. It is assumed that the speakers in different long audios are independent and the speaker IDs are not repeated. The high-dimensional hidden layer vector, language and speaker information obtained from the Mel-spectrogram feature map are input into the speech recognition decoder to obtain the text content corresponding to each segment of the current audio and the timestamp information of each word.

5. A sound reproduction method according to claim 1, characterized in that: The segmenting of the long audio and long text in the alignment result to obtain the segmentation result includes: The alignment results are segmented, where the long audio is the audio after noise reduction, and the long text is the content with timestamp after alignment; Set the silence duration thresholds θ1 and θ2, and the audio clip duration threshold θ max ,θ min , in seconds, where θ max >θ1>θ2>θ min , and the text quantity threshold is S1, and the segmentation rules from the left end to the right end of the audio include: If the silence duration is greater than or equal to θ1, the audio clip is greater than or equal to θ min and less than or equal to θ max , then the segmentation is done based on the center time of silence; If the silence duration is greater than or equal to θ2, the number of characters is greater than or equal to S1, and the audio clip is greater than or equal to θ min and less than or equal to θ max , then the segmentation is done based on the center time of silence; Other data that do not meet the conditions are automatically filtered until the current long audio is segmented. The final segmentation result is that the duration of each audio segment is θ min to θ max Not waiting.

6. A sound reproduction method according to claim 1, characterized in that: The step of extracting multimodal features from the segmentation results to obtain multimodal features includes: The segmented audio and text segments with language and speaker information are input into the pre-trained multi-language, multi-modal encoding model, and the text context is linked to its corresponding context in the joint multi-modal space to obtain multi-modal alignment information. The specific steps are as follows: When a text is used as the input of a pre-trained multi-language, multi-modal encoding model, a sentence of text uses the semantic capabilities of the large model to integrate contextual information and obtain the text representation vector of the target token. The components of the text encoder of the large model include convolutional layers, pooling layers, recurrent neural networks, generative adversarial networks, and attention mechanisms. When using audio as input to a pre-trained multi-language, multi-modal large model, the speech corresponding to the target token is input into the network to extract the prosodic representation; The audio encoder of the multimodal coding model includes convolutional layers, pooling layers, recurrent neural networks, generative adversarial networks and attention mechanisms.

7. A sound reproduction method according to claim 1, characterized in that: The sound reproduction training and reasoning according to the multimodal features specifically includes: The multimodal features are used for sound replication training and reasoning to obtain the replication effect. The implementation process specifically includes: Use massive data to train a multi-speaker synthesis model and use it as a base model. Use multimodal features to fine-tune the replica model. When the loss function value of the replica model no longer decreases, stop fine-tuning the training. When replicating model reasoning, the multimodal feature extraction model used during fine-tuning training is introduced to assist in the fine-tuning and control of emotions, pauses, and rhythm during reasoning, so that the model can accurately fit the speaker's pronunciation characteristics.

8. A sound reproduction method according to claim 1, characterized in that: The data generation during the sound replication reasoning specifically includes: during the sound replication reasoning, the speech speed, intonation, and volume gain of the audio are realized through a speech signal processing method.

9. A sound reproduction system, using a sound reproduction method according to any one of claims 1 to 8, characterized in that: The sound reproduction system comprises: The data acquisition module is used to obtain long audio from the network and perform audio processing to obtain pre-processed audio; A long audio noise reduction module, used to perform noise reduction processing on the pre-processed audio based on a deep learning noise reduction method to obtain noise-reduced speech; A long audio recognition and alignment module is used to perform long audio recognition and alignment on the noise reduction speech using a large multilingual speech recognition model to obtain an alignment result; A long audio and long text segmentation module, used for segmenting the long audio and long text in the alignment result to obtain a segmentation result; A multimodal feature extraction module, used for performing multimodal feature extraction on the segmentation result to obtain multimodal features; A sound replication module, used for performing sound replication training and reasoning according to the multimodal features; The data generation module is used to generate data during sound replication reasoning.

Images