Text visualization voice editing method and system, storage medium and equipment

Abstract

The invention provides a text visualization voice editing method and system, a storage medium and equipment, and the method comprises the steps: carrying out the feature extraction of an input original audio, obtaining a first sound feature, carrying out the multi-layer linear layer mapping processing of the first sound feature and a second sound feature, and generating an audio mark; converting the audio mark into a corresponding character content through a reverse mark, and generating a text mark for all changed characters in the editing process; converting the generated text mark and the mapped audio mark into a unified semantic mark; and performing feature fusion on the first sound feature and the second sound feature, and decoding the semantic mark into a synthetic audio by the fused sound feature through a conditional stream matching algorithm to obtain a voice editing result. According to the method and the device, the problems of complicated operation, poor interactivity and low efficiency in the voice editing process in the prior art are solved, so that the voice editing mode is more free, and meanwhile, the quality of the finally synthesized voice is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of voice editing technology, and in particular to a text visualization voice editing method, system, storage medium and device. Background Art

[0002] Voice editing technology refers to the processing and modification of voice signals. Its core is to modify and optimize voice content through editing, splicing, pitch adjustment, etc. This technology plays a vital role in the fields of voice assistants and intelligent customer service, and can improve the naturalness and accuracy of voice interaction, thereby enhancing the user experience.

[0003] Although voice editing technology has made some progress, the current mainstream technology is mostly based on traditional machine learning and has obvious limitations. For example, when processing long sequence sound files, such as mispronunciation, grammatical errors, content addition or redundant deletion, the operation process is complicated and requires repeated playback and confirmation of the modified time period. This operation method is not only inefficient but also lacks intuitiveness.

[0004] With the rapid development of artificial intelligence technology, speech-to-text (ASR) and text-to-speech (TTS) algorithms based on large language models (LLM) have gradually become mainstream due to their high naturalness and zero-shot capabilities. These technologies have improved the efficiency and quality of voice editing to a certain extent, but they still cannot meet users' needs for simple and intuitive editing of voice content.

[0005] In view of the above problems, the present invention proposes a text visualization voice editing method, system, storage medium and device, which can realize intuitive and efficient editing of voice content. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a text visualization voice editing method, system, storage medium and device, which solve the problems of complex operation, poor interactivity and low efficiency in the voice editing process in the prior art.

[0007] On the one hand, an embodiment of the present invention provides a text visualization voice editing method, comprising:

[0008] Extracting features from the input original audio to obtain a first sound feature;

[0009] Input the second sound feature, perform multi-layer linear layer mapping processing on the first sound feature and the second sound feature, and generate an audio tag;

[0010] Convert the audio mark to corresponding text content through reverse marking, and perform text editing on the text content;

[0011] Generate corresponding text tags for all texts changed during the editing process;

[0012] Convert the generated text tags and mapped audio tags into unified semantic tags;

[0013] The first sound feature and the second sound feature are feature-fused, and the fused sound features are decoded into a synthetic audio with a unified semantic tag through a conditional stream matching algorithm to obtain a speech editing result.

[0014] Preferably, before extracting features from the input original audio to obtain the first sound feature, a voice activity detection algorithm is used to identify valid vocal segments in the input audio to obtain de-redundant audio segments, and the de-redundant audio segments are standardized in length according to a block length parameter to output original audio with a uniform length.

[0015] Preferably, performing multi-layer linear layer mapping processing on the first sound feature and the second sound feature to generate an audio tag includes:

[0016] Inputting the first sound feature and the second sound feature into a Transformer model, wherein the Transformer model comprises multiple Transformer layers;

[0017] In each Transformer layer, the internal dependencies of the input sound feature sequence are modeled through the self-attention mechanism;

[0018] Information fusion between sound feature sequences through cross-attention mechanism;

[0019] The feedforward neural network performs nonlinear transformation and feature extraction on the fused features;

[0020] The fused sound features obtained after being processed by the multi-layer Transformer layer are input into the multi-layer linear layer. Through the mapping operation of the multi-layer linear layer, the features are gradually mapped to the feature space of the audio tag;

[0021] The mapped features are classified through the output layer to generate corresponding audio tags.

[0022] Preferably, the first sound feature comprises one of a filter bank feature, a Mel frequency cepstral coefficient feature and a logarithmic Mel frequency spectrum feature; the second sound feature is a basic feature extracted from an audio signal, and the basic feature comprises audio volume, pitch and timbre.

[0023] In a second aspect, an embodiment of the present invention further provides a text visualization voice editing system, comprising:

[0024] A speech encoder, used for extracting features from the input original audio to obtain a first sound feature;

[0025] A speech encoder, configured to input a second sound feature, perform multi-layer linear layer mapping processing on the first sound feature and the second sound feature, and generate an audio tag;

[0026] A reverse tagger, used for converting the audio tag into corresponding text content through reverse tagging;

[0027] A text editing module, including a text editing window, for editing text content;

[0028] The text marker is used to generate corresponding text markers for all the changed texts during the editing process;

[0029] A large model module for converting the generated text tags and mapped audio tags into unified semantic tags;

[0030] The speech decoder is used to fuse the first sound feature and the second sound feature. The fused sound features are decoded into synthetic audio using a conditional stream matching algorithm to obtain a speech editing result.

[0031] Preferably, the speech encoder includes a Transformer model, which includes multiple layers of Transformer layers and multiple layers of linear layers, and each Transformer layer includes a self-attention layer, a cross-attention layer, and a feedforward neural network layer.

[0032] Preferably, the large model module includes one of the large language models of Qwen2.5-7B and GLM4-9B.

[0033] Preferably, it also includes a prompt word module, which is used to guide the large model module to generate expected output according to different instructions through preset prompt word content.

[0034] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, a text visualization speech editing method is implemented through the following steps: extracting features from the input original audio to obtain a first sound feature; inputting a second sound feature, performing multi-layer linear layer mapping processing on the first sound feature and the second sound feature to generate an audio tag; converting the audio tag into corresponding text content through reverse tagging, and performing text editing on the text content; generating corresponding text tags for all changed texts during the editing process; converting the generated text tags and the mapped audio tags into a unified semantic tag; performing feature fusion on the first sound feature and the second sound feature, and decoding the unified semantic tag into synthetic audio through a conditional stream matching algorithm on the fused sound feature to obtain a speech editing result.

[0035] In a fourth aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a text visualization voice editing method is implemented through the following steps: feature extraction is performed on the input original audio to obtain a first sound feature; a second sound feature is input, and the first sound feature and the second sound feature are subjected to multi-layer linear layer mapping processing to generate an audio tag; the audio tag is converted into corresponding text content through reverse tagging, and the text content is subjected to text editing; corresponding text tags are generated for all changed texts during the editing process; the generated text tags and the mapped audio tags are converted into a unified semantic tag; the first sound feature and the second sound feature are subjected to feature fusion, and the fused sound feature uses a conditional stream matching algorithm to decode the unified semantic tag into synthetic audio to obtain a voice editing result.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] By adopting the audio-text multimodal tagging and conditional stream matching algorithm decoding method, users can edit text more freely and are no longer restricted by traditional voice editing methods. They can flexibly adjust the voice content and style to meet diverse editing needs. By utilizing the Transformer model and multi-layer linear layer mapping processing in the voice encoder, as well as the powerful semantic understanding and generation capabilities of the large model module, they can quickly perform operations such as feature extraction, tag conversion, and semantic fusion, greatly improving the efficiency of voice editing and reducing user waiting and processing time. By optimizing the naturalness and fluency of the synthesized audio through feature fusion and conditional stream matching algorithms, the synthesized voice is more natural and fluent in terms of pitch, rhythm, emotion, etc., achieving efficient, stable and reliable voice editing and synthesis, which can meet users' high standards for voice editing quality and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a text visualization voice editing method according to an embodiment of the present invention;

[0039] Figure 2 A flowchart of the steps of performing multi-layer linear layer mapping processing on the first sound feature and the second sound feature to generate an audio tag provided by the present invention;

[0040] Figure 3 A schematic diagram of the principle of a text visualization voice editing system provided by the present invention;

[0041] Figure 4 A schematic diagram of the structure of a computer device provided by the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0043] like Figure 1 As shown, an embodiment of the present invention provides a text visualization voice editing method, comprising:

[0044] Extracting features from the input original audio to obtain a first sound feature;

[0045] Input the second sound feature, perform multi-layer linear layer mapping processing on the first sound feature and the second sound feature, and generate an audio tag;

[0046] Convert the audio mark to corresponding text content through reverse marking, and perform text editing on the text content;

[0047] Generate corresponding text tags for all texts changed during the editing process;

[0048] Convert the generated text tags and mapped audio tags into unified semantic tags;

[0049] The first sound feature and the second sound feature are feature-fused, and the fused sound features are decoded into a synthetic audio with a unified semantic tag through a conditional stream matching algorithm to obtain a speech editing result.

[0050] In this embodiment, the input original audio is firstly feature extracted to obtain the first sound feature, and then the second sound feature is input, and the first sound feature and the second sound feature are subjected to multi-layer linear layer mapping processing to generate an audio tag. Among them, the multi-layer linear layer mapping is a deep learning technology, which can learn the complex relationship between audio features through multi-layer linear transformation and nonlinear activation function, and generate accurate audio tags. The audio tag is converted into the corresponding text content through reverse tagging, and the user can edit the converted text content, such as modification, deletion, addition and other operations. During the editing process, corresponding text tags are generated for all changed texts, and these text tags record the editing information of the text. The generated text tags and the mapped audio tags are converted into unified semantic tags. Finally, the first sound feature and the second sound feature are feature fused, and the fused sound feature decodes the unified semantic tag into synthetic audio through the conditional stream matching algorithm to obtain the speech editing result.

[0051] The method of multimodal tagging of audio text and decoding by conditional stream matching algorithm makes the voice editing method more free. Users can edit the text as needed, and the synthesized voice can accurately reflect the edited text content, and is no longer restricted by traditional voice editing methods. For example, traditional voice editing may require converting the voice into text first, and then re-synthesizing the voice after editing. This process often involves multiple steps and requires complex logical judgment and processing, such as voice recognition, text alignment, etc. The technical solution proposed in the present invention allows users to edit directly at the text level, and the voice synthesis will automatically adapt to the changes in the text; improving the efficiency and quality of voice editing. The deep fusion and processing of multimodal features make the voice editing process more efficient. In the text modification stage, there is no need to perform complex logical judgment. It only needs to process the edited text and voice fragments into text tags and voice tags, and then directly input the large language model (LLM) to synthesize semantically unified tags, reducing the intermediate steps and calculation amount. At the same time, the quality of the synthesized voice is significantly improved, more natural and smooth. The conditional stream matching algorithm can generate natural speech that matches the input semantic tags and sound features, making the synthesized speech closer to human natural pronunciation in terms of intonation, rhythm, emotion, etc., thereby improving the realism and comprehensibility of speech synthesis.

[0052] Preferably, before extracting features from the input original audio to obtain the first sound feature, a voice activity detection algorithm is used to identify valid vocal segments in the input audio to obtain de-redundant audio segments, and the de-redundant audio segments are divided and padded according to a block length parameter to output original audio of uniform length.

[0053] In this embodiment, the voice activity detection algorithm (VAD) is first used to analyze various features of the audio signal (such as energy, zero-crossing rate, spectrum, etc.) to determine whether the current audio frame contains human voices to effectively remove the silent segments and background noise in the audio, and only retain the part containing the effective human voice. By removing redundant audio segments, the amount of data to be processed is reduced, thereby improving the efficiency of subsequent feature extraction and voice editing. Then, the de-redundant audio segments are divided into blocks according to the preset block length parameters. In order to ensure that the length of each audio block is uniform, the audio blocks of insufficient length are also padded by padding, so that the audio blocks have a unified format in subsequent processing. After the de-redundant audio segments are divided into blocks and padded, the next step of feature extraction is performed.

[0054] like Figure 2 As shown, Figure 2 A flowchart of the steps of performing multi-layer linear layer mapping processing on the first sound feature and the second sound feature to generate an audio tag provided by the present invention.

[0055] The first sound feature and the second sound feature are subjected to multi-layer linear layer mapping processing to generate an audio tag, including:

[0056] Inputting the first sound feature and the second sound feature into a Transformer model, wherein the Transformer model comprises multiple Transformer layers;

[0057] In each Transformer layer, the internal dependencies of the input sound feature sequence are modeled through the self-attention mechanism;

[0058] Information fusion between sound feature sequences through cross-attention mechanism;

[0059] The feedforward neural network performs nonlinear transformation and feature extraction on the fused features;

[0060] The fused sound features obtained after being processed by the multi-layer Transformer layer are input into the multi-layer linear layer. Through the mapping operation of the multi-layer linear layer, the features are gradually mapped to the feature space of the audio tag;

[0061] The mapped features are classified through the output layer to generate corresponding audio tags.

[0062] In this embodiment, the Transformer model is a deep learning model based on the self-attention mechanism. The Transformer model includes multiple layers of Transformer layers, each of which is composed of a self-attention mechanism and a feedforward neural network. In each Transformer layer, the self-attention mechanism constructs the internal dependency relationship within the sequence by calculating the attention weight of each element in the sequence to other elements; then the information between the sound feature sequences is fused through the cross-attention mechanism, and then the fused features are nonlinearly transformed and extracted by the feedforward neural network, and the fused sound features obtained after processing by the multiple layers of Transformer layers are input into the multiple layers of linear layers. The multiple layers of linear layers gradually map the features to the feature space of the audio tag through linear transformation, and the output layer classifies the mapped features by using a classifier, and maps the features to different audio tag categories, thereby obtaining the final audio tag result.

[0063] Preferably, the first sound feature includes one of a filter bank feature, a Mel-frequency cepstral coefficient feature and a logarithmic Mel-frequency spectrum feature; the second sound feature is a basic feature extracted from an audio signal, and the basic feature includes audio volume, pitch and timbre.

[0064] In this embodiment, the first sound feature is one of the filter bank features, Mel-frequency cepstral coefficient features and logarithmic Mel-frequency spectrum features used to capture the spectral characteristics and acoustic characteristics of the audio; the second sound feature is the basic feature extracted from the audio signal, mainly including information such as the volume, pitch and timbre of the audio; in this embodiment, the second sound feature is the basic sound feature of the external input.

[0065] Preferably, the text editing of the text content includes modifying, adding and deleting the text content.

[0066] Specifically, when generating corresponding text tags for all the changed texts during the editing process, the texts modified and added during the editing process are mainly marked.

[0067] like Figure 3 As shown, Figure 3 A schematic diagram of the principle of a text visualization voice editing system provided by the present invention includes:

[0068] A speech encoder, used for extracting features from the input original audio to obtain a first sound feature;

[0069] A speech encoder, configured to input a second sound feature, perform multi-layer linear layer mapping processing on the first sound feature and the second sound feature, and generate an audio tag;

[0070] A reverse tagger, used for converting the audio tag into corresponding text content through reverse tagging;

[0071] A text editing module is used to edit text content;

[0072] The text marker is used to generate corresponding text markers for all the changed texts during the editing process;

[0073] A large model module for converting the generated text tags and mapped audio tags into unified semantic tags;

[0074] The speech decoder is used to fuse the first sound feature and the second sound feature. The fused sound features are decoded into synthetic audio using a conditional stream matching algorithm to obtain a speech editing result.

[0075] Preferably, the speech encoder includes a Transformer model, which includes multiple layers of Transformer layers and multiple layers of linear layers, and each Transformer layer includes a self-attention layer, a cross-attention layer, and a feedforward neural network layer.

[0076] Preferably, the large model module includes one of the large language models Qwen2.5-7B and GLM4-9B.

[0077] Preferably, it also includes a prompt word module, which is used to guide the large model module to generate expected output according to different instructions through preset prompt word content.

[0078] In this embodiment, the preset prompt word content may be "Please repeat this content, making sure that each word is the same". The prompt word mechanism guides the large model module to generate semantic tags that meet the expected output according to different instructions, so as to improve the output accuracy and consistency of the large model and ensure that the speech synthesis result meets expectations.

[0079] Through the text visualization speech editing system proposed in this embodiment, an efficient speech editing and synthesis process is realized. By editing text, users can simply and intuitively edit speech, and finally generate high-quality synthesized speech output. The text editing module in the system includes a text editing window, in which users can modify, delete or add the text content converted from audio tags. Each word and audio clip corresponds to the input tag of the large language model (LLM) one by one, which greatly improves the freedom of speech editing and enables users to flexibly adjust the speech content. Finally, the speech decoder uses the conditional stream matching algorithm to decode the unified semantic tags into synthesized audio. Specifically, the fused sound features serve as the control conditions of the conditional stream matching algorithm, guiding the algorithm to adjust various aspects of the speech when synthesizing audio, and can efficiently convert the unified semantic tags into synthesized audio, ensuring that the synthesized speech is not only semantically accurate, but also more natural and fluent in sound quality and expression, thereby providing high-quality speech editing and synthesis effects.

[0080] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above method embodiment is based on a text visualization voice editing system. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0081] Based on the same technical concept, a computer device is also provided. Figure 4As shown, it is a structural diagram of a computer device provided in an embodiment of the present invention, including a processor, a memory and a bus. Among them, the memory is used to store execution instructions, including internal memory and external memory; the internal memory here is also called internal memory, which is used to temporarily store the operation data in the processor, and the data exchanged with the external memory such as the hard disk, and the processor exchanges data with the external memory through the internal memory. The memory is specifically used to store and execute the program logic code corresponding to the text visualization voice editing system in the embodiment of the present invention, and the execution is controlled by the processor. That is, when the computer device is running, the processor and the memory communicate through the bus, so that the processor executes the application code stored in the memory, and then controls the complete reconstruction of a text visualization voice editing system in the embodiment. The processor may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit, a network processor, etc.; it may also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The disclosed methods, steps and logic block diagrams in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may be, but is not limited to, a random access memory, a read-only memory, a programmable read-only memory, an erasable read-only memory, an electrically erasable read-only memory, etc.

[0082] It can be understood that the above embodiments and Figure 4 The structure illustrated in the figure does not constitute a specific limitation on the computer device. In actual use, the computer device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A text visualization voice editing method, characterized in that: include: Extracting features from the input original audio to obtain a first sound feature; Input the second sound feature, perform multi-layer linear layer mapping processing on the first sound feature and the second sound feature, and generate an audio tag; Convert the audio mark to corresponding text content through reverse marking, and perform text editing on the text content; Generate corresponding text tags for all texts changed during the editing process; Convert the generated text tags and mapped audio tags into unified semantic tags; The first sound feature and the second sound feature are feature-fused, and the fused sound features are decoded into a synthetic audio with a unified semantic tag through a conditional stream matching algorithm to obtain a speech editing result.

2. A text visualization voice editing method according to claim 1, characterized in that: Before extracting features from the input original audio to obtain the first sound feature, a voice activity detection algorithm is used to identify valid vocal segments in the input audio to obtain de-redundant audio segments, and the de-redundant audio segments are divided and padded according to a block length parameter to output original audio with a uniform length.

3. A text visualization voice editing method according to claim 1, characterized in that: The first sound feature and the second sound feature are subjected to multi-layer linear layer mapping processing to generate an audio tag, including: Inputting the first sound feature and the second sound feature into a Transformer model, wherein the Transformer model comprises multiple Transformer layers; In each Transformer layer, the internal dependencies of the input sound feature sequence are modeled through the self-attention mechanism; Information fusion between sound feature sequences through cross-attention mechanism; The feedforward neural network performs nonlinear transformation and feature extraction on the fused features; The fused sound features obtained after being processed by the multi-layer Transformer layer are input into the multi-layer linear layer. Through the mapping operation of the multi-layer linear layer, the features are gradually mapped to the feature space of the audio tag; The mapped features are classified through the output layer to generate corresponding audio tags.

4. A text visualization voice editing method according to any one of claims 1 to 3, characterized in that: The first sound feature includes one of a filter bank feature, a Mel frequency cepstral coefficient feature and a logarithmic Mel frequency spectrum feature; the second sound feature is a basic feature extracted from an audio signal, and the basic feature includes audio volume, pitch and timbre.

5. A text visualization voice editing system, characterized in that: include: A speech encoder, used for extracting features from the input original audio to obtain a first sound feature; A speech encoder, configured to input a second sound feature, perform multi-layer linear layer mapping processing on the first sound feature and the second sound feature, and generate an audio tag; A reverse tagger, used for converting the audio tag into corresponding text content through reverse tagging; A text editing module, including a text editing window, for editing text content; The text marker is used to generate corresponding text markers for all the changed texts during the editing process; A large model module for converting the generated text tags and mapped audio tags into unified semantic tags; The speech decoder is used to fuse the first sound feature and the second sound feature. The fused sound features are decoded into synthetic audio using a conditional stream matching algorithm to obtain a speech editing result.

6. A text visualization voice editing system according to claim 5, characterized in that: The speech encoder includes a Transformer model, which includes multiple layers of Transformer layers and multiple layers of linear layers, and each Transformer layer includes a self-attention layer, a cross-attention layer, and a feedforward neural network layer.

7. A text visualization voice editing system according to claim 5, characterized in that: The large model module includes one of the large language models Qwen2.5-7B and GLM4-9B.

8. A text visualization voice editing system according to claim 5, characterized in that: It also includes a prompt word module, which is used to guide the large model module to generate expected output according to different instructions through preset prompt word content.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a text visualization voice editing method according to any one of claims 1 to 5 is implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for visualizing and voice editing text as described in any one of claims 1 to 5 is implemented.

Images