Data processing, training of a speech synthesis model, speech synthesis method, apparatus, device, readable storage medium and program product

Abstract

The application relates to a data processing method, a speech synthesis model training method, a speech synthesis method, a device, a readable storage medium and a program product. The method comprises the following steps: obtaining original audio data; performing speech transcription on the original audio data to obtain original transcription text; identifying paralanguage events in the original audio data based on a multi-modal processing model to generate paralanguage labels, and inserting the paralanguage labels into corresponding positions of the paralanguage events in the original transcription text to obtain target transcription text. The method can improve the robustness of speech synthesis.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing, speech synthesis model training, speech synthesis method, apparatus, device, readable storage medium, and program product. Background Technology

[0002] Currently, in order to improve the expressiveness of speech synthesis, paralinguistic information such as laughter, breathing sounds, and coughing sounds is usually added to the training text during the text-to-speech (TTS) process.

[0003] In related technologies, multimodal models are typically used to perform acoustic event detection (AED) on the training audio to identify paralinguistic information and corresponding start and end timestamps in the training text. Then, paralinguistic tags are inserted into the transcribed text output by Automatic Speech Recognition (ASR) based on the start and end timestamps. Finally, the transcribed text with inserted paralinguistic tags is used to train a text-to-speech model.

[0004] However, relying on start and end timestamp alignment mechanisms to insert sub-language tags into transcribed text introduces additional complexity and potential accuracy bottlenecks. The prediction accuracy of start and end timestamps directly affects the accuracy of tag insertion; deviations in start and end timestamps may lead to incorrect tag placement, thus affecting the naturalness of the final synthesized speech. Therefore, current speech synthesis methods are not very robust. Summary of the Invention

[0005] Therefore, it is necessary to provide a data processing, speech synthesis model training, speech synthesis method, device, equipment, readable storage medium, and program product that can improve the robustness of speech synthesis in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a data processing method, including:

[0007] Obtain the raw audio data;

[0008] The original audio data is transcribed into speech to obtain the original transcribed text;

[0009] Based on a multimodal processing model, sub-language events in the original audio data are identified to generate sub-language tags, and the sub-language tags are inserted into the corresponding positions of the sub-language events in the original transcribed text to obtain the target transcribed text.

[0010] In one embodiment, the method further includes:

[0011] Based on the multimodal processing model, acoustic feature recognition is performed on the original audio data to obtain at least one acoustic descriptive feature of the original audio data;

[0012] Based on the at least one acoustic description feature, a speech description text in a preset format is generated.

[0013] In one embodiment, after generating the speech description text in a preset format, the method further includes:

[0014] The target transcribed text and the speech description text are concatenated to obtain the input text sequence;

[0015] Acoustic features are extracted from the original audio data to obtain an acoustic feature sequence;

[0016] Training sample data pairs are generated based on the input text sequence and the acoustic feature sequence.

[0017] In one embodiment, concatenating the target transcribed text and the speech description text to obtain the input text sequence includes:

[0018] The word sequence corresponding to the speech description text, the word sequence corresponding to the start symbol, and the word sequence corresponding to the target transcribed text are sequentially concatenated to obtain the concatenated text.

[0019] The concatenated text is segmented into words to obtain the input text sequence.

[0020] In one embodiment, after the original audio data is transcribed into speech to obtain the original transcribed text, the method further includes:

[0021] Based on the multimodal processing model, semantic recognition is performed on the original audio data to obtain acoustic semantic understanding information, and semantic recognition is performed on the original transcribed text to obtain text semantic understanding information;

[0022] Based on the acoustic semantic understanding information and the text semantic understanding information, the original transcribed text is calibrated to obtain the calibrated transcribed text;

[0023] The step of inserting the sub-language tag into the corresponding position of the sub-language event in the original transcribed text to obtain the target transcribed text includes:

[0024] The sub-language tag is inserted into the corresponding position of the sub-language event in the calibrated transcribed text to obtain the target transcribed text.

[0025] Secondly, this application provides a training method for a speech synthesis model, including:

[0026] Based on the training sample data pairs, the speech synthesis model in the initial state is trained to obtain the trained speech synthesis model; wherein, the training sample data pairs are generated by the data processing method of the first aspect.

[0027] Thirdly, this application provides a speech synthesis method, including:

[0028] Receive text data of the target speech to be synthesized and prompt words, wherein the prompt words are descriptive text of the target speech;

[0029] The text data and the prompt words are concatenated into a speech reasoning input sequence;

[0030] Based on a pre-trained speech synthesis model, speech is synthesized from the speech inference input sequence to obtain the acoustic feature sequence of the target speech; the speech synthesis model is trained using the speech synthesis model training method of the second aspect.

[0031] The acoustic feature sequence is synthesized to generate a speech waveform file of the target speech.

[0032] Fourthly, this application provides a data processing apparatus, comprising:

[0033] The acquisition module is used to acquire raw audio data;

[0034] The transcription module is used to perform speech transcription on the original audio data to obtain the original transcribed text;

[0035] The model processing module is used to identify sub-language events in the original audio data based on a multimodal processing model, generate sub-language tags, and insert the sub-language tags into the corresponding positions of the sub-language events in the original transcribed text to obtain the target transcribed text.

[0036] In one embodiment, the model processing module is further configured to perform acoustic feature recognition on the original audio data based on the multimodal processing model to obtain at least one acoustic descriptive feature of the original audio data; and generate a speech description text in a preset format based on the at least one acoustic descriptive feature.

[0037] In one embodiment, the model processing module is further configured to concatenate the target transcribed text and the speech description text to obtain an input text sequence; extract acoustic features from the original audio data to obtain an acoustic feature sequence; and generate training sample data pairs based on the input text sequence and the acoustic feature sequence.

[0038] In one embodiment, the model processing module is further configured to sequentially concatenate the lexical sequence corresponding to the speech description text, the lexical sequence corresponding to the start symbol, and the lexical sequence corresponding to the target transcribed text to obtain the concatenated text; and to perform word segmentation on the concatenated text to obtain the input text sequence.

[0039] In one embodiment, the model processing module is further configured to perform semantic recognition on the original audio data based on the multimodal processing model to obtain acoustic semantic understanding information, and perform semantic recognition on the original transcribed text to obtain text semantic understanding information; calibrate the original transcribed text according to the acoustic semantic understanding information and the text semantic understanding information to obtain calibrated transcribed text; and insert the sub-language tag into the corresponding position of the sub-language event in the calibrated transcribed text to obtain the target transcribed text.

[0040] Fifthly, this application provides a training device for a speech synthesis model, comprising:

[0041] The sample acquisition module is used to acquire training sample data pairs;

[0042] The model training module is used to train the speech synthesis model in the initial state based on training sample data pairs to obtain the trained speech synthesis model; wherein, the training sample data pairs are generated by the data processing method of the first aspect.

[0043] Sixthly, this application provides a speech synthesis apparatus, comprising:

[0044] A receiving module is used to receive text data of the target speech to be synthesized and prompt words, wherein the prompt words are descriptive text of the target speech;

[0045] The concatenation module is used to concatenate the text data and the prompt words into a speech reasoning input sequence;

[0046] The recognition module is used to synthesize speech from the speech inference input sequence based on a pre-trained speech synthesis model to obtain the acoustic feature sequence of the target speech; the speech synthesis model is trained using the speech synthesis model training method of the second aspect.

[0047] The synthesis module is used to synthesize the acoustic feature sequence to generate a speech waveform file of the target speech.

[0048] Fifthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the data processing method of the first aspect, the training method of the speech synthesis model of the second aspect, or the speech synthesis method of the third aspect.

[0049] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data processing method of the first aspect, the training method for the speech synthesis model of the second aspect, or the speech synthesis method of the third aspect.

[0050] In a seventh aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method of the first aspect, the training method of the speech synthesis model of the second aspect, or the speech synthesis method of the third aspect.

[0051] The aforementioned data processing, speech synthesis model training, speech synthesis methods, devices, equipment, readable storage media, and program products acquire raw audio data; perform speech-to-text transcription on the raw audio data to obtain raw transcribed text; based on a multimodal processing model, identify paralinguistic events in the raw audio data to generate paralinguistic tags, and insert the paralinguistic tags into the corresponding positions of the paralinguistic events in the raw transcribed text to obtain the target transcribed text. Since both the identification of paralinguistic events and the insertion of paralinguistic tags into the raw transcribed text are achieved through the multimodal processing model, the paralinguistic tags can be directly inserted into the raw transcribed text during the process of the multimodal processing model identifying the raw audio data to obtain paralinguistic tags, eliminating the need for prediction and alignment of the start timestamp after the multimodal processing model outputs the paralinguistic events. This reduces the accuracy errors introduced by the prediction and alignment of the start timestamp, improving the robustness of speech synthesis. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 An application environment diagram of a data processing method provided in an embodiment of this application;

[0054] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0055] Figure 3 This is a schematic diagram of the structure of a multimodal processing model provided in an embodiment of this application;

[0056] Figure 4 A schematic diagram of an input text sequence provided in an embodiment of this application;

[0057] Figure 5 A flowchart illustrating a training method for a speech synthesis model provided in an embodiment of this application;

[0058] Figure 6 A schematic flowchart of a speech synthesis method provided in an embodiment of this application;

[0059] Figure 7 A structural block diagram of a data processing apparatus provided in an embodiment of this application;

[0060] Figure 8 A structural block diagram of a training device for a speech synthesis model provided in an embodiment of this application;

[0061] Figure 9 A structural block diagram of a speech synthesis device provided in an embodiment of this application;

[0062] Figure 10 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] The relevant technologies will be explained below.

[0065] Currently, in order to improve the expressiveness of speech synthesis, paralinguistic information such as laughter, breathing sounds, and coughing sounds are usually added to the training text during the text-to-speech process.

[0066] In related technologies, when training a text-to-speech (TTS) model, the first step is to collect a pre-recorded, high-quality audio and video dataset and pre-align the training audio and corresponding transcribed text in the dataset. Next, a multimodal model is used to perform Automatic Language Encoding (AED) on the training audio in the audio and video dataset, identifying when paralinguistic events such as laughter and inhalation occur in the training audio, and outputting the identified paralinguistic events along with their start and end timestamps. Then, an Automatic Speech Recognition (ASR) model is used to recognize the training audio and output the transcribed text. An alignment and arbitration module then inserts paralinguistic tags (e.g., laughter tags, inhalation tag, etc.) into the corresponding positions in the transcribed text based on the start and end timestamps. For example, if laughter is detected between 2.5 and 3.0 seconds, and this start and end timestamp is after word A in the transcribed text, a laughter tag is inserted after word A. Finally, the transcribed text with inserted paralinguistic tags is used as input, and the training audio is used as output to train the TTS model. In this way, the TTS model can synthesize the corresponding paralinguistic voice when a specific tag appears in the text.

[0067] However, the speech synthesis methods in related technologies have the following drawbacks:

[0068] First, the alignment mechanism relying on the start timestamp introduces additional complexity and potential accuracy bottlenecks. Whether the multimodal model directly predicts the start timestamp or performs timestamp alignment, it significantly increases the complexity of the system. Simultaneously, the accuracy of the start timestamp prediction directly affects the accuracy of paralinguistic label insertion; any slight deviation can lead to incorrect paralinguistic label placement, thus affecting the naturalness of the final synthesized speech and the robustness of speech synthesis.

[0069] Among these technologies, speech synthesis methods suffer from complex data processing and high computational costs, making them difficult to scale. Because they rely on pre-prepared high-quality datasets and require tools such as aligners (e.g., the Montreal Forced Aligner, MFA) for forced alignment of speech and text during processing, the data preprocessing is complex, time-consuming, and computationally expensive. This makes it difficult to scale to hundreds of thousands of hours of unlabeled audio data collected from the real world, severely limiting scalability.

[0070] Furthermore, the speech synthesis methods in related technologies do not fully utilize the capabilities of multimodal models, resulting in technological redundancy. Simply using multimodal models for basic acoustic event detection greatly wastes their comprehensive capabilities in text understanding, content calibration, contextual reasoning, and multi-dimensional feature description of speech.

[0071] Finally, the limited control dimension of speech synthesis restricts the expressive potential of the synthesized speech. Because speech synthesis only references a few paralinguistic events, the final trained TTS model can only synthesize corresponding voices based on the presence of discrete paralinguistic tags in the text. This limits the guidance and control of speech generation through higher-dimensional, finer-grained natural language descriptions. For example, it cannot achieve complex acoustic descriptions such as "reading with a more stable, magnetic middle-aged male voice" or "expressing with a light, whispering style" in the synthesized speech, resulting in a lower upper limit for the final expressiveness and controllability of the synthesized speech.

[0072] To address the aforementioned issues, embodiments of this application provide a data processing, speech synthesis model training, speech synthesis method, apparatus, device, readable storage medium, and program product. The identification of paralinguistic events and the insertion of paralinguistic tags into the original transcribed text are both achieved through a multimodal processing model. This allows for the direct insertion of paralinguistic tags into the original transcribed text during the process of the multimodal processing model identifying the original audio data and obtaining paralinguistic tags, eliminating the need for prediction and alignment of the start timestamp after the multimodal processing model outputs the paralinguistic events. This reduces the accuracy errors introduced by the prediction and alignment of the start timestamps and improves the robustness of speech synthesis.

[0073] The application scenarios of the data processing method provided in the embodiments of this application will be described below.

[0074] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0075] Terminal 102 can first send raw audio data to server 104. Then, server 104 performs speech-to-text transcription on the raw audio data to obtain raw transcribed text. Server 104 then uses a multimodal processing model to identify paralinguistic events in the raw audio data, generates paralinguistic tags, and inserts the paralinguistic tags into the corresponding positions of the paralinguistic events in the raw transcribed text to obtain the target transcribed text.

[0076] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0077] In one exemplary embodiment, such as Figure 2 As shown, a data processing method is provided, which can be applied to... Figure 1 Taking the server in the example, this data processing method includes S201 to S203:

[0078] S201. Obtain the raw audio data.

[0079] In some embodiments, the aforementioned raw audio data can be collected when training the speech synthesis model is required, or it can be pre-collected and stored in a database on a server; this application embodiment does not impose any limitations on this. After collecting a preset number (e.g., one thousand or ten thousand) of raw audio data, a raw audio dataset can be constructed for subsequent data processing.

[0080] For example, the raw audio data may include multiple target data sources. These target data sources can be any data source; there may be one or more target data sources. They can include data sources from public networks or data sources from specific databases. For example, target data sources may include podcast platforms, audiobook websites, video sharing websites, etc., and the preset quantity of raw audio data may be hundreds of thousands of hours.

[0081] In some embodiments, the original audio data can be any audio data, and this application does not limit this. That is, the source, language, quality, and format of the original audio data are not limited.

[0082] S202. Perform speech-to-text transcription on the original audio data to obtain the original transcribed text.

[0083] In some embodiments, after obtaining raw audio data from a target data source, each raw audio data can be input into a preset ASR model, so that the ASR model can perform speech transcription on each raw audio data to generate a raw transcribed text corresponding to each raw audio data, which contains timestamp information.

[0084] In some embodiments, after generating the original transcribed text corresponding to each original audio data, each original audio data and its corresponding original transcribed text can be combined into a text-audio data pair, and multiple text-audio data pairs can be combined into a data pair set.

[0085] For example, if the original audio data is a 10-second audio file "audio_001.wav" and the speech content of the original audio data is "Hello, world", then the ASR model will perform speech-to-text transcription on the original audio data to obtain the original transcribed text "Hello, world", and the original audio data "audio_001.wav" and the original transcribed text "Hello, world" will be combined into a text-audio data pair.

[0086] It should be noted that the ASR model only performs speech transcription, and the original transcribed text generated by the ASR model includes timestamps and paralinguistic information.

[0087] It should be understood that the embodiments of this application do not limit the ASR model, and can be any model that converts human speech signals into corresponding text files, such as the Whisper model. The ASR model can achieve efficient and accurate transcription of large-scale raw audio data.

[0088] This application's embodiments process massive amounts of unlabeled raw audio data, eliminating the reliance on pre-built high-quality datasets and complex tools such as forced alignment. This reduces the cost and time of data preparation by several orders of magnitude, making it possible to build speech synthesis systems that support ultra-large-scale timbre libraries and multiple languages. It greatly simplifies the data processing workflow and significantly reduces costs, possessing strong potential for large-scale applications.

[0089] S203. Based on the multimodal processing model, identify the sub-language events in the original audio data, generate sub-language tags, and insert the sub-language tags into the corresponding positions of the sub-language events in the original transcribed text to obtain the target transcribed text.

[0090] The target transcribed text includes a text sequence with inserted sub-language tags.

[0091] The aforementioned multimodal processing model can be considered a multimodal omni-based model. Multimodal processing models can simultaneously process and understand information from different modalities (e.g., text, audio, and images) and establish relationships between information from different modalities. Due to their powerful cross-modal understanding capabilities, multimodal processing models can be used for various tasks such as audio event detection and audio / video content analysis.

[0092] For example, after the server performs speech-to-text transcription on the original audio data to obtain text-audio data pairs, it treats each text-audio data pair as a processing unit and inputs it into a pre-trained multimodal processing model so that the multimodal processing model can insert sub-language labels for the processing unit to obtain the target transcribed text.

[0093] For example, Figure 3 This is a schematic diagram of the structure of a multimodal processing model provided in an embodiment of this application, as shown below. Figure 3 As shown, the multimodal processing model includes a paralinguistic insertion unit. This unit identifies paralinguistic events in the original audio data and generates paralinguistic tags. These tags are then inserted into the corresponding positions of the paralinguistic events in the original transcribed text to obtain the target transcribed text.

[0094] Paralinguistic events can include laughter, inhalation sounds, pauses, coughs, and other similar sounds. Correspondingly, the paralinguistic insertion unit can identify the acoustic features of specific paralinguistic events in the original audio data, thereby generating paralinguistic labels.

[0095] For example, since the original transcribed text is obtained by transcribing the original audio data, it contains timestamp information. After the sub-language insertion unit identifies the sub-language events in the original audio data, it can determine the corresponding position of the sub-language events in the original transcribed text based on the timestamp information in the original transcribed text, thereby inserting the sub-language tag into the corresponding position of the sub-language events in the original transcribed text to obtain the target transcribed text.

[0096] In some embodiments, after transcribing the original audio data to obtain the original transcribed text, the server can further perform semantic recognition on the original audio data based on a multimodal processing model to obtain acoustic semantic understanding information, and perform semantic recognition on the original transcribed text to obtain text semantic understanding information. Based on the acoustic and text semantic understanding information, the original transcribed text is then calibrated to obtain calibrated transcribed text. Correspondingly, the server can insert sub-language tags into the corresponding positions of sub-language events in the calibrated transcribed text to obtain the target transcribed text.

[0097] The calibration, which utilizes acoustic and textual semantic understanding information, can correct textual errors in the original transcribed text. Correspondingly, the calibrated transcribed text can be considered the corrected version of the original transcribed text.

[0098] Continue to refer to Figure 3 The multimodal processing model also includes a text calibration unit. For example, after receiving the input text-audio data pair, the text calibration unit first analyzes and processes the original transcribed text based on its acoustic understanding of the audio data and its semantic understanding of the original transcribed text, identifying and correcting any possible textual errors to obtain the calibrated transcribed text. Subsequently, the paralinguistic insertion unit identifies paralinguistic events in the audio data and generates corresponding paralinguistic tags. Finally, the paralinguistic insertion unit inserts the paralinguistic tags into the corresponding positions of the paralinguistic events in the calibrated transcribed text to obtain the target transcribed text.

[0099] In some embodiments, the server may also perform acoustic feature recognition on the raw audio data based on a multimodal processing model to obtain at least one acoustic descriptive feature of the raw audio data. Subsequently, the server generates a speech description text in a preset format based on the at least one acoustic descriptive feature.

[0100] The speech description text is used to indicate acoustic description features.

[0101] For example, the voice description text could be something like: "{"transcription_text": "The weather is so nice today [laughter], let's go for a walk.", "timbre_desc": "Middle-aged male, with a slightly hoarse and magnetic voice", "style_desc": "Narrative style, slow and steady speech", "emotion_desc": "Calm and slightly serious"}.

[0102] For example, continue to refer to Figure 3 The multimodal processing model also includes a speech description generation unit. This unit performs acoustic feature recognition on the raw audio data to obtain at least one acoustic description feature. Subsequently, based on a preset structured data format, the unit uses this at least one acoustic description feature as a key-value pair to generate speech description text.

[0103] Among them, the above acoustic description features may include timbre features, speech rate style features, emotional state features, prosody rhythm features, etc. The above structured data format may include the JSON (JavaScript Object Notation) format, and each acoustic description feature may be used as a key-value pair in the JSON format. In addition, the above structured data format may also be the Extensible Markup Language (XML) format, the YAML (YAML Ain't Markup Language) format, or other custom formats.

[0104] It should be noted that the key-value pairs in the JSON format are extensible, and more description dimensions can be added according to application requirements, that is, the acoustic description features can be increased or decreased according to the actual situation. For example, a scene description feature "Scene description: indoors, with slight reverberation" can be added.

[0105] Exemplarily, after receiving the input text-audio data pair, the multimodal processing model can perform multi-dimensional acoustic feature recognition on the original audio data in the text-audio data pair by the speech description generation unit, so as to obtain at least one acoustic description feature of the original audio data. Subsequently, according to the structured data format, the at least one identified acoustic description feature is respectively used as a key-value pair to generate a speech description text.

[0106] Among them, the processing instructions or training objectives of the speech description text generated by the above multimodal processing model may be, for example, "Please analyze the input text-audio data pair, and generate a speech description text in the JSON format according to its timbre, style, and emotion. The JSON contains the following key-value pairs: "timbre_desc" (timbre description), "style_desc" (style description), "emotion_desc" (emotion description), and "transcription_text" (transcribed text, including paralinguistic tags)".

[0107] Exemplarily, the input text-audio data pair is "audio_001.wav, hello, world". Correspondingly, through the multimodal processing model, the output target transcribed text is "hello, world [laughter]", and the output speech description text is "{\"timbre_desc\":\"Young male, loud voice, full of vitality\", \"style_desc\":\"Moderate speech rate, fluent expression\", \"emotion_desc\":\"Happy, confident, infectious\", \"transcription_text\":\"hello, world [laughter]\"}". <(

[0108] It should be noted that the multimodal processing model can first generate the target transcribed text and then generate the speech description text, or the multimodal processing model can first generate the speech description text and then generate the target transcribed text, or the multimodal processing model can generate both the target transcribed text and the speech description text simultaneously. This application embodiment does not impose any restrictions on this.

[0109] In this embodiment, the multimodal processing model directly calibrates and inserts sub-language tags into the original transcribed text, thereby transforming the complex, timestamp-based cross-modal alignment problem into a simpler and more efficient text-domain information processing problem. By avoiding the accuracy errors that may be introduced during timestamp prediction and alignment, the entire technical chain is shorter and more direct, significantly improving the system's robustness and operational efficiency. Furthermore, the multimodal processing model is elevated from a simple acoustic event detector to a model integrating content calibration, event annotation, and multi-dimensional feature description. This not only makes fuller use of the powerful cross-modal understanding capabilities of the multimodal processing model but also opens up new paths for its in-depth application in the field of speech processing.

[0110] In some embodiments, after obtaining the target transcribed text and speech description text corresponding to the text-audio data pair through the multimodal processing model, model training samples can be obtained based on the target transcribed text and speech description text.

[0111] It should be understood that the embodiments of this application do not limit how the model training samples are obtained from the target transcribed text and the speech description text. In some embodiments, the server concatenates the target transcribed text and the speech description text to obtain an input text sequence. Subsequently, the server extracts acoustic features from the original audio data to obtain an acoustic feature sequence. Finally, the server generates training sample data pairs based on the input text sequence and the acoustic feature sequence.

[0112] The acoustic feature sequence may include the Mel spectrum sequence.

[0113] For example, the server can sequentially concatenate the word sequence corresponding to the speech description text, the word sequence corresponding to the start symbol, and the word sequence corresponding to the target transcribed text to obtain the concatenated text. Then, the server performs word segmentation on the concatenated text to obtain the input text sequence.

[0114] For example, Figure 4 This is a schematic diagram of an input text sequence provided in an embodiment of this application, such as... Figure 4As shown, the server can obtain the target transcribed text and speech description text corresponding to each text audio data pair, and sequentially concatenate the token sequence of the speech description text, the token of a start symbol (SOS), and the token sequence of the target transcribed text to form a unified input text sequence.

[0115] For example, the server can also extract acoustic features from the original audio file, converting it into an acoustic feature sequence, which serves as the target output sequence for training the speech synthesis model. Subsequently, the server pairs the input text sequence and the extracted acoustic features to form a training sample data pair, ultimately creating a structured training dataset.

[0116] For example, if the calibration transcription text is "Hello, world [laughter]", the speech description text is "{"timbre_desc": "young male, loud and energetic voice", ...}", and the original audio file is "audio_001.wav", then the input text sequence in the model training samples can be "[token_of_timbre_desc] [token_of_style_desc] [token_of_emotion_desc] [SOS] [token_of_hello] [token_of_,] [token_of_world] [token_of_laughter]", and the acoustic feature sequence can be the Mel spectrum sequence corresponding to "audio_001.wav".

[0117] The data processing provided in this application involves acquiring raw audio data; performing speech-to-text transcription on the raw audio data to obtain raw transcribed text; and, based on a multimodal processing model, identifying paralinguistic events in the raw audio data to generate paralinguistic tags, and inserting the paralinguistic tags into the corresponding positions of the paralinguistic events in the raw transcribed text to obtain the target transcribed text. Since both the identification of paralinguistic events and the insertion of paralinguistic tags into the raw transcribed text are achieved through a multimodal processing model, the paralinguistic tags can be directly inserted into the raw transcribed text during the process of the multimodal processing model identifying the raw audio data to obtain paralinguistic tags. This eliminates the need for prediction and alignment of the start timestamp after the multimodal processing model outputs the paralinguistic events, thereby reducing the accuracy errors introduced by the prediction and alignment of the start timestamp and improving the robustness of speech synthesis.

[0118] Based on the above embodiments, Figure 5 A flowchart illustrating a training method for a speech synthesis model provided in this application embodiment is shown below. Figure 5 As shown, taking the application of this method to a server as an example, the training method of this speech synthesis model includes S301:

[0119] S301. Based on the training sample data pairs, train the speech synthesis model in the initial state to obtain the trained speech synthesis model.

[0120] The training sample data is generated using the data processing methods described in the above embodiments.

[0121] In some embodiments, after obtaining training sample data pairs, the server can use the training sample data pairs to train the initial speech synthesis model to obtain the trained speech synthesis model.

[0122] For example, after constructing a structured training dataset using training sample data, the initial speech synthesis model can be trained using the training sample data in the structured training dataset to obtain the trained speech synthesis model.

[0123] The initial speech synthesis model described above can be an untrained speech synthesis model. This speech synthesis model can be a Large Language Model (LLM) for TTS, employing a sequence-to-sequence architecture. The training objective for training the initial model can be to generate acoustic feature sequences that match the acoustic feature sequences in the model's training samples.

[0124] It should be understood that TTS models can convert text information into human speech. Based on neural networks, combined with TTS technology, TTS models can improve the naturalness, expressiveness, and timbre diversity of synthesized speech.

[0125] In this application, after training an initial-state speech synthesis model using a training dataset, the initial-state speech synthesis model can learn to generate corresponding acoustic feature sequences, thus obtaining a speech synthesis model. Subsequently, the speech inference input sequence is input into the speech synthesis model, which then generates the corresponding acoustic feature sequence based on this speech inference input sequence.

[0126] In this embodiment, the upstream multimodal processing model and the downstream speech synthesis model share the same structured description system. The structured speech description text output by the multimodal processing model can be directly and seamlessly used as the control input of the speech synthesis model, forming an efficient and unified data flow. This not only ensures the consistency between data annotation and speech synthesis but also lays the foundation for building a more powerful end-to-end speech generation model.

[0127] Figure 6 This is a flowchart illustrating a speech synthesis method provided in an embodiment of this application, as shown below. Figure 6 As shown, taking the application of this method to a server as an example, the speech synthesis method includes S401 to S404:

[0128] S401. Receive the text data of the target speech to be synthesized and the prompt words, where the prompt words are the descriptive text of the target speech.

[0129] In this application, when a user needs to convert a piece of text into target speech, they can input the text data of the target speech to be synthesized and the prompt word in the terminal, thereby causing the terminal to send a speech synthesis request carrying the text data of the target speech to be synthesized and the prompt word to the server.

[0130] For example, the target speech can include any segment of speech, and this application embodiment does not limit this. The text data can record the text corresponding to the target speech, and it may or may not carry secondary language tags. For example, secondary language tags can include laughter tags, inhalation sound tags, cough sound tags, etc.

[0131] For example, the aforementioned prompt words can be structured prompt words. Structured prompt words are used to indicate at least one acoustic descriptive feature of the target speech. Acoustic descriptive features may include, for example, timbre features, speech rate style features, emotional state features, prosodic rhythm features, etc. It should be noted that this application embodiment does not limit the data format of the structured prompt words; for example, the data format of the structured prompt words may include JSON (simplified musical notation).

[0132] For example, the text data in the speech synthesis request could be, for instance, “The weather is really nice today.”, and the structured prompts in the speech synthesis request could be, for instance, “{“timbre_desc”: “young woman with a sweet voice”, “style_desc”: “lively with a brisk speaking pace”, “emotion_desc”: “happy”}.

[0133] In this embodiment, by adding prompt words to the speech synthesis request to indicate at least one acoustic descriptive feature of the target speech, the user can flexibly and in combination describe multiple acoustic descriptive features such as timbre, style, and emotion using natural language during model inference. This enables refined and personalized control over speech synthesis, ensuring that the generated target speech is not limited to a single dimension of a few paralinguistic events, thus greatly enriching the expressiveness of synthesized speech.

[0134] S402. Concatenate the text data and prompt words into a speech reasoning input sequence.

[0135] In this step, after the server receives a speech synthesis request, it can concatenate the text data and prompts into a speech reasoning input sequence.

[0136] In some embodiments, the server may extract at least one acoustic descriptive feature value from the prompt words. Then, the server sequentially concatenates the lexical sequence corresponding to the at least one acoustic descriptive feature value, the lexical sequence corresponding to the start symbol, and the lexical sequence corresponding to the text data of the target speech to obtain the concatenated text. Finally, the server performs word segmentation on the concatenated text to obtain the speech inference input sequence.

[0137] For example, for a prompt word, the service can extract at least one acoustic descriptive feature value and concatenate them into a long text string, with the word sequence corresponding to the acoustic descriptive feature value first and the word sequence corresponding to the text data last. The word sequence corresponding to the acoustic descriptive feature value and the word sequence corresponding to the text data are separated by the word sequence corresponding to the start character (e.g., [SOS]). Subsequently, the server performs tokenization on the concatenated text to form a speech reasoning input sequence.

[0138] For example, the text data in the speech synthesis request could be, for instance, "The weather is really nice today.", and the structured prompts in the speech synthesis request could be, for instance, "{"timbre_desc": "young woman, sweet voice", "style_desc": "lively, brisk speech", "emotion_desc": "happy"}. Correspondingly, the concatenated speech inference input sequence could be, for instance, "[token_of_timbre_desc] [token_of_style_desc] [token_of_emotion_desc] [SOS] [token_of_today] [token_of_weather] [token_of_good] [token_of_."

[0139] S403. Based on the pre-trained speech synthesis model, perform speech synthesis on the speech inference input sequence to obtain the acoustic feature sequence of the target speech.

[0140] The speech synthesis model is trained using the same training method as the aforementioned speech synthesis model. For example, the training sample data pairs in the training dataset are generated from the target transcribed text and speech description text obtained by processing text-audio data pairs using a multimodal processing model. The target transcribed text includes text sequences with inserted sub-language labels, and the speech description text is used to indicate acoustic descriptive features.

[0141] S404. Synthesize the acoustic feature sequence to generate a speech waveform file of the target speech.

[0142] In this step, after the server obtains the acoustic feature sequence of the target speech synthesized by the speech synthesis model, it can synthesize the acoustic feature sequence into a speech waveform file of the target speech.

[0143] In some embodiments, the server can synthesize the acoustic feature sequence into a final speech waveform file using a vocoder.

[0144] For example, the text data in the speech synthesis request could be, for instance, "The weather is really nice today.", and the structured prompts in the speech synthesis request could be, for instance, "{"timbre_desc": "young woman, sweet voice", "style_desc": "lively, brisk speech", "emotion_desc": "happy"}. Accordingly, the generated speech waveform file could be a speech waveform file of "The weather is really nice today," synthesized from the sweet voice of a young woman, lively and brisk, expressing happiness.

[0145] It should be noted that the specific model selection for the speech synthesis model and vocoder can be replaced with other existing models, and this application embodiment does not impose any restrictions on this. Furthermore, the multimodal model and speech synthesis model involved in this application embodiment can be synthesized and trained into a unified, end-to-end model. This model receives text data and structured prompts as input, directly outputs high-quality speech waveform files, and implicitly completes all processes such as calibration, description, and generation internally.

[0146] The speech synthesis method provided in this application receives text data of the target speech to be synthesized and prompt words, where the prompt words are descriptive text of the target speech; concatenates the text data and prompt words into a speech inference input sequence; based on a pre-trained speech synthesis model, performs speech synthesis on the speech inference input sequence to obtain an acoustic feature sequence of the target speech; the speech synthesis model is trained using the aforementioned speech synthesis model training method; and synthesizes the acoustic feature sequence to generate a speech waveform file of the target speech. Since both the input speech inference input sequence during multimodal processing model inference and the input text-audio data pair during training include text and audio, the multimodal processing model can directly insert sub-language tags into the text data during the process of recognizing audio data to obtain sub-language tags. This eliminates the need for prediction and alignment of the start timestamp after the multimodal processing model outputs the sub-language event, thereby reducing the accuracy error introduced by the prediction and alignment of the start timestamp, reducing the processing links for sub-language tag insertion, and improving the robustness of speech synthesis.

[0147] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0148] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0149] In one exemplary embodiment, such as Figure 7 As shown, a data processing device 500 is provided, including: an acquisition module 501, a transcription module 502, and a model processing module 503, wherein:

[0150] The acquisition module 501 is used to acquire raw audio data.

[0151] The transcription module 502 is used to transcribe the original audio data into speech to obtain the original transcribed text.

[0152] The model processing module 503 is used to identify the sub-language events in the original audio data based on the multimodal processing model, generate sub-language tags, and insert the sub-language tags into the corresponding positions of the sub-language events in the original transcribed text to obtain the target transcribed text.

[0153] In one embodiment, the model processing module 503 is further configured to perform acoustic feature recognition on the original audio data based on a multimodal processing model to obtain at least one acoustic descriptive feature of the original audio data; and generate a speech description text in a preset format based on the at least one acoustic descriptive feature.

[0154] In one embodiment, the model processing module 503 is further configured to concatenate the target transcribed text and the speech description text to obtain an input text sequence; extract acoustic features from the original audio data to obtain an acoustic feature sequence; and generate training sample data pairs based on the input text sequence and the acoustic feature sequence.

[0155] In one embodiment, the model processing module 503 is further configured to sequentially concatenate the word sequence corresponding to the speech description text, the word sequence corresponding to the start symbol, and the word sequence corresponding to the target transcribed text to obtain the concatenated text; and to perform word segmentation on the concatenated text to obtain the input text sequence.

[0156] In one embodiment, the model processing module 503 is further configured to perform semantic recognition on the original audio data based on a multimodal processing model to obtain acoustic semantic understanding information, and perform semantic recognition on the original transcribed text to obtain text semantic understanding information; calibrate the original transcribed text according to the acoustic semantic understanding information and the text semantic understanding information to obtain calibrated transcribed text; and insert sub-language tags into the corresponding positions of sub-language events in the calibrated transcribed text to obtain the target transcribed text.

[0157] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0158] Based on the same inventive concept, this application also provides a training apparatus for a speech synthesis model to implement the training method for the speech synthesis model described above. The solution provided by this apparatus is similar to the implementation described in the above method. Therefore, the specific limitations of one or more speech synthesis model training apparatus embodiments provided below can be found in the limitations of the speech synthesis model training method described above, and will not be repeated here.

[0159] In one exemplary embodiment, such as Figure 8 As shown, a training device 600 for a speech synthesis model is provided, including: a sample acquisition module 601 and a model training module 602, wherein:

[0160] The sample acquisition module 601 is used to acquire training sample data pairs.

[0161] The model training module 602 is used to train the speech synthesis model in the initial state based on training sample data pairs to obtain the trained speech synthesis model; wherein the training sample data pairs are generated by the above data processing method.

[0162] The modules in the training device for the aforementioned speech synthesis model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0163] Based on the same inventive concept, this application also provides a speech synthesis apparatus for implementing the speech synthesis method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more speech synthesis apparatus embodiments provided below can be found in the limitations of the speech synthesis method described above, and will not be repeated here.

[0164] In one exemplary embodiment, such as Figure 9 As shown, a speech synthesis device 700 is provided, including: a receiving module 701, a splicing module 702, a recognition module 703, and a synthesis module 704, wherein:

[0165] The receiving module 701 is used to receive the text data of the target speech to be synthesized and the prompt words, wherein the prompt words are the descriptive text of the target speech.

[0166] The splicing module 702 is used to splice text data and prompt words into a speech reasoning input sequence.

[0167] The recognition module 703 is used to synthesize speech from the speech inference input sequence based on a pre-trained speech synthesis model to obtain the acoustic feature sequence of the target speech; the speech synthesis model is trained using the training method described above.

[0168] The synthesis module 704 is used to synthesize acoustic feature sequences to generate speech waveform files of the target speech.

[0169] The modules in the training device for the aforementioned speech synthesis model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0170] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements a data processing method, a speech synthesis model training method, or a speech synthesis method.

[0171] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0172] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described data processing method, speech synthesis model training method, or speech synthesis method.

[0173] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described data processing method, speech synthesis model training method, or speech synthesis method.

[0174] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the data processing method, the speech synthesis model training method, or the speech synthesis method described above.

[0175] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0176] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0177] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0178] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the raw audio data; The original audio data is transcribed into speech to obtain the original transcribed text; Based on a multimodal processing model, sub-language events in the original audio data are identified to generate sub-language tags, and the sub-language tags are inserted into the corresponding positions of the sub-language events in the original transcribed text to obtain the target transcribed text.

2. The method according to claim 1, characterized in that, The method further includes: Based on the multimodal processing model, acoustic feature recognition is performed on the original audio data to obtain at least one acoustic descriptive feature of the original audio data; Based on the at least one acoustic description feature, a speech description text in a preset format is generated.

3. The method according to claim 2, characterized in that, After generating the speech description text in the preset format, the method further includes: The target transcribed text and the speech description text are concatenated to obtain the input text sequence; Acoustic features are extracted from the original audio data to obtain an acoustic feature sequence; Training sample data pairs are generated based on the input text sequence and the acoustic feature sequence.

4. The method according to claim 3, characterized in that, The step of concatenating the target transcribed text and the speech description text to obtain the input text sequence includes: The word sequence corresponding to the speech description text, the word sequence corresponding to the start symbol, and the word sequence corresponding to the target transcribed text are sequentially concatenated to obtain the concatenated text. The concatenated text is segmented into words to obtain the input text sequence.

5. The method according to claim 1, characterized in that, After performing speech-to-text transcription on the original audio data to obtain the original transcribed text, the method further includes: Based on the multimodal processing model, semantic recognition is performed on the original audio data to obtain acoustic semantic understanding information, and semantic recognition is performed on the original transcribed text to obtain text semantic understanding information; Based on the acoustic semantic understanding information and the text semantic understanding information, the original transcribed text is calibrated to obtain the calibrated transcribed text; The step of inserting the sub-language tag into the corresponding position of the sub-language event in the original transcribed text to obtain the target transcribed text includes: The sub-language tag is inserted into the corresponding position of the sub-language event in the calibrated transcribed text to obtain the target transcribed text.

6. A training method for a speech synthesis model, characterized in that, The method includes: Based on the training sample data pairs, the speech synthesis model in the initial state is trained to obtain the trained speech synthesis model; wherein, the training sample data pairs are generated by the data processing method described in claim 3 or 4.

7. A speech synthesis method, characterized in that, The method includes: Receive text data of the target speech to be synthesized and prompt words, wherein the prompt words are descriptive text of the target speech; The text data and the prompt words are concatenated into a speech reasoning input sequence; Based on a pre-trained speech synthesis model, speech synthesis is performed on the speech inference input sequence to obtain the acoustic feature sequence of the target speech; the speech synthesis model is trained by the method described in claim 6. The acoustic feature sequence is synthesized to generate a speech waveform file of the target speech.

8. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire raw audio data; The transcription module is used to perform speech transcription on the original audio data to obtain the original transcribed text; The model processing module is used to identify sub-language events in the original audio data based on a multimodal processing model, generate sub-language tags, and insert the sub-language tags into the corresponding positions of the sub-language events in the original transcribed text to obtain the target transcribed text.

9. A training device for a speech synthesis model, characterized in that, The device includes: The sample acquisition module is used to acquire training sample data pairs; The model training module is used to train the speech synthesis model in the initial state based on training sample data pairs to obtain the trained speech synthesis model; wherein the training sample data pairs are generated by the data processing method described in claim 3 or 4.

10. A speech synthesis device, characterized in that, The device includes: A receiving module is used to receive text data of the target speech to be synthesized and prompt words, wherein the prompt words are descriptive text of the target speech; The concatenation module is used to concatenate the text data and the prompt words into a speech reasoning input sequence; The recognition module is used to perform speech synthesis on the speech inference input sequence based on a pre-trained speech synthesis model to obtain the acoustic feature sequence of the target speech; the speech synthesis model is trained by the method described in claim 6. The synthesis module is used to synthesize the acoustic feature sequence to generate a speech waveform file of the target speech.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the data processing method according to any one of claims 1-5, or the training method for the speech synthesis model according to claim 6, or the speech synthesis method according to claim 7.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data processing method according to any one of claims 1-5, or the training method for the speech synthesis model according to claim 6, or the speech synthesis method according to claim 7.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the data processing method according to any one of claims 1-5, or the training method for the speech synthesis model according to claim 6, or the speech synthesis method according to claim 7.

Images