A voice parsing method, device, equipment and storage medium thereof

Abstract

The embodiment of the application belongs to the technical field of voice decoupling and is applied to a voice feature analysis scene, and relates to a voice analysis method, device, equipment and storage medium thereof, comprising the following steps: obtaining voice data to be analyzed and inputting the voice data into a trained voice analysis model; extracting timbre information in the voice data to be analyzed through a timbre encoder in the model, extracting prosody information in the voice data to be analyzed based on a prosody encoder in the model; and outputting the timbre information and the prosody information as analysis results. The timbre encoder and the prosody encoder in the voice analysis model are trained in advance, and the timbre information / prosody information is extracted through the trained timbre encoder / prosody encoder subsequently. Compared with a conventional voice analysis method, the text information and the acoustic information are analyzed first, and then the acoustic information is decoupled to obtain the timbre information / prosody information, which improves the voice analysis efficiency to a certain extent and simplifies the analysis steps.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology and is applied in the scenario of speech feature parsing, and in particular to a speech parsing method, apparatus, device and its storage medium. Background Technology

[0002] Speech parsing technology refers to the analysis of speech data to obtain the text content and acoustic information within it. Currently, most speech parsing technologies can extract the text content and acoustic information from speech data. However, for the detailed information contained in the acoustic information, a supervised approach is often used to decouple rhythm, pitch, and tone.

[0003] However, in this approach, for example, there is a repetitive relationship between pitch and tone, and the supervised approach increases the complexity of model building, so it performs poorly in decoupling. It cannot achieve the expected decoupling effect, has low decoupling efficiency, and also increases the difficulty of speech parsing. Summary of the Invention

[0004] The purpose of this application is to provide a speech parsing method, apparatus, device and storage medium to solve the problem that the existing technology performs poorly in decoupling, which not only fails to achieve the expected decoupling effect, has low decoupling efficiency, and increases the difficulty of speech parsing.

[0005] To address the aforementioned technical problems, this application provides a speech parsing method, employing the following technical solution:

[0006] A speech parsing method includes the following steps:

[0007] Obtain the speech data to be parsed;

[0008] The speech data to be parsed is input into the trained speech parsing model, wherein the speech parsing model includes a timbre encoder and a prosody encoder;

[0009] Based on the timbre encoder, the timbre information contained in the speech data to be parsed is extracted, wherein the timbre information includes the voiceprint information of the speaker of the speech data to be parsed;

[0010] Based on the prosodic encoder, the prosodic information contained in the speech data to be parsed is extracted, wherein the prosodic information includes the speaker's speaking style, speech rate, rhythm, intonation variation, volume, and emotional information;

[0011] The timbre information and the prosody information are output as the analysis results.

[0012] Furthermore, before performing the step of inputting the speech data to be parsed into the trained speech parsing model, the method further includes:

[0013] Obtain the training speech set;

[0014] The training speech set is input into a pre-built speech parsing model to train the speech parsing model, thereby obtaining the trained speech parsing model.

[0015] Furthermore, the pre-built speech parsing model includes an autoencoder-based speech parsing model. The step of inputting the training speech set into the pre-built speech parsing model to train the speech parsing model and obtain the trained speech parsing model specifically includes:

[0016] Step 301: Input the training speech set into the autoencoder-based speech parsing model;

[0017] Step 302: Extract the text information and acoustic information contained in all speech data in the training speech set through a preset feature extraction component. The text information includes the text content information in each speech data, and the acoustic information includes timbre information and prosody information.

[0018] Step 303: Learn the timbre information contained in all speech data in the training speech set through the timbre encoder, wherein the timbre information includes the voiceprint information of the pronunciation target corresponding to each speech data;

[0019] Step 304: Learn the prosodic information contained in all speech data in the training speech set through the prosodic encoder, wherein the prosodic information includes speaking style, speech rate, rhythm, intonation variation, volume and emotion information in each speech data;

[0020] Step 305: Input the text information, the timbre information, and the prosody information into the decoder of the speech parsing model to obtain the decoding output result;

[0021] Step 306: Determine the loss value of the decoding output result relative to the training speech set based on a preset loss function;

[0022] Step 307: If the loss value does not meet the preset loss threshold, adjust the hyperparameters of the speech parsing model and re-execute steps 301 to 306 until the loss value meets the preset loss threshold to obtain the trained speech parsing model.

[0023] Furthermore, the timbre encoder consists of a locally connected layer with a ReLU activation function and two fully connected layers with ReLU activation functions. The step of learning the timbre information contained in all speech data in the training speech set through the timbre encoder specifically includes:

[0024] The acoustic information is input into the local connection layer with the ReLU activation function, and the acoustic information is feature-mapped according to the connection units set in the local connection layer to obtain a first mapping result;

[0025] The first mapping result is input into the two fully connected layers with ReLU activation functions to obtain the second mapping result after the first mapping result has undergone two fully connected mappings, which is used as the timbre information.

[0026] Furthermore, the prosodic encoder sequentially includes a sentence-level prosodic feature extraction unit, a character-level prosodic feature extraction unit, a word-level prosodic feature extraction unit, and a projection layer. The step of learning the prosodic information contained in all speech data in the training speech set through the prosodic encoder specifically includes:

[0027] The acoustic information is input into the sentence-level prosodic feature extraction unit to extract sentence-level prosodic features;

[0028] The sentence-level prosodic features are input into the character-level prosodic feature extraction unit to extract character-level prosodic features;

[0029] The character-level prosodic features are input into the word-level prosodic feature extraction unit to extract word-level prosodic features;

[0030] The word-level prosodic features are output through the projection layer to obtain the prosodic information.

[0031] Furthermore, the step of inputting the text information, the timbre information, and the prosody information into the decoder of the speech parsing model to obtain the decoding output result specifically includes:

[0032] The text information, timbre information, and prosody information input into the decoder are obtained;

[0033] The text information, timbre information, and prosodic information are fused to obtain the fused and reconstructed speech, which is used as the decoding output.

[0034] Furthermore, the step of determining the loss value of the decoding output result relative to the training speech set based on a preset loss function specifically includes:

[0035] According to the loss function:

[0036] L=a×l ph +b×l word +c×l recon +d×l sm ,

[0037] Calculate the loss value of the decoded output compared to the training speech set, where L represents the loss value, l ph Indicates word-level loss, l word Indicates word-level loss, l recon L represents the speech reconstruction loss. sm Let a represent the mutual information loss, b represent the word-level loss weight, c represent the speech reconstruction loss weight, and d represent the mutual information loss weight. a+b+c+d=1.

[0038] To address the aforementioned technical problems, this application also provides a speech parsing device, which employs the following technical solution:

[0039] A speech analysis device, comprising:

[0040] The voice data acquisition module is used to acquire the voice data to be parsed.

[0041] A speech data input module is used to input the speech data to be parsed into a trained speech parsing model, wherein the speech parsing model includes a timbre encoder and a prosody encoder;

[0042] The timbre information extraction module is used to extract the timbre information contained in the speech data to be parsed based on the timbre encoder, wherein the timbre information includes the voiceprint information of the speaker of the speech data to be parsed;

[0043] The prosodic information extraction module is used to extract the prosodic information contained in the speech data to be parsed based on the prosodic encoder, wherein the prosodic information includes the speaker's speaking style, speech rate, rhythm, intonation variation, volume, and emotional information.

[0044] The parsing result output module is used to output the timbre information and the prosody information as parsing results.

[0045] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0046] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the speech parsing method described above.

[0047] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0048] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the speech parsing method described above.

[0049] Compared with the prior art, the embodiments of this application have the following main advantages:

[0050] The speech parsing method described in this application involves: acquiring speech data to be parsed; inputting the speech data to be parsed into a trained speech parsing model, wherein the speech parsing model includes a timbre encoder and a prosodic encoder; extracting timbre information contained in the speech data to be parsed using the timbre encoder, and extracting prosodic information contained in the speech data to be parsed based on the prosodic encoder; and outputting the timbre information and the prosodic information as the parsing result. This application improves speech parsing efficiency and simplifies the parsing steps by pre-training the timbre encoder and prosodic encoder in the speech parsing model, and subsequently extracting timbre information / prosodic information using the trained timbre encoder / prosodic encoder. Compared to conventional speech parsing methods, which first parse text information and acoustic information, and then decouple the acoustic information to obtain timbre information / prosodic information, this method facilitates subsequent speech processing steps, such as speech reconstruction, speech synthesis, and speech interpretation. Applying this speech parsing method to financial speech application scenarios, such as intelligent customer service speech synthesis, can greatly reduce speech synthesis time and also allow for free synthesis based on timbre / prosodic information. Attached Figure Description

[0051] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0053] Figure 2 This is a flowchart of an embodiment of the speech parsing method according to this application;

[0054] Figure 3 This is a flowchart of a specific embodiment of training the prediction model in the speech parsing method described in this application;

[0055] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 203 shown;

[0056] Figure 5 yes Figure 4 A flowchart of a specific embodiment of step 402 shown;

[0057] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 205 shown;

[0058] Figure 7 This is a schematic diagram of the structure of one embodiment of the speech parsing device according to this application;

[0059] Figure 8 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0061] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0062] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0063] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0064] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0065] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0066] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0067] It should be noted that the speech parsing method provided in this application embodiment is generally executed by a terminal device, and correspondingly, the speech parsing device is generally installed in the terminal device.

[0068] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0069] Continue to refer to Figure 2 A flowchart of an embodiment of the speech parsing method according to this application is shown. The speech parsing method includes the following steps:

[0070] Step 201: Obtain the speech data to be parsed.

[0071] In this embodiment, the speech data to be parsed includes pre-collected speech packets and real-time collected speech of the target user. The specific difference depends on the application scenario of the speech parsing method, and no limitation is made here. For example, the speech parsing method of this application can be applied to a medical rehabilitation awakening scenario. This speech parsing method can be used to parse and obtain the timbre and / or prosody information of a familiar relative or lover speaking to the target patient. When the target patient is in a coma or waiting to be awakened, the parsed timbre and / or prosody information can be used to perform speech synthesis or awakening signal synthesis, making the synthesized speech more natural and closer to the actual speech of the speaker.

[0072] Step 202: Input the speech data to be parsed into the trained speech parsing model, wherein the speech parsing model includes a timbre encoder and a prosody encoder.

[0073] In this embodiment, before performing the step of inputting the speech data to be parsed into the trained speech parsing model, the method further includes: obtaining a training speech set; inputting the training speech set into a pre-built speech parsing model to train the speech parsing model, so as to obtain the trained speech parsing model.

[0074] In this embodiment, the pre-built speech parsing model includes an autoencoder-based speech parsing model, which includes an encoder part and a decoder part. The purpose of the encoder part is to parse the speech to obtain a low-dimensional feature representation, and the purpose of the decoder part is to perform high-dimensional reconstruction based on the low-dimensional feature representation to obtain the output speech, so that the output speech of the finally trained speech parsing model is as similar as possible to the input speech.

[0075] Continue to refer to Figure 3 , Figure 3 This is a flowchart of a specific embodiment of training the speech parsing model in the speech parsing method described in this application, including:

[0076] Step 301: Input the training speech set into the autoencoder-based speech parsing model;

[0077] Step 302: Extract the text information and acoustic information contained in all speech data in the training speech set through a preset feature extraction component. The text information includes the text content information in each speech data, and the acoustic information includes timbre information and prosody information.

[0078] In this embodiment, the preset feature extraction component includes at least a text feature extraction component and an acoustic feature extraction component. The text feature extraction component can be a speech recognition component that meets the minimum requirements for speech-text recognition, such as a natural language-based speech recognition component. The acoustic feature extraction component can be an open-source acoustic feature extraction tool called AcousticFeatureExtraction, which is based on the Python language and can efficiently extract various acoustic features from speech data and audio files.

[0079] Step 303: Learn the timbre information contained in all speech data in the training speech set through the timbre encoder, wherein the timbre information includes the voiceprint information of the pronunciation target corresponding to each speech data;

[0080] In this embodiment, the timbre encoder is composed of a locally connected layer with a ReLU activation function and two fully connected layers with ReLU activation functions.

[0081] Continue to refer to Figure 4 , Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 303 shown includes:

[0082] Step 401: Input the acoustic information into the local connection layer with the ReLU activation function, and perform feature mapping on the acoustic information according to the connection units set in the local connection layer to obtain a first mapping result;

[0083] Step 402: Input the first mapping result into the two fully connected layers with ReLU activation functions to obtain the second mapping result after the first mapping result has undergone two fully connected mappings, and use it as the timbre information.

[0084] Step 304: Learn the prosodic information contained in all speech data in the training speech set through the prosodic encoder, wherein the prosodic information includes speaking style, speech rate, rhythm, intonation variation, volume and emotion information in each speech data;

[0085] In this embodiment, the prosodic encoder sequentially includes a sentence-level prosodic feature extraction unit, a character-level prosodic feature extraction unit, a word-level prosodic feature extraction unit, and a projection layer.

[0086] Continue to refer to Figure 5 , Figure 5 yes Figure 3 A flowchart of a specific embodiment of step 304 shown includes:

[0087] Step 501: Input the acoustic information into the sentence-level prosodic feature extraction unit to extract sentence-level prosodic features;

[0088] Step 502: Input the sentence-level prosodic features into the character-level prosodic feature extraction unit to extract character-level prosodic features;

[0089] Step 503: Input the character-level prosodic features into the word-level prosodic feature extraction unit to extract word-level prosodic features;

[0090] Step 504: The word-level prosodic features are output through the projection layer to obtain the prosodic information.

[0091] Specifically, in order to fully extract prosodic features under different dimensions, the sentence-level prosodic feature extraction unit, the character-level prosodic feature extraction unit, and the word-level prosodic feature extraction unit are each composed of WaveNet dilated convolutional layers, Pooling layers, vector quantization representation layers, and multi-head attention mechanisms.

[0092] Step 305: Input the text information, the timbre information, and the prosody information into the decoder of the speech parsing model to obtain the decoding output result;

[0093] Continue to refer to Figure 6 , Figure 6 yes Figure 3 A flowchart of a specific embodiment of step 305 shown includes:

[0094] Step 601: Obtain the text information, timbre information, and prosody information input into the decoder;

[0095] Step 602: Perform feature fusion on the text information, the timbre information, and the prosodic information to obtain the fused and reconstructed speech, which is used as the decoding output result.

[0096] Step 306: Determine the loss value of the decoding output result relative to the training speech set based on a preset loss function;

[0097] In this embodiment, the step of determining the loss value of the decoding output result relative to the training speech set based on a preset loss function specifically includes: according to the loss function:

[0098] L=a×l ph +b×l word +c×l recon +d×l sm ,

[0099] Calculate the loss value of the decoded output compared to the training speech set, where L represents the loss value, lph Indicates word-level loss, l word Indicates word-level loss, l recon L represents the speech reconstruction loss. sm Let a represent the mutual information loss, b represent the word-level loss weight, c represent the speech reconstruction loss weight, and d represent the mutual information loss weight. a+b+c+d=1.

[0100] The aforementioned calculation parameters include the use of cross-entropy loss to calculate the word-level loss l. ph Specifically, based on the preset word-level loss function:

[0101] l ph =0.5×(l text (τ·c ij )+l timbre (τ·c ij )+l prosody (τ·c ij )),

[0102] Calculate the word-level loss, where l ph Indicates word-level loss, c ij ∈R N×N This represents the dot product similarity measure between information features, where τ is a learnable parameter, and ij is the word-level feature representation. Here, l text (τ·c ij The value represents the loss calculated based on the dot product similarity measure between textual information features. timbre (τ·c ij The value represents the loss calculated based on the dot product similarity measure between timbre information features. prosody (τ·c ij ) represents the loss value calculated based on the dot product similarity measure between prosodic information features.

[0103] Similarly, the word-level loss l is calculated using the cross-entropy loss method. word Specifically, based on the preset word-level loss function:

[0104] l word =0.5×(l text (τ·c ij )+l timbre (τ·c ij )+l prosody (τ·c ij )),

[0105] Calculate the word-level loss, where l word Indicates word-level loss, c ij ∈RN×N This represents the dot product similarity measure between information features, where τ is a learnable parameter, and ij is the word-level feature representation. Here, l text (τ·c ij The value represents the loss calculated based on the dot product similarity measure between textual information features. timbre (τ·c ij The value represents the loss calculated based on the dot product similarity measure between timbre information features. prosody (τ·c ij ) represents the loss value calculated based on the dot product similarity measure between prosodic information features.

[0106] Note that although l in the above two formulas text (τ·c ij ),l timbre (τ·c ij ) and l prosody (τ·c ij While they appear "same," the inputs "ij" in the function represent character-level and word-level features, respectively. Therefore, the feature dimensions considered during dot product similarity calculation differ. Consequently, the corresponding l values ​​in the character-level and word-level loss functions differ. text (τ·c ij ),l timbre (τ·c ij ) and l prosody (τ·c ij The output of ) does not have the same meaning.

[0107] The speech reconstruction loss is calculated using the Euclidean distance method. recon Specifically, according to the preset distance calculation formula:

[0108]

[0109] Calculate the speech reconstruction loss, where, X represents the decoding output result, and X represents the speech in the training speech set.

[0110] The mutual information loss l is calculated using a preset mutual information loss function. sm Specifically, according to the mutual information loss function:

[0111]

[0112] Calculate the mutual information loss, where x represents the feature of the text information, y represents the feature of the timbre information, and z represents the feature of the prosody information.

[0113] Step 307: If the loss value does not meet the preset loss threshold, adjust the hyperparameters of the speech parsing model and re-execute steps 301 to 306 until the loss value meets the preset loss threshold to obtain the trained speech parsing model.

[0114] Step 203: Extract the timbre information contained in the speech data to be parsed based on the timbre encoder, wherein the timbre information includes the voiceprint information of the speaker of the speech data to be parsed.

[0115] Step 204: Extract prosodic information contained in the speech data to be parsed based on the prosodic encoder, wherein the prosodic information includes the speaker's speaking style, speech rate, rhythm, intonation variation, volume, and emotional information.

[0116] Since the timbre encoder and the prosody encoder have already been trained, in steps 203 and 204, the timbre encoder and the prosody encoder are directly used to parse the speech data to be parsed, without having to first parse the acoustic information and then parse the timbre and prosody information from the acoustic information to obtain the timbre and prosody information.

[0117] Step 205: Output the timbre information and the rhythm information as the analysis results.

[0118] This application obtains speech data to be parsed; inputs the speech data to be parsed into a trained speech parsing model, wherein the speech parsing model includes a timbre encoder and a prosodic encoder; extracts timbre information contained in the speech data to be parsed using the timbre encoder, and extracts prosodic information contained in the speech data to be parsed based on the prosodic encoder; and outputs the timbre information and the prosodic information as the parsing result. This application improves speech parsing efficiency and simplifies the parsing steps to a certain extent compared to conventional speech parsing methods, which first parse text and acoustic information and then decouple the acoustic information to obtain timbre / prosodic information. This facilitates subsequent speech processing steps, such as speech reconstruction, speech synthesis, and speech interpretation. Applying this speech parsing method to financial speech application scenarios, such as intelligent customer service speech synthesis, can greatly reduce speech synthesis time and also allow for free synthesis based on timbre / prosodic information.

[0119] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0120] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0121] In this embodiment, the process involves acquiring speech data to be parsed; inputting the speech data into a trained speech parsing model, wherein the speech parsing model includes a timbre encoder and a prosodic encoder; extracting timbre information from the speech data to be parsed using the timbre encoder, and extracting prosodic information from the speech data to be parsed using the prosodic encoder; and outputting the timbre information and the prosodic information as the parsing result. This application pre-trains the timbre encoder and prosodic encoder in the speech parsing model, and subsequently extracts timbre information / prosodic information using the trained timbre encoder / prosodic encoder. Compared to conventional speech parsing methods, which first parse text and acoustic information and then decouple the acoustic information to obtain timbre / prosodic information, this method improves speech parsing efficiency to a certain extent, simplifies the parsing steps, and facilitates subsequent speech processing steps, such as speech reconstruction, speech synthesis, and speech interpretation. Applying this speech parsing method to financial speech application scenarios, such as intelligent customer service speech synthesis, can greatly reduce speech synthesis time and also allow for free synthesis based on timbre / prosodic information.

[0122] Further reference Figure 7 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a speech parsing device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0123] like Figure 7 As shown, the speech parsing device 700 described in this embodiment includes: a speech data acquisition module 701, a speech data input module 702, a timbre information extraction module 703, a prosody information extraction module 704, and a parsing result output module 705. Wherein:

[0124] The voice data acquisition module 701 is used to acquire voice data to be parsed.

[0125] The speech data input module 702 is used to input the speech data to be parsed into a trained speech parsing model, wherein the speech parsing model includes a timbre encoder and a prosody encoder;

[0126] The timbre information extraction module 703 is used to extract the timbre information contained in the speech data to be parsed based on the timbre encoder, wherein the timbre information includes the voiceprint information of the speaker of the speech data to be parsed;

[0127] The prosodic information extraction module 704 is used to extract the prosodic information contained in the speech data to be parsed based on the prosodic encoder, wherein the prosodic information includes the speaker's speaking style, speech rate, rhythm, intonation changes, volume, and emotional information.

[0128] The parsing result output module 705 is used to output the timbre information and the rhythm information as parsing results.

[0129] This application obtains speech data to be parsed; inputs the speech data to be parsed into a trained speech parsing model, wherein the speech parsing model includes a timbre encoder and a prosodic encoder; extracts timbre information contained in the speech data to be parsed using the timbre encoder, and extracts prosodic information contained in the speech data to be parsed based on the prosodic encoder; and outputs the timbre information and the prosodic information as the parsing result. This application improves speech parsing efficiency and simplifies the parsing steps to a certain extent compared to conventional speech parsing methods, which first parse text and acoustic information and then decouple the acoustic information to obtain timbre / prosodic information. This facilitates subsequent speech processing steps, such as speech reconstruction, speech synthesis, and speech interpretation. Applying this speech parsing method to financial speech application scenarios, such as intelligent customer service speech synthesis, can greatly reduce speech synthesis time and also allow for free synthesis based on timbre / prosodic information.

[0130] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0131] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0132] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0133] The computer device 8 includes a memory 8a, a processor 8b, and a network interface 8c that are interconnected via a system bus. It should be noted that only the computer device 8 with components 8a-8c is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0134] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0135] The memory 8a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 8a may be an internal storage unit of the computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the memory 8a may also be an external storage device of the computer device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 8. Of course, the memory 8a may include both the internal storage unit and its external storage device of the computer device 8. In this embodiment, the memory 8a is typically used to store the operating system and various application software installed on the computer device 8, such as computer-readable instructions for a speech parsing method. In addition, the memory 8a can also be used to temporarily store various types of data that have been output or will be output.

[0136] In some embodiments, the processor 8b may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 8b is typically used to control the overall operation of the computer device 8. In this embodiment, the processor 8b is used to execute computer-readable instructions stored in the memory 8a or to process data, such as executing computer-readable instructions for the speech parsing method.

[0137] The network interface 8c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 8 and other electronic devices.

[0138] The computer device proposed in this embodiment belongs to the field of speech decoupling technology and is applied in speech feature parsing scenarios. This application acquires speech data to be parsed; inputs the speech data to be parsed into a trained speech parsing model, wherein the speech parsing model includes a timbre encoder and a prosodic encoder; extracts timbre information contained in the speech data to be parsed using the timbre encoder, and extracts prosodic information contained in the speech data to be parsed based on the prosodic encoder; and outputs the timbre information and the prosodic information as the parsing result. This application pre-trains the timbre encoder and prosodic encoder in the speech parsing model, and then extracts timbre / prosodic information using the trained timbre encoder / prosodic encoder. Compared to conventional speech parsing methods, which first parse text and acoustic information and then decouple the acoustic information to obtain timbre / prosodic information, this method improves speech parsing efficiency to a certain extent, simplifies the parsing steps, and facilitates subsequent speech processing steps, such as speech reconstruction, speech synthesis, and speech interpretation. Applying this speech parsing method to financial speech application scenarios, such as intelligent customer service speech synthesis, can greatly reduce speech synthesis time and also enable free synthesis based on timbre / prosodic information.

[0139] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the speech parsing method described above.

[0140] The computer-readable storage medium proposed in this embodiment belongs to the field of speech decoupling technology and is applied in speech feature parsing scenarios. This application involves acquiring speech data to be parsed; inputting the speech data to be parsed into a trained speech parsing model, wherein the speech parsing model includes a timbre encoder and a prosodic encoder; extracting timbre information contained in the speech data to be parsed using the timbre encoder, and extracting prosodic information contained in the speech data to be parsed based on the prosodic encoder; and outputting the timbre information and the prosodic information as the parsing result. This application pre-trains the timbre encoder and prosodic encoder in the speech parsing model, and then extracts timbre / prosodic information using the trained timbre encoder / prosodic encoder. Compared to conventional speech parsing methods, which first parse text and acoustic information and then decouple the acoustic information to obtain timbre / prosodic information, this method improves speech parsing efficiency to a certain extent, simplifies the parsing steps, and facilitates subsequent speech processing steps, such as speech reconstruction, speech synthesis, and speech interpretation. Applying this speech parsing method to financial speech application scenarios, such as intelligent customer service speech synthesis, can greatly reduce speech synthesis time and also enable free synthesis based on timbre / prosodic information.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0142] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A speech analysis method, characterized by, The method comprises the following steps: acquiring voice data to be parsed; inputting the voice data to be parsed into a trained voice parsing model, wherein the voice parsing model comprises a timbre encoder and a prosody encoder, the pre-constructed voice parsing model comprises a voice parsing model based on an autoencoder, and the step of training the pre-constructed voice parsing model comprises: step 301, inputting a training voice set into the voice parsing model based on the autoencoder; step 302, extracting text information and acoustic information respectively contained in all voice data in the training voice set by a preset feature extraction component, wherein the text information comprises text content information in each piece of voice data, and the acoustic information comprises timbre information and prosody information; step 303, learning the timbre information respectively contained in all voice data in the training voice set by the timbre encoder, wherein the timbre information comprises voiceprint information of a pronunciation target corresponding to each piece of voice data; step 304, learning the prosody information respectively contained in all voice data in the training voice set by the prosody encoder, wherein the prosody information comprises speaking style, speech rate, rhythm, intonation change, volume, and emotion information in each piece of voice data; step 305, inputting the text information, the timbre information, and the prosody information into a decoder of the voice parsing model to obtain a decoding output result; step 306, determining a loss value of the decoding output result compared with the training voice set based on a preset loss function, wherein the loss value calculation by the preset loss function jointly utilizes a word-level loss, a word-level loss, a voice reconstruction loss, and a mutual information loss; step 307, if the loss value does not satisfy a preset loss threshold, adjusting hyperparameters of the voice parsing model, and re-executing steps 301 to 306 until the loss value satisfies the preset loss threshold, to obtain a trained voice parsing model; extracting timbre information contained in the voice data to be parsed based on the timbre encoder, wherein the timbre information comprises voiceprint information of a pronunciation person of the voice data to be parsed; extracting prosody information contained in the voice data to be parsed based on the prosody encoder, wherein the prosody information comprises speaking style, speech rate, rhythm, intonation change, volume, and emotion information of the pronunciation person; outputting the timbre information and the prosody information as a parsing result.

2. The voice parsing method of claim 1, wherein, The timbre encoder is composed of one local connection layer with a ReLU activation function and two fully connected layers with a ReLU activation function, and the step of learning the timbre information respectively contained in all voice data in the training voice set by the timbre encoder comprises: inputting the acoustic information into the local connection layer with the ReLU activation function, performing feature mapping on the acoustic information according to connection units arranged in the local connection layer, and obtaining a first mapping result; The first mapping result is input into the two-layer full connection layer with a ReLU activation function to obtain a second mapping result of the first mapping result after twice full connection mapping, as the timbre information.

3. The voice parsing method of claim 1, wherein, The prosody encoder sequentially comprises a sentence-level prosody feature extraction unit, a word-level prosody feature extraction unit, a word-level prosody feature extraction unit, and a projection layer, and the step of learning the prosody information contained in all voice data in the training voice set by the prosody encoder specifically comprises: The acoustic information is input into the sentence-level prosody feature extraction unit to extract sentence-level prosody features; The sentence-level prosody features are input into the word-level prosody feature extraction unit to extract word-level prosody features; The word-level prosody features are input into the word-level prosody feature extraction unit to extract word-level prosody features; The word-level prosody features are output through the projection layer to obtain the prosody information.

4. The voice parsing method of claim 1, wherein, The step of inputting the text information, the timbre information, and the prosody information into the decoder of the voice analysis model to obtain a decoding output result specifically comprises: The text information, the timbre information, and the prosody information input into the decoder are obtained; The text information, the timbre information, and the prosody information are feature-fused to obtain a fused reconstructed voice as the decoding output result.

5. The voice parsing method of claim 1, wherein, The step of determining a loss value of the decoding output result compared with the training voice set based on a preset loss function specifically comprises: According to the loss function: ， computing a loss value of the decoding output result compared to the training speech set, wherein L represents the loss value, represents a word-level loss, represents a word-level loss, represents a speech reconstruction loss, represents a mutual information loss, represents a word-level loss weight, represents a word-level loss weight, represents a speech reconstruction loss weight, represents a mutual information loss weight, .

6. A speech analysis apparatus characterized by comprising: It comprises: A voice data acquisition module is configured to acquire voice data to be analyzed. A voice data input module is configured to input the voice data to be analyzed into a voice analysis model trained, wherein the voice analysis model comprises a timbre encoder and a prosody encoder, the pre-constructed voice analysis model comprises a voice analysis model based on a self-encoder, and the step of training the pre-constructed voice analysis model comprises: Step 301, inputting a training voice set into a voice analysis model based on a self-encoder; Step 302, extracting text information and acoustic information contained in all voice data in the training voice set by a preset feature extraction component, wherein the text information comprises text content information in each voice data, and the acoustic information comprises timbre information and prosody information; Step 303, learning the timbre information contained in all voice data in the training voice set by the timbre encoder, wherein the timbre information comprises voiceprint information of a pronunciation target corresponding to each voice data; Step 304, learning the prosody information contained in all voice data in the training voice set by the prosody encoder, wherein the prosody information comprises speaking style, speech rate, rhythm, intonation change, volume, and emotion information in each voice data; Step 305, inputting the text information, the timbre information, and the prosody information into a decoder of the voice analysis model to obtain a decoding output result; In step 306, a loss value of the decoding output result compared with the training voice set is determined based on a preset loss function, wherein the loss value calculation using the preset loss function jointly utilizes a word level loss, a word level loss, a voice reconstruction loss, and a mutual information loss. In step 307, if the loss value does not satisfy a preset loss threshold, a hyperparameter of the voice parsing model is adjusted, and steps 301 to 306 are re-executed until the loss value satisfies the preset loss threshold, and a trained voice parsing model is obtained. A timbre information extraction module is configured to extract timbre information contained in the to-be-parsed voice data based on the timbre encoder, wherein the timbre information includes voiceprint information of a speaker of the to-be-parsed voice data. A prosody information extraction module is configured to extract prosody information contained in the to-be-parsed voice data based on the prosody encoder, wherein the prosody information includes speaking style, speech rate, rhythm, intonation change, volume, and emotion information of the speaker. An analysis result output module is configured to output the timbre information and the prosody information as an analysis result.

7. A computer device, comprising: A computer readable storage medium has computer readable instructions stored thereon, and the computer readable instructions are executed by a processor to implement the steps of the voice parsing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium has computer readable instructions stored thereon, and the computer readable instructions are executed by a processor to implement the steps of the voice parsing method according to any one of claims 1 to 5.

Images