Deep learning-based teaching voice naturalness optimization method and system

Abstract

The invention discloses a teaching speech naturalness optimization method and system based on deep learning, and belongs to the technical field of equipment speech optimization, and the method comprises the steps: constructing a semantic recognition model, and extracting statement emotion features; constructing a pronunciation emotion parameter simulation model to adjust voice parameters; constructing a GAN adversarial network to carry out fine-grained optimization; and realizing continuous learning according to user feedback. According to the invention, the naturalness and emotional expressive force of teaching voice can be effectively improved, and the user experience is improved.

Description

Technical Field

[0001] This invention belongs to the field of device voice optimization technology, specifically relating to a method and system for optimizing the naturalness of teaching voice based on deep learning. Background Technology

[0002] Speech naturalness is an evaluation of the similarity between the output speech of a speech synthesis model and a real human voice. It mainly measures the accuracy of the reproduction of parameters such as timbre, pitch, speech rate, and emotion. Speech naturalness directly affects the user's acceptance of the synthesized speech and is one of the key indicators for evaluating the performance of speech synthesis models. It is also one of the important factors affecting the user experience of various devices with pronunciation functions, especially in teaching speech output devices, where speech naturalness is very important. However, many problems still exist in the actual application scenarios of educational speech pronunciation hardware devices. Most existing devices use fixed parameter templates, lacking the ability to learn and improve independently, and have insufficient understanding of sentence semantics. For example, while "Publication No. CN120126447A is a method for generating personalized speech content" considers the dynamic adjustment of speech parameter models based on factors such as tone, pauses, and stress, it still needs improvement in real-time application on educational hardware devices. Currently, devices generally suffer from a strong mechanical feel in speech and a lack of emotional expression. They cannot flexibly adjust emotions according to the semantics of the sentence and the specific usage environment, resulting in overly mechanical speech that seriously affects the user experience.

[0003] Therefore, it is necessary to design a method and system for optimizing the naturalness of teaching speech based on deep learning. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for optimizing the naturalness of teaching voice based on deep learning, in order to address the shortcomings of the prior art, and to solve the technical problems of existing teaching voice devices having a strong mechanical feel, lack of emotional expression, and lack of self-learning ability.

[0005] The present invention adopts the following technical solution: A deep learning-based method for optimizing the naturalness of teaching speech includes the following steps: S1. Construct a semantic recognition model to perform semantic recognition and emotion feature extraction on the input teaching voice sentences, and output the emotion feature vector of the sentences; S2. Construct a pronunciation emotion parameter simulation model, and adjust the speech parameters according to the emotion feature vector of the statement. The speech parameters include fundamental frequency, speech rate, intonation and three-dimensional emotion vector. S3. Construct a GAN adversarial network, using the speech parameters adjusted in step S2 as input, and perform fine-grained optimization through adversarial training to generate the final synthesized speech; S4. Obtain the user's emotional expression feedback on the synthesized speech generated in step S3. Based on the feedback, extract the pronunciation emotion parameters and associated semantic feature vectors corresponding to the synthesized speech marked by the user as having the preferred emotional expression. Calculate the distance between the semantic feature vector of the newly input teaching speech and the stored preferred semantic feature vector. When the distance is less than a set threshold, apply the pronunciation emotion parameters corresponding to the stored preferred semantic feature vector to the parameter adjustment of the new input speech.

[0006] Preferably, in step S1, the semantic recognition model performs semantic recognition and emotion feature extraction, including: The input sentence is segmented into word sequences; each word sequence is tagged with part-of-speech tags; a word vector model is used to map words and their contexts into vector representations; semantic feature vectors are calculated by combining dialogue scenario and dialogue time features; based on the semantic feature vectors, sentiment features are extracted through a sentiment classification model, and the sentiment feature vector of the sentence is output.

[0007] Preferably, the word vector model is a Word2Vec model or a BERT model, and words are obtained through the BERT model. The vector representation is ,in, For vector dimensions.

[0008] Preferably, the sentiment feature vector of the statement :

[0009] in, For activation function, This is the weight matrix. For bias vector This is a vector representation of the dialogue scene. This is a vector representation of the dialogue time.

[0010] Preferably, in step S2, the adjustment of the voice parameters is performed using the following formula: baseband speech rate intonation 3D Emotion Vector , in, For sentiment feature vectors, , , , This is the weight matrix. , , For bias vectors, It is a three-dimensional bias vector.

[0011] Preferably, in step S2, the three-dimensional emotion vector =[ Pleasure level, Activation level (Dominance).

[0012] Preferably, in step S3, the process of fine-grained optimization through adversarial training includes: The adjusted speech parameters from step S2 are input into the generator of the GAN adversarial network. By generator Mapped to synthetic speech features; discriminator via GAN adversarial network. The synthesized speech features are distinguished from real speech features. Based on the distinction, the generator is optimized by minimizing a preset loss function. and discriminator The parameters are used for adversarial training to achieve fine-grained optimization and generate the final synthesized speech.

[0013] Preferably, the loss function is:

[0014] in, For random input noise to the generator, This represents the expectation operation. These are real speech features.

[0015] Preferably, in step S4, the parameter adjustment applied to the new input statement is achieved through the following formula:

[0016] in, For learning rate, The number of high-quality sentence samples, For the pronunciation emotion parameter vector of the new input sentence, For the first The bar is marked as the parameter vector of the preferred synthesized speech.

[0017] Secondly, a deep learning-based system for optimizing the naturalness of teaching speech includes: The voice signal receiving module is used to collect the voice signal to be emitted by the device, preprocess it, and obtain the preprocessed signal. The semantic recognition model is used to segment sentences into word sequences, combine the parts of speech of each word, and take into account the context of the sentence, the dialogue scene, and the dialogue time to obtain the sentiment tendency of the sentence and output the sentiment features of the sentence. The pronunciation emotion parameter simulation model is used to parameterize and decompose the specific pronunciation of the sentence to be pronounced, and determine the pronunciation method by changing the parameters according to the imported emotion features; GAN (Generative Adversarial Network) is used to achieve fine-grained control of pronunciation emotion parameters and optimize the final pronunciation effect; The voice output module uses hardware devices such as a vocoder to pronounce the voice to be output after changing various parameters. The user feedback module is used to provide user feedback. After listening to the simulated emotional voice emitted by the device, users can judge its simulation effect and naturalness, and provide feedback through the user feedback module. The data processing module is used to process user feedback data and the final data fed back after GAN adversarial network, and transmit the processed data to the data storage module. The data storage module is used to store user feedback data, adversarial data, various model data, past pronunciation parameter data, and other data. The user feedback module can transmit user feedback data to the data processing module for processing. For processed statements with good naturalness and emotional feedback, the data processing module can independently extract and store the specific adjustment parameters of the statement for easy use in the future and for model improvement.

[0018] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the aforementioned deep learning-based teaching speech naturalness optimization method.

[0019] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described deep learning-based teaching speech naturalness optimization method.

[0020] Fifthly, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the aforementioned deep learning-based teaching speech naturalness optimization method.

[0021] In a sixth aspect, embodiments of the present invention provide an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the above-described deep learning-based teaching speech naturalness optimization method.

[0022] Compared with the prior art, the present invention has at least the following beneficial effects: A deep learning-based method for optimizing the naturalness of teaching speech constructs a complete closed loop encompassing semantic recognition, parameter adjustment, adversarial optimization, and feedback learning. This four-step core process achieves end-to-end optimization of the naturalness of teaching speech. It accurately extracts emotional features at the semantic level to ensure targeted parameter adjustments; leverages GAN adversarial networks for fine-grained optimization to enhance speech naturalness; and incorporates a dynamic learning mechanism based on user feedback to allow the system to continuously adapt to the needs of teaching scenarios. The overall logic is rigorous, addressing the core shortcomings of existing devices—fixed parameters and lack of autonomous evolution—significantly improving the emotional adaptability and naturalness of teaching speech, making it suitable for various teaching statements and scenarios.

[0023] Furthermore, the semantic recognition process was refined, and the accuracy of semantic understanding was improved through part-of-speech tagging, word vector mapping, and multi-feature fusion. It can accurately capture the emotional tone of sentences, providing a reliable basis for subsequent pronunciation parameter adjustments and effectively solving the mechanical speech problem caused by insufficient semantic understanding.

[0024] Furthermore, the use of the BERT model was restricted, leveraging its powerful contextual understanding capabilities to significantly improve the accuracy of word semantic representation. This ensured the reliability of sentiment feature extraction, laying a solid foundation for generating natural and fluent instructional speech.

[0025] Furthermore, specific formulas for calculating sentiment features are provided, achieving an accurate mapping from semantic features to sentiment features through linear transformations of the weight matrix and bias vector. This mathematical expression ensures the feasibility and reproducibility of the method.

[0026] Furthermore, the mathematical relationships between pronunciation parameters were clarified, and the conversion from emotional features to pronunciation parameters was achieved through a linear model. This method offers precise and controllable parameter adjustment, enabling the generation of corresponding combinations of speech parameters based on different emotional features.

[0027] Furthermore, the specific composition of the three-dimensional emotion vector is defined, quantifying the abstract emotion concept into computable parameters. This quantified emotion representation method provides a clear technical path for speech emotion synthesis.

[0028] Furthermore, the optimization process of the GAN network is described in detail, achieving fine-grained control of pronunciation parameters through adversarial training between the generator and discriminator. This method can generate more natural and emotional speech, significantly improving the user experience.

[0029] Furthermore, the generator and discriminator parameters are optimized iteratively by minimizing this function. This loss function effectively measures the difference between synthesized speech features and real speech features, providing a clear optimization target for adversarial training, ensuring the stability and effectiveness of the training process, and making the generated synthesized speech closer to real speech in terms of timbre and emotional expression, thereby further improving the naturalness of the speech.

[0030] Furthermore, by introducing the learning rate and the number of high-quality samples, a dynamic balance in parameter adjustment can be achieved. This formula enables new statement parameters to retain their semantic adaptability while converging towards high-quality parameters recognized by users, avoiding over- or under-adjustment of parameters. This ensures that the system can quickly absorb high-quality user feedback, achieving continuous iterative improvement in optimization capabilities and enhancing the system's personalized adaptability.

[0031] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0032] In summary, this invention constructs a complete speech optimization system through deep learning, accurately identifies semantic emotions, quantitatively adjusts speech parameters, optimizes them with fine-grained GAN networks, and continuously evolves based on user feedback. This solves the problems of mechanical and emotionless speech in existing devices, significantly improves the naturalness and adaptability of teaching speech, and makes it suitable for various teaching scenarios.

[0033] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0034] Figure 1 This is a flowchart of the optimized method of the present invention; Figure 2 This is a flowchart illustrating the operation of the deep learning-based teaching speech naturalness optimization system of the present invention. Figure 3 A schematic diagram of a computer device provided in an embodiment of the present invention; Figure 4 This is a block diagram of a chip provided according to an embodiment of the present invention.

[0035] Among them, 60. Computer equipment; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access memory unit; 6202. Cache memory unit; 6203. Read-only memory unit; 6204. Program / utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0038] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0039] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0040] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0041] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0042] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0043] This invention provides a deep learning-based method for optimizing the naturalness of teaching speech. It realizes the input and adjustment functions of the speech to be emitted by teaching equipment. By decomposing the words and sentences to be pronounced and importing them into a semantic recognition model, the method understands the emotional features of the sentences based on the characteristics of parts of speech and usage context. The features are then imported into a constructed pronunciation emotion parameter simulation model. Various parameters in the pronunciation are adjusted based on the understood emotional features. At the same time, deep learning is used to refine and optimize the pronunciation effect using GAN adversarial networks, outputting emotional speech that conforms to the context and semantics. Finally, the method is continuously improved by combining user feedback to improve the accuracy and processing speed of the system.

[0044] Please see Figure 1 This invention discloses a method for optimizing the naturalness of teaching speech based on deep learning, comprising the following steps: S1. Construct a semantic recognition model to identify and analyze words in a sentence. By recognizing and analyzing the semantics of words, determine the sentiment of the sentence and extract features. The acquisition device collects the voice S to be emitted during the teaching process, extracts the words from the sentence structure, and segments the sentence into word sequences. For each word Part-of-speech tagging algorithm is used to obtain its part-of-speech tags. To more accurately understand word meanings, contextual information is incorporated, and word vector models (such as Word2Vec and BERT) are used to vectorize words. The words and their context words are mapped to vector representations. Let's assume that the words are obtained through the BERT model. The vector representation is ,in, Using vector dimensions and incorporating features such as the dialogue scenario C and dialogue time T, the sentiment feature vector of the statement is calculated using the following formula. :

[0045] in, For activation functions (such as the ReLU function). This is the weight matrix. For bias vector This is a vector representation of the dialogue scene. This is a vector representation of the dialogue time.

[0046] Based on semantic feature vectors The system uses a sentiment classification model (such as an LSTM-based classifier) ​​to determine the actual sentiment of a statement, obtain the sentiment tendency of the statement, and output the sentiment feature vector of the statement. .

[0047] The acquisition device collects the speech to be uttered during the teaching process, extracts each word from the sentence structure, segments the sentence into a word sequence, and combines the specific parts of speech (noun, adjective, etc.) of each word in the sentence to assist semantic understanding. At the same time, it combines the characteristics of the sentence context, dialogue scene, and dialogue time to improve the accuracy of word meaning comprehension, thereby improving the accuracy of overall semantic understanding of the sentence and judging the actual emotion of the sentence. In addition to dialogue scene and dialogue time, it considers incorporating multimodal information such as teaching courseware content and teacher body language. When explaining geometric figures, it combines the graphic display in the courseware and the teacher's gestures to more accurately understand the semantics of related sentences, judge the emotional tendency of sentences, reduce ambiguity, and output more accurate emotional features of sentences.

[0048] S2, construct a pronunciation emotion parameter simulation model, and change the final pronunciation effect by adjusting parameters such as the fundamental frequency, speech rate, intonation + three-dimensional emotion vector of the speech; The specific pronunciation of the sentence to be pronounced is parameterized and broken down, and the final pronunciation is determined by the fundamental frequency. Speech rate intonation and three-dimensional emotion vector =[ Pleasure level, Activation level The parameters include [dominance], etc. The values ​​of each parameter are derived from the extracted sentiment feature vector of the statement. The decision is made by mapping using the following formula:

[0049]

[0050]

[0051]

[0052] in, , , , This is the weight matrix. , , For bias vectors, It is a three-dimensional bias vector, and the values ​​of each parameter together ultimately determine the way of pronunciation.

[0053] A simulation model of pronunciation emotion parameters is constructed. When parameterizing the specific pronunciation methods of the sentences to be pronounced, the pleasantness, activation, and dominance in the three-dimensional emotion vector are defined in more detail. Pleasantness can be determined comprehensively based on the range of pitch variation, average pitch, and the degree of variation in speech rate; activation can be quantified by combining factors such as the energy intensity and duration of speech; dominance can be measured by the stress distribution and intonation fluctuations, making the values ​​of each parameter more accurately reflect emotional characteristics. To make parameter adjustment more flexible to adapt to different teaching scenarios and sentences, a dynamic adaptation mechanism is established. When explaining different levels of difficulty of the same knowledge point, the fundamental frequency, speech rate, and other parameters are automatically adjusted according to the students' cognitive level and learning progress. For students with weak foundations, the speech rate is appropriately reduced and the fundamental frequency clarity is increased; for students with better foundations, the speech rate is increased and intonation variation is enhanced, thereby improving teaching efficiency and students' learning experience.

[0054] The emotional features of the sentences extracted in step S1 are imported into the pronunciation emotion parameter simulation model in step S2. The imported emotional features are used to adjust the parameters of the pronunciation emotion parameter simulation model. By adjusting parameters such as fundamental frequency, speech rate, intonation and three-dimensional emotion vector, the emotion of the output speech is finally determined.

[0055] S3: Construct a GAN adversarial network and achieve fine-grained control of pronunciation emotion parameters through adversarial training to further optimize the final pronunciation effect; The fundamental frequency output from the vocal emotion parameter simulation model in step S2 Speech rate intonation and three-dimensional emotion vector Parameters are used as inputs to the generator architecture in GAN adversarial networks, and the generator... The input parameters are mapped to synthesized speech features through a series of neural network layers. Discriminator Used to determine if the input speech features are real speech features. Or generated speech features .

[0056] During adversarial training, the parameters of the generator and discriminator are optimized by minimizing the following loss function, mapped using the following formula:

[0057] in, For random input noise to the generator, This indicates the expected operation.

[0058] Through continuous adversarial training, fine-grained control of pronunciation and emotional parameters is achieved, further optimizing the final pronunciation effect.

[0059] The fundamental frequency, speech rate, intonation, and three-dimensional emotion vector output from the pronunciation emotion parameter simulation model in step S2 are used as the generator architecture in the GAN adversarial network. An existing emotion discrimination model is introduced as the discriminator in the GAN adversarial network and trained adversarially against the generator to achieve the goal of refining the pronunciation parameters.

[0060] S4 uses deep learning based on user feedback. It extracts the pronunciation emotion parameters of sentences with good emotional expression from user feedback, so that in the process of adjusting parameters for similar sentences in the future, the parameters tend to be closer to the previous parameter samples with good feedback.

[0061] Based on the sentences with good emotional expression in user feedback, the pronunciation emotional parameters are extracted. Let the first sentence in the user feedback be... The pronunciation emotion parameter vector of a high-quality sentence is In subsequent parameter adjustments for similar statements, the following formula is used to adjust the parameters of the new statement. More inclined to use previous parameter samples with better feedback:

[0062] in, For learning rate, This represents the number of high-quality sentence samples. Through continuous extraction, learning, and evolution, the naturalness of the speech is improved, while the system processing time is optimized and the pronunciation interval is shortened.

[0063] Deep learning is performed based on user feedback. Based on sentences with good emotional expression in user feedback, the pronunciation emotion parameters are extracted. This makes the parameter adjustments for similar sentences in the subsequent process more inclined to the previous parameter samples with good feedback. In actual use, the parameters are continuously extracted, learned and evolved to improve the naturalness of the speech, while optimizing the system processing time and shortening the pronunciation interval. In addition to extracting pronunciation emotion parameters from user feedback sentences with good emotional expression, the system conducts multi-dimensional analysis of user feedback data. This analysis examines the correlation between the time of user feedback, usage scenarios, user characteristics (such as age and learning ability), and pronunciation parameters, providing richer information for subsequent parameter adjustments and making the system's learning more targeted. For younger students, the better-sounding speech may be simpler and clearer in terms of speech speed and intonation. Based on this, the system can optimize speech parameters for students of the same age group and establish a continuous model evolution mechanism. Based on long-term accumulated user feedback data, the system regularly performs joint optimization of the semantic recognition model, the pronunciation emotion parameter simulation model, and the GAN adversarial network. In different semesters or academic years, as the teaching content and student groups change, the model automatically adjusts its parameters and structure to better adapt to new teaching needs and continuously improve the naturalness of speech and system performance.

[0064] Please see Figure 2 In another embodiment of the present invention, a teaching speech naturalness optimization system based on deep learning is provided. This system can be used to implement the above-mentioned teaching speech naturalness optimization method based on deep learning. Specifically, the teaching speech naturalness optimization system based on deep learning includes a speech signal receiving module, a semantic recognition model, a pronunciation emotion parameter simulation model, a GAN adversarial network, a speech output module, a user feedback module, a data processing module, and a data storage module.

[0065] The voice signal receiving module is used to collect the voice signal to be emitted by the device, preprocess it, and obtain the preprocessed signal. The semantic recognition model is used to segment sentences into word sequences, combine the parts of speech of each word, and take into account the context of the sentence, the dialogue scene, and the dialogue time to obtain the sentiment tendency of the sentence and output the sentiment features of the sentence. The pronunciation emotion parameter simulation model is used to parameterize and decompose the specific pronunciation of the sentence to be pronounced, and determine the pronunciation method by changing the parameters according to the imported emotion features; GAN (Generative Adversarial Network) is used to achieve fine-grained control of pronunciation emotion parameters and optimize the final pronunciation effect; The voice output module uses hardware devices such as a vocoder to pronounce the voice to be output after changing various parameters. The user feedback module is used to provide user feedback. After listening to the simulated emotional voice emitted by the device, users can judge its simulation effect and naturalness, and provide feedback through the user feedback module. The data processing module is used to process user feedback data and the final data fed back after GAN adversarial network, and transmit the processed data to the data storage module. The data storage module is used to store user feedback data, adversarial data, various model data, past pronunciation parameter data, and other data. The user feedback module transmits user feedback data to the data processing module for processing. For processed statements with good naturalness and sentiment feedback, the data processing module independently extracts and stores the specific adjustment parameters of the statement, facilitating subsequent direct use and model improvement.

[0066] This invention provides a terminal device comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve corresponding method flows or corresponding functions. The processor described in this embodiment can be used for operating a deep learning-based method for optimizing the naturalness of teaching speech, including: A semantic recognition model is constructed to perform semantic recognition and emotion feature extraction on the input teaching speech sentences, and output the emotion feature vector of the sentences. A pronunciation emotion parameter simulation model is constructed to adjust the speech parameters according to the emotion feature vector of the sentences. The speech parameters include fundamental frequency, speech rate, intonation, and a three-dimensional emotion vector. A GAN adversarial network is constructed, and the adjusted speech parameters are used as input. Fine-grained optimization is performed through adversarial training to generate the final synthesized speech. User feedback on the emotional expression of the generated synthesized speech is obtained. Based on the feedback, the pronunciation emotion parameters corresponding to the synthesized speech marked by the user as having the preferred emotional expression and their associated semantic feature vectors are extracted. The distance between the semantic feature vector of the newly input teaching speech sentence and the stored preferred semantic feature vector is calculated. When the distance is less than a set threshold, the pronunciation emotion parameters corresponding to the stored preferred semantic feature vector are applied to the parameter adjustment of the new input sentence.

[0067] Please see Figure 3 The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the method for estimating the concentration of radioactive iodine species in the containment vessel after an accident, as described in this embodiment. To avoid repetition, these details are not elaborated here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the deep learning-based teaching speech naturalness optimization system of this embodiment. To avoid repetition, these details are not elaborated here.

[0068] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 3 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0069] The processor 61 may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0070] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or RAM of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device 60.

[0071] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0072] Please see Figure 4 The terminal device is an electronic device 600, which is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0073] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0074] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0075] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0076] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0077] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem). This communication can be performed via input / output interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network, wide area network, and / or public network, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0078] Example 4 This invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the terminal device and extended storage media supported by the terminal device; it can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). More specific examples of the computer-readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical fiber, portable compact disk read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0079] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0080] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0081] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the deep learning-based teaching speech naturalness optimization method in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps: A semantic recognition model is constructed to perform semantic recognition and emotion feature extraction on the input teaching speech sentences, and output the emotion feature vector of the sentences. A pronunciation emotion parameter simulation model is constructed to adjust the speech parameters according to the emotion feature vector of the sentences. The speech parameters include fundamental frequency, speech rate, intonation, and a three-dimensional emotion vector. A GAN adversarial network is constructed, and the adjusted speech parameters are used as input. Fine-grained optimization is performed through adversarial training to generate the final synthesized speech. User feedback on the emotional expression of the generated synthesized speech is obtained. Based on the feedback, the pronunciation emotion parameters corresponding to the synthesized speech marked by the user as having the preferred emotional expression and their associated semantic feature vectors are extracted. The distance between the semantic feature vector of the newly input teaching speech sentence and the stored preferred semantic feature vector is calculated. When the distance is less than a set threshold, the pronunciation emotion parameters corresponding to the stored preferred semantic feature vector are applied to the parameter adjustment of the new input sentence.

[0082] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0083] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0084] To verify the effectiveness of the method described in this invention, a simulation experiment was designed and applied to a real-world teaching scenario. The experimental data are as follows: I. Experimental Setup and Evaluation Indicators Experimental data: 5,000 teaching voice sentences covering subjects such as mathematics, Chinese, and English were collected and recorded by 5 professional teachers, including different teaching emotions such as explanation, questioning, emphasis, and summarization.

[0085] Comparison System: Baseline System 1: Employs traditional parametric speech synthesis methods (such as HMM-based synthesis).

[0086] Baseline System 2: Employs an end-to-end deep learning model (such as Tacotron2), but does not introduce fine-grained sentiment parameter control and user feedback mechanisms.

[0087] The system of this invention fully implements the method and system described in this invention.

[0088] Evaluation indicators: Naturalness MOS score: 50 evaluators (including teachers and students) were invited to subjectively rate the naturalness of the synthesized speech using an average opinion score of 1-5 (1: very unnatural, 5: very natural).

[0089] Emotional Matching: Evaluators judge whether the emotion of the synthesized speech matches the semantics of the statement and the teaching scenario (1: no match, 5: perfect match).

[0090] Word error rate: An objective measure of the clarity and intelligibility of speech.

[0091] II. Experimental Results and Data Analysis

[0092] Data Analysis: Naturalness and Emotional Expression: The MOS score and emotional matching degree of the system of this invention are significantly higher than those of the two baseline systems. This proves that by driving the adjustment of emotional parameters through semantic understanding and combining it with GAN for fine-grained optimization, the mechanical feel of the speech can be effectively eliminated, and more natural and emotionally rich teaching speech can be generated.

[0093] Clarity: The significant reduction in word error rate proves that while optimizing emotional expression, the invention does not sacrifice speech clarity and intelligibility. On the contrary, it improves the accurate transmission of information through more natural rhythm (such as more reasonable pauses and stresses).

[0094] III. User Feedback and Verification of Learning Effectiveness To verify the effectiveness of the user feedback learning mechanism, A long-term experiment was conducted: Experimental Procedure: Initially, the system provided instructional audio services to a group of students and collected their feedback on the naturalness of the speech and the expression of emotion. After four weeks of continuous learning and parameter optimization, the same group of students evaluated the system again.

[0095] Experimental results: Initially, users' average satisfaction with the emotional expression in voice was 3.6 points.

[0096] After four weeks of learning, the average satisfaction score rose to 4.3.

[0097] Meanwhile, the average time for the system to process new sentences and generate optimized speech decreased from the initial 850ms to 720ms, improving the processing speed by about 15%.

[0098] In summary, this invention provides a method and system for optimizing the naturalness of teaching speech based on deep learning. It can input and adjust the speech to be emitted by teaching equipment, segment sentences into word sequences, and improve the accuracy of word meaning comprehension by combining the characteristics of each word in the sentence, context, dialogue scenario, and dialogue time. This improves the accuracy of overall semantic comprehension of the sentence, obtains the emotional tendency of the sentence, and outputs the emotional features of the sentence. These emotional features are then imported into a pronunciation emotional parameter simulation model, which adjusts various parameters of the model. By adjusting parameters such as fundamental frequency, speech rate, intonation, and three-dimensional emotional vector, the emotional tone of the output speech is ultimately determined. Simultaneously, a GAN adversarial network is constructed, and fine-grained control of the pronunciation emotional parameters is achieved through adversarial training, further optimizing the final pronunciation effect. This results in speech that is context- and semantically appropriate and emotionally charged, improving the naturalness of the teaching speech. Furthermore, deep learning, combined with user feedback, improves the emotional accuracy of the system's output speech and the speed of the system's speech processing.

[0099] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0100] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0101] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0102] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random-access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0106] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0109] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A method for optimizing the naturalness of teaching speech based on deep learning, characterized in that, Includes the following steps: S1. Construct a semantic recognition model to perform semantic recognition and emotion feature extraction on the input teaching voice sentences, and output the emotion feature vector of the sentences; S2. Construct a pronunciation emotion parameter simulation model, and adjust the speech parameters according to the emotion feature vector of the statement. The speech parameters include fundamental frequency, speech rate, intonation and three-dimensional emotion vector. S3. Construct a GAN adversarial network, using the speech parameters adjusted in step S2 as input, and perform fine-grained optimization through adversarial training to generate the final synthesized speech; S4. Obtain the user's emotional expression feedback on the synthesized speech generated in step S3. Based on the feedback, extract the pronunciation emotion parameters and associated semantic feature vectors corresponding to the synthesized speech marked by the user as having the preferred emotional expression. Calculate the distance between the semantic feature vector of the newly input teaching speech and the stored preferred semantic feature vector. When the distance is less than a set threshold, apply the pronunciation emotion parameters corresponding to the stored preferred semantic feature vector to the parameter adjustment of the new input speech.

2. The method for optimizing the naturalness of teaching speech based on deep learning according to claim 1, characterized in that, In step S1, the semantic recognition model performs semantic recognition and emotion feature extraction, including: The input sentence is segmented into word sequences; each word sequence is tagged with part-of-speech tags; a word vector model is used to map words and their contexts into vector representations; semantic feature vectors are calculated by combining dialogue scenario and dialogue time features; based on the semantic feature vectors, sentiment features are extracted through a sentiment classification model, and the sentiment feature vector of the sentence is output.

3. The method for optimizing the naturalness of teaching speech based on deep learning according to claim 2, characterized in that, The word vector model is either the Word2Vec model or the BERT model. Words are obtained through the BERT model. The vector representation is ,in, For vector dimensions.

4. The method for optimizing the naturalness of teaching speech based on deep learning according to claim 3, characterized in that, sentiment feature vector of a statement : in, For activation function, This is the weight matrix. For bias vector This is a vector representation of the dialogue scene. This is a vector representation of the dialogue time.

5. The method for optimizing the naturalness of teaching speech based on deep learning according to claim 1, characterized in that, In step S2, the voice parameters are adjusted using the following formula: baseband speech rate intonation 3D Emotion Vector , in, For sentiment feature vectors, , , , This is the weight matrix. , , For bias vectors, It is a three-dimensional bias vector.

6. The method for optimizing the naturalness of teaching speech based on deep learning according to claim 1, characterized in that, In step S2, the three-dimensional emotion vector =[ Pleasure level, Activation level (Dominance).

7. The method for optimizing the naturalness of teaching speech based on deep learning according to claim 1, characterized in that, Step S3, the process of fine-grained optimization through adversarial training, includes: The adjusted speech parameters from step S2 are input into the generator of the GAN adversarial network. By generator Mapped to synthetic speech features; discriminator via GAN adversarial network. The synthesized speech features are distinguished from real speech features. Based on the distinction, the generator is optimized by minimizing a preset loss function. and discriminator The parameters are used for adversarial training to achieve fine-grained optimization and generate the final synthesized speech.

8. The method for optimizing the naturalness of teaching speech based on deep learning according to claim 7, characterized in that, The loss function is: in, For random input noise to the generator, This represents the expectation operation. These are real speech features.

9. The method for optimizing the naturalness of teaching speech based on deep learning according to claim 1, characterized in that, In step S4, the parameter adjustment applied to the new input statement is achieved through the following formula: in, For learning rate, The number of high-quality sentence samples, For the pronunciation emotion parameter vector of the new input sentence, For the first The bar is marked as the parameter vector of the preferred synthesized speech.

10. A deep learning-based system for optimizing the naturalness of teaching speech, characterized in that, include: The voice signal receiving module is used to collect the voice signal to be emitted by the device, preprocess it, and obtain the preprocessed signal. The semantic recognition model is used to segment sentences into word sequences, combine the parts of speech of each word, and take into account the context of the sentence, the dialogue scene, and the dialogue time to obtain the sentiment tendency of the sentence and output the sentiment features of the sentence. The pronunciation emotion parameter simulation model is used to parameterize and decompose the specific pronunciation of the sentence to be pronounced, and determine the pronunciation method by changing the parameters according to the imported emotion features; GAN (Generative Adversarial Network) is used to achieve fine-grained control of pronunciation emotion parameters and optimize the final pronunciation effect; The voice output module uses hardware devices such as a vocoder to pronounce the voice to be output after changing various parameters. The user feedback module is used to provide user feedback. After listening to the simulated emotional voice emitted by the device, users can judge its simulation effect and naturalness, and provide feedback through the user feedback module. The data processing module is used to process user feedback data and the final data fed back after GAN adversarial network, and transmit the processed data to the data storage module. The data storage module is used to store user feedback data, adversarial data, various model data, past pronunciation parameter data, and other data. The user feedback module can transmit user feedback data to the data processing module for processing. For processed statements with good naturalness and emotional feedback, the data processing module can independently extract and store the specific adjustment parameters of the statement for easy use in the future and for model improvement.

Images