Voice reply method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a voice reply method and device, electronic equipment and a storage medium, belongs to the technical field of artificial intelligence, and is suitable for the technical field of medical treatment and the field of financial science and technology. The method comprises the steps of obtaining a sample text; performing speech synthesis on the sample text through a preset initial speech synthesis model to obtain synthesized speech; performing timbre feature conversion on the synthesized voice to obtain a target voice, and performing audio splitting on the target voice to obtain target sub-voices; performing language naturalness evaluation on the target sub-speech to obtain a naturalness score; performing semantic coherence evaluation on the target sub-speech according to the sample text and the naturalness score to obtain a semantic coherence score; screening the initial speech synthesis model according to the semantic coherence score to obtain a target speech synthesis model; and obtaining a question text, and performing voice reply on the question text through the target voice synthesis model. According to the embodiment of the invention, the quality of man-machine voice interaction can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and is applicable to the fields of medical technology and financial technology, and in particular to a voice response method and device, an electronic device, and a storage medium. Background Art

[0002] Currently, various service platforms offer voice assistant services, allowing users to interact with these assistants through voice. For example, in healthcare scenarios, users can seek treatment advice or psychological counseling from voice assistants. In the fintech sector, users can consult with voice assistants about business matters, and the assistants will display the text of the responses and also announce the corresponding voice responses.

[0003] Traditional solutions generate voice responses using the TTS (Text-to-Speech) method, which sequentially preprocesses the response text, performs acoustic modeling, and decodes it using a vocoder. However, the resulting speech lacks semantic coherence and naturalness, resulting in choppy conversations between users and voice assistants and the inability to achieve high-quality human-computer voice interaction. Therefore, improving the quality of human-computer voice interaction has become a pressing issue. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a voice response method and device, an electronic device and a storage medium, aiming to improve the quality of human-computer voice interaction.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a voice reply method, the method comprising:

[0006] Get sample text;

[0007] Performing speech synthesis on the sample text using a preset initial speech synthesis model to obtain synthesized speech;

[0008] Performing timbre feature conversion on the synthesized speech to obtain a target speech, and performing audio decomposition on the target speech to obtain target sub-speech;

[0009] Performing language naturalness evaluation on the target sub-speech to obtain a naturalness score;

[0010] performing semantic coherence evaluation on the target sub-speech according to the sample text and the naturalness score to obtain a semantic coherence score;

[0011] screening the initial speech synthesis model according to the semantic coherence score to obtain a target speech synthesis model;

[0012] Obtain question text, and provide a voice response to the question text through the target speech synthesis model.

[0013] In some embodiments, performing semantic coherence evaluation on the target sub-speech based on the sample text and the naturalness score to obtain a semantic coherence score includes:

[0014] Decomposing the sample text according to the target sub-speech to obtain a sample sub-text; wherein the sample sub-text is the text of the target sub-speech;

[0015] Performing contextual adaptation evaluation based on the sample sub-text and the target sub-speech to obtain an adaptation score;

[0016] The semantic coherence score is obtained by performing calculation based on the sample sub-text, the naturalness score, and the adaptation score.

[0017] In some embodiments, the calculating based on the sample subtext, the naturalness score, and the fit score to obtain the semantic coherence score includes:

[0018] Determining a semantic weight value of the sample subtext according to the sample subtext by using a preset semantic attention model, and using the semantic weight value as the speech weight value of the target sub-speech;

[0019] Calculating the semantic coherence sub-score based on the speech weight value, the naturalness score, and the adaptability score;

[0020] The sum of the semantic coherence sub-scores is calculated to determine the semantic coherence score.

[0021] In some embodiments, performing contextual adaptation evaluation based on the sample sub-text and the target sub-speech to obtain an adaptation score includes:

[0022] Obtaining a text position identifier of the sample subtext in the sample text, and obtaining a speech position identifier of the target subspeech in the target speech;

[0023] Filtering the text position identifiers according to position identifiers adjacent to the voice position identifier to obtain adjacent text identifiers;

[0024] determining an adjacent subtext from the sample subtext according to the adjacent text identifier;

[0025] A cross-modal evaluation is performed based on the adjacent sub-text and the target sub-speech to obtain the fitness score.

[0026] In some embodiments, performing speech synthesis on the sample text using a preset initial speech synthesis model to obtain synthesized speech includes:

[0027] Performing speech synthesis on the sample text using the initial speech synthesis model to obtain original speech;

[0028] Calculating the word accuracy between the original speech and the sample text;

[0029] Performing speech consistency evaluation on the original speech to obtain a consistency score;

[0030] Performing a prosodic diversity assessment on the original speech to obtain a prosodic diversity score;

[0031] The original speech is filtered according to the word accuracy, the consistency score and the prosodic diversity score to obtain the synthesized speech.

[0032] In some embodiments, performing prosodic diversity assessment on the original speech to obtain a prosodic diversity score includes:

[0033] Decomposing the original speech into sentences to obtain original sub-speech;

[0034] Calculating the speaking rate of the original sub-speech to obtain the speaking rate of the sub-speech;

[0035] Performing time domain fundamental frequency extraction on the original sub-speech to obtain a sub-speech fundamental frequency value;

[0036] Calculating a standard deviation of the sub-speech speech rate to obtain a first standard deviation value;

[0037] Calculating the standard deviation between the fundamental frequency values ​​of the sub-speech to obtain a second standard deviation value;

[0038] The first standard deviation value and the second standard deviation are used as the prosodic diversity score.

[0039] In some embodiments, calculating the word accuracy between the original speech and the sample text includes:

[0040] Converting the original speech into text to obtain speech text, and decomposing the speech text into words to obtain speech words;

[0041] Decomposing the sample text into words to obtain sample words;

[0042] Determining the number of different words from the sample words from the spoken words;

[0043] The word accuracy rate is obtained by calculating the ratio of the number of different words to the number of words in the speech words.

[0044] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a voice reply device, the device comprising:

[0045] Acquisition module, used to obtain sample text;

[0046] A speech synthesis module, configured to perform speech synthesis on the sample text using a preset initial speech synthesis model to obtain synthesized speech;

[0047] A speech processing module is used to convert the timbre characteristics of the synthesized speech to obtain a target speech, and to decompose the target speech into audio components to obtain target sub-speech;

[0048] A naturalness evaluation module, configured to evaluate the language naturalness of the target sub-speech to obtain a naturalness score;

[0049] a semantic coherence evaluation module, configured to perform semantic coherence evaluation on the target sub-speech based on the sample text and the naturalness score to obtain a semantic coherence score;

[0050] A screening module, configured to screen the initial speech synthesis model according to the semantic coherence score to obtain a target speech synthesis model;

[0051] The answer module is used to obtain the question text and give a voice answer to the question text through the target speech synthesis model.

[0052] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0053] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.

[0054] The present application proposes a method and device for voice answering, an electronic device, and a storage medium. The method obtains a sample text and performs voice synthesis using a preset initial voice synthesis model, then performs timbre feature conversion on the synthesized voice to obtain a target voice, and decomposes the target voice into multiple target sub-voices. The target sub-voices are then evaluated for language naturalness and semantic coherence, and the initial voice synthesis model is screened according to the evaluation score to determine the target voice synthesis model. Finally, based on the target voice synthesis model, voice generation is performed for the question text to be answered, thereby improving the quality of the final output voice in terms of natural expression and semantic coherence. In this way, the embodiment of the present application optimizes the screening of voice synthesis models by introducing a multi-dimensional evaluation mechanism, and obtains a target voice synthesis model that can generate voice content that is more in line with contextual expectations and user listening habits during the actual broadcast process, effectively improving the interactive coherence and naturalness of the voice assistant, solving the problems of rigidity and unsmoothness in semantic expression, and ultimately achieving high-quality human-computer voice interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of the voice reply method provided in an embodiment of the present application;

[0056] Figure 2 yes Figure 1 Flowchart of step S102 in FIG.

[0057] Figure 3 yes Figure 2 Flowchart of step S202 in FIG.

[0058] Figure 4 yes Figure 2 Flowchart of step S204 in FIG.

[0059] Figure 5 yes Figure 1 Flowchart of step S105 in FIG.

[0060] Figure 6 yes Figure 5 Flowchart of step S502 in FIG.

[0061] Figure 7 yes Figure 5 Flowchart of step S503 in FIG.

[0062] Figure 8 Schematic diagram of the structure of the voice reply device provided in an embodiment of the present application;

[0063] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0065] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0067] First, let’s analyze some of the terms used in this application:

[0068] Time-frequency basis domain: The time-frequency basis domain (TFBD) is a mathematical framework for jointly representing and analyzing signals in both the time and frequency domains. It is a key supporting tool for techniques such as pattern recognition and feature extraction in signal processing and artificial intelligence. By constructing basis functions with local characteristics in both the time and frequency domains, the TFBD describes the local variations of a signal at different times and frequencies, thereby enabling refined analysis of non-stationary signals. Common TFBD forms include wavelet transforms, windowed Fourier transforms, and Hilbert-Huang transforms. They are widely used in speech recognition, music analysis, bioelectric signal processing, seismic wave analysis, and image processing. By expanding on the TFBD, the model can extract multi-scale information related to the time-varying structure of the signal, helping to improve the expressiveness and discriminability of feature representations. As a unified framework connecting time and frequency domain analysis, the TFBD is an important foundation for intelligent computing tasks such as time-frequency energy distribution assessment, speech texture modeling, and time series anomaly detection. It has broad application value in artificial intelligence, communications engineering, and multimodal perception systems.

[0069] Mel-frequency cepstral coefficients (MFCCs): Mel-frequency cepstral coefficients are a parameter representation method for extracting speech or audio signal features based on the human auditory perception characteristics. They are an important feature extraction method in artificial intelligence technologies such as speech recognition, speech synthesis, and audio classification. This method simulates the human ear's nonlinear perception of different frequencies and processes the original speech signal through steps such as short-time Fourier transform, Mel filter bank weighting, logarithmic operation, and discrete cosine transform to obtain a set of spectral feature parameters with strong stability, low dimensionality, and good discriminative ability. Mel-frequency cepstral coefficients are widely used in various audio processing scenarios such as speech recognition, speaker recognition, emotion recognition, speech synthesis, and music retrieval. They are an important intermediate representation for converting continuous speech signals into a suitable analysis form for machine learning models. Because they compress speech feature information while retaining the frequency variations to which the human auditory system is sensitive, MFCCs have become one of the standard feature input forms for most speech processing systems and play an important role in improving the recognition accuracy and robustness of the model.

[0070] Currently, various service platforms offer voice assistant services, allowing users to interact with these assistants through voice. For example, in healthcare scenarios, users can seek treatment advice or psychological counseling from voice assistants. In the fintech sector, users can consult with voice assistants about business matters, and the assistants will display the text of the responses and also announce the corresponding voice responses.

[0071] Traditional solutions generate voice responses using the TTS (Text-to-Speech) method, which sequentially preprocesses the response text, performs acoustic modeling, and decodes it using a vocoder. However, the resulting speech lacks semantic coherence and naturalness, resulting in choppy conversations between users and voice assistants and the inability to achieve high-quality human-computer voice interaction. Therefore, improving the quality of human-computer voice interaction has become a pressing issue.

[0072] Based on this, the embodiments of the present application provide a voice response method and device, an electronic device and a storage medium, aiming to improve the quality of human-computer voice interaction.

[0073] The embodiments of the present application provide a voice reply method and device, electronic device and storage medium, which are specifically illustrated through the following embodiments. First, the voice reply method in the embodiments of the present application is described.

[0074] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0075] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0076] The voice reply method provided in the embodiment of the present application relates to the field of artificial intelligence technology and is applicable to the field of medical technology and financial technology. The voice reply method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the voice reply method, etc., but is not limited to the above forms.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0078] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0079] Figure 1 This is an optional flowchart of the voice reply method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0080] Step S101, obtaining a sample text;

[0081] Step S102, performing speech synthesis on the sample text using a preset initial speech synthesis model to obtain synthesized speech;

[0082] Step S103, performing timbre feature conversion on the synthesized speech to obtain a target speech, and performing audio decomposition on the target speech to obtain target sub-speech;

[0083] Step S104, evaluating the language naturalness of the target sub-speech to obtain a naturalness score;

[0084] Step S105, performing semantic coherence evaluation on the target sub-speech according to the sample text and the naturalness score to obtain a semantic coherence score;

[0085] Step S106, screening the initial speech synthesis model according to the semantic coherence score to obtain a target speech synthesis model;

[0086] Step S107: Obtain the question text and give a voice response to the question text through the target speech synthesis model.

[0087] In steps S101 to S107 shown in the embodiment of the present application, a sample text is obtained and speech synthesis is performed using a preset initial speech synthesis model, and then the synthesized speech is converted into timbre features to obtain a target speech, and the target speech is decomposed into multiple target sub-speech, and then the target sub-speech is evaluated for language naturalness and semantic coherence. The initial speech synthesis model is screened according to the evaluation score to determine the target speech synthesis model, and finally speech generation is performed for the question text to be answered based on the target speech synthesis model, thereby improving the quality of the final output speech in terms of natural expression and semantic coherence. In this way, the embodiment of the present application introduces a multi-dimensional evaluation mechanism to screen and optimize the speech synthesis model, and obtains a target speech synthesis model that can generate speech content that is more in line with contextual expectations and user listening habits during the actual broadcast process, effectively improving the interactive coherence and naturalness of the voice assistant, solving the problems of rigidity and unsmoothness in semantic expression, and ultimately achieving high-quality human-computer speech interaction.

[0088] In step S101 of some embodiments, sample text refers to text information, typically derived from question-and-answer content from a previous business interaction, or from standardized corpus pre-set for simulating and training a speech generation model. For example, in a medical setting, sample text might include, "I've been having headaches for the past few days. What should I do?" Such expressions can be used to guide the speech synthesis model to generate speech output with the characteristics of a medical consultation context. In a fintech setting, sample text might be, "I'd like to learn how to install my credit card bill?"

[0089] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S205:

[0090] Step S201, performing speech synthesis on the sample text through an initial speech synthesis model to obtain original speech;

[0091] Step S202, calculating the word accuracy between the original speech and the sample text;

[0092] Step S203, performing speech consistency evaluation on the original speech to obtain a consistency score;

[0093] Step S204, performing prosodic diversity evaluation on the original speech to obtain a prosodic diversity score;

[0094] Step S205 , filtering the original speech according to the word accuracy, vector similarity and prosodic diversity score to obtain synthesized speech.

[0095] In steps S201 to S205, the embodiment of the present application performs speech synthesis on the sample text using an initial speech synthesis model to obtain the original speech; calculates the word accuracy between the original speech and the sample text; performs speech consistency evaluation on the original speech to obtain a consistency score; performs prosodic diversity evaluation on the original speech to obtain a prosodic diversity score; and filters the original speech based on the word accuracy, consistency score, and prosodic diversity score to obtain the synthesized speech. In this way, the embodiment of the present application introduces evaluation indicators between text and speech at multiple levels, such as word dimension, consistency, and prosodic variation, to jointly screen the speech synthesis results, so that the retained synthesized speech has a high level of accuracy, consistency, and prosodic naturalness, thereby effectively improving the overall quality of speech output in the speech synthesis scenario.

[0096] In step S201 of some embodiments, the initial speech synthesis model is a model for implementing text-to-speech conversion, which can generate corresponding speech content based on the input sample text. The speech content generated by the initial speech synthesis model is also the original speech.

[0097] In one embodiment, the initial speech synthesis model is constructed using a generative speech language model (SLM). This model, trained on large-scale speech tagging data, is capable of predicting speech feature sequences from discrete text units. When a sample text is input into the SLM, the model sequentially generates a corresponding sequence of speech tags based on the text content and converts this sequence into an audio signal, resulting in the original speech.

[0098] See also Figure 3 In some embodiments, step S202 may include but is not limited to steps S301 to S304:

[0099] Step S301, converting the original speech into text to obtain speech text, and breaking down the speech text into words to obtain speech words;

[0100] Step S302, breaking down the sample text into words to obtain sample words;

[0101] Step S303, determining the number of different words from the sample words from the spoken words;

[0102] Step S304 , calculating the ratio of the number of different words to the number of phonetic words to obtain the word accuracy rate.

[0103] In the embodiment of the present application, steps S301 to S304 are as follows: converting the original speech into text to obtain speech text, decomposing the speech text into words to obtain speech words; decomposing the sample text into words to obtain sample words; determining the number of different words from the sample words in the speech words; and calculating the ratio of the number of different words to the number of words in the speech words to obtain the word accuracy. In this way, the embodiment of the present application establishes a correspondence between text and speech based on word content, and then evaluates the accuracy of the speech synthesis result from the lexical level. It can effectively identify the text expression deviation in the speech output, improve the expression accuracy of the speech content in the speech synthesis scenario, and ultimately solve the problem of disconnection between speech and text and inaccurate content expression.

[0104] In step S301 of some embodiments, converting the original speech into text refers to processing the original speech using a speech recognition model and outputting textual information corresponding to the original speech content, i.e., speech text. After obtaining the speech text, the speech text is further segmented using a word segmentation algorithm, dividing the continuous text into discrete word units according to word boundaries in the language to obtain speech words.

[0105] In some embodiments, the word decomposition in step S302 is based on the same principle as the word decomposition of the speech text in step S301, and will not be repeated here.

[0106] In step S303 of some embodiments, determining the number of different words from the sample words in the spoken words refers to comparing the spoken words with the sample words one by one, identifying the parts with inconsistent word content, and counting the total number of inconsistent words.

[0107] In some embodiments, step S304 calculates the word accuracy by calculating the ratio of the number of different words to the total number of spoken words. This involves calculating the ratio of the number of different words to the total number of spoken words and subtracting the ratio from one to obtain the accuracy value. The word accuracy quantifies the degree of lexical content match between the original speech and the sample text.

[0108] In step S203 of some embodiments, when the initial speech synthesis model performs speech synthesis, slight speaker drift may occur, that is, several speech samples of the same speaker may sound like different speakers. Therefore, it is necessary to quantify the degree to which the initial speech synthesis model does not exhibit speaker drift, which is also called consistency assessment.

[0109] In one embodiment, consistency assessment is performed using the ECAPA-TDNN speaker recognition model. Specifically, a number of original speech samples synthesized by an initial speech synthesis model are input into the ECAPA-TDNN model. The initial speech synthesis model is then evaluated for consistency, resulting in an embedding vector corresponding to each original speech. The similarities between these embedding vectors are then calculated, and the average of these similarities is calculated to obtain a consistency score.

[0110] See also Figure 4 In some embodiments, step S204 may include but is not limited to steps S401 to S406:

[0111] Step S401, decomposing the original speech into sentences to obtain original sub-speech;

[0112] Step S402, calculating the speech rate of the original sub-speech to obtain the speech rate of the sub-speech;

[0113] Step S403, extracting the fundamental frequency of the original sub-speech in the time domain to obtain the fundamental frequency value of the sub-speech;

[0114] Step S404, calculating the standard deviation of the sub-speech speech rate to obtain a first standard deviation value;

[0115] Step S405, calculating the standard deviation between the sub-speech fundamental frequency values ​​to obtain a second standard deviation value;

[0116] Step S406: Taking the first standard deviation and the second standard deviation as a prosody diversity score.

[0117] In steps S401 to S406 shown in the embodiment of the present application, the original speech is decomposed into sentences to obtain the original sub-speech; the speech rate of the original sub-speech is calculated to obtain the speech rate of the sub-speech; the time domain fundamental frequency of the original sub-speech is extracted to obtain the sub-speech fundamental frequency value; the standard deviation of the sub-speech speech rate is calculated to obtain a first standard deviation value; the standard deviation between the sub-speech fundamental frequency values ​​is calculated to obtain a second standard deviation value; the first standard deviation value and the second standard deviation value are used as the prosodic diversity score. In this way, the embodiment of the present application performs statistical analysis on the rhythm and pitch changes of the original sub-speech from the two dimensions of speech rate and fundamental frequency, quantifies the degree of change in the prosodic performance of the original speech, thereby improving the expression level and perceptual quality of the speech synthesis content, and ultimately solving the problem of lack of prosodic diversity in the synthesized speech.

[0118] In step S401 of some embodiments, sentence decomposition of the original speech refers to dividing the original speech into multiple audio segments with independent semantic content based on semantic clues such as the grammatical structure, pause boundaries or punctuation positions of the sentences in the original speech to obtain original sub-speech.

[0119] In step S402 of some embodiments, calculating the speaking rate of the original sub-speech means calculating the number of syllables per second based on the duration of each original sub-speech segment and the number of syllables in its corresponding speech text to obtain the speaking rate of the sub-speech.

[0120] In some embodiments, step S403 of the present invention involves extracting the time-domain fundamental frequency of the original sub-speech using a fundamental frequency extraction algorithm based on the time-domain waveform of the speech signal to extract a sequence of fundamental frequency values ​​for each original sub-speech at the speech frame level, thereby obtaining the sub-speech fundamental frequency value. The sub-speech fundamental frequency value is used to reflect the fluctuation characteristics of the speech in the pitch dimension.

[0121] In step S404 of some embodiments, calculating the standard deviation of the sub-speech speed refers to calculating the standard deviation of the speed values ​​corresponding to all original sub-speech to obtain a first standard deviation value. The first standard deviation value reflects the variation range of the original speech in the speed dimension.

[0122] In step S405 of some embodiments, calculating the standard deviation between the fundamental frequency values ​​of the sub-speech refers to performing standard deviation calculation on the set of fundamental frequency values ​​extracted from each original sub-speech to obtain a second standard deviation value. The second standard deviation value is used to represent the degree of fluctuation of the original speech in the fundamental frequency dimension.

[0123] In some embodiments, filtering the original speech based on word accuracy, vector similarity, and prosodic diversity scores in step S205 involves comprehensively evaluating multiple original speech samples based on the three evaluation metrics, eliminating original speech samples that do not meet the set quality standards and retaining original speech samples that meet the criteria as synthesized speech. Filtering methods may include threshold filtering, sorting filtering, or multi-metric weighted scoring filtering.

[0124] In one embodiment, minimum thresholds are set for word accuracy, vector similarity, and prosodic diversity scores. If the original speech is lower than the corresponding threshold in any indicator, it is considered to be of substandard quality and is removed from the candidate set.

[0125] In one embodiment, the above three indicators can be normalized and a weighted comprehensive score can be calculated according to a preset weighting coefficient. The scores are sorted according to the scores, and the original speech with scores in the top N% is selected as the synthesized speech, and the rest is filtered out.

[0126] In step S103 of some embodiments, by performing timbre feature conversion on the synthesized speech, the timbre performance of the generated speech can be made more consistent, thereby reducing the perceptual deviation caused by the timbre differences among speakers.

[0127] Voice feature conversion means: for the multiple speaker voices that may be contained in the synthesized speech, a target speaker voice is selected as a unified standard, and then the voice conversion model is used to convert the voice of the synthesized speech into a speech signal consistent with the voice of the target speaker.

[0128] In one embodiment, the timbre feature conversion is performed on the synthesized speech through the speech conversion model FreeVC to obtain the target speech.

[0129] Audio decomposition of the target speech involves segmenting it into predefined time intervals, sentence boundaries, or semantic structures, so that each audio segment corresponds to a relatively independent speech unit. After audio decomposition is complete, the resulting audio segments become the target sub-speech.

[0130] In step S104 of some embodiments, evaluating the naturalness of the target sub-speech involves performing a comprehensive computational analysis of multiple speech dimensions of the target sub-speech, such as fluency, reasonableness of pauses, and smoothness of intonation transitions, to quantitatively determine a naturalness score that reflects the naturalness of the speech. In one embodiment, the naturalness score is obtained by calculating the Mel-frequency cepstral coefficients of the target sub-speech and then inputting the Mel-frequency cepstral coefficients into a pre-set naturalness scoring network.

[0131] See also Figure 5 In some embodiments, step S105 includes but is not limited to steps S501 to S503:

[0132] Step S501: Decompose the sample text into sentences according to the target sub-speech to obtain a sample sub-text; wherein the sample sub-text is the text of the target sub-speech;

[0133] Step S502: evaluating contextual adaptability of the sample sub-text and the target sub-speech to obtain a adaptability score.

[0134] Step S503 : Calculate the semantic coherence score based on the sample sub-text, the naturalness score, and the adaptation score.

[0135] In steps S501 to S503 shown in the embodiment of the present application, the sample text is decomposed into sentences according to the target sub-speech to obtain a sample sub-text; contextual adaptation is evaluated based on the sample sub-text and the target sub-speech to obtain a adaptation score; and a semantic coherence score is calculated based on the sample sub-text, the naturalness score, and the adaptation score. In this way, the embodiment of the present application introduces a contextual adaptation index to measure the degree of fit between the speech content and the context, and further combines it with the naturalness score for fusion calculation, thereby achieving a quantitative assessment of the coherence of speech expression, so that the target speech synthesis model obtained by subsequent screening based on the semantic coherence score can synthesize semantically coherent speech.

[0136] In step S501 of some embodiments, sentence decomposition refers to searching for a text sentence corresponding to the sentence in the sample text based on the sentence content of the target sub-speech, that is, extracting a text segment that is semantically consistent with the target sub-speech from the sample text as a sample sub-text.

[0137] See also Figure 6 In some embodiments, step S502 includes but is not limited to steps S601 to S604:

[0138] Step S601, obtaining a text position identifier of a sample subtext in a sample text, and obtaining a speech position identifier of a target subspeech in a target speech;

[0139] Step S602, filtering text position markers according to position markers adjacent to the speech position marker to obtain adjacent text markers;

[0140] Step S603, determining adjacent subtexts from the sample subtexts according to the adjacent text identifiers;

[0141] Step S604: Perform cross-modal evaluation based on the adjacent sub-texts and the target sub-speech to obtain a fitness score.

[0142] Steps S601 to S604 shown in the embodiment of the present application are to obtain the text position identifier of the sample subtext in the sample text, and obtain the speech position identifier of the target sub-speech in the target speech; filter the text position identifiers according to the position identifiers adjacent to the speech position identifiers to obtain adjacent text identifiers; determine the adjacent sub-text from the sample sub-text according to the adjacent text identifiers; perform cross-modal evaluation based on the adjacent sub-texts and the target sub-speech to obtain a fitness score. In this way, the embodiment of the present application determines the semantic scope of the sample sub-text in the context by combining the speech position of the target sub-speech in the target speech with the text interval corresponding to its adjacent speech segment, and calculates the fitness of the target sub-speech in the current context based on the cross-modal evaluation method, thereby measuring whether the speech content constitutes a consistent and coherent expression with the surrounding text, and improving the situational fitness of the speech content and the context.

[0143] In step S601 of some embodiments, obtaining the text position identifier of the sample subtext in the sample text and obtaining the voice position identifier of the target sub-speech in the target speech means recording the starting position and ending position of the sample subtext in the sample text as text position identifiers, respectively; and simultaneously recording the starting time point and ending time point of the target sub-speech in the target speech as voice position identifiers.

[0144] In step S602 of some embodiments, the text position identifiers are screened based on position identifiers adjacent to the speech position identifier to obtain adjacent text identifiers. This involves using the speech position identifier of the target sub-speech as a reference point, searching forward or backward for speech segments adjacent to the target sub-speech, and extracting the range of text position identifiers mapped to these speech segments in the sample text. For example, if the target sub-speech is located at the fifth sentence in the target speech, the corresponding text identifiers of sentences 4 and 6 are used as adjacent text identifiers.

[0145] In some embodiments, determining adjacent subtexts from the sample subtext based on the adjacent text identifiers in step S603 refers to extracting the corresponding text content from the sample subtext based on the adjacent text identifiers to form the context text of the target sub-speech, i.e., the adjacent subtexts. For example, if the adjacent text identifiers are sentences 4 and 6, then the adjacent subtexts are the text content of sentences 4 and 6 in the sample text.

[0146] In step S604 of some embodiments, cross-modal evaluation involves performing a correlation analysis on data from two different modalities, text and speech, to determine whether the target sub-speech is appropriately expressed within the current text context, thereby determining the degree of fit between the speech content and the context. For example, if the target sub-speech expresses "OK, I'll be there in a moment," and its adjacent subtext contains "Please register at the front desk as soon as possible," the target sub-speech can be considered to have a strong semantic response relationship with the context, and the cross-modal evaluation result should be a high fit.

[0147] In one embodiment, cross-modal evaluation can be achieved by first extracting text semantic feature vectors from adjacent subtexts to obtain contextual encoding results; then extracting the corresponding speech semantic feature vectors from the target sub-speech; and then using a trained matching model to perform a judgment and output a fit score that represents the degree of contextual match between the speech and text. A higher score indicates that the speech content is more natural and reasonable in the current context.

[0148] Cross-modal evaluation is necessary because, in actual speech synthesis scenarios, relying solely on the naturalness of the speech itself or a one-to-one semantic match with the original text cannot effectively determine whether the speech is expressed appropriately in a specific context. This is especially true in synthesis tasks with strong context, such as multi-round conversations, coherent narratives, or written paragraphs. Speech output often needs to be combined with the surrounding text information to determine whether its semantics are coherent and the tone is appropriate. Cross-modal evaluation provides quantitative support for the contextual appropriateness of speech synthesis results by fusing information from both textual context and speech expression, addressing the existing issues of speech being disconnected from context and abrupt expressions.

[0149] See also Figure 7 In some embodiments, step S503 may include but is not limited to steps S701 to S703:

[0150] Step S701: Determine the semantic weight value of the sample subtext based on the sample subtext by using a preset semantic attention model, and use the semantic weight value as the speech weight value of the target sub-speech;

[0151] Step S702, calculating based on the speech weight value, the naturalness score and the adaptation score to determine the semantic coherence sub-score;

[0152] Step S703: Calculate the sum of the semantic coherence sub-scores to determine the semantic coherence score.

[0153] In steps S701 to S703 shown in the embodiment of the present application, the semantic weight value of the sample sub-text is determined by a preset semantic attention model, and the semantic weight value is used as the speech weight value of the target sub-speech; the semantic coherence sub-score is determined based on the speech weight value, the naturalness score and the fitness score; and the sum of the semantic coherence sub-scores is calculated to determine the semantic coherence score. In this way, the embodiment of the present application introduces a semantic attention model so that the key information in the sample sub-text can obtain corresponding weights in the target sub-speech, thereby clarifying the relative importance of different speech segments in semantic expression, and combining the naturalness score and the fitness score to achieve a multi-dimensional quantitative evaluation of the speech content.

[0154] In step S701 of some embodiments, a semantic attention model is preset to determine the semantic weight value of the sample sub-text based on the sample sub-text, and the semantic weight value is used as the speech weight value of the target sub-speech. This means that the importance of the sample sub-text is evaluated by using the attention mechanism, the weight score corresponding to the sample sub-text is determined, and then the weight score is used as the speech weight value of the target sub-speech, thereby determining the weight of the speech from the level of text semantics.

[0155] In step S702 of some embodiments, the semantic coherence sub-score is determined based on the speech weight value, the naturalness score and the adaptation score, which means that the naturalness score and the adaptation score are weighted based on the speech weight value to calculate the semantic coherence score of each target sub-speech.

[0156] In step S703 of some embodiments, calculating the sum of the semantic coherence sub-scores to determine the semantic coherence score refers to summing the semantic coherence sub-scores corresponding to all target sub-speech to determine the semantic coherence of the target speech.

[0157] In some embodiments, step S106 of the present invention involves screening the initial speech synthesis model based on the semantic coherence score. This involves determining the best-performing speech synthesis model among multiple candidate speech synthesis models based on their semantic coherence performance on the target sub-speech, and using it as the target speech synthesis model for subsequent speech generation. Specifically, a coherence score threshold can be set, and models with scores below the threshold can be eliminated as the target speech synthesis model. Alternatively, the scores of different models can be ranked, and the top-ranked model can be selected based on the ranking results.

[0158] In step S107 of some embodiments, the question text refers to the text to be answered, and giving a voice answer to the question text through the target speech synthesis model refers to inputting the question text into the target speech synthesis model.

[0159] See also Figure 8The present application also provides a voice reply device that can implement the above-mentioned voice reply method, and the device includes:

[0160] An acquisition module 801 is used to acquire a sample text;

[0161] The speech synthesis module 802 is used to perform speech synthesis on the sample text using a preset initial speech synthesis model to obtain synthesized speech;

[0162] The speech processing module 803 is used to convert the timbre characteristics of the synthesized speech to obtain the target speech, and to decompose the target speech into audio components to obtain target sub-speech.

[0163] The naturalness evaluation module 804 is used to evaluate the language naturalness of the target sub-speech and obtain a naturalness score;

[0164] Semantic coherence evaluation module 805, configured to evaluate the semantic coherence of the target sub-speech based on the sample text and the naturalness score to obtain a semantic coherence score;

[0165] A screening module 806 is configured to screen the initial speech synthesis model according to the semantic coherence score to obtain a target speech synthesis model;

[0166] The answering module 807 is used to obtain the question text and provide a voice answer to the question text through the target speech synthesis model.

[0167] The specific implementation of the voice reply device is basically the same as the specific embodiment of the above-mentioned voice reply method, and will not be repeated here.

[0168] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned voice reply method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0169] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0170] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0171] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the voice reply method of the embodiments of this application.

[0172] Input / output interface 903, used to implement information input and output;

[0173] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0174] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0175] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0176] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned voice reply method is implemented.

[0177] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0178] The speech reply method, speech reply device, electronic device and storage medium provided in the embodiment of the present application obtain sample text and perform speech synthesis using a preset initial speech synthesis model, and then perform timbre feature conversion on the synthesized speech to obtain a target speech, and then decompose the target speech into multiple target sub-speech, and then perform language naturalness evaluation and semantic coherence evaluation on the target sub-speech, and screen the initial speech synthesis model according to the evaluation score to determine the target speech synthesis model, and finally generate speech for the question text to be answered based on the target speech synthesis model, thereby improving the quality of the final output speech in terms of natural expression and semantic coherence. In this way, the embodiment of the present application optimizes the screening of speech synthesis models by introducing a multi-dimensional evaluation mechanism, and obtains a target speech synthesis model that can generate speech content that is more in line with context expectations and user listening habits during the actual broadcast process, effectively improving the interactive coherence and naturalness of the voice assistant, solving the problems of rigidity and unsmoothness in semantic expression, and ultimately achieving high-quality human-computer speech interaction.

[0179] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0180] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0181] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0182] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0183] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0184] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0186] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0188] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0189] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A voice reply method, characterized in that: The method comprises: Get sample text; Performing speech synthesis on the sample text using a preset initial speech synthesis model to obtain synthesized speech; Performing timbre feature conversion on the synthesized speech to obtain a target speech, and performing audio decomposition on the target speech to obtain target sub-speech; Performing language naturalness evaluation on the target sub-speech to obtain a naturalness score; performing semantic coherence evaluation on the target sub-speech according to the sample text and the naturalness score to obtain a semantic coherence score; screening the initial speech synthesis model according to the semantic coherence score to obtain a target speech synthesis model; Obtain question text, and provide a voice response to the question text through the target speech synthesis model.

2. The method according to claim 1, characterized in that The performing semantic coherence evaluation on the target sub-speech according to the sample text and the naturalness score to obtain a semantic coherence score includes: Decomposing the sample text according to the target sub-speech to obtain a sample sub-text; wherein the sample sub-text is the text of the target sub-speech; Performing contextual adaptation evaluation based on the sample sub-text and the target sub-speech to obtain an adaptation score; The semantic coherence score is obtained by performing calculation based on the sample sub-text, the naturalness score, and the adaptation score.

3. The method according to claim 2, characterized in that The calculating according to the sample subtext, the naturalness score, and the adaptability score to obtain the semantic coherence score includes: Determining a semantic weight value of the sample subtext according to the sample subtext by using a preset semantic attention model, and using the semantic weight value as the speech weight value of the target sub-speech; Calculating the semantic coherence sub-score based on the speech weight value, the naturalness score, and the adaptability score; The sum of the semantic coherence sub-scores is calculated to determine the semantic coherence score.

4. The method according to claim 2, characterized in that The performing contextual adaptation evaluation based on the sample sub-text and the target sub-speech to obtain an adaptation score includes: Obtaining a text position identifier of the sample subtext in the sample text, and obtaining a speech position identifier of the target subspeech in the target speech; Filtering the text position identifiers according to position identifiers adjacent to the voice position identifier to obtain adjacent text identifiers; determining an adjacent subtext from the sample subtext according to the adjacent text identifier; A cross-modal evaluation is performed based on the adjacent sub-text and the target sub-speech to obtain the fitness score.

5. The method according to any one of claims 1 to 4, characterized in that The method of performing speech synthesis on the sample text by using a preset initial speech synthesis model to obtain synthesized speech includes: Performing speech synthesis on the sample text using the initial speech synthesis model to obtain original speech; Calculating the word accuracy between the original speech and the sample text; Performing speech consistency evaluation on the original speech to obtain a consistency score; Performing a prosodic diversity assessment on the original speech to obtain a prosodic diversity score; The original speech is filtered according to the word accuracy, the consistency score and the prosodic diversity score to obtain the synthesized speech.

6. The method according to claim 5, characterized in that The performing prosodic diversity evaluation on the original speech to obtain a prosodic diversity score includes: Decomposing the original speech into sentences to obtain original sub-speech; Calculating the speaking rate of the original sub-speech to obtain the speaking rate of the sub-speech; Performing time domain fundamental frequency extraction on the original sub-speech to obtain a sub-speech fundamental frequency value; Calculating a standard deviation of the sub-speech speech rate to obtain a first standard deviation value; Calculating the standard deviation between the fundamental frequency values ​​of the sub-speech to obtain a second standard deviation value; The first standard deviation value and the second standard deviation are used as the prosodic diversity score.

7. The method according to claim 5, characterized in that The calculating of the word accuracy between the original speech and the sample text includes: Converting the original speech into text to obtain speech text, and decomposing the speech text into words to obtain speech words; Decomposing the sample text into words to obtain sample words; Determining the number of different words from the sample words from the spoken words; The word accuracy rate is obtained by calculating the ratio of the number of different words to the number of words in the speech words.

8. A voice answering device, characterized in that: The device comprises: Acquisition module, used to obtain sample text; A speech synthesis module, configured to perform speech synthesis on the sample text using a preset initial speech synthesis model to obtain synthesized speech; A speech processing module is used to convert the timbre characteristics of the synthesized speech to obtain a target speech, and to decompose the target speech into audio components to obtain target sub-speech; A naturalness evaluation module, configured to evaluate the language naturalness of the target sub-speech to obtain a naturalness score; a semantic coherence evaluation module, configured to perform semantic coherence evaluation on the target sub-speech based on the sample text and the naturalness score to obtain a semantic coherence score; A screening module, configured to screen the initial speech synthesis model according to the semantic coherence score to obtain a target speech synthesis model; The answer module is used to obtain the question text and give a voice answer to the question text through the target speech synthesis model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images