Voice interaction method and device, computer program product and electronic equipment

Abstract

The invention discloses a voice interaction method and device, a computer program product and electronic equipment. The method relates to the field of artificial intelligence, and comprises the following steps: receiving voice information sent by a target user, and identifying the voice information to obtain first text content; generating a reply text of the first text content through a large language model to obtain second text content; obtaining user information of a target user, generating a user portrait according to the user information, and determining an audio generation requirement of the target user according to the user portrait; and generating audio reply information according to the audio generation requirement and the second text content, and sending the audio reply information to the user side. Through the method and the device, the problem of relatively low accuracy of audio information generated by a voice interaction system in related technologies is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a voice interaction method, device, computer program product, and electronic device. Background Technology

[0002] In the service scenarios of financial institutions, voice interaction systems are becoming increasingly popular as a key technology to enhance customer experience.

[0003] However, current voice interaction systems suffer from two problems in practical applications: First, in open and noisy environments, the accuracy of speech recognition drops significantly, leading to misunderstandings of customer service robots' instructions and incorrect responses. Second, the response style of voice interaction robots is the same for all customers, lacking personalized settings and unable to be flexibly adjusted according to different users, especially for professional terminology and complex needs in the financial field. This results in some customers having difficulty understanding the responses, affecting service quality.

[0004] There is currently no effective solution to the problem of low accuracy of audio information generated by voice interaction systems in related technologies. Summary of the Invention

[0005] The main objective of this application is to provide a voice interaction method, device, computer program product, and electronic device to solve the problem of low accuracy of audio information generated by voice interaction systems in related technologies.

[0006] To achieve the above objectives, according to one aspect of this application, a voice interaction method is provided. The method includes: receiving voice information sent by a target user and recognizing the voice information to obtain first text content; generating a response text of the first text content using a large language model to obtain second text content; obtaining user information of the target user, generating a user profile based on the user information, and determining the target user's audio generation requirements based on the user profile; generating audio response information based on the audio generation requirements and the second text content, and sending the audio response information to the user terminal.

[0007] Optionally, recognizing the speech information to obtain the first text content includes: preprocessing the speech information to obtain processed speech information; performing a recognition operation on the processed speech information to obtain initial text content; and inputting the initial text content and the processed speech information into a text filling model to obtain the first text content, wherein the text filling model is used to correct the initial text content into the first text content based on the processed speech information.

[0008] Optionally, the speech information is preprocessed to obtain the processed speech information by: processing the spectral data of each audio frame in the speech information using a convolutional neural network model to obtain a noise mask for each audio frame, wherein the noise mask is used to characterize the noise intensity of the audio frame; inputting the noise mask of each audio frame into a recurrent neural network model to obtain the frequency gain vector of each audio frame; and adjusting the frequency of each audio frame using the frequency gain vector of each audio frame to obtain the processed speech information.

[0009] Optionally, generating audio response information based on audio generation requirements and the second text content includes: generating an initial audio spectrogram of the second text content using an audio generation model; determining the initial audio spectrogram as a candidate audio spectrogram if no abnormal frames exist in the initial audio spectrogram; marking the abnormal frames in the initial audio spectrogram if abnormal frames exist, and inputting the initial audio spectrogram with marked abnormal frames and the second text content into a text-to-speech model to obtain a corrected initial audio spectrogram, and determining the corrected initial audio spectrogram as a candidate audio spectrogram; adjusting the candidate audio spectrogram according to audio generation requirements to obtain a target audio spectrogram, and generating audio response information based on the target audio spectrogram.

[0010] Optionally, the text-to-speech model is trained as follows: obtain M sample audio spectrograms and the sample text content of each sample audio spectrogram, where M is a positive integer; perform a masking operation on each sample audio spectrogram to obtain M abnormal sample spectrograms; take each sample audio spectrogram, the sample text content of the sample audio spectrogram, and the abnormal sample spectrogram as a set of sample data to obtain M sets of sample data; use the M sets of sample data as the training set to train the initial conversion model to obtain the text-to-speech model.

[0011] Optionally, adjusting the candidate audio spectrograms according to the audio generation requirements to obtain the target audio spectrogram includes: if there is a speech rate requirement in the audio generation requirements, scaling the time axis of the candidate audio spectrograms according to the speech rate requirement to obtain the target audio spectrogram; if there is a timbre requirement in the audio generation requirements, adjusting the high-frequency energy distribution and formant structure of the candidate audio spectrograms according to the speech rate requirement to obtain the target audio spectrogram; if there is an emotional requirement in the audio generation requirements, adjusting the fundamental frequency curve of the candidate audio spectrograms according to the emotional requirement to obtain the target audio spectrogram.

[0012] Optionally, adjusting the candidate audio spectrograms according to the audio generation requirements to obtain the target audio spectrogram includes: determining whether there are visual recognition requirements in the audio generation requirements; if there are visual recognition requirements in the audio generation requirements, acquiring the image information of the target user and recognizing the visual information based on the image information; and adjusting the candidate audio spectrograms according to the visual information to obtain the target audio spectrogram.

[0013] To achieve the above objectives, according to another aspect of this application, a voice interaction device is provided. The device includes: a receiving unit for receiving voice information sent by a target user and recognizing the voice information to obtain first text content; a first generating unit for generating a response text of the first text content using a large language model to obtain second text content; a first acquiring unit for acquiring user information of the target user, generating a user profile based on the user information, and determining the target user's audio generation requirements based on the user profile; and a second generating unit for generating audio response information based on the audio generation requirements and the second text content, and sending the audio response information to the user terminal.

[0014] Optionally, the receiving unit includes: a preprocessing module for preprocessing the speech information to obtain processed speech information; a recognition module for recognizing the processed speech information to obtain initial text content; and an input module for inputting the initial text content and the processed speech information into a text filling model to obtain first text content, wherein the text filling model is used to correct the initial text content into the first text content based on the processed speech information.

[0015] Optionally, the preprocessing module includes: a processing submodule, used to process the spectral data of each audio frame in the speech information using a convolutional neural network model to obtain a noise mask for each audio frame, wherein the noise mask is used to characterize the noise intensity of the audio frame; an input submodule, used to input the noise mask of each audio frame into a recurrent neural network model to obtain the frequency gain vector of each audio frame; and a first adjustment submodule, used to adjust the frequency of each audio frame using the frequency gain vector of each audio frame to obtain the processed speech information.

[0016] Optionally, the second generation unit includes: a first generation module, used to generate an initial audio spectrogram of the second text content using an audio generation model; a determination module, used to determine the initial audio spectrogram as a candidate audio spectrogram if there are no abnormal frames in the initial audio spectrogram; a marking module, used to mark the abnormal frames in the initial audio spectrogram if there are abnormal frames in the initial audio spectrogram, and input the initial audio spectrogram and the second text content after the abnormal frame marking operation is completed into a text-to-speech model to obtain a corrected initial audio spectrogram, and determine the corrected initial audio spectrogram as a candidate audio spectrogram; and a second generation module, used to adjust the candidate audio spectrogram according to the audio generation requirements to obtain a target audio spectrogram, and generate audio response information based on the target audio spectrogram.

[0017] Optionally, the text-to-speech model is trained using the following apparatus: a second acquisition unit, used to acquire M sample audio spectrograms and acquire the sample text content of each sample audio spectrogram, where M is a positive integer; a masking unit, used to perform a masking operation on each sample audio spectrogram to obtain M abnormal sample spectrograms; a determination unit, used to take each sample audio spectrogram, the sample text content of the sample audio spectrogram, and the abnormal sample spectrogram as a set of sample data to obtain M sets of sample data; and a training unit, used to train the initial conversion model using the M sets of sample data as a training set to obtain the text-to-speech model.

[0018] Optionally, the second generation module includes: a scaling submodule, used to scale and adjust the time axis of the candidate audio spectrogram according to the speech rate requirement when there is a speech rate requirement in the audio generation requirements, to obtain the target audio spectrogram; a second adjustment submodule, used to adjust the high-frequency energy distribution and formant structure of the candidate audio spectrogram according to the speech rate requirement when there is a timbre requirement in the audio generation requirements, to obtain the target audio spectrogram; and a third adjustment submodule, used to adjust the fundamental frequency curve of the candidate audio spectrogram according to the emotional requirement when there is an emotional requirement in the audio generation requirements, to obtain the target audio spectrogram.

[0019] Optionally, the second generation module includes: a judgment submodule, used to determine whether there is a visual recognition requirement in the audio generation requirements; an acquisition submodule, used to acquire the image information of the target user when there is a visual recognition requirement in the audio generation requirements, and to identify the visual information based on the image information; and a fourth adjustment submodule, used to adjust the candidate audio spectrogram based on the visual information to obtain the target audio spectrogram.

[0020] To achieve the above objectives, according to another aspect of this application, an electronic device is provided, the electronic device including a memory storing an executable program; and a processor for running the program, wherein the program executes the above-described voice interaction method when it runs.

[0021] To achieve the above objectives, according to another aspect of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the above-described voice interaction method.

[0022] In this embodiment, the system receives voice information sent by a target user, recognizes the voice information to obtain first text content, generates a response text of the first text content using a large language model to obtain second text content, obtains user information of the target user, generates a user profile based on the user information, and determines the audio generation requirements of the target user based on the user profile, generates audio response information based on the audio generation requirements and second text content, and sends the audio response information to the user terminal. By recognizing voice information to obtain first text content and using a model to generate second text content, the system ensures accurate understanding of the input voice. Furthermore, by determining audio generation requirements based on the user profile and adjusting the second text content and audio features when generating audio response information, the system ensures a high degree of matching between the generated audio response information and the user, achieving the goal of enabling the user to understand the audio response information. This improves the accuracy of audio response information and the matching degree between the audio response information and the user, thus solving the technical problem of low accuracy of audio information generated by voice interaction systems in related technologies. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0024] Figure 1 A hardware structure block diagram of a computer terminal for implementing a voice interaction method is shown.

[0025] Figure 2 This is a flowchart of the voice interaction method provided according to Embodiment 1 of this application;

[0026] Figure 3 This is a schematic diagram of a voice interaction device provided according to Embodiment 2 of this application;

[0027] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] It should be noted that the voice interaction methods, devices, computer program products, and electronic devices defined in this disclosure can be used in the field of artificial intelligence, or in any field other than artificial intelligence. The application fields of the voice interaction methods, devices, computer program products, and electronic devices defined in this disclosure are not limited.

[0032] It should be noted that all information, user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) used in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse use. If the user chooses to refuse, the process will proceed to the expert decision-making process. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface. After receiving consent from the aforementioned user or organization, the relevant information is obtained. Users can view the purpose of data use in real time through the authorization interface and have the right to withdraw authorization or delete data at any time. After authorization is withdrawn, the system will terminate the relevant data processing within 24 hours.

[0033] The embodiments or examples disclosed herein are not exhaustive, but merely illustrative of some embodiments or examples, and are not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment or example can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment or example can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment or example can be arbitrarily interchanged. Furthermore, optional methods or examples in a particular embodiment or example can be arbitrarily combined; moreover, embodiments or examples can be arbitrarily combined. For example, some or all steps of different embodiments or examples can be arbitrarily combined, and a particular embodiment or example can be arbitrarily combined with optional methods or examples of other embodiments or examples.

[0034] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0035] Mel spectrogram: A widely used audio representation in acoustic signal processing and speech recognition, it reflects how the intensity of an audio signal changes with frequency and time.

[0036] E2TTS model: End-to-End Text-to-Speech Model, is a deep learning system that directly generates corresponding speech output from text input.

[0037] Convolutional Neural Network (CNN) model: A deep learning model used to process spatial and temporal data, such as images and audio signals. Through structures such as convolutional layers, pooling layers, and fully connected layers, it can identify and extract local features and patterns from the input data.

[0038] Noise mask: In audio processing, a noise mask is a binary or continuous matrix used to indicate which parts of an audio signal are occupied by noise. It can characterize the noise intensity of each audio frame and is an important component of noise suppression algorithms.

[0039] Recurrent Neural Network (RNN) model: A deep learning model used to process sequential data, such as time series analysis. Through the recurrent structure, it can retain historical information in the sequence and is suitable for analyzing the time-varying features of speech signals, such as frequency gain.

[0040] Frequency gain vector: In audio processing, the frequency gain vector is used to describe how to adjust the intensity of different frequency components in an audio signal to improve signal quality or achieve specific purposes, such as enhancing speech clarity or making personalized tone adjustments.

[0041] Example 1

[0042] According to an embodiment of this application, an embodiment of a voice interaction method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware structure block diagram of a computer terminal for implementing a voice interaction method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, processing devices such as microprocessors or programmable logic devices), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface, a universal serial bus port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0044] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0045] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the voice interaction method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned voice interaction method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0047] The display may be, for example, a touchscreen LCD display that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0048] Under the aforementioned operating environment, this application provides the following: Figure 2 The voice interaction method shown. Figure 2 This is a flowchart of the voice interaction method provided in Embodiment 1 of this application, as follows: Figure 2 As shown, the method includes:

[0049] Step S201: Receive voice information sent by the target user and recognize the voice information to obtain the first text content.

[0050] It should be noted that the execution subject in this embodiment can be a voice interaction system. This system can recognize the voice information sent by the user and output corresponding audio response information according to the content of the voice information, thereby completing the voice interaction operation between the user and the system.

[0051] It should be noted that voice information refers to the audio signal input by the user through a voice device (e.g., a mobile phone microphone), which includes the user's voice commands or inquiries.

[0052] Specifically, when the system receives voice information from the target user, it can first send the audio signal to the preprocessing module. Through deep adaptive noise filtering technology, convolutional neural networks and recurrent neural networks are used to analyze and suppress background noise, ensuring the clarity of the target speech.

[0053] After initial processing of the voice information, speech recognition technology is used to parse it to obtain the first text content. When a user asks a question to the voice interaction system, such as "Check account balance," the system first captures the voice command through the microphone. Then, the preprocessing module converts the voice signal into a Mel spectrogram, and the speech recognition engine uses a trained deep learning model to analyze the Mel spectrogram, extract key voice features, and convert it into the text "Check account balance."

[0054] Step S202: Generate the response text of the first text content through the large language model to obtain the second text content.

[0055] Specifically, after obtaining the first text content, the system can call a pre-trained large language model to identify and analyze the text content, and retrieve the corresponding response content from the system's database based on the analysis results. This allows the system to generate an appropriate response text based on the user's question and context, thus obtaining the second text content and ensuring an accurate response to the user's input.

[0056] Step S203: Obtain user information of the target user, generate a user profile based on the user information, and determine the audio generation requirements of the target user based on the user profile.

[0057] It should be noted that user information may include user voice samples, historical interaction records, personal preferences, etc., and should only be obtained with the user's permission. If the user does not allow access to user information, the corresponding audio response information can be generated directly based on the second text content. The user profile is a description of user characteristics generated based on the comprehensive user information. The audio generation requirements can be the audio requirements of the target user predicted based on the user profile, such as speech rate, pitch, and voice style, which are used to guide the speech synthesis process.

[0058] Specifically, the system retrieves target user information from the database to build a user profile. Assuming the user prefers a clear and gentle voice, the system will determine audio generation requirements based on this profile, such as using a gentle tone and clear pronunciation. Furthermore, the system will analyze user voice samples to learn the user's timbre and expression habits to generate a more personalized voice. For example, if the user typically speaks at a slower pace, the system will adjust its audio output to match this preference when generating audio responses, ensuring that the response is both accurate and suited to the user's personal habits.

[0059] It should be noted that the system can utilize speaker embedding technology to adjust the parameters of the text-to-speech synthesizer based on the speech features in the user profile, in order to generate a more personalized voice.

[0060] Step S204: Generate audio response information based on the audio generation requirements and the second text content, and send the audio response information to the user terminal.

[0061] Specifically, after determining the audio generation requirements and the second text content, a non-autoregressive text-to-speech model can be used to adaptively adjust the second text content according to the audio generation requirements. Simultaneously, audio features are determined based on the audio generation requirements, thereby quickly generating an audio response while ensuring speech quality and intelligibility. Furthermore, during the generation process, the system can utilize user audio prompts to adjust the generator parameters, synthesizing a response that better matches the user's speech characteristics.

[0062] For example, assuming the second text content is "account balance is 10,000 yuan," the system can use a non-autoregressive text-to-speech model to quickly generate a speech signal that matches the text, while ensuring the speech style conforms to pre-determined user preferences, such as using a slower speech rate and a softer tone. The generated audio response is encoded into a digital signal in real time and then sent to the user's device via a network protocol for the user to listen to. By employing an efficient non-autoregressive model, the system can complete speech synthesis in a very short time, greatly reducing waiting time, providing users with immediate voice feedback, and enhancing the real-time interactive experience.

[0063] In summary, after receiving user voice information, a personalized response text is generated using a large language model. Audio generation requirements are determined by constructing a user profile, and finally, an audio response matching the user's preferences is generated and sent to the user. This approach, through deep learning technology, effectively suppresses noise in complex environments, achieves personalized speech synthesis and recognition, and dynamically adjusts the output audio. This not only significantly improves the accuracy and efficiency of voice interaction but also greatly enhances the matching degree between the output audio and the user, thereby improving the service quality of financial institutions using the voice interaction system to provide services to users and ensuring the efficiency of users using the voice interaction system for business consultation or processing.

[0064] The voice interaction method provided in this application involves receiving voice information sent by a target user and recognizing the voice information to obtain first text content; generating a response text of the first text content using a large language model to obtain second text content; acquiring user information of the target user, generating a user profile based on the user information, and determining the target user's audio generation requirements based on the user profile; generating audio response information based on the audio generation requirements and second text content, and sending the audio response information to the user terminal. By recognizing voice information to obtain first text content and using a model to generate second text content, the method ensures accurate understanding of the input voice. Furthermore, by determining the audio generation requirements based on the user profile and adjusting the second text content and audio features using the audio generation requirements when generating the audio response information, the method ensures a high degree of matching between the generated audio response information and the user, thereby enabling the user to understand the audio response information. This achieves the technical effect of improving the accuracy of audio response information and the matching degree between audio response information and the user, thus solving the technical problem of low accuracy of audio information generated by voice interaction systems in related technologies.

[0065] To ensure the accuracy of the first text content obtained through recognition, optionally, in the voice interaction method provided in this application embodiment, recognizing the voice information to obtain the first text content includes: preprocessing the voice information to obtain processed voice information; performing a recognition operation on the processed voice information to obtain initial text content; and inputting the initial text content and the processed voice information into a text filling model to obtain the first text content, wherein the text filling model is used to correct the initial text content into the first text content based on the processed voice information.

[0066] It should be noted that preprocessing involves noise filtering, standardization, and feature extraction of the raw speech information to improve the accuracy and efficiency of subsequent recognition. Text completion models are used to correct and enrich text content. Based on the processed speech information, they can correct and refine the initial text content to generate more accurate text.

[0067] Specifically, when the system receives voice information sent by the target user, the voice may contain interference factors such as environmental noise and the speaker's breathing. Therefore, preprocessing is required to analyze the environmental noise in real time and dynamically adjust the filtering parameters to reduce the impact of noise on the voice signal. In the preprocessing operation, a convolutional neural network is first used to perform preliminary filtering of the audio signal to remove noise components in the spectrum. Then, a recurrent neural network is used to further analyze the time series characteristics of the signal, identify and remove residual noise, as well as possible non-verbal sounds from the speaker, such as coughing or breathing sounds, thereby obtaining the processed voice information.

[0068] Furthermore, after obtaining the processed speech information, the speech information can be input into the recognition engine. The engine uses a deep learning-based speech recognition model, such as the Transformer architecture, to convert the audio signal into a Mel spectrogram. By matching the patterns in the Mel spectrogram, the audio signal is converted into a text description to obtain the initial text content.

[0069] Finally, after obtaining the initial text content, it is necessary to further refine and optimize the initial text content using a text filling model to obtain a more accurate first text content.

[0070] It's important to note that the text completion model is a neural network model that modifies the initial text content based on processed speech information. This model not only corrects speech recognition errors but also refines the text according to the speaker's tone, intonation, and contextual information, making it closer to the speaker's original meaning and expression habits. For example, if the initial text is: "Query last month's transaction records," the text completion model can modify it to: "Quickly query last month's transaction records," based on the urgency implied in the processed speech information. This improves the accuracy of the text while preserving the speaker's tone and emotion, thus ensuring a high degree of match between the response and the user's spoken speech.

[0071] This embodiment ensures the accuracy of the recognized text content by preprocessing the speech information and correcting the text information after recognition.

[0072] To ensure the accuracy of the preprocessing operation, optionally, in the voice interaction method provided in this application embodiment, the voice information is preprocessed to obtain the processed voice information, including: processing the spectral data of each audio frame in the voice information using a convolutional neural network model to obtain a noise mask for each audio frame, wherein the noise mask is used to characterize the noise intensity of the audio frame; inputting the noise mask of each audio frame into a recurrent neural network model to obtain the frequency gain vector of each audio frame; and adjusting the frequency of each audio frame using the frequency gain vector of each audio frame to obtain the processed voice information.

[0073] Specifically, during the preprocessing of speech information, the audio signal is first converted into a spectrogram using a Fast Fourier Transform (FFT). Then, a convolutional neural network (CNN) model is used to perform in-depth analysis of the audio frame spectral data in the spectrogram. This CNN model identifies noise patterns in the spectrogram using multiple convolutional layers and generates a noise mask by learning the differences between noise and clear speech in the training data. Each pixel value reflects the noise intensity at the corresponding frequency and time point. For example, suppose a user says, "Please check your account balance," in a noisy environment. The CNN model analyzes the spectrogram, identifies the spectral distribution of background noise, and generates a corresponding noise mask for each audio frame.

[0074] Furthermore, after obtaining the noise mask, the system further processes the noise mask using a recurrent neural network model to generate a frequency gain vector. The recurrent neural network model can capture the time dependence between audio frames. By analyzing the noise intensity changes in each audio frame, it predicts the frequency gain vector and then uses the frequency gain vector to adjust the spectral composition of the audio signal in subsequent steps to reduce the impact of noise. By analyzing the noise mask of the user's speech through the recurrent neural network model, it can learn how to appropriately adjust the frequency components of the audio signal while maintaining speech clarity to achieve the best noise suppression effect.

[0075] Finally, after obtaining the frequency gain vector, the spectrum of each audio frame can be adjusted based on the guidance of the frequency gain vector. Through multiplication or addition operations, the frequency components with high noise intensity in the spectrum are attenuated, while the frequency components of the clear speech signal are maintained or enhanced, resulting in the adjusted audio frame. The adjusted audio frames are then combined to obtain the processed speech information, thereby eliminating noise while ensuring the clarity and intelligibility of the user's speech.

[0076] This embodiment maintains high accuracy and stability of speech recognition by identifying and suppressing noise in speech information, thereby improving the accuracy of subsequent text recognition operations.

[0077] To ensure the accuracy of the audio response information, optionally, in the voice interaction method provided in this application embodiment, generating audio response information based on audio generation requirements and second text content includes: generating an initial audio spectrogram of the second text content using an audio generation model; if there are no abnormal frames in the initial audio spectrogram, determining the initial audio spectrogram as a candidate audio spectrogram; if there are abnormal frames in the initial audio spectrogram, marking the abnormal frames in the initial audio spectrogram, and inputting the initial audio spectrogram and second text content after completing the abnormal frame marking operation into a text-to-speech model to obtain a corrected initial audio spectrogram, and determining the corrected initial audio spectrogram as a candidate audio spectrogram; adjusting the candidate audio spectrogram according to the audio generation requirements to obtain a target audio spectrogram, and generating audio response information based on the target audio spectrogram.

[0078] It should be noted that the audio generation model is used to generate audio feature representations related to text content, such as Mel spectrograms. It is a non-autoregressive deep learning model capable of generating the entire audio feature sequence without predicting each audio frame individually. The initial audio spectrogram is the audio feature map directly generated by the audio generation model based on the second text content. Abnormal frames refer to audio frames that significantly deviate from normal speech features. The text-to-speech model is a deep learning model used to generate high-quality speech output from text input, capable of generating audio features that match the input text, such as Mel spectrograms. The target audio spectrogram is an audio spectrogram adjusted to meet specific audio generation requirements.

[0079] Specifically, after obtaining the second text content, the second text content can be input into the audio generation model. The audio generation model converts the text information into speech feature parameters, such as the fundamental frequency, pitch, timbre and duration of the speech. Then, the speech feature parameters are used to generate a Mel spectrogram to obtain the initial audio spectrogram.

[0080] Furthermore, the generated initial audio spectrogram needs to undergo anomaly frame detection to ensure audio quality. The system can use anomaly detection models to identify anomaly frames in the audio spectrogram caused by model prediction errors, noise interference, or other reasons. If no anomaly frames are detected, it indicates that the audio spectrogram quality is good. In this case, the system directly identifies the initial audio spectrogram as a candidate audio spectrogram and proceeds to the next step. If anomaly frames are detected, a text-to-speech model can be used to correct them. The text-to-speech model predicts and replaces the anomaly frames based on the text content and the characteristics of surrounding audio frames to generate a corrected initial audio spectrogram, i.e., a candidate audio spectrogram. It should be noted that the correction process may include adjusting frequency components, pitch, or duration to ensure the coherence and naturalness of the entire audio output.

[0081] Finally, after obtaining the candidate audio spectrograms, the system can adjust them according to the target user's audio generation requirements, such as speech rate and pitch, to obtain the target audio spectrogram. Then, it can generate audio response information to provide feedback to the user based on the target audio spectrogram. For example, if the user prefers a slower speech rate, the system will adjust the duration parameters of the syllables in the audio spectrogram to generate a slower speech output.

[0082] This embodiment ensures the accuracy of the audio response information generated based on the processed target audio spectrogram by processing the initial audio spectrogram.

[0083] To ensure the accuracy of the output of the text-to-speech model, optionally, in the voice interaction method provided in this application embodiment, the text-to-speech model is trained in the following manner: M sample audio spectrograms are obtained, and the sample text content of each sample audio spectrogram is obtained, where M is a positive integer; a masking operation is performed on each sample audio spectrogram to obtain M abnormal sample spectrograms; each sample audio spectrogram, along with the sample text content of the sample audio spectrogram and the abnormal sample spectrograms, is used as a set of sample data to obtain M sets of sample data; the M sets of sample data are used as a training set to train the initial conversion model to obtain the text-to-speech model.

[0084] Specifically, during model training, the system first randomly selects M audio samples from a pre-set database. These samples contain diverse speech features, such as different speaking speeds, intonations, accents, and background noise. For each sample, speech recognition technology is used to extract the corresponding sample text content and the sample audio spectrogram of each audio file. Taking "Please check your account balance" as an example, the system extracts this text description from the corresponding user recording and simultaneously obtains the Mel-spectrum of the speech as the sample audio spectrogram.

[0085] Furthermore, a masking operation needs to be performed on each sample audio spectrogram. This involves randomly selecting several audio frames and setting their feature values ​​on the spectrogram to zero or a specific value to generate abnormal sample spectrograms. This simulates abnormal data that may be encountered in real-world applications, such as missing audio frames due to network instability or spectrogram anomalies caused by environmental noise. For example, in the Mel spectrogram corresponding to "Please check your account balance," the system may randomly mask the audio frames for the "account" or "balance" portion to test the model's processing capability under conditions of missing information.

[0086] At this point, each sample audio spectrogram, its corresponding sample text content, and the abnormal sample spectrogram after masking can be combined into a complete set of sample data, thus obtaining M sample data. Then, the M sample data can be used as a training set to train the model and obtain the final text-to-speech model. This ensures that the model can learn not only from normal audio spectrograms but also from abnormal situations, thereby improving its performance when facing unknown or complex environments.

[0087] For example, during training, the model learns to predict the correct spectral features of the “account” or “balance” section based on the text content and features of other audio frames, even if the audio frames for the “account” or “balance” section are masked, thus avoiding distortion or omissions when generating audio.

[0088] In this embodiment, by using masking operations to generate anomalous sample spectrograms during model training, the robustness and accuracy of the text-to-speech model are enhanced.

[0089] To ensure the accuracy of the target audio spectrogram, optionally, in the voice interaction method provided in this application embodiment, adjusting the candidate audio spectrogram according to the audio generation requirements to obtain the target audio spectrogram includes: if there is a speech rate requirement in the audio generation requirements, scaling the time axis of the candidate audio spectrogram according to the speech rate requirement to obtain the target audio spectrogram; if there is a timbre requirement in the audio generation requirements, adjusting the high-frequency energy distribution and formant structure of the candidate audio spectrogram according to the speech rate requirement to obtain the target audio spectrogram; if there is an emotional requirement in the audio generation requirements, adjusting the fundamental frequency curve of the candidate audio spectrogram according to the emotional requirement to obtain the target audio spectrogram.

[0090] It should be noted that the speech rate requirement refers to the speech output speed set in the audio generation requirements, including but not limited to acceleration, deceleration, or maintaining the original speed. The timbre requirement refers to adjusting the sound quality characteristics of the speech output in the audio generation requirements, such as brightness and fullness, to match the voice of a specific speaker or the sound effect needs of a specific context. The emotion requirement refers to adjusting the emotional tone of the speech output according to the audio generation requirements. Time axis scaling adjustment refers to adjusting the speech rate by changing the time resolution of the audio spectrogram. High-frequency energy distribution and formant structure adjustment involves adjusting the energy distribution and formant positions in the high-frequency regions of the audio spectrum to change the timbre, making the sound closer to the characteristics of the target speaker or specific emotional expression. Fundamental frequency curve adjustment adds emotional tone by modifying the fundamental frequency curve of the audio.

[0091] Specifically, when the audio generation requirements include a speech rate requirement, the system will scale and adjust the time axis of the candidate audio spectrogram. If acceleration is required, the system will increase the speech rate by compressing the time axis, that is, reducing the duration of each syllable, but must ensure that the clarity and intelligibility of the speech are not compromised; if deceleration is required, the system will slow down the speech rate by expanding the time axis and increasing the duration of each syllable, while maintaining the natural fluency of the speech.

[0092] When timbre requirements are specified, the system adjusts the high-frequency energy distribution and formant structure in the candidate audio spectrograms based on those requirements. The adjustment process involves identifying and modifying frequency components in the Mel spectrogram that affect timbre, such as boosting high-frequency energy to increase the brightness of the sound, or adjusting the timbre according to the formant structure of a specific speaker to mimic their vocal characteristics.

[0093] When the audio generation requirements include emotional requirements, the system analyzes these requirements and adjusts the fundamental frequency curve in the candidate audio spectrogram. Adjusting the fluctuations and trends of the fundamental frequency curve can make the speech express more emotional color, such as making the voice more excited or calmer, and can be flexibly adjusted according to user needs or changes in context.

[0094] This embodiment achieves highly personalized control over speech rate, timbre, and emotion by finely adjusting the time axis, high-frequency energy distribution, formant structure, and fundamental frequency curve of the audio spectrogram, thereby enhancing the flexibility and naturalness of voice interaction and improving the matching degree between the output audio and the user.

[0095] To ensure the accuracy of the target audio spectrogram, optionally, in the voice interaction method provided in this application embodiment, adjusting the candidate audio spectrogram according to the audio generation requirements to obtain the target audio spectrogram includes: determining whether there are visual recognition requirements in the audio generation requirements; if there are visual recognition requirements in the audio generation requirements, acquiring the image information of the target user and recognizing visual information based on the image information; adjusting the candidate audio spectrogram according to the visual information to obtain the target audio spectrogram.

[0096] It should be noted that visual recognition requirements can refer to the need to recognize users' facial expressions or movements in order to assist in the personalization and contextual adaptation of voice interaction.

[0097] Specifically, in some cases, visual recognition may also be involved in generating the target audio spectrogram. In this case, it is necessary to capture video streams or still images of the target user during the interaction process through cameras or other image acquisition devices, so as to make targeted adjustments to the candidate audio spectrogram by analyzing facial expressions and body language.

[0098] For example, if visual information indicates that the user is smiling, suggesting that they are in a good mood, the system may adjust the fundamental frequency curve of the audio, add a slight vibrato, or raise the pitch to make the speech sound more cheerful and friendly; if the user shows a confused expression, the system may slow down the speech rate to make the response clearer and easier for the user to understand.

[0099] It should be noted that user authorization is required before collecting image information. If the user does not authorize the process, this step cannot be performed.

[0100] This embodiment achieves the technical effect of improving the matching degree between the target audio spectrogram and the user by adjusting the candidate audio spectrogram by acquiring the user's image information.

[0101] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0102] Example 2

[0103] This application also provides a voice interaction device. It should be noted that the voice interaction device of this application can be used to execute the voice interaction method provided in the above embodiments. The voice interaction device provided in this application is described below.

[0104] According to an embodiment of this application, an apparatus for implementing the above-described voice interaction method is also provided. Figure 3 This is a schematic diagram of a voice interaction device according to Embodiment 2 of this application, as shown below. Figure 3 As shown, the device includes:

[0105] The receiving unit 31 is used to receive voice information sent by the target user and to recognize the voice information to obtain the first text content.

[0106] The first generation unit 32 is used to generate the response text of the first text content through the large language model, so as to obtain the second text content.

[0107] The first acquisition unit 33 is used to acquire user information of the target user, generate a user profile based on the user information, and determine the audio generation requirements of the target user based on the user profile.

[0108] The second generation unit 34 is used to generate audio response information according to the audio generation requirements and the second text content, and send the audio response information to the user terminal.

[0109] The voice interaction device provided in this application embodiment receives voice information sent by a target user through a receiving unit 31 and recognizes the voice information to obtain first text content; a first generation unit 32 generates a response text of the first text content through a large language model to obtain second text content; a first acquisition unit 33 acquires user information of the target user, generates a user profile based on the user information, and determines the audio generation requirements of the target user based on the user profile; a second generation unit 34 generates audio response information based on the audio generation requirements and the second text content, and sends the audio response information to the user terminal. By recognizing the voice information to obtain the first text content, and using the model to generate the second text content, the device ensures accurate understanding of the input voice. Based on the user profile, the device determines the audio generation requirements, and then adjusts the second text content and audio features using the audio generation requirements when generating the audio response information, thereby ensuring the matching degree between the generated audio response information and the user. This achieves the goal of enabling the user to understand the audio response information, thus improving the accuracy of the audio response information and the matching degree between the audio response information and the user, and solving the technical problem of low accuracy of audio information generated by voice interaction systems in related technologies.

[0110] Optionally, in the voice interaction device provided in the embodiments of this application, the receiving unit 31 includes: a preprocessing module for preprocessing voice information to obtain processed voice information; a recognition module for recognizing the processed voice information to obtain initial text content; and an input module for inputting the initial text content and the processed voice information into a text filling model to obtain first text content, wherein the text filling model is used to correct the initial text content into the first text content based on the processed voice information.

[0111] Optionally, in the voice interaction device provided in this application embodiment, the preprocessing module includes: a processing submodule, used to process the spectral data of each audio frame in the voice information using a convolutional neural network model to obtain a noise mask for each audio frame, wherein the noise mask is used to characterize the noise intensity of the audio frame; an input submodule, used to input the noise mask of each audio frame into a recurrent neural network model to obtain a frequency gain vector for each audio frame; and a first adjustment submodule, used to adjust the frequency of each audio frame using the frequency gain vector of each audio frame to obtain the processed voice information.

[0112] Optionally, in the voice interaction device provided in this application embodiment, the second generation unit 34 includes: a first generation module, used to generate an initial audio spectrogram of the second text content using an audio generation model; a determination module, used to determine the initial audio spectrogram as a candidate audio spectrogram when there are no abnormal frames in the initial audio spectrogram; a marking module, used to mark the abnormal frames in the initial audio spectrogram when there are abnormal frames in the initial audio spectrogram, and input the initial audio spectrogram and the second text content after the abnormal frame marking operation is completed into a text-to-speech model to obtain a corrected initial audio spectrogram, and determine the corrected initial audio spectrogram as a candidate audio spectrogram; and a second generation module, used to adjust the candidate audio spectrogram according to the audio generation requirements to obtain a target audio spectrogram, and generate audio response information according to the target audio spectrogram.

[0113] Optionally, in the voice interaction device provided in this application embodiment, the text-to-speech model is trained by the following devices: a second acquisition unit, used to acquire M sample audio spectrograms and acquire the sample text content of each sample audio spectrogram, where M is a positive integer; a masking unit, used to perform a masking operation on each sample audio spectrogram to obtain M abnormal sample spectrograms; a determination unit, used to take each sample audio spectrogram, the sample text content of the sample audio spectrogram, and the abnormal sample spectrogram as a set of sample data to obtain M sets of sample data; and a training unit, used to train the initial conversion model using the M sets of sample data as a training set to obtain the text-to-speech model.

[0114] Optionally, in the voice interaction device provided in this application embodiment, the second generation module includes: a scaling submodule, used to scale and adjust the time axis of the candidate audio spectrogram according to the speech rate requirement when there is a speech rate requirement in the audio generation requirements, to obtain a target audio spectrogram; a second adjustment submodule, used to adjust the high-frequency energy distribution and formant structure of the candidate audio spectrogram according to the speech rate requirement when there is a timbre requirement in the audio generation requirements, to obtain a target audio spectrogram; and a third adjustment submodule, used to adjust the fundamental frequency curve of the candidate audio spectrogram according to the emotion requirement when there is an emotion requirement in the audio generation requirements, to obtain a target audio spectrogram.

[0115] Optionally, in the voice interaction device provided in the embodiments of this application, the second generation module includes: a judgment submodule, used to judge whether there is a visual recognition requirement in the audio generation requirements; an acquisition submodule, used to acquire the image information of the target user when there is a visual recognition requirement in the audio generation requirements, and to identify the visual information based on the image information; and a fourth adjustment submodule, used to adjust the candidate audio spectrogram based on the visual information to obtain the target audio spectrogram.

[0116] It should be noted that the receiving unit 31, the first generating unit 32, the first acquiring unit 33, and the second generating unit 34 mentioned above correspond to steps S201 to S204 in Embodiment 1. The instances and application scenarios implemented by each of the above units and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and can run in the computer terminal 10 provided in Embodiment 1.

[0117] Example 3

[0118] Embodiments of this application may provide an electronic device. Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4 (Only one is shown) processor 1002, memory 1004, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0119] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0120] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0121] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0122] Example 4

[0123] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the voice interaction method provided in Embodiment 1.

[0124] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0125] Embodiments of this application also provide a computer program product, which, when executed on a data processing device, is a program adapted to perform the steps of a voice interaction method.

[0126] Embodiments of this application also provide a computer-readable storage medium, which includes a stored executable program, wherein the executable program controls the device where the computer-readable storage medium is located to perform the above-described voice interaction method when it runs.

[0127] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0128] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0132] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0133] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A voice interaction method, characterized in that, include: Receive voice information sent by the target user, and recognize the voice information to obtain the first text content; The response text for the first text content is generated using a large language model, thus obtaining the second text content; Obtain the user information of the target user, generate a user profile based on the user information, and determine the audio generation requirements of the target user based on the user profile; An audio response message is generated based on the audio generation requirements and the second text content, and then sent to the user terminal.

2. The method according to claim 1, characterized in that, The first text content obtained by recognizing the voice information includes: The voice information is preprocessed to obtain processed voice information; The processed speech information is then subjected to a recognition operation to obtain the initial text content; The initial text content and the processed speech information are input into a text filling model to obtain the first text content, wherein the text filling model is used to correct the initial text content into the first text content based on the processed speech information.

3. The method according to claim 2, characterized in that, The preprocessed voice information obtained by preprocessing the voice information includes: The spectral data of each audio frame in the speech information is processed using a convolutional neural network model to obtain a noise mask for each audio frame, wherein the noise mask is used to characterize the noise intensity of the audio frame. The noise mask of each audio frame is input into the recurrent neural network model to obtain the frequency gain vector of each audio frame. The frequency of each audio frame is adjusted using the frequency gain vector of each audio frame to obtain the processed speech information.

4. The method according to claim 1, characterized in that, Generating audio response information based on the audio generation requirements and the second text content includes: The initial audio spectrogram of the second text content is generated using an audio generation model; If no abnormal frames are found in the initial audio spectrogram, the initial audio spectrogram is determined as a candidate audio spectrogram. If there are abnormal frames in the initial audio spectrogram, the abnormal frames in the initial audio spectrogram are marked, and the initial audio spectrogram after the abnormal frame marking operation and the second text content are input into the text-to-speech model to obtain the corrected initial audio spectrogram, and the corrected initial audio spectrogram is determined as the candidate audio spectrogram. The candidate audio spectrogram is adjusted according to the audio generation requirements to obtain the target audio spectrogram, and the audio response information is generated based on the target audio spectrogram.

5. The method according to claim 4, characterized in that, The text-to-speech model is trained in the following manner: Obtain M sample audio spectrograms and obtain the sample text content for each sample audio spectrogram, where M is a positive integer; Perform a masking operation on the audio spectrogram of each sample to obtain M abnormal sample spectrograms; Each sample audio spectrogram, along with the sample text content of the sample audio spectrogram and the abnormal sample spectrogram, are treated as a set of sample data, resulting in M ​​sets of sample data. The initial conversion model is trained using the M sets of sample data to obtain the text-to-speech model.

6. The method according to claim 4, characterized in that, The candidate audio spectrogram is adjusted according to the audio generation requirements to obtain the target audio spectrogram, including: If a speech rate requirement exists in the audio generation requirements, the time axis of the candidate audio spectrogram is scaled and adjusted according to the speech rate requirement to obtain the target audio spectrogram; If timbre requirements are included in the audio generation requirements, the high-frequency energy distribution and formant structure of the candidate audio spectrogram are adjusted according to the speech rate requirements to obtain the target audio spectrogram. If there is an emotional requirement in the audio generation requirements, the fundamental frequency curve of the candidate audio spectrogram is adjusted according to the emotional requirement to obtain the target audio spectrogram.

7. The method according to claim 4, characterized in that, The candidate audio spectrogram is adjusted according to the audio generation requirements to obtain the target audio spectrogram, including: Determine whether there are visual recognition requirements in the audio generation requirements; If the visual recognition requirement exists in the audio generation requirements, the image information of the target user is acquired, and visual information is recognized based on the image information; The candidate audio spectrogram is adjusted based on the visual information to obtain the target audio spectrogram.

8. A voice interaction device, characterized in that, include: The receiving unit is used to receive voice information sent by the target user and to recognize the voice information to obtain the first text content; The first generation unit is used to generate the response text of the first text content through a large language model, so as to obtain the second text content; The first acquisition unit is used to acquire user information of the target user, generate a user profile based on the user information, and determine the audio generation requirements of the target user based on the user profile. The second generation unit is used to generate audio response information according to the audio generation requirements and the second text content, and send the audio response information to the user terminal.

9. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the voice interaction method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program executes the voice interaction method according to any one of claims 1 to 7 when it runs.

Images