Dialogue response method and device based on artificial intelligence, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a dialogue response method and device based on artificial intelligence, equipment and a medium, and the method comprises the steps: obtaining the current input voice of a target user; a target response text and a target rhythm style vector of the current input voice are generated through the target multi-mode audio language model and a preset target context database, and the target context database comprises at least one piece of context data of a target user; the context data comprises dialogue information and rhythm style vectors corresponding to the dialogue information; inputting the target response text and the target rhythm style vector into a target style text-to-speech model for speech synthesis processing, generating a target response speech, and extracting a current speech rhythm style vector of the current input speech through the target style text-to-speech model, and updating the target context database according to the current voice rhythm style vector and the current input voice. The rhythm of the answer voice is more natural, and the response speed is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based dialogue response method, device, equipment and medium. Background Art

[0002] With the development of artificial intelligence, human-computer dialogue technology has become more and more mature; and with the rapid development of large-scale language models, it has significantly promoted the development of text-based chatbots. Existing chatbots have a high degree of coherence when answering.

[0003] However, existing chatbots (such as those in banking systems) have unnatural rhythm problems when speaking, and when generating speech, they need to first convert the acquired speech into text through an automatic speech recognition (ASR) model before proceeding to the next step. There are inherent delay limitations in the automatic speech recognition process, which affects the robot's response speed. Summary of the invention

[0004] The present application provides an artificial intelligence dialogue response method, device, computer equipment and medium to solve the technical problems of unnatural dialogue rhythm and slow response speed.

[0005] In a first aspect, a dialogue response method based on artificial intelligence is provided, wherein the dialogue response method based on artificial intelligence is applied to an intelligent dialogue system, wherein the intelligent dialogue system includes a target multimodal audio language model and a target style text-to-speech model, including:

[0006] Get the target user's current input voice;

[0007] Generate a target response text and a target prosody style vector of the current input speech by using the target multimodal audio language model and a preset target context database, wherein the target context database includes at least one context data of the target user, and the context data includes dialogue information and a prosody style vector corresponding to the dialogue information;

[0008] The target response text and the target prosodic style vector are input into the target style text-to-speech model for speech synthesis processing to generate a target response speech, and the current speech prosodic style vector of the current input speech is extracted through the target style text-to-speech model, and the target context database is updated according to the current speech prosodic style vector and the current input speech.

[0009] In a second aspect, an artificial intelligence-based dialogue response device is provided, wherein the artificial intelligence-based dialogue response device is deployed in an intelligent dialogue system, wherein the intelligent dialogue system includes a target multimodal audio language model and a target style text-to-speech model, including:

[0010] A voice acquisition unit, used to acquire the current input voice of the target user;

[0011] A first generating unit, configured to generate a target response text and a target prosody style vector of the current input speech by using the target multimodal audio language model and a preset target context database, wherein the target context database includes at least one context data of the target user, and the context data includes dialogue information and a prosody style vector corresponding to the dialogue information;

[0012] A second generating unit, configured to input the target response text and the target prosodic style vector into the target style text-to-speech model for speech synthesis processing to generate a target response speech;

[0013] An updating unit is used to extract a current speech prosody style vector of the current input speech through the target style text-to-speech model, and update the target context database according to the current speech prosody style vector and the current input speech.

[0014] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned artificial intelligence-based dialogue response method when executing the computer program.

[0015] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned artificial intelligence-based dialogue response method are implemented.

[0016] In the scheme implemented by the above-mentioned artificial intelligence-based dialogue response method, device, computer equipment and storage medium, the current input voice of the target user can be obtained; the target response text and the target prosodic style vector of the current input voice are generated through the target multimodal audio language model and the preset target context database, and the target context database includes at least one context data of the target user, and the context data includes dialogue information and the prosodic style vector corresponding to the dialogue information; the target response text and the target prosodic style vector are input into the target style text-to-speech model for speech synthesis processing to generate the target response speech, and the current speech prosodic style vector of the current input speech is extracted through the target style text-to-speech model, and the target context database is updated according to the current speech prosodic style vector and the current input speech. In the present application, by utilizing the provided intelligent dialogue system, after acquiring the current input voice of the target user, the target multimodal audio language model is used to generate not only the corresponding response text but also the rhythmic style vector. The target style text-to-speech model generates a target response voice with a specific rhythmic style based on the obtained response text and the rhythmic style vector, so that the rhythm of the answer voice is more natural. In addition, after acquiring the user voice, the present application directly processes the voice through a pre-trained target multimodal audio language model, without the need to convert the voice into text before further processing, thereby improving the response speed of the chat robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 It is a schematic diagram of an application environment of a conversation response method based on artificial intelligence in an embodiment of the present application;

[0019] Figure 2 It is a flowchart of a dialogue response method based on artificial intelligence in one embodiment of the present application;

[0020] Figure 3 It is a sub-flow diagram of a dialogue response method based on artificial intelligence in one embodiment of the present application;

[0021] Figure 4 It is a system framework diagram of a conversation response method based on artificial intelligence in one embodiment of the present application;

[0022] Figure 5 is a schematic diagram of a training process of a target multimodal audio language model in an embodiment of the present application;

[0023] Figure 6 It is a structural diagram of a dialogue response device based on artificial intelligence in one embodiment of the present application;

[0024] Figure 7 is a structural schematic diagram of a computer device in one embodiment of the present application;

[0025] Figure 8 It is another structural schematic diagram of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The artificial intelligence-based dialogue response method provided in the embodiment of the present application can be applied in Figure 1 In an application environment, the client communicates with the server through a network. The server can receive the current input speech of the target user through the client, and generate a target response text and a target prosodic style vector of the current input speech through a target multimodal audio language model and a preset target context database, wherein the target context database includes at least one context data of the target user, and the context data includes dialogue information and a prosodic style vector corresponding to the dialogue information; the target response text and the target prosodic style vector are input into a target style text-to-speech model for speech synthesis processing to generate a target response speech, and the target response speech is played through the corresponding client, and the current speech prosodic style vector of the current input speech is extracted through the target style text-to-speech model, and the target context database is updated according to the current speech prosodic style vector and the current input speech. In the present application, for entities such as bank question-and-answer robots, the intelligent dialogue system provided by the present application is used. After obtaining the current input voice of the target user, the target multimodal audio language model is used to generate not only the corresponding response text, but also the rhythmic style vector. The target style text-to-speech model generates a target response voice with a specific rhythmic style based on the obtained response text and the rhythmic style vector, so that the rhythm of the answer voice is more natural. In addition, after obtaining the user voice, the present application directly processes the voice through a pre-trained target multimodal audio language model, without the need to convert the voice into text before further processing, thereby improving the response speed of the robot.

[0028] For example, in finance-related scenarios, the intelligent dialogue system is deployed in a financial chatbot, and users can use the financial chatbot to ask questions about finance, such as how the stock market is doing today. The financial robot then generates a response voice with a specific rhythmic style based on the corresponding context. In medical business-related scenarios, the intelligent dialogue system is deployed in a medical chatbot, and users use the medical robot to ask questions related to diseases and medical treatment. The medical chatbot then generates a response voice with a specific rhythmic style based on the corresponding context, thereby realizing smart medical treatment.

[0029] The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented by an independent server or a server cluster consisting of multiple servers. The present application is described in detail below through specific embodiments.

[0030] See also Figure 2 As shown, Figure 2 A flowchart of an artificial intelligence-based dialogue response method provided in an embodiment of the present application is provided. The artificial intelligence-based dialogue response method is applied to an intelligent dialogue system, and the intelligent dialogue system includes a target multimodal audio language model and a target style text-to-speech model, and includes the following steps:

[0031] S110: Acquire the current input voice of the target user.

[0032] The dialogue response method provided in the present application can be applied to the intelligent dialogue system of the chat robot. The intelligent dialogue system is usually implemented through a server, which can receive the voice sent by the user through the client in real time.

[0033] Among them, the intelligent dialogue system will infer whether the speaker (target user) has finished speaking based on a preset time threshold. For example, if the speaker's silence time exceeds the time threshold, the system will use the voice obtained in the current dialogue stage as the input voice.

[0034] S120, generating a target response text and a target prosodic style vector of the current input speech through the target multimodal audio language model and a preset target context database, wherein the target context database includes at least one context data of the target user, and the context data includes dialogue information and a prosodic style vector corresponding to the dialogue information.

[0035] In this embodiment, after obtaining the current input speech, the current input speech is input into the target multimodal audio language model, so that the target multimodal audio language model generates a corresponding target response text and a target prosodic style vector based on the target context database.

[0036] Specifically, the target multimodal audio language model in this embodiment is a multimodal audio large language model (LLM), which is used for spoken language understanding, text and style response generation, wherein the target multimodal audio language model includes an audio encoder for processing input speech.

[0037] Furthermore, multiple context databases are stored in the local memory, and the target context database may be a context database corresponding to the login account of the target user. At this time, before generating the target response text and the target prosodic style vector of the current input speech through the target multimodal audio language model and the preset target context database, the method further includes: obtaining a target user identifier (such as a login account) corresponding to the target user; determining the context database corresponding to the target user identifier in the preset multiple context databases as the target context database

[0038] In some embodiments, in order to further improve the user experience, the target multimodal audio language model in this embodiment also combines the target timbre style vector when generating text and style. At this time, step S120 includes:

[0039] Get the target timbre style vector;

[0040] The target response text and the target prosody style vector are generated through the target multimodal audio language model, the target timbre style vector and the target context database.

[0041] The target timbre style vector can be determined by the user's timbre selection instruction before the conversation begins; the timbre style vector corresponding to the timbre selection instruction among the preset multiple timbre style vectors is determined as the target timbre style vector.

[0042] In the embodiment of the present application, the timbre style vector indicates that the robot emits different timbres, such as the timbre of a loli, the timbre of a normal male, the timbre of a normal female, the timbre of a child, etc., and the rhythm style vector indicates the speaker's speaking characteristics such as pause habits, lexical stress, formant transition, and speaking rate. The intelligent dialogue system in the present application can use the specified timbre to emit a speech with a rhythm corresponding to the target user, so that the rhythm of the response answer voice is similar to that of the target user's voice, thereby making the rhythm of the dialogue more fluent and improving the user experience.

[0043] In addition, if the user does not make a timbre selection, the system uses a default timbre style vector as the target timbre style vector, wherein the default timbre style vector can be a normal female timbre or other preset timbre, which can be manually set according to actual conditions.

[0044] Furthermore, in some embodiments, the default timbre style vector of the system can also be automatically adjusted, for example, the timbre style vector that is selected most times within a preset time period (for example, within the last 24 hours) is used as the default timbre style vector.

[0045] In some embodiments, see Figure 3 , the generating the target response text and the target prosody style vector by using the target multimodal audio language model, the target timbre style vector and the target context database comprises:

[0046] S1201. Input the target timbre style vector and the current input speech into the target multimodal audio language model, so that the target multimodal audio language model generates a plurality of candidate response data based on the target context database, wherein the candidate response data includes candidate response texts and candidate prosody style vectors.

[0047] In this embodiment, in addition to being used to instruct the target style text-to-speech model to perform speech synthesis of the corresponding timbre, the target timbre style vector is also used to instruct the target multimodal audio language model to generate a response text that is more in line with the timbre style. For example, when other conditions are the same, the response text generated according to different timbre style vectors is different.

[0048] S1202: Determine the response data confidence of each candidate response data according to the text confidence of each candidate response text and the style confidence of each candidate prosodic style vector.

[0049] Specifically, the text confidence of each candidate response text and the style confidence of each candidate prosodic style vector are confidences determined by the target multimodal audio language model. In this embodiment, different weights are preset for the text confidence and the style confidence, respectively, and then the response data confidence is determined in combination with the different weights.

[0050] For example, for each candidate response data, the confidence of the response data is determined according to formula (1), and formula (1) is as follows:

[0051] C = α·c1+β·c2; (1)

[0052] Among them, C is the response data confidence, α is the weight corresponding to the text confidence, c1 is the text confidence, β is the weight corresponding to the style confidence, and c2 is the style confidence.

[0053] S1203: Determine the candidate response data with the highest confidence level as the target response data, and determine the candidate response text and the candidate prosodic style vector in the target response data as the target response text and the target prosodic style vector.

[0054] In this embodiment, after the response data confidence of each candidate response data is obtained, the candidate response data with the highest response data confidence is determined as the target response data.

[0055] In this embodiment, when the system performs dialogue response processing, the timbre style and the rhythm style are decoupled to avoid the timbre interfering with the rhythm calculation, and the rhythm calculation accuracy is high.

[0056] S130, inputting the target response text and the target prosodic style vector into the target style text-to-speech model for speech synthesis processing to generate a target response speech, and extracting the current speech prosodic style vector of the current input speech through the target style text-to-speech model, and updating the target context database according to the current speech prosodic style vector and the current input speech.

[0057] Specifically, the target style text-to-speech model in this embodiment is a style-based text-to-speech (TTS) model, which includes a style encoder. The style encoder in the target style text-to-speech model can extract the prosodic style of the speech.

[0058] Among them, if it is necessary to refer to the target timbre style vector when responding to a conversation, at this time, it is necessary to input the target response text, the target timbre style vector and the target rhythmic style vector into the target style text-to-speech model for speech synthesis processing to generate the target response speech.

[0059] After generating the target response speech, this embodiment extracts the current speech prosody style vector of the current input speech through the style encoder in the target style text-to-speech model, and updates the target context database according to the current speech prosody style vector and the current input speech.

[0060] It can be seen that the target style text-to-speech model in the present application extracts the current speech prosody style vector of the current input speech after generating the target response speech. The speech prosody style vector extraction step does not affect the generation efficiency of the target response speech, thereby ensuring the generation efficiency of the response speech.

[0061] Since speech is inherently longer than its transcribed text, if the target multimodal audio language model operates in the text field, its reference context is longer than the context of the speech-to-speech model. In some embodiments, the dialogue information stored in the target context database is text dialogue information. In this case, the intelligent dialogue system also includes a target speech recognition model; after obtaining the current input speech of the target user, the method also includes:

[0062] Performing text transcription processing on the current input speech through the target speech recognition model to obtain a current dialogue text, and using the current dialogue text as the current dialogue information;

[0063] The updating of the target context database according to the current speech prosody style vector and the current input speech comprises:

[0064] Generate current context data according to the current dialogue information and the current speech prosody style vector; and add the current context data to the target context database.

[0065] It can be seen that the dialogue information stored in the context database of this embodiment is text information, so that the target multimodal audio language model generates more accurate response text.

[0066] For ease of understanding, the artificial intelligence-based dialogue response method provided in this embodiment is described in detail below. Figure 4 :

[0067] The target multimodal audio language model obtains speech input x through the audio encoder n , and then refer to the target timbre style vector θin and the target context database c n-1 , generate the target response text t n+1 and the target prosodic style vector s n+1 , then t n+1 and n+1 The target style text-to-speech model is fed into the target style text-to-speech model according to t n+1 and n+1 Synthesized target response speech x n+1 , and plays it back to the user.

[0068] While playing back the target response speech, the system uses the target speech recognition model to convert the input speech x n Transcribe to text n , and use the style encoder of the target style text-to-speech model from x n Extract the current speech prosodic style vector s n , then t n and n Aggregated into the target context database, the updated target context data is The user processes the next voice input x n+2 .

[0069] like Figure 4 As shown, the target multimodal audio language model applies an additional linear projection head ρ when generating responses. out, to decode the speaking style corresponding to the response text.

[0070] It should be noted that in order to make the rhythm of the response speech more fluent and achieve a more natural style and coherent content, the target multimodal audio language model provided in this embodiment is trained by combining text and rhythmic style. Figure 5 , including the following training steps:

[0071] S210: Create an initial multimodal audio language model.

[0072] S220, obtaining a plurality of sample data, wherein the sample data includes a speech sample, a current context sample library, a prosody style sample vector, and a response sample text, wherein the current context sample library includes at least one context sample, and the context sample includes dialogue sample information and a training style sample vector.

[0073] S230: For each sample data, generate a prosody style output vector and a response output text of the corresponding speech sample through the initial multimodal audio language model and the current context sample library.

[0074] S240. For each sample data, determine a first loss value according to the prosodic style output vector and the prosodic style sample vector, and determine a second loss value according to the response sample text and the response output text.

[0075] S250: Determine target loss values ​​corresponding to each of the sample data respectively according to each of the first loss values, each of the second loss values, a preset style weight, and a preset text weight.

[0076] S260. Converging the initial multimodal audio language model through each of the target loss values ​​to obtain the target multimodal audio language model.

[0077] Specifically, for each sample data, the first loss value is the rhythmic style sample vector With rhythmic style n+1 The distance between the output vectors is calculated as follows:

[0078]

[0079] The second loss value L2 is the cross entropy loss between the maximum probability of the response output text and the response sample text, where when generating the response text, the goal of the initial multimodal audio language model is to maximize the given input speech x n and the preceding text containing style and transcription c n-1 Next, the response output text t n+1 The maximum probability is: in,θin For the style input projection (i.e., the timbre style sample vector, if the model needs to input the reference timbre, it needs to be combined θin Perform model training, otherwise there is no need to combine θin Model training), φ AE is the audio encoder, φ LM Train the parameters for the model.

[0080] The target loss value is the weighted sum of the first loss value and the second loss value, for example, L = L1 + λL2, where λ is a hyperparameter used to balance different loss terms. In this case, the style weight is 1 and the text weight is λ.

[0081] It can be seen that the model takes into account both the coherence of text content and the consistency of style during the training process, so as to generate response text that is both natural and in line with the context.

[0082] To summarize, in the present application, by utilizing the provided intelligent dialogue system, after acquiring the current input voice of the target user, the target multimodal audio language model is used to generate not only the corresponding response text but also the rhythmic style vector. The target style text-to-speech model generates a target response voice with a specific rhythmic style based on the obtained response text and the rhythmic style vector, so that the rhythm of the answer voice is more natural. In addition, after acquiring the user voice, the present application directly processes the voice through the pre-trained target multimodal audio language model, without the need to convert the voice into text before further processing, thereby improving the response speed of the chat robot.

[0083] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0084] In one embodiment, a conversation response device based on artificial intelligence is provided, and the conversation response device based on artificial intelligence corresponds one-to-one with the conversation response method based on artificial intelligence in the above embodiment. Figure 6 As shown, the artificial intelligence-based dialogue response device includes a speech acquisition unit 601, a first generation unit 602, a second generation unit 603, and an update unit 604. The functional modules are described in detail as follows:

[0085] The voice acquisition unit 601 is used to acquire the current input voice of the target user;

[0086] A first generating unit 602 is configured to generate a target response text and a target prosody style vector of the current input speech by using the target multimodal audio language model and a preset target context database, wherein the target context database includes at least one context data of the target user, and the context data includes dialogue information and a prosody style vector corresponding to the dialogue information;

[0087] The second generating unit 603 is used to input the target response text and the target prosodic style vector into the target style text-to-speech model for speech synthesis processing to generate a target response speech;

[0088] The updating unit 604 is configured to extract the current speech prosody style vector of the current input speech through the target style text-to-speech model, and update the target context database according to the current speech prosody style vector and the current input speech.

[0089] In some embodiments, when the first generating unit 602 performs the step of generating the target response text and the target prosody style vector of the current input speech through the target multimodal audio language model and the preset target context database, it is specifically used to:

[0090] Get the target timbre style vector;

[0091] The target response text and the target prosody style vector are generated through the target multimodal audio language model, the target timbre style vector and the target context database.

[0092] In some embodiments, when the first generating unit 602 executes the step of obtaining the target timbre style vector, it is specifically used to:

[0093] Get the user's tone selection instruction;

[0094] A timbre style vector corresponding to the timbre selection instruction among a plurality of preset timbre style vectors is determined as the target timbre style vector.

[0095] In some embodiments, when the first generating unit 602 performs the step of generating the target response text and the target prosody style vector by using the target multimodal audio language model, the target timbre style vector and the target context database, it is specifically used to:

[0096] Inputting the target timbre style vector and the current input speech into the target multimodal audio language model, so that the target multimodal audio language model generates a plurality of candidate response data based on the target context database, the candidate response data including candidate response texts and candidate prosody style vectors;

[0097] Determining the response data confidence of each candidate response data according to the text confidence of each candidate response text and the style confidence of each candidate prosodic style vector;

[0098] The candidate response data with the highest confidence level is determined as the target response data, and the candidate response text and the candidate prosodic style vector in the target response data are determined as the target response text and the target prosodic style vector.

[0099] In some embodiments, the intelligent dialogue system further includes a target speech recognition model; and the artificial intelligence-based dialogue response device further includes:

[0100] A transcription unit, configured to perform text transcription processing on the current input speech through the target speech recognition model to obtain a current dialogue text, and use the current dialogue text as the current dialogue information;

[0101] At this time, the updating unit 604 is specifically used for:

[0102] Generate current context data according to the current dialogue information and the current speech prosody style vector;

[0103] The current context data is added to the target context database.

[0104] In some embodiments, the artificial intelligence-based dialogue response device further includes:

[0105] An identification acquisition unit, used to acquire a target user identification corresponding to the target user;

[0106] A first determining unit is configured to determine a context database corresponding to the target user identifier among a plurality of preset context databases as the target context database.

[0107] In some embodiments, the artificial intelligence-based dialogue response device further includes:

[0108] A creation unit for creating an initial multimodal audio language model;

[0109] A sample acquisition unit, configured to acquire a plurality of sample data, wherein the sample data includes a speech sample, a current context sample library, a prosody style sample vector, and a response sample text, wherein the current context sample library includes at least one context sample, and the context sample includes dialogue sample information and a training style sample vector;

[0110] A third generating unit is used to generate a prosody style output vector and a response output text of a corresponding speech sample for each sample data through the initial multimodal audio language model and the current context sample library;

[0111] a second determining unit, configured to determine, for each of the sample data, a first loss value according to the prosodic style output vector and the prosodic style sample vector, and to determine a second loss value according to the response sample text and the response output text;

[0112] A third determining unit, configured to determine target loss values ​​corresponding to the sample data respectively according to the first loss values, the second loss values, the preset style weights, and the preset text weights;

[0113] A convergence unit is used to converge the initial multimodal audio language model through each of the target loss values ​​to obtain the target multimodal audio language model.

[0114] The present application provides a dialogue response device based on artificial intelligence. After obtaining the current input voice of the target user, the target multimodal audio language model is used to generate not only the corresponding response text but also the rhythmic style vector. The target style text-to-speech model generates a target response voice with a specific rhythmic style based on the obtained response text and the rhythmic style vector, so that the rhythm of the answer voice is more natural. In addition, after obtaining the user voice, the present application directly processes the voice through a pre-trained target multimodal audio language model, without the need to convert the voice into text before further processing, thereby improving the response speed of the chat robot.

[0115] For the specific definition of the artificial intelligence-based dialogue response device, please refer to the definition of the artificial intelligence-based dialogue response method above, which will not be repeated here. Each module in the above-mentioned dialogue response device can be implemented in whole or in part by software, hardware, and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0116] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes a function or step on the server side of a dialogue response method based on artificial intelligence.

[0117] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the client side of a dialogue response method based on artificial intelligence.

[0118] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. An intelligent dialogue system is deployed in the computer device, the intelligent dialogue system including a target multimodal audio language model and a target style text-to-speech model. When the processor executes the computer program, the following steps are implemented:

[0119] Get the target user's current input voice;

[0120] Generate a target response text and a target prosody style vector of the current input speech by using the target multimodal audio language model and a preset target context database, wherein the target context database includes at least one context data of the target user, and the context data includes dialogue information and a prosody style vector corresponding to the dialogue information;

[0121] The target response text and the target prosodic style vector are input into the target style text-to-speech model for speech synthesis processing to generate a target response speech, and the current speech prosodic style vector of the current input speech is extracted through the target style text-to-speech model, and the target context database is updated according to the current speech prosodic style vector and the current input speech.

[0122] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0123] Get the target user's current input voice;

[0124] Generate a target response text and a target prosody style vector of the current input speech by using the target multimodal audio language model and a preset target context database, wherein the target context database includes at least one context data of the target user, and the context data includes dialogue information and a prosody style vector corresponding to the dialogue information;

[0125] The target response text and the target prosodic style vector are input into the target style text-to-speech model for speech synthesis processing to generate a target response speech, and the current speech prosodic style vector of the current input speech is extracted through the target style text-to-speech model, and the target context database is updated according to the current speech prosodic style vector and the current input speech.

[0126] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0128] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0129] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A dialogue response method based on artificial intelligence, characterized in that: The artificial intelligence-based dialogue response method is applied to an intelligent dialogue system, which includes a target multimodal audio language model and a target style text-to-speech model, including: Get the target user's current input voice; Generate a target response text and a target prosody style vector of the current input speech by using the target multimodal audio language model and a preset target context database, wherein the target context database includes at least one context data of the target user, and the context data includes dialogue information and a prosody style vector corresponding to the dialogue information; The target response text and the target prosodic style vector are input into the target style text-to-speech model for speech synthesis processing to generate a target response speech, and the current speech prosodic style vector of the current input speech is extracted through the target style text-to-speech model, and the target context database is updated according to the current speech prosodic style vector and the current input speech.

2. The method according to claim 1, characterized in that The step of generating a target response text and a target prosodic style vector of the current input speech by using the target multimodal audio language model and a preset target context database includes: Get the target timbre style vector; The target response text and the target prosody style vector are generated through the target multimodal audio language model, the target timbre style vector and the target context database.

3. The method according to claim 2, characterized in that The step of obtaining the target timbre style vector includes: Get the user's tone selection instruction; A timbre style vector corresponding to the timbre selection instruction among a plurality of preset timbre style vectors is determined as the target timbre style vector.

4. The method according to claim 2, characterized in that: The generating the target response text and the target prosody style vector by using the target multimodal audio language model, the target timbre style vector and the target context database comprises: Inputting the target timbre style vector and the current input speech into the target multimodal audio language model, so that the target multimodal audio language model generates a plurality of candidate response data based on the target context database, the candidate response data including candidate response texts and candidate prosody style vectors; Determining the response data confidence of each candidate response data according to the text confidence of each candidate response text and the style confidence of each candidate prosodic style vector; The candidate response data with the highest confidence level is determined as the target response data, and the candidate response text and the candidate prosodic style vector in the target response data are determined as the target response text and the target prosodic style vector.

5. The method according to claim 1, characterized in that The intelligent dialogue system further includes a target speech recognition model; after obtaining the current input speech of the target user, the method further includes: Performing text transcription processing on the current input speech through the target speech recognition model to obtain a current dialogue text, and using the current dialogue text as the current dialogue information; The updating of the target context database according to the current speech prosody style vector and the current input speech comprises: Generate current context data according to the current dialogue information and the current speech prosody style vector; The current context data is added to the target context database.

6. The method according to any one of claims 1 to 5, characterized in that Before generating the target response text and the target prosodic style vector of the current input speech through the target multimodal audio language model and the preset target context database, the method further includes: Obtaining a target user identifier corresponding to the target user; A context database corresponding to the target user identifier among a plurality of preset context databases is determined as the target context database.

7. The method according to claim 1, before obtaining the current input voice of the target user, the method further comprises: Creating an initial multimodal audio language model; Acquire a plurality of sample data, wherein the sample data includes a speech sample, a current context sample library, a prosody style sample vector, and a response sample text, wherein the current context sample library includes at least one context sample, and the context sample includes dialogue sample information and a training style sample vector; For each of the sample data, generating a prosodic style output vector and a response output text of the corresponding speech sample through the initial multimodal audio language model and the current context sample library; For each of the sample data, determining a first loss value according to the prosodic style output vector and the prosodic style sample vector, and determining a second loss value according to the response sample text and the response output text; Determine the target loss values ​​corresponding to the sample data respectively according to the first loss values, the second loss values, the preset style weight and the preset text weight; The initial multimodal audio language model is converged through each of the target loss values ​​to obtain the target multimodal audio language model.

8. A dialogue response device based on artificial intelligence, characterized in that: The artificial intelligence-based dialogue response device is deployed in an intelligent dialogue system, and the intelligent dialogue system includes a target multimodal audio language model and a target style text-to-speech model, including: A voice acquisition unit, used to acquire the current input voice of the target user; A first generating unit, configured to generate a target response text and a target prosody style vector of the current input speech by using the target multimodal audio language model and a preset target context database, wherein the target context database includes at least one context data of the target user, and the context data includes dialogue information and a prosody style vector corresponding to the dialogue information; A second generating unit, configured to input the target response text and the target prosodic style vector into the target style text-to-speech model for speech synthesis processing to generate a target response speech; An updating unit is used to extract a current speech prosody style vector of the current input speech through the target style text-to-speech model, and update the target context database according to the current speech prosody style vector and the current input speech.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the artificial intelligence-based dialogue response method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the artificial intelligence-based dialogue response method as claimed in any one of claims 1 to 7 are implemented.

Images