A method, device, storage medium and electronic device for executing a conversation service

Abstract

In the conversation service execution method provided in this specification, the user's user information and the user's input speech in the current round are input into a pre-trained conversation model. User features are obtained based on the user's user information, and audio features and content features are obtained based on the user's input speech in the current round. The obtained user features, audio features, and content features are then fused to obtain fused features. Based on the obtained fused features, a reply speech to the user in the current round is determined, and the reply speech is used to reply to the user. When using this method to determine the reply speech to the user, the user features, content features of the conversation context, and audio features are comprehensively considered to obtain a multimodal fused feature. The reply speech is then determined based on the multimodal fused feature, ensuring the optimal reply speech is obtained through multi-faceted considerations.

Description

Technical Field

[0001] This specification relates to the field of Internet technology, and in particular to a method, device, storage medium, and electronic device for executing a conversational service. Background Art

[0002] Currently, many businesses are using artificial intelligence (AI) conversational methods to replace human interaction in order to save manpower. In some scenarios, AI conversational methods are used to communicate with users. The method used by AI in conversational operations not only affects the success of the associated business but also directly impacts the user experience. Therefore, how conversational services are executed is crucial.

[0003] Existing technologies for conversational services primarily rely on manual experience to configure the transition relationships between a limited number of state machines and the AI's response phrases. This approach considers possible scenarios and user utterances during a conversation, storing each scenario as a state in a state machine. The transition relationships between states are determined based on the user's responses, and further determines the full range of responses the AI ​​can provide to the user in each state.

[0004] However, relying solely on human experience makes it difficult to consider all possible scenarios and all possible user responses. Furthermore, it's difficult to guarantee that each transition between states and the AI's responses to users are optimal. As can be seen, current approaches to executing conversational services are relatively simplistic, lacking flexibility and richness, and failing to ensure that AI conversations can effectively handle a wide range of business needs. Summary of the Invention

[0005] This specification provides a method, device, storage medium, and electronic device for executing a conversation service to at least partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a method for executing a conversation service, including:

[0008] Get the input voice of the user in the current round;

[0009] Inputting the user information of the user into a pre-trained dialogue model to obtain user features through the user information processing subnet in the dialogue model; inputting the input speech of the current round and the dialogue speech of the specified historical round into the feature extraction subnet in the dialogue model to extract audio features and content features through the feature extraction subnet;

[0010] Inputting the user features, the audio features, and the content features into a fusion subnet in the conversation model, so as to fuse the user features, the audio features, and the content features through the fusion subnet to obtain fusion features;

[0011] Determining a reply voice to the user in the current round based on the fusion features;

[0012] Reply to the user using the determined reply voice.

[0013] Optionally, the user information of the user is input into a pre-trained dialogue model to obtain user features through a user information processing subnet in the dialogue model, specifically including:

[0014] Inputting the user's basic information and behavior information into a pre-trained dialogue model to extract user basic features corresponding to the basic information and user behavior features corresponding to the behavior information through an extraction layer in a user information processing subnet in the dialogue model;

[0015] The user basic features and the user behavior features are fused through the fusion layer in the user information processing subnet to obtain user features.

[0016] Optionally, the input speech of the current round and the dialogue speech of the specified historical round are input into the feature extraction subnet in the dialogue model to extract audio features and content features through the feature extraction subnet, specifically including:

[0017] Determine the input content of the current round based on the input voice of the current round, and determine the dialogue content of the specified historical round based on the dialogue voice of the specified historical round;

[0018] Inputting the input speech of the current round and the dialogue speech of the specified historical round into the feature extraction subnet in the dialogue model, so as to extract audio features through the audio feature extraction layer in the feature extraction subnet;

[0019] The input content of the current round and the dialogue content of the specified historical round are input into the feature extraction subnet in the dialogue model, so as to extract content features through the content feature extraction layer in the feature extraction subnet.

[0020] Optionally, determining a reply voice to the user in the current round based on the fusion feature specifically includes:

[0021] Selecting a reply audio to reply to the user in the current round from among various preset standard audios according to the fusion features, and selecting a reply content to reply to the user in the current round from among various preset standard contents;

[0022] The reply audio and the reply content are synthesized to obtain a reply voice.

[0023] Optionally, pre-training the dialogue model specifically includes:

[0024] Obtain voice and user information from historical manual conversations;

[0025] Determine a sample round in each round of historical manual dialogue, and select a designated round from each round before the sample round as a sample historical round;

[0026] Determining the labeled speech according to the speech of the sample round in the historical manual conversation;

[0027] Inputting the user information into the dialogue model to obtain the user features to be optimized through the user information processing subnet in the dialogue model; inputting the speech input by the user in the sample round and the speech in the sample historical round into the feature extraction subnet in the dialogue model to extract the audio features and content features to be optimized through the feature extraction subnet;

[0028] Inputting the user features to be optimized, the audio features to be optimized, and the content features to be optimized into a fusion subnet in the conversation model, so as to fuse the user features to be optimized, the audio features to be optimized, and the content features to be optimized through the fusion subnet to obtain fused features to be optimized;

[0029] According to the fusion features to be optimized, selecting the reply audios replied to the user in the sample rounds from the preset standard audios, and selecting the reply contents replied to the user in the sample rounds from the preset standard contents;

[0030] synthesizing the reply audio and the round content to obtain a reply voice;

[0031] The dialogue model is trained with the optimization of minimizing the difference between the reply speech and the annotated speech.

[0032] Optionally, determining the annotated speech according to the speech of the sample round in the historical manual conversation specifically includes:

[0033] The audio of the voice reply to the user in the sample round is used as the designated audio, and the standard audio with the highest matching degree with the designated audio among the preset standard audios is used as the marked audio;

[0034] The content of the voice reply to the user in the sample round is used as the designated content, and the standard content with the highest matching degree with the designated content among the preset standard contents is used as the marked content;

[0035] The annotated audio and the annotated content are synthesized to obtain annotated speech.

[0036] Optionally, the dialogue model is trained with minimizing the difference between the reply speech and the annotated speech as an optimization goal, specifically including:

[0037] The dialogue model is trained with the optimization goals of minimizing the difference between the reply audio and the annotated audio, and minimizing the difference between the reply content and the annotated content.

[0038] This specification provides a conversation service execution device, the device comprising:

[0039] The acquisition module obtains the input voice input by the user in the current round;

[0040] An extraction module inputs the user's user information into a pre-trained dialogue model to obtain user features through a user information processing subnet in the dialogue model; inputs the input speech of the current round and the dialogue speech of a specified historical round into a feature extraction subnet in the dialogue model to extract audio features and content features through the feature extraction subnet;

[0041] a fusion module, inputting the user features, the audio features, and the content features into a fusion subnet in the conversation model, so as to fuse the user features, the audio features, and the content features through the fusion subnet to obtain a fusion feature;

[0042] a determination module, which determines a reply voice to reply to the user in the current round based on the fusion features;

[0043] The reply module uses the determined reply voice to reply to the user.

[0044] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned conversation service execution method is implemented.

[0045] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned conversation service execution method is implemented.

[0046] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0047] In the conversation service execution method provided in this specification, the user's user information and the user's input speech in the current round are input into a pre-trained conversation model. User features are obtained based on the user's user information, and audio features and content features are obtained based on the user's input speech in the current round. The obtained user features, audio features, and content features are then fused to obtain fused features. Based on the obtained fused features, a reply speech to the user in the current round is determined, and the reply speech is used to reply to the user. When using this method to determine the reply speech to the user, the user features, content features of the conversation context, and audio features are comprehensively considered to obtain a multimodal fused feature. The reply speech is then determined based on the multimodal fused feature, ensuring the optimal reply speech is obtained through multi-faceted considerations. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0049] Figure 1 A flowchart of a method for executing a conversation service in this specification;

[0050] Figure 2 This is a structural diagram of a dialogue model in this specification;

[0051] Figure 3 A schematic diagram of a conversation service execution device provided in this specification;

[0052] Figure 4 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0053] With the continuous development and popularization of technology, various industries are launching more and more services for users. However, the increase in services also means more communication with users. Whether it is promoting new services to users, following up and obtaining feedback from users during service execution, or providing after-sales and customer service to solve user problems, frequent dialogue and communication with users is required.

[0054] The sheer volume of work makes it difficult to communicate with all users in a timely manner relying solely on manual services. Therefore, AI can be leveraged to replace humans in simple conversations, allowing limited human resources to be more effectively allocated. However, AI currently performs poorly in conversations. Existing AI typically responds to users based on predefined procedures or based solely on the user's content. This restrictive operating model often fails to provide a positive user experience, even if it can address customer needs.

[0055] In order to solve the above problems, this specification provides a method for executing a conversation service by utilizing multimodal features to determine the content of a reply to a user.

[0056] To make the purpose, technical solutions, and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0057] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0058] Figure 1 This is a flowchart of a method for executing a conversation service in this specification, which specifically includes the following steps:

[0059] S100: Acquire the input voice input by the user in the current round.

[0060] All steps in the conversation service execution method provided in this specification can be implemented by any electronic device with computing capabilities, such as a server, a terminal, and the like.

[0061] In this manual, a round is defined as a user's voice input and the AI's response to that voice. Each round consists of a user inputting a voice segment, followed by the AI's response to the user's voice input in the current round. The user's voice input in the current round is the user's speech in that round, and responding based on the user's voice input in the current round is the foundation of AI dialogue. Therefore, the user's voice input in the current round must first be obtained.

[0062] S102: Input the user information of the user into a pre-trained dialogue model to obtain user features through the user information processing subnet in the dialogue model; input the input speech of the current round and the dialogue speech of the specified historical round into the feature extraction subnet in the dialogue model to extract audio features and content features through the feature extraction subnet.

[0063] In the method for executing a dialogue service provided in this specification, a pre-trained dialogue model may be used to determine the reply voice to the user. The structure of the dialogue model may be as follows: Figure 2 As shown, the dialogue model may include a user information processing subnet, a feature extraction subnet, a fusion subnet, and a reply voice determination subnet.

[0064] It should be noted that all actions involving user information in this manual, such as obtaining and using user information, are legal and will only be implemented with the user's consent.

[0065] The user's user information is input into the dialogue model, and the user features are extracted through the user information processing subnet in the dialogue model, wherein the user information can be acquired in advance. Similarly, the input voice input by the user in the current round and the dialogue voice of the specified historical round can be input into the dialogue model, and the audio features and content features are extracted through the feature extraction subnet in the model. In this specification, the specified historical round can be any round before the current round in this dialogue, such as the previous round of the current round; the dialogue voice of the specified historical round can include the user's input voice in the specified historical round and the reply voice of the AI ​​to the user.

[0066] The feature extraction subnet can extract the audio features and content features of a speech. The audio features are obtained based on the timbre, pitch, etc. of the speech and are used to characterize the emotional color of the speech. The content features are obtained based on the text content of the speech and are used to characterize the specific content that the speech wants to express.

[0067] S104: Input the user features, the audio features, and the content features into a fusion subnet in the conversation model, so as to fuse the user features, the audio features, and the content features through the fusion subnet to obtain fusion features.

[0068] The feature fusion subnet in the conversation model is used to fuse the user features, audio features, and content features obtained in step S102 to obtain fused features. A variety of fusion methods can be used, such as concatenation and weighting.

[0069] S106: Determine a reply voice to the user in the current round based on the fusion features.

[0070] In this step, the multimodal fusion features determined in step S104 can be input into the reply speech determination subnet in the dialogue model to obtain the reply speech that should be sent to the user in the current round. When determining the reply speech, the complete reply speech can be directly determined and replied to the user, or the reply audio and reply content can be determined separately and then synthesized into a reply speech before replying to the user.

[0071] S108: Reply to the user using the determined reply voice.

[0072] The reply voice determined in step S106 is used to reply to the user in the current round.

[0073] When conducting a conversation using the conversation service execution method provided in this specification, in addition to determining the reply voice based on the content of the user's input voice in the current round, the method also considers the impact of the voice audio and user information on the optimal reply voice. Furthermore, the method also optimizes the reply voice based on the voice of previous rounds in the current conversation. In other words, the method comprehensively considers multiple factors when determining the reply voice, combining user characteristics, audio characteristics, and content characteristics into multimodal features. These multimodal features are then used to determine the final reply voice, ensuring the optimal reply voice is obtained in all circumstances.

[0074] In step S102, the pre-acquired user information may include the user's basic information and the user's behavior information, wherein the user's basic information may include personal information such as the user's age, gender, marital status, and work status, and the user's behavior information includes but is not limited to the number of times the user has performed a specified behavior, the user's use of specified software or items, the user's recent consumption, etc. In dialogue scenarios for different businesses, the user's behavior information may contain different content.

[0075] In this case, when inputting user information into a pre-trained conversation model, the user's basic information and behavior information can be input into the pre-trained conversation model. The extraction layer in the user information processing subnet of the conversation model then extracts basic user features corresponding to the basic information and user behavior features corresponding to the behavior information. The fusion layer in the user information processing subnet then fuses the basic user features and the behavior features to obtain user features. Methods that can be used to fuse the basic user features and the behavior features include, but are not limited to, concatenation and weighting, and are not limited in this specification.

[0076] On the other hand, in step S102, there are also various methods for extracting audio features and content features. This specification provides one method as an example. When extracting audio features and content features, the input content of the current round can be determined based on the input speech of the current round, and the conversation content of a specified historical round can be determined based on the conversation speech of a specified historical round. The input speech of the current round and the conversation speech of the specified historical round are input into the feature extraction subnet of the conversation model to extract audio features through the audio feature extraction layer of the feature extraction subnet. The input content of the current round and the conversation content of the specified historical round are input into the feature extraction subnet of the conversation model to extract content features through the content feature extraction layer of the feature extraction subnet.

[0077] Before extracting audio features, the speech is typically preprocessed, which may include operations such as framing and windowing. After preprocessing, the speech intensity, Mel-Frequency Cepstral Coefficients (MFCCs), and Identity Vectors (I-Vectors) can be extracted as audio features. Alternatively, various neural networks can be used to extract the speech spectrogram to obtain audio features.

[0078] During speech preprocessing, speech can also be converted to text, representing the speech content in text. Because text semantics are often complex, even the same text can convey different meanings in different contexts. Therefore, contextual relationships must be considered when extracting content features. Simple neural networks often fail to capture rich semantics, so models based on long-short-term memory (LSTM) networks can be used to extract content features. Specifically, language models such as TF-IDF, Word2vec, and BERT can be used to extract content features.

[0079] In step S106, a variety of methods can be used to determine the reply voice to the user in the current round based on preset features. For example, an algorithm can be used to enable a model to generate a reply voice to the user in real time based on the preset features, or the optimal voice can be selected from various preset voices as the reply voice to reply to the user. Specifically, based on the fusion features, the reply audio to the user in the current round can be selected from various preset standard audios, and the reply content to the user in the current round can be selected from various preset standard contents; the reply audio and the reply content are synthesized to obtain the reply voice.

[0080] In the conversation service execution method provided in this specification, the conversation model used in the method needs to be pre-trained. In this method, the model is mainly trained in a supervised manner using data from historical manual conversations. Specifically, the voice and user information in the historical manual conversations can be obtained; a sample round is determined in each round of the historical manual conversation, and a specified round is selected from each round before the sample round as a sample historical round; the labeled voice is determined based on the voice of the sample round in the historical manual conversation; the user information is input into the conversation model to obtain the user features to be optimized through the user information processing subnet in the conversation model; the voice input by the user in the sample round and the voice of the sample historical round are input into the feature extraction subnet in ... user information input in the sample round and the voice of the sample historical round are input into the feature extraction subnet in the conversation model to obtain the user features to be optimized through the user information processing subnet in the conversation model; the user information input in the sample round and the voice of the sample historical round are input into the feature extraction subnet in the conversation model to obtain the user features to be optimized through the user information processing subnet in the conversation model; the user information input in the sample round and the voice of the sample historical round are input into the feature extraction subnet in the conversation model to obtain the user features to be optimized through the user information processing subnet in the conversation model The feature extraction subnet extracts the audio features to be optimized and the content features to be optimized; the user features to be optimized, the audio features to be optimized, and the content features to be optimized are input into the fusion subnet in the dialogue model, so as to fuse the user features to be optimized, the audio features to be optimized, and the content features to be optimized through the fusion subnet to obtain the fusion features to be optimized; according to the fusion features to be optimized, the reply audio to the user in the sample round is selected from each preset standard audio, and the reply content to the user in the sample round is selected from each preset standard content; the reply audio and the round content are synthesized to obtain a reply voice; the dialogue model is trained with the difference between the reply voice and the annotated voice being minimized as the optimization.

[0081] When training the model, we first need to obtain speech from historical human conversations as training samples. We then determine sample rounds and sample historical rounds within each round of the human conversation. Sample historical rounds should be selected from rounds preceding the sample round. The sample rounds are the target rounds for this training, meaning that the model must ultimately determine the response speech for the sample rounds.

[0082] Since the model needs to determine the reply speech for the sample round, and in the manual conversation used as the training sample, the manual reply in each round is close to the optimal reply, the optimal reply speech in the sample round can be determined based on the manual reply speech in the sample round and used as the annotated speech. Specifically, the audio of the speech replying to the user in the sample round can be used as the designated audio, and the standard audio with the highest degree of match with the designated audio among the preset standard audios can be used as the annotated audio; the content of the speech replying to the user in the sample round can be used as the designated content, and the standard content with the highest degree of match with the designated content among the preset standard content can be used as the annotated content; the annotated audio and the annotated content can be synthesized to obtain the annotated speech.

[0083] Subsequently, the user features to be optimized, the voice features to be optimized, and the content features to be optimized are determined. The method for determining the user features to be optimized, the audio features to be optimized, and the content features to be optimized can be the same as the method for determining the user features, audio features, and content features in step S102. Similarly, the method for fusing the user features to be optimized, the audio features to be optimized, and the content features to be optimized to obtain the fused features to be optimized can also be the same as the method for obtaining the fused features in step S104.

[0084] After collecting all the data from historical manual conversations, a heuristic algorithm will be used to summarize all possible reply audio and reply content in any situation and input them into the model as the preset standard audio and standard content. During the training process, when the model selects the reply audio and reply content from the preset standard audio and standard content based on the fusion features to be optimized, and synthesizes the reply audio and reply content to obtain the reply voice, it is necessary to determine whether the reply voice currently determined by the model is the best reply voice. The best reply voice at this time is the annotated voice determined in the above training process. If so, this round of training ends. Otherwise, the difference between the reply voice and the annotated voice can be minimized as the optimization, and the conversation model can be trained.

[0085] Since each speech consists of two parts, audio and content, the reply speech can be split into two parts during optimization. The reply audio and reply content are optimized separately, and then resynthesized into the reply speech. Specifically, the difference between the reply audio and the annotated audio can be minimized, and the difference between the reply content and the annotated content can be minimized as the optimization goal. The dialogue model is trained, specifically for the following: Figure 2 The model parameters of the user information processing subnet, feature extraction subnet, fusion subnet, and reply voice determination subnet in the model shown are adjusted.

[0086] When using this method to train a conversation model, the model can not only learn the content that should be replied in various situations, but also fully learn the audio required when facing different scenarios, so that the model can reasonably choose the content of the reply during the conversation. At the same time, the emotional color of the model's conversation can be as close as possible to human conversation, giving users a good conversation experience.

[0087] The above is the conversation service execution method provided in this specification. Based on the same idea, this specification also provides a corresponding conversation service execution device, such as Figure 3 shown.

[0088] Figure 3 This is a diagram of a conversation service execution device provided in this specification, specifically including:

[0089] An acquisition module 200 acquires the input voice input by the user in the current round;

[0090] Extraction module 202 inputs the user's user information into a pre-trained conversation model to obtain user features through the user information processing subnet in the conversation model; inputs the input speech of the current round and the conversation speech of the specified historical round into the feature extraction subnet in the conversation model to extract audio features and content features through the feature extraction subnet;

[0091] A fusion module 204 inputs the user features, the audio features, and the content features into a fusion subnet in the conversation model, so as to fuse the user features, the audio features, and the content features through the fusion subnet to obtain a fusion feature;

[0092] A determination module 206 determines a reply voice to the user in the current round based on the fusion feature;

[0093] The reply module 208 uses the determined reply voice to reply to the user.

[0094] In an alternative embodiment:

[0095] The extraction module 202 is specifically used to input the user's basic information and behavior information into a pre-trained dialogue model, so as to extract the user's basic features corresponding to the basic information and the user's behavior features corresponding to the behavior information through the extraction layer in the user information processing subnet in the dialogue model; and fuse the user's basic features and the user's behavior features through the fusion layer in the user information processing subnet to obtain user features.

[0096] In an alternative embodiment:

[0097] The extraction module 202 is specifically used to determine the input content of the current round based on the input voice of the current round, and determine the dialogue content of the specified historical round based on the dialogue voice of the specified historical round; input the input voice of the current round and the dialogue voice of the specified historical round into the feature extraction subnet in the dialogue model to extract audio features through the audio feature extraction layer in the feature extraction subnet; input the input content of the current round and the dialogue content of the specified historical round into the feature extraction subnet in the dialogue model to extract content features through the content feature extraction layer in the feature extraction subnet.

[0098] In an alternative embodiment:

[0099] The fusion module 204 is specifically used to select a reply audio to reply to the user in the current round from each preset standard audio according to the fusion feature, and select a reply content to reply to the user in the current round from each preset standard content; synthesize the reply audio and the reply content to obtain a reply voice.

[0100] In an alternative embodiment:

[0101] The device also includes a training module 210, which is specifically used to obtain voice and user information in historical artificial conversations; determine a sample round in each round of the historical artificial conversation, and select a specified round from each round before the sample round as a sample historical round; determine the labeled voice according to the voice of the sample round in the historical artificial conversation; input the user information into the conversation model to obtain the user features to be optimized through the user information processing subnet in the conversation model; input the voice input by the user in the sample round and the voice of the sample historical round into the feature extraction subnet in the conversation model to extract the audio features to be optimized and the content to be optimized through the feature extraction subnet. content features; inputting the user features to be optimized, the audio features to be optimized, and the content features to be optimized into the fusion subnet in the dialogue model, so as to fuse the user features to be optimized, the audio features to be optimized, and the content features to be optimized through the fusion subnet to obtain the fusion features to be optimized; according to the fusion features to be optimized, selecting the reply audio to be replied to the user in the sample round from each preset standard audio, and selecting the reply content to be replied to the user in the sample round from each preset standard content; synthesizing the reply audio and round content to obtain a reply voice; and training the dialogue model with the difference between the reply voice and the annotated voice being minimized as the optimization.

[0102] In an alternative embodiment:

[0103] The training module 210 is specifically used to use the audio of the voice replying to the user in the sample round as the designated audio, and use the standard audio with the highest degree of matching with the designated audio among the preset standard audios as the annotated audio; use the content of the voice replying to the user in the sample round as the designated content, and use the standard content with the highest degree of matching with the designated content among the preset standard content as the annotated content; synthesize the annotated audio and the annotated content to obtain the annotated voice.

[0104] In an alternative embodiment:

[0105] The training module 210 is specifically used to train the dialogue model with the optimization goals of minimizing the difference between the reply audio and the annotated audio, and minimizing the difference between the reply content and the annotated content.

[0106] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 Provides a method for executing a conversational service.

[0107] This manual also provides Figure 4 The schematic structure diagram of the electronic device shown in FIG. Figure 4 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0108] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0109] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0110] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0111] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0112] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0113] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0117] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0118] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0119] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0120] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0121] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0123] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0124] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of this application.

Claims

1. A method for executing a conversational service, characterized in that: include: Get the input voice of the user in the current round; Inputting the user's user information into a pre-trained dialogue model to obtain user features through a user information processing subnet in the dialogue model; the user information includes basic information and behavior information of the user; Inputting the input speech of the current round and the dialogue speech of the specified historical round into the feature extraction subnet in the dialogue model to extract audio features and content features through the feature extraction subnet; Inputting the user features, the audio features, and the content features into a fusion subnet in the conversation model, so as to fuse the user features, the audio features, and the content features through the fusion subnet to obtain fusion features; Determining a reply voice to the user in the current round based on the fusion features; Replying to the user using the determined reply voice; Pre-training the dialogue model, specifically including: Obtain voice and user information from historical manual conversations; Determine a sample round in each round of historical manual dialogue, and select a designated round from each round before the sample round as a sample historical round; Determining the labeled speech according to the speech of the sample round in the historical manual conversation; Inputting the user information into the dialogue model to obtain the user features to be optimized through the user information processing subnet in the dialogue model; inputting the speech input by the user in the sample round and the speech in the sample historical round into the feature extraction subnet in the dialogue model to extract the audio features and content features to be optimized through the feature extraction subnet; Inputting the user features to be optimized, the audio features to be optimized, and the content features to be optimized into a fusion subnet in the conversation model, so as to fuse the user features to be optimized, the audio features to be optimized, and the content features to be optimized through the fusion subnet to obtain fused features to be optimized; According to the fusion features to be optimized, selecting the reply audios replied to the user in the sample rounds from the preset standard audios, and selecting the reply contents replied to the user in the sample rounds from the preset standard contents; synthesizing the reply audio and the round content to obtain a reply voice; The dialogue model is trained with the optimization goal of minimizing the difference between the reply speech and the annotated speech.

2. The method according to claim 1, wherein Inputting the user's user information into a pre-trained conversation model to obtain user features through the user information processing subnet in the conversation model specifically includes: Inputting the user's basic information and behavior information into a pre-trained dialogue model to extract user basic features corresponding to the basic information and user behavior features corresponding to the behavior information through an extraction layer in a user information processing subnet in the dialogue model; The user basic features and the user behavior features are fused through the fusion layer in the user information processing subnet to obtain user features.

3. The method according to claim 1, wherein Inputting the input speech of the current round and the dialogue speech of the specified historical round into the feature extraction subnet in the dialogue model to extract audio features and content features through the feature extraction subnet, specifically including: Determine the input content of the current round based on the input voice of the current round, and determine the dialogue content of the specified historical round based on the dialogue voice of the specified historical round; Inputting the input speech of the current round and the dialogue speech of the specified historical round into the feature extraction subnet in the dialogue model, so as to extract audio features through the audio feature extraction layer in the feature extraction subnet; The input content of the current round and the dialogue content of the specified historical round are input into the feature extraction subnet in the dialogue model, so as to extract content features through the content feature extraction layer in the feature extraction subnet.

4. The method according to claim 1, wherein Determining a reply voice to the user in the current round based on the fusion features specifically includes: Selecting a reply audio to reply to the user in the current round from among various preset standard audios according to the fusion features, and selecting a reply content to reply to the user in the current round from among various preset standard contents; The reply audio and the reply content are synthesized to obtain a reply voice.

5. The method according to claim 1, wherein Determining the labeled speech according to the speech of the sample round in the historical manual conversation specifically includes: The audio of the voice reply to the user in the sample round is used as the designated audio, and the standard audio with the highest matching degree with the designated audio among the preset standard audios is used as the marked audio; The content of the voice reply to the user in the sample round is used as the designated content, and the standard content with the highest matching degree with the designated content among the preset standard contents is used as the marked content; The annotated audio and the annotated content are synthesized to obtain annotated speech.

6. The method according to claim 5, wherein The dialogue model is trained with the minimum difference between the reply speech and the annotated speech as the optimization goal, specifically including: The dialogue model is trained with the optimization goals of minimizing the difference between the reply audio and the annotated audio, and minimizing the difference between the reply content and the annotated content.

7. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

Images