System audio generation method and device based on multiple rounds of dialogues, equipment and medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to a system audio generation method and device based on multi-round dialogues, equipment and a medium, and the method comprises the steps: obtaining a user text, an audio, a system text and an audio of each dialogue round generated by interaction between a user and a multi-round dialogue system, and for each dialogue round, fusing the user text and the audio, and fusing the system text and the audio, and short-term abstract characteristics of the user and the system are obtained. And based on a system text, carrying out bidirectional semantic fusion on short-term abstract characteristics of the two to obtain user and system abstract characteristics. And interacting all abstract features based on a system text to obtain a long-term feature. And obtaining the total dialogue round number, mapping the total dialogue round number into an intimacy embedding vector, and generating the target system audio of the next dialogue round in combination with the target system text, the long-term feature and the intimacy embedding vector of the next dialogue round. The method can be applied to the business fields of financial science and technology, insurance, medical treatment and the like, and personalized voice interaction requirements can be met.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and is applied to online processing business scenarios such as fintech, insurance, and healthcare. In particular, it relates to a system audio generation method, apparatus, device, and medium based on multi-turn dialogue. Background Technology

[0002] Driven by the development of Large Language Model (LLM) technology, intelligent dialogue systems have made significant progress. LLM possesses powerful internal reasoning, external knowledge acquisition, and the ability to accurately grasp the context of a dialogue, enabling intelligent dialogue systems to generate high-quality, context-appropriate response texts and providing users with a more intelligent and natural communication experience.

[0003] After LLM completes text generation, Text-to-Speech (TTS) technology is key to giving dialogue systems realistic voice interaction capabilities. It converts text content into fluent and natural speech, making communication more intuitive and convenient. With the introduction of a "turn-to-turn" mechanism, the system can remember historical dialogue turns, better simulate the rhythm of human communication, and update user data based on long-term interaction, gaining a deeper understanding and adapting to user preferences, thus enhancing the personalization of communication. However, existing technologies have significant shortcomings in voice interaction. Although LLM-based text generation already possesses advanced features such as "gradual updates" and "actively initiating dialogue," the subsequent TTS module lacks adaptability. Currently, the TTS stage outputs a monotonous tone and style. While it can initially control this through emotional tags, it cannot dynamically and continuously control fine-grained variations in speech rate and tone in response to changes in the number of interactions, intimacy, and context, making it difficult to meet users' demands for high-quality, personalized voice interaction. Summary of the Invention

[0004] The purpose of this application is to propose a system audio generation method, apparatus, computer device, and storage medium based on multi-turn dialogue, so as to solve the problem that existing dialogue systems cannot control the fine-grained dynamic style features of the output speech at a fine-grained level when facing multi-turn dialogue and interaction changes.

[0005] Firstly, a system audio generation method based on multi-turn dialogue is provided, which adopts the following technical solution: The system acquires user text, user audio, system text, and system audio from multiple dialogue rounds generated by the user's interaction with the multi-turn dialogue system. For each dialogue round, it fuses the user text and user audio, and the system text and system audio, to obtain short-term user summary features and short-term system summary features for each dialogue round. Based on the system text of each dialogue round, it performs bidirectional semantic fusion processing on the short-term user summary features and short-term system summary features for each dialogue round to obtain user summary features and system summary features for each dialogue round. Based on the system text of multiple dialogue rounds, it performs feature interaction on all user summary features and system summary features to obtain long-term features. It obtains the total number of dialogue rounds between the user and the multi-turn dialogue system and maps the number of dialogue rounds to an intimacy embedding vector. It obtains the target system text of the multi-turn dialogue system in the next dialogue round and fuses and transforms the target system text, long-term features, and intimacy embedding vector to generate the target system audio of the multi-turn dialogue system in the next dialogue round.

[0006] Secondly, a system audio generation device based on multi-turn dialogue is provided, which adopts the following technical solution: The acquisition module is used to acquire user text, user audio, system text, and system audio generated from multiple dialogue rounds of interaction between the user and the multi-turn dialogue system. The fusion module is used to fuse user text and user audio, and system text and system audio for each dialogue round, to obtain user short-term summary features and system short-term summary features for each dialogue round. The semantic fusion module is used to perform bidirectional semantic fusion processing on the user short-term summary features and system short-term summary features of each dialogue round based on the system text of each dialogue round, so as to obtain the user summary features and system summary features of each dialogue round. The interaction module is used to perform feature interaction between all user summary features and system summary features based on the system text from multiple dialogue rounds to obtain long-term features. The mapping module is used to obtain the total number of dialogue rounds between the user and the multi-turn dialogue system, and map the number of dialogue rounds into an affinity embedding vector; The fusion and transformation module is used to obtain the target system text of the multi-turn dialogue system in the next dialogue round, and to fuse and transform the target system text, long-term features and intimacy embedding vector to generate the target system audio of the multi-turn dialogue system in the next dialogue round.

[0007] Thirdly, a computer device is provided, which adopts the following technical solution: The system acquires user text, user audio, system text, and system audio from multiple dialogue rounds generated by the user's interaction with the multi-turn dialogue system. For each dialogue round, it fuses the user text and user audio, and the system text and system audio, to obtain short-term user summary features and short-term system summary features for each dialogue round. Based on the system text of each dialogue round, it performs bidirectional semantic fusion processing on the short-term user summary features and short-term system summary features for each dialogue round to obtain user summary features and system summary features for each dialogue round. Based on the system text of multiple dialogue rounds, it performs feature interaction on all user summary features and system summary features to obtain long-term features. It obtains the total number of dialogue rounds between the user and the multi-turn dialogue system and maps the number of dialogue rounds to an intimacy embedding vector. It obtains the target system text of the multi-turn dialogue system in the next dialogue round and fuses and transforms the target system text, long-term features, and intimacy embedding vector to generate the target system audio of the multi-turn dialogue system in the next dialogue round.

[0008] Fourthly, a computer-readable storage medium is provided, which adopts the following technical solution: The system acquires user text, user audio, system text, and system audio from multiple dialogue rounds generated by the user's interaction with the multi-turn dialogue system. For each dialogue round, it fuses the user text and user audio, and the system text and system audio, to obtain short-term user summary features and short-term system summary features for each dialogue round. Based on the system text of each dialogue round, it performs bidirectional semantic fusion processing on the short-term user summary features and short-term system summary features for each dialogue round to obtain user summary features and system summary features for each dialogue round. Based on the system text of multiple dialogue rounds, it performs feature interaction on all user summary features and system summary features to obtain long-term features. It obtains the total number of dialogue rounds between the user and the multi-turn dialogue system and maps the number of dialogue rounds to an intimacy embedding vector. It obtains the target system text of the multi-turn dialogue system in the next dialogue round and fuses and transforms the target system text, long-term features, and intimacy embedding vector to generate the target system audio of the multi-turn dialogue system in the next dialogue round.

[0009] Compared with existing technologies, the embodiments of this application have the following main advantages: By acquiring user text and audio, as well as system text and audio, from multi-turn dialogues, and fusing bimodal data of users and systems within a single turn, short-term summary features of users and systems are generated. This can completely preserve the semantic and speech details of a single-turn interaction, avoiding the one-sidedness caused by the lack of single-modal data. Based on the single-turn system text, bidirectional semantic fusion of user and system short-term summary features can strengthen the semantic association of interactions within a single turn, improve semantic consistency, and provide an accurate foundation for multi-turn feature interactions. Combining all summary features of multi-turn system text interactions to construct long-term features can capture the dynamic changes in semantics and emotion in multi-turn dialogues, solving the problem of text-to-speech difficulty in adapting to context. Furthermore, the dialogue turns are mapped to intimacy vectors, which are fused with target text and long-term features to generate target audio, achieving fine-grained dynamic control and meeting personalized needs. Attached Figure Description

[0010] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 A flowchart of an embodiment of a system audio generation method based on multi-turn dialogue according to this application; Figure 3 This is a schematic diagram of a structure of an embodiment of the system audio generation apparatus based on multi-turn dialogue according to this application; Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0013] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0015] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0016] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0017] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0018] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0019] It should be noted that the system audio generation method based on multi-turn dialogue provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the system audio generation device based on multi-turn dialogue is generally set in the server / terminal device.

[0020] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0021] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of a system audio generation method based on multi-turn dialogue according to this application. The system audio generation method based on multi-turn dialogue includes the following steps: Step S201: Obtain user text, user audio, system text, and system audio from multiple dialogue rounds generated by the user's interaction with the multi-turn dialogue system.

[0022] In this context, "user" refers to an individual who interacts with a multi-turn dialogue system via voice. They are the initiator of the interaction and the provider of feedback, completing information exchange with the system by inputting text or voice signals. Examples include individuals consulting a smart customer service system about business issues or engaging in daily conversations with a smart assistant.

[0023] Multi-turn dialogue systems refer to intelligent systems with continuous interaction capabilities. Built upon the contextual understanding capabilities of large language models and text-to-speech technology, they can receive multiple rounds of user input and continuously generate responses to achieve continuous multi-turn communication with users. Examples include intelligent customer service systems that support multi-turn insurance and medical consultations, and intelligent assistants that can provide long-term companionship.

[0024] In this context, multiple dialogue turns refer to a continuous set of interactive units formed during the interaction between the user and the multi-turn dialogue system. Specifically, it consists of the current dialogue turn and the c most recent historical dialogue turns. Here, c is a preset memory capacity parameter, the value of which can be pre-configured according to the hardware computing power, interaction scenario complexity, and other requirements of the actual application scenario. Its core function is to limit the size of the historical window when modeling long-term features, avoiding computational redundancy or semantic interference caused by including too many irrelevant historical turns, and ensuring that long-term features can accurately capture recent key interaction information.

[0025] User text refers to the text information that users input into the multi-turn dialogue system in each round of the conversation. For example, text content such as "check this month's bill" or "adjust appointment time" entered by the user.

[0026] User audio refers to the voice signals input by the user into the multi-turn dialogue system in each round of conversation, used to provide the user's voice modal characteristics. For example, a user speaking "How to activate the service" through their mobile phone microphone.

[0027] In this context, system text refers to the textual responses generated by a multi-turn dialogue system in each round of the dialogue. It represents the system's semantic intent and the content of the information conveyed, and is used to provide users with query results, execute instructions, or initiate new interactions. For example, an intelligent customer service system might generate the textual response "Your bill for this month is XX yuan" in response to a user's inquiry.

[0028] Among them, system audio refers to the voice signal output by the multi-turn dialogue system in each turn of the dialogue. It is generated by converting the system text into speech using text-to-speech technology, and it represents the system's voice expression style, emotional tone and intimacy characteristics, and is used to provide users with voice responses.

[0029] Step S202: For each dialogue round, the user text and user audio, and the system text and system audio are fused to obtain the user short-term summary features and the system short-term summary features for each dialogue round.

[0030] Fusion refers to the integration of different modal data (text and audio) from the same subject (user or system) in the same dialogue round at the feature level. Through feature splicing, element-level addition, or attention weighting, the scattered single-modal features are transformed into unified comprehensive features to make up for the information limitations of single-modal data.

[0031] Among them, the user short-term summary feature refers to the feature vector obtained by fusing user text and user audio in each dialogue round, which represents the user's semantic intent, emotional state and voice features in that round.

[0032] Among them, the system short-term summary features refer to the feature vectors obtained by fusing the system text and system audio in each dialogue round, which represent the system's semantic response, speech style and emotional expression in that round.

[0033] Step S203: Based on the system text of each dialogue round, perform bidirectional semantic fusion processing on the user short-term summary features and system short-term summary features of each dialogue round to obtain the user summary features and system summary features of each dialogue round.

[0034] Among them, bidirectional semantic fusion processing refers to the process of bidirectional information transmission and integration between the user's short-term summary features and the system's short-term summary features for each dialogue round, based on the system text of each dialogue round, through semantic association modeling, attention interaction and other technologies.

[0035] Among them, the user summary feature refers to the feature vector obtained by performing bidirectional semantic fusion processing on the user short-term summary feature of each dialogue round and the system short-term summary feature of the corresponding round, which represents the comprehensive semantic and feature information of the user's interaction with the system in that round.

[0036] Among them, the system summary feature refers to the feature vector obtained by performing bidirectional semantic fusion processing on the system short-term summary feature of each dialogue round and the user short-term summary feature of the corresponding round, which represents the comprehensive semantic and feature information of the system after interacting with the user in that round.

[0037] Step S204: Based on the system text from multiple dialogue rounds, perform feature interaction between all user summary features and system summary features to obtain long-term features.

[0038] Feature interaction refers to the process of integrating and modeling the information across rounds of user summary features and system summary features based on system text from multiple dialogue rounds, using techniques such as gating, attention mechanisms, and Transformer encoding.

[0039] Among them, long-term features refer to feature vectors obtained by interacting user summary features and system summary features across multiple dialogue rounds. These vectors represent global information such as semantic preferences, sentiment tendencies, and style adaptations between users and the system during long-term interactions.

[0040] Step S205: Obtain the total number of dialogue rounds between the user and the multi-turn dialogue system, and map the number of dialogue rounds to an intimacy embedding vector.

[0041] Among them, the number of dialogue rounds refers to the total number of cumulative interaction rounds between the user and the multi-turn dialogue system from the first interaction to the current round. It represents the duration and intimacy of the interaction between the two parties and is used to map it into an intimacy embedding vector to reflect the intimacy of the interaction.

[0042] The intimacy embedding vector refers to a low-dimensional feature vector obtained by mapping the number of dialogue rounds to a preset vector space. It represents the level of intimacy between the user and the system and is used to provide feature support for the intimacy expression of the target system's audio. For example, 10 dialogue rounds can be mapped to a 5-dimensional vector representing "medium intimacy".

[0043] Step S206: Obtain the target system text of the multi-turn dialogue system in the next dialogue round, and fuse and transform the target system text, long-term features and intimacy embedding vector to generate the target system audio of the multi-turn dialogue system in the next dialogue round.

[0044] The next dialogue round refers to the next interaction unit between the user and the multi-turn dialogue system after the current dialogue round ends. It represents the response to be generated in the subsequent interaction and is used to clarify the generation scenario of the target system's audio. For example, after the third round of interaction, the fourth round of interaction is the next dialogue round.

[0045] Among them, the target system text refers to the text response generated by the multi-turn dialogue system for the next dialogue turn. It can be generated by a large language model based on long-term features, current needs and historical context, and is used as the text basis for the audio generation of the target system.

[0046] The fusion and transformation process refers to the integration of target system text, long-term features, and intimacy embedding vectors at the feature level, and then converting the integrated features and text information into speech signals using text-to-speech technology to generate personalized target system audio. Target system audio refers to the final speech signal output by the multi-turn dialogue system in the next dialogue round, used to provide users with voice feedback that aligns with long-term interaction features and intimacy.

[0047] This application's embodiments acquire user text and audio, as well as system text and audio, from multi-turn dialogues. It then fuses bimodal data from both the user and system within a single turn to generate short-term summary features for both the user and system. This fully preserves the semantic and vocal details of single-turn interactions, avoiding the bias caused by missing single-modal data. Based on the single-turn system text, bidirectional semantic fusion of short-term summary features for both the user and system strengthens the semantic association within a single turn, improves semantic consistency, and provides a precise foundation for multi-turn feature interactions. Combining all summary features from multi-turn system text interactions to construct long-term features captures the dynamic changes in semantics and emotion during multi-turn dialogues, solving the problem of text-to-speech difficulty in adapting to context. Finally, the dialogue turns are mapped to intimacy vectors, which are then fused with the target text and long-term features to generate target audio, achieving fine-grained dynamic control and meeting personalized needs.

[0048] In some optional implementations of this embodiment, step 202, for each dialogue turn, involves fusing user text and user audio, and system text and system audio, to obtain user short-term summary features and system short-term summary features for each dialogue turn. This specifically includes the following steps: A text encoder is used to semantically represent the user text and system text in each dialogue round of the multi-turn dialogue system, resulting in user text vectors and system text vectors for each dialogue round. An audio encoder is used to extract acoustic features from the user audio and system audio in each dialogue round, resulting in user speech vectors and system speech vectors for each dialogue round. The user text vectors, user speech vectors, and system text vectors and system speech vectors are then jointly modeled using a multilayer perceptron-based fusion network to obtain user short-term summary features and system short-term summary features for each dialogue round.

[0049] The core function of a text encoder is to transform input text into a low-dimensional, dense vector representation that characterizes the text's semantic information, syntactic structure, and contextual relationships. For example, a text encoder can process the user text "Query this month's spending details" and output a user text vector containing the semantic information of that text.

[0050] Semantic representation refers to the abstract expression of the core meaning, semantic logic, and contextual relationships of text using mathematical vectors. User text vectors are low-dimensional dense vectors obtained by semantically representing user text in each round of dialogue using a text encoder.

[0051] Here, the system text vector refers to the low-dimensional dense vector obtained by semantically representing the system text in each dialogue turn of a multi-turn dialogue system through a text encoder. The audio encoder refers to an acoustic feature extraction model built based on digital signal processing and deep learning techniques. Its core function is to process the raw audio signal and extract key acoustic features.

[0052] Acoustic feature extraction refers to the process of analyzing and transforming raw audio signals (such as user audio and system audio) using an audio encoder to extract the physical characteristics and expressive features of the audio. User speech vectors are low-dimensional dense vectors obtained by extracting acoustic features from user audio in each round of dialogue using an audio encoder. They represent the acoustic features (such as intonation, timbre, and speech rate) and emotional tendencies of the user's speech.

[0053] Among them, the system speech vector refers to the low-dimensional dense vector obtained by extracting acoustic features of the system audio in each dialogue round of the multi-turn dialogue system through an audio encoder, which represents the acoustic features of the system speech (such as speech style, emotional color, and pronunciation rhythm).

[0054] Among them, the fusion network based on multilayer perceptron refers to a multimodal feature fusion model built with multilayer perceptron (MLP) as its core. Its core function is to receive feature vectors from different modalities (such as text vectors and speech vectors), and perform joint modeling through linear transformation and nonlinear activation to explore the potential correlations between modalities. Joint modeling refers to the process of collaboratively learning and integrating feature vectors from multiple modalities or multiple sources through a specific model (such as a fusion network based on multilayer perceptron).

[0055] In one embodiment, a pre-trained Sentence-Bert model can be used as a text encoder to semantically represent the text inputs from the user and the system in each dialogue round, i.e., the user text and the system text, to obtain the text vector representation for each dialogue round, i.e., the user text vector. and system text vectors An audio encoder is used to extract acoustic features from the speech signals (user audio and system audio) in each dialogue round between the user and the system, resulting in a speech vector representation for each round, i.e., the user speech vector. and system speech vectors .Will and The user short-term summary features for each dialogue turn are obtained by jointly modeling using a fusion network based on a multilayer perceptron. and system short summary features This enables nonlinear mapping and interaction between modes.

[0056] This application utilizes a text encoder to semantically represent the user text and system text at each round, transforming them into user and system text vectors containing semantic logic, thus avoiding semantic biases from direct text use. An audio encoder extracts the acoustic features of the user audio and system audio, obtaining user and system speech vectors that carry speech details, compensating for insufficient text information. Then, a fusion network based on a multilayer perceptron is used to jointly model the text and speech vectors on both the user and system sides, mining the inherent correlations between the two modal data and generating short-term user and system summary features that combine semantic and acoustic characteristics, providing a precise single-round foundation for subsequent processing.

[0057] In some optional implementations, step 203, based on the system text of each dialogue round, performs bidirectional semantic fusion processing on the user short-term summary features and system short-term summary features of each dialogue round to obtain the user summary features and system summary features of each dialogue round, specifically including the following steps: For each dialogue round, the system text, user short-term summary features, and system short-term summary features are integrated to obtain a joint feature sequence for each dialogue round. Based on the joint feature sequence, query vectors, key vectors, and value vectors are constructed. Attention is then calculated based on these vectors to obtain the joint features for each dialogue round. Based on the original sequence lengths of the user and system short-term summary features for each dialogue round, the joint features are split into system-side sub-features and user-side features. A first feature enhancement network is used to perform nonlinear feature transformation on the system-side sub-features to obtain the system summary features for each dialogue round. A second feature enhancement network is used to perform nonlinear feature transformation on the user-side sub-features to obtain the user summary features for each dialogue round.

[0058] In this attention calculation, the query vector, key vector, and value vector are constructed based on the joint features of each dialogue turn. The query vector expresses the intent to query information, representing the focus of the current model, and can be used to find the most relevant part in the joint feature sequence. The key vector is used to calculate similarity with the query vector to determine the degree of association between each part and the query vector. The value vector contains the actual information content and is the information carrier that is ultimately extracted and focused on. In the attention calculation, attention weights are calculated by matching the query vector and the key vector, thereby determining the degree of attention paid to the value vector at different positions, and ultimately mining the key information in the joint features. Attention calculation is an operation based on the query vector, key vector, and value vector during the user's interaction with the multi-turn dialogue system.

[0059] The joint feature is the result obtained after attention calculation based on query vector, key vector, and value vector during the user's interaction with the multi-turn dialogue system. It integrates key information from both the system and user sides in each dialogue turn, focusing on and extracting important information from the joint feature sequence, and comprehensively reflecting the interaction status and key connections between the two parties in the current turn.

[0060] The system-side features and user-side features are obtained based on the joint features of each dialogue turn during the user's interaction with the multi-turn dialogue system. Specifically, they are two parts obtained by splitting the joint features according to the original sequence lengths of the user's short-term summary features and the system's short-term summary features for each dialogue turn. The system-side features mainly contain key information about the system in the dialogue turn, such as the semantic emphasis and tone of the system's response; the user-side features mainly contain key information about the user in the dialogue turn, such as the user's questioning intent and emotional inclination.

[0061] Among them, the first feature enhancement network is a network structure specifically designed for nonlinear feature transformation of system-side sub-features, while the second feature enhancement network is a network structure designed for nonlinear feature transformation of user-side sub-features.

[0062] Each system summary feature comprehensively and deeply encompasses key information about the system during the corresponding dialogue round, covering the semantic focus and tone of the system's responses. Each user summary feature contains key information about the user during the corresponding dialogue round, such as the user's questioning intent, emotional inclination, language habits, and long-term interests.

[0063] In one example, taking a fintech intelligent customer service dialogue system, during multiple rounds of user-customer consultations about financial products, the system acquires user text (e.g., "Want to learn about high-yield financial products"), user audio, system text (e.g., "Recommend products with an annualized return of over 5%), and system audio from multiple dialogue rounds. For each round of dialogue, the user text and audio, and the system text and audio, are first fused separately to obtain short-term summary features for both the user and system sides. Next, feature integration is performed. After normalizing the relevant text and features, the system text vector is fused with the normalized system short-term summary features, resulting in a linear transformation feature on the system side; the same applies to the user side. The sequence dimensions of both are concatenated into a joint feature sequence, and query, key, and value vectors are constructed. Attention is then used to calculate the joint features, which are then split into system-side and user-side sub-features. Finally, different feature enhancement networks are used to nonlinearly transform the sub-features to obtain system summary features and user summary features.

[0064] This application integrates relevant features from each dialogue turn to obtain a joint feature sequence. This process combines system text, user, and system short-term summary features to comprehensively capture dialogue information. Constructing query, key, and value vectors and performing attention calculations to obtain joint features allows for precise focus on key information. The joint features are then broken down into system-side and user-side sub-features, which are non-linearly transformed using feature enhancement networks to deeply mine system and user features. The resulting system summary features and user summary features more accurately reflect the characteristics of both parties in the dialogue, providing rich evidence for subsequent personalized voice generation and effectively improving the adaptability and personalization of voice interaction.

[0065] In some optional implementations, the step "to perform feature integration processing on the system text, user short-term summary features, and system short-term summary features for each dialogue round to obtain a joint feature sequence for each dialogue round" specifically includes the following steps: For each dialogue round, the user short-term summary features and system short-term summary features are normalized to obtain normalized user short-term summary features and normalized system short-term summary features. For each dialogue round, the system text vector corresponding to the system text of each dialogue round is fused with the normalized system short-term summary features to obtain fused features. The fused features are scaled and shifted to obtain system-side scaled and shifted features. The system-side scaled and shifted features are then linearly transformed to obtain system-side linearly transformed features. For each dialogue round, the normalized user short-term summary features are scaled and shifted to obtain user-side scaled and shifted features. The user-side scaled and shifted features are then linearly transformed to obtain user-side linearly transformed features. The system-side linearly transformed features and user-side linearly transformed features are concatenated along their sequence dimensions to obtain the joint feature sequence for each dialogue round.

[0066] Among them, the normalized user short-term summary features can characterize the key features of users in short-term conversations, including but not limited to users' language habits, emotional tendencies, and points of interest.

[0067] Among them, the normalized short-term summary features of the system mainly characterize the key features of the system in the short-term dialogue process, such as the semantic focus and tone of the system's response text, as well as the acoustic features of the system's audio, which can reflect the system's state and performance in this round of dialogue.

[0068] Among them, the fused features combine the semantic information carried by the system text with the key features of the system in short-term dialogues, and comprehensively reflect the system's state and key information in each round of dialogue.

[0069] The system-side scaling and offset feature adjustment adjusts the numerical range and position of the feature, enabling it to better adapt to the needs of subsequent processing. Specifically, the scaling operation changes the numerical value of the feature, while the offset operation changes the reference position of the feature. This process effectively adjusts the distribution of the feature in the numerical space.

[0070] Among them, the system-side linear transformation feature maps the system-side scaling and offset features to a more suitable feature space through linear transformation, so that the relationships between features are presented in a way that is easier for the model to understand and process. This feature can effectively highlight the key information and characteristics of the system in the dialogue.

[0071] Among them, the user-side scaling and offset features change the feature value through scaling operations and change the feature reference position through offset operations, thereby adjusting the distribution of user short-term summary features in the numerical space so that the features can better adapt to the needs of subsequent processing.

[0072] Among them, the user-side linear transformation feature maps the user-side scaling and offset feature from the original feature space to a new space that is more conducive to model understanding and processing through linear transformation, which can more prominently represent the user's key information and potential needs in the dialogue.

[0073] Sequence dimension splicing refers to the operation of merging the system-side linear transformation features and the user-side linear transformation features in the sequence dimension for each dialogue turn during the interaction between the user and the multi-turn dialogue system.

[0074] The joint feature sequence is obtained by concatenating the system-side linear transformation features and the user-side linear transformation features along the sequence dimension for each dialogue round during the user's interaction with the multi-turn dialogue system.

[0075] In one example, within an intelligent customer service scenario in the financial insurance sector, users interact with a multi-turn dialogue system to inquire about insurance products. The system acquires user text (e.g., "I want to know the coverage of critical illness insurance"), user audio, system text (e.g., "This critical illness insurance covers a variety of serious illnesses, specifically including...") and system audio from multiple dialogue rounds. For each dialogue round, the user's short-term summary features and the system's short-term summary features are first normalized to eliminate the influence of dimensions and make the data more comparable. For example, audio features of different lengths and numerical ranges are unified into a specific interval. Next, the system text vector is fused with the normalized system short-term summary features. The fused features combine text semantics and audio feature information. After scaling and offsetting the fused features, a linear transformation is performed to obtain the system-side linear transformation features, which enhances the expression of system-related features. Similarly, the normalized user short-term summary features are scaled, offset, and linearly transformed to obtain the user-side linear transformation features, highlighting user characteristics. Finally, the dimensions of the system-side and user-side linear transformation feature sequences are concatenated to obtain a joint feature sequence.

[0076] This application's embodiments normalize the short-term summary features of users and the system for each dialogue turn, eliminating differences in feature dimensions and ensuring the data is on a uniform scale, facilitating subsequent processing and fusion. Fusing the system text vector with the normalized system short-term summary features integrates text semantics and audio features, enriching information expression. Scaling, offsetting, and linearly transforming the fused features enhance system-side features. Similarly, after processing user-side features, the dimensions of the system-side and user-side linearly transformed feature sequences are concatenated into a joint feature sequence. This series of operations fully integrates multi-dimensional information, accurately capturing the interaction details between users and the system in the dialogue, providing a solid data foundation for generating more context-appropriate and personalized speech.

[0077] In some optional implementations, step 204, based on the system text from multiple dialogue rounds, involves feature interaction between all user summary features and system summary features to obtain long-term features, specifically including the following steps: The user summary features and system summary features of each dialogue round are concatenated along the sequence dimension to obtain the concatenated features of each dialogue round. The concatenated features of each dialogue round are stacked along the round dimension to obtain a multi-round joint feature sequence. The system text features corresponding to the system text of each dialogue round are fused with the concatenated features of the corresponding dialogue round at the element level to obtain the fused multi-round fused features. The multi-round fused features are then enhanced with features and interaction modeling to obtain interaction features. The text features corresponding to the system text of all dialogue rounds are input into a gating adjustment unit to obtain a gating weight vector. The interaction features and the gating weight vector are then weighted and fused to obtain the gated post-encoded features. Finally, the gated post-encoded features are deeply encoded and global feature extracted using a multi-round feature deep encoding unit to obtain long-term features.

[0078] Among them, concatenation along the sequence dimension represents the integration of features from different sources but belonging to the same dialogue turn, thereby obtaining a more comprehensive and representative feature representation. For example, suppose there are two feature vectors A=[1,2,3] and B=[4,5,6], the new vector obtained after concatenation along the sequence dimension is [1,2,3,4,5,6], that is, the two features are arranged and combined sequentially along the sequence dimension.

[0079] The splicing features of each dialogue round represent the fusion result of the features of both the user and the system in the same dialogue round, comprehensively reflecting the overall characteristics of the dialogue in that round.

[0080] Stacking by round dimension refers to the process in a multi-turn dialogue system where the concatenated features obtained from each dialogue round are sequentially superimposed along the direction of the dialogue round. The aim is to integrate the concatenated features from multiple rounds into a coherent feature sequence, used to characterize the relationships and trends between rounds throughout the multi-turn dialogue process.

[0081] Among them, the multi-turn joint feature sequence represents the comprehensive features of the interaction in each turn of the entire multi-turn dialogue process. It includes the fusion features of the user and system interaction information in each turn of the multi-turn dialogue, reflecting the relationship and change trend of the dialogue across multiple turns. Element-level fusion refers to the operation of fusing different features with the same dimension according to the element correspondence relationship in the multi-turn dialogue system processing flow, such as element-level addition and element-level multiplication.

[0082] Among them, multi-turn fusion features represent the deep integration of system text information and comprehensive information from user and system interactions within the same dialogue turn. Feature enhancement and interaction modeling aim to improve feature quality and uncover the interaction relationships between features. By optimizing and modeling multi-turn fusion features, the dependencies and semantic information between features can be better captured.

[0083] Interaction features characterize the complex interactive relationships and deep semantic information between features in multi-turn dialogues after enhancement processing, reflecting the core content and changing trends of user intent in multi-turn dialogues. The gating unit is a module in a multi-turn dialogue system used to process text features and generate gating weight vectors. It takes the text features corresponding to all rounds of system text as input, performs internal calculations, and outputs a gating weight vector.

[0084] The gating weight vector is the result obtained after inputting the text features corresponding to the system text in all rounds into the gating adjustment unit. It aims to represent the importance of each round's system text features within the overall features using a single vector. The gating-encoded features represent the weighted encoded features of the interaction features, taking into account the importance of the system text features (represented by the gating weight vector), comprehensively reflecting the core information in multi-turn dialogues and the key influence of system text features.

[0085] Among them, the multi-turn feature deep encoding unit is a neural network module used in multi-turn dialogue systems to perform deep processing and global feature extraction on gated post-encoded features. Deep encoding and global feature extraction represent the process of gradually extracting comprehensive, deep, and long-term memory feature representations from gated post-encoded features. In this way, the core content and changing trends of user needs in multi-turn dialogues can be grasped more accurately.

[0086] In one example, this technical solution is applied in a multi-turn dialogue system for intelligent medical consultation. The system collects multi-turn data of patient-system interaction, including user text describing symptoms, user audio of speaking tone, and diagnostic suggestions and corresponding audio from the system. For each round of dialogue, the user's and system's text and audio are first fused to obtain short-term summary features. Then, based on the system text, the two are semantically fused bidirectionally to obtain user and system summary features. The summary features of each round are concatenated sequentially and stacked round by round to form a multi-round joint feature sequence, reflecting the overall consultation. Next, the system text features are fused with the concatenated features at the element level, and interaction features are obtained through feature enhancement and interaction modeling. All rounds of system text features are input into a gating adjustment unit to obtain a weight vector, which is then weighted and fused with the interaction features. Finally, a multi-round feature deep encoding unit is used to extract long-term features such as the patient's long-term health trends.

[0087] This application embodiment integrates key information from both parties in each dialogue round by concatenating the user summary features and system summary features of each dialogue round along the sequence dimension, forming a more complete local feature set. Stacking and concatenating features along the round dimension yields a multi-round joint feature sequence, preserving the historical order and connections of the dialogue. Element-level fusion of system text features and concatenated features deeply integrates text semantics with dialogue features. Interaction features are obtained through feature enhancement and interaction modeling, uncovering complex relationships between features. The weight vector output by the gating adjustment unit is weighted and fused with the interaction features, and then deep-encoded to extract long-term features. This accurately captures long-term user preferences and overall dialogue trends, providing strong support for generating high-quality, dynamically changing, and user-centric personalized voice.

[0088] In some alternative implementations, the step "enhancing and modeling the interaction of the multi-round fused features to obtain interactive features" specifically includes the following steps: The multi-round fusion features are normalized to obtain normalized multi-round fusion features. The normalized multi-round fusion features are then scaled and shifted to obtain scaled and shifted features. The scaled and shifted features are then transformed by a multilayer perceptron to obtain local nonlinear features. Attention models are then applied to the scaled and shifted features to obtain global dependency features. Finally, the local nonlinear features and global dependency features are fused at the element level to obtain interactive features.

[0089] The multilayer perceptron transformation is an operation performed on scaled and offset features during the feature enhancement and interactive modeling steps, implemented using a multilayer perceptron. In this process, the scaled and offset features are first input into the multilayer perceptron, which consists of multiple neurons, including an input layer, hidden layers, and an output layer. The input layer receives the scaled and offset features, while the hidden layers perform nonlinear transformations on the input data. Each hidden layer neuron performs a weighted summation of the input data using weights and biases, followed by nonlinear processing through an activation function. Different hidden layers can learn features at different levels of abstraction, uncovering complex local relationships and nonlinear patterns in the original features. The output layer outputs local nonlinear features, which are ultimately the output of the last layer of the multilayer perceptron.

[0090] Among them, local nonlinear features aim to effectively represent the complex feature relationships between local data in the original features that cannot be described by simple linear relationships through the nonlinear transformation operation of the multilayer perceptron.

[0091] Attention modeling is an operation performed on scaled and offset features during feature processing. Its core principle is to assign different weights to each feature by calculating the similarity or correlation between feature vectors, thereby representing the dependencies between features and focusing on the feature parts most important to the current task.

[0092] The global dependency feature is the result obtained when performing attention modeling on the scaled and offset features. It characterizes the global dependency between features by calculating the degree of correlation between feature vectors and assigning weights to each feature.

[0093] In one example, this technical solution is applied in the scenario of intelligent medical consultation. The patient has multiple rounds of conversations with the system, and the system obtains the user text (such as symptom description), user audio (the tone reflects the degree of anxiety), system text (diagnostic suggestions), and system audio in each round. The multi-round fusion features are normalized to eliminate the influence of dimension, and then scaled and offset to enhance the feature expression ability. The scaled and offset features are subjected to a multi-layer perceptron transformation to挖掘 local complex non-linear relationships, obtaining local non-linear features to accurately capture the details of the patient's symptoms. Through attention modeling, global dependent features are obtained to grasp the overall consultation trend. The two are element-wise fused into interaction features to comprehensively integrate information.

[0094] In the embodiment of this application, by normalizing the multi-round fusion features, the dimension differences between different features are eliminated, making the data at a similar scale, providing a stable basis for subsequent processing. The scaled and offset features obtained after the scaling and offset operations enhance the feature expression ability and discrimination. Performing a multi-layer perceptron transformation on it can挖掘 the complex local non-linear relationships in the features and capture the subtle changes in the conversation. Attention modeling can obtain global dependent features and grasp the overall context of the conversation. Element-wise fusing the local non-linear features and global dependent features into interaction features can comprehensively integrate local details and global information, enabling the system to more accurately perceive the conversation situation and providing strong support for generating speech that fits the context, emotions, and changes in intimacy.

[0095] In some optional implementation manners, in step 206, the target system text, long-term features, and intimacy embedding vectors are subjected to a fusion transformation process to generate the target system audio of the multi-round dialogue system in the next dialogue round, which specifically includes the following steps: The target system text is input into a phoneme sequence generation model to obtain a phoneme sequence, and a phoneme hidden feature extraction model is used to extract phoneme hidden features from the phoneme sequence; the long-term features, intimacy embedding vectors, and phoneme hidden features are concatenated to obtain fusion features; through an acoustic decoder, the fusion features are subjected to an acoustic feature transformation process to generate a target Mel spectrogram; through a vocoder, the target Mel spectrogram is converted into waveform speech to obtain the target system audio of the multi-round dialogue system in the next dialogue round.

[0096] Among them, the phoneme sequence generation model is based on speech synthesis technology, processes and analyzes the input text, audio data, and word segmentation information, constructs a network structure, and uses a specific algorithm to search for paths to generate a phoneme sequence. The phoneme sequence is the phoneme sequence obtained by converting the input target system text through the phoneme sequence generation model. For example, "你好" will be converted into a phoneme sequence similar to: nǐ / hǎo. The phoneme sequence is an intermediate representation form of speech synthesis.

[0097] The phoneme latent feature extraction model, built on a deep learning architecture, extracts hidden features from phoneme sequences that are crucial for speech synthesis but difficult to obtain using conventional methods. These features contain latent information such as prosody and emotion. Phoneme latent features are features extracted from phoneme sequences using a specific model and are hidden within the phoneme sequence; they are not directly observable. They contain rich information related to speech characteristics, such as prosody, pronunciation habits, and emotion.

[0098] Among them, fusion features are comprehensive features obtained by combining different features such as long-term features, proximity embedding vectors, and phoneme latent features through methods such as direct splicing in fields such as multimodal information processing and speech synthesis. These features reflect the collaborative relationships between different modalities.

[0099] The acoustic decoder is a key component of the speech synthesis system, responsible for processing and converting fused features into acoustic features. Acoustic feature conversion is the process in speech synthesis where the acoustic decoder processes fused features (composed of long-term features, proximity embedding vectors, and latent phoneme features). Through operations based on signal processing theory and machine learning algorithms, information such as multi-turn dialogue history, proximity, and latent phoneme speech characteristics from the fused features is transformed into the acoustic features required for speech generation.

[0100] The target Mel spectrum, in speech synthesis, is the result of acoustic feature transformation processing of the fused features by the acoustic decoder. It represents the frequency characteristics of the speech signal. Based on the Mel frequency scale that simulates human ear perception, it maps linear frequencies to Mel frequencies, more accurately reflecting the frequency and energy distribution of speech. The vocoder is a key tool in the speech synthesis system. Based on signal reconstruction technology, it transforms the target Mel spectrum output by the acoustic decoder into an audible speech waveform.

[0101] In one example, within an intelligent customer service scenario in the insurance field, a user interacts with a multi-turn dialogue system multiple times to inquire about car insurance-related issues. The system first acquires the user's text, audio, system text, and system audio from multiple dialogue rounds generated by the user's interaction with the system. The target system text, "Your car insurance premium is calculated based on factors such as vehicle model and age; there are currently promotional offers," is input into a phoneme sequence generation model to obtain the corresponding phoneme sequence. Then, a phoneme latent feature extraction model is used to extract latent phoneme features reflecting pronunciation stress and intonation from the phoneme sequence. These latent features, combined with long-term features obtained from long-term interactions and the proximity embedding vector mapped from the number of dialogue rounds, are concatenated with the latent phoneme features to form a fused feature. An acoustic decoder performs acoustic feature transformation on the fused feature to generate the target Mel spectrum. Finally, a vocoder converts the target Mel spectrum into waveform speech.

[0102] This application embodiment obtains the phoneme sequence by generating the target system text input phoneme sequence and extracts latent features using a phoneme latent feature extraction model, which can accurately capture details at the text speech level. By combining long-term features, proximity embedding vectors, and phoneme latent features to form fused features, it fully considers users' long-term interaction habits and proximity. The target Mel spectrum is generated through acoustic feature conversion processing by an acoustic decoder, and then converted into waveform speech by a vocoder. This allows for dynamic and continuous changes in speech rate, tone, and emotional tone according to different situations, making the generated target system audio more tailored to user needs and providing users with a high-quality, personalized voice interaction experience.

[0103] It should be emphasized that, to further ensure the privacy and security of the aforementioned user text, user audio, system text, system audio, user short-term summary features, system short-term summary features, user summary features, system summary features, long-term features, intimacy embedding vector, target system text, and target system audio, these user text, user audio, system text, system audio, user short-term summary features, system short-term summary features, user summary features, system summary features, long-term features, intimacy embedding vector, target system text, and target system audio can also be stored in a blockchain node.

[0104] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0105] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0106] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0107] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a system audio generation device based on multi-turn dialogue, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0108] like Figure 3 As shown, the system audio generation device 400 based on multi-turn dialogue in this embodiment includes: an acquisition module 401, a fusion module 402, a semantic fusion module 403, an interaction module 404, a mapping module 405, and a fusion conversion module 406. Wherein: The acquisition module 401 is used to acquire user text, user audio, system text and system audio generated from multiple dialogue rounds generated by the user's interaction with the multi-turn dialogue system. The fusion module 402 is used to fuse user text and user audio, and system text and system audio for each dialogue round, to obtain user short-term summary features and system short-term summary features for each dialogue round. The semantic fusion module 403 is used to perform bidirectional semantic fusion processing on the user short-term summary features and system short-term summary features of each dialogue round based on the system text of each dialogue round, so as to obtain the user summary features and system summary features of each dialogue round. Interaction module 404 is used to perform feature interaction between all user summary features and system summary features based on the system text from multiple dialogue rounds to obtain long-term features; The mapping module 405 is used to obtain the total number of dialogue rounds between the user and the multi-turn dialogue system, and map the number of dialogue rounds into an affinity embedding vector. The fusion and transformation module 406 is used to obtain the target system text of the multi-turn dialogue system in the next dialogue round, and to perform fusion and transformation processing on the target system text, long-term features and intimacy embedding vector to generate the target system audio of the multi-turn dialogue system in the next dialogue round.

[0109] This application's embodiments acquire user text and audio, as well as system text and audio, from multi-turn dialogues. It then fuses bimodal data from both the user and system within a single turn to generate short-term summary features for both the user and system. This fully preserves the semantic and vocal details of single-turn interactions, avoiding the bias caused by missing single-modal data. Based on the single-turn system text, bidirectional semantic fusion of short-term summary features for both the user and system strengthens the semantic association within a single turn, improves semantic consistency, and provides a precise foundation for multi-turn feature interactions. Combining all summary features from multi-turn system text interactions to construct long-term features captures the dynamic changes in semantics and emotion during multi-turn dialogues, solving the problem of text-to-speech difficulty in adapting to context. Finally, the dialogue turns are mapped to intimacy vectors, which are then fused with the target text and long-term features to generate target audio, achieving fine-grained dynamic control and meeting personalized needs.

[0110] In one embodiment, the fusion module 402 includes: The semantic representation submodule is used to perform semantic representation on the user text and system text in each dialogue round of the multi-turn dialogue system using a text encoder, so as to obtain the user text vector and system text vector for each dialogue round; The feature extraction submodule is used to extract acoustic features from the user audio and system audio in each dialogue round of the multi-turn dialogue system using an audio encoder, so as to obtain the user speech vector and system speech vector for each dialogue round. The modeling submodule is used to jointly model user text vectors and user speech vectors, as well as system text vectors and system speech vectors, through a fusion network based on a multilayer perceptron, to obtain user short-term summary features and system short-term summary features for each dialogue turn.

[0111] This application utilizes a text encoder to semantically represent the user text and system text at each round, transforming them into user and system text vectors containing semantic logic, thus avoiding semantic biases from direct text use. An audio encoder extracts the acoustic features of the user audio and system audio, obtaining user and system speech vectors that carry speech details, compensating for insufficient text information. Then, a fusion network based on a multilayer perceptron is used to jointly model the text and speech vectors on both the user and system sides, mining the inherent correlations between the two modal data and generating short-term user and system summary features that combine semantic and acoustic characteristics, providing a precise single-round foundation for subsequent processing.

[0112] In one embodiment, the semantic fusion module 403 includes: The integration submodule is used to perform feature integration processing on the system text, user short-term summary features and system short-term summary features of each dialogue round to obtain the joint feature sequence of each dialogue round. The construction submodule is used to build query vectors, key vectors, and value vectors based on the joint feature sequence, and to perform attention calculation based on the query vectors, key vectors, and value vectors to obtain the joint features for each dialogue round; The splitting submodule is used to split the joint features into system-side sub-features and user-side features based on the original sequence length of the user short-term summary features and system short-term summary features for each dialogue round. The first transformation submodule is used to perform nonlinear feature transformation on the system side features through the first feature enhancement network to obtain the system summary features for each dialogue round; The second submodule is used to perform nonlinear feature transformation on the user-side sub-features through the second feature enhancement network to obtain the user summary features for each dialogue round.

[0113] This application integrates relevant features from each dialogue turn to obtain a joint feature sequence. This process combines system text, user, and system short-term summary features to comprehensively capture dialogue information. Constructing query, key, and value vectors and performing attention calculations to obtain joint features allows for precise focus on key information. The joint features are then broken down into system-side and user-side sub-features, which are non-linearly transformed using feature enhancement networks to deeply mine system and user features. The resulting system summary features and user summary features more accurately reflect the characteristics of both parties in the dialogue, providing rich evidence for subsequent personalized voice generation and effectively improving the adaptability and personalization of voice interaction.

[0114] In one embodiment, the integration submodule is further configured to normalize the user short-term summary features and system short-term summary features for each dialogue round, respectively, to obtain normalized user short-term summary features and normalized system short-term summary features; for each dialogue round, the system text vector corresponding to the system text of each dialogue round is fused with the normalized system short-term summary features to obtain fused features; the fused features are scaled and shifted to obtain system-side scaled and shifted features, and then linearly transformed to obtain system-side linearly transformed features; for each dialogue round, the normalized user short-term summary features are scaled and shifted to obtain user-side scaled and shifted features, and then linearly transformed to obtain user-side linearly transformed features; the system-side linearly transformed features and user-side linearly transformed features are concatenated along their sequence dimensions to obtain a joint feature sequence for each dialogue round.

[0115] This application's embodiments normalize the short-term summary features of users and the system for each dialogue turn, eliminating differences in feature dimensions and ensuring the data is on a uniform scale, facilitating subsequent processing and fusion. Fusing the system text vector with the normalized system short-term summary features integrates text semantics and audio features, enriching information expression. Scaling, offsetting, and linearly transforming the fused features enhance system-side features. Similarly, after processing user-side features, the dimensions of the system-side and user-side linearly transformed feature sequences are concatenated into a joint feature sequence. This series of operations fully integrates multi-dimensional information, accurately capturing the interaction details between users and the system in the dialogue, providing a solid data foundation for generating more context-appropriate and personalized speech.

[0116] In one embodiment, the interaction module 404 includes: The decomposition submodule is used to concatenate the user summary features and system summary features of each dialogue round along the sequence dimension to obtain the concatenated features of each dialogue round. The stacking submodule is used to stack the concatenated features of each dialogue round according to the round dimension to obtain a multi-round joint feature sequence; The fusion submodule is used to perform element-level fusion of the system text features corresponding to the system text of each dialogue round with the concatenation features of the corresponding dialogue round to obtain the fused multi-round fusion features. The interaction submodule is used to enhance and model the interaction of multi-round fused features to obtain interactive features. The input submodule is used to input the text features corresponding to the system text of all dialogue rounds into the gating adjustment unit to obtain the gating weight vector; The weighted submodule is used to weight and fuse the interaction features and the gating weight vector to obtain the gating post-encoded features; The feature extraction submodule is used to perform deep encoding and global feature extraction on the gated post-encoded features through multiple rounds of feature deep encoding units to obtain long-term features.

[0117] This application embodiment integrates key information from both parties in each dialogue round by concatenating the user summary features and system summary features of each dialogue round along the sequence dimension, forming a more complete local feature set. Stacking and concatenating features along the round dimension yields a multi-round joint feature sequence, preserving the historical order and connections of the dialogue. Element-level fusion of system text features and concatenated features deeply integrates text semantics with dialogue features. Interaction features are obtained through feature enhancement and interaction modeling, uncovering complex relationships between features. The weight vector output by the gating adjustment unit is weighted and fused with the interaction features, and then deep-encoded to extract long-term features. This accurately captures long-term user preferences and overall dialogue trends, providing strong support for generating high-quality, dynamically changing, and user-centric personalized voice.

[0118] In one embodiment, the interaction submodule is further configured to normalize the multi-round fusion features to obtain normalized multi-round fusion features, scale and offset the normalized multi-round fusion features to obtain scaled and offset features, perform multilayer perceptron transformation on the scaled and offset features to obtain local nonlinear features, perform attention modeling on the scaled and offset features to obtain global dependency features, and fuse the local nonlinear features and global dependency features at the element level to obtain interaction features.

[0119] This application's embodiments normalize the multi-turn fusion features, eliminating dimensional differences between different features and bringing the data to a similar scale, providing a stable foundation for subsequent processing. The scaled and offset features obtained after scaling and offset operations enhance their expressive power and discriminative ability. Applying a multilayer perceptron transform to these features can uncover complex local nonlinear relationships and capture subtle changes in the dialogue. Attention modeling can obtain globally dependent features, grasping the overall context of the dialogue. Element-level fusion of local nonlinear features and globally dependent features into interactive features comprehensively integrates local details and global information, enabling the system to more accurately perceive the dialogue situation and providing strong support for generating speech that fits context, emotion, and changes in intimacy.

[0120] In one embodiment, the fusion conversion module 406 includes: The phoneme extraction submodule is used to input the target system text into the phoneme sequence generation model to obtain the phoneme sequence, and then use the phoneme latent feature extraction model to extract the phoneme latent features from the phoneme sequence. The splicing submodule is used to splice long-term features, proximity embedding vectors, and phoneme latent features to obtain fused features; The processing submodule is used to perform acoustic feature transformation processing on the fused features through an acoustic decoder to generate the target Mel spectrum; The conversion submodule is used to convert the target Mel spectrum into waveform speech using a vocoder, so as to obtain the target system audio for the next dialogue round of the multi-turn dialogue system.

[0121] This application embodiment obtains the phoneme sequence by generating the target system text input phoneme sequence and extracts latent features using a phoneme latent feature extraction model, which can accurately capture details at the text speech level. By combining long-term features, proximity embedding vectors, and phoneme latent features to form fused features, it fully considers users' long-term interaction habits and proximity. The target Mel spectrum is generated through acoustic feature conversion processing by an acoustic decoder, and then converted into waveform speech by a vocoder. This allows for dynamic and continuous changes in speech rate, tone, and emotional tone according to different situations, making the generated target system audio more tailored to user needs and providing users with a high-quality, personalized voice interaction experience.

[0122] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0123] Computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only computer device 6 with memory 61, processor 62, and network interface 63 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0124] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0125] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 61 may also include both the internal storage unit and the external storage device of the computer device 6. In this embodiment, the memory 61 is typically used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions for a system audio generation method based on multi-turn dialogue. In addition, memory 61 can also be used to temporarily store various types of data that have been output or will be output.

[0126] In some embodiments, processor 62 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 62 is typically used to control the overall operation of computer device 6. In this embodiment, processor 62 is used to execute computer-readable instructions stored in memory 61 or to process data, such as executing computer-readable instructions for a system audio generation method based on multi-turn dialogue.

[0127] The network interface 63 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 6 and other electronic devices.

[0128] This application's embodiments acquire user text and audio, as well as system text and audio, from multi-turn dialogues. It then fuses bimodal data from both the user and system within a single turn to generate short-term summary features for both the user and system. This fully preserves the semantic and vocal details of single-turn interactions, avoiding the bias caused by missing single-modal data. Based on the single-turn system text, bidirectional semantic fusion of short-term summary features for both the user and system strengthens the semantic association within a single turn, improves semantic consistency, and provides a precise foundation for multi-turn feature interactions. Combining all summary features from multi-turn system text interactions to construct long-term features captures the dynamic changes in semantics and emotion during multi-turn dialogues, solving the problem of text-to-speech difficulty in adapting to context. Finally, the dialogue turns are mapped to intimacy vectors, which are then fused with the target text and long-term features to generate target audio, achieving fine-grained dynamic control and meeting personalized needs.

[0129] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the system audio generation method based on multi-turn dialogue as described above.

[0130] This application's embodiments acquire user text and audio, as well as system text and audio, from multi-turn dialogues. It then fuses bimodal data from both the user and system within a single turn to generate short-term summary features for both the user and system. This fully preserves the semantic and vocal details of single-turn interactions, avoiding the bias caused by missing single-modal data. Based on the single-turn system text, bidirectional semantic fusion of short-term summary features for both the user and system strengthens the semantic association within a single turn, improves semantic consistency, and provides a precise foundation for multi-turn feature interactions. Combining all summary features from multi-turn system text interactions to construct long-term features captures the dynamic changes in semantics and emotion during multi-turn dialogues, solving the problem of text-to-speech difficulty in adapting to context. Finally, the dialogue turns are mapped to intimacy vectors, which are then fused with the target text and long-term features to generate target audio, achieving fine-grained dynamic control and meeting personalized needs.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0132] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

[0133] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A system audio generation method based on multi-turn dialogue, characterized in that, Includes the following steps: Acquire user text, user audio, system text, and system audio from multiple dialogue rounds generated by the user's interaction with the multi-turn dialogue system; For each dialogue round, the user text and user audio, and the system text and system audio are fused to obtain the user short-term summary features and system short-term summary features for each dialogue round. Based on the system text of each dialogue round, bidirectional semantic fusion processing is performed on the user short-term summary features and system short-term summary features of each dialogue round to obtain the user summary features and system summary features of each dialogue round. Based on the system text from the multiple dialogue rounds, feature interaction is performed on all the user summary features and the system summary features to obtain long-term features; Obtain the total number of dialogue rounds between the user and the multi-turn dialogue system, and map the number of dialogue rounds into an intimacy embedding vector; The target system text of the multi-turn dialogue system in the next dialogue round is obtained, and the target system text, the long-term features and the intimacy embedding vector are fused and transformed to generate the target system audio of the multi-turn dialogue system in the next dialogue round.

2. The method according to claim 1, characterized in that, The step of fusing the user text and user audio, and the system text and system audio for each dialogue round to obtain the user short-term summary features and system short-term summary features for each dialogue round specifically includes: A text encoder is used to semantically represent the user text and system text in each dialogue round between the user and the multi-turn dialogue system, so as to obtain the user text vector and system text vector for each dialogue round. An audio encoder is used to extract acoustic features from the user audio and system audio in each dialogue round between the user and the multi-turn dialogue system, so as to obtain the user speech vector and system speech vector for each dialogue round. The user text vector and the user speech vector, as well as the system text vector and the system speech vector, are jointly modeled through a multilayer perceptron-based fusion network to obtain the user short-term summary features and the system short-term summary features for each dialogue turn.

3. The method according to claim 1, characterized in that, The step of performing bidirectional semantic fusion processing on the user short-term summary features and system short-term summary features of each dialogue round based on the system text of each dialogue round to obtain the user summary features and system summary features of each dialogue round specifically includes: The system text, user short-term summary features, and system short-term summary features of each dialogue round are integrated to obtain a joint feature sequence for each dialogue round. Based on the joint feature sequence, a query vector, a key vector, and a value vector are constructed. Based on the query vector, key vector, and value vector, attention calculation is performed to obtain the joint features for each dialogue round. Based on the original sequence length of the user short-term summary features and system short-term summary features for each dialogue round, the joint features are split into system-side sub-features and user-side features; The system side features are transformed nonlinearly using a first feature enhancement network to obtain the system summary features for each dialogue round. The user-side sub-features are transformed nonlinearly using a second feature enhancement network to obtain the user summary features for each dialogue round.

4. The method according to claim 3, characterized in that, The step of performing feature integration processing on the system text, user short-term summary features, and system short-term summary features of each dialogue round to obtain the joint feature sequence of each dialogue round specifically includes: The user short-term summary features and system short-term summary features for each dialogue round are normalized to obtain normalized user short-term summary features and normalized system short-term summary features. For each dialogue round, the system text vector corresponding to the system text of each dialogue round is fused with the normalized system short-term summary features to obtain the fused features; The fused features are scaled and offset to obtain system-side scaled and offset features. The system-side scaled and offset features are then linearly transformed to obtain system-side linearly transformed features. For each dialogue round, the normalized user short-term summary features are scaled and offset to obtain user-side scaled-off features. The user-side scaled-off features are then linearly transformed to obtain user-side linearly transformed features. The system-side linear transformation features and the user-side linear transformation features are concatenated along their sequence dimensions to obtain the joint feature sequence for each dialogue round.

5. The method according to claim 1, characterized in that, The step of obtaining long-term features by performing feature interaction between all user summary features and system summary features based on system text from multiple dialogue rounds specifically includes: The user summary features and system summary features of each dialogue round are concatenated along the sequence dimension to obtain the concatenated features of each dialogue round. The concatenated features of each dialogue round are stacked according to the round dimension to obtain a multi-round joint feature sequence; The system text features corresponding to the system text of each dialogue round are fused with the concatenation features of the corresponding dialogue round at the element level to obtain the fused multi-round fused features. The multi-round fusion features are enhanced and interactively modeled to obtain interactive features; Input the text features corresponding to the system text of all dialogue rounds into the gating adjustment unit to obtain the gating weight vector; The interaction features and the gating weight vector are weighted and fused to obtain the gating post-encoded features; Long-term features are obtained by performing deep encoding and global feature extraction on the gated post-encoded features through multiple rounds of feature deep encoding units.

6. The method according to claim 5, characterized in that, The step of performing feature enhancement and interaction modeling on the multi-round fusion features to obtain interaction features specifically includes: The multi-round fusion features are normalized to obtain normalized multi-round fusion features. The normalized multi-round fusion features are then scaled and offset to obtain scaled and offset features. The scaled and offset features are subjected to a multilayer perceptron transformation to obtain local nonlinear features; Attention modeling is performed on the scaled and offset features to obtain globally dependent features; The local nonlinear features and the global dependency features are fused at the element level to obtain the interactive features.

7. The method according to claim 1, characterized in that, The step of fusing and transforming the target system text, the long-term features, and the intimacy embedding vector to generate the target system audio of the multi-turn dialogue system in the next dialogue round specifically includes: The target system text is input into the phoneme sequence generation model to obtain a phoneme sequence. Then, a phoneme latent feature extraction model is used to extract phoneme latent features from the phoneme sequence. The long-term features, the intimacy embedding vector, and the phoneme latent features are concatenated to obtain the fused features; The fused features are subjected to acoustic feature transformation processing using an acoustic decoder to generate the target Mel spectrum; The target Mel spectrum is converted into waveform speech using a vocoder to obtain the target system audio of the multi-turn dialogue system in the next dialogue round.

8. A system audio generation device based on multi-turn dialogue, characterized in that, include: The acquisition module is used to acquire user text, user audio, system text, and system audio generated from multiple dialogue rounds generated by the user's interaction with the multi-turn dialogue system. The fusion module is used to fuse the user text and the user audio, and the system text and the system audio for each dialogue round, respectively, to obtain the user short-term summary features and the system short-term summary features for each dialogue round; The semantic fusion module is used to perform bidirectional semantic fusion processing on the user short-term summary features and system short-term summary features of each dialogue round based on the system text of each dialogue round, so as to obtain the user summary features and system summary features of each dialogue round. The interaction module is used to perform feature interaction between all the user summary features and the system summary features based on the system text of the multiple dialogue rounds to obtain long-term features; The mapping module is used to obtain the total number of dialogue rounds between the user and the multi-turn dialogue system, and map the number of dialogue rounds into an affinity embedding vector. The fusion and transformation module is used to obtain the target system text of the multi-turn dialogue system in the next dialogue round, and to perform fusion and transformation processing on the target system text, the long-term features and the intimacy embedding vector to generate the target system audio of the multi-turn dialogue system in the next dialogue round.

9. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the system audio generation method based on multi-turn dialogue as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions that, when executed by a processor, implement the steps of the system audio generation method based on multi-turn dialogue as described in any one of claims 1 to 7.

Images