A dynamic multi-modal expression parameter decision method and system based on user portrait and dialogue context

Abstract

This application discloses a dynamic multimodal expression parameter decision-making method and system based on user profiles and dialogue context, belonging to the field of intelligent voice interaction technology. This method acquires user psychological profile features, multi-turn dialogue context, and response text, and extracts emotional semantic features; it then fuses multi-source features through a hierarchical hyperbolic embedding network to generate a unified hyperbolic embedding representation, and extracts long-range dependency features through a selective state-space network; the data is input into a dynamic routing hybrid expert decision-making network, which performs three types of decisions in parallel: timbre matching, speech rate control, and emotional expression, and aggregates and outputs structured multimodal expression parameters. This invention can characterize the hierarchical association of user psychological features, achieve collaborative decision-making of timbre, speech rate, and emotion, and provide fine-grained continuous control parameters.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction, and in particular to a dynamic multimodal expression parameter decision-making method and system based on user profiles and dialogue context. Background Technology

[0002] In AI-powered mental health support scenarios, digital human counselors leverage intelligent voice interaction systems to generate speech parameters adapted to the dialogue context, achieving human-like communication. Current mainstream multimodal speech parameter generation solutions generally employ Euclidean space vector splicing to construct user profiles, independent branch models to predict various speech parameters separately, and discrete classification methods to output speech rate and emotional control levels.

[0003] The existing technical solutions have three inherent limitations: First, using Euclidean space vector splicing to complete user profile modeling can only flatly integrate user psychological characteristics, communication preferences and dialogue history information, making it difficult to depict the hierarchical tree-like relationship structure between the three types of information. In long-term, multi-round psychological counseling dialogues, as the number of dialogue rounds accumulates, the feature coupling relationship is continuously lost, leading to a gradual decline in the ability to represent user state.

[0004] Secondly, the existing architecture sets up independent prediction tasks for timbre selection, speech rate control, and emotional expression. Each branch model is trained separately and the inference process is not constrained by each other, lacking a joint optimization mechanism for multi-dimensional parameters. Conflicts are likely to occur between the output speech parameters, resulting in a disconnect between the rhythm and emotional tone of the digital human's speech expression and insufficient coherence.

[0005] Third, existing solutions use a discrete classification output mode for speech rate and emotion-related parameters, which can only output a limited range of control values ​​and cannot generate continuous, fine-grained adjustment parameters. However, psychological counseling scenarios require a high level of empathic ability in speech, necessitating smooth and gentle dynamic adjustment of speech according to the user's emotions, a requirement that the discrete output mode cannot meet.

[0006] Although some studies in the industry have attempted to alleviate the above problems by increasing the number of model parameters and expanding the dialogue sample set, they have not solved the core problems of missing profile-level representation, multimodal parameter decoupling prediction, and discrete parameter output at the underlying modeling architecture level. They still cannot support adaptive voice expression decision-making in long-term psychological counseling dialogues. In practice, there are common problems such as stiff expression of digital humans, insufficient empathy, and mismatch between output voice and user communication preferences. Summary of the Invention

[0007] The purpose of this application is to provide a dynamic multimodal expression parameter decision-making method and system based on user profiles and dialogue context, which can effectively represent the hierarchical tree-like association structure of user psychological characteristics, realize collaborative decision-making of tone, speech rate and emotional expression, and provide fine-grained continuous control parameters.

[0008] Firstly, this application proposes a dynamic multimodal expression parameter decision-making method based on user profiles and dialogue context, the method comprising: Obtain the user's psychological profile feature vector, the multi-turn dialogue history context sequence, and the current reply text; perform semantic encoding on the current reply text and extract emotional semantic features; The user psychological profile feature vector, the multi-turn dialogue history context sequence, and the emotional semantic features are input into a hierarchical hyperbolic embedding network. Multi-source feature fusion is performed through a hyperbolic space feature aggregation operator to generate a unified hyperbolic embedding representation. The unified hyperbolic embedding representation is input into a long context modeling network based on a selective state space mechanism. Long-range dependency features are extracted through the parameterized state recursive equation that depends on the input, and the fused hidden layer state sequence is output. The fused hidden state sequence is input into a dynamic routing hybrid expert decision network. Expert weights are allocated through a gating routing mechanism, and three decision branches—timbre matching, speech rate control, and emotion expression—are executed in parallel. In the timbre matching branch, the fusion hidden layer state sequence is mapped to the hyperbolic feature space, the hyperbolic distance between the fusion hidden layer state sequence and each timbre prototype vector in the timbre prototype library is calculated, and the timbre template index corresponding to the minimum hyperbolic distance is output. In the speech rate control branch, the fused hidden layer state sequence is input into the continuous prosodic dynamic programming network, and a word-by-word speech rate control curve is generated through the continuous time state evolution model, and a continuous speech rate parameter sequence is output. In the emotion expression branch, the fused hidden state sequence is input into the multi-emotion distribution modeling head, and the mixed weights of each emotion dimension are generated through emotion basis vector space mapping, and the mixed weight vector of emotion dimension is output. By aggregating the timbre template index, the continuous speech rate parameter sequence, and the emotional dimension mixed weight vector, a set of structured multimodal expression parameters is output.

[0009] Optionally, the step of inputting the user psychological profile feature vector, the multi-turn dialogue history context sequence, and the emotional semantic features into a hierarchical hyperbolic embedding network, and performing multi-source feature fusion through a hyperbolic space feature aggregation operator to generate a unified hyperbolic embedding representation includes: The user psychological profile feature vector, the multi-turn dialogue history context sequence, and the emotional semantic features are respectively lexicalized and encoded to generate three types of discrete lexical sequences; The three types of discrete word sequences are mapped to Euclidean space word vector sequences through a pre-trained word embedding layer; the Euclidean space word vector sequences are mapped to the Poincaré disk hyperbolic space through a hyperbolic index mapping layer to generate hyperbolic word vector sequences. The unified hyperbolic embedding representation is generated by element-wise aggregation of hyperbolic word vector sequences using hyperbolic space feature aggregation operators.

[0010] Optionally, the step of inputting the unified hyperbolic embedding representation into a long context modeling network based on a selective state-space mechanism, extracting long-range dependency features through parameterized state recursion equations dependent on the input, and outputting a fused hidden layer state sequence includes: Construct an input-dependent parameterized state recursive equation, which includes a time-varying state transition matrix, a time-varying input projection matrix, and a time-varying output projection matrix. The three types of matrix parameters are dynamically generated by the unified hyperbolic embedding representation through a linear projection layer. The parameterized state recursive equation is solved recursively along the time dimension using a parallel scanning algorithm to generate the hidden state vector for each time step. The hidden state vectors at each time step are mapped from hyperbolic space back to Euclidean space through a hyperbolic logarithmic mapping layer to generate the fused hidden layer state sequence.

[0011] Optionally, the step of inputting the fused hidden state sequence into a dynamic routing hybrid expert decision network, allocating expert weights through a gating routing mechanism, and executing three decision branches in parallel—timbre matching, speech rate control, and emotion expression—includes: Construct a hybrid expert network architecture, which includes a set of timbre expert subnetworks, a set of speech rate expert subnetworks, and a set of emotion expert subnetworks; The fused hidden layer state sequence is linearly projected onto the expert weight space using a learnable gating matrix, and the routing weight distribution of each expert sub-network is generated by the Softmax activation function. Based on the routing weight distribution, the Top-K expert subnetworks are selected for forward computation to generate timbre decision features, speech rate decision features, and emotion decision features, respectively.

[0012] Optionally, in the timbre matching branch, mapping the fusion hidden state sequence to the hyperbolic feature space, calculating the hyperbolic distance between the fusion hidden state sequence and each timbre prototype vector in the timbre prototype library in the hyperbolic feature space, and outputting the timbre template index corresponding to the minimum hyperbolic distance, includes: The timbre prototype vectors corresponding to each timbre category are pre-set in the timbre prototype library through contrastive learning pre-training, and the timbre prototype vectors are stored in the hyperbolic feature space. The fused hidden layer state sequence is mapped to the hyperbolic feature space through a hyperbolic projection layer to generate a query feature vector. Calculate the hyperbolic geodesic distance between the query feature vector and each timbre prototype vector in the timbre prototype library; sort all hyperbolic geodesic distances in ascending order, and extract the index value corresponding to the first position of the sort as the timbre template index.

[0013] Optionally, in the speech rate control branch, the fused hidden layer state sequence is input into a continuous prosodic dynamic programming network, and a word-by-word speech rate control curve is generated through a continuous-time state evolution model, outputting a continuous speech rate parameter sequence, including: Using the fused hidden layer state sequence as the initial hidden state, an integral time interval is set; a continuous-time state evolution function is constructed through a residual network, wherein the continuous-time state evolution function takes the instantaneous hidden state and the global fused features as inputs and outputs the state change gradient; The continuous-time state evolution function is integrally solved by an adaptive step-size numerical solver to generate the hidden state at the termination time. The hidden state at the termination time is input into the word-by-word regression header to generate a speech rate value sequence with the same length as the current reply text, which is used as the continuous speech rate parameter sequence.

[0014] Optionally, in the emotion expression branch, the fused hidden state sequence is input into the multi-emotion distribution modeling head, and mixed weights for each emotion dimension are generated through emotion basis vector space mapping, and an emotion dimension mixed weight vector is output, including: Construct an emotional basis vector set, which contains a preset number of orthogonal basis vectors corresponding to emotional categories; The fused hidden layer state sequence is mapped to the emotion feature space through a parameterized feature transformation network to generate complex amplitude coefficients corresponding to each emotion basis vector; Calculate the squared modulus of each complex amplitude coefficient to obtain the observation probability value of each emotion dimension; normalize all observation probability values ​​to generate the mixed weight vector of the emotion dimension.

[0015] Secondly, this application also proposes a dynamic multimodal expression parameter decision system based on user profiles and dialogue context, the system comprising: The data acquisition module is configured to acquire user psychological profile feature vectors, multi-turn dialogue history context sequences and current response text, perform semantic encoding on the current response text, and extract emotional semantic features; The hyperbolic feature embedding module is configured to input the user psychological profile feature vector, the multi-turn dialogue history context sequence and the emotional semantic features into a hierarchical hyperbolic embedding network, and perform multi-source feature fusion through a hyperbolic space feature aggregation operator to generate a unified hyperbolic embedding representation. The long context modeling module is configured to input the unified hyperbolic embedding representation into a long context modeling network based on a selective state space mechanism, extract long-range dependency features through the parameterized state recursion equation of the input dependency, and output a fused hidden layer state sequence. The multi-expert joint decision-making module is configured to input the fused hidden layer state sequence into the dynamic routing hybrid expert decision-making network, allocate expert weights through a gating routing mechanism, and execute three decision branches in parallel: timbre matching, speech rate control, and emotion expression. The multi-expert joint decision-making module includes a timbre matching unit, a speech rate control unit, and an emotion expression unit. The timbre matching unit is configured to map the fusion hidden layer state sequence to the hyperbolic feature space, calculate the hyperbolic distance between the fusion hidden layer state sequence and each timbre prototype vector in the timbre prototype library in the hyperbolic feature space, and output the timbre template index corresponding to the minimum hyperbolic distance; The speech rate control unit is configured to input the fused hidden layer state sequence into a continuous prosodic dynamic programming network, generate a word-by-word speech rate control curve through a continuous-time state evolution model, and output a continuous speech rate parameter sequence. The emotion expression unit is configured to input the fused hidden layer state sequence into the multi-emotion distribution modeling head, generate the mixed weights of each emotion dimension through emotion basis vector space mapping, and output the emotion dimension mixed weight vector. The parameter aggregation module is configured to aggregate the timbre template index, the continuous speech rate parameter sequence, and the emotional dimension mixed weight vector, and output a set of structured multimodal expression parameters.

[0016] Thirdly, this application also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0017] Fourthly, this application also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0018] Compared with the prior art, this application has the following advantages: This application provides a dynamic multimodal expression parameter decision method and system based on user profiles and dialogue context. It uses a hierarchical hyperbolic embedding network to aggregate multi-source features in hyperbolic space to effectively represent the user's psychological characteristics in a hierarchical tree structure. It uses a dynamic routing hybrid expert decision network to achieve synergistic optimization of timbre matching, speech rate control and emotional expression, and generates a continuous speech rate parameter sequence and a mixed weight vector of emotional dimension to provide fine-grained control. It can significantly improve the naturalness and coherence of speech expression and the degree of anthropomorphism. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a dynamic multimodal expression parameter decision-making method based on user profile and dialogue context, provided for an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a dynamic multimodal expression parameter decision system based on user profile and dialogue context, provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] like Figure 1 As shown, this application proposes a dynamic multimodal expression parameter decision method based on user profiles and dialogue context. The method includes: S1. Obtain the user's psychological profile feature vector, the multi-turn dialogue history context sequence and the current reply text, perform semantic encoding on the current reply text, and extract emotional semantic features; User psychological profile feature vectors can be generated manually or semi-automatically through methods such as questionnaires and historical behavior data analysis to summarize the user's psychological tendencies. The multi-turn dialogue history context sequence can consist of the original text of the previous rounds of dialogue. The current response text is the real-time text for which the system needs to generate speech. Subsequently, the current response text is semantically encoded. For example, traditional methods such as the bag-of-words model or TF-IDF can be used to vectorize the text, and sentiment semantic features are extracted from the semantic encoding result, for example, by using preset keyword matching or sentiment dictionary queries to determine the overall sentiment tendency of the text.

[0023] S2. Input the user psychological profile feature vector, multi-turn dialogue history context sequence and emotional semantic features into the hierarchical hyperbolic embedding network, and perform multi-source feature fusion through the hyperbolic space feature aggregation operator to generate a unified hyperbolic embedding representation. This network can be designed to receive different types of input features and map them directly into a hyperbolic space. For example, these features can be projected into the hyperbolic space separately through independent linear transformation layers, and then multi-source feature fusion can be performed using a hyperbolic space feature aggregation operator. This aggregation operator can be a hyperbolic averaging operation that weights and averages multiple hyperbolic vectors to generate a unified hyperbolic embedding representation.

[0024] S3. Input the unified hyperbolic embedding representation into the long context modeling network based on the selective state space mechanism, extract long-range dependency features through the parameterized state recursion equation of the input dependency, and output the fused hidden layer state sequence. This network can be constructed as a model capable of processing sequential data, with its internal state updated based on the input. Through an input-dependent parameterized state recursion equation, the network can dynamically adjust its memory and processing of historical information based on the current input, thereby extracting long-range dependent features. For example, this recursion equation could be a linear recursive relationship, with coefficients determined by the current input, and output a fused hidden state sequence that reflects the model's comprehensive understanding of both the current and historical context.

[0025] S4. Input the fused hidden state sequence into the dynamic routing hybrid expert decision network, allocate expert weights through a gating routing mechanism, and execute three decision branches in parallel: timbre matching, speech rate control, and emotion expression. This network can contain multiple independent decision modules, each responsible for processing specific expression parameters. Through a gated routing mechanism, the network can assign a weight to each decision module based on the input fusion hidden layer state sequence, thereby determining which modules' outputs contribute more to the final decision. For example, the gated routing mechanism could be a fully connected layer whose output, after normalization, serves as expert weights and executes three decision branches: timbre matching, speech rate control, and emotion expression.

[0026] S5. In the timbre matching branch, the fusion hidden state sequence is mapped to the hyperbolic feature space. The hyperbolic distance between the fusion hidden state sequence and each timbre prototype vector in the timbre prototype library is calculated. The timbre template index corresponding to the minimum hyperbolic distance is output. The sequence can be projected into a hyperbolic feature space using a hyperbolic layer. Then, the hyperbolic distance between this fused hidden layer state sequence and each timbre prototype vector in the timbre prototype library is calculated. The timbre prototype library can store feature vectors of predefined timbre categories (e.g., male, female, and children's voices). By comparing the hyperbolic distances between the query features and these prototype vectors, the timbre template index corresponding to the minimum hyperbolic distance is output, for example, selecting the closest timbre category.

[0027] S6. In the speech rate control branch, the fused hidden state sequence is input into the continuous prosodic dynamic programming network, and the word-by-word speech rate control curve is generated through the continuous time state evolution model, and the continuous speech rate parameter sequence is output. This network can be designed to generate a sequence representing speech rate changes based on the hidden state of the input. Using a continuous-time state evolution model, the network can simulate the continuous change of speech rate over time. For example, it could be a sequence generator based on a recurrent neural network, whose output represents the speech rate value at each time step, thereby generating a word-by-word speech rate control curve and outputting a continuous sequence of speech rate parameters.

[0028] S7. In the emotion expression branch, the fused hidden state sequence is input into the multi-emotion distribution modeling head, and the mixed weights of each emotion dimension are generated through emotion basis vector space mapping, and the mixed weight vector of emotion dimension is output. This modeling head can be configured to predict the intensity of text across multiple preset sentiment dimensions (e.g., happiness, sadness, calm) based on the hidden state of the input. Through sentiment basis vector space mapping, the modeling head can map the input features into a space composed of these sentiment dimensions. For example, it can directly output the score of each sentiment dimension through a fully connected layer, thereby generating mixed weights for each sentiment dimension and outputting a mixed weight vector of sentiment dimensions.

[0029] S8, aggregated timbre template index, continuous speech rate parameter sequence and emotion dimension mixed weight vector, outputs a set of structured multimodal expression parameters.

[0030] This aggregation process can simply combine these parameters into a data format, such as a JSON object containing timbre IDs, a list of speech rate values, and a dictionary of emotion weights, thereby outputting a structured set of multimodal expression parameters for use by subsequent speech synthesis modules.

[0031] In some possible embodiments, user psychological profile feature vectors, multi-turn dialogue history context sequences, and emotional semantic features are input into a hierarchical hyperbolic embedding network. Multi-source feature fusion is performed using a hyperbolic space feature aggregation operator to generate a unified hyperbolic embedding representation, including: S21. Perform word-based encoding on the user psychological profile feature vector, the multi-turn dialogue historical context sequence, and the emotional semantic features to generate three types of discrete word sequences. Lexical encoding is the process of converting raw input data into a sequence of discrete units that the model can process. For user psychological profile feature vectors, they can be discretized into lexical units representing different psychological dimensions or intensity levels, for example, through quantization or clustering. For multi-turn dialogue history context sequences, which are usually text sequences, word segmentation or subword encoding techniques can be used to convert them into lexical sequences, such as using algorithms like BPE (Byte Pair Encoding) or WordPiece. For sentiment semantic features, they can be quantized into lexical units representing different sentiment categories or intensity levels. This step aims to unify heterogeneous input data into a discrete, embeddable representation, laying the foundation for subsequent embedding operations.

[0032] S22. The three types of discrete word sequences are mapped to Euclidean space word vector sequences through a pre-trained word embedding layer; the Euclidean space word vector sequences are mapped to the Poincaré disk hyperbolic space through a hyperbolic index mapping layer to generate hyperbolic word vector sequences. Pre-trained word embedding layers are neural network layers that convert discrete lexical units into continuous, dense vector representations (i.e., word vectors). These word vectors are typically pre-trained on large corpora using self-supervised learning (such as Word2Vec, GloVe, BERT, etc.) to capture the semantic and syntactic relationships between lexical units. Mapping lexical sequences to Euclidean space word vector sequences allows the model to handle continuous numerical representations and leverages the rich semantic knowledge inherent in the pre-trained model to provide high-quality initial representations for subsequent hyperbolic space mapping.

[0033] Hyperbolic space, particularly the Poincaré disk model, is a geometric space with negative curvature, characterized by its ability to more effectively represent data with hierarchical structures or tree-like relationships. The hyperbolic exponential mapping layer is a mathematical transformation that maps vectors in Euclidean space to the hyperbolic space of the Poincaré disk. This mapping can represent hierarchical relationships and long-distance dependencies, which are difficult to capture in Euclidean space, with more compact and natural distance relationships in hyperbolic space. It is particularly suitable for processing features with complex hierarchical semantics, thereby better capturing the intrinsic connections between user psychology, dialogue context, and emotions.

[0034] S23. Element-wise aggregation of hyperbolic word vector sequences is performed using hyperbolic space feature aggregation operators to generate a unified hyperbolic embedding representation. The hyperbolic space feature aggregation operator is an aggregation operation defined in hyperbolic geometry, used to fuse multiple hyperbolic word vector sequences into a unified hyperbolic embedding representation. Unlike averaging or weighted summing in Euclidean space, aggregation operators in hyperbolic space need to consider the characteristics of hyperbolic geometry, such as using hyperbolic averaging, hyperbolic weighted summing, or aggregation methods based on the hyperbolic Möbius transform. Element-wise aggregation means aggregating features at each position in the sequence, ultimately generating a unified hyperbolic embedding representation that comprehensively reflects the user's psychological profile, dialogue history, and emotional semantics. This representation, in hyperbolic space, can better capture the complex hierarchy and semantic relationships between these multi-source features.

[0035] In some possible embodiments, a unified hyperbolic embedding representation is input to a long context modeling network based on a selective state-space mechanism. Long-range dependency features are extracted through parameterized state recursive equations dependent on the input, and the output is a fused hidden layer state sequence, including: S31. Construct input-dependent parameterized state recursive equations. The parameterized state recursive equations include time-varying state transition matrix, time-varying input projection matrix, and time-varying output projection matrix. The parameters of the three types of matrices are dynamically generated by a unified hyperbolic embedding representation through a linear projection layer. S32. Solve the parameterized state recursive equation along the time dimension using a parallel scanning algorithm to generate the hidden state vector for each time step. S33. The hidden state vectors of each time step are mapped from hyperbolic space back to Euclidean space through a hyperbolic logarithmic mapping layer to generate a fused hidden state sequence.

[0036] As one implementation of this embodiment, the parameterized state recursion equation for input dependence constructed in this application has parameters that are not fixed but dynamically adjusted according to the current input (i.e., the unified hyperbolic embedding representation). This means that the model can adjust its internal state evolution logic according to the real-time context of the dialogue, thereby capturing long-range dependencies more flexibly and accurately, allowing the model to exhibit different dynamic behaviors when processing different inputs, and enhancing the model's adaptability and expressiveness. The parameterized state recursion equation includes a time-varying state transition matrix, a time-varying input projection matrix, and a time-varying output projection matrix. In the state-space model, the state transition matrix determines how the current hidden state evolves into the next hidden state, the input projection matrix determines how the current input affects the current hidden state, and the output projection matrix determines how the hidden state is mapped to the output. These matrices are designed to be "time-varying," meaning their values ​​change with time steps (or sequence positions) rather than remaining fixed. This time-varying nature allows the model to adapt to contextual information at different positions in the sequence, thereby modeling long-range dependencies more precisely. The three types of matrix parameters are dynamically generated by the unified hyperbolic embedding representation through a linear projection layer. Unified hyperbolic embeddings incorporate comprehensive information about user mental profiles, dialogue history, and current sentiment. These high-dimensional, semantically rich unified hyperbolic embeddings are transformed into parameters for constructing the aforementioned time-varying matrix through a linear projection layer. The linear projection layer typically consists of learnable weights and biases, capable of mapping input features to different parameter spaces. This dynamic generation mechanism ensures that the parameters of the state recursion equations reflect the complex context of the current dialogue in real time, enabling a highly context-aware extraction process for long-range dependent features.

[0037] In solving the aforementioned parameterized state recursive equations, this application employs a parallel scanning algorithm to recursively solve along the time dimension to generate hidden state vectors for each time step. Traditional sequence models typically process sequences using a sequential recursive approach, which limits computational efficiency, especially when dealing with long sequences. The parallel scanning algorithm is an efficient computational method that allows parallel computation of all time steps in the sequence without sacrificing the model's expressive power. By transforming the recursive relation into a parallelizable form (e.g., using the associative law), this algorithm significantly accelerates the extraction of long-range dependency features, improving the model's training and inference efficiency. Despite employing a parallel scanning algorithm, its core remains the recursive computation of states along the time dimension of the sequence. This means that the hidden state of each time step is calculated based on the hidden state of the previous time step and the input of the current time step, thus ensuring the gradual accumulation and modeling of the sequence context. After solving using the parallel scanning algorithm, the model generates a corresponding hidden state vector for each time step in the sequence. These hidden state vectors are a compressed representation of the current time step and all preceding contextual information, containing rich long-range dependency features and providing comprehensive contextual information for subsequent decision branches.

[0038] For ease of subsequent processing, this application uses a hyperbolic logarithmic mapping layer to map the hidden state vectors of each time step from hyperbolic space back to Euclidean space, generating a fused hidden state sequence. Hyperbolic space has negative curvature, making it suitable for representing data with hierarchical structures or tree-like relationships, while Euclidean space is flat and more suitable for linear operations and traditional neural network processing. The hyperbolic logarithmic mapping layer is a mathematical transformation that maps points in hyperbolic space (i.e., the hidden state vectors of each time step) to corresponding points in Euclidean space. This mapping is necessary because subsequent dynamic routing hybrid expert decision networks may be better suited to operate in Euclidean space, or for compatibility with other modules in Euclidean space. After the transformation by the hyperbolic logarithmic mapping layer, the hidden state vectors of all time steps together constitute a sequence in Euclidean space, namely the fused hidden state sequence. This sequence not only preserves the long-range dependency features extracted from hyperbolic space but also transforms them into a representation that is easier for subsequent modules to process, providing a unified and easy-to-operate input for decision branches such as timbre matching, speech rate control, and emotion expression.

[0039] In some possible embodiments, the fused hidden state sequence is input into a dynamic routing hybrid expert decision network. Expert weights are allocated through a gated routing mechanism, and three decision branches—timbre matching, speech rate control, and emotion expression—are executed in parallel, including: S41. Construct a hybrid expert network architecture, which includes a set of timbre expert subnetworks, a set of speech rate expert subnetworks, and a set of emotion expert subnetworks. S42. The fused hidden layer state sequence is linearly projected onto the expert weight space through a learnable gating matrix, and the routing weight distribution of each expert sub-network is generated through the Softmax activation function. S43. Select the Top-K expert subnetworks according to the routing weight distribution for forward computation, and generate timbre decision features, speech rate decision features and emotion decision features respectively.

[0040] Dynamic routing hybrid expert decision networks are an advanced machine learning architecture whose core consists of multiple "expert" subnetworks and a "gating" network. The gating network dynamically selects or combines the outputs of different experts based on the fused hidden state sequence of the input, enabling the model to handle heterogeneous data or perform multiple tasks. Each expert focuses on processing a specific aspect of the input data or solving a specific sub-problem. This design allows the system to dynamically activate or weight different expert subnetworks based on the characteristics of the input (fused hidden state sequence) to handle the three different decision tasks of timbre, speech rate, and emotion more finely, thus better adapting to the expressive needs of different contexts than a single model.

[0041] Gated routing is key to achieving dynamic selection and weighting. It linearly projects the fused hidden state sequence onto the expert weight space using a learnable gating matrix. This learnable gating matrix is ​​a parameter matrix that continuously optimizes during training, learning how to map complex input information to the evaluation of expert importance. Subsequently, the original weights in the expert weight space are processed by a Softmax activation function to generate a normalized routing weight distribution. The Softmax activation function transforms any real-valued vector into a probability distribution, ensuring that the sum of the weights of all expert subnetworks is 1, and that each weight is between 0 and 1. This facilitates subsequent expert selection or weighted combination. This mechanism allows the system to intelligently allocate computational resources and decision priorities based on the characteristics of the current fused hidden state sequence; for example, when the demand for emotional expression is strong, the emotional expert subnetwork may receive higher weights.

[0042] The three decision branches—timbre matching, speech rate control, and emotion expression—are three independent or semi-independent task processing paths in the hybrid expert decision network, each focusing on generating expression parameters for a specific modality. This decomposition breaks down the complex speech expression parameter decision task into more manageable and optimizable subtasks, with each branch employing algorithms and models optimized for its specific modality. The hybrid expert network architecture comprises sets of timbre expert subnetworks, speech rate expert subnetworks, and emotion expert subnetworks, providing a structured framework for processing multimodal outputs. Each expert subnetwork set can consist of one or more independent neural networks; for example, the timbre expert subnetwork set might include multiple experts specializing in different timbre styles, and the speech rate expert subnetwork set might include experts specializing in different speech rate ranges. These subnetworks can be fully connected layers, convolutional layers, recurrent layers, or Transformer layers, etc., and are specifically designed and trained for their respective modalities, thereby improving the accuracy and flexibility of decision-making.

[0043] After generating the routing weight distribution, the system selects Top-K expert subnetworks for forward computation based on this distribution. The Top-K selection mechanism introduces sparsity, reduces unnecessary computation, and allows the model to focus on the most relevant experts, thereby improving efficiency and decision-making specificity. K can be a preset constant or dynamically determined. The selected Top-K expert subnetworks receive a fused hidden state sequence (or its gated weighted version) as input and independently perform their internal computations to generate timbre decision features, speech rate decision features, and sentiment decision features, respectively. These decision features, generated by their respective expert subnetworks, serve as intermediate representations for subsequent decisions on specific modality parameters (timbre template index, continuous speech rate parameter sequence, and sentiment dimension mixed weight vector). They provide more discriminative, expert-processed information specific to the modality, laying the foundation for the final parameter output.

[0044] In some possible embodiments, in the timbre matching branch, the fused hidden state sequence is mapped to the hyperbolic feature space, the hyperbolic distance between the fused hidden state sequence and each timbre prototype vector in the timbre prototype library is calculated, and the timbre template index corresponding to the minimum hyperbolic distance is output, including: S51. Pre-set the timbre prototype vectors corresponding to each timbre category in the timbre prototype library, which are obtained through contrastive learning pre-training. The timbre prototype vectors are stored in the hyperbolic feature space. S52. Map the fused hidden layer state sequence to the hyperbolic feature space through a hyperbolic projection layer to generate a query feature vector; S53. Calculate the hyperbolic geodesic distance between the query feature vector and each timbre prototype vector in the timbre prototype library; sort all hyperbolic geodesic distances in ascending order, and extract the index value corresponding to the first sorted value as the timbre template index.

[0045] As an implementation, a timbre prototype library is a pre-built database storing feature vectors representing different timbre categories. These prototype vectors are not randomly generated but pre-trained using a contrastive learning method. Contrastive learning trains the model by constructing positive sample pairs (e.g., expressions of the same timbre in different contexts) and negative sample pairs (different timbres), ensuring that features of the same timbre are close to each other in hyperbolic space, while features of different timbres are far apart. This training method helps learn timbre representations with strong discriminative power and robustness. Storing these prototype vectors in a hyperbolic feature space is chosen because hyperbolic geometry has a natural advantage in modeling data with hierarchical structures or complex relationships, and can more effectively capture subtle differences and potential hierarchical relationships between timbres.

[0046] The fused hidden state sequence is a comprehensive representation generated by the system after integrating information such as user psychological profile, dialogue history context, and current response text. It contains key information that determines timbre selection. To effectively compare it with the hyperbolic timbre prototype vectors in the timbre prototype library, this fused hidden state sequence needs to be transformed into the same hyperbolic feature space. The hyperbolic projection layer is a specially designed neural network layer whose function is to project the input (e.g., the fused hidden state sequence, which may be in Euclidean space) into a hyperbolic geometric space through nonlinear transformations and hyperbolic exponential mapping, thereby generating a query feature vector in the same geometric space as the timbre prototype vectors.

[0047] In hyperbolic feature space, the standard for measuring the similarity or difference between two vectors is hyperbolic geodesic distance. Unlike straight-line distance in Euclidean space, hyperbolic geodesic distance is the length of the shortest path between two points on a hyperbolic manifold. For example, in the Poincaré disk model, the calculation of hyperbolic geodesic distance involves the hyperbolic sine function and the norm of the vector, which can more accurately reflect the true distance relationship between features in hyperbolic space. By calculating the hyperbolic geodesic distance between the query feature vector and each timbre prototype vector in the timbre prototype library, the degree of matching between the required timbre in the current dialogue context and various preset timbre prototypes can be quantified.

[0048] After calculating the hyperbolic geodesic distances between the query feature vector and all timbre prototype vectors, these distance values ​​are sorted in ascending order. The purpose of sorting is to find the timbre prototype vector that is closest to the query feature vector, as a smaller distance indicates a higher degree of matching. After sorting, the index of the timbre prototype vector corresponding to the distance value at the top of the list is selected as the timbre template index that best fits the current context. This index can then be used to guide the speech synthesis system in selecting a specific timbre template to generate a speech response with the corresponding timbre.

[0049] In some possible embodiments, in the speech rate control branch, the fused hidden state sequence is input into a continuous prosodic dynamic programming network, and a word-by-word speech rate control curve is generated through a continuous-time state evolution model, outputting a continuous speech rate parameter sequence, including: S61. Using the fused hidden state sequence as the initial hidden state, the integral time interval is set; a continuous-time state evolution function is constructed through a residual network. The continuous-time state evolution function takes the instantaneous hidden state and the global fused features as inputs and outputs the state change gradient. The fused hidden state sequence is obtained after processing by a hierarchical hyperbolic embedding network and a long context modeling network. It contains rich contextual information such as user psychological profile feature vectors, multi-turn dialogue history context sequences, and emotional semantic features. As the initial hidden state, it can provide a comprehensive contextual foundation for subsequent dynamic speech rate planning. At the same time, an integral time interval is defined, which defines the scope of the speech rate evolution model in the time dimension. It is usually associated with the expected pronunciation duration of the current response text, providing a clear time boundary for the continuous time state evolution process.

[0050] Residual networks, by introducing skip connections, effectively alleviate the vanishing gradient problem in deep network training, enabling them to learn more complex and refined nonlinear mappings. Here, residual networks are used to model the continuous temporal changes of speech rate-related hidden states. The continuous-time state evolution function receives instantaneous hidden states and global fusion features as input. The instantaneous hidden states represent the current state of speech rate evolution, while the global fusion features provide overall contextual information throughout the evolution process, ensuring that the dynamic adjustment of speech rate remains consistent with the overall dialogue context. The core output of this function is the state change gradient, i.e., the instantaneous rate of change of the speech rate-related hidden states at the current moment, which provides the basis for subsequent continuous-time integration.

[0051] S62. The continuous-time state evolution function is integrated using an adaptive step-size numerical solver to generate the terminal hidden state. The adaptive step-size numerical solver is an efficient and accurate numerical integration method that dynamically adjusts the integration step size based on the intensity of the state change gradient. For example, when the state changes rapidly, the solver uses a smaller step size to capture details; when the state changes gradually, it uses a larger step size to improve computational efficiency. This mechanism ensures that while maintaining computational efficiency, it can accurately simulate the continuous evolution of the speech rate hidden state throughout the entire integration time interval, ultimately obtaining a terminal hidden state that fully reflects contextual information and dynamic prosodic changes.

[0052] S63. Input the terminal hidden state into the word-by-word regression head to generate a speech rate value sequence of the same length as the current reply text, serving as a continuous speech rate parameter sequence. The word-by-word regression head is a specially designed neural network module whose function is to map the high-dimensional terminal hidden state into a series of specific numerical values ​​that can be used to control the speech rate. This regression head can generate a corresponding speech rate value for each character or word in the current reply text based on the rich information contained in the terminal hidden state. In this way, a speech rate value sequence of the same length as the current reply text is generated as a continuous speech rate parameter sequence. This ensures that the generated speech rate parameters are precisely aligned with the text content, achieving fine-grained speech rate control at the word-by-word level.

[0053] In some possible embodiments, in the emotion expression branch, the fused hidden state sequence is input into the multi-emotion distribution modeling head, and mixed weights for each emotion dimension are generated through emotion basis vector space mapping. The output emotion dimension mixed weight vector includes: S71. Construct a set of sentiment basis vectors. This set contains a predetermined number of orthogonal basis vectors corresponding to a set of sentiment categories. These categories can be flexibly defined according to the needs of the actual application scenario. For example, they can cover core sentiment dimensions such as joy, sadness, anger, surprise, and neutrality. The orthogonal nature of the basis vectors ensures that these sentiment dimensions are independent of each other and have a unified dimension in the sentiment space. This helps to decouple different sentiment dimensions and lays the foundation for subsequent quantitative analysis. In practice, these sentiment basis vectors can be manually defined using domain expert knowledge or automatically extracted through unsupervised learning on large-scale sentiment-labeled data (e.g., using principal component analysis or independent component analysis) to capture the main directions of variation in the sentiment space.

[0054] S72. The fused hidden state sequence is mapped to the sentiment feature space through a parameterized feature transformation network, generating complex amplitude coefficients corresponding to each sentiment basis vector. The parameterized feature transformation network is a learnable neural network module whose function is to transform the fused hidden state sequence output from the long context modeling network (which integrates user psychological profile feature vectors, multi-turn dialogue historical context sequences, and sentiment semantic features) into a feature space specifically for sentiment modeling. This network can adopt various neural network architectures, such as multilayer perceptron (MLP), convolutional neural network (CNN), or recurrent neural network (RNN), and its internal parameters are optimized during training using the backpropagation algorithm. Generating complex amplitude coefficients is a core technical point of this scheme. Complex numbers can simultaneously represent amplitude and phase information. In this context, the magnitude of the complex amplitude can characterize the strength or importance of the corresponding sentiment basis vector, while its phase can encode subtle changes in sentiment or contextual dependencies, thus providing a richer sentiment representation capability than a single real number. For example, a complex amplitude coefficient... Its model This represents the activation level of the corresponding emotional basis vector.

[0055] S73. Calculate the squared modulus of each complex amplitude coefficient to obtain the observation probability value of each emotion dimension; normalize all observation probability values ​​to generate a mixed weight vector of emotion dimensions.

[0056] The square of the modulus of the complex amplitude coefficient (i.e. The magnitude of a wave function can be understood as the "energy" or "intensity" of the corresponding emotional dimension in the current context, which is similar to the principle of representing probability using the squared modulus of a wave function in quantum mechanics. By calculating the squared modulus, the representation in the complex field is converted into intensity values ​​in the real field. These intensity values ​​reflect the potential contribution of each emotional dimension to the current expression. Subsequently, these intensity values ​​are normalized, for example, by using the Softmax function or simply dividing each value by the sum of all intensity values, to ensure that the sum of the weights of all emotional dimensions is 1. This results in a probability distribution-formed emotional dimension mixture weight vector, where each element represents the relative importance or mixing proportion of the corresponding emotional category in the final multimodal expression.

[0057] It should be noted that the above description describes some embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0058] Based on the same inventive concept, corresponding to any of the above-described embodiments, this application also proposes a dynamic multimodal expression parameter decision system based on user profiles and dialogue context, such as... Figure 2 As shown; The data acquisition module 100 is configured to acquire user psychological profile feature vectors, multi-turn dialogue history context sequences and current reply text, perform semantic encoding on the current reply text, and extract emotional semantic features. The hyperbolic feature embedding module 200 is configured to input user psychological profile feature vectors, multi-turn dialogue history context sequences and emotional semantic features into a hierarchical hyperbolic embedding network, and perform multi-source feature fusion through a hyperbolic space feature aggregation operator to generate a unified hyperbolic embedding representation. The long context modeling module 300 is configured to input a unified hyperbolic embedding representation into a long context modeling network based on a selective state space mechanism, extract long-range dependency features through the parameterized state recursion equation of the input dependency, and output a fused hidden layer state sequence. The multi-expert joint decision-making module 400 is configured to input the fused hidden state sequence into the dynamic routing hybrid expert decision-making network, allocate expert weights through a gating routing mechanism, and execute three decision branches in parallel: timbre matching, speech rate control, and emotion expression. The multi-expert joint decision-making module 400 includes a timbre matching unit 410, a speech rate control unit 420, and an emotion expression unit 430; The timbre matching unit 410 is configured to map the fusion hidden layer state sequence to the hyperbolic feature space, calculate the hyperbolic distance between the fusion hidden layer state sequence and each timbre prototype vector in the timbre prototype library in the hyperbolic feature space, and output the timbre template index corresponding to the minimum hyperbolic distance; The speech rate control unit 420 is configured to input the fused hidden layer state sequence into a continuous prosodic dynamic programming network, generate a word-by-word speech rate control curve through a continuous-time state evolution model, and output a continuous speech rate parameter sequence. The emotion expression unit 430 is configured to input the fused hidden state sequence into the multi-emotion distribution modeling head, generate the mixed weights of each emotion dimension through emotion basis vector space mapping, and output the emotion dimension mixed weight vector; The parameter aggregation module 500 is configured to aggregate timbre template indexes, continuous speech rate parameter sequences, and emotional dimension mixed weight vectors, and output a set of structured multimodal expression parameters.

[0059] The apparatus described above is used to implement the corresponding methods in the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0060] Based on the same inventive concept, this application also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the above-described dynamic multimodal expression parameter decision method based on user profile and dialogue context.

[0061] Based on the same inventive concept, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described dynamic multimodal expression parameter decision method based on user profile and dialogue context.

[0062] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A dynamic multimodal expression parameter decision-making method based on user profiles and dialogue context, characterized in that, The method includes: Obtain the user's psychological profile feature vector, the multi-turn dialogue history context sequence, and the current reply text; perform semantic encoding on the current reply text and extract emotional semantic features; The user psychological profile feature vector, the multi-turn dialogue history context sequence, and the emotional semantic features are input into a hierarchical hyperbolic embedding network. Multi-source feature fusion is performed through a hyperbolic space feature aggregation operator to generate a unified hyperbolic embedding representation. The unified hyperbolic embedding representation is input into a long context modeling network based on a selective state space mechanism. Long-range dependency features are extracted through the parameterized state recursive equation that depends on the input, and the fused hidden layer state sequence is output. The fused hidden state sequence is input into a dynamic routing hybrid expert decision network. Expert weights are allocated through a gating routing mechanism, and three decision branches—timbre matching, speech rate control, and emotion expression—are executed in parallel. In the timbre matching branch, the fusion hidden layer state sequence is mapped to the hyperbolic feature space, the hyperbolic distance between the fusion hidden layer state sequence and each timbre prototype vector in the timbre prototype library is calculated, and the timbre template index corresponding to the minimum hyperbolic distance is output. In the speech rate control branch, the fused hidden layer state sequence is input into the continuous prosodic dynamic programming network, and a word-by-word speech rate control curve is generated through the continuous time state evolution model, and a continuous speech rate parameter sequence is output. In the emotion expression branch, the fused hidden state sequence is input into the multi-emotion distribution modeling head, and the mixed weights of each emotion dimension are generated through emotion basis vector space mapping, and the mixed weight vector of emotion dimension is output. By aggregating the timbre template index, the continuous speech rate parameter sequence, and the emotional dimension mixed weight vector, a set of structured multimodal expression parameters is output.

2. The method according to claim 1, characterized in that, The step of inputting the user psychological profile feature vector, the multi-turn dialogue history context sequence, and the emotional semantic features into a hierarchical hyperbolic embedding network, and performing multi-source feature fusion through a hyperbolic space feature aggregation operator to generate a unified hyperbolic embedding representation includes: The user psychological profile feature vector, the multi-turn dialogue history context sequence, and the emotional semantic features are respectively lexicalized and encoded to generate three types of discrete lexical sequences; The three types of discrete word sequences are mapped to Euclidean space word vector sequences through a pre-trained word embedding layer; the Euclidean space word vector sequences are mapped to the Poincaré disk hyperbolic space through a hyperbolic index mapping layer to generate hyperbolic word vector sequences. The unified hyperbolic embedding representation is generated by element-wise aggregation of hyperbolic word vector sequences using hyperbolic space feature aggregation operators.

3. The method according to claim 1, characterized in that, The unified hyperbolic embedding representation input is used in a long context modeling network based on a selective state-space mechanism to extract long-range data through parameterized state recursion equations dependent on the input. Dependent on features, output the fused hidden layer state sequence, including: Construct an input-dependent parameterized state recursive equation, which includes a time-varying state transition matrix, a time-varying input projection matrix, and a time-varying output projection matrix. The three types of matrix parameters are dynamically generated by the unified hyperbolic embedding representation through a linear projection layer. The parameterized state recursive equation is solved recursively along the time dimension using a parallel scanning algorithm to generate the hidden state vector for each time step. The hidden state vectors at each time step are mapped from hyperbolic space back to Euclidean space through a hyperbolic logarithmic mapping layer to generate the fused hidden layer state sequence.

4. The method according to claim 1, characterized in that, The process involves inputting the fused hidden state sequence into a dynamic routing hybrid expert decision network, allocating expert weights through a gating routing mechanism, and executing three decision branches in parallel: timbre matching, speech rate control, and emotion expression. This includes: Construct a hybrid expert network architecture, which includes a set of timbre expert subnetworks, a set of speech rate expert subnetworks, and a set of emotion expert subnetworks; The fused hidden layer state sequence is linearly projected onto the expert weight space using a learnable gating matrix, and the routing weight distribution of each expert sub-network is generated by the Softmax activation function. Based on the routing weight distribution, the Top-K expert subnetworks are selected for forward computation to generate timbre decision features, speech rate decision features, and emotion decision features, respectively.

5. The method according to claim 1, characterized in that, In the timbre matching branch, the fusion hidden state sequence is mapped to the hyperbolic feature space, the hyperbolic distance between the fusion hidden state sequence and each timbre prototype vector in the timbre prototype library is calculated, and the timbre template index corresponding to the minimum hyperbolic distance is output, including: The timbre prototype vectors corresponding to each timbre category are pre-set in the timbre prototype library through contrastive learning pre-training, and the timbre prototype vectors are stored in the hyperbolic feature space. The fused hidden layer state sequence is mapped to the hyperbolic feature space through a hyperbolic projection layer to generate a query feature vector. Calculate the hyperbolic geodesic distance between the query feature vector and each timbre prototype vector in the timbre prototype library; sort all hyperbolic geodesic distances in ascending order, and extract the index value corresponding to the first position of the sort as the timbre template index.

6. The method according to claim 1, characterized in that, In the speech rate control branch, the fused hidden layer state sequence is input into a continuous prosodic dynamic programming network. A word-by-word speech rate control curve is generated through a continuous-time state evolution model, and a continuous speech rate parameter sequence is output, including: Using the fused hidden layer state sequence as the initial hidden state, an integral time interval is set; a continuous-time state evolution function is constructed through a residual network, wherein the continuous-time state evolution function takes the instantaneous hidden state and the global fused features as inputs and outputs the state change gradient; The continuous-time state evolution function is integrally solved by an adaptive step-size numerical solver to generate the hidden state at the termination time. The hidden state at the termination time is input into the word-by-word regression header to generate a speech rate value sequence with the same length as the current reply text, which is used as the continuous speech rate parameter sequence.

7. The method according to claim 1, characterized in that, In the emotion expression branch, the fused hidden state sequence is input into the multi-emotion distribution modeling head, and mixed weights for each emotion dimension are generated through emotion basis vector space mapping. The output emotion dimension mixed weight vector includes: Construct an emotional basis vector set, which contains a preset number of orthogonal basis vectors corresponding to emotional categories; The fused hidden layer state sequence is mapped to the emotion feature space through a parameterized feature transformation network to generate complex amplitude coefficients corresponding to each emotion basis vector; Calculate the squared modulus of each complex amplitude coefficient to obtain the observation probability value of each emotion dimension; normalize all observation probability values ​​to generate the mixed weight vector of the emotion dimension.

8. A dynamic multimodal expression parameter decision system based on user profiles and dialogue context, characterized in that, The system includes: The data acquisition module is configured to acquire user psychological profile feature vectors, multi-turn dialogue history context sequences and current response text, perform semantic encoding on the current response text, and extract emotional semantic features; The hyperbolic feature embedding module is configured to input the user psychological profile feature vector, the multi-turn dialogue history context sequence and the emotional semantic features into a hierarchical hyperbolic embedding network, and perform multi-source feature fusion through a hyperbolic space feature aggregation operator to generate a unified hyperbolic embedding representation. The long context modeling module is configured to input the unified hyperbolic embedding representation into a long context modeling network based on a selective state space mechanism, extract long-range dependency features through the parameterized state recursion equation of the input dependency, and output a fused hidden layer state sequence. The multi-expert joint decision-making module is configured to input the fused hidden layer state sequence into the dynamic routing hybrid expert decision-making network, allocate expert weights through a gating routing mechanism, and execute three decision branches in parallel: timbre matching, speech rate control, and emotion expression. The multi-expert joint decision-making module includes a timbre matching unit, a speech rate control unit, and an emotion expression unit. The timbre matching unit is configured to map the fusion hidden layer state sequence to the hyperbolic feature space, calculate the hyperbolic distance between the fusion hidden layer state sequence and each timbre prototype vector in the timbre prototype library in the hyperbolic feature space, and output the timbre template index corresponding to the minimum hyperbolic distance; The speech rate control unit is configured to input the fused hidden layer state sequence into a continuous prosodic dynamic programming network, generate a word-by-word speech rate control curve through a continuous-time state evolution model, and output a continuous speech rate parameter sequence. The emotion expression unit is configured to input the fused hidden layer state sequence into the multi-emotion distribution modeling head, generate the mixed weights of each emotion dimension through emotion basis vector space mapping, and output the emotion dimension mixed weight vector. The parameter aggregation module is configured to aggregate the timbre template index, the continuous speech rate parameter sequence, and the emotional dimension mixed weight vector, and output a set of structured multimodal expression parameters.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images