Large voice model system based on elderly language expression characteristic perception

Abstract

The invention relates to the field of voice generation, and particularly discloses a voice large model system based on elderly language expression characteristic perception, which generates a vector comprehensively reflecting the current state of a user by capturing and quantifying rhythm characteristics such as speed and pause in a voice stream of an elderly user in real time and combining dialogue history. The state vector is input into a strategy network, the core function of the network is to execute dynamic mapping, and low-order acoustic features are intelligently translated into high-order strategy vectors capable of guiding large model behaviors. The strategy vector is used for dynamically constructing and configuring prompt instructions and reasoning parameters (such as temperature and depth) of a large model, so that the response style and details are accurately regulated and controlled. In this way, the problem of lack of a mapping mechanism is solved, a bridge from sensing to adaptation is built, and refined and self-adaptive regulation and control of large model behaviors are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of voice generation, and more specifically, to a voice large model system based on the perception of language expression characteristics of the elderly. BACKGROUND

[0002] With the intensification of the aging trend of the population and the popularization of artificial intelligence technology, how to use advanced technology to serve the elderly population and bridge the digital divide has become the focus of social attention. In the field of human-computer interaction, voice large model systems, as cutting-edge technologies, are gradually integrated into all aspects of daily life. However, the unique language expression habits of elderly users, such as slower speech, more frequent pauses, or the use of fillers, are often not fully understood and adapted by existing systems. These language expression characteristics not only serve as a medium for communication, but also carry rich information about the current cognitive state, emotional needs, and communication intentions of the elderly user. Therefore, building a voice large model system that can actively perceive and intelligently adapt to the language expression characteristics of the elderly is crucial for improving their interaction experience and achieving personalized care and service quality with warmth.

[0003] However, existing technologies have significant limitations in achieving this goal. The core architecture of mainstream voice assistants or large model applications typically focuses on accurate recognition of voice content, i.e., directly converting user voice signals into text, and then performing semantic understanding and response generation. This processing approach generally lacks deep perceptual capabilities for rhythm, emotion, and other paralinguistic information contained in voice streams, and is unable to translate these dynamically changing perceptual features into fine-grained control of large model inference behavior. That is, there is a lack of a core bridge that effectively connects the perception and adaptation of existing technologies, i.e., a dynamic mapping mechanism from user expression rhythm to large model inference strategy. The absence of such a mapping mechanism is rooted in the technological gap between the front-end voice physical signal and the back-end abstract inference strategy of the large model. Specifically, the front-end perceived speech rate, pause ratio, and other continuous, physically meaningful acoustic features, while the inference behavior of the large model is controlled by highly abstract discrete parameters and text instructions such as temperature, depth of thought, and there is a serious mismatch in interface and semantics between the two. Existing technologies are unable to establish a dynamic mapping model that understands the context of the conversation history, and cannot accurately and adaptively adjust the depth of thought, response style, and content detail of the LLM according to the user's real-time, subtle expression rhythm changes.

[0004] Therefore, how to design an effective intermediate layer to intelligently translate low-level acoustic rhythm features into high-level, executable LLM inference strategies has become a key technical problem in realizing truly personalized and individualized elderly services, and needs to be solved urgently. SUMMARY

[0005] To solve the above technical problems, the present application is proposed. According to the voice large model system based on the language expression characteristics perception of the elderly, comprising:

[0006] The original audio stream rhythm voice analysis module is used for voice recognition and multi-dimensional rhythm feature extraction on the original audio stream of the target elderly object to obtain a rhythm feature vector and a voice recognition text.

[0007] The user state coding module is used for context-aware user state coding on the rhythm feature vector and the dialogue history embedding to obtain a current state vector.

[0008] The strategy vector generation module is used for inputting the current state vector into the trained strategy network to obtain a strategy vector.

[0009] The prompt configuration construction module is used for constructing a final prompt and reasoning configuration of the voice recognition text based on the strategy vector.

[0010] The reply text generation module is used for inputting the final prompt and reasoning configuration of the voice recognition text into the voice large model to obtain a final reply text.

[0011] The reply text conversion module is used for converting the final reply text into a reply voice signal.

[0012] Compared with the prior art, the voice large model system based on the language expression characteristics perception of the elderly provided by the present application constructs a strategy network based on deep reinforcement learning as an intelligent decision-making hub connecting the front-end voice perception and the back-end large model reasoning. Specifically, by capturing and quantifying the rhythm features such as speech rate and pause in the voice stream of the elderly user in real time, and combining the dialogue history, a vector reflecting the current state of the user is generated. The state vector is input into the strategy network, and the core function of the network is to perform dynamic mapping to intelligently translate low-order acoustic features into high-order strategy vectors that can guide the behavior of the large model. The strategy vector is used to dynamically construct and configure the prompt instructions and reasoning parameters (such as temperature and depth) of the large model, so as to accurately regulate the style and detail of its response. In this way, the mapping mechanism missing due to the semantic gap between the front-end physical signal and the back-end abstract control parameter mentioned in the background technology is solved, a bridge from perception to adaptation is built, and fine and adaptive regulation of the behavior of the large model is realized. BRIEF DESCRIPTION OF DRAWINGS

[0013] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description thereof taken in conjunction with the accompanying drawings, in which: The accompanying drawings provide exemplary embodiments of the application and serve as an aid in understanding the application. They are a part of and constitute portions of this specification that illustrate the application. These drawings are not intended to limit the scope of the application and are merely included to describe and explain illustrative embodiments of the application. In the drawings:

[0014] Figure 1 A block diagram of a voice large model system based on perception of language expression characteristics of the elderly according to embodiments of the present application.

[0015] Figure 2 A data flow diagram of a voice large model system based on perception of language expression characteristics of the elderly according to embodiments of the present application.

[0016] Figure 3 A block diagram of a raw audio stream rhythm voice analysis module in a voice large model system based on perception of language expression characteristics of the elderly according to embodiments of the present application.

[0017] Figure 4 A block diagram of a prompt configuration construction module in a voice large model system based on perception of language expression characteristics of the elderly according to embodiments of the present application.

[0018] Figure 5 A block diagram of a reply text generation module in a voice large model system based on perception of language expression characteristics of the elderly according to embodiments of the present application. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the present disclosure are shown. As this disclosure may, and certainly does, include a variety of embodiments, no limitation as to the scope of the present disclosure is intended to be imputed to any embodiment described herein or illustrated in the drawings. Rather, it is contemplated that the present disclosure includes as its embodiments other specific embodiments not described herein but which are consistent with the principles of the present disclosure.

[0020] The present application is proposed in view of the foregoing problems in the prior art. Figure 1 A block diagram of a voice large model system based on perception of language expression characteristics of the elderly according to embodiments of the present application. Figure 2 A data flow diagram of a voice large model system based on perception of language expression characteristics of the elderly according to embodiments of the present application. Specifically, as Figure 1 and Figure 2As shown, the voice large model system 100 based on the language expression characteristics perception of the elderly according to the embodiment of the present application comprises: an original audio stream rhythm voice analysis module 110, configured to perform voice recognition and multi-dimensional rhythm feature extraction on the original audio stream of the target elderly object acquired to obtain a rhythm feature vector and a voice recognition text; a user state coding module 120, configured to perform context-aware user state coding on the rhythm feature vector and the dialogue history embedding to obtain a current state vector; a strategy vector generation module 130, configured to input the current state vector into a trained strategy network to obtain a strategy vector; a prompt configuration construction module 140, configured to construct a final prompt and reasoning configuration of the voice recognition text based on the strategy vector; a reply text generation module 150, configured to input the final prompt and reasoning configuration of the voice recognition text into a voice large model to obtain a final reply text; and a reply text conversion module 160, configured to convert the final reply text into a reply voice signal.

[0021] Specifically, the original audio stream rhythm voice analysis module 110 is configured to perform voice recognition and multi-dimensional rhythm feature extraction on the original audio stream of the target elderly object acquired to obtain a rhythm feature vector and a voice recognition text. It can be understood that when performing voice interaction with an elderly user, it is far from enough to only understand the literal meaning of his speech. The language expression manner of the elderly, including their unique speech rate, pause and intonation change, contains rich state information far beyond the text content, such as cognitive load, emotional fluctuation or communication intention. The prior art generally ignores the in-depth mining of these paralanguage information, resulting in a harsh and lack of humanized interactive experience. In order to make up for this defect and truly realize intelligent response to the deep needs of the elderly user, the present application first needs to perform voice recognition and multi-dimensional rhythm feature extraction on the original audio stream of the target elderly object acquired, to parallelly analyze two important information dimensions from the original, single-dimensional audio signal: one is the content dimension carrying the user's explicit instructions, i.e. the voice recognition text; the other is the manner dimension reflecting the user's potential state, i.e. the multi-dimensional rhythm feature vector, to provide a basic input for the subsequent modules to establish a comprehensive and three-dimensional user state portrait.

[0022] In one possible implementation, Figure 3 A block diagram of the original audio stream rhythm voice analysis module in the voice large model system based on the language expression characteristics perception of the elderly according to the embodiment of the present application is shown in FIG. 2. As shown, the original audio stream rhythm voice analysis module 110 comprises: a voice recognition unit 111, configured to perform voice recognition on the original audio stream to obtain the voice recognition text; an acoustic feature extraction unit 112, configured to extract acoustic features from the original audio stream; and a rhythm feature vector combination unit 113, configured to combine the acoustic features into the rhythm feature vector after normalization. Figure 3 As shown, the original audio stream rhythm voice analysis module 110 comprises: a voice recognition unit 111, configured to perform voice recognition on the original audio stream to obtain the voice recognition text; an acoustic feature extraction unit 112, configured to extract acoustic features from the original audio stream; and a rhythm feature vector combination unit 113, configured to combine the acoustic features into the rhythm feature vector after normalization.

[0023] In the above embodiment, the sensor data frame acquisition module 110 includes the following: this process takes a specific input as an example: a segment of raw audio stream recorded by the target elderly user, the content of which is: "that... I want to ask... what's the weather like today?". The audio stream is input to the speech recognition unit in the form of digital waveform data. The speech recognition unit 111 relies on a depth-optimized end-to-end automatic speech recognition model. The overall architecture of this model precisely integrates convolutional neural networks, long short-term memory networks, and connectionist temporal classification loss functions to ensure efficient and accurate processing. The first step is audio preprocessing and feature extraction. The input raw audio waveform is first subjected to frame processing, i.e., it is cut into a series of short-time audio frames with a length of 25 milliseconds, and there is a 10-millisecond overlap between frames. The 25-millisecond frame length aims to balance the quasi-stationary characteristics of the speech signal and the accuracy requirements of spectral analysis, while the 10-millisecond overlap ensures smooth transition between frames and avoids truncating key acoustic features at the frame boundary, thereby ensuring continuity of information. Then, a Hamming window function is applied to each frame to reduce spectral leakage, and a fast Fourier transform is performed to convert it from the time domain to the frequency domain to obtain a spectrogram. Next, the spectrogram is passed through a Mel filter bank, which is set to simulate the hearing characteristics of the human ear and better capture the key frequency information of the speech. Finally, the energy values after the filter bank are taken logarithmically and subjected to discrete cosine transform, ultimately obtaining a set of Mel frequency cepstral coefficient vectors that can compactly represent the core features of each audio frame. After the entire preprocessing process, the original one-dimensional audio waveform data is converted into a two-dimensional feature matrix, with each column representing a time frame and each row representing a dimension of Mel frequency cepstral coefficients. Next, the feature matrix is fed into the convolutional neural network layer of the model. This part is stacked by several one-dimensional convolutional layers, batch normalization layers, and activation functions such as ReLU. The convolution kernel slides in the time dimension to capture local time-frequency patterns in the Mel frequency cepstral coefficient features, such as the rapid transition from voiced to unvoiced phonemes or the stable formant structure of vowels. After the convolutional neural network layer, the output feature sequence is input into the multi-layer bidirectional long short-term memory network. Long short-term memory network is a special recurrent neural network whose internal gating mechanism (input gate, forget gate, output gate) enables it to effectively learn and remember long-distance dependencies in time series, and the bidirectional structure can process the feature sequence from both forward and backward directions. This allows the model to utilize not only previous acoustic information but also subsequent information when making judgments about any time frame, which is crucial for accurately identifying context-dependent phonemes and words. For example, in identifying the word "weather", the bidirectional long short-term memory network can utilize all the speech information before "day" and after "air" to make a more reliable judgment.The output of the bidirectional long short-term memory network is a sequence of hidden states containing rich context information. This sequence then passes through a fully connected layer, where the number of neurons in this layer is equal to the total number of all characters that the speech model can recognize. For example, it is the total number of thousands of common Chinese characters, English letters, numbers, plus a special blank symbol. Subsequently, a Softmax activation function acts on the output of the fully connected layer, converting the output vector for each time frame into a probability distribution vector. In this way, the model generates a probability of occurrence for all possible characters for each input audio frame. The last step is decoding. The decoding process uses the connectionist temporal classification algorithm. This algorithm receives the probability distribution matrix output by the Softmax layer and searches for the path of the character sequence with the highest probability. The uniqueness of connectionist temporal classification lies in its ability to automatically handle repeated characters and blank symbols in the output sequence. For example, the model may output a series of identical characters for a continuously pronounced phoneme, such as "天天天气", and the connectionist temporal classification decoder will merge them into a single "天". At the same time, it will remove all blank symbols inserted between characters, which represent moments of non-pronunciation or pronunciation transitions. Finally, for the example audio, the final result output by the decoder is a string of text for speech recognition: "那个我想问一下今天天气怎么样?". It is worth mentioning that regarding the acquisition of model parameters, the learnable parameters such as the weights and biases of this model are obtained through training on a large amount of data. First, it is pre-trained on a large-scale corpus containing hundreds of thousands of hours of general speech data, enabling the model to learn the general laws of language and acoustics. Subsequently, it is fine-tuned on a specific dataset specially collected, which contains thousands of hours of speech data of the elderly (covering different accents, speech rates, and common expressions). The fine-tuning process continuously adjusts the model parameters through the backpropagation algorithm and optimizer to minimize the connectionist temporal classification loss function, so as to make the difference between the text output by the model and the true annotated text as small as possible, ensuring that it has a very high recognition accuracy for the target elderly user group.

[0024] Meanwhile, the acoustic feature extraction unit 112 processes the same segment of the original audio stream to extract the multi-dimensional rhythm information contained therein. In one possible implementation, the acoustic features include speech rate, pause ratio, filler word technique, and fundamental frequency contour variance. That is, the speech rate, pause ratio, and frequency of filler word usage collectively constitute key indicators of the user's speech fluency, which can very intuitively and quantitatively reflect the user's current cognitive load, degree of hesitation, or thinking state when recalling information. Meanwhile, the fundamental frequency contour variance captures the user's emotional color and degree of involvement from the prosody level, effectively distinguishing between a segment of expression that is emotionally charged and highlighted, or a segment that is flat in tone and possibly with a tired or indifferent emotion. Therefore, this combination aims to comprehensively and deeply perceive the user's internal state from the two core dimensions of cognition and emotion, providing necessary and rich basis for achieving truly intelligent and humanized interaction response. Specifically, the extraction of the speech rate aims to objectively measure the speed at which the user speaks. The speech rate is defined as the number of syllables contained in a unit of effective speaking time. Its calculation process relies on the cooperation of the speech recognition unit. First, the unit receives two inputs: the original audio stream, and the corresponding text output by the speech recognition unit "that I want to ask today, how is the weather?". Then, using a pre-trained acoustic model, the forced alignment technique is used to accurately match each syllable in the text (e.g., "nà", "ge", "wǒ", "xiǎng", etc.) with the corresponding time segment in the original audio stream. The acoustic model is specifically a composite model based on Gaussian Mixture Model-Hidden Markov Model, and its core architecture is to establish an independent Hidden Markov Model for each basic pronunciation unit in the language (i.e., phonemes). Each Hidden Markov Model is composed of multiple states, which represent the start, middle, and end stages of a phoneme pronunciation process, and the transition probabilities between states accurately describe the dynamic process of the acoustic features evolving over time during pronunciation. In addition, each state is associated with a Gaussian Mixture Model, which calculates the probability of observing a specific acoustic feature vector (such as the Mel-frequency cepstral coefficient extracted from the audio) under the current pronunciation state. The parameters of the acoustic model, including state transition probabilities and the mean, variance, and weight of the Gaussian Mixture Model, are obtained by performing the Expectation-Maximization (EM) algorithm training on a large-scale and accurately phoneme-level time-labeled speech corpus. When performing the forced alignment task, the model takes the original audio stream and the known text as input, and through the Viterbi decoding algorithm, it finds a Hidden Markov Model state sequence path that can generate the observed audio with the maximum posterior probability under the strict constraint of the phoneme sequence converted from the text. This decoded optimal path naturally binds each phoneme in the text, or even each syllable composed of phonemes, with a unique and accurate time frame in the original audio stream.The process generates a start and end timestamp for each syllable. By accumulating the duration of all syllables, the total effective pronunciation duration can be obtained. For example, for the example audio with a total duration of 5 seconds, the total pronunciation duration of all syllables is calculated to be 3.5 seconds after forced alignment. At the same time, the total number of syllables in the text is counted to be 13. Finally, the speech rate is calculated by the formula total number of syllables / effective pronunciation duration, i.e. 13 divided by 3.5, resulting in a speech rate value of approximately 3.7 syllables per second. The extraction of the pause ratio feature is used to quantify the degree of hesitation or thinking time in the user's speech. This process is implemented through a voice activity detection algorithm that distinguishes between speech segments and silent segments in the audio by analyzing underlying acoustic parameters such as short-time energy and zero-crossing rate. In order to precisely define an effective pause, a silence duration threshold is set, which is determined based on statistical analysis of a large amount of elderly speech data, for example 250 milliseconds. During processing, the algorithm scans the entire 5-second audio stream and identifies all silent segments with a duration exceeding 250 milliseconds. In the example audio, there may be a 300-millisecond pause between that and I want, and a 450-millisecond pause between ask and today. The algorithm accumulates the duration of all these effective pauses to obtain a total pause duration of 750 milliseconds, i.e. 0.75 seconds. The pause ratio is the ratio of the total pause duration to the total audio duration, and the calculation result is 0.75 seconds divided by 5 seconds, i.e. 15%. The implementation of the filler word technique is mainly a text-based counting process to identify non-fluent speech markers used by the user during expression. The unit maintains a predefined filler word dictionary inside, which is summarized and organized for common oral language used by the elderly, including words such as that, this, um, ah, and so on. The input of this processing flow is the text generated by the speech recognition unit: "that I want to ask you today, how is the weather?". Through an efficient string matching algorithm, it is checked whether there are filler words defined in the dictionary in the text. In the current example, the algorithm successfully matches the beginning of that and records the number of occurrences. Since it only appears once, the output value of the filler word count is 1. The extraction of the fundamental frequency contour variance aims to capture the amplitude of the user's intonation, which is closely related to emotional state and emphasis of speech. The fundamental frequency, i.e. the pitch of the sound, is the basic frequency of vocal cord vibration. The extraction process first performs frame-by-frame processing on the audio signal through autocorrelation function or cepstrum analysis. For each frame of audio that is judged to be voiced (i.e. the vocal cords have vibration), the fundamental frequency value is calculated. From this, a sequence of fundamental frequency values that changes over time can be obtained, which constitutes the fundamental frequency contour of the speech. Finally, the variance of all fundamental frequency values in this contour sequence is calculated. A higher variance value, for example 300 Hz squared, means that the user's intonation is fluctuating, with a larger amplitude, which may indicate that the user is more emotionally involved or is emphasizing something.Conversely, a lower F0 variance value indicates a more even tone, and a potentially calmer or tired emotion. After all the parallel processing procedures are completed, the acoustic feature extraction unit will aggregate the four raw feature values calculated separately. For this example, the final output is a structured data set containing the names of the features and their corresponding raw numerical values: {speech rate: 3.71 syllables per second, pause ratio: 15%, filler count: 1, F0 contour variance: 300 Hz squared}.

[0025] Finally, these raw acoustic features with different physical units and numerical ranges are sent to the rhythm feature vector combination 113 unit for normalization and integration. The first step is normalization. In order to eliminate the scale difference between different features, the Z-score standardization method is adopted here. The calculation formula of this method is: standardized value = (original value - mean) / standard deviation. The mean and standard deviation in this formula are fixed parameters obtained by comprehensive statistical analysis of a large-scale, high-variety of old people's speech benchmark database in advance. This database contains thousands of speech samples of old people of different ages, genders, regions and health conditions, ensuring the universality and representativeness of the statistical parameters obtained. For example, through statistical analysis, the distribution parameters of each feature are determined in advance as follows: the mean of speech rate is 4.5 syllables per second, and the standard deviation is 0.5; the mean of pause ratio is 10%, and the standard deviation is 4%; the mean of filler count is 0.5, and the standard deviation is 0.8; the mean of fundamental frequency contour variance is 250 Hz square, and the standard deviation is 50. In the processing flow, the unit will substitute each original feature value into the above formula for calculation. For the input data in the example: the standardized value of speech rate is (3.71-4.5) / 0.5, and the calculation result is -1.58. The standardized value of pause ratio is (15-10) / 4, and the calculation result is 1.25. The standardized value of filler count is (1-0.5) / 0.8, and the calculation result is 0.625. The standardized value of fundamental frequency contour variance is (300-250) / 50, and the calculation result is 1.0. In this way, all acoustic features are converted into dimensionless, zero-centered, one-standard-deviation standardized values. The second step is vector combination. After all the feature values are normalized, the unit arranges and splices these standardized values in a predefined and fixed order to form a single, multi-dimensional mathematical vector. This order is the key to ensuring that the subsequent model can correctly interpret the meaning of each dimension of the vector. For example, the order is set as: [speech rate, pause ratio, filler count, fundamental frequency contour variance]. According to this, the four standardized values calculated in the previous step are filled into the corresponding positions of the vector in turn. Finally, the rhythm feature vector combination unit completes its processing and outputs a four-dimensional rhythm feature vector: [-1.58, 1.25, 0.625, 1.0], which not only integrates all acoustic features, but also has comparable dimensions within the vector, which can objectively and comprehensively reflect the user's current language expression rhythm characteristics.

[0026] In particular, the user state encoding module 120 is configured to encode the rhythm feature vector and the dialogue history embedding based on context awareness to obtain a current state vector. Accordingly, although the previous steps successfully parse the user's speech into content text and instant rhythm features, these two kinds of information are still essentially isolated and instantaneous snapshots. The user's real state, especially the deep meaning of his / her intention and needs, does not depend only on the expression of the current sentence, but is closely related to the evolution of the entire dialogue. For example, the same slow speech speed may indicate deep thinking after answering a complex question, but may mean fatigue or uncertainty at the beginning of the dialogue. Without combining historical interaction information, the interpretation of instant rhythm features will be one-sided or even wrong. Therefore, the present application encodes the rhythm feature vector and the dialogue history embedding based on context awareness to effectively integrate the rhythm features of the current utterance with the past dialogue context, and generates a comprehensive, dynamic and context-rich current state vector through a nonlinear encoding method, thereby providing a more profound and accurate decision basis for subsequent strategy generation than a single feature.

[0027] In one possible implementation, the user state encoding module 120 includes a rhythm dialogue vector splicing unit 121 configured to splice the rhythm feature vector and the dialogue history embedding to obtain a combined vector; and a combined vector fusion dimension reduction unit 122 configured to perform nonlinear fusion and dimension reduction on the combined vector according to the following formula to obtain the current state vector:

[0028] Hidden1 = ReLU (W1 Combinedt + b1)

[0029] Currentt = W2 Hidden1 + b2

[0030] wherein W1, b1, b2 and W2 are learnable weight parameters, Combinedt is the combined vector, ReLU is a ReLU activation function, Hidden1 is a hidden combined vector, and Currentt is the current state vector. t t

[0031] ​​In the above implementation, the user state encoding module 120 includes the following: the dialogue history embedding is a fixed-dimensional vector, which aims to compress all the dialogue content before the current round into a semantic representation. The generation process of the embedding is as follows: first, extract the dialogue records of the last N rounds, including the user's speech text and the reply text of the party. The specific size of this N value is a pre-set hyperparameter, and the selection aims to balance the richness of context information and computational efficiency, and the best value is determined through experimental verification. Then, the text sequences are spliced into a long text in chronological order. This long text is input into a pre-trained, large-scale text encoder model. The text encoder model specifically adopts a deep bidirectional model based on the Transformer architecture, and its core is composed of multiple layers of stacked encoder layers. Each encoder layer contains two key sublayers: a multi-head self-attention mechanism layer and a position feedforward network layer. Around each sublayer, residual connection and layer normalization operations are applied to ensure the stability of information flow and the convergence of the training process. The multi-head self-attention mechanism enables the model to simultaneously focus on all other word tokens in the input text when processing each word token, and calculates their mutual association weights, thereby capturing rich context dependency relationships; while the position feedforward network further enhances the model's representation ability by independently transforming the representation of each position output by the self-attention layer. The specific encoding process begins with tokenizing the spliced long text and adding a special classification marker at the start position of the text sequence. Then, an initial input vector that integrates word embedding, position embedding and paragraph embedding is generated for each word token in the sequence. The vector sequence then passes through the entire encoder layer stack from bottom to top. In each layer, information is processed and refined through self-attention mechanism and feedforward network. Finally, the final hidden state vector corresponding to the initial classification marker in the model output layer is taken as the aggregated semantic representation of the entire dialogue history. This fixed-dimensional vector, for example, 256-dimensional, is the dialogue history embedding, which condenses all the context information of the recent dialogue.

[0032] First, the rhythm dialogue vector splicing unit 121 receives the four-dimensional rhythm feature vector and the 256-dimensional dialogue history embedding described above. The operation performed by this unit is feature splicing, that is, concatenating the two vectors end to end to form a longer combined vector. In this example, it appends the rhythm feature vector to the dialogue history embedding to generate a combined vector with a dimension of 260. This combined vector contains both the user's current speaking style, i.e., the rhythm feature, and the background of the conversation, i.e., the dialogue history.

[0033] Next, this combined vector is passed to the combined vector fusion and dimension reduction unit 122. The core of this unit is a small, two-layer feedforward neural network, whose function is to perform nonlinear information fusion and feature compression on the combined vector. Its processing strictly follows the following formula: First, compute the hidden combined vector Hidden1 = ReLU(W1·Combined t + b1). In this formula, W1 is a learnable weight matrix with dimension 64 x 260, and b1 is a learnable bias vector with dimension 64 x 1. Combined t First, perform matrix multiplication with the weight matrix W1 to obtain a 64 x 1 vector. This process can be understood as a weighted sum of the 260 input features, extracting sixty-four new, more abstract features. Then, add the bias vector v1. Finally, pass the result through the ReLU activation function. The ReLU function performs the operation of taking the maximum between zero and each element of the vector, i.e., if the element value is less than zero, output zero, otherwise output the original value. This nonlinear transformation is crucial, as it enables the model to learn complex, nonlinear relationships between the rhythm features and the dialogue history. For example, the model can learn that when the dialogue history indicates that the user is reminiscing, a slower speaking rate (the corresponding dimension in the rhythm feature vector is negative) should be interpreted as focused, rather than tired. Second, compute the final current state vector Current t = W2·Hidden1 + b2. In this formula, W2 is a learnable weight matrix with dimension 32 x 64, and b2 is a learnable bias vector with dimension 32 x 1. The hidden combined vector Hidden1 generated in the first step is matrix multiplied with the weight matrix W2, linearly transforming it from 64 dimensions to 32 dimensions. Then, add the bias vector b2. This step does not use a nonlinear activation function and is a linear dimension reduction and integration process that combines the abstract features extracted by the hidden layer into the final, 32-dimensional current state vector Current t It is worth mentioning that the learnable weight parameters W1, b1, b2, and W2 are learned through end-to-end optimization during the training process of the entire large model system. Finally, the unit outputs a 32-dimensional current state vector, which is a low-dimensional, dense representation that is no longer a simple concatenation of original features, but a highly condensed state code that deeply fuses the user's current expression rhythm and historical dialogue context.

[0034] In particular, in constructing an interaction framework capable of deeply perceiving user states, the core challenge lies in how to effectively fuse two information sources with different natures: one is the rhythm feature vector, which is low-dimensional, sparse, and each dimension has a clear physical meaning; the other is the dialogue history embedding, which is high-dimensional, dense, and represents abstract semantics. If a simple feature concatenation is used and then encoded by a linear layer, a series of technical problems will inevitably arise. First, the high-dimensional semantic embedding is orders of magnitude higher than the low-dimensional rhythm feature. In the optimization process of gradient descent, the model will instinctively prioritize optimizing the weights related to the massive semantic information, causing the rhythm feature, although weak but crucial state signal, to be submerged and its value cannot be fully reflected. Second, the interaction between the rhythm feature and the dialogue history is not a simple linear superposition, but a complex nonlinear modulation relationship. For example, when the semantic background is seeking help and the expression rhythm is hesitant and slow, the system should enter the empathy and pacification mode. This cross-modal multiplicative effect is difficult to capture efficiently through simple linear fusion. Therefore, a more precise modulation and fusion mechanism is needed. This step is to deeply and bidirectionally modulate the rhythm feature vector and the dialogue history embedding before formal encoding, to solve the signal submersion problem, explicitly model the nonlinear interaction between the two, and ultimately generate a more structured and balanced information representation combined vector, laying a solid foundation for the accurate decision-making of the subsequent strategy network.

[0035] In one possible preferred embodiment, the user state encoding module 120 is configured to:

[0036] The rhythm feature vector and the dialogue history embedding are jointly associated projected to obtain a jointly associated projection matrix, i.e.,

[0037]

[0038] wherein V1 is the rhythm feature vector, the dimension is m, T is the transpose operation, V2 is the dialogue history embedding, the dimension is n, is the matrix multiplication, and M1 is the common correlation projection matrix. It should be understood that, in order to preliminarily establish the global and pairwise correlation between the rhythm feature vector and the dialogue history embedding, a basic interactive mapping space is provided for subsequent feature modulation. The present application explicitly captures the potential interaction strength between each dimension of the rhythm feature and each dimension of the semantic embedding by projecting the vectors of two different modalities into a common correlation space. This processing method goes beyond simple linear combination and builds a matrix that can reflect the fine-grained dependency between cross-modal features. The generation of the common correlation projection matrix is achieved by calculating the outer product of the rhythm feature vector and the dialogue history embedding. Each element in the matrix quantitatively represents the correlation degree between a specific rhythm feature (such as speech rate) and a specific semantic dimension in the dialogue history embedding, thereby providing comprehensive and underlying interactive information for subsequent modulation process.

[0039] The key gating response of each feature value of the rhythm feature vector to the dialogue history embedding to obtain a plurality of dialogue history embedding response vectors, and arranging the plurality of dialogue history embedding response vectors into a key gating response matrix, that is:

[0040]

[0041] M2=[V 21 ,V 22 ,…,V 2i ]

[0042] wherein v 1i is each feature value in the rhythm feature vector, is the point multiplication of vectors according to position, exp is the exponential function value with the natural constant e as the base, V 21 ,V 22 ,…,V 2i is each dialogue history embedding response vector in the plurality of dialogue history embedding response vectors, and [V 21 ,V 22 ,…,V 2iis the arrangement of each dialogue history embedding response vector, and M2 is the key gating response matrix. Accordingly, in order to enable the model to explicitly learn and express, according to the current semantic context, to what extent each rhythm feature should adjust or affect the semantic feature. This is a dynamic, content-based gating mechanism that makes rhythm features no longer simple additional information, but a controller that can actively modulate semantic representation. In specific implementation, for the i-th feature value in the rhythm feature vector, an exponential decay function is applied to the entire dialogue history embedding vector to generate a corresponding dialogue history embedding response vector. The smaller the rhythm feature value of the exponential function, the stronger the inhibitory effect on the semantic embedding, and vice versa, thereby achieving a fine nonlinear control. The m n-dimensional dialogue history embedding response vectors calculated for all m rhythm feature values are arranged in order to finally form a key gating response matrix with a dimension of m times n. Each row of the matrix represents the dynamic modulation view of the corresponding rhythm feature on the overall semantic information.

[0043] The gating fusion of the rhythm feature vector and the dialogue history embedding is performed for the common correlation projection matrix and the key gating response matrix respectively to obtain the modulated rhythm feature vector and the modulated dialogue history embedding, that is,

[0044]

[0045] where w1 and w2 are numerical scaling hyperparameters, V1' T is the modulated rhythm feature vector, V2' Tis to modulate the dialogue history embedding. That is, using the interaction information matrix generated in the previous two steps as a dynamic filter or gate, the original rhythm feature vector and dialogue history embedding are weighted and reshaped respectively, so as to generate a new feature representation after deep interaction and mutual adjustment. Through this gating fusion mechanism, it can be ensured that the low-dimensional rhythm feature signal will not be overwhelmed by the high-dimensional semantic features, and its influence can be adaptively adjusted according to the semantic context. In implementation, the generation of the modulated rhythm feature vector is obtained by a series of matrix operations on the original rhythm feature vector by the transpose of the co-association projection matrix and the critical gating response matrix, and scaled by a learnable numerical scaling parameter. Similarly, the generation of the modulated dialogue history embedding is obtained by a similar matrix operation on the original dialogue history embedding by the transpose of the co-association projection matrix and the critical gating response matrix, and scaled by another learnable numerical scaling parameter. The numerical scaling hyperparameter is experimentally determined by performing a systematic hyperparameter search (such as grid search) on an independent validation dataset, with the goal of maximizing the final performance indicator of the downstream policy network. In a specific embodiment, after the hyperparameter optimization process, the numerical scaling hyperparameter w1 used to modulate the rhythm feature vector can be set to 1.2, and the numerical scaling hyperparameter w2 used to modulate the dialogue history embedding can be set to 0.8. These two parallel computing processes enable the two features to be mapped to the same associated representation space and control and fuse information based on the associated mapping of the gating response, thereby generating a more discriminative state representation.

[0046] The modulated rhythm feature vector and the modulated dialogue history embedding are concatenated to obtain a combined vector; the combined vector is nonlinearly fused and dimensionally reduced to obtain a current state vector. Finally, after completing the deep modulation of the original feature vector, the two modulated vectors that have been mutually enhanced and optimized in information structure are concatenated, and through a nonlinear transformation, they are further fused into a final state representation with lower dimension and more dense information. Nonlinear fusion can capture higher-order complex relationships between modulated features, while dimension reduction helps to generate a more robust and more general state vector to meet the input requirements of the downstream policy network. After the pre-modulation step, the generated combined vector has a better structured information decoupling representation, such as the part of the vector that carries the rhythm-modulated semantic information, so that the downstream policy decoding head can more easily learn the specific information it is concerned about. In particular, the implementation process of these two steps is the same as in the above embodiment, and is not described.

[0047] Specifically, the policy vector generation module 130 is configured to input the current state vector into the trained policy network to obtain the policy vector. It should be understood that the current state vector itself is only a mathematical description of how the user is now, and it cannot directly drive the large language model to make adaptive responses. There is a key decision-making gap between understanding the user's state and controlling the model's behavior: how to intelligently and dynamically translate this abstract state representation into a specific set of instructions that can accurately guide the large language model to generate content, style, and depth. Without an effective decision-making center, the early depth perception cannot be converted into final intelligent adaptation. Therefore, a policy generation module is needed to use a well-trained policy network to map the input current state vector to a set of explicit and executable policy parameters, providing a complete and customized action blueprint for subsequent prompt construction and text generation steps, thereby converting the understanding of the user's state into control of the model's behavior.

[0048] In one possible implementation, the policy vector generation module 130 includes a shared representation vector generation unit 131 configured to input the current state vector into a shared hidden layer of the policy network to obtain a shared representation vector, and a vector decoding and assembling unit 132 configured to perform parallel decoding on the shared representation vector and assemble the policy vector to obtain the policy vector, the policy vector including a selected style, temperature, depth, and text instruction.

[0049] In the above implementation, the policy vector generation module 130 includes the following: the policy network is a deep feedforward neural network with an integrated multi-head architecture. The network is composed of a shared input layer, one or more shared hidden layers, and four parallel, task-specific output heads or decoders. The advantage of this architecture is that the network can first learn a general, highly abstract shared representation from the current state vector, which captures all the core information needed for decision-making, and then make specialized, independent decisions for different policy parameters based on this unified representation. In particular, the learnable parameters such as weights and biases of the network are trained in a large-scale simulated interactive environment through deep reinforcement learning. During training, the network acts as an agent that outputs policies based on the input state vector and interacts with a simulated user. The reward signal calculated based on the interaction results (such as the fluency of the conversation, the success rate of the task, the user's satisfaction score, etc.) is used to continuously update the network's parameters through algorithms such as proximal policy optimization, enabling the network to learn how to generate policy combinations that can achieve the highest long-term reward based on different user states.

[0050] The first step is performed by the shared representation generation unit 131. This unit receives the 32-dimensional current state vector output by the user state encoding module. This vector is fed into the shared hidden layer of the policy network. The shared hidden layer can be composed of one or two fully connected layers with a non-linear activation function such as GELU. Specifically, the input 32-dimensional current state vector is first matrix-multiplied with the weight matrix of the first shared hidden layer, e.g., of dimension 128x32, and added with the bias vector, and then passed through the GELU activation function. This process maps the input low-dimensional state representation into a higher-dimensional feature space, e.g., of dimension 128. In this higher-dimensional space, the complex patterns embedded in the original state information are further amplified and decoupled, resulting in a shared representation vector that is richer in information and more conducive to subsequent decision-making.

[0051] The second step is performed by the vector decoding assembly unit 132. This unit receives the 128-dimensional shared representation vector output by the shared representation generation unit and inputs it into four independent decoding heads in parallel to generate the four components of the policy vector, respectively.

[0052] The first decoding head is responsible for generating the selected style. Style refers to the overall tone and communication approach of the reply text, which is a classification task selected from a pre-defined style library. The style library pre-defines multiple styles suitable for interaction with the elderly, such as: concise and clear, patient guidance, empathetic comfort, information checking. This decoding head is a classification head with the structure of a fully connected layer, the number of output neurons equal to the number of styles in the style library, which is 4 in this example, followed by a Softmax activation function. After the 128-dimensional shared representation vector is input into this decoding head, the Softmax function outputs a four-dimensional probability distribution vector, where each element of the vector corresponds to the probability of selecting the four styles: concise and clear, patient guidance, empathetic comfort, and information checking. For example, if the input current state vector indicates that the user's speech speed is slow and slightly paused, but the conversation history shows that it is a regular information query, the shared representation vector may make the output probability of the concise and clear style the highest, e.g., [0.7, 0.2, 0.05, 0.05]. This unit selects the class with the highest probability as the output, so the output value of the selected style is concise and clear.

[0053] The second decoding head is responsible for generating temperature. Temperature is a continuous value parameter that controls the randomness of the large language model when generating text, and is between zero and one. A lower temperature makes the output more deterministic and factual, while a higher temperature makes the output more creative and diverse. This decoding head is a regression head, which is a fully connected layer with only one output neuron, followed by a Sigmoid activation function, and linearly maps the output range to the actual required temperature interval, for example, 0.1 to 1. After the shared representation vector is input into this decoding head, a specific floating-point value is output. In this example, since the user is making a factual query, an accurate answer is required, and the policy network will tend to generate a lower temperature value, for example, 0.3.

[0054] The third decoding head is responsible for generating depth. Depth is an integer parameter that controls the length of the inference chain inside the large language model, which determines how many steps of thinking the model will go through before generating the final reply. For example, a depth of one means a direct answer, and a depth of three means that the model will first generate a two-step internal thinking process, and then generate the final answer based on it. This is a classification task that selects from a preset depth set, for example, {1, 2, 3, 4}. The structure of this decoding head is similar to the style decoding head, which is a fully connected layer with a Softmax activation function, and the number of output neurons is equal to the number of available depth levels. For a simple weather query, the reasoning of depth is unnecessary, so the policy network will most likely choose a depth of one. The output probability distribution of this decoding head may be [0.9, 0.05, 0.03, 0.02], and the final output depth value is one.

[0055] The fourth decoding head is responsible for generating text instructions. Text instructions are a short natural language instruction for the large language model to refer to when generating a reply. Unlike the previous three parameters, its generation is a text generation task. This decoding head can be a small sequence generation model based on a recurrent neural network such as LSTM or a Transformer decoder structure. The shared representation vector is used as the initial hidden state of the decoder. The decoder then generates word tokens one by one in an autoregressive manner until an end token is generated. For example, based on the current state, the policy network determines that this is a direct and explicit request, so the decoding head may generate a text instruction such as: "Directly answer the user's question about the weather, and the language should be concise and clear."

[0056] Finally, the vector decoding assembly unit assembles the outputs generated by the above four decoding heads in parallel, selects the style as concise and clear, the temperature as 0.3, the depth as one, and the text instruction as "Directly answer the user's question about the weather, and the language should be concise and clear", and assembles them into a structured data object, namely {selected style: concise and clear, temperature: 0.3, depth: 1, text instruction: "Directly answer the user's question about the weather, and the language should be concise and clear"}, which is the final policy vector.

[0057] Specifically, the prompt configuration module 140 is configured to construct a final prompt and inference configuration for the speech recognition text based on the policy vector. It can be understood that the preceding policy vector generation module has successfully decoded a highly condensed user state vector into a structured decision scheme containing style, temperature, depth, and text instructions. However, this scheme itself is still a set of abstract, discrete parameters and instruction fragments, which is not the final input that can be directly understood and executed by the large language model. The behavior of the large language model is guided by a carefully constructed, complete text prompt and a set of precise inference configuration parameters. There is a missing link between policy decision and model execution that materializes and formats the policy. Therefore, in the present application, a final prompt and inference configuration for the speech recognition text need to be constructed. This can systematically and automatically convert the abstract policy vector output upstream into two specific, standard inputs that can be directly used by the downstream large language model: one is a final prompt text containing complete context and explicit instructions, and the other is a precise inference configuration that controls the generation process.

[0058] In one possible implementation, Figure 4 A block diagram of the prompt configuration module in the speech large model system based on the language expression characteristics of the elderly according to an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the prompt configuration module 140 includes a style instruction extraction unit 141, a template selection unit 142, a final prompt generation unit 143, and a temperature depth extraction unit 144. Figure 4 The style instruction extraction unit 141 is configured to extract the selected style and text instructions from the policy vector. The template selection unit 142 is configured to extract a selected template adapted to the selected style from a predefined prompt template library according to the selected style. The final prompt generation unit 143 is configured to fill the text instructions and the speech recognition text into the preset placeholders in the selected template to obtain the final prompt. The temperature depth extraction unit 144 is configured to extract the temperature and depth from the policy vector to obtain the inference configuration.

[0059] In the above implementation, the prompt configuration module 140 includes the following: first, the style instruction extraction unit 141 starts working. The function of this unit is to parse the input policy vector and extract two parts directly related to the construction of the text prompt. It reads the selected style field and the text instruction field in the policy vector. According to the input of the present example, the selected style extracted by this unit is the string concise and clear, and the text instruction extracted is the string directly answering the user's question about the weather, and the language should be concise and clear.

[0060] Next, the template selection unit 142 receives the selected style value, i.e., concise and clear, from the style instruction extraction unit. The core task of this unit is to look up the matching in the pre-defined prompt template library according to this style value. It is worth mentioning that the pre-defined prompt template library is a pre-constructed, structured data resource, which can be in the form of a key-value pair storage or a lookup table. The keys of this library are the style names corresponding to the selected style in the strategy vector, such as concise and clear, patient guidance, empathetic pacification, and information checking, while the values are the text template strings adapted to this style, containing specific placeholders. These templates are carefully designed by human-computer interaction experts and prompt engineering experts according to the communication psychology and cognitive characteristics of the elderly users, as well as the behavioral characteristics of large language models. They provide a stable, efficient and stylized framework for reply generation in different communication scenarios. For example, the prompt template library may contain the following entries: Key: concise and clear, Value: "System instruction: {text instruction}\nUser question: {speech recognition text}\nYour answer:". Key: patient guidance, Value: "You are a patient and tactful helper. Your task is: {text instruction}. Please note that the user may need more detailed steps or explanations. The user's question is: {speech recognition text}. Please answer with encouraging and guiding language:". Key: empathetic pacification, Value: "You are a partner full of empathy and good at listening. Your core task is: {text instruction}. Before answering, please show understanding and care for the user's emotions. The user said: {speech recognition text}. Your response should be warm and powerful:". Key: information checking, Value: "You are a meticulous information checker. Your task is: {text instruction}. Before providing the final information, you need to first repeat and confirm the user's request. The user's request is: {speech recognition text}. Please start your checking and answering:". Among them, {text instruction} and {speech recognition text} are pre-set placeholders for subsequent dynamic filling. In this application, take concise and clear as the key, query the template library, and retrieve the corresponding template string. In this example, the unit successfully matches and obtains the template: "System instruction: {text instruction}\nUser question: {speech recognition text}\nYour answer:".

[0061] Subsequently, the final prompt generation unit 143 performs the task of constructing the core. This unit receives three inputs: the selected template provided by the template selection unit, the text instruction provided by the style instruction extraction unit, and the most original speech recognition text. Its workflow is a precise string filling or formatting operation. It will go through all the placeholders in the selected template and replace them with the corresponding actual content. Specifically, it finds the {text instruction} placeholder in the template and replaces it with a direct answer to the user's question about the weather, with the language being concise and to the point. It finds the {speech recognition text} placeholder in the template and replaces it with "That I want to ask is what's the weather like today?". After completing the replacement of all placeholders, a complete and specific final prompt text is generated. For this example, the generated final prompt is: "System instruction: directly answer the user's question about the weather, the language should be concise and to the point. User question: That I want to ask is what's the weather like today? Your answer: " This multi-line string will be used as the first part of the main output of this module.

[0062] At the same time, the temperature depth extraction unit 144 processes the input strategy vector in parallel. The function of this unit is similar to that of the style instruction extraction unit, but its goal is to extract numerical parameters related to the reasoning process configuration. It reads the temperature field and depth field in the strategy vector. According to the input of this example, the unit extracts the temperature value as a floating-point number 0.3 and the depth value as an integer 1. Subsequently, the unit encapsulates these two extracted numerical parameters into a structured reasoning configuration object. For example, it can be an object in the form of a key-value pair: {“temperature”:0.3,“depth”:1}.

[0063] Specifically, the reply text generation module 150 is used to input the final prompt of the speech recognition text and the reasoning configuration into the speech large model to obtain the final reply text. Accordingly, after the layer-by-layer processing of the previous modules, a set of final prompts containing complete context and clear instructions and a set of reasoning configurations precisely quantifying the generation process control parameters are ready. This blueprint is the wisdom of the entire perception and decision-making process. However, the blueprint itself cannot automatically become the final result. It needs a powerful execution engine that can deeply understand every detail of the blueprint and convert it into a high-quality, high-fitting natural language reply. Without a module that can faithfully execute and finely control the generation process, all previous perception and strategy will be in vain. Therefore, a final reply text generation module needs to be established to call and finely control a powerful speech large model, taking the constructed final prompt and reasoning configuration as input, and through a rigorous process that may include internal reasoning enhancement and controlled decoding, finally generating a final reply text that fully meets the strategy expectations and is personalized.

[0064] In one possible implementation, Figure 5 A block diagram of a reply text generation module in a voice large model system based on the language expression characteristics of the elderly according to an embodiment of the present application. As shown, the reply text generation module 150 includes a prompt enhancement unit 151 for generating an enhanced prompt based on internal inference chain generation based on an iterative inference loop and context enhancement for the final prompt based on the depth in the inference configuration, the depth being used to control the number of iterations of the iterative inference loop; and a prompt decoding unit 152 for generating the final reply text based on controlled decoding generation based on the enhanced context for the enhanced prompt based on the temperature in the inference configuration, the temperature being used to modify the probability distribution of each word piece. Figure 5

[0065] ​In the above implementation, the reply text generation module 150 includes the following: first, the prompt enhancement unit 151 receives the final prompt and the reasoning configuration, and immediately extracts the depth parameter therefrom. The function of this unit is to determine, according to the depth value, whether and how to contextually enhance the final prompt. The depth parameter here is used to control an internal reasoning chain generation process based on an iterative reasoning loop, the number of loops of which is determined by the depth value minus one. This process aims to allow the large model to perform one or more steps of self-thinking or drafting before generating the final answer, and to use these thinking processes as additional context to improve the logic, accuracy and depth of the final reply. In the internal logic of this unit, the depth value will be checked first. If the depth value is greater than one, the iterative reasoning loop is started. In order to more clearly illustrate this process, a more complex alternative example is introduced here: for example, the user's input is "I want to go for a walk in the park, but I'm afraid of the rain, what's the weather like in the afternoon, is it suitable to go out?", and the depth value generated by the strategy network is 3. In this case, the iterative reasoning loop will be executed twice, i.e. the depth value 3 minus 1. In the first loop, the unit will append a guiding instruction such as "please first think about the first step:" at the end of the original final prompt, and then input this new, temporary prompt into the speech large model for an internal call. The speech large model here has a core architecture of a large-scale, decoder-only Transformer-based autoregressive language model. The model is composed of an input embedding layer and dozens of stacked decoder layers. Each decoder layer contains a core masked multi-head self-attention mechanism layer and a position feedforward network layer, and is stably coupled through residual connection and layer normalization. The masked self-attention mechanism is the key, which ensures that the model can only focus on all previous word tokens when predicting any word token in the sequence, thereby strictly following the causal order of language generation. When the temporary prompt is input into the model for internal call, its processing process begins with text tokenization, i.e. the prompt string is decomposed into a series of word tokens, and each word token and its position information is converted into a high-dimensional vector through the input embedding layer. The vector sequence then passes through all the decoder layers from bottom to top. In each layer, the masked multi-head self-attention mechanism calculates the association weight between each word token and all previous word tokens in the sequence, thereby constructing a dynamic hidden state representation with a deep understanding of the entire prompt context. After completing the encoding of the entire prompt sequence, the model begins to generate thinking text in a self-recursive manner. It first calculates the probability distribution of all word tokens in the vocabulary as the next word token based on the final hidden state of the last word token through a language model head, i.e. a linear layer plus a Softmax function. Then, a word token is selected from the distribution according to a sampling strategy.The newly generated token is then appended to the end of the input sequence, forming a new, longer sequence, and serving as input for the next step of prediction, and the cycle continues until the model generates a special sequence end token. Through the above process, the large model can generate a preliminary analysis such as: "1. Analyze the weather: need to query the specific weather forecast for the afternoon, including temperature, rainfall probability, wind force. 2. Analyze user intent and concerns: the user's core demand is to walk, mainly worried about the rain." Subsequently, the unit will append the generated thinking content to the back of the original final prompt. In the second cycle, the unit will again append the guiding instructions at the end of the current enhanced prompt, such as "\nPlease make the second step of thinking based on the above analysis:", and call the large model again. The model can continue to generate: "According to the query, the rainfall probability is 20% in the afternoon, the temperature is 24 degrees Celsius, and there is a slight wind. The risk of the user's concern (rain) is low, while the conditions for the core demand (walking) are good. Therefore, it is recommended to go out, but it is suggested to remind the user to pay attention to weather changes." After completing two cycles, the complete text containing the two-step thinking process constitutes the final enhanced prompt.

[0066] However, returning to the example in the present application, the depth value generated by the strategy network is 1. In this case, the prompt enhancement unit checks that the depth value is not greater than 1, so the condition of the iterative reasoning cycle is not met, and the cycle number is zero. The unit will not perform any generation operation of the internal reasoning chain. At this time, the output enhanced prompt of the unit will be exactly the same as the input final prompt. That is, the enhanced prompt is: "System instruction: directly answer the user's question about the weather, the language should be concise and to the point\nUser question: That I want to ask is what is the weather like today?\nYour answer:". This ensures that for simple and direct requests, the processing flow is efficient and without redundancy, avoiding unnecessary overthinking.

[0067] Next, the prompt decoding unit 152 receives the enhanced prompt output by the prompt enhancement unit, as well as the temperature parameter in the inference configuration. This unit is the executor that actually generates the final reply text, and its core is a large-scale autoregressive language model based on a Transformer architecture containing only the decoder part. Its internal structure consists of multiple layers of stacked decoder layers, each of which contains a masked multi-head self-attention mechanism, a cross-attention mechanism (in this scenario, the keys and values of the cross-attention come from the encoding of the prompt text), and a feedforward network. The implementation process of this unit is as follows: First, the enhanced prompt text is input into the language model. The model converts the text into a series of vectors through its embedding layer and encodes the vectors through multiple layers of decoder to understand the semantics and instructions of the entire prompt. After encoding is complete, the model begins to generate word tokens of the final reply text in a self-recursive manner. When generating each new word token, the following steps are performed inside the model: First, based on the sequence of generated word tokens and the understanding of the enhanced prompt, the model outputs a high-dimensional hidden state vector through its last layer of decoder. This vector is then passed through a linear transformation layer and a Softmax function to calculate the probability distribution of all possible word tokens appearing at the current position in the vocabulary. This original probability distribution reflects the natural tendency of the model without any intervention. At this point, the temperature parameter begins to play its key and fine-tuning role. Before applying the Softmax function, the model divides each element in the original vector of logical values output by the linear transformation layer by the temperature value. In this example, the temperature is 0.3. Since dividing by a number less than one amplifies the differences in the original logical values. The logical value of the word token with a higher original probability will become higher, while the logical value of the word token with a lower original probability will become relatively lower. When this adjusted logical value vector is passed through the Softmax function again, the resulting final probability distribution becomes more sharp or steep. This means that the probability of the word token with the highest probability is significantly higher than that of all other word tokens. Subsequently, a sampling algorithm (such as kernel sampling or greedy search) selects the next word token from this temperature-regulated, sharp probability distribution. Due to the sharpness of the distribution, the sampling algorithm will most likely choose the word token with the highest probability. This process causes the model's output to tend to be more deterministic, more factual, and closer to the most common expression in the training data. This is highly consistent with the requirements of concise and clear style and the nature of fact-checking tasks. This cycle of "calculating probability distribution -> temperature regulation -> sampling selecting word token -> appending new word token to sequence" is repeated until the model generates a special sequence end marker or reaches the pre-set maximum generation length. Finally, all sequentially generated word tokens are combined to form a complete and coherent sentence or paragraph.For this example, based on the instructions in the augmented prompt and the low temperature setting of 0.3, the prompt decoding unit can generate the following final reply text: "It is a sunny afternoon today, with a temperature of 25 degrees Celsius and no rain, which is perfect for you to go out and be active." This text string is the final output of the reply text generation module, which not only answers the user's question, but also its style and certainty are precisely controlled by the front-end perception and policy decision.

[0068] Specifically, the reply text conversion module 160 is configured to convert the final reply text into a reply speech signal. That is, the entire intelligent decision and content generation process has been completed, and a personalized reply text that fully meets the user's current state and needs has been produced. However, for elderly users who primarily interact in voice, a pure text output does not complete the entire interactive loop. The user's input is natural speech, and the most expected and barrier-free way of receiving it is also speech. If the interaction ends with cold text, not only will the naturalness and warmth of the experience be reduced, but it may also cause inconvenience to users with visual impairments. Therefore, the final text-to-speech conversion is needed to seamlessly bridge the internally generated, machine-readable text information to the user's audible, anthropomorphic speech signal, to maintain consistency with the initial interaction mode, and thus complete a complete, natural, and humane voice interaction.

[0069] In one possible implementation, the reply text conversion module 160 includes the following: the core function of this module is to perform high-quality text-to-speech (TTS) synthesis. Its input is the final reply text from the reply text generation module. In the preceding example, the input is a string: "It is a sunny afternoon today, with a temperature of 25 degrees Celsius and no rain, which is perfect for you to go out and be active." The final output of this module is a playable, digitized reply speech signal, such as a pulse code modulation (PCM) format audio waveform data.

[0070] The implementation of this module relies on an advanced deep neural network-based speech synthesis model. This model is mainly composed of two cascaded parts: an acoustic model responsible for converting text sequences into acoustic features (such as mel-spectrograms), and a vocoder responsible for converting acoustic features into final audio waveforms.

[0071] The first step is text frontend processing. When the final reply text is input, a text processing frontend will first analyze and normalize it. This includes converting non-standard words in the text into pronounceable forms, such as converting the Arabic numeral 25 and the unit Celsius to the Chinese character twenty-five Celsius. Then, word segmentation and part-of-speech tagging are performed, and the normalized text sequence is further converted into a phoneme sequence, such as converting today into the pinyin sequence "jīntiān". This phoneme sequence is the input representation that the acoustic model can directly understand.

[0072] The second step is acoustic feature prediction. This step is performed by a non-autoregressive acoustic model (such as a FastSpeech 2-based architecture). The input of this model is the phoneme sequence generated in the previous step. It contains an encoder and a decoder based on the Transformer's self-attention mechanism. The phoneme sequence is first passed through the encoder to generate a hidden representation for each phoneme that contains contextual information. The key to this model is the integration of a variance adapter that can explicitly predict multiple variable information in speech. Specifically, the variance adapter receives the output of the encoder and predicts three key acoustic attributes in parallel: duration, which predicts the length of time each phoneme should be pronounced. Pitch, which predicts the fundamental frequency profile of the entire sentence, i.e. the ups and downs of intonation. Energy, which predicts the volume of each phoneme. These predictions make the synthesized speech more natural and vivid in pauses, rhythm and prosody. After the prediction is completed, the hidden representation of the phoneme is expanded according to the predicted duration, and the predicted pitch and energy information are added, and then sent to the decoder. The decoder finally decodes these high-dimensional representations integrated with rich prosodic information into a two-dimensional mel-spectrogram in parallel. Mel-spectrogram is a kind of spectrogram that simulates the hearing characteristics of human ears, which compactly represents the energy distribution of speech in the time-frequency domain.

[0073] The third step is audio waveform synthesis. This process is performed by a neural vocoder based on a generative adversarial network (GAN), such as HiFi-GAN. The input to this vocoder is the mel-spectrogram generated by the acoustic model. It contains a generator and a discriminator inside. The generator is a deep convolutional neural network whose task is to upsample the input mel-spectrogram and gradually restore the high temporal resolution, one-dimensional original audio waveform. The discriminator plays a role in the training phase, and its task is to distinguish between the fake audio synthesized by the generator and the real studio-level true audio. Through this adversarial training, the generator is continuously optimized until its synthesized audio is indistinguishable from the real recording in various acoustic indicators, thus ensuring the high fidelity and naturalness of the final output speech. In particular, regarding the acquisition of model parameters, the weights and biases of the entire speech synthesis model are trained on a large-scale, single-speaker high-fidelity speech database. The recording content of this database is carefully designed to cover a wide range of phoneme combinations and prosodic patterns. In particular, to adapt to the hearing habits of older users, the selected speaker's tone is usually clear, soft, and moderate in speed, and the prediction range of parameters such as duration and pitch can be fine-tuned during the training process to generate speech that is slightly slower in overall speed and clearer in pronunciation.

[0074] Finally, for the input text "Today the weather is sunny, the temperature is 25 degrees Celsius, there is no rain, and it is suitable for you to go out and exercise.", the module will output a corresponding speech signal. This speech signal not only has accurate content, but its tone, speed, and intonation are also optimized, sounding clear, natural, and full of affinity, and can be directly used for playback, thus completing the last step of interaction with the user.

[0075] In summary, the speech large model system 100 based on the language expression characteristics of the elderly is illustrated based on the embodiments of the present application, which constructs a policy network based on deep reinforcement learning as an intelligent decision-making hub connecting the front-end speech perception and the back-end large model inference. Specifically, by capturing and quantifying the rhythm features such as speech rate and pause in the speech stream of the elderly user in real time, and combining the dialogue history, a vector that comprehensively reflects the current state of the user is generated. This state vector is input into the policy network, and the core function of this network is to perform dynamic mapping, intelligently translating low-order acoustic features into high-order policy vectors that can guide the behavior of the large model. The policy vector is used to dynamically construct and configure the prompt instructions and inference parameters (such as temperature and depth) of the large model, so as to accurately regulate the style and detail of its response. In this way, the problem of missing mapping mechanism mentioned in the background technology due to the semantic gap between the front-end physical signal and the back-end abstract control parameter is solved, building a bridge from perception to adaptation, and realizing fine and adaptive regulation of the behavior of the large model.

[0076] Having described various implementations of the disclosure above, it is to be understood that the above descriptions are intended to be illustrative and not exhaustive with respect to disclosed implementations. Also, not all of the implementations described above are necessarily equivalent, and many modifications and variations are possible in light of the above teachings. Thus, it is intended that the scope of the technology be defined by the claims appended hereto rather than the description and examples described above, which were for illustration of implementations only.

Claims

1. A large-scale speech model system based on the perception of language expression characteristics of the elderly, characterized in that, include: The original audio stream rhythm speech analysis module is used to perform speech recognition and multi-dimensional rhythm feature extraction on the acquired original audio stream of the target elderly object to obtain rhythm feature vector and speech recognition text; The user state encoding module is used to perform context-aware user state encoding on rhythm feature vectors and dialogue history embeddings to obtain the current state vector. The policy vector generation module is used to input the current state vector into the trained policy network to obtain the policy vector; The prompt configuration construction module is used to construct the final prompt and inference configuration for speech recognition text based on policy vectors; The response text generation module is used to input the final prompts and inference configurations of the speech recognition text into the speech model to obtain the final response text; The response text conversion module is used to convert the final response text into a response voice signal.

2. The speech model system based on the perception of language expression characteristics of the elderly as described in claim 1, characterized in that, The original audio stream rhythmic speech analysis module includes: A speech recognition unit is used to perform speech recognition on the original audio stream to obtain the speech-recognized text. The acoustic feature extraction unit is used to extract acoustic features from the raw audio stream; The rhythm feature vector combination unit is used to combine the acoustic features into the rhythm feature vector after normalization.

3. The speech model system based on the perception of language expression characteristics of the elderly as described in claim 2, characterized in that, Acoustic features include speech rate, pause ratio, filler word technique, and fundamental frequency profile variance.

4. The speech model system based on the perception of language expression characteristics of the elderly as described in claim 1, characterized in that, The user status coding module includes: The rhythmic dialogue vector concatenation unit is used to concatenate rhythmic feature vectors and dialogue history embeddings to obtain a combined vector. The combined vector fusion and dimensionality reduction unit is used to perform nonlinear fusion and dimensionality reduction on the combined vectors to obtain the current state vector using the following formula: Hidden1=ReLU(W1·Combined t +b1) Current t =W2·Hidden1+b2 Where W1, b1, b2, and W2 are learnable weight parameters, Combined t For the combination vector, ReLU is the ReLU activation function, Hidden1 is the hidden combination vector, and Current is the activation function. t This is the current state vector.

5. The speech model system based on the perception of the language expression characteristics of the elderly as described in claim 1, characterized in that, The strategy vector generation module includes: The shared representation vector generation unit is used to input the current state vector into the shared hidden layer of the policy network to obtain the shared representation vector; The vector decoding and assembly unit is used to perform parallel decoding and policy vector assembly on the shared representation vector to obtain the policy vector, which includes selected style, temperature, depth and text instructions.

6. The speech model system based on the perception of the language expression characteristics of the elderly as described in claim 5, characterized in that, The prompt configuration construction module includes: The style instruction extraction unit is used to extract selected style and text instructions from the strategy vector; The template selection unit is used to extract a selected template that matches the selected style from a predefined prompt template library. The final prompt generation unit is used to fill the text command and the speech recognition text into the preset placeholders in the selected template to obtain the final prompt; A temperature and depth extraction unit is used to extract temperature and depth from the policy vector to obtain the inference configuration.

7. The speech model system based on the perception of the language expression characteristics of the elderly according to claim 6, characterized in that, The reply text generation module includes: The prompt enhancement unit is used to generate an internal inference chain and enhance the context based on the depth in the inference configuration to obtain an enhanced prompt, wherein the depth is used to control the number of iterative inference loops; The prompt decoding unit is used to perform controlled decoding of the enhanced prompt based on the enhanced context to generate the final response text, based on the temperature in the inference configuration, wherein the temperature is used to modify the probability distribution of each word.

Images