Child language cognitive rehabilitation auxiliary system and method based on adaptive co-creation mechanism

Abstract

The application discloses a kind of children language cognitive rehabilitation auxiliary system and method based on adaptive co-creation mechanism, belong to medical artificial intelligence and psychological rehabilitation engineering field.The system includes five big core modules, and is connected in the form of data stream closed loop between modules, forms sustainable optimization interactive system, which is by fusing the semantic vector of children voice, emotion vector and task vector, via dynamic weight generation network and cross-modal attention mechanism to construct fusion state vector, and using attention enhanced gated recurrent unit and strategy network to generate co-creation control instruction, and then synthesis interactive voice signal with emotional expressiveness.The application can realize the synchronous fusion and linkage feedback of semantic understanding and emotion recognition, dynamically adjust interactive strategy according to the real-time state of children, and optimize strategy network parameters through online self-learning, improve the naturalness, coherence and individual adaptability of rehabilitation interaction.

Description

A Child Language and Cognitive Rehabilitation Assistance System and Method Based on Adaptive Co-creation Mechanism Technical Field

[0001] This invention belongs to the field of medical artificial intelligence and psychological rehabilitation engineering, and in particular relates to a language and cognitive rehabilitation assistance system and method for children based on an adaptive co-creation mechanism. Background Technology

[0002] Child psychology and cognitive impairment rehabilitation is an important area of ​​interdisciplinary research in medicine, psychology, and artificial intelligence. For typical issues such as autism spectrum disorder (ASD), language delay, and emotion regulation disorders, traditional intervention methods mainly rely on therapists providing one-on-one guidance through language training, storytelling, and cognitive behavioral therapy to help children improve their language expression, emotion recognition, and social interaction skills.

[0003] With the development of artificial intelligence technology, researchers have begun to introduce natural language processing, speech recognition, affective computing, and human-computer interaction technologies into the field of pediatric rehabilitation to assist therapists in completing language training and emotional guidance tasks. In particular, the emergence of large language models has enabled systems to possess strong semantic understanding and natural language generation capabilities, demonstrating potential value in scenarios such as education, psychological intervention, and cognitive rehabilitation.

[0004] However, existing pediatric rehabilitation systems still have many limitations in practical applications, including insufficient emotional understanding, inadequate adaptation to individual differences, insufficient depth of interaction, and difficulty in quantifying rehabilitation effects. These shortcomings directly affect the continuity and effectiveness of interventions and also limit the clinical translation and promotion of artificial intelligence in the field of rehabilitation.

[0005] Existing solutions mainly fall into the following technical categories. One category is rehabilitation systems based on templates and fixed dialogue scripts. These systems rely on manually designed scenarios and statements, lacking true semantic understanding. Another category is intelligent rehabilitation training systems based on speech recognition and sentiment analysis. These systems use acoustic features to identify emotions and provide fixed feedback, but they are mostly at a shallow recognition stage and struggle to dynamically adjust to context. A third category is evaluation models based on machine learning. These utilize deep neural networks to improve the accuracy of specific tasks (such as pronunciation correction and emotion classification), but they are often limited to single functions and lack cross-modal fusion and content generation capabilities.

[0006] Current AI-assisted rehabilitation systems still have significant shortcomings. First, existing systems lack personalized generation capabilities, with most operating based on templates or fixed logic. This leads to a mechanical and repetitive training process, causing children's interest to wane and intervention compliance to suffer. Second, the accuracy of emotion recognition is limited, making it difficult to capture complex psychological states such as anxiety, loneliness, and shyness, and lacking effective reassurance mechanisms after recognition. Third, existing systems lack sufficient interaction depth, lacking multi-turn semantic understanding mechanisms, and are unable to cope with children's ambiguous and varied expressions to maintain natural dialogue. Finally, rehabilitation assessment systems lack scientific rigor and continuity, relying heavily on simple indicators and failing to develop reliable long-term rehabilitation curves from multiple dimensions such as language complexity and emotional changes. These deficiencies mainly stem from the failure to fully utilize the semantic capabilities of large models and the lack of comprehensive multimodal information processing mechanisms. Summary of the Invention

[0007] The technical problem this invention aims to solve is that existing pediatric psychological and cognitive rehabilitation systems suffer from low accuracy, insufficient interaction depth, and poor individualized adaptability in language comprehension, emotion recognition, and intervention feedback. Existing methods generally rely on fixed scripts or manually set training templates, failing to dynamically generate intervention content based on children's immediate emotional states and language expressions. Furthermore, they lack a continuous rehabilitation assessment mechanism, resulting in a mechanical rehabilitation process, delayed feedback, and difficulty in quantifying effects. Therefore, this invention provides a pediatric language and cognitive rehabilitation auxiliary system and method based on an adaptive co-creation mechanism, applicable to clinical psychological intervention, rehabilitation training, and auxiliary educational scenarios.

[0008] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0009] In a first aspect, the present invention provides a language and cognitive rehabilitation assistance system for children based on an adaptive co-creation mechanism, comprising:

[0010] The state vector acquisition module is used to acquire the child's original speech signal in each round of interaction in real time. The speech recognition unit converts the original speech signal into a text information sequence and extracts the semantic vector from it. The emotion recognition unit obtains the emotion vector from the original speech signal. The rehabilitation task management module generates the task vector before the interaction begins. Thus, the semantic vector, emotion vector and task vector together constitute a multimodal vector.

[0011] The multimodal fusion module performs linear mapping on multimodal vectors to obtain multimodal features. A dynamic weight generation network generates a set of dynamic weights, which are then used to weight the multimodal features to generate initial fusion features. Based on the multimodal features, a cross-modal attention weight matrix and a similarity matrix are calculated. The two matrices are multiplied element-wise and normalized to obtain a composite fusion weight matrix. The composite fusion weight matrix is ​​then used to weight the initial fusion features element-wise to obtain a fusion state vector.

[0012] The temporal policy decision module is used to weight and aggregate the fused state vector through an attention-enhanced gated recurrent unit to generate the temporal policy hidden state and input it into the policy network to generate co-creation control instructions.

[0013] The speech generation and emotion rendering module is used to parse the co-creation control commands to generate target text, call the speech synthesis engine to convert the target text into an output speech signal, and inject emotional expression into the speech signal generation process to finally obtain an interactive speech signal for interaction with children.

[0014] The policy self-learning module is used to evaluate the system operation process and update parameters after each round of interaction, so as to achieve continuous learning and performance improvement of the policy network.

[0015] Based on the above scheme, each module can be implemented in the following preferred manner.

[0016] As a preferred embodiment of the first aspect mentioned above, in the state vector acquisition module, a pre-trained Chinese natural language processing model is used to perform deep semantic encoding on the text information sequence to obtain a semantic vector; the emotion vector is obtained by dimensionality reduction mapping of statistical features through a multilayer perceptron, wherein the statistical features are Mel-frequency cepstral coefficients and prosodic features obtained from the original speech signal; the rehabilitation task management module retrieves three types of task information from the preset rehabilitation course map based on the current interaction progress pointer: the current task type number, the task target difficulty level, and the task completion status, and converts the task information into a numerical vector representation through a lookup table method or one-hot encoding to obtain the task vector.

[0017] As a preferred embodiment of the first aspect mentioned above, in the multimodal fusion module, the semantic vector, emotion vector, and task vector each pass through a linear mapping layer to obtain semantic features, emotion features, and task features that constitute multimodal features. The dynamic weight generation network takes the context vector constructed from various real-time indicators as input, and after processing by a multilayer perceptron, the output contains a set of dynamic weights, with each weight corresponding to a feature. Among them, the real-time indicators include: speech recognition confidence, emotion change trend, speech rate stability, and task stage.

[0018] As a preferred embodiment of the first aspect above, the cross-modal attention weight matrix uses semantic features as the query vector, and concatenates emotion features and task features to serve as both the key vector and the value vector, and obtains them through attention calculation; the similarity between semantic features, emotion features, and task features is calculated to obtain a similarity matrix.

[0019] As a preferred embodiment of the first aspect, in the time-series policy decision-making module, the policy network adopts a multilayer perceptron composed of fully connected layers, and the generated co-creation control instructions consist of three control components: policy control component, language control component, and emotion control component.

[0020] As a preferred embodiment of the first aspect mentioned above, in the speech generation and emotion rendering module, firstly, based on natural language generation technology, target text is generated according to the interaction behavior type in the policy control component and the language control component; secondly, the speech synthesis engine is invoked to convert the target text into an outputtable speech signal according to the tone style parameter in the language control component and the output rhythm parameter in the emotion control component; then, the emotion rendering intensity parameter in the emotion control component is input into the parameter mapping network to generate a high-dimensional emotion feature vector; finally, the high-dimensional emotion feature vector is injected into the acoustic parameter space, mapped to three prosodic parameters—pitch, energy level, and formant distribution—in the vocoder stage and added to the speech signal, outputting an interactive speech signal synthesized by the vocoder.

[0021] As a preferred option in the first aspect mentioned above, in the strategy self-learning module, the first... Round-robin strategy network parameters After interacting with the children, by the first The overall rehabilitation return is calculated by weighting and summing the task completion score, interaction fluency, and emotional improvement in the round-robin interaction. And as a result of the expected comprehensive rehabilitation benefits The unbiased estimate is derived from the sentiment vector. and high-dimensional emotion feature vector Calculate the sentiment consistency index The comprehensive rehabilitation reward is added to the weighted emotional consistency index to obtain the target value, and the gradient of the policy network parameters with respect to the target value is calculated. The preset learning rate is then applied. Multiplying the calculated gradient by the parameter adjustment amount yields the parameter adjustment amount, which is then used to adjust the first parameter. Round-robin strategy network parameters The updated policy network parameters are obtained by adding the adjustment amount.

[0022] Furthermore, the updated policy network parameters The calculation method is as follows:

[0023]

[0024] in, This indicates an assignment operation; This represents the gradient operator; The weighting coefficient for the emotional consistency index.

[0025] As a preferred embodiment of the first aspect mentioned above, the task completion score For the first The ratio of the number of sub-task objectives successfully triggered by the child during the interactive round to the preset total number of task objectives; the smoothness of the interaction. The data incorporates response latency and interruption counts obtained from interaction logs; the stated mood improvement level... The emotional consistency index is the difference between the emotional valence at the end of the interaction and the emotional valence at the beginning of the interaction. The calculation method of the emotional consistency index is as follows: first calculate the dot product of the emotional vector and the high-dimensional emotional feature vector, then calculate the product of the magnitude of the emotional vector and the magnitude of the high-dimensional emotional feature vector, and divide the dot product result by the product result.

[0026] Furthermore, the specific calculation methods for task completion score, interaction fluency, and emotional consistency index are as follows:

[0027]

[0028]

[0029]

[0030] in, For the first The number of sub-task objectives successfully triggered by the child during the round-robin interaction; For the first The total number of preset task objectives in the wheel interaction; This indicates the response latency obtained from the interaction log; This indicates the number of interruptions obtained from the interaction log; and These are the preset first tolerance threshold and the second tolerance threshold, respectively; This represents the magnitude of the vector.

[0031] Secondly, the present invention provides an interaction method based on an adaptive co-creation mechanism, which includes the following steps:

[0032] S1. Acquire the child's real-time voice signal during the current interaction, and input the real-time voice signal into the child language cognitive rehabilitation assistance system based on the adaptive co-creation mechanism described in any of the above-mentioned first aspect schemes. The state vector acquisition module generates a new multimodal vector based on the child's voice signal, the multimodal fusion module generates a new fusion state vector based on the new multimodal vector, the temporal strategy decision module generates a new co-creation control command based on the new fusion state vector, the voice generation and emotion rendering module generates a new interactive voice signal based on the new co-creation control command, and plays the new interactive voice signal to the child through an audio output device to conduct voice dialogue interaction.

[0033] S2. After the current interaction ends, the policy self-learning module evaluates the operation process of the children's language cognitive rehabilitation assistance system, updates the policy network parameters, and returns to S1 to execute the next round of interaction until the preset rehabilitation task is completed or the maximum number of interactions is reached.

[0034] In a third aspect, the present invention provides a computer electronic device, which includes a memory and a processor;

[0035] The memory is used to store a computer program;

[0036] The processor is used to implement the children's language cognitive rehabilitation assistance system based on the adaptive co-creation mechanism according to any one of the above first aspect solutions when executing the computer program.

[0037] The present invention has the following beneficial effects compared with the prior art:

[0038] Through the dynamic weight fusion structure, the children's language cognitive rehabilitation assistance system constructs a unified representation space among semantic understanding, emotion recognition and task orientation, enabling the speech generation process to simultaneously respond to language logic and emotional changes. Compared with the traditional rehabilitation assistance system based only on keywords or fixed templates, the present invention can achieve synchronous understanding and linkage feedback in terms of semantics and emotions, making the interaction more natural and the response closer to the state of children.

[0039] The present invention adopts an emotion-modulated adaptive strategy generation mechanism, which is specifically composed of the feature dynamic weighting algorithm of the "multi-modal fusion module" and the attention-enhanced gated recurrent unit and policy network of the "temporal strategy decision module", enabling the decision-making process to dynamically adjust the interaction strategy according to the semantic content, acoustic features and real-time emotional state of the children's speech. This mechanism generates a co-creation control instruction for guiding the speech feedback content and expression mode by integrating semantic information, emotion signals and task goals, realizing the synchronous output of semantic coherence and emotion coordination.

[0040] In addition, the children's language cognitive rehabilitation assistance system can dynamically adjust the policy parameters according to the subtle changes in speech emotions, enabling synchronous semantic generation, tone control and content feedback. Compared with the static system relying on preset processes, the present invention can instantaneously update the interaction strategy during the interaction process, significantly improving the fluency, continuity and emotion guidance effect of the dialogue.

[0041] Furthermore, by introducing a parameter update method based on comprehensive rehabilitation rewards and emotional consistency indicators, this invention enables the children's language and cognitive rehabilitation assistance system to continuously optimize its strategy network through multiple rounds of interaction, gradually adapting it to the language and emotional characteristics of different children. Traditional rehabilitation assistance systems often rely on offline training and manual annotation, resulting in poor adaptability and difficulty in updating. In contrast, this invention can self-correct and optimize during operation, maintaining long-term stability and individual adaptability. Attached Figure Description

[0042] Figure 1 is a flowchart of the core algorithm of the children's language and cognitive rehabilitation assistive system of the present invention;

[0043] Figure 2 is a schematic diagram of the module composition of the children's language and cognitive rehabilitation auxiliary system of the present invention;

[0044] Figure 3 is a schematic diagram of the strategy network parameter update of the present invention;

[0045] Figure 4 is a schematic diagram showing the comparison of children's pre-test and post-test scores in this embodiment;

[0046] Figure 5 is a flowchart of the interaction method of the present invention;

[0047] Figure 6 is a schematic diagram of the composition of a computer electronic device according to the present invention. Detailed Implementation

[0048] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0049] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0050] As shown in Figure 1, in a preferred embodiment of the present invention, the above-mentioned children's language and cognitive rehabilitation assistance system based on an adaptive co-creation mechanism includes:

[0051] The state vector acquisition module is used to acquire the child's original speech signal in each round of interaction in real time. The speech recognition unit converts the original speech signal into a text information sequence and extracts the semantic vector from it. The emotion recognition unit obtains the emotion vector from the original speech signal. The rehabilitation task management module generates the task vector before the interaction begins. Thus, the semantic vector, emotion vector and task vector together constitute a multimodal vector.

[0052] The multimodal fusion module performs linear mapping on multimodal vectors to obtain multimodal features. A dynamic weight generation network generates a set of dynamic weights, which are then used to weight the multimodal features to generate initial fusion features. Based on the multimodal features, a cross-modal attention weight matrix and a similarity matrix are calculated. The two matrices are multiplied element-wise and normalized to obtain a composite fusion weight matrix. The composite fusion weight matrix is ​​then used to weight the initial fusion features element-wise to obtain a fusion state vector.

[0053] The temporal policy decision module is used to weight and aggregate the fused state vector through an attention-enhanced gated recurrent unit to generate the temporal policy hidden state and input it into the policy network to generate co-creation control instructions.

[0054] The speech generation and emotion rendering module is used to parse the co-creation control commands to generate target text, call the speech synthesis engine to convert the target text into an output speech signal, and inject emotional expression into the speech signal generation process to finally obtain an interactive speech signal for interaction with children.

[0055] The policy self-learning module is used to evaluate the system operation process and update parameters after each round of interaction, so as to achieve continuous learning and performance improvement of the policy network.

[0056] It should be noted that in the state vector acquisition module of this invention, the text information sequence is subjected to deep semantic encoding by a pre-trained Chinese natural language processing model to obtain a semantic vector; the emotion vector is obtained by dimensionality reduction mapping of statistical features through a multilayer perceptron, wherein the statistical features are Mel frequency cepstral coefficients and prosodic features obtained from the original speech signal; the rehabilitation task management module retrieves three types of task information from the preset rehabilitation course map based on the current interaction progress pointer: the current task type number, the task target difficulty level, and the task completion status, and converts the task information into a numerical vector representation by a lookup table method or one-hot encoding to obtain the task vector.

[0057] It should be noted that in the multimodal fusion module of this invention, the semantic vector, emotion vector, and task vector each pass through a linear mapping layer to obtain semantic features, emotion features, and task features that constitute multimodal features. The dynamic weight generation network takes the context vector constructed from various real-time indicators as input, and after processing by a multilayer perceptron, the output contains a set of dynamic weights, with each weight corresponding to a feature. Among them, the real-time indicators include: speech recognition confidence, emotion change trend, speech rate stability, and task stage.

[0058] It should be noted that the cross-modal attention weight matrix uses semantic features as the query vector, and concatenates emotion features and task features to serve as both the key vector and the value vector, and is obtained through attention calculation; the similarity between semantic features, emotion features, and task features is calculated to obtain a similarity matrix.

[0059] It should be noted that in the temporal strategy decision-making module of the present invention, the strategy network adopts a multilayer perceptron composed of fully connected layers, and the generated co-creation control instructions consist of three control components: strategy control component, language control component, and emotion control component.

[0060] It should be noted that in the speech generation and emotion rendering module, firstly, based on natural language generation technology, target text is generated according to the interaction behavior type in the policy control component and the language control component; secondly, the speech synthesis engine is called to convert the target text into an output speech signal according to the tone style parameter in the language control component and the output rhythm parameter in the emotion control component; then, the emotion rendering intensity parameter in the emotion control component is input into the parameter mapping network to generate a high-dimensional emotion feature vector; finally, the high-dimensional emotion feature vector is injected into the acoustic parameter space, mapped to three prosodic parameters—pitch, energy level, and formant distribution—in the vocoder stage and added to the speech signal, outputting an interactive speech signal synthesized by the vocoder.

[0061] It should be noted that in the strategy self-learning module of this invention, the first... After the strategy network parameters of the round-robin interaction are interacted with by the child, the first... The task completion score, interaction fluency, and emotional improvement in each round of interaction are weighted and summed to calculate the comprehensive rehabilitation reward, which serves as an unbiased estimate of the expected comprehensive rehabilitation reward. An emotion consistency index is calculated using the emotion vector and high-dimensional emotion feature vector. The comprehensive rehabilitation reward is then added to the weighted emotion consistency index to obtain the target value. The gradient of the policy network parameters with respect to this target value is calculated. The preset learning rate is multiplied by the calculated gradient to obtain the parameter adjustment amount. The updated policy network parameters are obtained by adding the adjusted amount to the policy network parameters obtained from the round-trip interaction.

[0062] Furthermore, the task completion score is the [number]th [unit]. The ratio of the number of sub-task objectives successfully triggered by the child during the round of interaction to the preset total number of task objectives; the interaction smoothness incorporates response delays and interruption counts obtained from the interaction log; the emotion improvement is the difference between the emotional valence at the end of the interaction and the emotional valence at the beginning of the interaction.

[0063] Furthermore, the calculation method of the emotion consistency index is as follows: first calculate the dot product of the emotion vector and the high-dimensional emotion feature vector, then calculate the product of the magnitude of the emotion vector and the magnitude of the high-dimensional emotion feature vector, and divide the dot product result by the product result.

[0064] This invention constructs a three-dimensional fusion loop of "semantics-emotion-task" by jointly modeling the semantic features, emotional features, and rehabilitation goal information contained in children's original speech signals, thereby achieving the coordinated operation of language understanding, emotion recognition, and task regulation. As shown in Figure 2, the core modules of this invention mainly include a state vector acquisition module, a multimodal fusion module, a temporal strategy decision-making module, a speech generation and emotion rendering module, and a strategy self-learning module. Each module is executed sequentially according to the data flow order to achieve semantic understanding, emotion recognition, rehabilitation goal alignment, and dynamic interactive strategy generation of children's speech. The specific processing procedures in each module are described in detail below.

[0065] In this embodiment, the state vector acquisition module receives the child's raw speech signal as input. The raw speech signal is then processed in parallel by two units: the speech recognition unit first uses an automatic speech recognition engine based on a deep neural network (an end-to-end RNN-T architecture model) to convert the input raw speech signal into a text information sequence. This text information sequence is then input into a pre-trained Chinese natural language processing model (BERT-base-Chinese) for deep semantic encoding to obtain the first... A semantic vector with fixed dimensions at each time step. The emotion recognition unit performs a standard acoustic feature extraction process on the same original speech signal, specifically including: signal pre-emphasis, frame-by-frame windowing, Fast Fourier Transform (FFT), and Mel filter bank filtering. Finally, it obtains Mel frequency cepstral coefficients (MFCCs) through Discrete Cosine Transform (DCT). Simultaneously, it extracts prosodic features such as fundamental frequency (F0), short-time energy, and syllable duration. The Mel frequency cepstral coefficients and prosodic features are used as statistical features. These statistical features are then dimensionality-reduced and mapped by a multilayer perceptron (MLP) to generate an emotion vector. .

[0066] Furthermore, to ensure consistency in the temporal granularity of the two modalities, in this embodiment, semantic features and emotional features are aligned at the sentence-level time unit and unified into a time step.

[0067] In addition, a task vector is formed based on the currently set rehabilitation task objectives or sub-task objectives. Specifically, the rehabilitation task management module is a state tracker based on existing task graph technology. Internally, it stores a preset rehabilitation course graph and task logic table. Before the interaction begins, the module retrieves the current task type number, task difficulty level, and task completion status from the rehabilitation course graph based on the current interaction progress pointer. It then converts these into a numerical vector representation by querying the task logic table or using one-hot encoding, thus accurately representing the rehabilitation goal configuration for this round of interaction—that is, the rehabilitation task goal or sub-task goal for this round of interaction. Finally, the semantic vector... Emotion Vector Task vector Together constitute the first Multimodal vectors at each time step .

[0068] In this embodiment, the three state vectors representing different dimensions (i.e., the semantic vector, emotion vector, and task vector) are input into the multimodal fusion module. This module performs multi-dimensional joint modeling, specifically using three fully connected networks with shared parameters as linear mapping layers. Each linear mapping layer processes one state vector, thereby mapping the semantic vector, emotion vector, and task vector to a unified dimensional space to obtain semantic features. Emotional characteristics With task characteristics To ensure consistency in subsequent weighted fusion calculations, a dynamic weight generation network is introduced in the multimodal fusion module to adaptively adjust the importance of different modalities based on the child's real-time state. This network receives a context vector constructed from real-time metrics generated during acoustic feature extraction (such as speech recognition confidence, emotion change trends, speech rate stability, and task stage) as input. Its internal multilayer perceptron (MLP) processes this context vector, and the output contains a set of dynamic weights, namely the first weight. Second weight Third weight Each weight corresponds to a feature of a modality. Each weight is then normalized using softmax, so that... This ensures the scale stability of the initial fused features. Therefore, the three weights mentioned above are used to weight the semantic features. Emotional characteristics With task characteristics Weighting is performed to obtain initial fusion features. It serves only as a baseline feature for the linear weighting stage:

[0069]

[0070] Furthermore, the three weights mentioned above are adaptively adjusted. Based on the dynamic changes in weights, the system can automatically amplify the influence of the corresponding modality when a child experiences emotional fluctuations, decreased attention, or changes in speech fluency. For example, when a decreased emotion or slower speech rate is detected, the dynamic weight generation network automatically reduces the output values ​​of the speech recognition confidence-related weights. After Softmax normalization, this forces the weights representing the emotional modality to... The relative increase enhances the proportion of emotional information in the initial fusion features; when the semantic vector is clearer and contains words related to the target, it improves... and This enhances the expression of semantic and task information. Through the aforementioned adaptive dynamic weighting, the system can automatically rely on emotion or task context when semantics are unclear, or increase the weight of emotional modalities when emotions fluctuate, thus achieving complementary enhancement between modalities.

[0071] To further enhance modal coupling capabilities, the multimodal fusion module also introduces a lightweight cross-modal correlation modeling mechanism. This mechanism includes two sub-processes: attention-based cross-modal association modeling and similarity-based lightweight coupling modeling. First, an attention-based cross-modal association modeling method is used, with semantic features... As a query vector, using sentiment features and task characteristics The concatenated vectors are used as key / value pairs to calculate the cross-modal attention weight matrix. This matrix serves as a preliminary modality association factor, used to characterize the interactive dependencies between semantic, emotional, and task modalities. Building upon this, this embodiment further performs lightweight coupling modeling based on similarity. By calculating the cosine similarity or dot product similarity between different modalities, the cross-modal attention weight matrix is ​​subjected to secondary adjustment and weighted smoothing to obtain a composite fusion weight matrix. ;in, The similarity matrix is ​​calculated based on three modalities: semantic, emotional, and task-related features. This indicates an element-wise multiplication operation. This represents the normalization function.

[0072] The composite fusion weight matrix obtained through the above method can adaptively enhance feature information with high semantic-emotion consistency or semantic-task relevance, thereby improving the stability and effectiveness of multimodal fusion results. In this way, when semantic and emotional consistency is high, the weight of the emotional modality is increased; when semantic and task relevance is high, the weight of the task modality is strengthened. This fusion structure achieves lightweighting through parameter sharing and low-rank decomposition, significantly reducing computational complexity. After the above two-level modeling, the system can identify implicit relationships such as "the impact of emotional changes on the reliability of semantic expression" and "the constraint effect of the task on semantic interpretation."

[0073] After obtaining the composite fusion weight matrix, the system combines it with the aforementioned initial fusion features. Perform element-wise weighted operations to generate a more stable, information-rich, and context-aware fusion state vector. ,symbol This indicates element-wise weighting.

[0074] It should be noted that in this invention, the "Attention-Enhanced GRU" used in the temporal policy decision module is an existing network structure used to weighted aggregate the fused state vector, thereby highlighting key information while preserving temporal dependencies. Specifically, the fused state vector is first input into the self-attention layer of the Attention-Enhanced GRU, and the correlation between the fused state vector and historical interaction trajectories is calculated, thereby outputting a reference vector containing historical context information. Then, the fused state vector is concatenated with the reference vector to form a comprehensive feature, which is then input into the gated recurrent unit for state update. The hidden state output at each time step is used as the hidden state of the timing policy. :

[0075]

[0076] Among them, the historical interaction trajectory is from the previous The sequence of hidden states formed by the temporal strategy at each time step. The sequence is pre-stored in memory; Indicates a splicing operation; In other words, the above comprehensive characteristics can be regarded as the first The fusion state vector and reference vector of each time step; That is, the first The hidden state output at each time step; This represents a gated loop unit, whose internal memory mechanism automatically saves historical semantics, emotions, and task context information. The system makes a comprehensive judgment based on the combined features of the above-mentioned comprehensive features and the joint features of historical interaction trajectories to ensure that the strategy reasoning process simultaneously satisfies semantic continuity, emotional consistency, and goal orientation.

[0077] After comprehensively judging the integrated features and historical interaction trajectories, the aforementioned temporal policy hidden states are further input into the policy network. The multi-parameter output layer, which is placed in the first... The vector output at each time step serves as the co-creation control command. This system is used to achieve dynamic control and real-time strategy adjustment in continuous interaction processes. In this embodiment, the co-creation control command generated in each round of interaction consists of three control components: a strategy control component, a language control component, and an emotion control component. These three control components are different dimensions of the co-creation control command. Among them, the strategy control component is used to determine the type of interaction behavior (such as encouragement and guidance, error correction prompts, emotional soothing, task advancement, etc.); the language control component is used to adjust the semantic depth, lexical level, and tone style of the generated target text; and the emotion control component is used to adjust the emotional rendering intensity, output rhythm, and prompting method of the interactive voice signal.

[0078] The aforementioned co-created control instructions serve as input to the speech generation and emotion rendering module, which comprises two parts: speech generation and emotion rendering. In speech generation, firstly, based on existing Natural Language Generation (NLG) technology, target text is generated according to the interaction behavior type in the policy control component and the language control component. Secondly, the co-created control instructions are parsed to obtain the tone style parameters in the language control component and the output rhythm parameters in the emotion control component. Then, a text-to-speech (TTS) engine is invoked to convert the target text into an outputtable speech signal. In emotion rendering, firstly, the co-created control instructions are parsed to obtain the emotion rendering intensity parameters, which are then used as input. A fully connected pre-trained parameter mapping network is used to perform nonlinear transformation and dimensionality expansion on these parameters, thereby generating a high-dimensional emotion feature vector. This mapping process transforms a single-dimensional intensity value into a feature distribution containing multi-dimensional acoustic details, enabling continuous and nuanced control over the emotional intensity and prosodic details of the output speech, rather than simply switching discrete emotion tags. Secondly, the high-dimensional emotion feature vector is injected into the acoustic parameter space and mapped to prosodic parameters such as pitch, energy level, and formant distribution during the vocoder stage, thereby outputting an interactive speech signal synthesized by the vocoder. The signal integrates semantic content, tone and emotional expression during its generation process, and matches it with the child's real-time semantic needs, emotional state and rehabilitation goals.

[0079] It should be emphasized that the "language control component" mentioned above is the first part of the policy network in the [missing information]. The co-creation control instructions output at each time step are a direct component of the policy network. The policy network receives the temporal policy latent state, which contains the child's current multimodal characteristics and historical interaction trajectory, as input. After nonlinear mapping calculations in its internal fully connected layers, it generates a multidimensional control vector, including a language control component, in real time, which is the co-creation control instruction. Therefore, the language control component is essentially a dynamic indicator calculated in real time by the system based on the child's current cognitive load (such as response delay, emotional fluctuations, etc.), rather than a fixed value preset by the user.

[0080] To enable the policy network to possess long-term adaptive and optimization capabilities, this invention incorporates a policy self-learning module. Its input includes the co-creation control instruction sequence generated by the policy network during the current round of interaction. and its corresponding set of interactive voice signals In addition, the module also receives multiple sets of external evaluation parameters: task completion status parameters (task completion score, sub-task goal flag, task execution time) provided by the rehabilitation task management module; interaction fluency provided by the interaction log (a log system based on standard timestamps and event statistics), including response latency, number of interruptions, and language coherence score, as reference data for calculating comprehensive rehabilitation returns; and the child's emotion vectors output by the state vector acquisition module and the speech generation and emotion rendering module, respectively. With the high-dimensional emotion feature vector generated by the system .

[0081] No. Comprehensive rehabilitation benefits of wheel interaction Calculated by a weighted function to quantify the overall effect of the current interaction:

[0082]

[0083]

[0084]

[0085] in, , , These are the fourth, fifth, and sixth weights, which are adaptively set by the system based on the task type. This indicates the score for task completion, which is the [number]th [level]. The number of sub-task objectives successfully triggered by the child during the round-robin interaction With the The total number of preset task objectives in the wheel interaction The ratio is used to measure the quality of a child's completion of the current rehabilitation task; It indicates the fluency of interaction and is used to reflect the coherence and fluency of children in the process of language expression; This indicates the response latency obtained from the interaction log; This indicates the number of interruptions obtained from the interaction log; and These are the preset first tolerance threshold and the second tolerance threshold, respectively; Indicates the degree of mood improvement, used to describe the child's mood improvement during the period. The degree of improvement in emotional state during the interaction is the difference between the emotional valence at the end of the interaction and the emotional valence at the beginning of the interaction. If the difference is positive, it indicates that the child's emotions have improved and is used to motivate the system to provide positive emotional guidance. If the difference is 0 or negative, the current state is maintained and no motivation is provided.

[0086] To assess the consistency between the system's output emotions and children's actual emotions, the system introduces an emotion consistency index. Calculate using the following formula:

[0087]

[0088] in, This represents the vector magnitude. The numerator is... and The dot product is used to measure the directional similarity between the two; the denominator is the product of their moduli, used for normalization. This formula essentially adopts the form of cosine similarity, with a value range of [-1, 1]. The closer the value is to 1, the more consistent the system's output emotion is with the child's emotional state.

[0089] During the strategy network parameter update phase, the system incorporates both the comprehensive rehabilitation reward and the emotional consistency index into the optimization objective. The parameter update relationship is expressed as follows:

[0090]

[0091] in, Indicates the first The strategy network parameters for round-robin interaction. This represents the updated policy network parameters; This indicates an assignment operation; The learning rate; This represents the gradient operator; This indicates an expectation of comprehensive rehabilitation benefits; This represents the weighting coefficient for the emotional consistency index. In the actual calculation process, the system will use the [missing value] as the weighting coefficient. The actual comprehensive rehabilitation benefits obtained through wheel interaction As an unbiased estimate of the expected comprehensive rehabilitation return (i.e., Monte Carlo sampling estimate based on a single interaction trajectory), the policy network parameters are calculated and the gradient is updated in conjunction with the policy gradient theorem.

[0092] Therefore, by introducing an emotion consistency index as a constraint into the optimization objective, this invention constructs a comprehensive rehabilitation reward that includes interaction fluency and emotion improvement, in addition to the conventional task completion score. This forces the strategy network to maximize both "task completion effect" and "emotional adaptation effect" during the training process, avoiding the neglect of children's psychological experience in order to complete the task, and improving the stability and emotional synergy of the strategy network in the rehabilitation guidance process.

[0093] Finally, as shown in Figure 3, the policy self-learning module outputs the updated policy network parameters. Feedback to the policy network This information is used for the next round of interactive decision-making. Through multiple rounds of interaction and parameter iteration, the system continuously optimizes in terms of task completion accuracy, semantic understanding accuracy, and emotional adaptability, forming an adaptive interaction strategy with continuity, stability, and emotional sensitivity in the time dimension. This makes the rehabilitation guidance process more natural, gentle, and in line with children's cognitive and emotional characteristics.

[0094] It should also be noted that the core algorithm of this invention is the Structure Adaptive Co-creation Decision Engine (ACDE), and its strategy generation mechanism based on "semantic-emotion-task" fusion modeling and emotion modulation self-learning. These are key components for the system to achieve dynamic co-creation and individualized adjustment, and are not replaceable. The temporal policy decision module is responsible for fusing multimodal inputs and outputting co-creation control commands, and is the core of the system's overall closed-loop logic. Without changing the framework of this invention, peripheral modules can be replaced according to the application scenario or computing power. Different deep model structures can be used for speech recognition and emotion recognition. For example, Conformer or Wav2Vec2.0 models can be used in high-performance environments to improve recognition accuracy, while DeepSpeech2 or lightweight RNN-CTC models can be used on mobile devices to ensure real-time performance. Different pre-trained language models, such as BERT, RoBERTa, or ERNIE, can also be selected based on the language type during the semantic encoding stage to generate high-quality semantic vectors.

[0095] To better demonstrate the specific implementation and technical effects of the present invention, the children's language and cognitive rehabilitation assistance system based on the adaptive co-creation mechanism described above will be applied to a specific example.

[0096] Example

[0097] The specific implementation process of the children's language and cognitive rehabilitation assistance system based on the adaptive co-creation mechanism used in this embodiment is as described above and will not be repeated here.

[0098] This experiment was conducted at an educational and rehabilitation institution in a certain region, recruiting 20 children aged 5 to 10 years old. The experiment consisted of two phases: a usability experiment phase and a formal experiment phase.

[0099] In the usability experiment phase, this embodiment used the Children's Language and Cognitive Rehabilitation Assistive System to conduct basic function tests on four children, including voice input, drawing creation, and plot selection. The system invokes the state vector acquisition module, the temporal strategy decision-making module, and the voice generation and emotion rendering module in real time to complete voice recognition, feedback generation, and plot presentation. Subsequently, the system interface and interactive experience were evaluated using a child-friendly system usability scale. The results showed that the system provided by this invention has good operability and interactivity, and the tested children were able to complete the tasks independently and maintain a high level of interest in using it.

[0100] In the formal experimental phase, to verify the practical effect of this invention in promoting children's narrative learning and cognitive expression, this embodiment designed an experimental process of "pre-test—co-creation interaction—transfer creation—post-test". The system, combining a state vector acquisition module and a speech generation and emotion rendering module, implemented an adaptive co-creation mechanism of "problem—feedback—scaffolding," dynamically guiding children to improve the plot and expressive content. The system background automatically records speech input, number of interactions, response time, and drawing behavior data.

[0101] After the experiment, based on the children's pre- and post-test questionnaire results, interaction records, and system logs, assessments were conducted on dimensions such as language expression, core idea comprehension, plot generalization, and transferability. As shown in Figure 4, after using the system of this invention, children exhibited significant improvements in the structural completeness of story narration, language coherence, and emotional expression. Furthermore, feedback from interviews with teachers and parents indicated that the system effectively improved children's attention span and learning interest. Educators believe that this system can stimulate children's language imagination and narrative abilities, promote emotional expression and creative thinking training, and has good potential for rehabilitation and educational support.

[0102] In summary, by integrating the semantic understanding capabilities of a large language model, speech emotion recognition technology, and a multimodal generation mechanism, this invention constructs an intelligent cognitive rehabilitation assistance system for children that can autonomously generate rehabilitation content, achieve dynamic emotional feedback, and conduct long-term effect evaluation, thereby significantly improving the individualization, emotional responsiveness, and overall intelligence level of rehabilitation intervention.

[0103] It should also be noted that the adaptive co-creation mechanism-based children's language and cognitive rehabilitation assistance system in the above embodiments can essentially be executed through method steps or computer programs. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides an interactive method based on an adaptive co-creation mechanism, corresponding to the adaptive co-creation mechanism-based children's language and cognitive rehabilitation assistance system provided in the above embodiments, as shown in Figure 5, which includes the following steps:

[0104] S1. Acquire the child's real-time voice signal during the current interaction, and input the real-time voice signal into the child language cognitive rehabilitation assistance system based on the adaptive co-creation mechanism described in the above embodiment. The state vector acquisition module generates a new multimodal vector based on the child's voice signal, the multimodal fusion module generates a new fusion state vector based on the new multimodal vector, the temporal strategy decision module generates a new co-creation control command based on the new fusion state vector, the voice generation and emotion rendering module generates a new interactive voice signal based on the new co-creation control command, and plays the new interactive voice signal to the child through the audio output device to conduct voice dialogue interaction;

[0105] S2. After the current interaction ends, the strategy self-learning module evaluates the operation of the children's language and cognitive rehabilitation assistance system, updates the strategy network parameters, and returns to S1 to execute the next round of interaction until the preset rehabilitation task is completed or the maximum number of interactions is reached.

[0106] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the children's language and cognitive rehabilitation auxiliary system based on the adaptive co-creation mechanism provided in the above embodiment, as shown in FIG6, which includes a memory and a processor;

[0107] The memory is used to store computer programs;

[0108] The processor is configured to implement the child language and cognitive rehabilitation assistive system based on the adaptive co-creation mechanism in the above embodiments when executing the computer program.

[0109] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0110] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0111] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0112] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A language and cognitive rehabilitation assistance system for children based on an adaptive co-creation mechanism, characterized in that, include: The state vector acquisition module is used to acquire the child's original speech signal in each round of interaction in real time. The speech recognition unit converts the original speech signal into a text information sequence and extracts the semantic vector from it. The emotion recognition unit obtains the emotion vector from the original speech signal. The rehabilitation task management module generates the task vector before the interaction begins. Thus, the semantic vector, emotion vector, and task vector together constitute a multimodal vector. The multimodal fusion module is used to perform linear mapping on the multimodal vector to obtain multimodal features. A set of dynamic weights is generated through a dynamic weight generation network. After weighting the multimodal features, the initial fusion features are generated. Based on multimodal features, a cross-modal attention weight matrix and a similarity matrix are calculated. The two matrices are multiplied element-wise and normalized to obtain a composite fusion weight matrix. The composite fusion weight matrix is ​​used to weight the initial fusion features element-wise to obtain a fusion state vector. The temporal policy decision module is used to weight and aggregate the fusion state vector through a gated recurrent unit based on attention enhancement to generate a temporal policy hidden state, which is then input into the policy network to generate co-creation control instructions. The speech generation and emotion rendering module is used to parse the co-creation control instructions to generate target text, and calls the speech synthesis engine to convert the target text into an output speech signal. Emotional expression is injected during the speech signal generation process to finally obtain an interactive speech signal for interaction with children. The policy self-learning module is used to evaluate the system operation process and update parameters after each round of interaction, so as to achieve continuous learning and performance improvement of the policy network. In the state vector acquisition module, the text information sequence is subjected to deep semantic encoding by a pre-trained Chinese natural language processing model to obtain a semantic vector; the emotion vector is obtained by dimensionality reduction mapping of statistical features through a multilayer perceptron, and the statistical features are Mel frequency cepstral coefficients and prosodic features obtained from the original speech signal. The rehabilitation task management module retrieves three types of task information from the preset rehabilitation course map based on the current interaction progress pointer: the current task type number, the task target difficulty level, and the task completion status. Then, it converts the task information into a numerical vector representation by using a lookup table or one-hot encoding to obtain the task vector.

2. The children's language and cognitive rehabilitation assistance system based on an adaptive co-creation mechanism as described in claim 1, characterized in that, In the multimodal fusion module, semantic vectors, emotion vectors, and task vectors each pass through a linear mapping layer to obtain semantic features, emotion features, and task features, which constitute multimodal features. The dynamic weight generation network takes the context vector constructed from various real-time indicators as input, and after processing by a multilayer perceptron, the output contains a set of dynamic weights, with each weight corresponding to a feature. The real-time indicators include: speech recognition confidence, emotion change trend, speech rate stability, and task stage.

3. The children's language and cognitive rehabilitation assistance system based on an adaptive co-creation mechanism as described in claim 2, characterized in that, The cross-modal attention weight matrix uses semantic features as the query vector, and concatenates emotion features and task features to serve as both the key vector and the value vector, and is obtained through attention calculation; the similarity between semantic features, emotion features, and task features is calculated to obtain a similarity matrix.

4. The children's language and cognitive rehabilitation assistance system based on an adaptive co-creation mechanism as described in claim 1, characterized in that, In the time-series policy decision-making module, the policy network adopts a multilayer perceptron composed of fully connected layers, and the generated co-creation control instructions consist of three control components: policy control component, language control component, and emotion control component.

5. The children's language and cognitive rehabilitation assistance system based on an adaptive co-creation mechanism as described in claim 4, characterized in that, In the speech generation and emotion rendering module, firstly, based on natural language generation technology, target text is generated according to the interaction behavior type in the policy control component and the language control component. Secondly, the speech synthesis engine is called to convert the target text into an output speech signal according to the tone style parameter in the language control component and the output rhythm parameter in the emotion control component. Then, the emotion rendering intensity parameter in the emotion control component is input into the parameter mapping network to generate a high-dimensional emotion feature vector. Finally, the high-dimensional emotion feature vector is injected into the acoustic parameter space, mapped to three prosodic parameters—pitch, energy level, and formant distribution—in the vocoder stage, and added to the speech signal to output the interactive speech signal synthesized by the vocoder.

6. The children's language and cognitive rehabilitation assistance system based on an adaptive co-creation mechanism as described in claim 5, characterized in that, In the strategy self-learning module, after interacting with the child using the strategy network parameters from the i-th round of interaction, the comprehensive rehabilitation reward is calculated by weighting and summing the task completion score, interaction fluency, and emotion improvement degree of the i-th round of interaction, and used as an unbiased estimate of the expected comprehensive rehabilitation reward. The emotion consistency index is calculated from the emotion vector and the high-dimensional emotion feature vector. The comprehensive rehabilitation reward is added to the weighted emotion consistency index to obtain the target value, and the gradient of the strategy network parameters with respect to the target value is calculated. The preset learning rate is multiplied by the calculated gradient to obtain the parameter adjustment amount. The strategy network parameters from the i-th round of interaction are added to the adjustment amount to obtain the updated strategy network parameters.

7. The children's language and cognitive rehabilitation assistance system based on an adaptive co-creation mechanism as described in claim 6, characterized in that, The task completion score is the ratio of the number of sub-task objectives successfully triggered by the child in the i-th round of interaction to the preset total number of task objectives; the interaction fluency incorporates response delays and interruption counts obtained from the interaction log; the emotion improvement is the difference between the emotional valence at the end of the interaction and the emotional valence at the beginning of the interaction; the emotion consistency index is calculated as follows: first, calculate the dot product of the emotion vector and the high-dimensional emotion feature vector, then calculate the product of the magnitude of the emotion vector and the magnitude of the high-dimensional emotion feature vector, and divide the dot product result by the product result.

8. A computer electronic device, characterized in that, It includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement, when executing the computer program, the child language and cognitive rehabilitation assistance system based on the adaptive co-creation mechanism as described in any one of claims 1 to 7.

Images