Multi-mode synchronous control method for driving robot to narrate with body

Abstract

The invention relates to a multi-mode synchronous control method for driving a robot to narrate, which comprises the following steps: receiving to-be-described text data, and standardizing the to-be-described text data to obtain preprocessed text data; performing hierarchical decoding on the preprocessed text data based on a large language model to obtain structured feature annotation data; generating a multi-modal expression sequence according to the structured feature annotation data, and performing timestamp alignment synchronization on the multi-modal expression sequence to obtain a collaborative multi-modal instruction; and carrying out bandwidth allocation and execution scheduling on the collaborative multi-modal instruction by adopting a dynamic priority strategy, and driving robot hardware to execute a corresponding multi-modal expression action. The diversity of robot emotion expression content is improved, and interaction delay is reduced.

Description

Technical Field

[0001] This invention belongs to the field of embodied intelligence technology, and in particular relates to a multimodal synchronous control method for driving embodied narratives in robots. Background Technology

[0002] Existing technologies for emotional expression in intelligent robots primarily focus on single-modal or simple multimodal control implementation. Some technologies attempt to combine voice, facial expressions, and actions, but the lack of a unified emotional expression framework and precise time synchronization mechanisms leads to significant inconsistencies between multimodal expressions. In recent years, with the development of large language model technology, some research has begun to explore the application of natural language processing capabilities to robot control, but these studies largely remain at the level of text understanding and simple command generation. Meanwhile, existing technologies mainly rely on retrieval and matching from local sound effect and music libraries for auxiliary content generation, lacking real-time generation capabilities and making it difficult to provide personalized background music and environmental sound effects support for different storylines.

[0003] Existing technologies suffer from the following shortcomings: First, multimodal expression lacks a unified emotion modeling framework; the generation of speech, facial expressions, and actions is independent, resulting in insufficient overall expressiveness and temporal asynchrony issues. Second, the fine-grained control of emotion expression is limited; existing technologies mostly employ discrete emotion classification methods, failing to achieve continuous quantification of emotion intensity and dynamic mapping of multimodal parameters. Third, hardware resource utilization is inefficient; small robots lack effective priority scheduling mechanisms when executing multimodal commands, leading to high execution delays for critical action commands. These shortcomings severely restrict the application effectiveness of intelligent robots in scenarios such as education, companionship, and emotional interaction. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of the invention is to provide a multimodal synchronous control method for driving embodied narratives in robots.

[0005] This invention provides a multimodal synchronous control method for driving embodied narratives in robots, comprising: S1: Receive the text data to be described, and perform standardization processing on the text data to be described to obtain preprocessed text data; S2: Based on the large language model, perform hierarchical decoding on the preprocessed text data to obtain structured feature annotation data; S3: Generate a multimodal expression sequence based on the structured feature annotation data, and perform timestamp alignment and synchronization on the multimodal expression sequence to obtain a collaborative multimodal instruction; S4: A dynamic priority strategy is used to allocate bandwidth and schedule the execution of the collaborative multimodal instructions, driving the robot hardware to execute the corresponding multimodal expression actions.

[0006] According to the multimodal synchronous control method for driving a robot with embodied narrative provided by the present invention, step S2 further includes: S21: Perform role recognition on the preprocessed text data to extract a role list including main roles and secondary roles; S22: For each character in the character list, analyze the character features of each character in the character list using a large language model, construct a character feature vector matrix, and obtain the character feature vector; S23: Divide the preprocessed text data into multiple segments, perform sentiment analysis on each segment, identify the sentiment type and quantify the sentiment intensity level to obtain a sentiment feature vector; S24: Extract environmental description information from the preprocessed text data using a scene recognition algorithm to obtain a scene feature vector; S25: Integrate and label the character feature vector, the emotion feature vector, and the scene feature vector to form structured feature annotation data.

[0007] According to the multimodal synchronous control method for driving a robot with embodied narrative provided by the present invention, step S23 further includes: S231: Input the segmented text fragments into the sentiment classification model to identify and obtain sentiment type labels; S232: Calculate the emotional intensity value of each text segment using an emotional intensity quantization algorithm, and map the emotional intensity value to the corresponding level according to a preset intensity threshold range to obtain an emotional feature vector.

[0008] According to the multimodal synchronous control method for driving embodied narrative in a robot provided by the present invention, the emotion intensity grading mapping relationship established in step S232 includes multiple emotion intensity levels, specifically including: When the emotional intensity value is mapped to the first emotional level, the joint swing range is set to ±10 degrees, the voice pitch change rate is ±3%, and the facial expression change speed is 0.3 times the speed corresponding to the first intensity emotional expression parameters; When the emotional intensity value is mapped to the second emotional level, the joint swing range is set to ±15 degrees, the voice pitch change rate is ±5%, and the facial expression change speed is 0.5 times the speed corresponding to the second intensity emotional expression parameters; When the emotional intensity value is mapped to the third emotional level, the corresponding third intensity emotional expression parameters are set as follows: the joint swing range is ±25 degrees, the voice pitch change rate is ±10%, and the facial expression change speed is 1x. When the emotional intensity value is mapped to the fourth emotional level, the joint swing range is set to ±35 degrees, the voice pitch change rate is ±15%, and the facial expression change speed is 1.5 times the normal speed, corresponding to the fourth intensity emotional expression parameters. When the emotional intensity value is mapped to the fifth emotional level, the joint swing range is set to ±45 degrees, the voice pitch change rate is ±20%, and the facial expression change speed is 2 times faster, corresponding to the fifth intensity emotional expression parameters.

[0009] According to the multimodal synchronous control method for driving embodied narratives of a robot provided by the present invention, step S3 further includes: S31: Decode the structured feature annotation data based on a hierarchical decoder architecture to output an initial action parameter sequence; wherein, the hierarchical decoder architecture includes an attention LSTM layer and a language generation LSTM layer; S32: Adjust the amplitude of the initial action parameter sequence according to the emotion intensity level in the emotion feature vector to obtain the adjusted optimized action parameter sequence; S33: Match facial expression patterns and action patterns corresponding to the character feature vector and emotional feature vector from the preset expression library to generate a sequence of facial expression changes; S34: The optimized action parameter sequence and the facial expression change sequence are organized in chronological order to obtain an initial multimodal expression sequence including multiple expression units; wherein, each expression unit includes a speech segment, an facial expression change node, and an action execution node; S35: Calculate the start time and duration of the speech segment, and perform consistency alignment processing on the timestamps of the facial expression change node, the action execution node, and the speech segment; S36: The time deviation between multiple expression units in the initial multimodal expression sequence is detected by a time deviation calculation algorithm. When the time deviation exceeds a preset synchronization threshold, the timestamps of the expression change node and the action execution node are corrected to obtain a collaborative multimodal instruction.

[0010] According to the multimodal synchronous control method for driving a robot with embodied narrative provided by the present invention, step S31 further includes: S311: Input the structured feature annotation data into the attention LSTM layer, extract key event features and role association features through the attention mechanism, and generate intermediate feature representations; S312: Input the intermediate feature representation into the language generation LSTM layer, and generate the initial action parameter sequence through the recurrent neural network.

[0011] According to the multimodal synchronous control method for driving a robot with embodied narrative provided by the present invention, step S32 further includes: S321: Extract the emotional intensity level value from the emotional feature vector; S322: Retrieve the corresponding expression parameter configuration based on the emotional intensity level value; S323: The expression parameter configuration is applied to the initial action parameter sequence to obtain an optimized action parameter sequence.

[0012] According to the multimodal synchronous control method for driving embodied narratives of a robot provided by the present invention, step S35 includes: S351: Extract the start and end timestamps of the speech segment and calculate the time span of the speech segment; S352: Traverse the expression change nodes and the action execution nodes to obtain the preset timestamp of each node; S353: Adjust the preset timestamp according to the time span so that the preset timestamp and the speech segment remain associated on the timeline to obtain an optimized timestamp; S354: Verify the optimized timestamps to ensure that the timestamps of the facial expression change nodes and the action execution nodes fall within the time span of the speech segment.

[0013] According to the multimodal synchronous control method for driving a robot with embodied narrative provided by the present invention, step S4 further includes: S41: Classify and label the action instructions in the collaborative multimodal instructions to obtain classification and labeling results including key action instructions and non-key action instructions; S42: Based on the classification and labeling results, dynamically allocate communication bandwidth and computing resources according to a preset resource allocation ratio to obtain resource allocation results; S43: Establish an instruction execution queue, insert the critical action instruction into the first preset position of the queue, and insert the non-critical action instruction into the second preset position of the queue, wherein the first preset position has a higher priority than the second preset position; S44: Extract instructions sequentially from the instruction execution queue and send them to the execution unit of the robot hardware to control the robot hardware to perform corresponding actions.

[0014] According to the multimodal synchronous control method for driving embodied narratives of a robot provided by the present invention, step S4 further includes: S51: Call the music generation interface and the sound effect generation interface to generate background music data and ambient sound effect data; the music generation interface is implemented based on the MusicLM system, and the sound effect generation interface is implemented based on the AutoFoley system. S52: Integrate the background music data and the environmental sound effect data with the multimodal expression action on the timeline to obtain an enhanced multimodal output expression action.

[0015] This invention significantly improves the emotional consistency and temporal synchronization of robot multimodal expressions by employing a hierarchical decoding processing technology based on a large language model, combined with an emotional intensity hierarchical mapping mechanism and a dynamic priority scheduling strategy. Secondly, by introducing a hierarchical decoder architecture consisting of an attention LSTM layer and a language generation LSTM layer, this invention can accurately map structured feature-annotated data into a sequence of action parameters including joint angle parameters, facial expression change timestamps, and speech tone parameters. This ensures that the robot's expressive actions highly match the emotional features of the text content, greatly improving the naturalness and accuracy of the expression. Furthermore, by establishing a multi-level emotional intensity quantification system and configuring corresponding joint swing range, speech pitch change rate, and facial expression change speed parameters for each level, this invention achieves continuous control of emotional expression from coarse to fine granular, enabling the robot to simulate subtle differences in human emotional expression and enhancing emotional resonance and interactive affinity with users. This invention also uses timestamps to... The synchronization mechanism precisely synchronizes speech segments, facial expression change nodes, and action execution nodes, controlling the time deviation between multimodal expressions within a preset threshold range. This effectively eliminates common problems such as mismatch between facial expressions and speech, and disconnect between actions and content, significantly enhancing the fluency and coordination of the overall expression. Furthermore, by using a dynamic priority strategy to differentiate bandwidth allocation and execution scheduling for key and non-key action instructions, low-latency execution of key expressive actions is ensured even under limited hardware resources, resulting in a significant improvement in system response speed and resource utilization efficiency. In addition, this invention uses generative AI interfaces such as MusicLM and AutoFoley to create background music and environmental sound effects in real time, enriching the layering and immersion of the robot's expression. It can provide personalized auxiliary content for different scenarios without relying on a large amount of pre-stored resources, greatly reducing content production costs and development cycles, and expanding the applicability and commercial value of robots in various application scenarios such as education, entertainment, and companionship. Attached Figure Description

[0016] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0017] Figure 1 This is a schematic flowchart of a multimodal synchronous control method for driving a robot with embodied narrative, provided by an embodiment of the present invention. Figure 2 This is a schematic diagram of the method for hierarchical decoding of preprocessed text data based on a large language model, provided in an embodiment of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0019] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0020] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0022] To better understand this invention, the technical terms appearing in the embodiments of this invention will be explained in detail below.

[0023] Embodied Intelligence: refers to the intelligent behavior generated by an intelligent agent through interaction with its body and environment, completing a closed-loop interaction of "perception-understanding-reasoning-execution", and is an important direction of the next generation of artificial intelligence.

[0024] Hierarchical Decoder Architecture: A neural network architecture that decomposes multimodal instructions into symbolic task trees and maps them to robot action sequences, including attention LSTM layers and language generation LSTM layers.

[0025] Dynamic Priority Strategy: An algorithm that allocates communication bandwidth and computing resources based on action type (critical action / non-critical action) to ensure low-latency execution of critical action instructions.

[0026] Emotional Intensity Grading Model: A quantitative model that divides emotional intensity into 1-5 levels to guide the amplitude and speed of multimodal expression in robots.

[0027] AIGC (AI generated content): This refers to the technology that uses AI to automatically generate content such as text, images, audio, and video.

[0028] MusicLM: A music generation system based on a large model that can generate corresponding music fragments based on text descriptions.

[0029] AutoFoley: An environmental sound effect generation system based on large models, capable of generating corresponding sound effects based on text keywords.

[0030] Dynamic priority strategy: An algorithm that allocates communication bandwidth and computing resources based on action type (critical action / non-critical action) to ensure low-latency execution of critical action instructions.

[0031] Emotional intensity grading model: A quantitative model that divides emotional intensity into 1-5 levels to guide the amplitude and speed of robot multimodal expression.

[0032] Hierarchical decoder architecture: A neural network architecture that decomposes multimodal instructions into symbolic task trees and maps them into robot action sequences, including attention LSTM layers and language generation LSTM layers.

[0033] MusicLM: A music generation system based on a large model that can generate corresponding music fragments based on text descriptions.

[0034] AutoFoley: An environmental sound effect generation system based on large models, capable of generating corresponding sound effects based on text keywords.

[0035] The embodiments of the present invention are described below with reference to the figures.

[0036] like Figure 1 As shown, the present invention provides a multimodal synchronization control method for driving embodied narratives in robots, comprising: S1: Receive the text data to be described, and perform standardization processing on the text data to be described to obtain preprocessed text data.

[0037] Furthermore, this invention receives text data to be described from the user terminal through a story content input module. The source of this text data includes text files directly input by the user, text content converted through speech recognition technology, or story content scraped from a webpage. Upon receiving the text data, this invention performs standardization processing, specifically including removing special symbols, standardizing character encoding formats, converting full-width characters to half-width characters, and removing redundant spaces and line breaks. Subsequently, this invention performs sentence segmentation on the standardized text, dividing it into independent sentence units based on punctuation marks such as periods, question marks, and exclamation marks. After sentence segmentation, this invention performs lexical analysis on each sentence, identifying parts-of-speech tags such as nouns, verbs, and adjectives, and simultaneously establishing a grammatical dependency tree for the sentence. After the above processing, the original text data to be described is converted into preprocessed text data with a clear structure and uniform format, where each sentence unit contains complete part-of-speech tags and grammatical structure information.

[0038] S2: Based on the large language model, the preprocessed text data is decoded in layers to obtain structured feature annotation data.

[0039] like Figure 2 As shown, step S2 further includes: S21: Perform role recognition on the preprocessed text data to extract a role list including main roles and secondary roles.

[0040] In step S21, this invention utilizes a named entity recognition algorithm to perform role recognition processing on the preprocessed text data. The named entity recognition algorithm is implemented based on a conditional random field model or a BERT pre-trained model. By scanning each sentence in the preprocessed text data, it identifies named entities representing people, animals, or anthropomorphic objects. During the recognition process, this invention constructs a character-level feature vector for each sentence. The feature vector includes the part-of-speech tag, information about adjacent characters within the context window, and the character's position within the sentence. Subsequently, this invention inputs the feature vector into the named entity recognition model, and the model outputs a tag sequence for each character. Tag types include BIO annotation formats such as B-PER (beginning of name), I-PER (middle of name), and O (non-entity). By parsing the tag sequence, this invention extracts the complete character name and counts the frequency of each character in the preprocessed text data. Based on the frequency, this invention marks characters with a frequency higher than a preset threshold as primary characters and characters with a frequency lower than the preset threshold but higher than a minimum threshold as secondary characters, thus obtaining a character list containing primary and secondary characters.

[0041] S22: For each character in the character list, analyze the character features of each character in the character list using a large language model, construct a character feature vector matrix, and obtain the character feature vector.

[0042] Furthermore, for each character in the character list, this invention extracts all sentences containing that character from the preprocessed text data to form a set of character-related sentences. Subsequently, this invention inputs the set of character-related sentences into a large language model. The large language model uses prompt word engineering technology to construct analysis instructions. The format of the prompt words is "Analyze the personality traits, language style, and behavioral habits of the character [character name] in the following sentences, and output a feature description in JSON format."

[0043] After receiving cue words and a set of character-related sentences, the large language model uses a self-attention mechanism to calculate the semantic relationships between words in the sentences, identifying adjectives describing the character's personality, verbs reflecting the character's behavior, and dialogue content that embodies the character's language style. After processing, the large language model outputs JSON data containing fields for personality traits, language style, and behavioral habits.

[0044] Subsequently, this invention parses the JSON data, converting the descriptive terms in the personality traits field into numerical vectors of personality dimensions, the descriptions in the language style field into numerical vectors of language features, and the descriptions in the behavior habits field into numerical vectors of behavior patterns. Then, this invention concatenates the personality dimension vectors, language feature vectors, and behavior pattern vectors to form a single character's feature vector. This process is repeated for all characters in the character list, ultimately arranging the feature vectors of each character row-wise to construct a character feature vector matrix.

[0045] S23: Divide the preprocessed text data into multiple segments, perform sentiment analysis on each segment, identify the sentiment type and quantify the sentiment intensity level to obtain a sentiment feature vector.

[0046] In step S23, the present invention divides the preprocessed text data into multiple segments based on the plot development. The segmentation criteria include scene transition markers, time jump markers, or changes in the number of character dialogue rounds. Each segment contains several consecutive sentences, and the segment length is dynamically determined according to the text content. After the segmentation is completed, the present invention performs sentiment analysis on each segment to finally obtain a sentiment feature vector.

[0047] Step S23 further includes: S231: Input the segmented text fragments into the sentiment classification model to identify and obtain sentiment type labels.

[0048] In step S231, the present invention inputs each segmented text fragment into the sentiment classification model. The sentiment classification model is fine-tuned based on a RoBERTa pre-trained language model, and its structure includes a 12-layer Transformer encoder and a fully connected classification layer. When inputting the text fragment, the present invention first converts the text into a sequence of tokens and assigns positional and paragraph codes to each token. The token sequence is converted into a 768-dimensional vector representation through an embedding layer, and then input into the Transformer encoder for multi-head self-attention calculation. The first vector [CLS] of the encoder output sequence contains the semantic representation of the entire fragment. Then, the present invention inputs this vector into the fully connected classification layer. The classification layer uses a softmax activation function and outputs the probability distribution of the text fragment in terms of sentiment types such as happiness, sadness, tension, surprise, anger, and calmness. Finally, the present invention selects the sentiment type with the highest probability value as the sentiment type label of the fragment.

[0049] S232: Calculate the emotional intensity value of each text segment using an emotional intensity quantization algorithm, and map the emotional intensity value to the corresponding level according to a preset intensity threshold range to obtain an emotional feature vector.

[0050] Further, in step S232, this invention calculates the sentiment intensity value of each text segment using a sentiment intensity quantification algorithm. This algorithm is based on a sentiment dictionary and syntactic structure analysis. Specifically, this invention first extracts sentiment words from the text segment. Sentiment words include adjectives, adverbs, and verbs with emotional connotations. The sentiment dictionary pre-labels each sentiment word with a sentiment polarity value, ranging from -1 to 1, where negative values ​​represent negative sentiment and positive values ​​represent positive sentiment. After extracting sentiment words, this invention calculates the sum of the absolute values ​​of the polarity values ​​of all sentiment words in the segment as the initial sentiment intensity value. Subsequently, this invention analyzes degree adverbs and negation words in the segment. Degree adverbs such as "very" and "especially" enhance sentiment intensity, while negation words such as "not" and "no" reverse sentiment polarity. This invention adjusts the initial sentiment intensity value by a factor based on the number of degree adverbs and reverses the sign of the sentiment polarity based on the number of negation words. After adjustment, the final sentiment intensity value of the segment is obtained.

[0051] This invention predefines five intensity threshold ranges, with boundary values ​​of 0.2, 0.4, 0.6, and 0.8, respectively. When the emotional intensity value falls between 0 and 0.2, it is mapped to level 1; between 0.2 and 0.4, level 2; between 0.4 and 0.6, level 3; between 0.6 and 0.8, level 4; and exceeding 0.8, level 5. Ultimately, this invention encodes the emotional type label and emotional intensity level together into an emotional feature vector. The first dimension of the vector represents the one-hot encoding of the emotional type, and the second dimension represents the numerical value of the emotional intensity level.

[0052] The emotional intensity grading mapping relationship established in step S232 includes multiple emotional intensity levels, specifically: when the emotional intensity value is mapped to the first emotional level, the joint swing range is set to ±10 degrees, the voice pitch change rate is ±3%, and the facial expression change speed is 0.3 times the normal speed, corresponding to the first intensity emotional expression parameters; when the emotional intensity value is mapped to the second emotional level, the joint swing range is set to ±15 degrees, the voice pitch change rate is ±5%, and the facial expression change speed is 0.5 times the normal speed, corresponding to the second intensity emotional expression parameters; when the emotional intensity value is mapped to the third emotional level... When the emotional intensity value is mapped to the third emotional level, the joint swing range is set to ±25 degrees, the voice pitch change rate is ±10%, and the facial expression change speed is 1x. When the emotional intensity value is mapped to the fourth emotional level, the joint swing range is set to ±35 degrees, the voice pitch change rate is ±15%, and the facial expression change speed is 1.5x. When the emotional intensity value is mapped to the fifth emotional level, the joint swing range is set to ±45 degrees, the voice pitch change rate is ±20%, and the facial expression change speed is 2x.

[0053] Furthermore, the joint swing range refers to the offset of the rotation angle of the robot's shoulder, elbow, and neck joints relative to the neutral position; the speech pitch change rate is the amplitude of the change of the fundamental frequency of the synthesized speech relative to the reference pitch; and the facial expression change speed refers to the multiple by which the time for the robot's facial expression to transition from a neutral state to the target expression is extended to the standard speed.

[0054] S24: Extract environmental description information from the preprocessed text data using a scene recognition algorithm to obtain a scene feature vector.

[0055] This invention extracts environmental description information from preprocessed text data using a scene recognition algorithm. The scene recognition algorithm is based on keyword matching and semantic role labeling (SLA). First, this invention constructs a scene keyword library, which includes nouns representing locations such as "forest," "castle," and "seaside," words representing time such as "morning," "dusk," and "night," and words representing weather such as "sunny," "rainy," and "cloudy." This invention iterates through each sentence in the preprocessed text data, detecting whether the sentence contains words from the scene keyword library. Upon detecting a keyword, this invention extracts the complete semantic structure of the sentence containing that keyword, identifying location, time, and manner arguments in the sentence through SLA. Subsequently, this invention converts the identified location, time, and weather information into scene feature vectors. The scene feature vectors employ multi-dimensional numerical encoding: the first dimension represents the one-hot encoding of the location category, the second dimension represents the one-hot encoding of the time period, and the third dimension represents the one-hot encoding of the weather state. For each segment in the preprocessed text data, this invention extracts the corresponding scene feature vector, forming a scene feature vector sequence equal to the number of segments.

[0056] S25: Integrate and label the character feature vector, the emotion feature vector, and the scene feature vector to form structured feature annotation data.

[0057] S3: Generate a multimodal expression sequence based on the structured feature annotation data, and perform timestamp alignment and synchronization on the multimodal expression sequence to obtain a collaborative multimodal instruction.

[0058] This invention integrates and annotates the character feature vector, the emotion feature vector, and the scene feature vector. During the integration process, this invention establishes feature associations using segments as the basic unit. For each segment, this invention extracts the character identifiers involved in the segment and retrieves the corresponding character's feature vector from the character feature vector matrix. Subsequently, this invention concatenates the character feature vector, the segment's emotion feature vector, and the scene feature vector in a fixed order to form a comprehensive feature vector for a single segment. The dimension of the comprehensive feature vector is equal to the sum of the dimensions of the character feature vector, the emotion feature vector, and the scene feature vector. This invention adds metadata tags to the comprehensive feature vector of each segment. The tag content includes the segment's position index in the text, the segment's start timestamp, a list of characters involved in the segment, and the segment's emotion type identifier. All segments' comprehensive feature vectors and their metadata tags are organized into a sequence structure according to the text order; this sequence structure is the structured feature annotation data. The structured feature annotation data is stored in JSON format, with each segment corresponding to an object in a JSON array. The object contains a feature vector field, a character field, an emotion field, a scene field, and a timestamp field.

[0059] Step S3 further includes: S31: Decode the structured feature annotation data based on a hierarchical decoder architecture to output an initial action parameter sequence; wherein, the hierarchical decoder architecture includes an attention LSTM layer and a language generation LSTM layer.

[0060] Step S31 further includes: S311: Input the structured feature annotation data into the attention LSTM layer, extract key event features and role association features through the attention mechanism, and generate intermediate feature representations.

[0061] Furthermore, this invention inputs the structured feature-annotated data into an attention LSTM layer in segment order, with the comprehensive feature vector of each segment serving as a time step. After input, the attention LSTM layer processes the input sequence through a forget gate, an input gate, and an output gate, outputting a sequence of hidden state vectors. Subsequently, this invention applies an attention mechanism to calculate the importance weight of each hidden state vector, obtains a similarity score by taking the dot product of the query vector and the hidden state vector, and obtains the attention weight distribution after softmax normalization. Finally, this invention multiplies each hidden state vector by its attention weight and sums the results to obtain a fixed-dimensional intermediate feature representation, which encodes the key event features and role association features of the text.

[0062] S312: Input the intermediate feature representation into the language generation LSTM layer, and generate the initial action parameter sequence through the recurrent neural network.

[0063] Furthermore, this invention uses intermediate feature representations as the initial hidden states of the language generation LSTM layer. During decoding, this invention inputs a start marker symbol, and the language generation LSTM layer outputs a probability distribution of action parameter types, including joint angle parameters, facial expression change timestamp parameters, and speech pitch parameters. Subsequently, this invention selects the parameter type with the highest probability, uses its vector representation as the input for the next time step, and repeats decoding until a termination marker is output. During the decoding process, this invention assigns specific values ​​to each parameter type; for example, the joint angle parameter format is "shoulder joint: 15 degrees", the facial expression parameter format is "smile: 2.5 seconds", and the speech parameter format is "pitch: +8%, speech rate: 1.2 times". Finally, all parameters are combined in the order of generation to form the initial action parameter sequence.

[0064] S32: Adjust the amplitude of the initial action parameter sequence according to the emotional intensity level in the emotional feature vector to obtain the adjusted optimized action parameter sequence.

[0065] Step S32 further includes: S321: Extract the emotional intensity level value from the emotional feature vector.

[0066] In step S321, the present invention extracts the emotion intensity level value from the second dimension of the emotion feature vector. The first dimension of the emotion feature vector stores the one-hot encoding of the emotion type, and the second dimension stores integer values ​​from 1 to 5. The present invention accesses the second dimension through an index, reads the emotion intensity level value, and stores it in a temporary variable.

[0067] S322: Retrieve the corresponding expression parameter configuration based on the emotional intensity level value.

[0068] In step S322, the present invention retrieves the expression parameter configuration in the mapping table according to the emotional intensity level value. The mapping table adopts a dictionary structure, with the level value as the key and the value as a configuration object containing the joint swing range, the rate of change of voice pitch and the speed of change of facial expression. When searching, the present invention performs a hash lookup with the level value as the key and reads the three parameter values ​​in the configuration object.

[0069] S323: The expression parameter configuration is applied to the initial action parameter sequence to obtain an optimized action parameter sequence.

[0070] In step S323, the present invention iterates through the initial sequence of motion parameters. For joint angle parameters, the present invention trims the original angle values ​​to the configured joint swing range; for voice pitch parameters, the present invention scales the original pitch change rate according to the ratio of the configured pitch change rate to the baseline value; for facial expression timestamp parameters, the present invention divides the original time interval by the configured facial expression change rate to obtain the adjusted time interval. Finally, the present invention reorganizes the adjusted parameter items into an optimized sequence of motion parameters.

[0071] S33: Match facial expression patterns and action patterns corresponding to the character feature vector and emotional feature vector from the preset expression library to generate a sequence of facial expression changes.

[0072] In step S33, the present invention aims to match facial expression patterns and action patterns from an expression library. Specifically, the present invention extracts emotion types from emotion feature vectors, filters candidate objects suitable for emotion type matching in the facial expression pattern library, calculates the cosine similarity between the personality dimension vector and the candidate objects, and selects the facial expression pattern with the highest similarity. For action patterns, the present invention extracts behavior pattern vectors from character feature vectors, filters matching candidate objects in the action pattern library, and selects the action pattern with the highest similarity. Finally, the present invention extracts control parameters from the matched pattern objects, arranges them in chronological order, and associates them with timestamps to form a sequence of facial expression changes.

[0073] S34: Organize the optimized action parameter sequence and the facial expression change sequence in chronological order to obtain an initial multimodal expression sequence including multiple expression units; wherein, each expression unit includes a speech segment, an facial expression change node, and an action execution node.

[0074] In step S34, the present invention first inputs the fragmented text and speech intonation parameters into a speech synthesis engine to generate speech fragments containing audio data and duration, and assigns a cumulative timestamp to each speech fragment. Subsequently, the present invention encapsulates the control parameters in the facial expression change sequence into facial expression change nodes, and encapsulates the joint angle parameters in the optimized action parameter sequence into action execution nodes, calculating the absolute timestamp of each node based on the start timestamp of the corresponding speech fragment. Finally, the present invention combines speech fragments, facial expression change nodes, and action execution nodes within the same time window into expression units, arranging them in chronological order to form an initial multimodal expression sequence.

[0075] S35: Calculate the start time and duration of the speech segment, and perform consistency alignment processing on the timestamps of the facial expression change node, the action execution node, and the speech segment.

[0076] Step S35 further includes: S351: Extract the start and end timestamps of the speech segment and calculate the time span of the speech segment.

[0077] Furthermore, in step S351, the present invention traverses each expression unit in the initial multimodal expression sequence, extracts the start timestamp and duration from the speech segment object, and the end timestamp is obtained by adding the start timestamp and the duration. The time span is equal to the duration. After extraction, the present invention stores these three time values ​​in a temporary field of the expression unit object.

[0078] S352: Traverse the expression change nodes and the action execution nodes to obtain the preset timestamp of each node.

[0079] Furthermore, the present invention iterates through the list of expression change nodes and the list of action execution nodes of the expression unit, reads the timestamp field of each node object, and stores the timestamp value in the corresponding preset timestamp variable to form a preset timestamp list for expression nodes and a preset timestamp list for action nodes.

[0080] S353: Adjust the preset timestamp according to the time span so that the preset timestamp and the voice segment remain associated on the timeline to obtain an optimized timestamp.

[0081] In step S353, the present invention determines whether the preset timestamp falls between the start and end timestamps of the speech segment. When the preset timestamp is less than the start timestamp, it is set as the start timestamp; when the preset timestamp is greater than the end timestamp, it is set as the end timestamp; when the preset timestamp is between the two, it remains unchanged. The adjusted timestamp is the optimized timestamp, and the present invention writes the optimized timestamp back to the timestamp field of the node.

[0082] S354: Verify the optimized timestamps to ensure that the timestamps of the facial expression change nodes and the action execution nodes fall within the time span of the speech segment.

[0083] In step S354, the present invention iterates through each facial expression change node and action execution node, reads the optimized timestamp, and calculates the difference between it and the start and end timestamps of the speech segment. Verification passes when the difference is non-negative and the optimized timestamp does not exceed the end timestamp. When verification fails, the present invention records the abnormal node identifier and timestamp value, marks the node as needing correction, and counts the number of nodes that passed and failed verification.

[0084] S36: The time deviation between multiple expression units in the initial multimodal expression sequence is detected by a time deviation calculation algorithm. When the time deviation exceeds a preset synchronization threshold, the timestamps of the expression change node and the action execution node are corrected to obtain a collaborative multimodal instruction.

[0085] Specifically, in step S36, the present invention first traverses adjacent expression unit pairs, extracts the end timestamp of the speech segment of the previous unit and the start timestamp of the next unit, and calculates the difference between the two as the time interval. The present invention also calculates the deviation between the timestamp of the expression node and the start timestamp of the speech segment within the same unit, as well as the deviation between the timestamp of the action node and the start timestamp of the speech segment. When the absolute value of the deviation exceeds a synchronization threshold of 50 milliseconds, the present invention calculates the ideal offset of the node. The ideal offset is proportionally allocated according to the relative position of the node and the time span of the speech segment, and the node timestamp is corrected to the speech segment start timestamp plus the ideal offset. After correction, the present invention converts the initial multimodal expression sequence into a cooperative multimodal instruction sequence. Each instruction includes an instruction type, target device, parameter, and execution timestamp field, and all instructions are arranged in ascending order of execution timestamp.

[0086] S4: A dynamic priority strategy is used to allocate bandwidth and schedule the execution of the collaborative multimodal instructions, driving the robot hardware to execute the corresponding multimodal expression actions.

[0087] Step S4 further includes: S41: Classify and label the action instructions in the collaborative multimodal instructions to obtain classification and labeling results including key action instructions and non-key action instructions.

[0088] In step S41, this invention filters out all instructions of the action control type and extracts joint names and angle values. Specifically, this invention presets a key joint list including head, eye, and hand joints. When a joint name is in the list or the emotional intensity level is 4 or 5, it is marked as a key action instruction; otherwise, it is marked as a non-key action instruction. This invention adds a priority label to each action instruction, setting key action instructions to high priority and non-key action instructions to low priority. All annotation results are stored in a classification annotation result object containing two lists.

[0089] S42: Based on the classification and labeling results, dynamically allocate communication bandwidth and computing resources according to a preset resource allocation ratio to obtain resource allocation results.

[0090] Furthermore, during allocation, this invention allocates resources at a 3:1 ratio, meaning critical action instructions account for 75% and non-critical action instructions account for 25%. This invention counts the number of both types of instructions and reads the total system bandwidth capacity and total computation time. This invention multiplies the total bandwidth by 0.75 to obtain the bandwidth quota for critical instructions and by 0.25 to obtain the bandwidth quota for non-critical instructions, using the same ratio for computation resources. After calculation, this invention divides each quota by the corresponding number of instructions to obtain the unit bandwidth and unit computation time for each instruction. All allocation results are stored in a mapping table with the instruction identifier as the key and tuples containing bandwidth quota and computation time quota as values.

[0091] S43: Establish an instruction execution queue, insert the critical action instruction into the first preset position of the queue, and insert the non-critical action instruction into the second preset position of the queue, wherein the first preset position has a higher priority than the second preset position.

[0092] In step S43, this invention establishes a priority queue based on a binary heap. The comparison function first compares the priority tags and then compares the execution timestamps. Specifically, this invention traverses the cooperative multimodal instruction sequence, inserts each instruction into the queue, and performs a float operation to maintain the heap properties during insertion. Critical action instructions are located in the first preset position near the root node due to their high priority tags, while non-critical action instructions are located in the second preset position farthest from the root node due to their low priority tags.

[0093] S44: Extract instructions sequentially from the instruction execution queue and send them to the execution unit of the robot hardware to control the robot hardware to perform corresponding actions.

[0094] In step S44, this invention extracts the top element of the heap using a dequeue function, removes it, and then performs a sinking operation to maintain the heap properties. After extracting the instruction, this invention reads the target device field and parameter field, selects the CAN bus communication interface according to the target device, encapsulates the parameters into a CAN data frame containing the device address, instruction code, and parameter values, and sends it. After receiving the data frame, the execution unit parses it and drives the hardware: the voice playback module outputs a sound signal, the expression driving module adjusts the servo angle to change facial expressions, and the joint driving module controls the motor rotation to change limb posture. This invention continues to extract and issue instructions until the queue is empty.

[0095] Step S4 is followed by: S51: Call the music generation interface and the sound effect generation interface to generate background music data and ambient sound effect data; the music generation interface is implemented based on the MusicLM system, and the sound effect generation interface is implemented based on the AutoFoley system; S52: Integrate the background music data and the ambient sound effect data with the multimodal expression action on the timeline to obtain the enhanced multimodal output expression action.

[0096] Furthermore, this invention extracts sentiment type and scene feature vectors from structured feature annotation data, constructing a text prompt in the format "Generate a background music in the style of [sentiment type] and [scene type], with a duration of [duration] seconds". Subsequently, this invention calls the MusicLM system API interface via HTTP protocol, sending a JSON request containing the prompt and duration parameters, and receiving the returned WAV format audio data as background music data.

[0097] For sound effect generation, this invention extracts keywords such as "battle" and "rain" from the scene feature vector, constructs sound effect cue words, and calls the AutoFoley system API to receive the returned environmental sound effect data. Subsequently, this invention analyzes the total duration of the collaborative multimodal instruction sequence and sets the playback start timestamp of the background music data as the execution timestamp of the first instruction. Based on the scene transition flags in the scene feature vector, this invention inserts environmental sound effect playback instructions at the corresponding timestamp positions.

[0098] In addition, this invention creates an audio mixing instruction that aligns the audio data of background music, ambient sound effects and speech segments according to timestamps, sets the volume ratio to 30% for background music, 20% for ambient sound effects and 50% for speech, and the integrated instruction sequence is the enhanced multimodal output expression action.

[0099] This invention also provides a robot emotion expression system based on a large model, which, as an execution subject, executes a robot emotion expression method based on a large model provided by this invention, including: Story content input module: Receives story text (such as AIGC text after a user asks a question) and supports multiple input formats (text, speech-to-text, etc.).

[0100] Story analysis and processing module: Utilizes Large Language Model (LLM) to perform in-depth analysis of story content, extracting character features, emotional features, and scene features through layered decoding.

[0101] Multimodal generation module: Generates a coordinated sequence of speech output, facial expressions, and body movements based on the parsing results.

[0102] Robot control module: Sends the generated multimodal expression commands to the robot hardware to control it to execute corresponding actions. The robot control module adopts a dynamic priority strategy, allocating bandwidth resources through the controller, with critical action commands having higher priority than non-critical commands, and communication latency ≤50ms.

[0103] AIGC auxiliary module: It generates sound effects and background music by calling the MusicLM and AutoFoley systems through API interfaces. The API interfaces support the real-time generation of music styles and sound effect types that match the storyline based on emotional characteristics.

[0104] Interactive feedback module: Supports a response mechanism that allows users to interrupt the story with gestures / voice and generate new plot points. When a user interruption command is detected, the system pauses the current plot and calls the large model to generate a new plot branch.

[0105] This invention provides a multimodal synchronous control method for driving embodied storytelling in robots, which can be used in: children's education scenarios: educational robots help children better understand and remember story content through vivid multimodal expression, improving learning outcomes; family entertainment and companionship scenarios: companion robots enhance emotional connections and interactive stickiness with family members through expressive storytelling; virtual digital human scenarios: algorithms are adapted to screen-displayed characters through API interfaces to achieve emotional storytelling by digital humans; game NPC scenarios: algorithms are integrated into game engines such as Unity to achieve plot-driven NPC interaction, enhancing the game experience.

[0106] This invention uses a large model to parse story text, generate multimodal expressions (voice, facial expressions, actions) that match the story's emotions and character traits in real time, and achieve efficient execution of these expressions on small robot hardware, while supporting real-time interaction and personalized adjustments with users.

[0107] Compared with existing technologies, the present invention has the following significant advantages: This invention makes stories more vivid and interesting through comprehensive expression of voice tone, facial expressions, and body movements, effectively attracting children's attention. Secondly, it enhances the emotional connection and interactive stickiness between users and robots through multimodal interaction and personalized expression. Thirdly, it reduces the reliance on large amounts of labeled data and lowers development costs and time through large-model prompt word engineering and optimized prompt strategies. In addition, it utilizes AIGC technology to automatically generate auxiliary content, significantly improving content production efficiency and shortening output time.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal synchronous control method for driving embodied narratives in robots, characterized in that, include: S1: Receive the text data to be described, and perform standardization processing on the text data to be described to obtain preprocessed text data; S2: Based on the large language model, perform hierarchical decoding on the preprocessed text data to obtain structured feature annotation data; S3: Generate a multimodal expression sequence based on the structured feature annotation data, and perform timestamp alignment and synchronization on the multimodal expression sequence to obtain a collaborative multimodal instruction; S4: A dynamic priority strategy is used to allocate bandwidth and schedule the execution of the collaborative multimodal instructions, driving the robot hardware to execute the corresponding multimodal expression actions.

2. The multimodal synchronous control method for driving embodied narrative in a robot according to claim 1, characterized in that, Step S2 further includes: S21: Perform role recognition on the preprocessed text data to extract a role list including main roles and secondary roles; S22: For each character in the character list, analyze the character features of each character in the character list using a large language model, construct a character feature vector matrix, and obtain the character feature vector; S23: Divide the preprocessed text data into multiple segments, perform sentiment analysis on each segment, identify the sentiment type and quantify the sentiment intensity level to obtain a sentiment feature vector; S24: Extract environmental description information from the preprocessed text data using a scene recognition algorithm to obtain a scene feature vector; S25: Integrate and label the character feature vector, the emotion feature vector, and the scene feature vector to form structured feature annotation data.

3. The multimodal synchronous control method for driving embodied narratives in a robot according to claim 2, characterized in that, Step S23 further includes: S231: Input the segmented text fragments into the sentiment classification model to identify and obtain sentiment type labels; S232: Calculate the emotional intensity value of each text segment using an emotional intensity quantization algorithm, and map the emotional intensity value to the corresponding level according to a preset intensity threshold range to obtain an emotional feature vector.

4. The multimodal synchronous control method for driving embodied narratives in a robot according to claim 3, characterized in that, The emotional intensity grading mapping relationship established in step S232 includes multiple emotional intensity levels, specifically including: When the emotional intensity value is mapped to the first emotional level, the joint swing range is set to ±10 degrees, the voice pitch change rate is ±3%, and the facial expression change speed is 0.3 times the speed corresponding to the first intensity emotional expression parameters; When the emotional intensity value is mapped to the second emotional level, the joint swing range is set to ±15 degrees, the voice pitch change rate is ±5%, and the facial expression change speed is 0.5 times the speed corresponding to the second intensity emotional expression parameters; When the emotional intensity value is mapped to the third emotional level, the corresponding third intensity emotional expression parameters are set as follows: the joint swing range is ±25 degrees, the voice pitch change rate is ±10%, and the facial expression change speed is 1x. When the emotional intensity value is mapped to the fourth emotional level, the joint swing range is set to ±35 degrees, the voice pitch change rate is ±15%, and the facial expression change speed is 1.5 times the normal speed, corresponding to the fourth intensity emotional expression parameters. When the emotional intensity value is mapped to the fifth emotional level, the joint swing range is set to ±45 degrees, the voice pitch change rate is ±20%, and the facial expression change speed is 2 times faster, corresponding to the fifth intensity emotional expression parameters.

5. The multimodal synchronous control method for driving a robot with embodied narrative according to claim 1, characterized in that, Step S3 further includes: S31: Decode the structured feature annotation data based on a hierarchical decoder architecture to output an initial action parameter sequence; wherein, the hierarchical decoder architecture includes an attention LSTM layer and a language generation LSTM layer; S32: Adjust the amplitude of the initial action parameter sequence according to the emotion intensity level in the emotion feature vector to obtain the adjusted optimized action parameter sequence; S33: Match facial expression patterns and action patterns corresponding to the character feature vector and emotional feature vector from the preset expression library to generate a sequence of facial expression changes; S34: The optimized action parameter sequence and the facial expression change sequence are organized in chronological order to obtain an initial multimodal expression sequence including multiple expression units; wherein, each expression unit includes a speech segment, an facial expression change node, and an action execution node; S35: Calculate the start time and duration of the speech segment, and perform consistency alignment processing on the timestamps of the facial expression change node, the action execution node, and the speech segment; S36: The time deviation between multiple expression units in the initial multimodal expression sequence is detected by a time deviation calculation algorithm. When the time deviation exceeds a preset synchronization threshold, the timestamps of the expression change node and the action execution node are corrected to obtain a collaborative multimodal instruction.

6. The multimodal synchronous control method for driving embodied narrative in a robot according to claim 5, characterized in that, Step S31 further includes: S311: Input the structured feature annotation data into the attention LSTM layer, extract key event features and role association features through the attention mechanism, and generate intermediate feature representations; S312: Input the intermediate feature representation into the language generation LSTM layer, and generate the initial action parameter sequence through the recurrent neural network.

7. A multimodal synchronous control method for driving a robot with embodied narrative as described in claim 5, characterized in that, Step S32 further includes: S321: Extract the emotional intensity level value from the emotional feature vector; S322: Retrieve the corresponding expression parameter configuration based on the emotional intensity level value; S323: The expression parameter configuration is applied to the initial action parameter sequence to obtain an optimized action parameter sequence.

8. A multimodal synchronous control method for driving embodied narratives in a robot according to claim 5, characterized in that, Step S35 includes: S351: Extract the start and end timestamps of the speech segment and calculate the time span of the speech segment; S352: Traverse the expression change nodes and the action execution nodes to obtain the preset timestamp of each node; S353: Adjust the preset timestamp according to the time span so that the preset timestamp and the speech segment remain associated on the timeline to obtain an optimized timestamp; S354: Verify the optimized timestamps to ensure that the timestamps of the facial expression change nodes and the action execution nodes fall within the time span of the speech segment.

9. A multimodal synchronous control method for driving a robot with embodied narrative as described in claim 1, characterized in that, Step S4 further includes: S41: Classify and label the action instructions in the collaborative multimodal instructions to obtain classification and labeling results including key action instructions and non-key action instructions; S42: Based on the classification and labeling results, dynamically allocate communication bandwidth and computing resources according to a preset resource allocation ratio to obtain resource allocation results; S43: Establish an instruction execution queue, insert the critical action instruction into the first preset position of the queue, and insert the non-critical action instruction into the second preset position of the queue, wherein the first preset position has a higher priority than the second preset position; S44: Extract instructions sequentially from the instruction execution queue and send them to the execution unit of the robot hardware to control the robot hardware to perform corresponding actions.

10. A multimodal synchronous control method for driving a robot with embodied narrative according to claim 1, characterized in that, Step S4 is followed by: S51: Call the music generation interface and the sound effect generation interface to generate background music data and ambient sound effect data; the music generation interface is implemented based on the MusicLM system, and the sound effect generation interface is implemented based on the AutoFoley system. S52: Integrate the background music data and the environmental sound effect data with the multimodal expression action on the timeline to obtain an enhanced multimodal output expression action.

Images