A humanoid robot interaction system

Abstract

This invention relates to the field of robot interaction technology and discloses an embodied humanoid robot interaction system. The system includes: interaction perception, intent parsing, action generation, and feedback calibration modules. The interaction perception module collects a first interaction signal within a first preset time period and converts it into perceptual features after preprocessing. The intent parsing module identifies the behavioral entity of the object to be interacted with, uses its semantic set as a reference label, and trains a parsing network model by combining reference samples and perceptual features. The action generation module collects a second interaction signal within a second preset time period, extracts the encoder and decoder of the parsing network model, generates instructions based on the encoder, constructs a rule mapping execution unit, and optimizes the mapping relationship through the decoder. The feedback calibration module collects a third interaction signal within a third preset time period, classifies it into regular signals and irregular signals, and processes them through calibration by the mapping execution unit or generation by the encoder, respectively.

Description

Technical Field

[0001] This invention relates to the field of robot interaction technology, specifically to an interactive system for an embodied humanoid robot. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, humanoid robots are gradually moving from the laboratory to practical applications, such as home services, medical assistance, and educational companionship. These robots need to interact with humans in real-time and naturally, the core of which lies in accurately understanding human behavioral intentions and generating responses that conform to the logic of the scenario. However, existing interaction systems still face many challenges in complex and dynamic environments.

[0003] Human interaction behaviors are often diverse and uncertain; the same intention may be expressed through different actions, language, or facial expressions, and environmental interference (such as noise and changes in lighting) further increases the difficulty of signal acquisition. Traditional interaction systems often rely on single-modal signal processing, such as judging intent solely through voice commands or visual gestures, resulting in limited perceptual dimensions and difficulty in handling multimodal interaction scenarios. For example, when a human expresses a need simultaneously through gestures and language, a single-modal system may ignore one of the signals, thus misinterpreting the intent.

[0004] In the intent parsing stage, existing technologies often use predefined rule bases for matching. This approach is effective for fixed scenarios, but its adaptability is extremely poor when faced with unknown scenarios or new interaction patterns. To expand application scenarios, rules need to be manually updated, which is not only inefficient but also fails to cover all possible interaction situations. In addition, some systems have introduced machine learning models for intent recognition, but model training relies on a large amount of labeled data. The dynamic changes in human behavior in actual interactions can lead to insufficient model generalization ability and the phenomenon of "overfitting," meaning that the model can accurately identify samples in the training set but has a large parsing error for new samples.

[0005] The coordination between action generation and feedback calibration is also a weakness of existing systems. In most systems, the action generation module and the intent parsing module operate independently, and the generated actions may be out of sync with the current interaction state. For example, when a human's behavior changes midway, the robot may still execute the preset action according to the initially parsed intent, causing an interruption in the continuity of the interaction. Furthermore, feedback mechanisms often employ simple error correction and lack effective strategies for handling irregular signals. When interaction signals not included in the rule base appear, the system often stagnates or executes incorrect actions, severely impacting the user experience.

[0006] Furthermore, existing interactive systems lack sufficient temporal processing capabilities. Human interaction is a continuous and dynamic process, with signals at different stages exhibiting temporal correlation. However, traditional systems often process the interaction process in independent time segments, neglecting the continuity and dependency of signals over time. For example, in long dialogue scenarios, the robot may forget previous interaction information, leading to subsequent responses that conflict with the context. These issues collectively hinder the realization of truly natural interaction in practical applications for humanoid robots, limiting their widespread application in complex scenarios. Summary of the Invention

[0007] The purpose of this invention is to provide an interactive system for a humanoid robot to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides an interactive system for a humanoid robot, the system comprising:

[0009] The interactive sensing module is used to collect a first interactive signal within a first preset time period; and to preprocess the first interactive signal and convert it into a feature space to obtain the sensing features.

[0010] The intent parsing module is used to identify the behavioral entities of the object to be interacted with, and uses the semantic set corresponding to the behavioral entities as the baseline labels; the parsing network model is trained based on the baseline samples and perceptual features.

[0011] An action generation module is used to acquire a second interaction signal within a second preset time period and perform a generation operation on the second interaction signal, including:

[0012] Extract the encoder and decoder from the parsed network model;

[0013] The encoder encodes the target interaction signal to obtain the generation instructions of the interaction system;

[0014] Based on the second interaction signal and the generated semantic set, a rule mapping execution unit is constructed;

[0015] The mapping relationship of the execution unit is optimized based on the decoder's reverse optimization rules;

[0016] The feedback calibration module is used to acquire a third interactive signal within a third preset time period and perform calibration operations on the third interactive signal, including:

[0017] The third interactive signal is classified into: regular signals and irregular signals.

[0018] If it is a regular signal, the third interaction signal is calibrated based on the mapping execution unit;

[0019] If the signal is irregular, the third interaction signal is generated and processed based on the encoder.

[0020] Preferably, the first interactive signal is preprocessed and converted into a feature space to obtain perceptual features;

[0021] The preprocessing includes: signal filtering to remove environmental noise, electromagnetic interference and invalid frequency bands from the first interactive signal, to obtain a preprocessed signal of the first interactive signal;

[0022] The preprocessed signal is converted into a feature space to obtain the initial perceptual features;

[0023] The initial perceptual features are processed temporally using a one-dimensional temporal neural network to obtain the perceptual features.

[0024] Preferably, an analytical network model is trained based on benchmark samples and perceptual features;

[0025] The parsing network model includes: encoder and decoder;

[0026] Perceptual features are input into the encoder to obtain a generated semantic set, and baseline labels are input into the decoder to determine the matching probability against the generated semantic set;

[0027] If the absolute difference between the mean of P consecutive matching probabilities and the preset stability threshold is less than or equal to the preset tolerance threshold, then the training is stable, and an analytical network model is obtained.

[0028] Preferably, a rule mapping execution unit is constructed based on the second interaction signal and the generated semantic set, including:

[0029] After preprocessing the second interaction signal, the preprocessed signal is used to extract elements based on semantic analysis tools to obtain M interaction elements.

[0030] Extract N semantic entries from the generated semantic set;

[0031] Establish a mapping relationship between M interactive elements and N semantic entries, specifically including:

[0032] Initialize and generate a mapping scheme that meets the constraints; wherein, the mapping scheme includes the mapping relationship between each semantic entry and several interactive elements;

[0033] The constraints include: each semantic entry has at least one mapping relationship with an interaction element, each semantic entry has at most M mapping relationships with interaction elements, and each semantic entry and each interaction element can only have one mapping relationship.

[0034] A rule mapping execution unit is constructed based on the mapping scheme.

[0035] Preferably, the mapping relationship of the rule mapping execution unit is optimized based on the decoder, including determining the evaluation value of the rule mapping execution unit, as follows:

[0036] The rule mapping execution unit generates a corresponding set of rule semantics for the second interaction signal;

[0037] A decoder based on a parsing network model determines the matching probability of a set of rule semantics;

[0038] The absolute difference between the matching probability and the preset stable threshold is used as the evaluation value of the rule mapping execution unit;

[0039] If the absolute difference between the evaluated value and the preset stability threshold is greater than the preset tolerance threshold, the mapping relationship of the mapping scheme is updated based on the gradient adjustment method until the absolute difference between the evaluated value and the preset stability threshold is less than or equal to the preset tolerance threshold.

[0040] Preferably, the third interaction signal is classified into: regular signals and irregular signals, including:

[0041] If the absolute difference between the evaluation value of the rule mapping execution unit and the preset stability threshold is less than or equal to the preset tolerance threshold, then the corresponding second interaction signal is used as the rule signal.

[0042] Both the second and third interaction signals were preprocessed, feature extracted, and intent labeled.

[0043] Among them, intent annotation is based on the intent annotation tool to obtain the intent type of each interaction element corresponding to the second and third interaction signals;

[0044] Based on the temporal order of the interaction elements in the second and third interaction signals, corresponding intent sequences are constructed respectively; wherein, the i-th unit of the intent sequence represents the intent type of the i-th interaction element in the second or third interaction signal;

[0045] Calculate the similarity value of the corresponding intent sequences, specifically including:

[0046] Load a sliding detection window to convert the intent sequence into sequence features of dimension H; the length of the sliding detection window is dynamically set based on the dimension of the intent sequence.

[0047] Calculate the similarity value of the intent combination of the h-th unit in the corresponding sequence features; where the intent combination is extracted from the intent sequence based on the sliding detection window;

[0048] If the similarity value is greater than or equal to the preset similarity threshold, the third interaction signal is classified as a regular signal; otherwise, the third interaction signal is classified as an irregular signal.

[0049] Preferably, the similarity value of the intention combination of the h-th unit in the corresponding sequence features is calculated, including:

[0050] Take the intention combination of the h-th unit in the sequence features corresponding to the second interaction signal and the intention combination of the h-th unit in the sequence features corresponding to the third interaction signal, count the number of the same intention types between the two, and the ratio of the number of the total number of intention types between the two is the similarity value.

[0051] Preferably, the feature space is a spatiotemporal feature space.

[0052] Preferably, the generation and processing of the third interaction signal based on the encoder includes:

[0053] The perceptual features of the third interaction signal are input into the encoder, and deep features are extracted through the coding layer;

[0054] Deep features are input into a fully connected layer for dimensionality transformation to obtain action control parameters;

[0055] Generate joint movement commands for the humanoid robot based on motion control parameters.

[0056] Preferably, the calibration processing of the third interaction signal based on the mapping execution unit includes:

[0057] The interaction elements of the third interaction signal are matched with the mapping relationship of the mapping execution unit to obtain the corresponding semantic entries;

[0058] Based on semantic entries, a preset action template is invoked to generate initial motion instructions;

[0059] The feedback signal after the humanoid robot executes the initial motion command is collected, the parameter deviation of the initial motion command is adjusted, and the calibrated motion command is obtained.

[0060] Compared with the prior art, the beneficial effects of the present invention are:

[0061] This system, through a multi-module collaborative design, effectively solves the core problems of embodied humanoid robots in complex interaction scenarios. The interaction perception module employs multimodal signal acquisition and feature space transformation, integrating various interaction signals (such as visual movements, voice tone, and body posture) within a first preset time period. After preprocessing, it forms comprehensive perceptual features, overcoming the limitations of traditional single-modal signal processing. This processing method can cover multi-dimensional expressions in human interaction, and even in the presence of environmental interference, it can retain key information through feature space mapping, reducing the possibility of signal loss or misjudgment.

[0062] The intent parsing module introduces a mechanism that associates behavioral entities with semantic sets. Using the behavioral entities of the object to be interacted with as baseline labels, and combining baseline samples with perceptual features, it trains a parsing network model, enabling the model to capture core intents from dynamically changing interaction signals. Compared to traditional predefined rule bases, this model can adapt to new interaction patterns through sample learning without human intervention. Especially in unknown scenarios, it can autonomously expand the parsing scope through feature association, avoiding the problem of lagging rule base updates. Simultaneously, the semantic mapping based on behavioral entities makes intent parsing closer to the logical habits of human interaction, reducing misunderstandings caused by semantic ambiguity.

[0063] The action generation module achieves dynamic conversion between interactive signals and generated instructions through the collaborative work of the encoder and decoder. Within a second preset time period, system instructions are generated based on the encoder's encoding result of the target signal. These instructions are then combined with the second interactive signal to construct a rule-mapping execution unit. The decoder then optimizes the mapping relationship in reverse, ensuring that the generated actions not only conform to the current interactive intent but also maintain continuity with historical interactive states. This design breaks down the separation between action generation and intent parsing, ensuring the robot's response actions have continuity in the temporal dimension. For example, in multi-turn dialogues or continuous action interactions, subsequent actions can be adjusted based on preceding information, avoiding logical gaps.

[0064] The feedback calibration module classifies and processes third-party interaction signals, providing differentiated processing strategies for different signal types. For regular signals, calibration is performed using an optimized mapping execution unit to ensure the stability of routine interactions; for irregular signals, the encoder regenerates processing instructions, enabling the system to flexibly respond to sudden or unexpected interaction behaviors. This classification processing mechanism avoids the limitations of traditional feedback relying solely on a single correction method, maintaining smooth interaction even in the face of sudden human actions or environmental changes, and reducing system stagnation or erroneous responses.

[0065] The system completes signal acquisition and processing within different preset time periods through the sequential coordination of four modules: perception, analysis, generation, and calibration. This forms a closed-loop interactive link, enabling the embodied humanoid robot to achieve more natural and intelligent human-computer interaction in complex dynamic environments and adapt to diverse practical application scenarios. Attached Figure Description

[0066] Figure 1 This is a schematic diagram illustrating the working principle of the humanoid robot interaction system described in this invention.

[0067] Figure 2 A flowchart for signal preprocessing and timing processing;

[0068] Figure 3 A flowchart for analyzing the training process of a network model;

[0069] Figure 4 A flowchart for constructing the rule mapping execution unit;

[0070] Figure 5 This is a flowchart for signal classification and processing. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] Please see Figures 1-5 This invention provides an interactive system for an embodied humanoid robot. The system includes an interaction perception module, an intent parsing module, an action generation module, and a feedback calibration module. These modules work collaboratively to achieve dynamic interaction between the robot and objects. Specific implementation details are as follows:

[0073] The interactive perception module acquires a first interactive signal within a first preset duration, preprocesses it, and converts it into a feature space to obtain perceptual features. The first preset duration can be set according to the actual interaction scenario, for example, 5 seconds. During this period, sensors (such as visual sensors, voice sensors, and tactile sensors) acquire signals such as the actions, voice, and touch of the object to be interacted with as the first interactive signal. The preprocessing stage filters the acquired signals, removing environmental noise (such as background noise and light interference), electromagnetic interference (such as electromagnetic signals generated by device operation), and invalid frequency bands (such as signals outside the frequency range of human actions or voice), resulting in a preprocessed signal. This preprocessed signal is then converted into a feature space to form initial perceptual features. A one-dimensional temporal neural network then performs temporal processing on the initial perceptual features, ultimately obtaining perceptual features containing time-series information.

[0074] The intent parsing module first identifies the behavioral entities of the object to be interacted with, such as the entities corresponding to the object's body movements and speech content, and uses the semantic set corresponding to the behavioral entity as the baseline label. For example, the semantic set corresponding to "waving" may include "greeting" and "signaling attention". Based on the baseline samples and the aforementioned perceptual features, a parsing network model is trained. This model includes an encoder and a decoder. During training, the perceptual features are input into the encoder to obtain the generated semantic set, and the baseline labels are input into the decoder to determine the matching probability of the generated semantic set. If the absolute difference between the mean of the matching probabilities of P consecutive times (P can be set to 10) and a preset stability threshold is less than or equal to a preset tolerance threshold, the training is considered stable, and the parsing network model is obtained.

[0075] The action generation module acquires the second interaction signal within a second preset duration (e.g., 3 seconds) and executes the generation task. First, it extracts the encoder and decoder from the parsing network model, using the encoder to encode the target interaction signal to obtain the generation instruction. Next, it constructs a rule mapping execution unit based on the second interaction signal and the generated semantic set: after preprocessing the second interaction signal, it extracts M interaction elements (such as the amplitude and frequency of the action, keywords in the speech, etc.) using semantic analysis tools; it extracts N semantic entries from the generated semantic set; and it initializes a mapping scheme that meets the constraints, including that each semantic entry maps to at least one interaction element, maps to at most M interaction elements, and each mapping relationship is unique. Based on this scheme, it constructs the rule mapping execution unit. Then, it optimizes the mapping relationship through the decoder: it calculates the matching probability of the rule semantic set generated by the rule mapping execution unit, and uses the absolute difference between this probability and a preset stability threshold as the evaluation value. If the evaluation value exceeds the preset tolerance threshold, it updates the mapping scheme using a gradient adjustment method until the evaluation value meets the requirements.

[0076] The feedback calibration module collects the third interaction signal and performs calibration within a third preset duration (e.g., 2 seconds). First, the third interaction signal is classified into regular and irregular signals: if the evaluation value of the regular mapping execution unit meets the requirements, the corresponding second interaction signal is a regular signal. After preprocessing and extracting elements from the second and third interaction signals, the intent type of each interaction element is obtained through an intent annotation tool. Then, an intent sequence is constructed based on the temporal order of the interaction elements, where the i-th unit of the sequence corresponds to the intent type of the i-th interaction element. A sliding detection window (the window length is dynamically set according to the dimension of the intent sequence) is loaded to convert the intent sequence into a sequence feature of dimension H. The similarity value of the intent combination of the h-th unit in the corresponding sequence feature (i.e., the ratio of the number of identical intent types to the total number) is calculated. If the similarity value is greater than or equal to a preset threshold, the third interaction signal is a regular signal; otherwise, it is an irregular signal. For regular signals, calibration is performed based on the mapping execution unit: the interaction elements of the third interaction signal are matched with the mapping relationship to obtain semantic entries, a preset action template is called to generate initial motion instructions, feedback signals after robot execution are collected to adjust parameter deviations, and calibrated instructions are obtained. For irregular signals, encoder-based processing is used: the perceptual features of the third interaction signal are input into the encoder, deep features are extracted through the coding layer, and then the dimensions are transformed through the fully connected layer to obtain motion control parameters. Based on these parameters, joint motion commands are generated.

[0077] Example 1:

[0078] This embodiment focuses on the preprocessing and sensory feature acquisition of the first interactive signal. Appropriate processing strategies must be adopted based on the different types of signals to ensure the accuracy of subsequent feature extraction. The first interactive signal may include various types such as visual signals, speech signals, and tactile signals. The operational focus in the preprocessing stage differs for different types of signals.

[0079] For visual signals, which are primarily acquired through devices such as cameras and depth sensors, information such as the body movements and facial expressions of the object to be interacted with is included. The signal filtering in the preprocessing stage needs to focus on removing environmental noise. For example, in scenes with complex lighting, there may be interference from direct sunlight, alternating shadows, etc. Multi-frame image fusion algorithms can be used to average the continuously acquired image frames, weakening the impact of instantaneous light and shadow changes. For electromagnetic interference generated by the device itself, such as high-frequency noise generated by the sensor circuitry, wavelet thresholding can be used to decompose the image signal at multiple scales, filter out noisy coefficients, and perform thresholding to retain effective signal components. Furthermore, visual signals may contain invalid frequency bands, such as pixel changes exceeding the normal human movement speed range (e.g., rapidly flashing lights). A speed threshold can be set to filter out pixel regions with movement speeds exceeding the threshold, resulting in a preprocessed visual signal.

[0080] For speech signals, which are collected through devices such as microphones, information such as the speaker's language content and intonation is included. During preprocessing, environmental noise mainly manifests as background noise (such as the sound of air conditioners running or other people talking). An adaptive noise cancellation algorithm can be used to collect environmental reference noise and generate inverse sound waves in real time to cancel the background noise. Electromagnetic interference may come from electromagnetic radiation generated by nearby electronic devices, causing periodic current noise in the speech signal. This can be filtered by notch filters targeting specific interference frequencies (such as 50Hz or 60Hz). Invalid frequency bands include infrasound below 20Hz and ultrasound above 20kHz. These signals, which are outside the range of human speech, can be filtered by a combination of high-pass and low-pass filters, retaining the effective speech signal in the 20Hz-20kHz range to form the preprocessed speech signal.

[0081] For tactile signals, they are collected by pressure sensors on the robot's surface, reflecting information such as the touch force and contact position. During preprocessing, environmental noise may be thermal noise from the sensor itself, which can be smoothed by mean filtering to reduce random fluctuations in continuous pressure values. Electromagnetic interference may cause abnormal jumps in the pressure signal, which can be eliminated by setting an amplitude threshold to remove jump signals that exceed the normal touch force range. Invalid frequency bands include low-frequency drift outside the sensor's response range (such as baseline shift caused by temperature changes), which can be removed by a high-pass filter to remove slowly changing baseline drift, resulting in a preprocessed tactile signal.

[0082] When converting preprocessed signals into feature space, an appropriate conversion method must be selected based on the signal type. For example, visual signals can be converted into spatial feature vectors based on pixel coordinates by extracting spatial features such as edges and textures through convolution operations; speech signals can be converted into frequency domain features through Mel-frequency cepstral coefficients (MFCC) conversion, forming spectral feature vectors; tactile signals can be converted into multidimensional feature vectors by extracting feature parameters such as pressure peaks and contact area. These feature vectors together constitute the initial perceptual features.

[0083] When performing temporal processing on initial perceptual features using a one-dimensional temporal neural network, the network structure can adopt an architecture consisting of an input layer, multiple LSTM layers, and an output layer. The input layer receives the initial perceptual features, and the LSTM layers capture the temporal information within the features through gating mechanisms (input gate, forget gate, output gate). For example, for continuous limb movements, it can remember the previous movement trends (such as the direction in which the arm begins to rise) and combine this with the current movement state (such as the angle at which the arm is raised) to form a feature representation that includes temporal correlation. For speech signals, it can capture the temporal changes in intonation (such as the rising pitch of an interrogative tone), concatenating discrete spectral features into a temporally coherent feature sequence. After processing by the LSTM layers, the output layer outputs the final perceptual features, which contain both spatial features of the signal and integrate temporal series information, enabling a more comprehensive reflection of the dynamic changes during the interaction process.

[0084] Example 2:

[0085] The training process of a parsing network model relies on a large number of benchmark samples and perceptual features. Through the collaborative operation of the encoder and decoder, the model parameters are gradually optimized to achieve stable output. The selection of benchmark samples needs to cover diverse interaction scenarios, including signal data under different environmental conditions (such as bright indoors, dimly lit corridors, and noisy public places), different interaction objects (such as adults, children, and the elderly), and different interaction methods (such as fast body movements, slow gestures, clear speech, and unclear pronunciation). For example, for the behavior of "waving," the benchmark samples should include signals generated by different speeds (fast waving, slow waving), different amplitudes (small amplitude waving, large amplitude waving), and different body parts (single-hand waving, two-hand waving), while also covering data collected in different background environments (such as backgrounds with multiple people walking around or scenes with continuous background noise) to ensure the comprehensiveness and representativeness of the samples.

[0086] The encoder of the parsing network model employs a multi-layer convolutional neural network structure, with each layer containing convolutional layers, batch normalization layers, and activation functions. The convolutional layers use sliding convolution operations on the input perceptual features with convolutional kernels of different sizes (e.g., 3×3, 5×5) to extract key local information from the features, such as changes in the direction of body movements and spectral peaks of speech signals. The batch normalization layers standardize the convolutional features to stabilize their distribution. The ReLU activation function is used to enhance the model's ability to fit non-linear features. The encoder's output layer transforms the extracted features into a semantic set through a fully connected layer. This set is represented as a vector, with each element corresponding to a probability value for a semantic category, such as the probability distribution of semantics like "greeting," "saying goodbye," and "gesturing over."

[0087] The decoder employs a Transformer architecture, consisting of multiple encoder layers and one decoder layer. Each encoder layer includes a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism performs global correlation analysis on the baseline label and the generated semantic set, calculating the similarity between different semantics, such as the semantic correlation between "waving" and "greeting." The feedforward neural network performs a non-linear transformation on the features output by the attention mechanism, further enhancing the accuracy of semantic matching. The decoder takes the baseline label (represented as a one-hot vector) and the generated semantic set as input. By calculating the cross-entropy loss between the two, the matching probability is obtained, which reflects the degree of agreement between the generated semantic set and the baseline label.

[0088] During training, the preset stability threshold is determined based on the consistency of annotations on the benchmark samples. For example, when more than 80% of annotators agree on the semantics of a certain behavioral entity, the probability value corresponding to this consistency ratio (e.g., 0.8) can be set as the preset stability threshold. The preset tolerance threshold is set according to the actual interaction accuracy requirements, such as 0.05, which allows a deviation of ±0.05 between the mean of the matching probability and the stability threshold. In each training iteration, the perceptual features are input into the encoder to generate a semantic set, and the benchmark labels are input into the decoder to calculate the matching probability. The matching probabilities are recorded P times (e.g., 10 times) and the mean is calculated. If the absolute difference between the mean and the preset stability threshold is ≤0.05 for 10 consecutive times, the model training is considered to have reached a stable state, training is stopped, the current model parameters are saved, and the final parsed network model is obtained.

[0089] During training, if the difference between the average matching probability of P consecutive iterations and the stability threshold exceeds the tolerance threshold, the model parameters need to be adjusted and training continued. Adjustment methods include increasing the number of training iterations, adjusting the learning rate (e.g., reducing the learning rate when the loss decreases slowly), optimizing the convolutional kernel size, or the number of attention heads in the Transformer. Through this training mechanism, the parsing network model can gradually learn the mapping relationship between perceptual features and semantic sets, enabling the semantic sets generated by the encoder to accurately reflect the behavioral intent of the object to be interacted with, and allowing the decoder to stably verify the generated semantic sets.

[0090] Example 3:

[0091] The process of constructing the rule mapping execution unit requires combining the second interaction signal with the generated semantic set, and establishing a stable mapping relationship through multi-step processing. The preprocessing procedure for the second interaction signal is consistent with that for the first interaction signal, and corresponding filtering measures are taken according to the signal type. For example, if the second interaction signal is a voice signal, background noise (such as the sound of indoor electrical appliances running), electromagnetic interference (such as high-frequency signals generated by nearby equipment), and invalid frequency bands (such as low-frequency signals below 20Hz) need to be removed; if it is a body movement signal, light and shadow interference (such as the flickering of light spots shining through the window) and abnormal signals beyond the range of human movement (such as rapid non-human movements) are filtered to ensure that the preprocessed signal can accurately reflect the true intention of the interaction object.

[0092] The choice of semantic analysis tools must be adapted to the type of the second interaction signal. For speech-based second interaction signals, a Transformer-based natural language processing model can be used to extract M interaction elements through word segmentation, part-of-speech tagging, and entity recognition, such as verbs and nouns like "take," "cup," and "table," as well as time adverbs like "now" and "immediately." For body movement signals, a skeletal keypoint detection algorithm can be used to extract interaction elements of the movement, such as the angle of arm swing, the degree of hand opening and closing, and the duration of the movement. These interaction elements must have clear physical or semantic meanings and can serve as the basic units for mapping relationships.

[0093] The N semantic entries in the generated semantic set are determined by the encoder output of the parsing network model, and each entry corresponds to a possible interaction intent. For example, when the second interaction signal is "pointing to the water glass and saying 'Please bring it here'", the generated semantic set may contain N semantic entries such as "requesting to take the item", "specifying the target as the water glass", and "demanding immediate execution". These entries need to cover various intents that may be involved in the interaction process and have a certain degree of independence from each other to avoid semantic overlap that could lead to mapping confusion.

[0094] When initializing the mapping scheme, three constraints must be strictly followed. The first constraint ensures that each semantic entry is mapped to at least one interaction element, avoiding isolated semantic entries without corresponding elements. The second constraint limits each semantic entry to a maximum of M interaction elements, preventing complex mapping relationships caused by associating a single semantic entry with too many elements. The third constraint guarantees that there is only one mapping between each semantic entry and each interaction element, avoiding logical conflicts caused by duplicate mappings. For example, when M=4 (interaction elements are A, B, C, D) and N=2 (semantic entries are X, Y), the initial mapping scheme might be X mapped to A and B, and Y mapped to C and D, which satisfies all constraints.

[0095] In establishing the mapping relationship, it is necessary to calculate the relevance between each interactive element and the semantic entry. The formula for calculating the relevance is as follows:

[0096] ;

[0097] In the formula, k is the index variable of the interactive element, and its value ranges from 1 to M (M is the total number of interactive elements extracted from the second interactive signal), which is used to traverse all M interactive elements to complete the summation operation. This represents the degree of association between the i-th semantic item and the j-th interactive element. This represents the co-occurrence frequency of the i-th semantic entry and the j-th interactive element (i.e., the number of times they appear simultaneously in the same interaction process). This represents the sum of co-occurrence frequencies of the i-th semantic item and all M interactive elements. This formula yields the correlation degree between each semantic item and each interactive element; a higher correlation degree indicates a stronger connection between the two during the interaction process.

[0098] Based on the relevance ranking, interaction elements are selected and mapped for each semantic entry from highest to lowest relevance until the constraints are met. For example, for semantic entry X, if its relevance to interaction element A is the highest, followed by B and C, then X is mapped to A and B first. If the constraint of "at least one and at most M" is already met, then adding mapping elements to X stops. Interaction elements are assigned to other semantic entries in the same way, ultimately forming a complete mapping scheme.

[0099] The rule mapping execution unit is built around this mapping scheme, storing the mapping relationship between semantic entries and interactive elements through data structures (such as hash tables). The keys are semantic entries, and the values ​​are the corresponding sets of interactive elements. This execution unit needs to have fast query capabilities; when a new interactive element is input, it should be able to quickly match the corresponding semantic entry, providing a rule basis for subsequent action generation. Simultaneously, the mapping scheme needs to be stored in an extensible format to facilitate adjustments during subsequent reverse optimization via the decoder, ensuring that the rule mapping execution unit can adapt to different interaction scenarios.

[0100] Example 4:

[0101] The mapping relationship of the rule mapping execution unit, which is based on the decoder's reverse optimization, needs to be adjusted step by step through multiple operations to achieve the accuracy of the mapping relationship. First, the rule mapping execution unit generates a set of rule semantics for the second interaction signal. This process relies on the constructed mapping scheme to match the interaction elements in the second interaction signal with the corresponding semantic entries, combining them to form a complete semantic expression. For example, if the second interaction signal contains interaction elements such as "arm extended forward" or "finger pointing to a water cup", the rule mapping execution unit will integrate the corresponding semantic entries such as "indicating an item" or "requesting to take" according to the mapping relationship to generate a set of rule semantics.

[0102] The decoder of the parsing network model then determines the matching probability of the rule semantic set based on the degree of similarity between the rule semantic set and the baseline label. The baseline label is a standard semantic set determined based on the behavioral entity of the object to be interacted with. For example, the baseline label for "pointing to the water cup and issuing a take command" might include fixed items such as "target is water cup" and "action requirement is take". The decoder calculates the matching probability by comparing the overlap and logical coherence of each semantic item in the rule semantic set with those in the baseline label. For example, if 80% of the semantic items in the rule semantic set are consistent with the baseline label and the logical order is correct, the matching probability might be 0.8.

[0103] The evaluation value is determined based on the absolute difference between the matching probability and the preset stability threshold. The preset stability threshold is derived from the statistical analysis of the matching probabilities between the baseline label and the correct semantic set in historical interaction data. Assuming that the average matching probability between the correct semantic set and the baseline label is 0.85 in a large number of samples, the preset stability threshold can be set to 0.85. The preset tolerance threshold is set according to the accuracy requirements of the interaction scenario, such as 0.05, which allows the evaluation value to fluctuate within the range of 0.85 ± 0.05. If the matching probability of the rule semantic set is 0.82, and the absolute difference between it and 0.85 is 0.03, which is less than 0.05, then the evaluation value meets the requirements. If the matching probability is 0.79, and the difference is 0.06, which exceeds the tolerance threshold, then the reverse optimization process needs to be initiated.

[0104] Reverse optimization uses a gradient adjustment method to update the mapping relationship of the mapping scheme. The core of the gradient adjustment method is to adjust the mapping weights of interactive elements and semantic entries based on the direction of deviation between the evaluation value and the tolerance threshold. For example, when the evaluation value exceeds the tolerance threshold, the decoder will locate the semantic entries in the rule semantic set that do not match the baseline label, analyze the interactive elements corresponding to the entries, and if it is found that the association between a certain interactive element and the semantic entry is weak (such as the "wrist rotation" element being incorrectly associated with the "request to take" semantic), the mapping weight between the element and the semantic entry will be reduced, while the weight of elements with a higher degree of association with the semantic entry (such as "pointing finger") will be increased.

[0105] During the adjustment process, it is necessary to ensure that the mapping relationship still meets the preset constraints. For example, if a semantic entry was originally associated with 3 interactive elements, the adjustment must ensure that the number of associated elements remains within the range of 1 to M, and that the mapping relationship between each element and the semantic entry is unique. Assuming M is 5, and a semantic entry was originally associated with A, B, and C, the adjusted elements can be reduced to A and B, or increased to A, B, C, and D, but cannot be less than 1 or more than 5, while avoiding duplicate mappings between A and the semantic entry.

[0106] After each adjustment, the rule mapping execution unit regenerates the rule semantic set based on the new mapping scheme, and the decoder recalculates the matching probability and evaluation value. This process is repeated until the absolute difference between the evaluation value and the preset stability threshold is less than or equal to the preset tolerance threshold. For example, after multiple adjustments, if the matching probability of the rule semantic set increases to 0.83, and the difference between it and 0.85 is 0.02, which meets the tolerance requirement, then optimization stops, and the current mapping relationship is saved.

[0107] During the reverse optimization process, for semantic entries involving multiple interaction elements, the overall matching degree between the element combination and the semantic entry needs to be evaluated. For example, the semantic entry "request assistance" may be associated with multiple elements such as "waving", "saying 'help'", and "pointing to a difficult location". If adjusting the weight of only one element cannot achieve the target evaluation value, the weights of multiple elements need to be adjusted simultaneously, such as increasing the weight of "saying 'help'" and decreasing the weight of "waving", until the matching degree between the element combination and the semantic entry meets the requirements.

[0108] Example 5:

[0109] The classification and calibration of the third interaction signal must be carried out step by step according to the process to achieve accurate response to the interaction signal. The premise of classification is to determine the benchmark of the rule signal. That is, when the absolute difference between the evaluation value of the rule mapping execution unit and the preset stability threshold is less than or equal to the preset tolerance threshold, the corresponding second interaction signal is marked as a rule signal and used as a reference standard for subsequent classification.

[0110] The preprocessing of the second and third interaction signals follows the same approach as the previous signal processing, removing environmental noise, electromagnetic interference, and invalid frequency bands based on the signal type. For example, if the signal is speech, background noise and electromagnetic interference from equipment need to be filtered out; if it is a body movement signal, changes in lighting and abnormal data exceeding the normal range of movement are removed. The preprocessed signals then enter the element extraction stage, where semantic analysis tools are used to extract interaction elements, such as keywords in speech and the angle and speed of body movements.

[0111] The intent annotation tool labels each interaction element with an intent type based on a pre-defined intent type library. This library covers common interaction intents such as "attract attention," "request an action," "refuse a command," and "indicate a target." For example, the interaction element "raising an arm to chest level" might be labeled "attract attention"; "pointing a finger to a water glass on the table" might be labeled "indicate a target"; and the spoken phrase "no need" might be labeled "refuse a command." Each interaction element corresponds to a unique intent type, ensuring the clarity of the annotation results.

[0112] When constructing the intent sequence, arrange them according to the chronological order in which the interaction elements appear. For example, if the interaction elements of the second interaction signal are "raising arm", "pointing to the water glass", and "saying 'get it'", and the corresponding intent types are "attracting attention", "indicating a target", and "requesting an action", then its intent sequence is [attracting attention, indicating a target, requesting an action]. If the third interaction signal includes "waving", "pointing to the bookshelf", and "saying 'get the book'", the corresponding intent sequence is [attracting attention, indicating a target, requesting an action].

[0113] The length of the sliding detection window is dynamically set according to the dimensions of the intent sequence. If the intent sequence contains 5 units, the window length can be set to 2 or 3; if the sequence contains 8 units, the window length can be set to 3 or 4. The window slides across the intent sequence, dividing the sequence into multiple overlapping subsequences, each of which serves as a unit of the sequence feature. For example, if the intent sequence is [attention, target indication, request action, wait for response], and the window length is set to 2, the generated sequence feature is [[attention, target indication], [target indication, request action], [request action, wait for response]], where each subsequence is a unit of the sequence feature.

[0114] When calculating the similarity value, the intent combination of each unit in the corresponding sequence features of the second and third interaction signals is compared. For example, if the first unit of the second interaction signal sequence feature is [attention-seeking, target indication], and the first unit of the third interaction signal sequence feature is also [attention-seeking, target indication], then the number of identical intent types is 2, the total number of intent types is 2, and the similarity value is 1.0. If the first unit of the third interaction signal sequence feature is [attention-seeking, rejection instruction], then the number of identical intent types is 1, the total number is 2, and the similarity value is 0.5. The final similarity value is obtained by averaging the similarity values ​​of all units.

[0115] The preset similarity threshold is set based on the stability requirements of the interaction scenario. If the interaction mode in the scenario is relatively fixed, the threshold can be set to 0.8; if there are many variable interaction methods in the scenario, the threshold can be set to 0.6. When the similarity value between the third interaction signal and the second interaction signal is greater than or equal to the preset threshold, it is determined to be a regular signal; otherwise, it is determined to be an irregular signal.

[0116] For rule-based signals, calibration is performed through a mapping execution unit. The interaction elements of the third interaction signal are matched with the mapping relationships in the mapping execution unit to obtain the corresponding semantic entries, such as matching "indicating a target + requesting an operation" to the semantic entry "requesting to take a specified item". Preset motion templates are invoked to generate initial motion commands, such as commands to move the robotic arm to a specified position or to grasp an item. Feedback signals after the robot executes the commands are collected, such as the deviation between the actual position and the target position of the robotic arm, and the magnitude of the grasping force. Based on the feedback signals, the parameters of the initial motion commands are adjusted, such as correcting joint rotation angles and adjusting the grasping force, to obtain calibrated motion commands.

[0117] For irregular signals, generation and processing are performed based on an encoder. The perceptual features of the third interaction signal are input into the encoder, and the encoding layer extracts deep features through multi-layer processing, such as trajectory features of irregular movements and spectral features of special speech. The fully connected layer performs dimensionality transformation on the deep features to obtain parameters corresponding to robot joint control, such as the rotation angle, speed, and duration of each joint. Based on these parameters, joint motion commands are generated to drive the robot to perform actions adapted to irregular interaction scenarios.

[0118] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0119] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A humanoid robot interaction system, characterized in that, include: The interactive sensing module is used to collect the first interactive signal within a first preset time period; The first interactive signal is preprocessed and converted into a feature space to obtain the perceptual features; The intent parsing module is used to identify the behavioral entities of the object to be interacted with, and uses the semantic set corresponding to the behavioral entities as the base label. An analytical network model is trained based on benchmark samples and perceptual features; An action generation module is used to acquire a second interaction signal within a second preset time period and perform a generation operation on the second interaction signal, including: Extract the encoder and decoder from the parsed network model; The encoder encodes the target interaction signal to obtain the generation instructions of the interaction system; Based on the second interaction signal and the generated semantic set, a rule mapping execution unit is constructed; The mapping relationship of the execution unit is optimized based on the decoder's reverse optimization rules; The feedback calibration module is used to acquire a third interactive signal within a third preset time period and perform calibration operations on the third interactive signal, including: The third interactive signal is classified into: regular signals and irregular signals. If it is a regular signal, the third interaction signal is calibrated based on the mapping execution unit; If the signal is irregular, the third interaction signal is generated and processed based on the encoder.

2. The humanoid robot interaction system according to claim 1, characterized in that, The first interactive signal is preprocessed and converted into a feature space to obtain the perceptual features; The preprocessing includes: signal filtering to remove environmental noise, electromagnetic interference and invalid frequency bands from the first interactive signal, to obtain a preprocessed signal of the first interactive signal; The preprocessed signal is converted into a feature space to obtain the initial perceptual features; The initial perceptual features are processed temporally using a one-dimensional temporal neural network to obtain the perceptual features.

3. The humanoid robot interaction system according to claim 2, characterized in that, An analytical network model is trained based on benchmark samples and perceptual features; The parsing network model includes: encoder and decoder; Perceptual features are input into the encoder to obtain a generated semantic set, and baseline labels are input into the decoder to determine the matching probability against the generated semantic set; If the absolute difference between the mean of P consecutive matching probabilities and the preset stability threshold is less than or equal to the preset tolerance threshold, then the training is stable, and an analytical network model is obtained.

4. The humanoid robot interaction system according to claim 3, characterized in that, Based on the second interaction signal and the generated semantic set, a rule mapping execution unit is constructed, including: After preprocessing the second interaction signal, the preprocessed signal is used to extract elements based on semantic analysis tools to obtain M interaction elements. Extract N semantic entries from the generated semantic set; Establish a mapping relationship between M interactive elements and N semantic entries, specifically including: Initialize and generate a mapping scheme that meets the constraints; wherein, the mapping scheme includes the mapping relationship between each semantic entry and several interactive elements; The constraints include: each semantic entry has at least one mapping relationship with an interaction element, each semantic entry has at most M mapping relationships with interaction elements, and each semantic entry and each interaction element can only have one mapping relationship. A rule mapping execution unit is constructed based on the mapping scheme.

5. The humanoid robot interaction system according to claim 4, characterized in that, The mapping relationship of the rule mapping execution unit is optimized based on the decoder, including determining the evaluation value of the rule mapping execution unit, as follows: The rule mapping execution unit generates a corresponding set of rule semantics for the second interaction signal; A decoder based on a parsing network model determines the matching probability of a set of rule semantics; The absolute difference between the matching probability and the preset stable threshold is used as the evaluation value of the rule mapping execution unit; If the absolute difference between the evaluated value and the preset stability threshold is greater than the preset tolerance threshold, the mapping relationship of the mapping scheme is updated based on the gradient adjustment method until the absolute difference between the evaluated value and the preset stability threshold is less than or equal to the preset tolerance threshold.

6. The humanoid robot interaction system according to claim 5, characterized in that, The third interactive signal is classified into: regular signals and irregular signals, including: If the absolute difference between the evaluation value of the rule mapping execution unit and the preset stability threshold is less than or equal to the preset tolerance threshold, then the corresponding second interaction signal is used as the rule signal. Both the second and third interaction signals were preprocessed, feature extracted, and intent labeled. Among them, intent annotation is based on the intent annotation tool to obtain the intent type of each interaction element corresponding to the second and third interaction signals; Based on the temporal order of the interaction elements in the second and third interaction signals, corresponding intent sequences are constructed respectively; wherein, the i-th unit of the intent sequence represents the intent type of the i-th interaction element in the second or third interaction signal; Calculate the similarity value of the corresponding intent sequences, specifically including: Load a sliding detection window to convert the intent sequence into sequence features of dimension H; the length of the sliding detection window is dynamically set based on the dimension of the intent sequence. Calculate the similarity value of the intent combination of the h-th unit in the corresponding sequence features; where the intent combination is extracted from the intent sequence based on the sliding detection window; If the similarity value is greater than or equal to the preset similarity threshold, the third interaction signal is classified as a regular signal; otherwise, the third interaction signal is classified as an irregular signal.

7. The humanoid robot interaction system according to claim 6, characterized in that, Calculate the similarity value of the intent combination of the h-th unit in the corresponding sequence features, including: Take the intention combination of the h-th unit in the sequence features corresponding to the second interaction signal and the intention combination of the h-th unit in the sequence features corresponding to the third interaction signal, count the number of the same intention types between the two, and the ratio of the number of the total number of intention types between the two is the similarity value.

8. The humanoid robot interaction system according to claim 7, characterized in that, The feature space is a spatiotemporal feature space.

9. The humanoid robot interaction system according to claim 8, characterized in that, The third interactive signal is generated and processed based on the encoder, including: The perceptual features of the third interaction signal are input into the encoder, and deep features are extracted through the coding layer; Deep features are input into a fully connected layer for dimensionality transformation to obtain action control parameters; Generate joint movement commands for the humanoid robot based on motion control parameters.

10. The humanoid robot interaction system according to claim 9, characterized in that, The calibration process for the third interactive signal is based on the mapping execution unit, including: The interaction elements of the third interaction signal are matched with the mapping relationship of the mapping execution unit to obtain the corresponding semantic entries; Based on semantic entries, a preset action template is invoked to generate initial motion instructions; The feedback signal after the humanoid robot executes the initial motion command is collected, the parameter deviation of the initial motion command is adjusted, and the calibrated motion command is obtained.

Images