Robot Adaptive Human-Computer Interaction and Task Instruction Understanding System

Abstract

This invention belongs to the field of human-computer interaction technology. It discloses a robot adaptive human-computer interaction and task instruction understanding system, comprising: synchronously acquiring multimodal interaction signals from the user and identifying user state information; performing semantic understanding on voice instructions in the multimodal interaction signals to generate a structured task representation; dynamically evaluating the task executability of the structured task representation and identifying potential risk points; dynamically generating personalized interaction strategies based on the structured task representation, task executability, and potential risk points; performing human-computer interaction according to the personalized interaction strategies, capturing user feedback signals in real time, dynamically correcting the structured task representation, and determining the final task representation; and generating and executing sequential task steps based on the final task representation. This invention enables intelligent processing of the entire process from user intent parsing to task execution, enhancing the robot's adaptability in complex dynamic environments and improving the user experience.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and more specifically, to a robot adaptive human-computer interaction and task instruction understanding system. Background Technology

[0002] With the rapid development of artificial intelligence and intelligent robot technology, robot application scenarios are becoming increasingly diversified, especially in fields that require frequent human-machine collaboration (such as intelligent manufacturing, medical services, special education, and elderly assistance). In order to ensure the accuracy of task execution and the efficiency of collaboration, it is necessary to rely on multimodal technologies, including speech recognition, computer vision, and natural language processing, to work together to achieve accurate understanding of human instructions and precise execution of robot behavior.

[0003] However, in existing technologies, most robot human-computer interaction systems operate using a single-modal independent processing or simple information fusion approach. They mainly rely on preset instruction templates or fixed interaction processes to simply process user input (such as keyword matching, fixed syntax parsing, etc.). This mode lacks comprehensive perception, dynamic evaluation, and intelligent response to user state, environmental context, and task constraints, making it difficult to cope with ambiguous instructions and implicit information in multi-turn interactions. Taking the complex instruction understanding scenario commonly seen in practical applications as an example, omitted information, referential relationships, and context-dependent content in user expressions can easily lead to misunderstandings, such as instruction ambiguity, incomplete task parsing, and incorrect execution steps. This can result in interaction failure, increased task execution risk, or reduced user experience, thereby reducing the adaptability and reliability of existing human-computer interaction systems in complex dynamic environments.

[0004] In view of this, the present invention proposes a robot adaptive human-computer interaction and task instruction understanding system to solve the above problems. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a robot adaptive human-computer interaction and task instruction understanding system, comprising:

[0006] The signal acquisition module is used to synchronously acquire the user's multimodal interaction signals;

[0007] The status recognition module is used to sense and process multimodal interaction signals and identify user status information;

[0008] The instruction parsing module is used to perform context-enhanced semantic understanding of voice instructions in multimodal interaction signals based on user state information, and generate structured task representations.

[0009] The task evaluation module is used to collect task execution constraint data, dynamically evaluate the executability of the structured task representation based on the task execution constraint data, and identify potential risk points.

[0010] The strategy generation module is used to dynamically generate personalized interaction strategies based on structured task representation, task executability, and potential risk points.

[0011] The task determination module is used to control the robot to interact with the user according to the personalized interaction strategy, capture user feedback signals in real time, dynamically correct the structured task representation based on the user feedback signals, and determine the final task representation.

[0012] The task execution module is used to generate time-series task steps based on the final task representation, and to control the robot to execute the final task representation according to the time-series task steps.

[0013] Furthermore, multimodal interaction signals include command signals and facial expression signals; command signals include voice commands and gesture commands.

[0014] The user status information identified includes:

[0015] Preset framing parameters, and perform framing processing on voice commands based on the framing parameters to obtain multiple short-time voice frames; apply a window function to each short-time voice frame to obtain the window function weighted frame corresponding to each short-time voice frame;

[0016] Prosodic feature parameters are extracted from all window function weighted frames, and both the prosodic feature parameters and facial expression signals are input into the trained state prediction model to predict the corresponding user state information; among which, the user state information includes emotion labels and interaction intention intensity.

[0017] Furthermore, the content of generating structured task representations includes:

[0018] Voice commands are converted into corresponding text content in real time and marked as task text; corresponding task entities are extracted from gesture commands, including task object and task location; task entities are added to the task text to obtain complete text;

[0019] Both the complete text and user status information are converted into standardized formats to obtain standardized task information and standardized status information; the standardized task information and standardized status information are combined to generate a structured task representation.

[0020] Furthermore, the dynamic evaluation of the executability of structured task representations includes:

[0021] Collect multiple sets of historical execution constraint data, and group the task execution constraint data and data of the same type in each set of historical execution constraint data into a data set;

[0022] Compare the data in the same dataset. If all the data in the dataset are the same, mark the data in the dataset as static data. If all the data in the dataset are different, mark the data in the dataset as dynamic data.

[0023] A pre-defined fluctuation set is used, which includes the fluctuation range corresponding to each type of dynamic data. The fluctuation range corresponding to each type of dynamic data is obtained from the fluctuation set. Based on the fluctuation range, the fluctuation interval corresponding to each dynamic data in the task execution constraint data is calculated in turn, and the fluctuation interval of each dynamic data is set to be uniformly distributed.

[0024] A value is randomly selected from each fluctuation range, and a set of fluctuation constraint data is constructed; this process is repeated until a total of [number] values ​​are constructed. Group fluctuation constraint data; add all static data to each group of fluctuation constraint data respectively to obtain Sets of random constraint data; under the conditions of each set of random constraint data, the structured task representation is simulated and executed in sequence to obtain the execution result corresponding to each set of random constraint data. The execution result includes execution success and execution failure.

[0025] The execution result is the number of successfully executed random constraint data points, marked as the success count; the success count is then compared with... The ratio between these two values ​​gives the task's executability.

[0026] Furthermore, the steps to identify potential risk points include:

[0027] Step S1: Randomly select a type of dynamic data and mark it as the current data;

[0028] Step S2: Based on the current data, Group fluctuation constraint data is divided into Group variable control combinations, each group variable control combination contains Group fluctuation constraint data;

[0029] Step S3: Control the corresponding combinations of the same set of variables. Compare the execution results of the group fluctuation constraint data, if If the execution results of the group volatility constraint data are not all the same, the corresponding variable control portfolio will be marked as a risk portfolio;

[0030] Step S4: For the current data... If the group variables are controlled by combination analysis, If there are risky combinations in the group variable control portfolio, then the dynamic data corresponding to the current data will be regarded as potential risk points;

[0031] Step S5: Repeat steps S1 to S4 until all dynamic data in the task execution constraint data is marked as current data, then the loop ends and all potential risk points are obtained.

[0032] Furthermore, the content of dynamically generated personalized interaction strategies includes:

[0033] Based on the feasibility of the task and potential risks, the interaction mode is determined; the interaction modes include rapid execution mode, negotiation interaction mode, and explanation and suggestion mode.

[0034] If the interaction mode is fast execution mode, the structured task representation will be used as the final task representation, and the task execution module will be executed.

[0035] If the interaction mode is not the fast execution mode, the interaction style is determined based on the user state information corresponding to the standardized state information in the structured task representation; the number of risk combinations corresponding to each potential risk point is counted and used as the risk severity of each potential risk point; a risk set is preset, and the risk coefficient corresponding to each potential risk point is obtained from the risk set in sequence, and the product of the risk severity of each potential risk point and the corresponding risk coefficient is calculated in sequence to obtain the interaction priority of each potential risk point.

[0036] Based on the interaction style, the corresponding interactive voice features are obtained from the preset voice mapping table; based on the interaction mode, interaction style and the interaction priority of each potential risk point, the interactive output content is dynamically generated; the interactive voice features and interactive output content are integrated to form a personalized interaction strategy.

[0037] Furthermore, the content of determining the interaction mode includes:

[0038] If no potential risk points are found, the interaction mode is determined to be the fast execution mode;

[0039] If potential risks exist, the task's executability will be compared with the preset execution threshold.

[0040] If the task's executability is greater than or equal to the execution threshold, then the interaction mode is determined to be the negotiation interaction mode.

[0041] If the task's executability is less than the execution threshold, then the interaction mode is determined to be the explanation and suggestion mode;

[0042] The content for determining the interaction style includes:

[0043] Based on the emotion tags in the user's status information, obtain the corresponding emotion state; from the preset emotion set, obtain the emotion code corresponding to the emotion state;

[0044] A weight set is preset, which includes weight coefficients corresponding to the intensity of emotion encoding and interaction intention. Based on the weight set, the emotion encoding and the intensity of interaction intention are weighted and summed to obtain the user interaction tendency score. Corresponding score intervals are set for different interaction styles in sequence. The user interaction tendency score is compared with the score interval of each interaction style in sequence to filter out the interaction style corresponding to the score interval of the user interaction tendency score.

[0045] Furthermore, the content of setting corresponding score ranges for different interaction styles includes:

[0046] Collect multiple sets of historical state information, calculate the user interaction tendency score corresponding to each set of historical state information in turn, and mark it as the historical interaction tendency score; obtain the interaction style corresponding to each set of historical state information in turn, and match the historical interaction tendency score corresponding to each set of historical state information with the corresponding interaction style.

[0047] The average interaction tendency score for each interaction style is calculated by averaging the historical interaction tendency scores. Each interaction style is then sorted from largest to smallest according to its average interaction tendency score to generate a style sequence. The average interaction tendency scores for every two adjacent interaction styles in the style sequence are averaged to obtain a boundary score. Based on the two boundary scores for each interaction style, a corresponding score range is set for each interaction style.

[0048] Furthermore, the content of dynamically generated interactive output includes:

[0049] The number of potential risk points is counted to obtain the number of risks; the maximum score is calculated, and the ratio of the user interaction tendency score to the maximum score is calculated to obtain the interaction ratio;

[0050] Calculate the product of the number of risks and the percentage of interactions to obtain the number of interactions; sort all potential risk points according to their corresponding interaction priorities from highest to lowest, and obtain the top-ranked ones. Potential risk points in the location are marked as interactive risk points. The number of interactions rounded up;

[0051] If the interaction mode is the negotiation interaction mode, then the interaction output content is dynamically generated based on the interaction mode, interaction style, structured task representation, and all interaction risk points.

[0052] If the interaction mode is the description and suggestion mode, then obtain the risk combination corresponding to each interaction risk point; analyze each risk combination in turn to obtain the boundary set corresponding to each risk combination; each boundary set includes two adjacent variable control combinations with different execution results within the corresponding risk combination; take the value of the current data within the variable control combination with the execution failure result in each boundary set as the risk value of the corresponding risk interaction point;

[0053] Based on the interaction mode, interaction style, structured task representation, all interaction risk points, and the risk value of each interaction risk point, the interactive output content is dynamically generated.

[0054] Furthermore, determining the content of the final task representation includes:

[0055] User feedback signals are converted into corresponding text content in real time and marked as feedback text; both feedback text and interactive output content are converted into standardized formats to obtain standardized interactive information; the standardized interactive information is fused with standardized task information to obtain the final task representation.

[0056] The technical effects and advantages of the robot adaptive human-computer interaction and task instruction understanding system of this invention are as follows:

[0057] By comprehensively sensing and dynamically analyzing users' multimodal interaction signals, the system accurately understands users' intentions and emotional states, enhancing the intelligence, flexibility, and user experience of human-computer interaction. Employing methods for quantitatively assessing task executability and accurately identifying potential risks, the robot can proactively perceive the execution environment and constraints, achieving risk avoidance and strategy optimization during task execution, forming a closed-loop adaptive control system, and improving the safety and reliability of task execution. Based on task executability, potential risks, and user state information, the system dynamically generates differentiated interaction strategies, effectively mitigating task execution risks and improving the personalization and intelligence of human-computer interaction. Real-time capture and dynamic correction of user feedback during interaction ensures the accuracy and executability of the final task representation and generates directly executable sequential task steps, improving the flexibility and stability of task execution. This embodiment integrates multiple cutting-edge technologies such as multimodal perception, contextual understanding, intelligent decision-making, and adaptive interaction to achieve end-to-end intelligent processing from user intention parsing to task execution, thereby enhancing the robot's adaptability in complex dynamic environments, improving user experience, increasing task execution efficiency, and meeting diverse application needs. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the robot adaptive human-computer interaction and task instruction understanding system of Embodiment 1 of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Example 1

[0061] Please see Figure 1 As shown, the robot adaptive human-computer interaction and task instruction understanding system described in this embodiment includes a signal acquisition module, a state recognition module, an instruction parsing module, a task evaluation module, a strategy generation module, a task determination module, and a task execution module; each module is connected via wired and / or wireless means to realize data transmission between modules.

[0062] The signal acquisition module is used to synchronously acquire the user's multimodal interaction signals.

[0063] Multimodal interaction signals include command signals and facial expression signals;

[0064] Command signals include voice commands and gesture commands;

[0065] Voice commands refer to explicit operational commands or requests expressed orally by the user using natural language, such as turning on lights or moving materials; voice commands are acquired through a microphone array integrated within the robot. Gesture commands refer to action commands conveyed by the user through hand, arm, or body movements, such as pointing to the living room light or pointing to assembly line number 1 in the workshop; gesture commands are acquired by a camera integrated within the robot to obtain a full-body image of the user (i.e., an image of the user's entire body), and then processed by an action recognition algorithm; action recognition algorithms include OpenPose (for skeletal keypoint detection), MediaPipe (real-time gesture tracking), and Spatiotemporal Graph Convolutional Network (ST-GCN), etc.

[0066] Facial expressions refer to the natural facial features of a user, such as smiling, frowning, surprise, and confusion. Facial expressions are obtained by acquiring images of the user's face (i.e., images of the user's face) through a camera integrated into the robot and applying facial expression recognition algorithms to these images. Examples of facial expression recognition algorithms include convolutional neural network (CNN) expression classifiers, FER models, and FACS-based analysis methods.

[0067] Command signals are used to convey the specific task requirements that the user wants the robot to perform; facial expression signals are used to reflect the user's emotional state to help optimize human-computer interaction decisions.

[0068] The status recognition module is used to sense and process multimodal interaction signals and identify user status information.

[0069] The user status information identified includes:

[0070] Preset framing parameters are used to segment the speech command into frames, resulting in multiple short-time speech frames. These parameters include the sampling rate, frame length, and frame shift, which are preset by those skilled in the art based on actual conditions. The duration of each short-time speech frame is typically 20-30ms. A window function is applied to each short-time speech frame (i.e., each short-time speech frame is multiplied by a window function), resulting in a window-weighted frame corresponding to each short-time speech frame to reduce spectral leakage. Window functions include, for example, the Hamming window and the Hanning window.

[0071] Prosodic feature extraction methods are employed to extract prosodic feature parameters from all window function weighted frames. These methods include autocorrelation, short-time energy, energy envelope peak detection, and speech activity detection. The prosodic feature parameters include pitch, volume, speech rate, and pause frequency. Autocorrelation is used to extract pitch, short-time energy to extract volume, energy envelope peak detection to extract speech rate, and speech activity detection to extract pause frequency. The prosodic feature parameters and facial expression signals are input into a trained state prediction model to predict the corresponding user state information. It should be noted that the prosodic feature extraction methods are all existing technologies, and the specific processes are not elaborated upon here.

[0072] User status information includes emotion tags and the intensity of willingness to interact;

[0073] Emotional labels are numerical labels corresponding to emotional states; different emotional states correspond to different emotional labels. Emotional state refers to the user's current emotional experience, such as joy, sadness, or anger. Interaction willingness intensity represents the user's level of enthusiasm for actively participating in human-computer interaction, with a corresponding value range of [range missing]. Prosodic feature parameters can reflect the rhythm, stress, and pause patterns of speech. High pitch, high volume, fast speech rate, and low pause frequency usually correspond to pleasant or excited emotions and a high degree of willingness to interact, while low pitch, low volume, slow speech rate, and high pause frequency usually correspond to depressed or tired emotions and a low degree of willingness to interact. Facial expression signals can provide visual information and work together with prosodic feature parameters to enhance the judgment of emotional state and the strength of willingness to interact, enabling the system to more accurately perceive the user's current emotional experience and level of participation.

[0074] The state prediction model is a deep neural network model, which includes an input layer, hidden layers, and an output layer. Each hidden layer contains multiple neurons, and each neuron is connected to the neurons in the next layer. The connections contain weights that determine the importance and influence of data transmitted in the neural network. An activation function is applied to each neuron between the hidden layer and the output layer. The activation function introduces non-linearity, allowing the network to learn more complex patterns and features. The deep neural network model is a current technology, and the specific training process will not be described in detail here.

[0075] The instruction parsing module is used to perform context-enhanced semantic understanding of voice instructions in multimodal interaction signals based on user state information, and generate structured task representations.

[0076] The content of generating structured task representations includes:

[0077] By calling the API interface of cloud-based speech recognition services (such as Baidu, iFlytek, etc.), the voice commands are converted into corresponding text content in real time and marked as task text; the regular expression matching method is used to extract the corresponding task entity from the gesture command, the task entity includes the task object and the task position; the task entity is added to the task text to obtain the complete text; it should be noted that the regular expression matching method is an existing technology, and the specific process will not be described in detail here.

[0078] Both the complete text and user status information are converted into a standardized format to obtain standardized task information and standardized status information. The standardized task information and standardized status information are then combined to generate a structured task representation. The complete text and user status information are converted into a standardized format through preset standardization rules within the robot. These standardization rules are pre-designed by those skilled in the art based on the robot's system functional requirements and task processing requirements. The standardized format is a unified data representation used to represent the complete text and user status information, enabling the robot to understand them directly.

[0079] For example, if the task text is "turn on the light" and the gesture instruction is "point to the light in the living room", then the task object corresponding to the gesture instruction is the light and the task location is the living room. If the light and the living room are added to the task text, then the complete text obtained is "turn on the light in the living room".

[0080] It should be understood that context-enhanced semantic understanding of voice commands involves supplementing the task text converted from voice commands with user state information and task entities extracted from gesture commands, enabling the system to more accurately understand user intent and thus form a complete, accurate, and executable structured task representation.

[0081] The task evaluation module is used to collect task execution constraint data, dynamically evaluate the executability of the structured task representation based on the task execution constraint data, and identify potential risk points.

[0082] Task execution constraint data includes both self-constraints and environmental constraints;

[0083] Self-constraints refer to the physical, functional, and resource limitations within the robot itself, including but not limited to energy constraints (such as battery capacity and power consumption limits), kinematic constraints (such as maximum speed and turning radius), and dynamic constraints (such as load capacity and maximum thrust). Self-constraints are obtained through the robot's specifications and the built-in battery management system.

[0084] Environmental constraints refer to the external conditions and limitations imposed on the robot's task execution by the external environment in which it operates, including but not limited to spatial structure (such as passage width, passage slope, etc.) and dynamic environment (such as moving obstacles, crowd density, etc.). Environmental constraints are obtained by acquiring dynamic environmental data through the robot's built-in multimodal sensors (such as LiDAR, cameras, etc.) and processing it in combination with environmental modeling algorithms (such as SLAM algorithm, grid map, topology map, etc.) and dynamic inference algorithms (such as Kalman filter and particle filter for dynamic object tracking, trajectory prediction network, etc.). Among them, dynamic environmental data refers to the environmental information that changes over time and is perceived by the robot in real time through multimodal sensors, such as moving obstacles, crowd flow, etc.

[0085] The dynamic evaluation of the executability of structured task representations includes:

[0086] Multiple sets of historical execution constraint data are collected. This historical execution constraint data is the task execution constraint data collected by the robot before it performs structured task representation in the past, and is obtained from the robot's built-in database. Data of the same type in each set of historical execution constraint data are grouped into a data set, with each data set corresponding one-to-one with the data type in the task execution constraint data. Data within the same data set are compared. If all data values ​​in a data set are identical, the corresponding data set is marked as static data. If not all data values ​​in a data set are identical, the corresponding data set is marked as dynamic data. Static data includes, for example, maximum speed, load capacity, and channel width; dynamic data includes, for example, battery level, power consumption limits, moving obstacles, and crowd density.

[0087] A preset fluctuation set, which includes the fluctuation range corresponding to each type of dynamic data, is preset by those skilled in the art according to the actual situation. For example, the fluctuation range... , The process involves: obtaining the fluctuation range corresponding to each type of dynamic data from the fluctuation set; calculating the fluctuation interval corresponding to each dynamic data in the task execution constraint data based on the fluctuation range, and setting the fluctuation interval of each dynamic data to a uniform distribution; specifically: calculating the product of each dynamic data with the maximum and minimum values ​​of its corresponding fluctuation range to obtain the maximum and minimum variables of each dynamic data; calculating the sum of each dynamic data with its corresponding maximum variable to obtain the maximum fluctuation value of each dynamic data; calculating the difference between each dynamic data and its corresponding minimum variable to obtain the minimum fluctuation value of each dynamic data; and constructing the fluctuation interval corresponding to each dynamic data based on its maximum and minimum fluctuation values.

[0088] A value is randomly selected from each fluctuation range, and a set of fluctuation constraint data is constructed; this process is repeated until a total of [number] values ​​are constructed. Group fluctuation constraint data, It is an integer greater than 1, and The set of fluctuation constraint data are all different; by adding all static data to each set of fluctuation constraint data, we obtain... Sets of random constraint data; under the conditions of each set of random constraint data, the structured task representation is simulated and executed in sequence to obtain the execution result corresponding to each set of random constraint data. The execution result includes execution success and execution failure.

[0089] The execution result is the number of successfully executed random constraint data points, marked as the success count; the success count is then compared with... The ratio between these two values ​​gives the task's executability.

[0090] The simulation execution of the structured task representation includes the following:

[0091] A physical simulator (such as Gazebo, Webots, etc.) is pre-built by a person skilled in the art based on all static data, and then... The set of fluctuation constraint data is sequentially input into the physics simulator, where it is simulated and executed. The structured task representation is used to obtain the execution result corresponding to each set of fluctuation constraint data; that is, whenever a set of fluctuation constraint data is input into the physical simulator, the physical simulator simulates and executes a structured task representation.

[0092] The steps to identify potential risk points include:

[0093] Step S1: Randomly select a type of dynamic data and mark it as the current data;

[0094] Step S2: Based on the current data, Group fluctuation constraint data is divided into Group variable control combinations, each group variable control combination contains Group fluctuation constraint data, Among them, each group of variable control combinations corresponds to Within the group of fluctuation constraint data, only the value of the current data is different, while the values ​​of the rest of the dynamic data are the same.

[0095] For example, there are three types of dynamic data: the fluctuation range of dynamic data 1 is 1 and 2, the fluctuation range of dynamic data 2 is 3 and 4, and the fluctuation range of dynamic data 3 is 5. Therefore, the fluctuation constraint data includes... , , as well as If we take dynamic data 1 as the current data, then dynamic data 1 corresponds to two sets of variable control combinations. Variable control combination 1 includes... and Variable control combination 2 includes as well as ;

[0096] Step S3: Control the corresponding combinations of the same set of variables. Compare the execution results of the group fluctuation constraint data, if If the execution results of the group fluctuation constraint data are all the same, then no label will be marked for the corresponding variable control combination. If the execution results of the group volatility constraint data are not all the same, the corresponding variable control portfolio will be marked as a risk portfolio;

[0097] Step S4: For the current data... If the group variables are controlled by combination analysis, If a risk portfolio exists within the group variable control portfolio, then the dynamic data corresponding to the current data is considered as a potential risk point. If there is no risk portfolio in the group variable control portfolio, then the dynamic data corresponding to the current data will not be regarded as a potential risk point;

[0098] Step S5: Repeat steps S1 to S4 until all dynamic data in the task execution constraint data is marked as current data, then the loop ends and all potential risk points are obtained.

[0099] It should be understood that the purpose of dynamically assessing task feasibility and identifying potential risks is to provide robots with scientific decision-making basis and proactive risk management capabilities, thereby providing key decision inputs for the subsequent generation of personalized interaction strategies and realizing an intelligent closed loop from risk perception to strategy optimization.

[0100] Specifically, the probabilistic assessment of task executability directly influences the direction of personalized interaction strategy generation. When executability is high, a direct execution interaction strategy can be adopted, requiring only simple confirmation. When executability is at a medium level, a more cautious interaction strategy needs to be generated, such as explaining the potential risks to the user in detail and soliciting their opinions. When executability is low, the reasons why the task cannot be executed need to be clearly explained. At the same time, the identified potential risk points provide precise optimization directions for personalized interaction strategies. For example, when battery level is identified as a risk point, the personalized interaction strategy will proactively ask the user, "The current battery level is low; should we charge it before executing the task?" When crowd density is identified as a risk point, the personalized interaction strategy will suggest, "A dense crowd has been detected; it is recommended to execute the task in 5 minutes." This strategy generation mechanism based on quantitative assessment and risk identification enables the robot to dynamically adjust its interaction with the user according to actual constraints and risk conditions, ensuring both the success rate of task execution and improving the personalization and intelligence level of the user experience, thereby achieving adaptive human-computer interaction that is "suitable for the situation and takes appropriate measures based on the risks."

[0101] The strategy generation module is used to dynamically generate personalized interaction strategies based on structured task representation, task executability, and potential risk points.

[0102] The content of dynamically generated personalized interaction strategies includes:

[0103] Based on the feasibility of the task and potential risks, the interaction mode is determined; the interaction modes include rapid execution mode, negotiation interaction mode, and explanation and suggestion mode.

[0104] If the interaction mode is fast execution mode, the structured task representation will be used as the final task representation, and the task execution module will be executed.

[0105] If the interaction mode is not a fast execution mode, the interaction style is determined based on the user state information corresponding to the standardized state information in the structured task representation; the number of risk combinations corresponding to each potential risk point is counted and used as the risk severity of each potential risk point; a risk set is preset, which includes the risk coefficient corresponding to each potential risk point, and is preset by those skilled in the art according to the actual situation; the risk coefficient corresponding to each potential risk point is obtained sequentially from the risk set, and the product of the risk severity of each potential risk point and the corresponding risk coefficient is calculated sequentially to obtain the interaction priority of each potential risk point;

[0106] Based on the interaction style, the corresponding interactive voice features are obtained from the preset voice mapping table; based on the interaction mode, interaction style and the interaction priority of each potential risk point, the interactive output content is dynamically generated; the interactive voice features and interactive output content are integrated to form a personalized interaction strategy.

[0107] Among these, the lower the risk coefficient, the better the potential risk that the robot can automatically avoid or that requires no user intervention; conversely, the higher the risk coefficient, the better the potential risk that significantly impacts task feasibility and requires user intervention. For example, moving obstacles may affect the robot's task execution, but the robot can automatically avoid them, so the risk coefficient for moving obstacles is relatively low; while insufficient battery power will directly prevent the robot from completing the task and requires user intervention to charge it, so the risk coefficient for insufficient battery power is relatively high.

[0108] The voice mapping table includes interactive voice features corresponding to different interaction styles, which are preset by those skilled in the art according to the actual situation. Interactive voice features include tone, volume, and speech rate. Interactive styles include enthusiastic and detailed (proactive and information-rich interaction with strong interactivity and emotional expression, suitable for scenarios where users have a high willingness to interact and are in a positive mood), patient and caring (gentle and considerate interaction with moderate information, focusing on user experience and emotional care, suitable for scenarios where users have a moderate willingness to interact or are in a sensitive mood), and concise and efficient (simple and direct interaction with less information, emphasizing task execution, suitable for scenarios where users have a low willingness to interact or are in a negative mood and want to complete tasks quickly).

[0109] The content that determines the interaction mode includes:

[0110] If there are no potential risk points, the interaction mode is determined to be the fast execution mode. In the fast execution mode, the corresponding structured task representation has high executability and controllable risk. The robot can directly execute the task without complex interaction with the user.

[0111] If potential risks exist, the executability of the task will be compared with a preset execution threshold, which will be preset by a person skilled in the art based on the actual situation.

[0112] If the task executability is greater than or equal to the execution threshold, the interaction mode is determined to be the negotiation interaction mode. In the negotiation interaction mode, the corresponding structured task has medium executability and potential risks. The robot needs to interact with the user, prompt the potential risks and solicit the user's opinions.

[0113] If the task executability is less than the execution threshold, the interaction mode is determined to be the explanation and suggestion mode. In the explanation and suggestion mode, the executability of the corresponding structured task representation is low and the risk is significant. The robot needs to explain to the user why the structured task representation cannot be executed.

[0114] The content for determining the interaction style includes:

[0115] Based on the emotion tags in the user's status information, the corresponding emotion state is obtained; from a preset emotion set, the emotion code corresponding to the emotion state is obtained; the emotion set includes emotion codes corresponding to different emotion states, and the value range of the emotion codes is [range missing]. The emotional codes are pre-set by those skilled in the art based on the actual situation; the more positive and proactive the emotional state, the larger the corresponding emotional code, such as joy and excitement; conversely, the more negative and passive the emotional state, the smaller the corresponding emotional code, such as sadness and frustration.

[0116] A pre-set weight set is used, which includes weight coefficients corresponding to the emotion encoding and the intensity of interaction intention. These weights are pre-set by those skilled in the art based on the actual situation. According to the weight set, the emotion encoding and the intensity of interaction intention are weighted and summed to obtain the user interaction tendency score. Corresponding score intervals are set for different interaction styles in sequence. The user interaction tendency score is compared with the score interval of each interaction style in sequence to select the interaction style corresponding to the score interval of the user interaction tendency score, which is then used as the interaction style when the robot interacts with the user.

[0117] The content for setting corresponding score ranges for different interaction styles includes:

[0118] Collect multiple sets of historical state information, which are user state information identified by the robot when interacting with the user at historical moments, obtained from the robot's built-in database; calculate the user interaction tendency score corresponding to each set of historical state information in turn, and mark it as the historical interaction tendency score; obtain the interaction style corresponding to each set of historical state information in turn, and match the historical interaction tendency score corresponding to each set of historical state information with the corresponding interaction style.

[0119] The average interaction tendency score for each interaction style is calculated by averaging the historical interaction tendency scores. Each interaction style is then sorted from largest to smallest according to its average interaction tendency score to generate a style sequence. The average interaction tendency scores for every two adjacent interaction styles in the style sequence are averaged to obtain a boundary score. Based on the two boundary scores for each interaction style, a corresponding score range is set for each interaction style.

[0120] Specifically, if the interaction style is ranked first in the style sequence, the maximum value of the emotion encoding (i.e., 1) and the maximum value of the interaction intention intensity (i.e., 1) are obtained, and the maximum value of the user interaction tendency score is calculated according to the weight set and marked as the maximum score; the score range of the corresponding interaction style is set according to the boundary score and the maximum score of the corresponding interaction style; similarly, if the interaction style is ranked last in the style sequence, the score range of the corresponding interaction style is set according to the boundary score and 0 of the corresponding interaction style; it should be noted that since the minimum value of the emotion encoding and the minimum value of the interaction intention intensity are both 0, the minimum value of the calculated user interaction tendency score is also 0.

[0121] The dynamically generated interactive output content includes:

[0122] The process involves counting potential risk points to obtain the total number of risks; calculating the ratio of user interaction tendency score to the maximum score to obtain the interaction percentage; multiplying the number of risks by the interaction percentage to obtain the total number of interactions; and then sorting all potential risk points from highest to lowest according to their corresponding interaction priority, obtaining the top-ranked risk points. Potential risk points in the location are marked as interactive risk points. The number of interactions rounded up;

[0123] If the interaction mode is a negotiation interaction mode, then based on the interaction mode, interaction style, structured task representation, and all interaction risk points, the NLG (Natural Language Generation) method is used to dynamically generate the interaction output content; the NLG method is an existing technology, and the specific process will not be elaborated on here.

[0124] If the interaction mode is the description and suggestion mode, then obtain the risk combination corresponding to each interaction risk point; analyze each risk combination in turn to obtain the boundary set corresponding to each risk combination; each boundary set includes two adjacent variable control combinations with different execution results within the corresponding risk combination; take the value of the current data within the variable control combination with the execution failure result in each boundary set as the risk value of the corresponding risk interaction point;

[0125] Based on the interaction mode, interaction style, structured task representation, all interaction risk points and the risk value of each interaction risk point, the NLG (Natural Language Generation) method is used to dynamically generate the interaction output content.

[0126] For example, the interaction mode is a negotiation interaction mode, the interaction style is patient and caring, the structured task is to move materials to production line 2, and the interaction risk points are battery power (currently 35%) and crowd density (5 people detected near production line 2); therefore, the dynamically generated interaction output content is:

[0127] I understand you need to move materials from production line 2; however, before we begin, I'd like to confirm two things with you: First, my current battery level is 35%, which I expect to be able to complete this task, but I'll need to recharge it afterward; do you think now is an appropriate time to proceed? Second, I've detected several workers moving around near production line 2. To ensure safety, I will slow down my movement and maintain a safe distance; this may take an extra 3-5 minutes, is that acceptable to you? If you agree to these two points, we can begin the task now.

[0128] The task determination module is used to control the robot to interact with the user according to the personalized interaction strategy, capture user feedback signals in real time, dynamically correct the structured task representation based on the user feedback signals, and determine the final task representation.

[0129] The final task representation includes:

[0130] User feedback signals are converted into corresponding text content in real time and marked as feedback text; user feedback signals are multi-turn voice commands given by the user and the robot during the interaction process; both feedback text and interaction output content are converted into standardized formats to obtain standardized interaction information; the standardized interaction information and standardized task information are input into a trained NLP model (such as T5, BART, BERT2BERT, etc.) to obtain the final task representation; the NLP model is an existing technology, and the specific training process will not be described in detail here.

[0131] The task execution module is used to generate time-series task steps based on the final task representation, and to control the robot to execute the final task representation according to the time-series task steps.

[0132] The steps for generating a time-series task based on the final task representation include:

[0133] The Hierarchical Task Network (HTN) planning method is adopted to decompose the final task representation and generate multiple temporal task steps that the robot can directly execute. The temporal task steps refer to the executable action sequence after HTN decomposition and sorting. The Hierarchical Task Network (HTN) planning method is an existing technology, and the specific implementation process will not be described in detail here.

[0134] For example, if the final task is represented as installing part A onto part B, then after HTN decomposition and sorting, the sequential task steps can be obtained as follows: grab part A, move to the position of part B, install part A onto part B, and release part A, thus completing the final task representation.

[0135] This embodiment achieves accurate understanding of user intentions and emotional states by comprehensively sensing and dynamically analyzing multimodal interaction signals, thereby enhancing the intelligence, flexibility, and user experience of human-computer interaction. It employs methods for quantitatively assessing task executability and accurately identifying potential risk points, enabling the robot to proactively perceive the execution environment and constraints. This allows for risk avoidance and strategy optimization during task execution, forming a closed-loop adaptive control system and improving the safety and reliability of task execution. Based on task executability, potential risk points, and user state information, it dynamically generates differentiated interaction strategies, effectively mitigating task execution risks and improving the personalization and intelligence of human-computer interaction. Real-time capture and dynamic correction of user feedback during interaction ensures the accuracy and executability of the final task representation and generates directly executable sequential task steps, improving the flexibility and stability of task execution. This embodiment integrates multiple cutting-edge technologies such as multimodal perception, contextual understanding, intelligent decision-making, and adaptive interaction to achieve end-to-end intelligent processing from user intention parsing to task execution. This enhances the robot's adaptability in complex dynamic environments, improves user experience, increases task execution efficiency, and meets diverse application needs.

[0136] Example 2

[0137] This application also provides an electronic device. The electronic device may include one or more processors and one or more memories. The memories store computer-readable code, which, when executed by the one or more processors, can perform the robot adaptive human-machine interaction and task instruction understanding system described above.

[0138] The methods or systems according to embodiments of this application can also be implemented using the architecture of the electronic device shown in this application. The electronic device may include a bus, one or more CPUs, ROM, RAM, a communication port connected to a network, input / output, a hard disk, etc. Storage devices in the electronic device, such as ROM or hard disk, may store the robot adaptive human-machine interaction and task instruction understanding system provided in this application. Furthermore, the electronic device may also include a user interface. Of course, the architecture shown in this application is merely exemplary; when implementing different devices, one or more components in the electronic device shown in this application may be omitted according to actual needs.

[0139] Example 3

[0140] One embodiment of this application discloses a computer-readable storage medium. The computer-readable storage medium stores computer-readable instructions. When the computer-readable instructions are executed by a processor, a robot adaptive human-computer interaction and task instruction understanding system according to an embodiment of this application, as described with reference to the above figures, can be executed. The storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0141] Furthermore, according to embodiments of this application, the processes described above with reference to the schematic diagrams can be implemented as computer software programs. For example, this application provides a non-transitory machine-readable storage medium storing machine-readable instructions that can be executed by a processor to perform instructions corresponding to the method steps provided in this application, such as a robot adaptive human-computer interaction and task instruction understanding system. When this computer program is executed by a central processing unit (CPU), it performs the functions defined in the method of this application.

[0142] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0143] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0144] In the description of this invention, it should be understood that the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0145] In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0146] In the description of this invention, "several" means one or more, and "a large number" means two or more.

[0147] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0148] All formulas in this manual are dimensionless and calculated numerically. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0149] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A robot adaptive human-computer interaction and task instruction understanding system, characterized in that, include: The signal acquisition module is used to synchronously acquire the user's multimodal interaction signals; the multimodal interaction signals include command signals and facial expression signals; the command signals include voice commands and gesture commands. The status recognition module is used to sense and process multimodal interaction signals and identify user status information; The instruction parsing module is used to perform context-enhanced semantic understanding of voice instructions in multimodal interaction signals based on user state information, and generate structured task representations. The content of generating structured task representations includes: Voice commands are converted into corresponding text content in real time and marked as task text; corresponding task entities are extracted from gesture commands, including task object and task location; task entities are added to task text to obtain complete text; Both the complete text and user status information are converted into a standardized format to obtain standardized task information and standardized status information; the standardized task information and standardized status information are combined to generate a structured task representation. The task evaluation module is used to collect task execution constraint data, dynamically evaluate the executability of the structured task representation based on the task execution constraint data, and identify potential risk points. The strategy generation module is used to dynamically generate personalized interaction strategies based on structured task representation, task executability, and potential risk points. The task determination module is used to control the robot to interact with the user according to the personalized interaction strategy, capture user feedback signals in real time, dynamically correct the structured task representation based on the user feedback signals, and determine the final task representation. The task execution module is used to generate time-series task steps based on the final task representation, and to control the robot to execute the final task representation according to the time-series task steps.

2. The robot adaptive human-computer interaction and task instruction understanding system according to claim 1, characterized in that, The user status information identified includes: Preset framing parameters, and perform framing processing on voice commands based on the framing parameters to obtain multiple short-time voice frames; apply a window function to each short-time voice frame to obtain the window function weighted frame corresponding to each short-time voice frame; Prosodic feature parameters are extracted from all window function weighted frames, and both the prosodic feature parameters and facial expression signals are input into the trained state prediction model to predict the corresponding user state information; among which, the user state information includes emotion labels and interaction intention intensity.

3. The robot adaptive human-computer interaction and task instruction understanding system according to claim 2, characterized in that, The dynamic evaluation of the executability of structured task representations includes: Collect multiple sets of historical execution constraint data, and group the task execution constraint data and data of the same type in each set of historical execution constraint data into a data set; Compare the data in the same dataset. If all the data in the dataset are the same, mark the data in the dataset as static data. If all the data in the dataset are different, mark the data in the dataset as dynamic data. A pre-defined fluctuation set is used, which includes the fluctuation range corresponding to each type of dynamic data. The fluctuation range corresponding to each type of dynamic data is obtained from the fluctuation set. Based on the fluctuation range, the fluctuation interval corresponding to each dynamic data in the task execution constraint data is calculated in turn, and the fluctuation interval of each dynamic data is set to be uniformly distributed. A value is randomly selected from each fluctuation range, and a set of fluctuation constraint data is constructed; this process is repeated until a total of [number] values ​​are constructed. Group fluctuation constraint data; add all static data to each group of fluctuation constraint data respectively to obtain Sets of random constraint data; under the conditions of each set of random constraint data, the structured task representation is simulated and executed in sequence to obtain the execution result corresponding to each set of random constraint data. The execution result includes execution success and execution failure. The execution result is the number of successfully executed random constraint data points, marked as the success count; the success count is then compared with... The ratio between these two values ​​gives the task's executability.

4. The robot adaptive human-computer interaction and task instruction understanding system according to claim 3, characterized in that, The steps to identify potential risk points include: Step S1: Randomly select a type of dynamic data and mark it as the current data; Step S2: Based on the current data, Group fluctuation constraint data is divided into Group variable control combinations, each group variable control combination contains Group fluctuation constraint data; Step S3: Control the corresponding combinations of the same set of variables. Compare the execution results of the group fluctuation constraint data, if If the execution results of the group volatility constraint data are not all the same, the corresponding variable control portfolio will be marked as a risk portfolio; Step S4: For the current data... If the group variables are controlled by combination analysis, If there are risky combinations in the group variable control portfolio, then the dynamic data corresponding to the current data will be regarded as potential risk points; Step S5: Repeat steps S1 to S4 until all dynamic data in the task execution constraint data is marked as current data, then the loop ends and all potential risk points are obtained.

5. The robot adaptive human-computer interaction and task instruction understanding system according to claim 4, characterized in that, The content of dynamically generated personalized interaction strategies includes: Based on the feasibility of the task and potential risks, the interaction mode is determined; the interaction modes include rapid execution mode, negotiation interaction mode, and explanation and suggestion mode. If the interaction mode is fast execution mode, the structured task representation will be used as the final task representation, and the task execution module will be executed. If the interaction mode is not the fast execution mode, the interaction style is determined based on the user state information corresponding to the standardized state information in the structured task representation; the number of risk combinations corresponding to each potential risk point is counted and used as the risk severity of each potential risk point; a risk set is preset, and the risk coefficient corresponding to each potential risk point is obtained from the risk set in sequence, and the product of the risk severity of each potential risk point and the corresponding risk coefficient is calculated in sequence to obtain the interaction priority of each potential risk point. Based on the interaction style, the corresponding interactive voice features are obtained from the preset voice mapping table; based on the interaction mode, interaction style and the interaction priority of each potential risk point, the interactive output content is dynamically generated; the interactive voice features and interactive output content are integrated to form a personalized interaction strategy.

6. The robot adaptive human-computer interaction and task instruction understanding system according to claim 5, characterized in that, The content that determines the interaction mode includes: If no potential risk points are found, the interaction mode is determined to be the fast execution mode; If potential risks exist, the task's executability will be compared with the preset execution threshold. If the task's executability is greater than or equal to the execution threshold, then the interaction mode is determined to be the negotiation interaction mode. If the task's executability is less than the execution threshold, then the interaction mode is determined to be the explanation and suggestion mode; The content for determining the interaction style includes: Based on the emotion tags in the user's status information, obtain the corresponding emotion state; from the preset emotion set, obtain the emotion code corresponding to the emotion state; A weight set is preset, which includes weight coefficients corresponding to the intensity of emotion encoding and interaction intention. Based on the weight set, the emotion encoding and the intensity of interaction intention are weighted and summed to obtain the user interaction tendency score. Corresponding score intervals are set for different interaction styles in sequence. The user interaction tendency score is compared with the score interval of each interaction style in sequence to filter out the interaction style corresponding to the score interval of the user interaction tendency score.

7. The robot adaptive human-computer interaction and task instruction understanding system according to claim 6, characterized in that, The content for setting corresponding score ranges for different interaction styles includes: Collect multiple sets of historical state information, calculate the user interaction tendency score corresponding to each set of historical state information in turn, and mark it as the historical interaction tendency score; obtain the interaction style corresponding to each set of historical state information in turn, and match the historical interaction tendency score corresponding to each set of historical state information with the corresponding interaction style. The average interaction tendency score for each interaction style is calculated by averaging the historical interaction tendency scores. Each interaction style is then sorted from largest to smallest according to its average interaction tendency score to generate a style sequence. The average interaction tendency scores for every two adjacent interaction styles in the style sequence are averaged to obtain a boundary score. Based on the two boundary scores for each interaction style, a corresponding score range is set for each interaction style.

8. The robot adaptive human-computer interaction and task instruction understanding system according to claim 7, characterized in that, The dynamically generated interactive output content includes: The number of potential risk points is counted to obtain the number of risks; the maximum score is calculated, and the ratio of the user interaction tendency score to the maximum score is calculated to obtain the interaction ratio; Calculate the product of the number of risks and the percentage of interactions to obtain the number of interactions; sort all potential risk points according to their corresponding interaction priority from highest to lowest, and obtain the top-ranked ones. Potential risk points in the location are marked as interactive risk points. The number of interactions rounded up; If the interaction mode is the negotiation interaction mode, then the interaction output content is dynamically generated based on the interaction mode, interaction style, structured task representation, and all interaction risk points. If the interaction mode is the description and suggestion mode, then obtain the risk combination corresponding to each interaction risk point; analyze each risk combination in turn to obtain the boundary set corresponding to each risk combination; each boundary set includes two adjacent variable control combinations with different execution results within the corresponding risk combination; take the value of the current data within the variable control combination with the execution failure result in each boundary set as the risk value of the corresponding risk interaction point; Based on the interaction mode, interaction style, structured task representation, all interaction risk points, and the risk value of each interaction risk point, the interactive output content is dynamically generated.

9. The robot adaptive human-computer interaction and task instruction understanding system according to claim 8, characterized in that, The final task representation includes: User feedback signals are converted into corresponding text content in real time and marked as feedback text; both feedback text and interactive output content are converted into standardized formats to obtain standardized interactive information; the standardized interactive information is fused with standardized task information to obtain the final task representation.

Images