Robot self-adaptive man-machine interaction and task instruction understanding system

Abstract

The invention belongs to the technical field of man-machine interaction, and discloses a robot self-adaptive man-machine interaction and task instruction understanding system. Comprising the following steps: synchronously collecting a multi-mode interaction signal of a user, and identifying user state information; semantic understanding is carried out on the voice instruction in the multi-mode interaction signal, and structured task representation is generated; dynamically evaluating the task performability represented by the structured task, and identifying potential risk points; according to the structured task representation, the task performability and the potential risk point, a personalized interaction strategy is dynamically generated; performing man-machine interaction according to the personalized interaction strategy, capturing a user feedback signal in real time, performing dynamic correction on the structured task representation, and determining a final task representation; generating and executing a time sequence task according to the final task representation; according to the method, the whole process from user intention analysis to task execution can be intelligentized, and the adaptive capacity of the robot in a complex dynamic environment and the user experience are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, more particularly, to a robot adaptive human-computer interaction and task instruction understanding system. BACKGROUND

[0002] With the rapid development of artificial intelligence and intelligent robot technology, the application scenarios of robots are increasingly diversified, especially in fields that require frequent human-robot collaboration (such as intelligent manufacturing, medical services, special education, and elderly assistance, etc.). In order to ensure the accuracy of task execution and the efficiency of collaboration, it is necessary to use multi-modal technologies such as speech recognition, computer vision, and natural language processing to work together to achieve accurate understanding of human instructions and precise execution of robot behavior.

[0003] However, in the prior art, most robot human-computer interaction systems use single modal independent processing or simple information fusion mode, mainly relying on preset instruction templates or fixed interaction processes to simply process user input (such as keyword matching, fixed syntax parsing, etc.). This mode lacks comprehensive perception, dynamic evaluation, and intelligent response to user state, environmental context, and task constraints, making it difficult to deal with ambiguous instructions and implicit information problems in multi-round interactions. For example, in the complex instruction understanding scenarios commonly seen in practical applications, the omission information, reference relationship, and context-dependent content in user expression can easily lead to understanding bias, such as instruction ambiguity, incomplete task analysis, and execution step errors, which can cause interaction failure, increased risk of task execution, or reduced user experience, thereby reducing the adaptability and reliability of existing human-computer interaction systems in complex dynamic environments.

[0004] In view of this, the present application proposes a robot adaptive human-computer interaction and task instruction understanding system to solve the above problems. SUMMARY

[0005] In order to overcome the above-mentioned defects of the prior art, in order to achieve the above-mentioned purpose, the present application provides the following technical solution: a robot adaptive human-computer interaction and task instruction understanding system, comprising: a signal acquisition module for synchronously acquiring multi-modal interaction signals of a user; a state recognition module for perceiving and processing the multi-modal interaction signals and recognizing user state information; an instruction analysis module for performing context-enhanced semantic understanding of voice instructions in the multi-modal interaction signals based on the user state information, and generating structured task representation; a task evaluation module for collecting task execution constraint data, dynamically evaluating the task executability of the structured task representation based on the task execution constraint data, and identifying potential risk points; The strategy generation module is configured to dynamically generate a personalized interaction strategy according to the structured task representation, the task executability, and the potential risk points. The task determination module is configured to control the robot to perform human-robot interaction with the user according to the personalized interaction strategy, capture a user feedback signal in real time, dynamically correct the structured task representation based on the user feedback signal, and determine a final task representation. The task execution module is configured to generate a time-series task step according to the final task representation, and control the robot to execute the final task representation according to the time-series task step.

[0006] Further, the multi-modal interaction signal includes an instruction signal and an expression signal; the instruction signal includes a voice instruction and a gesture instruction. The content of the recognized user state information includes: The preset framing parameter is used for performing framing processing on the voice instruction to obtain a plurality of short-time voice frames; a window function is respectively applied to each short-time voice frame to obtain a window function weighted frame corresponding to each short-time voice frame. The prosodic feature parameters are extracted from all the window function weighted frames, and the prosodic feature parameters and the expression signal are input into the trained state prediction model to predict corresponding user state information; wherein the user state information includes an emotion label and an interaction intention strength.

[0007] Further, the content of generating the structured task representation includes: The voice instruction is converted into corresponding text content in real time and marked as a task text; a corresponding task entity is extracted from the gesture instruction, the task entity including a task object and a task location; the task entity is supplemented into the task text to obtain a complete text; The complete text and the user state information are converted into standardized formats to obtain standardized task information and standardized state information; the standardized task information and the standardized state information are combined to generate the structured task representation.

[0008] Further, the content of dynamically evaluating the task executability of the structured task representation includes: A plurality of groups of historical execution constraint data are collected, and the same type of data in the task execution constraint data and each group of historical execution constraint data are taken as a group of data set; The data in the same group of data set are compared respectively, if the numerical values of each data in the data set are all the same, the data corresponding to the corresponding data set is marked as static data, if the numerical values of each data in the data set are not all the same, the data corresponding to the corresponding data set is marked as dynamic data; a preset fluctuation set, the fluctuation set including a fluctuation range corresponding to each dynamic data; from the fluctuation set, a fluctuation range corresponding to each dynamic data is obtained; based on the fluctuation range, a fluctuation interval corresponding to each dynamic data in the task execution constraint data is sequentially calculated, and the fluctuation interval of each dynamic data is set as uniform distribution; a number value is randomly selected from each fluctuation interval, and a set of fluctuation constraint data is constructed; by analogy, a total of sets of fluctuation constraint data are constructed; all static data are added into each set of fluctuation constraint data respectively, and sets of random constraint data are obtained; under the condition of each set of random constraint data, the structured task representation is sequentially simulated to be executed, and an execution result corresponding to each set of random constraint data is obtained, the execution result including execution success and execution failure; the number of random constraint data with execution success in the execution result is counted and marked as a success number; a ratio between the success number and is calculated, and the task executability is obtained.

[0009] Further, the step of identifying the potential risk point comprises: Step S1: a dynamic data is randomly selected and marked as a current data; Step S2: according to the current data, the sets of fluctuation constraint data are divided into sets of variable control combinations, each set of variable control combination containing sets of fluctuation constraint data; Step S3: execution results of the sets of fluctuation constraint data corresponding to the same set of variable control combination are compared, if the execution results of the sets of fluctuation constraint data are not all the same, the corresponding variable control combination is marked as a risk combination; Step S4: the sets of variable control combinations corresponding to the current data are analyzed, if the risk combination exists in the sets of variable control combinations, the dynamic data corresponding to the current data is taken as a potential risk point; Step S5: the steps S1 to S4 are cycled until all dynamic data in the task execution constraint data are marked as the current data, the cycle is ended, and all potential risk points are obtained.

[0010] Further, the content of dynamically generating the individualized interaction strategy comprises: determining an interaction mode according to the task executability and the potential risk point; the interaction mode includes a fast execution mode, a negotiation interaction mode and an explanation suggestion mode; If the interaction mode is the fast execution mode, the structured task representation is taken as the final task representation, and the task execution module is executed; If the interaction mode is not the fast execution mode, the interaction style is determined according to the user state information corresponding to the standardized state information in the structured task representation; the number of risk combinations corresponding to each potential risk point is counted and taken as the risk severity of each potential risk point; a risk set is preset, and the risk coefficient corresponding to each potential risk point is sequentially obtained from the risk set, and the product of the risk severity and the corresponding risk coefficient of each potential risk point is sequentially calculated to obtain the interaction priority of each potential risk point; According to the interaction style, the corresponding interaction voice feature is obtained from the preset voice mapping table; according to the interaction mode, the interaction style and the interaction priority of each potential risk point, the interaction output content is dynamically generated; the interaction voice feature and the interaction output content are integrated to form a personalized interaction strategy.

[0011] Further, the content of determining the interaction mode includes: If there is no potential risk point, the interaction mode is determined to be the fast execution mode; If there is a potential risk point, the task executability is compared with the preset execution threshold; If the task executability is greater than or equal to the execution threshold, the interaction mode is determined to be the negotiation interaction mode; If the task executability is less than the execution threshold, the interaction mode is determined to be the suggestion mode; The content of determining the interaction style includes: According to the emotion label in the user state information, the corresponding emotion state is obtained; from the preset emotion set, the emotion code corresponding to the emotion state is obtained; A preset weight set is included in the weight set, and the weight set includes the weight coefficient corresponding to the emotion code and the interaction willingness intensity; the emotion code and the interaction willingness intensity are weighted and summed to obtain the user interaction tendency score; different interaction styles are sequentially set with corresponding score intervals, and the user interaction tendency score is sequentially compared with the score interval of each interaction style to filter out the interaction style corresponding to the score interval of the user interaction tendency score.

[0012] Further, the content of sequentially setting the corresponding score interval for different interaction styles includes: A plurality of groups of historical state information are collected, and the user interaction tendency score corresponding to each group of historical state information is sequentially calculated and marked as a historical interaction tendency score; the interaction style corresponding to each group of historical state information is sequentially obtained, and the historical interaction tendency score corresponding to each group of historical state information is matched with the corresponding interaction style; The historical interaction tendency scores corresponding to each interaction style are sequentially calculated in mean, to obtain the average interaction tendency scores corresponding to each interaction style;Each interaction style is sorted according to the corresponding average interaction tendency score from large to small to generate a style sequence;The average interaction tendency corresponding to each two adjacent interaction styles in the style sequence is calculated in mean to obtain a limit score;According to the two limit scores corresponding to each interaction style, the corresponding score interval of each interaction style is sequentially set.

[0013] Further, the content of the dynamically generated interaction output content includes: The number of potential risk points is counted to obtain a risk number;The maximum score is calculated, and the ratio of the user interaction tendency score to the maximum score is calculated to obtain an interaction proportion; The product of the risk number and the interaction proportion is calculated to obtain an interaction number;All potential risk points are sorted according to the corresponding interaction priority from large to small to obtain the potential risk points ranked in the front The interaction risk points, The interaction number is rounded up. If the interaction mode is a negotiation interaction mode, the interaction output content is dynamically generated according to the interaction mode, the interaction style, the structured task representation, and all the interaction risk points; If the interaction mode is a suggestion mode, the risk combination corresponding to each interaction risk point is obtained;Each risk combination is sequentially analyzed to obtain the boundary set corresponding to each risk combination;Each boundary set includes two adjacent variable control combinations in the corresponding risk combination, and the execution results of the two adjacent variable control combinations are different;The value corresponding to the current data in the variable control combination with the execution result of execution failure in each boundary set is taken as the risk value of the corresponding risk interaction point; The interaction output content is dynamically generated according to the interaction mode, the interaction style, the structured task representation, all the interaction risk points, and the risk value of each interaction risk point.

[0014] Further, the content of determining the final task representation includes: The user feedback signal is converted into corresponding text content in real time, and is marked as feedback text;The feedback text and the interaction output content are converted into a standardized format to obtain standardized interaction information;The standardized interaction information and the standardized task information are fused to obtain the final task representation.

[0015] The technical effects and advantages of the robot adaptive human-computer interaction and task instruction understanding system of the application are: By comprehensively sensing the multi-modal interaction signals of the user and dynamically analyzing, accurate understanding of the user's intention and emotional state is realized, and the intelligence, flexibility and user experience of human-computer interaction are improved; by using the method of quantitatively evaluating task executability and accurately identifying potential risk points, the robot can actively perceive the execution environment and constraint conditions, realize risk avoidance and strategy optimization in the task execution process, form a closed-loop adaptive control, and improve the safety and reliability of task execution; based on the task executability, potential risk points and user state information, a differentiated interaction strategy is dynamically generated, which can effectively avoid task execution risks and improve the personalization and intelligence level of human-computer interaction; real-time capture of user feedback and dynamic correction during the interaction process ensure the accuracy and executability of the final task representation, and generate a directly executable time sequence task step, improving the flexibility and stability of task execution; the embodiment integrates multiple frontier technologies such as multi-modal perception, context understanding, intelligent decision-making, adaptive interaction, realizes the intelligentization of the whole process from user intention analysis to task execution, thereby enhancing the adaptability and user experience of the robot in complex dynamic environment, improving the task execution efficiency, and meeting the diversified application requirements. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The figure is a schematic diagram of the robot adaptive human-computer interaction and task instruction understanding system of embodiment 1 of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0018] Embodiment 1 Please refer to Figure 1 The robot adaptive human-computer interaction and task instruction understanding system described in the embodiment includes a signal acquisition module, a state recognition module, an instruction analysis module, a task evaluation module, a strategy generation module, a task determination module and a task execution module; each module is connected by wired and / or wireless means to realize data transmission between modules.

[0019] The signal acquisition module is used for synchronously acquiring multi-modal interaction signals of the user.

[0020] The multi-modal interaction signals include instruction signals and expression signals; The instruction signals include voice instructions and gesture instructions; The voice instruction refers to an explicit operation command or request expressed by a user through natural language orally, such as turning on a light, moving materials, etc. The voice instruction is acquired through a microphone array integrated in the robot. The gesture instruction refers to an action command conveyed by a user through hand, arm or body movement, such as pointing to a lamp in a living room or pointing to No. 1 assembly line in a workshop, etc. The gesture instruction is acquired by acquiring a whole body image (i.e. an image of the whole body of the user) of the user through a camera integrated in the robot, and is acquired after applying an action recognition algorithm to the whole body image. The action recognition algorithm is, for example, OpenPose (for skeleton key point detection), MediaPipe (real-time gesture tracking), spatio-temporal graph convolutional network (ST-GCN), etc. The expression signal refers to an expression feature naturally revealed by a user's face, such as smiling, frowning, surprise, confusion, etc. The expression signal is acquired by acquiring a face image (i.e. an image of the face of the user) of the user through a camera integrated in the robot, and is acquired after applying a facial expression recognition algorithm to the face image. The facial expression recognition algorithm is, for example, a convolutional neural network (CNN) expression classifier, a FER model, a FACS-based analysis method, etc. The instruction signal is used to convey a specific task requirement that the user wants the robot to perform. The expression signal is used to reflect the emotional state of the user to assist in optimizing human-robot interaction decisions.

[0021] The state recognition module is configured to perceive and process the multi-modal interaction signal, and recognize user state information.

[0022] The content of the recognized user state information includes: The voice instruction is subjected to frame processing based on preset frame parameters, to obtain a plurality of short-time voice frames. The frame parameters include a sampling rate, a frame length and a frame shift, which are pre-set by a person skilled in the art according to actual conditions. The time length of each short-time voice frame is usually 20-30 ms. A window function is applied to each short-time voice frame (i.e. each short-time voice frame is multiplied by a window function) to obtain a window function weighted frame corresponding to each short-time voice frame, so as to reduce spectral leakage. The window function is, for example, a Hamming window, a Hanning window, etc. A prosody feature extraction method is used to extract prosody feature parameters from all window function weighted frames. The prosody feature extraction method includes autocorrelation method, short-time energy method, energy envelope peak detection method and speech activity detection method. The prosody feature parameters include pitch, volume, speech rate and pause frequency. The autocorrelation method is used to extract the pitch, the short-time energy method is used to extract the volume, the energy envelope peak detection method is used to extract the speech rate, and the speech activity detection method is used to extract the pause frequency. The prosody feature parameters and the expression signal are input into a trained state prediction model to predict corresponding user state information. It should be noted that the prosody feature extraction method is a prior art, and the specific process will not be described in detail here.

[0023] wherein the user state information comprises an emotion label and an interaction willingness intensity; The emotion label is a digital label corresponding to an emotion state, and emotion labels corresponding to different emotion states are all different. The emotion state is the current emotional experience state of the user, such as joy, sadness, anger, etc. The interaction willingness intensity is the degree of initiative of the user to participate in human-computer interaction, and the corresponding value interval is The prosodic feature parameter can reflect the rhythm, stress and pause mode of the voice, wherein high pitch, high volume, fast speech speed and low pause frequency usually correspond to happy or excited emotions and higher interaction willingness intensity, low pitch, low volume, slow speech speed and high pause frequency usually correspond to low or tired emotions and lower interaction willingness; the expression signal can provide visual intuitive information and work together with the prosodic feature parameter to enhance the judgment of the emotion state and the interaction willingness intensity, so that the system can more accurately perceive the current emotional experience and the degree of initiative of the user; The state prediction model is a deep neural network model, which includes an input layer, a hidden layer and an output layer. Each hidden layer includes a plurality of neurons, and each neuron is connected with the next layer of neurons, and the connection includes a weight which determines the importance and influence of data transmission in the neural network. An activation function is applied to each neuron between the hidden layer and the output layer, which introduces nonlinearity and allows the network to learn more complex patterns and features. The deep neural network model is a prior art, and the specific training process will not be described in detail here.

[0024] The instruction analysis module is configured to perform context-enhanced semantic understanding on the voice instruction in the multi-modal interaction signal according to the user state information, and generate a structured task representation.

[0025] The content of generating the structured task representation includes: By calling the API interface of the cloud voice recognition service (such as Baidu, Xunfei, etc.), the voice instruction is converted into corresponding text content in real time, and is marked as a task text. A regular matching method is used to extract the corresponding task entity from the gesture instruction, and the task entity includes a task object and a task position. The task entity is supplemented to the task text to obtain a complete text. It should be noted that the regular matching method is a prior art, and the specific process will not be described in detail here. The complete text and the user state information are converted into a standardized format to obtain standardized task information and standardized state information; the standardized task information and the standardized state information are combined to generate a structured task representation; wherein the complete text and the user state information are converted into a standardized format through a robot internally preset standardization rule, the standardization rule is pre-designed by a person skilled in the art according to the system function requirement and the task processing requirement of the robot; the standardized format is a unified data representation form, which is used to represent the complete text and the user state information, so that the robot can directly understand.

[0026] For example, the task text is to turn on the light, and the gesture instruction is to point to the light in the living room; then the task object corresponding to the gesture instruction is the light, and the task location is the living room; the light and the living room are supplemented to the task text, and then the complete text obtained is to turn on the light in the living room; It should be understood that the semantic understanding of the context enhancement of the voice instruction is to supplement the task text converted from the voice instruction according to the user state information and the task entity extracted from the gesture instruction, so that the system can more accurately understand the user's intention, thereby forming a complete, accurate and executable structured task representation.

[0027] The task evaluation module is used to collect task execution constraint data, dynamically evaluate the task executability of the structured task representation based on the task execution constraint data, and identify potential risk points.

[0028] The task execution constraint data includes self-constraints and environmental constraints; The self-constraints refer to the physical, functional and resource limitations of the robot itself, including but not limited to energy constraints (such as battery capacity, power consumption limitations, etc.), kinematic constraints (such as maximum speed, turning radius, etc.), dynamic constraints (such as load capacity, maximum thrust, etc.), etc.; the self-constraints are obtained through the specification parameters of the robot and the built-in battery management system; The environmental constraints refer to the external conditions and limitations imposed on the task execution by the external environment in which the robot is located, including but not limited to spatial structure (such as passage width, passage slope, etc.), dynamic environment (such as moving obstacles, crowd density, etc.), etc.; the environmental constraints are obtained by processing the dynamic environment data through the built-in multi-modal sensors (such as lidar, camera, etc.) of the robot, and combining the environmental modeling algorithm (such as SLAM algorithm, grid map, topological map, etc.) and the dynamic inference algorithm (such as Kalman filter and particle filter for dynamic object tracking, trajectory prediction network, etc.); wherein the dynamic environment data refers to the environment information that changes over time and is perceived by the robot in real time through the multi-modal sensors, such as moving obstacles, crowd flow, etc.

[0029] The content of dynamically evaluating the task executability of the structured task representation includes: Collect a plurality of sets of historical execution constraint data, the historical execution constraint data being task execution constraint data collected by the robot before a historical execution of a structured task representation, obtained from a database built in the robot; each set of data in the historical execution constraint data is taken as a data set, and the data set corresponds to a data type in the task execution constraint data; the data in the same data set are compared, and if the values of each data in the data set are the same, the corresponding data set is marked as static data, and if the values of each data in the data set are not the same, the corresponding data set is marked as dynamic data; wherein the static data is, for example, maximum speed, load capacity, channel width, etc., and the dynamic data is, for example, battery power, power consumption limit, moving obstacle, crowd density, etc. A preset fluctuation set is provided, and the fluctuation set includes a fluctuation range corresponding to each dynamic data, which is set by a person skilled in the art according to the actual situation, and the fluctuation range is, for example, , and the like; the fluctuation range corresponding to each dynamic data is obtained from the fluctuation set; based on the fluctuation range, the fluctuation interval corresponding to each dynamic data in the task execution constraint data is calculated in sequence, and the fluctuation interval of each dynamic data is set to be uniformly distributed; specifically, the product of the maximum value and the minimum value of each dynamic data and the corresponding fluctuation range is calculated in sequence to obtain the maximum variable and the minimum variable of each dynamic data; the sum of each dynamic data and the corresponding maximum variable is calculated in sequence to obtain the maximum fluctuation value of each dynamic data; the difference between each dynamic data and the corresponding minimum variable is calculated in sequence to obtain the minimum fluctuation value of each dynamic data; and the fluctuation interval corresponding to each dynamic data is constructed according to the maximum fluctuation value and the minimum fluctuation value of each dynamic data. A value is randomly selected from each fluctuation interval, and a set of fluctuation constraint data is constructed; in the same way, a total of sets of fluctuation constraint data are constructed, which is an integer greater than 1, and each set of fluctuation constraint data is different; all static data are added to each set of fluctuation constraint data to obtain sets of random constraint data; the structured task representation is simulated and executed in sequence under the condition of each set of random constraint data, and an execution result corresponding to each set of random constraint data is obtained, the execution result including execution success and execution failure; The number of random constraint data with execution success is counted and marked as a success number; the ratio between the success number and is calculated to obtain the task executability.

[0030] The content of simulating and executing the structured task representation in sequence includes: The physical simulator (such as Gazebo, Webots, etc.) is pre-constructed by a person skilled in the art according to all static data, and the dynamic data is input into the physical simulator The group fluctuation constraint data is sequentially input into the physical simulator, and the execution result of the structured task representation is simulated in the physical simulator The execution result corresponding to each group of fluctuation constraint data is obtained, that is, each time a group of fluctuation constraint data is input into the physical simulator, the structured task representation is simulated once in the physical simulator.

[0031] The step of identifying the potential risk point comprises: Step S1: randomly selecting a dynamic data and marking it as current data; Step S2: according to the current data, the group fluctuation constraint data is divided into group variable control combinations, each of which contains group fluctuation constraint data, ; wherein the group fluctuation constraint data corresponding to each group of variable control combinations only has different values of the current data, and the values of the remaining dynamic data are consistent; For example, there are three dynamic data, the fluctuation interval of dynamic data 1 is 1 and 2, the fluctuation interval of dynamic data 2 is 3 and 4, and the fluctuation interval of dynamic data 3 is 5, so the fluctuation constraint data includes , , and ; taking dynamic data 1 as the current data, the dynamic data 1 corresponds to two groups of variable control combinations, the variable control combination 1 includes and , and the variable control combination 2 includes and ; Step S3: comparing the execution results of the group fluctuation constraint data corresponding to the same group of variable control combinations, if the execution results of the group fluctuation constraint data are all the same, the corresponding variable control combination is not marked, if the execution results of the group fluctuation constraint data are not all the same, the corresponding variable control combination is marked as a risk combination; Step S4: analyzing the group variable control combinations corresponding to the current data, if there is a risk combination in the group variable control combinations, the dynamic data corresponding to the current data is taken as a potential risk point, if there is no risk combination in the group variable control combinations, the dynamic data corresponding to the current data is not taken as a potential risk point; Step S5: Circulating step S1 to step S4 until all dynamic data in the task execution constraint data are marked as current data, and the circulation ends, and all potential risk points are obtained.

[0032] It should be appreciated that the purpose of dynamically evaluating task executability and identifying potential risk points is to provide scientific decision-making basis and active risk management capability for the robot, so as to provide key decision input for subsequent personalized interaction strategy generation, and realize intelligent closed loop from risk perception to strategy optimization. Specifically, the probabilistic evaluation result of task executability directly affects the generation direction of personalized interaction strategy; when the executability is high, the interaction strategy of direct execution can be used, and only simple confirmation is needed; when the executability is at a medium level, a more cautious interaction strategy needs to be generated, such as explaining the possible risks to the user in detail and seeking advice; when the executability is low, the reason why the task cannot be executed needs to be explicitly stated; at the same time, the identified potential risk points provide accurate optimization direction for the personalized interaction strategy; for example, when the battery power is identified as a risk point, the personalized interaction strategy will actively ask the user "the current power is low, do you want to charge first and then execute the task"; when the crowd density is identified as a risk point, the personalized interaction strategy will suggest "detecting the crowd density, it is suggested to execute the task after 5 minutes"; such strategy generation mechanism based on quantitative evaluation and risk identification enables the robot to dynamically adjust the interaction mode with the user according to the actual constraint conditions and risk conditions, which not only guarantees the success rate of task execution, but also improves the individualization and intelligence level of user experience, so as to realize the adaptive human-computer interaction of "adapting to the situation and taking measures according to the risk".

[0033] The strategy generation module is configured to dynamically generate a personalized interaction strategy according to the structured task representation, the task executability, and the potential risk points.

[0034] The content of dynamically generating a personalized interaction strategy includes: According to the task executability and the potential risk points, an interaction mode is determined; the interaction mode includes a quick execution mode, a negotiation interaction mode, and an explanation and suggestion mode; If the interaction mode is the quick execution mode, the structured task representation is taken as the final task representation, and the task execution module is executed. If the interaction mode is not the fast execution mode, an interaction style is determined according to user state information corresponding to standardized state information in the structured task representation; the number of risk combinations corresponding to each potential risk point is counted and taken as the risk severity of each potential risk point; a risk set is preset, and the risk set includes a risk coefficient corresponding to each potential risk point, which is preset by a person skilled in the art according to actual conditions; the risk coefficient corresponding to each potential risk point is sequentially obtained from the risk set, and the product of the risk severity of each potential risk point and the corresponding risk coefficient is sequentially calculated to obtain the interaction priority of each potential risk point; According to the interaction style, corresponding interaction voice features are obtained from a preset voice mapping table; interaction output content is dynamically generated according to the interaction mode, the interaction style, and the interaction priority of each potential risk point; and the interaction voice features and the interaction output content are integrated to form a personalized interaction strategy.

[0035] Among them, the smaller the risk coefficient corresponding to the potential risk point that the robot can automatically avoid or does not need user intervention; and the larger the risk coefficient corresponding to the potential risk point that has a significant impact on the task executability and needs user intervention; for example, moving obstacles will affect the execution of the robot, but the robot can automatically avoid moving obstacles, so the risk coefficient corresponding to the moving obstacle is small; and insufficient battery power will directly cause the robot to be unable to complete the task, and needs user charging intervention, so the risk coefficient corresponding to the battery power is large; The voice mapping table includes interaction voice features corresponding to different interaction styles, which are preset by a person skilled in the art according to actual conditions; the interaction voice features include tone, volume, and speed; the interaction style is, for example, enthusiastic and detailed type (interactive initiative, information rich, strong interactivity and emotional expression, suitable for scenarios with high user interaction willingness and positive emotions), patient and caring type (interactive gentle and considerate, moderate information, focusing on user experience and emotional care, suitable for scenarios with medium user interaction willingness or sensitive emotions), concise and efficient type (interactive concise and direct, less information, focusing on task execution, suitable for scenarios with low user interaction willingness or negative emotions and the desire to quickly complete the task), etc.

[0036] The content for determining the interaction mode includes: If there is no potential risk point, the interaction mode is determined to be the fast execution mode; wherein the task executability corresponding to the structured task representation is high and the risk is controllable in the fast execution mode, and the robot can directly execute the task without complex interaction with the user; If there is a potential risk point, the task executability is compared with a preset execution threshold; the execution threshold is preset by a person skilled in the art according to actual conditions; If the task executability is greater than or equal to the execution threshold, it is determined that the interaction mode is a negotiation interaction mode; in the negotiation interaction mode, the task executability corresponding to the structured task representation is moderate and there is a potential risk, the robot needs to interact with the user, prompt the potential risk and seek the user's opinion; If the task executability is less than the execution threshold, it is determined that the interaction mode is an explanation suggestion mode; in the explanation suggestion mode, the task executability corresponding to the structured task representation is low and the risk is significant, the robot needs to explain to the user the reason why the structured task representation cannot be executed.

[0037] The content of determining the interaction style includes: According to the emotion label in the user state information, the corresponding emotion state is obtained; from a preset emotion set, an emotion code corresponding to the emotion state is obtained; the emotion set includes emotion codes corresponding to different emotion states, and the value range of the emotion code is which is pre-set by a person skilled in the art according to actual conditions; wherein, the emotion code corresponding to a more positive and more active emotion state is larger, for example, joy, excitement, etc.; on the contrary, the emotion code corresponding to a more negative and more passive emotion state is smaller, for example, sadness, frustration, etc. A preset weight set includes weight coefficients corresponding to the emotion code and the interaction willingness intensity, which is pre-set by a person skilled in the art according to actual conditions; the emotion code and the interaction willingness intensity are weighted and summed according to the weight set to obtain a user interaction tendency score; different interaction styles are sequentially set with corresponding score intervals, the user interaction tendency score is sequentially compared with the score interval of each interaction style, the interaction style corresponding to the score interval in which the user interaction tendency score is located is screened out, and is used as the interaction style when the robot interacts with the user.

[0038] The content of sequentially setting the corresponding score interval for different interaction styles includes: A plurality of groups of historical state information are collected, the historical state information is user state information recognized by the robot when the robot interacts with the user at a historical time, and is obtained through a database built in the robot; the user interaction tendency score corresponding to each group of historical state information is sequentially calculated and marked as a historical interaction tendency score; the interaction style corresponding to each group of historical state information is sequentially obtained, and the historical interaction tendency score corresponding to each group of historical state information is matched with the corresponding interaction style; The average interaction tendency score corresponding to each interaction style is obtained by sequentially performing mean calculation on the historical interaction tendency scores corresponding to each interaction style; each interaction style is sorted according to the corresponding average interaction tendency score from large to small to generate a style sequence; the average interaction tendency corresponding to each two adjacent interaction styles in the style sequence is calculated by mean calculation to obtain a limit score; the score interval of each interaction style is sequentially set according to the two limit scores corresponding to each interaction style; If the interaction style is ranked first in the style sequence, the maximum value (i.e. 1) of the emotion code and the maximum value (i.e. 1) of the interaction willingness intensity are obtained, and the maximum value of the user interaction tendency score is calculated according to the weight set and marked as the maximum score; the score interval of the corresponding interaction style is set according to the limit score and the maximum score of the corresponding interaction style; similarly, if the interaction style is ranked last in the style sequence, the score interval of the corresponding interaction style is set according to the limit score and 0 of the corresponding interaction style; it should be noted that the minimum value of the emotion code and the minimum value of the interaction willingness intensity are both 0, so the minimum value of the user interaction tendency score calculated is also 0.

[0039] The content of the dynamically generated interaction output content includes: The number of potential risk points is counted to obtain a risk number; the ratio of the user interaction tendency score to the maximum score is calculated to obtain an interaction proportion; the product of the risk number and the interaction proportion is calculated to obtain an interaction number; all potential risk points are sorted according to the corresponding interaction priority from large to small to obtain the potential risk points ranked in the front , and marked as interaction risk points, , which is the interaction number rounded up; If the interaction mode is a negotiation interaction mode, the NLG (Natural Language Generation) method is used to dynamically generate the interaction output content according to the interaction mode, the interaction style, the structured task representation, and all the interaction risk points; the NLG method is a prior art, and the specific process will not be described in detail here; If the interaction mode is a suggestion mode, the risk combination corresponding to each interaction risk point is obtained; each risk combination is sequentially analyzed to obtain a boundary set corresponding to each risk combination; each boundary set includes two adjacent variable control combinations in the corresponding risk combination and the execution results are different; the value of the current data in the variable control combination with the execution result of execution failure is taken as the risk value of the corresponding risk interaction point; The NLG (Natural Language Generation) method is used to dynamically generate the interaction output content according to the interaction mode, the interaction style, the structured task representation, all the interaction risk points, and the risk value of each interaction risk point.

[0040] For example, the interaction mode is a negotiation interaction mode, the interaction style is a patient caring type, the structured task representation is to carry materials to the No. 2 assembly line, and the interaction risk point is the battery power (currently 35%) and the crowd density (5 people are detected near the No. 2 assembly line); therefore, the dynamically generated interaction output content is: I understand that you need to carry materials to the No. 2 assembly line; however, before starting, I would like to confirm two conditions with you: first, my current battery power is 35%, and I can complete this carrying task, but I need to charge after completion; do you think it is appropriate to perform now? Second, I detect that several workers are active near the No. 2 assembly line, and in order to ensure safety, I will reduce the moving speed and maintain a safe distance; this may take an additional 3-5 minutes; can you accept this? If you agree to these two points, I can start the task now.

[0041] The task determination module is configured to control the robot to perform human-robot interaction with the user according to the personalized interaction strategy, and to capture user feedback signals in real time, and to dynamically modify the structured task representation based on the user feedback signals to determine a final task representation.

[0042] The content of determining the final task representation includes: The user feedback signals are converted into corresponding text content in real time and marked as feedback text; the user feedback signals are a plurality of rounds of voice instructions fed back by the user during the interaction with the robot; the feedback text and the interaction output content are converted into a standardized format to obtain standardized interaction information; the standardized interaction information and the standardized task information are input into a trained NLP model (such as T5, BART, BERT2BERT, etc.) to obtain the final task representation; the NLP model is a prior art, and the specific training process will not be described in detail here.

[0043] The task execution module is configured to generate a time sequence task step according to the final task representation, and to control the robot to execute the final task representation according to the time sequence task step.

[0044] The content of generating a time sequence task step according to the final task representation includes: A hierarchical task network (HTN) planning method is used to decompose the final task representation to generate a plurality of time sequence task steps that can be directly executed by the robot; the time sequence task step refers to an executable action sequence after HTN decomposition and sorting; the hierarchical task network (HTN) planning method is a prior art, and the specific implementation process will not be described in detail here.

[0045] For example, the final task representation is to install part A on part B, and after HTN decomposition and sorting, the time sequence task steps that can be obtained are in turn: grasping part A, moving to the position of part B, installing part A on part B, and releasing part A, thereby completing the final task representation.

[0046] The embodiment comprehensively perceives multi-modal interaction signals of a user and dynamically analyzes to accurately understand the user's intention and emotional state, and improve the intelligence, flexibility and user experience of human-computer interaction; the method of quantitatively evaluating task executability and accurately identifying potential risk points enables the robot to actively perceive the execution environment and constraint conditions, realize risk avoidance and strategy optimization in the task execution process, form a closed-loop adaptive control, and improve the safety and reliability of task execution; based on the task executability, potential risk points and user state information, a differentiated interaction strategy is dynamically generated, which can effectively avoid task execution risks and improve the personalization and intelligence level of human-computer interaction; real-time capture of user feedback and dynamic correction during the interaction process ensure the accuracy and executability of the final task representation, and generate a directly executable time sequence task step, improving the flexibility and stability of task execution; the embodiment integrates multiple frontier technologies such as multi-modal perception, context understanding, intelligent decision-making, adaptive interaction, realizes the intelligentization of the whole process from user intention analysis to task execution, thereby enhancing the adaptability and user experience of the robot in a complex dynamic environment, improving the task execution efficiency, and meeting the diversified application requirements.

[0047] Embodiment 2 The application also provides an electronic device. The electronic device can include one or more processors and one or more memories. The memory stores computer readable code which, when executed by the one or more processors, can perform the robot adaptive human-computer interaction and task instruction understanding system as described above.

[0048] The method or system according to the embodiments of the application can also be implemented by means of the architecture of the electronic device shown in the application. The electronic device can include a bus, one or more CPUs, ROM, RAM, a communication port connected to a network, input / output, a hard disk, etc. The storage device in the electronic device, such as ROM or hard disk, can store the robot adaptive human-computer interaction and task instruction understanding system provided by the application. Further, the electronic device can also include a user interface. Of course, the architecture shown in the application is only exemplary, and when implementing different devices, one or more components of the electronic device shown in the application can be omitted according to actual needs.

[0049] Embodiment 3 One embodiment of the present application discloses a computer readable storage medium. The computer readable storage medium stores computer readable instructions. When the computer readable instructions are run by a processor, the robot adaptive human-robot interaction and task instruction understanding system according to the embodiments of the present application described above with reference to the accompanying drawings can be executed. The storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and cache memory, etc. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0050] In addition, according to the embodiments of the present application, the processes described above with reference to the schematic diagrams can be implemented as a computer software program. For example, the present application provides a non-transitory machine readable storage medium storing machine readable instructions executable by a processor to perform instructions corresponding to the method steps provided by the present application, for example: the robot adaptive human-robot interaction and task instruction understanding system. When this computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed.

[0051] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0052] It should be noted that in this document, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of another identical element in the process, method, article or device comprising the element.

[0053] In the description of the present application, it should be understood that the terms "first", "second" and the like are only used to distinguish description, and cannot be understood as indicating or implying relative importance.

[0054] In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0055] In the description of the application, the meaning of "several" is one or more, and the meaning of "a large number" is two or more.

[0056] In the description of the present application, the description of the terms "one embodiment", "several embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0057] For the formulas in the present application, the values are calculated by de-dimensioning, the formulas are obtained by collecting a large amount of data to simulate the most recent real situation, and the preset parameters and threshold values in the formulas are set by the person skilled in the art according to the actual situation.

[0058] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and purposes of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A robotic adaptive human-robot interaction and task instruction understanding system, characterized in that, The method comprises the following steps: a signal acquisition module is used to synchronously acquire multi-modal interaction signals of a user; a state recognition module is used to perform perception processing on the multi-modal interaction signals to recognize user state information; an instruction analysis module is used to perform context-enhanced semantic understanding on voice instructions in the multi-modal interaction signals according to the user state information, and generate structured task representation; a task evaluation module is used to collect task execution constraint data, dynamically evaluate task executability of the structured task representation based on the task execution constraint data, and identify potential risk points; a strategy generation module is used to dynamically generate an individualized interaction strategy according to the structured task representation, the task executability and the potential risk points; a task determination module is used to control a robot to perform human-robot interaction with the user according to the individualized interaction strategy, capture user feedback signals in real time, dynamically correct the structured task representation based on the user feedback signals, and determine final task representation; a task execution module is used to generate time sequence task steps according to the final task representation, and control the robot to execute the final task representation according to the time sequence task steps.

2. The robotic adaptive human interaction and task instruction understanding system of claim 1, wherein, The multi-modal interaction signals include instruction signals and expression signals; the instruction signals include voice instructions and gesture instructions; The content of the recognized user state information includes: preset framing parameters are used to perform framing processing on the voice instructions based on the framing parameters to obtain a plurality of short-time voice frames; a window function is applied to each short-time voice frame respectively to obtain a window function weighted frame corresponding to each short-time voice frame; rhythm feature parameters are extracted from all window function weighted frames, and the rhythm feature parameters and the expression signals are input into a trained state prediction model to predict corresponding user state information; wherein the user state information includes emotion labels and interaction intention intensity.

3. The robotic adaptive human interaction and task instruction understanding system of claim 2, wherein, The content of the generated structured task representation includes: the voice instructions are converted into corresponding text content in real time and marked as task text; task entities including task objects and task positions are extracted from the gesture instructions; the task entities are supplemented into the task text to obtain complete text; the complete text and the user state information are converted into standardized formats to obtain standardized task information and standardized state information; the standardized task information and the standardized state information are combined to generate structured task representation.

4. The robotic adaptive human interaction and task instruction understanding system of claim 3, wherein, The content of the dynamically evaluated task executability of the structured task representation includes: a plurality of groups of historical execution constraint data are collected, and the task execution constraint data and the same type of data in each group of historical execution constraint data are all taken as a group of data set; the data in the same group of data set are compared respectively, if the numerical values of each data in the data set are all the same, the corresponding data of the corresponding data set is marked as static data, if the numerical values of each data in the data set are not all the same, the corresponding data of the corresponding data set is marked as dynamic data; The preset fluctuation set includes a fluctuation range corresponding to each dynamic data, and the fluctuation range corresponding to each dynamic data is obtained from the fluctuation set; the fluctuation range is used to sequentially calculate a fluctuation interval corresponding to each dynamic data in the task execution constraint data, and the fluctuation interval of each dynamic data is set to be uniformly distributed; A value is randomly selected from each fluctuation interval, and a set of fluctuation constraint data is constructed; in the same way, a total of sets of fluctuation constraint data are constructed; all static data are added to each set of fluctuation constraint data respectively, to obtain sets of random constraint data; under the condition of each set of random constraint data, the structured task representation is sequentially simulated to be executed, to obtain an execution result corresponding to each set of random constraint data, the execution result including execution success and execution failure; The statistical execution result is the number of successfully executed random constraint data, and is marked as the success number; the ratio between the success number and the total number of constraint data is calculated to obtain the task executability.

5. The robotic adaptive human interaction and task instruction understanding system of claim 4, wherein, The step of identifying the potential risk points comprises: Step S1: randomly selecting one dynamic data and marking it as the current data; Step S2: According to the current data, the group fluctuation constraint data is divided into group variable control combinations, each of which contains group fluctuation constraint data; Step S3: corresponding to the same group of variable control combination Compare the execution results of the group fluctuation constraint data, if If the execution results of the group fluctuation constraint data are not all the same, mark the corresponding variable control combination as a risk combination; Step S4: For the current data If the control combination of group variables is analyzed, If there is a risk combination in the group variable control combination, the dynamic data corresponding to the current data will be used as a potential risk point; Step S5: repeating steps S1 to S4 until all dynamic data in the task execution constraint data are marked as the current data, and the cycle is ended, and all potential risk points are obtained.

6. The robotic adaptive human interaction and task instruction understanding system of claim 5, wherein, The content of dynamically generating the personalized interaction strategy comprises: According to the task executability and the potential risk points, the interaction mode is determined, which comprises a quick execution mode, a negotiation interaction mode and an explanation suggestion mode; If the interaction mode is the quick execution mode, the structured task representation is taken as the final task representation, and the task execution module is executed; If the interaction mode is not the quick execution mode, the interaction style is determined according to the user state information corresponding to the standardized state information in the structured task representation; the number of risk combinations corresponding to each potential risk point is counted and taken as the risk severity of each potential risk point; a risk set is preset, and the risk coefficient corresponding to each potential risk point is sequentially obtained from the risk set, and the product of the risk severity and the corresponding risk coefficient of each potential risk point is sequentially calculated to obtain the interaction priority of each potential risk point; According to the interaction style, the corresponding interaction voice feature is obtained from the preset voice mapping table; the interaction output content is dynamically generated according to the interaction mode, the interaction style and the interaction priority of each potential risk point; the interaction voice feature and the interaction output content are integrated to form a personalized interaction strategy.

7. The robotic adaptive human interaction and task instruction understanding system of claim 6, wherein, The content of determining the interaction mode comprises: If there is no potential risk point, the interaction mode is determined to be the quick execution mode; If there is a potential risk point, the task executability is compared with a preset execution threshold; If the task executability is greater than or equal to the execution threshold, the interaction mode is determined to be the negotiation interaction mode; If the task executability is less than the execution threshold, the interaction mode is determined to be the explanation suggestion mode; The content of determining the interaction style comprises: According to the emotion label in the user state information, the corresponding emotion state is obtained; the emotion coding corresponding to the emotion state is obtained from the preset emotion set; A preset weight set comprises a weight coefficient corresponding to the emotion coding and the interaction willingness intensity; the emotion coding and the interaction willingness intensity are weighted and summed according to the weight set to obtain a user interaction tendency score; different interaction styles are sequentially set with corresponding score intervals, and the user interaction tendency score is sequentially compared with the score interval of each interaction style to screen out the interaction style corresponding to the score interval where the user interaction tendency score is located.

8. The robotic adaptive human interaction and task instruction understanding system of claim 7, wherein, The content of sequentially setting the corresponding score interval for different interaction styles comprises: A plurality of sets of historical state information are collected, and a user interaction tendency score corresponding to each set of historical state information is calculated in sequence and marked as a historical interaction tendency score; an interaction style corresponding to each set of historical state information is obtained in sequence, and the historical interaction tendency score corresponding to each set of historical state information is matched with the corresponding interaction style; The historical interaction tendency scores corresponding to each interaction style are calculated in sequence to obtain an average interaction tendency score corresponding to each interaction style; each interaction style is sorted according to the corresponding average interaction tendency score from large to small to generate a style sequence; the average interaction tendency scores corresponding to two adjacent interaction styles in the style sequence are calculated in sequence to obtain a boundary score; and the corresponding score interval of each interaction style is set in sequence according to the two boundary scores corresponding to each interaction style.

9. The robotic adaptive human interaction and task instruction understanding system of claim 8, wherein, The content of the dynamically generated interaction output content includes: The number of potential risk points is counted to obtain a risk number; the maximum score is calculated, and the ratio of the user interaction tendency score to the maximum score is calculated to obtain an interaction proportion; The product of the risk quantity and the interaction proportion is calculated to obtain an interaction quantity; all potential risk points are sorted according to the corresponding interaction priorities from large to small to obtain potential risk points ranked in front and marked as interaction risk points, The interaction quantity is rounded up. If the interaction mode is a negotiation interaction mode, the interaction output content is dynamically generated according to the interaction mode, the interaction style, the structured task representation, and all the interaction risk points; If the interaction mode is an explanation suggestion mode, a risk combination corresponding to each interaction risk point is obtained; each risk combination is analyzed in sequence to obtain a boundary set corresponding to each risk combination; each boundary set includes two adjacent variable control combinations in the corresponding risk combination, and the execution results of the two adjacent variable control combinations are different; the value corresponding to the current data in the variable control combination with the execution result of execution failure in each boundary set is taken as the risk value of the corresponding risk interaction point; The interaction output content is dynamically generated according to the interaction mode, the interaction style, the structured task representation, all the interaction risk points, and the risk value of each interaction risk point.

10. The robotic adaptive human interaction and task instruction understanding system of claim 9, wherein, The content of determining the final task representation includes: The user feedback signal is converted into corresponding text content in real time and marked as feedback text; the feedback text and the interaction output content are converted into a standardized format to obtain standardized interaction information; and the standardized interaction information and the standardized task information are fused to obtain the final task representation.

Images