Robot control method, computer device and computer readable storage medium

Abstract

The application provides a robot control method, computer equipment and a computer readable storage medium. The method comprises: acquiring multi-modal data of a user, the multi-modal data comprising posture data, sound data and force control data; performing feature extraction and fusion processing on the multi-modal data to obtain multi-modal fusion features; determining context information of human-computer interaction according to environmental information of an area where a robot is located and historical interaction data of the robot; generating a response strategy according to the multi-modal fusion features and the context information, and controlling movement of the robot according to the response strategy, so as to improve naturalness and accuracy of human-computer interaction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a robot control method, a computer device and a computer readable storage medium. BACKGROUND

[0002] With the continuous expansion of the application field of robots, the interaction demand between robots and humans is increasing, and the requirement for human-computer interaction technology is also increasing.

[0003] In the prior art, the interaction between robots and humans mostly depends on traditional computer interfaces based on keyboards and mice and traditional human-computer interfaces such as touch screens, which lack naturalness and adaptability and are difficult to achieve smooth communication and highly personalized response. And most of the existing human-computer interaction systems are based on single sensory input, which limits the perception ability of the system and the naturalness of the interaction. There are also preliminary multi-modal systems that try to combine visual and voice systems, but lack deep data fusion and advanced decision-making mechanisms, resulting in insufficient efficiency and accuracy of interaction.

[0004] Therefore, the existing human-computer interaction has the disadvantages of single interaction mode, data dependency, and lack of autonomous decision-making ability. SUMMARY

[0005] The present application aims to solve the problems of the prior art that the existing human-computer interaction has the limitations of single interaction mode, data dependency, and lack of autonomous decision-making ability.

[0006] To achieve the above-mentioned purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0007] In a first aspect, the embodiments of the present application provide a robot control method, which comprises:

[0008] Obtaining multi-modal data of a user, the multi-modal data comprising posture data, sound data and force control data;

[0009] Performing feature extraction and fusion processing on the multi-modal data to obtain multi-modal fusion features;

[0010] Determining context information of human-computer interaction according to environmental information of an area where the robot is located and historical interaction data of the robot;

[0011] Generating a response strategy according to the multi-modal fusion features and the context information, and controlling the motion of the robot according to the response strategy.

[0012] As an optional implementation manner, after the robot motion is controlled according to the response strategy, the method further includes:

[0013] obtaining feedback data input by the user based on the motion information of the robot;

[0014] correcting the response strategy according to the feedback data.

[0015] As an optional implementation manner, the multi-modal data of the user is obtained by:

[0016] obtaining posture data of the user by a visual sensor, the posture data including facial expression, gesture and body posture of the user;

[0017] obtaining voice data of the user by a voice sensor, the voice data including voice instruction and non-verbal sound of the user;

[0018] obtaining force control data of the user by a force sensor, the force control data including touch interaction signal and push-pull interaction signal of the user.

[0019] As an optional implementation manner, the multi-modal data is subjected to feature extraction and fusion processing to obtain multi-modal fusion features, including:

[0020] extracting posture features from the posture data by a visual encoder in a neural network model;

[0021] extracting voice features from the voice data by a voice encoder in the neural network model;

[0022] extracting force control features from the force control data by a force encoder in the neural network model;

[0023] obtaining the multi-modal fusion features from the posture features, the voice features and the force control features by a decoder in the neural network model.

[0024] As an optional implementation manner, the multi-modal fusion features are obtained from the posture features, the voice features and the force control features, including:

[0025] determining first attention weight of the posture features, second attention weight of the voice features and third attention weight of the force control features according to the posture features, the voice features and the force control features;

[0026] obtaining the multi-modal fusion features according to the posture features, the voice features, the force control features, the first attention weight, the second attention weight and the third attention weight.

[0027] As an optional implementation manner, the generating a response strategy according to the multi-modal fusion feature and the context information comprises:

[0028] generating an initial response strategy according to the multi-modal fusion feature by using an imitation learning algorithm;

[0029] determining a reward signal corresponding to the initial response strategy according to the multi-modal fusion feature and the context information by using a reinforcement learning algorithm, and iteratively correcting the initial response strategy according to the reward signal, and taking the initial response strategy at the end of iteration as the response strategy.

[0030] As an optional implementation manner, the controlling the robot motion according to the response strategy comprises:

[0031] determining a to-be-executed action of the robot according to the response strategy, and generating voice data corresponding to the to-be-executed action;

[0032] generating a control instruction according to the to-be-executed action and the voice data, and controlling the robot to execute the to-be-executed action while outputting the voice data according to the control instruction.

[0033] As an optional implementation manner, the generating voice data corresponding to the to-be-executed action comprises:

[0034] generating a natural language response matched with the to-be-executed action according to the to-be-executed action;

[0035] generating voice data corresponding to the to-be-executed action by voice synthesis according to the natural language response.

[0036] In a second aspect, an embodiment of the present application provides a robot control device, the device comprising:

[0037] an acquisition module configured to acquire multi-modal data of a user, the multi-modal data comprising posture data, voice data and force control data;

[0038] a processing module configured to perform feature extraction and fusion processing on the multi-modal data to obtain a multi-modal fusion feature;

[0039] a determination module configured to determine context information of human-computer interaction according to environment information of an area where a robot is located and historical interaction data of the robot;

[0040] a control module configured to generate a response strategy according to the multi-modal fusion feature and the context information, and control the robot motion according to the response strategy.

[0041] As an optional implementation, the apparatus further comprises a correction module, and the correction module is configured to:

[0042] obtain feedback data input by the user based on motion information of the robot;

[0043] correct the response strategy according to the feedback data.

[0044] As an optional implementation, the obtaining module is specifically configured to:

[0045] obtain posture data of the user through a visual sensor, the posture data comprising facial expressions, gestures and body postures of the user;

[0046] obtain voice data of the user through a voice sensor, the voice data comprising voice instructions and non-verbal sounds of the user;

[0047] obtain force control data of the user through a force sensor, the force control data comprising touch interaction signals and push-pull interaction signals of the user.

[0048] As an optional implementation, the processing module is specifically configured to:

[0049] extract posture features from the posture data by a visual encoder in a neural network model;

[0050] extract voice features from the voice data by a voice encoder in the neural network model;

[0051] extract force control features from the force control data by a force encoder in the neural network model;

[0052] obtain the multi-modal fusion features from the posture features, the voice features and the force control features by a decoder in the neural network model.

[0053] As an optional implementation, the processing module is specifically configured to:

[0054] determine a first attention weight of the posture features, a second attention weight of the voice features and a third attention weight of the force control features according to the posture features, the voice features and the force control features;

[0055] obtain the multi-modal fusion features according to the posture features, the voice features, the force control features, the first attention weight, the second attention weight and the third attention weight.

[0056] As an optional implementation, the control module is specifically configured to:

[0057] According to the multi-modal fusion feature, an imitation learning algorithm is used to generate an initial response strategy;

[0058] According to the multi-modal fusion feature and the context information, a reinforcement learning algorithm is used to determine a reward signal corresponding to the initial response strategy, and the initial response strategy is iteratively corrected according to the reward signal, and the initial response strategy at the end of iteration is taken as the response strategy.

[0059] As an optional implementation manner, the control module is specifically configured to:

[0060] According to the response strategy, a to-be-executed action of the robot is determined, and voice data corresponding to the to-be-executed action is generated;

[0061] According to the to-be-executed action and the voice data, a control instruction is generated, and the robot is controlled to execute the to-be-executed action while outputting the voice data according to the control instruction.

[0062] As an optional implementation manner, the control module is specifically configured to:

[0063] According to the to-be-executed action, a natural language response matched with the to-be-executed action is generated;

[0064] According to the natural language response, voice data corresponding to the to-be-executed action is generated through voice synthesis.

[0065] In a third aspect, an embodiment of the present application provides a computer device, including a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, the processor executes the machine readable instructions, and the steps of the robot control method in the first aspect are executed.

[0066] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is run by a processor, the steps of the robot control method in the first aspect are executed.

[0067] The present application has the following beneficial effects:

[0068] The application provides a robot control method, a computer device and a computer readable storage medium. Multimodal data in the process of user interaction with a robot is acquired, including posture data collected under visual sense, sound data collected under auditory sense and force control data collected under tactile sense in the process of user interaction with the robot. Feature extraction is performed on the multimodal data to obtain key features under multiple senses, and feature fusion is performed on the extracted key features under multiple senses to obtain fused multimodal fusion features. The multimodal fusion features integrate key features under visual, auditory and tactile senses, and can comprehensively perceive the interactive intention of the user. Environmental information of a region where the robot is located and historical interaction data of the user and the robot are acquired, various external factors related to the current human-computer interaction scene in the environmental information and each historical interaction behavior in the historical interaction data are combined, and context information of the current human-computer interaction is obtained through context understanding. Based on the multimodal fusion features and the context information, a response strategy is generated, and the robot is controlled according to the response strategy. By combining the multimodal fusion features under multiple senses and the context information of the current human-computer interaction, the interactive intention of the user and the current interactive environment state can be more comprehensively and accurately perceived, and the accuracy of the response strategy and the naturalness, accuracy and intelligent degree of human-computer interaction are improved. BRIEF DESCRIPTION OF DRAWINGS

[0069] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0070] Figure 1 A flowchart of a robot control method provided by the embodiments of the application is shown.

[0071] Figure 2 Another flowchart of a robot control method provided by the embodiments of the application is shown.

[0072] Figure 3 A flowchart of acquiring multimodal data of a user in a robot control method provided by the embodiments of the application is shown.

[0073] Figure 4 A flowchart of obtaining multimodal fusion features in a robot control method provided by the embodiments of the application is shown.

[0074] Figure 5 Another flowchart of obtaining multimodal fusion features in a robot control method provided by the embodiments of the application is shown.

[0075] Figure 6A flowchart of a response generation strategy of a robot control method provided by an embodiment of the present application is shown in FIG. 1.

[0076] Figure 7 A flowchart of a robot motion control method provided by an embodiment of the present application is shown in FIG. 2.

[0077] Figure 8 A flowchart of a voice data generation method provided by an embodiment of the present application is shown in FIG. 3.

[0078] Figure 9 A module structure diagram of a robot control device provided by an embodiment of the present application is shown in FIG. 4.

[0079] Figure 10 A structure diagram of a computer device provided by an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0080] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in detail with reference to the accompanying drawings in the embodiments of the present application. It should be understood that the accompanying drawings in the present application are only intended to illustrate and describe the present application, and are not intended to limit the scope of the present application. In addition, it should be understood that the accompanying drawings are not drawn to scale. The flowcharts show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or removed from the flowcharts by those skilled in the art under the guidance of the content of the present application.

[0081] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the present application.

[0082] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0083] In the field of robotics, the demand for human-robot interaction is increasing, and the requirements for human-computer interaction technology are becoming higher. The existing human-computer interaction has the limitations of single interaction mode, data dependency, and lack of autonomous decision-making ability, resulting in low efficiency and accuracy of human-computer interaction.

[0084] Based on the above problems, the embodiment of the present application proposes a robot control method, which integrates visual, auditory and tactile sensory inputs to comprehensively capture and extract multi-modal fusion features of user's body posture, voice and tactile force control. This multi-modal fusion feature enables a more comprehensive understanding of user's intentions and environmental status during human-computer interaction, thereby providing a more rich and natural interaction experience. Reinforcement learning algorithm is used to enable the robot to learn autonomously during interaction with the environment, reducing the dependence on a large amount of labeled data. Through iterative feedback correction, the response strategy can be automatically adjusted and optimized, thereby improving the generalization ability, adaptability and accuracy of robot control.

[0085] Figure 1 The flowchart of the robot control method provided by the embodiment of the present application, the execution subject of the method can be any computer device with computing processing capability. As shown in the figure, Figure 1 The method comprises:

[0086] S101, acquiring multi-modal data of a user, the multi-modal data comprising posture data, sound data and force control data.

[0087] Optionally, the multi-modal data in the process of user interaction with the robot is collected from multiple senses, including posture data, sound data and force control data in the process of user interaction with the robot. Among them, the posture data can be the interaction data collected under the visual sense, the sound data can be the interaction data collected under the auditory sense, and the force control data can be the interaction data collected under the tactile sense.

[0088] Specifically, the posture data can reflect the posture information of each part of the user's body, such as the posture information of the face and the hand. The sound data can reflect the user's language expression and emotional state, and the force control data can reflect the interaction information between the user and the object or the environment.

[0089] The posture data, sound data and force control data in the collected multi-modal data are respectively subjected to data cleaning and standardization and other preprocessing operations, wherein the data cleaning includes filtering and denoising of the multi-modal data. The cleaned multi-modal data is subjected to normalization processing to unify the formats of the posture data, the formats of the sound data and the formats of the force control data, respectively, to obtain standardized posture data, sound data and force control data, so as to improve the credibility and accuracy of the multi-modal data.

[0090] S102, feature extraction and fusion processing are performed on the multi-modal data to obtain multi-modal fusion features.

[0091] Optionally, visual key features, auditory key features and tactile key features are extracted from the preprocessed multi-modal data through feature extraction, and the visual key features, the auditory key features and the tactile key features are fused to obtain fused multi-modal fusion features.

[0092] Through feature extraction and feature fusion on the preprocessed multi-modal data, the information of various sensory inputs in the multi-modal data is effectively integrated, and more representative and informative multi-modal fusion features are obtained, so that the multi-modal fusion features integrate key features under the multi-sensory of vision, hearing and touch, so as to comprehensively perceive the interactive intention of the user, and provide data support for the naturalness and accuracy of subsequent interaction with the robot.

[0093] S103, according to the environmental information of the area where the robot is located and the historical interaction data of the robot, the context information of human-computer interaction is determined.

[0094] Optionally, the environmental information of the area where the robot is located represents various external factors related to the current human-computer interaction scene, to assist in identifying the interactive intention of the user. The environmental information can include physical environment information and social environment information. The physical environment information can be the time, place and state of surrounding objects and devices of the current interaction, etc. The social environment information can be the interaction occasion and the number of people present in the interaction occasion.

[0095] The historical interaction data of the user and the robot records each historical interaction behavior. By analyzing the historical interaction data, the interactive preference, habit and behavior pattern of the user can be determined. For example, the user has always focused on interacting with the robot by voice and inquiring about the dynamics of the technology field.

[0096] The environmental information of the area where the robot is located and the historical interaction data of the user and the robot are obtained, and combined with various external factors related to the current human-computer interaction scene and each historical interaction behavior, the context of the current human-computer interaction is understood, and the context information of the current human-computer interaction is obtained, to improve the accuracy of identifying the interactive intention of the user and enhance the human-computer interaction effect and user satisfaction.

[0097] S104, according to the multi-modal fusion features and the context information, a response strategy is generated, and the robot motion is controlled according to the response strategy.

[0098] Optionally, the multi-modal fusion feature includes comprehensive information of posture data, sound data and force control data after extraction and fusion, and integrates key features in the multi-sense of vision, hearing and touch. The current human-computer interaction context information combines environmental information and historical interaction data, and represents various external factors related to the current human-computer interaction scene and each historical interaction behavior.

[0099] The multi-modal fusion feature is input into a pre-trained decision model as the main data, and the current human-computer interaction context information is input into the pre-trained decision model as the auxiliary data. The decision model predicts and outputs a response strategy according to the input multi-modal fusion feature and context information. The motion trajectory of the robot is planned according to the response strategy, and each joint motion of the robot is controlled based on the motion trajectory. Based on the multi-modal fusion feature and the context information, a more reasonable response strategy is generated and the robot motion is accurately controlled, and a more intelligent and natural human-computer interaction is realized.

[0100] In this embodiment, multi-modal data in the process of user interaction with the robot is acquired, including posture data collected in the visual sense, sound data collected in the auditory sense, and force control data collected in the tactile sense in the process of user interaction with the robot. Feature extraction is performed on the multi-modal data to obtain key features in the multi-sense, and feature fusion is performed on the extracted key features in the multi-sense to obtain multi-modal fusion features after fusion. The multi-modal fusion feature integrates key features in the multi-sense of vision, hearing and touch, and can comprehensively perceive the user's interaction intention. The environmental information of the area where the robot is located and the historical interaction data of the user and the robot are acquired, and the context information of the current human-computer interaction is obtained by combining various external factors related to the current human-computer interaction scene in the environmental information and each historical interaction behavior in the historical interaction data. Based on the multi-modal fusion feature and the context information, a response strategy is generated, and the robot motion is controlled according to the response strategy. By combining the multi-modal fusion feature in the multi-sense and the context information of the current human-computer interaction, the user's interaction intention and the current interaction environment state can be more comprehensively and accurately perceived, and the accuracy of the response strategy and the naturalness, accuracy and intelligence of the human-computer interaction are improved.

[0101] Figure 2 Another flowchart of the robot control method provided by the embodiments of the present application is shown in FIG. 10. After the step S104 of controlling the robot motion according to the response strategy, the step further includes: Figure 2

[0102] S201, acquire feedback data input by the user based on the motion information of the robot.

[0103] ​Optionally, after the current human-robot interaction ends, feedback data input by the user based on the motion information of the robot can be obtained through voice interaction, interface interaction, visual monitoring or wearable devices. Specifically, the voice feedback of the user is captured in real time by the microphone built in the robot as feedback information. Alternatively, the feedback information is input by inputting text on the evaluation interface or selecting evaluation options (such as star rating or satisfaction score, etc.). Alternatively, visual monitoring is performed by using the camera of the robot itself or the camera of the surrounding interactive environment, and the positive feedback information or negative feedback information of the user is obtained by analyzing the expression and body language of the user. Alternatively, the user can also wear a specific wearable device, such as a smart bracelet, to monitor the heart rate and other physiological feedback information of the user.

[0104] Exemplarily, the feedback data can be evaluation information of the user on the motion performance state of the robot in the form of language, text, etc. Alternatively, the feedback information is obtained by observing the behavior and action of the user. For example, during the execution of the action of the robot, the user makes actions such as avoiding, frowning, shaking head, etc., which may indicate that the user is not satisfied or feels uncomfortable with the motion performance state of the robot. On the contrary, the user makes actions such as nodding, smiling, actively approaching, etc., which may indicate that the user approves the motion performance state of the robot.

[0105] S202, correcting the response strategy according to the feedback data.

[0106] Optionally, according to the feedback data input by the user, the link to be corrected is determined. For example, if the user feedbacks that the motion of the robot is not accurate when picking up the object, the action link of picking up the object in the response strategy is taken as the link to be corrected.

[0107] According to the link to be corrected and the feedback data input by the user, the feedback data is integrated into the decision model, the model parameters in the decision model are updated by incremental learning in stages using an online learning algorithm, and the response strategy is corrected and output by the decision model according to the input multi-modal fusion features, context information and feedback data. By dynamically adjusting the response strategy, the motion of the robot is dynamically adjusted.

[0108] In this embodiment, after the current human-robot interaction ends, feedback data input by the user based on the motion information of the robot is obtained through voice interaction, interface interaction, visual monitoring or wearable devices. According to the feedback data input by the user, the link to be corrected is determined, and based on the link to be corrected and the feedback data input by the user, the model parameters in the decision model are updated by incremental learning in stages using an online learning algorithm, and the response strategy is corrected. The response strategy is dynamically adjusted based on the feedback data in the interactive process, and the motion of the robot is dynamically adjusted, so as to realize a more smooth, natural and accurate interactive experience.

[0109] The process of acquiring multi-modal data of the user is described in detail as follows.

[0110] Figure 3 The flowchart of the process of acquiring multi-modal data of the user in the robot control method provided by the embodiments of the present application is shown in Figure 3 The step of acquiring multi-modal data of the user in step S101 includes the following steps.

[0111] S301, acquiring pose data of the user through a visual sensor, the pose data including facial expression, gesture and body pose of the user.

[0112] Optionally, at least one visual sensor is deployed in the human-machine interaction environment, and the pose data of the user is acquired through the visual sensor. Specifically, the visual sensor can be a camera, and the facial expression, gesture and body pose of the user when interacting with the robot are captured by real-time acquisition of video stream or image of the user when interacting with the robot, so as to obtain the pose data of the user, thereby providing a visual sensory data source for human-machine interaction.

[0113] For example, the visual sensor can be deployed in front of or beside the user, so as to clearly and comprehensively capture the pose data of the user. For example, the visual sensor captures the video stream or image of the user pointing to the direction of the water cup on the table as the pose data of the user.

[0114] S302, acquiring sound data of the user through a sound sensor, the sound data including voice instruction and non-verbal sound of the user.

[0115] Optionally, at least one sound sensor is also deployed in the human-machine interaction environment, and the sound data of the user is acquired through the sound sensor. Specifically, the sound sensor can be a microphone, and the voice instruction and non-verbal sound of the user when interacting with the robot are captured by real-time acquisition of sound of the user when interacting with the robot, so as to obtain the sound data of the user, thereby providing an auditory sensory data source for human-machine interaction. The non-verbal sound can be sound accompanied by changes in the body pose of the user, such as applause, stomping, etc.

[0116] For example, the sound sensor can be deployed close to the user, so as to clearly and effectively capture the sound data of the user. For example, the sound sensor captures the voice instruction "help me get a water cup from the table" and the rubbing sound of the clothes when the user points to the water cup on the table as the sound data of the user.

[0117] S303, acquiring force control data of the user through a force sensor, the force control data including touch interaction signal and push-pull interaction signal of the user.

[0118] Optionally, at least two force sensors are also deployed in the human-computer interaction environment, and the force control data of the user is acquired through the force sensors. Specifically, the force sensors can include piezoresistive force sensors and strain gauge force sensors, the touch interaction signals when the user interacts with the robot are collected through the piezoresistive force sensors, the push-pull interaction signals when the user interacts with the robot are collected through the strain gauge force sensors, and the force control data of the user is obtained to provide a data source for the tactile sense of the human-computer interaction.

[0119] Exemplarily, the force sensors can be deployed on the objects touched by the user to accurately capture the force control data of the user. For example, when the user holds a water cup placed on a table, the force control sensors installed on the water cup detect the force of the user's actions such as touching, pressing, pushing, pulling or holding the water cup, and obtain the touch interaction signals and the push-pull interaction signals of the user as the voice data of the user.

[0120] In this embodiment, the visual sensor, the sound sensor and the force sensor are deployed in the human-computer interaction environment, the facial expressions, gestures and body postures when the user interacts with the robot are captured through the visual sensor to obtain the posture data of the user, and a data source for the visual sense of the human-computer interaction is provided. The voice instructions and non-verbal sounds when the user interacts with the robot are captured through the sound sensor to obtain the voice data of the user, and a data source for the auditory sense of the human-computer interaction is provided. The touch interaction signals and the push-pull interaction signals when the user interacts with the robot are captured through the force sensor to obtain the force control data of the user, and a data source for the tactile sense of the human-computer interaction is provided. By acquiring the data sources of the visual, auditory and tactile senses in the human-computer interaction process, the interactive intention of the user can be comprehensively and accurately perceived.

[0121] Hereinafter, the process of feature extraction and fusion processing of multi-modal data to obtain multi-modal fusion features is described in detail.

[0122] Figure 4 The flowchart of the robot control method provided by the embodiment of the present application to obtain multi-modal fusion features is shown in Figure 4 As shown in the above step S102, the step of feature extraction and fusion processing of multi-modal data to obtain multi-modal fusion features includes:

[0123] S401, extracting posture features according to the posture data by a visual encoder in the neural network model.

[0124] Optionally, a visual encoder is designed for the pose data of the visual sensor in advance by a deep neural network model, and the pose features of the visual modality are extracted by the visual encoder. The visual encoder can be a convolutional neural network (CNN), which automatically learns the local features of each image or video frame in the pose data and gradually extracts the pose features through a hierarchical structure.

[0125] Specifically, the visual encoder performs facial expression recognition on the pose data, extracts facial expression features of the user by analyzing the movement of the facial muscles of the user during human-computer interaction. The visual encoder performs gesture recognition on the pose data, extracts the movement trajectory of the user's gestures during human-computer interaction, including gesture shape, speed, acceleration, and gesture duration, and other gesture features. The visual encoder performs body pose estimation on the pose data, extracts body pose features of the user by identifying the positions and angles of each body joint of the user during human-computer interaction, such as standing, sitting, walking, etc. The facial expression features, gesture features, and body pose features are used as the pose features in the visual modality.

[0126] S402, extracting sound features from the sound data by a sound encoder in the neural network model.

[0127] Optionally, a sound encoder is designed for the sound data of the sound sensor in advance by a deep neural network model, and the sound features of the auditory modality are extracted by the sound encoder. The sound encoder can be a recurrent neural network (RNN), which processes the time series information in the audio signal in the sound data and extracts the sound features.

[0128] Specifically, the sound encoder performs speech recognition on the sound data, and extracts speech instruction features by extracting features such as pitch, frequency, and formant of the speech. The sound encoder performs non-verbal sound recognition on the sound data, and extracts non-verbal sound features such as hand-clapping sound and knocking sound that accompany the body posture of the user during human-computer interaction. These non-verbal sounds can be associated with the body posture of the user to provide additional interaction information. The speech instruction features and non-verbal sound features are used as the sound features in the auditory modality.

[0129] S403, extracting force control features from the force control data by a force encoder in the neural network model.

[0130] Optionally, a force encoder is designed in advance for the force control data of the force sensor through the deep neural network model, and the force control features of the haptic modality are extracted through the force encoder. The force encoder can also be an RNN, the input layer of the RNN receives the sequence of the force control data, and after processing through one or more RNN layers, the force control feature representation is obtained through the fully connected layer, so as to better capture the trend of force change.

[0131] Specifically, the force encoder performs touch recognition on the force control data, extracts touch features such as the force of touch, the duration of touch, and the position of the touch point. The force encoder performs push-pull action recognition on the force control data, extracts push-pull action features by analyzing the force change, speed, and direction of push-pull. The force encoder performs interactive mode recognition on the force control data, identifies the interactive intention of the user such as selection, sliding, and dragging, etc. by combining touch and push-pull action, and obtains interactive action features. The touch features, push-pull action features, and interactive action features are taken as the force control features under the haptic modality.

[0132] S404, obtaining the multi-modal fusion features according to the posture features, the sound features, and the force control features by a decoder in the neural network model.

[0133] Optionally, a decoder based on an attention mechanism is designed in advance through the deep neural network model, and the posture features, the sound features, and the force control features are fused by the decoder through the attention mechanism to form unified multi-modal fusion features.

[0134] The decoder based on the attention mechanism can dynamically allocate attention weights of different modal features, so as to automatically focus on more important modal features according to the interactive task requirements, and improve the feature fusion effect.

[0135] In this embodiment, a visual encoder, a sound encoder, and a force encoder are designed for the posture data of the visual modality, the sound data of the auditory modality, and the force control data of the haptic modality respectively through the deep neural network model, and each encoder extracts the features of the corresponding modality. The posture features under the visual modality are extracted from the posture data by the visual encoder in the neural network model, the sound features under the auditory modality are extracted from the sound data by the sound encoder, and the force control features under the haptic modality are extracted from the force control data by the force encoder. A decoder based on an attention mechanism is designed through the deep neural network model, and the posture features under the visual modality, the sound features under the auditory modality, and the force control features under the haptic modality are fused by the decoder through the attention mechanism to form unified multi-modal fusion features. By dynamically allocating the attention weights of the modal features, the more important modal features are automatically focused on, and the feature fusion effect is improved.

[0136] In the following, the process of obtaining the multi-modal fusion features according to the posture features, the sound features, and the force control features is described in detail.

[0137] Figure 5 Another flow diagram of obtaining a multi-modal fusion feature in the robot control method provided by the embodiments of the present application is shown in FIG. 5. Figure 5 As shown in FIG. 5, the step S404 of obtaining a multi-modal fusion feature according to the posture feature, the sound feature and the force control feature includes:

[0138] S501, determining a first attention weight of the posture feature, a second attention weight of the sound feature and a third attention weight of the force control feature according to the posture feature, the sound feature and the force control feature.

[0139] Optionally, the decoder based on the attention mechanism utilizes a multi-modal attention network to input the posture feature in the visual modality, the sound feature in the auditory modality and the force control feature in the tactile modality as vectors, perform linear transformation on the posture feature, the sound feature and the force control feature, and map the three features to the same dimension.

[0140] In the same dimension, the first attention score of the posture feature, the second attention score of the sound feature and the third attention score of the force control feature are calculated, each attention score is converted into an attention weight by a Softmax function, and the first attention weight of the posture feature, the second attention weight of the sound feature and the third attention weight of the force control feature are obtained.

[0141] S502, obtaining a multi-modal fusion feature according to the posture feature, the sound feature, the force control feature, the first attention weight, the second attention weight and the third attention weight.

[0142] Optionally, the features of each modality are weighted and summed according to the first attention weight of the posture feature, the second attention weight of the sound feature and the third attention weight of the force control feature, and a multi-modal fusion feature is obtained.

[0143] Specifically, a first product of the posture feature and the first attention weight, a second product of the sound feature and the second attention weight, and a third product of the force control feature and the third attention weight are calculated, and the sum of the first product, the second product and the third product is taken as the multi-modal fusion feature. Through this way of dynamically allocating weights, the fused multi-modal feature can more accurately reflect the interactive intention of the user, so that the robot can more effectively understand the user's demand and make a more appropriate interactive response.

[0144] In this embodiment, the attention mechanism-based decoder utilizes a multi-modal attention network to obtain first attention weights of the pose features, second attention weights of the sound features, and third attention weights of the force control features according to the pose features in the visual mode, the sound features in the auditory mode, and the force control features in the tactile mode. The multi-modal fusion features are obtained by weighted summation of the features in each mode according to the first attention weights of the pose features, the second attention weights of the sound features, and the third attention weights of the force control features. The attention weights of the features in each mode are dynamically allocated based on the current interaction requirement, so that the multi-modal fusion features after fusion more accurately reflect the user's interaction intention, and effective fusion of the multi-modal features is realized.

[0145] Next, the process of generating a response strategy according to the multi-modal fusion features and the context information is described in detail.

[0146] Figure 6 The flowchart of the process of generating a response strategy of the robot control method provided in this embodiment is shown in FIG. 6. Figure 6 As shown in FIG. 6, the step of generating a response strategy according to the multi-modal fusion features and the context information in step S104 includes:

[0147] S601, generating an initial response strategy using an imitation learning algorithm according to the multi-modal fusion features.

[0148] Optionally, the multi-modal fusion features are input into a pre-trained decision model, which can be a fully connected neural network response strategy model. The multi-modal fusion features are mapped to a response strategy using an imitation learning algorithm, and an initial response strategy is predicted and output.

[0149] Specifically, the decision model uses an imitation learning algorithm to make autonomous decisions according to the multi-modal fusion features, and predicts and outputs a natural initial response strategy by imitating human behavior.

[0150] S602, determining a reward signal corresponding to the initial response strategy using a reinforcement learning algorithm according to the multi-modal fusion features and the context information, and iteratively correcting the initial response strategy according to the reward signal, and taking the initial response strategy at the end of iteration as the response strategy.

[0151] Optionally, the multi-modal fusion features and the context information of the current human-computer interaction are input into the decision model, the expected effect of the initial response strategy is evaluated using a reinforcement learning algorithm, and the initial response strategy is iteratively corrected according to the evaluation result.

[0152] Specifically, the agent is pre-designed in the decision model, the agent can receive the reward signals returned based on the multi-modal fusion features, the context information of the current human-robot interaction and the initial response strategy at each time step, and iteratively correct the initial response strategy based on the reward signals. The iteration is stopped when the optimal initial response strategy is learned, and the initial response strategy at the end of the iteration is taken as the response strategy.

[0153] In this embodiment, the decision model adopts an imitation learning algorithm, autonomously decides according to the multi-modal fusion features, and outputs a natural initial response strategy by imitating human behavior prediction. The multi-modal fusion features and the context information of the current human-robot interaction are input into the decision model, the reinforcement learning algorithm is used to determine the reward signal corresponding to the initial response strategy, and the initial response strategy is iteratively corrected according to the reward signal, and the initial response strategy at the end of the iteration is taken as the response strategy. By using the multi-modal fusion features and the context information, the optimal response strategy is learned through continuous trial and error, and the autonomous decision performance in human-robot interaction is improved.

[0154] Next, the process of controlling the robot motion according to the response strategy is described in detail.

[0155] Figure 7 The flowchart of the robot control method provided by the embodiments of the present application for controlling the robot motion is shown in Figure 7 As shown in the above step S104, the step of controlling the robot motion according to the response strategy includes:

[0156] S701, according to the response strategy, determining the action to be executed by the robot, and generating voice data corresponding to the action to be executed.

[0157] Optionally, a pre-defined action space list is obtained, which contains various actions that the robot can perform. The motion trajectory and action of the robot are planned according to the response strategy, the action index with the maximum matching probability of the response strategy is found, the corresponding action is determined from the action space list based on the action index, as the action to be executed by the robot, such as reaching out, picking up and putting down objects, etc., and the action to be executed is converted into voice data, generating voice data corresponding to the action to be executed.

[0158] S702, generating a control instruction according to the action to be executed and the voice data, and controlling the robot to execute the action to be executed while outputting the voice data according to the control instruction.

[0159] Optionally, a control instruction is generated according to the action to be executed by the robot and the corresponding voice data, and the control instruction is sent to the robot controller, so that the robot controller controls the robot to output the voice data while executing the action to be executed based on the control instruction.

[0160] Specifically, the robot controller ensures the synchronization of the robot action execution and the voice output based on the control instruction, for example, when the robot starts to execute the action of picking up the cup, the voice prompt of "picking up the cup" is played at the same time.

[0161] In this embodiment, according to the response strategy, the action to be executed by the robot is determined, and the action to be executed is converted into voice data to generate voice data corresponding to the action to be executed. The control instruction is generated according to the action to be executed by the robot and the corresponding voice data, and the robot controller controls the robot to output the voice data at the same time of executing the action to be executed based on the control instruction. A more natural, accurate and intuitive human-computer interaction is realized.

[0162] Next, the process of generating voice data corresponding to the action to be executed is described in detail.

[0163] Figure 8 The flowchart of generating voice data of the robot control method provided in the embodiments of the present application is shown as Figure 8 As shown in the above step S701, the step of generating voice data corresponding to the action to be executed includes:

[0164] S801, according to the action to be executed, a natural language response matching the action to be executed is generated.

[0165] Optionally, the action to be executed is identified, and the action intention is parsed. According to the action intention, a target natural language template matching the action intention is selected from the predefined natural language template. The specific action information such as picking up the cup is filled into the target natural language template to generate an initial natural language response, and the initial natural language response is checked for grammar, semantics and language style to obtain the natural language response.

[0166] The grammar check can ensure that the generated natural language response conforms to the grammar rules and avoid syntax errors. The semantic check can ensure that the semantics are clear and easy to understand, and avoid using too complex or professional terms. The language style check according to the current user's identity and interaction scene can keep the consistency of the language style. For example, if the user is a child, in the child interaction scene, a simple and lively language style can be used.

[0167] S802, according to the natural language response, the voice data corresponding to the action to be executed is generated by voice synthesis.

[0168] Optionally, based on the preset text-to-speech library, the generated natural language response is converted into voice data by voice synthesis technology, and the voice data corresponds to the action to be executed by the robot, so that the robot outputs the voice data at the same time of executing the action to be executed.

[0169] In this embodiment, the action to be performed by the robot is identified, and an action intention is obtained by analysis. According to the action intention, a natural language response matching the action to be performed is generated. The generated natural language response is converted into voice data corresponding to the action to be performed by the robot through voice synthesis technology, so that the robot outputs the voice data while performing the action to be performed. A more rich human-computer interaction experience is provided.

[0170] Based on the same inventive concept, the embodiments of the present application also provide a robot control device corresponding to the robot control method. Since the principle of the device in the embodiments of the present application solves the problem similar to the robot control method in the embodiments of the present application, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described here.

[0171] Figure 9 The module structure diagram of the robot control device provided in the embodiments of the present application is shown in Figure 9 The device comprises:

[0172] The acquisition module 901 is configured to acquire multi-modal data of the user, and the multi-modal data comprises posture data, sound data and force control data.

[0173] The processing module 902 is configured to perform feature extraction and fusion processing on the multi-modal data to obtain multi-modal fusion features.

[0174] The determination module 903 is configured to determine context information of human-computer interaction according to environmental information of an area where the robot is located and historical interaction data of the robot.

[0175] The control module 904 is configured to generate a response strategy according to the multi-modal fusion features and the context information, and control the motion of the robot according to the response strategy.

[0176] As an optional implementation, the device further comprises a correction module 905, and the correction module 905 is configured to:

[0177] Obtain feedback data input by the user based on the motion information of the robot.

[0178] Correct the response strategy according to the feedback data.

[0179] As an optional implementation, the acquisition module 901 is specifically configured to:

[0180] Obtain the posture data of the user through a visual sensor, and the posture data comprises facial expressions, gestures and body postures of the user.

[0181] Obtain the sound data of the user through a sound sensor, and the sound data comprises voice instructions and non-verbal sounds of the user.

[0182] The force control data of the user is acquired by the force sensor, and the force control data includes touch interaction signals and push-pull interaction signals of the user.

[0183] As an optional implementation, the processing module 902 is specifically configured to:

[0184] The posture feature is extracted from the posture data by a visual encoder in the neural network model.

[0185] The sound feature is extracted from the sound data by a sound encoder in the neural network model.

[0186] The force control feature is extracted from the force control data by a force encoder in the neural network model.

[0187] The multi-modal fusion feature is obtained from the posture feature, the sound feature, and the force control feature by a decoder in the neural network model.

[0188] As an optional implementation, the processing module 902 is specifically configured to:

[0189] The first attention weight of the posture feature, the second attention weight of the sound feature, and the third attention weight of the force control feature are determined according to the posture feature, the sound feature, and the force control feature.

[0190] The multi-modal fusion feature is obtained according to the posture feature, the sound feature, the force control feature, the first attention weight, the second attention weight, and the third attention weight.

[0191] As an optional implementation, the control module 904 is specifically configured to:

[0192] The initial response strategy is generated by using an imitation learning algorithm according to the multi-modal fusion feature.

[0193] The reward signal corresponding to the initial response strategy is determined by using a reinforcement learning algorithm according to the multi-modal fusion feature and the context information, the initial response strategy is iteratively corrected according to the reward signal, and the initial response strategy at the end of the iteration is taken as the response strategy.

[0194] As an optional implementation, the control module 904 is specifically configured to:

[0195] The action to be performed by the robot is determined according to the response strategy, and the voice data corresponding to the action to be performed is generated.

[0196] The control instruction is generated according to the action to be performed and the voice data, and the robot is controlled to perform the action to be performed while outputting the voice data according to the control instruction.

[0197] As an optional implementation, the control module 904 is specifically configured to:

[0198] Generate a natural language response that matches the action to be performed.

[0199] Based on the natural language response, speech data corresponding to the action to be performed is generated through speech synthesis.

[0200] This application also provides a computer device, such as... Figure 10 The diagram shown is a schematic representation of the structure of a computer device provided in an embodiment of this application, including: a processor 101, a memory 102, and a bus 103. The memory 102 stores machine-readable instructions executable by the processor 101 (e.g., ...). Figure 9 The device includes the acquisition module 901, processing module 902, determination module 903, control module 904, and correction module 905 (and the corresponding execution instructions). When the computer device is running, the processor 101 and the memory 102 communicate through the bus 103. When the machine-readable instructions are executed by the processor 101, the steps of the robot control method in the above embodiment are executed.

[0201] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the robot control method described above.

[0202] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0203] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. When the functions are realized in the form of software function units and sold or used as an independent product, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0204] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. A robot control method characterized by, The method comprises the following steps: acquiring multi-modal data of a user, the multi-modal data comprising posture data, sound data and force control data; performing feature extraction and fusion processing on the multi-modal data to obtain multi-modal fusion features; determining context information of human-machine interaction according to environmental information of a region where a robot is located and historical interaction data of the robot; generating a response strategy according to the multi-modal fusion features and the context information, and controlling movement of the robot according to the response strategy; the feature extraction and fusion processing on the multi-modal data to obtain multi-modal fusion features comprises the following steps: extracting posture features from the posture data by a visual encoder in a neural network model; extracting sound features from the sound data by a sound encoder in the neural network model; extracting force control features from the force control data by a force encoder in the neural network model; obtaining the multi-modal fusion features from the posture features, the sound features and the force control features by a decoder in the neural network model; the obtaining of the multi-modal fusion features from the posture features, the sound features and the force control features comprises the following steps: determining first attention weights of the posture features, second attention weights of the sound features and third attention weights of the force control features according to the posture features, the sound features and the force control features; obtaining the multi-modal fusion features according to the posture features, the sound features, the force control features, the first attention weights, the second attention weights and the third attention weights.

2. The method of claim 1, wherein, after the controlling of the movement of the robot according to the response strategy, the method further comprises the following steps: acquiring feedback data input by a user based on movement information of the robot; correcting the response strategy according to the feedback data.

3. The method of claim 1, wherein, the acquiring of the multi-modal data of the user comprises the following steps: acquiring posture data of the user by a visual sensor, the posture data comprising facial expressions, gestures and body postures of the user; acquiring sound data of the user by a sound sensor, the sound data comprising voice instructions and non-verbal sounds of the user; acquiring force control data of the user by a force sensor, the force control data comprising touch interaction signals and push-pull interaction signals of the user.

4. The method of claim 1, wherein, the generating of the response strategy according to the multi-modal fusion features and the context information comprises the following steps: generating an initial response strategy by using an imitation learning algorithm according to the multi-modal fusion features; determining a reward signal corresponding to the initial response strategy by using a reinforcement learning algorithm according to the multi-modal fusion features and the context information, and iteratively correcting the initial response strategy according to the reward signal, and taking the initial response strategy at the end of the iteration as the response strategy.

5. The method of claim 1, wherein, the controlling of the movement of the robot according to the response strategy comprises the following steps: determining a to-be-executed action of the robot according to the response strategy, and generating voice data corresponding to the to-be-executed action; According to the action to be performed and the voice data, a control instruction is generated, and the robot is controlled to perform the action to be performed while outputting the voice data according to the control instruction.

6. The method of claim 5, wherein, The voice data corresponding to the action to be performed is generated, including: According to the action to be performed, a natural language response matching the action to be performed is generated; According to the natural language response, voice data corresponding to the action to be performed is generated through voice synthesis.

7. A computer device, comprising: Including: A processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device runs, the processor and the memory communicate through the bus, the processor executes the machine readable instructions, and the steps of the robot control method in any one of claims 1 to 6 are executed.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to execute the steps of the robot control method in any one of claims 1 to 6.

Images