Humanoid robot multi-modal interaction method and device based on large language model

Abstract

The invention relates to a humanoid robot multi-mode interaction method and device based on a large language model, and the method comprises the steps: obtaining the voice of a user, and converting the voice into an original input text in response to a received user output voice command which is matched with a preset wake-up instruction of a humanoid robot; based on the original input text, generating a text matched with the current situation by utilizing a large language model, and for an output text corresponding to the text matched with the current situation, converting the output text into voice and outputting the voice; the method comprises the following steps: analyzing interaction contents of a large language model, and selecting a continuous human shape action sequence with the highest matching degree from a plurality of preset candidate human shape action sequences; on the premise of considering the size difference between the virtual human body and the actual robot and the joint angle error, the generated humanoid action sequence is mapped to the humanoid robot, and multi-modal interaction is achieved. The method has the advantages of high-precision low-delay real-time voice recognition, high voice action matching degree and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to a multimodal interaction method and device of a humanoid robot based on a large language model. Background Art

[0002] In recent years, the development of deep learning technology has promoted the progress of robotics, especially in the field of humanoid robots. The application of deep learning technology has shown great potential for human-computer interaction. These technologies have achieved basic intelligent interaction. However, when dealing with complex sentences or professional terms, voice interaction still has problems such as low conversion accuracy, insufficient context understanding, mechanical voice expression and lack of emotion, no support for multiple languages ​​and inability to generate long texts. It shows obvious limitations when facing diverse and complex user needs.

[0003] In addition, there is often a disconnect between semantics and actions in existing humanoid robot interactions. Most robots rely on a preset action library. When interacting, the robot cannot make accurate matching actions according to semantic changes, which makes it look stiff and unnatural. Moreover, the preset actions usually cover limited scenarios and cannot meet the real-time and changing interaction needs. Especially when faced with a large amount of non-standardized interaction content, the robot often cannot accurately perform the corresponding actions. This further limits the application of humanoid robots in interaction.

[0004] In order to meet the above challenges, a multimodal interaction method is urgently needed that can achieve precise matching of voice and action. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a humanoid robot multimodal interaction method and device based on a large language model, so as to solve or partially solve the problem that the robot's multimodal expression ability is insufficient in complex interactive tasks, resulting in unsatisfactory matching between speech and action.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] One aspect of the present invention provides a humanoid robot multimodal interaction method based on a large language model, comprising the following steps:

[0008] Step S1, in response to receiving a user output voice command and matching the wake-up instruction preset by the humanoid robot, obtaining the user's voice and converting it into original input text;

[0009] Step S2, based on the original input text, using a large language model to generate text matching the current context, and for output text corresponding to the text matching the current context, converting the output text into speech and then outputting it;

[0010] Step S3, by parsing the interaction content of the large language model, selecting a continuous humanoid action sequence with the highest matching degree from a plurality of preset candidate humanoid action sequences;

[0011] Step S4, taking into account the size difference between the virtual human body and the actual robot, as well as the joint angle error, the generated humanoid action sequence is mapped to the humanoid robot to achieve multimodal interaction.

[0012] As a preferred technical solution, the step S2 includes the following sub-steps:

[0013] Step S201, based on the original input text, the preset interaction scenario and the context information, using a large language model to generate text matching the current context;

[0014] Step S202: Based on the text matching the current context, a large language model is used to generate corresponding output text, and the output text is converted into speech for output.

[0015] As a preferred technical solution, the step S4 comprises the following steps:

[0016] Step S401, obtaining a plurality of key marking points selected from the digital human body model, and normalizing the key marking points to the coordinate system of the humanoid robot using a robot kinematic chain;

[0017] Step S402, for the normalized key markers, use a pre-trained multi-layer perceptron network to obtain the output target joint angle;

[0018] Step S403, taking the target joint angle as the optimization object, minimizing the position loss and the rotation loss as the goal, and optimizing the target joint angles of multiple parts by using a genetic algorithm to obtain the final target joint angle;

[0019] Step S404, controlling each part in a multi-threaded manner, and controlling the actuator of each part to approach the final target joint angle through differential interpolation.

[0020] As a preferred technical solution, during the training process of the multi-layer perceptron network, minimizing Cartesian space loss is taken as the optimization goal.

[0021] As a preferred technical solution, in step S403, the initial joint angles of multiple parts are obtained by inverse kinematics solution, and the target joint angles are optimized using a genetic algorithm until a preset error threshold is reached.

[0022] As a preferred technical solution, the position loss is the smoothed L1 distance between the body position and the predicted position, and the rotation loss is the distance between the quaternion calculated using the SMQL method and the predicted quaternion.

[0023] As a preferred technical solution, the parts include the trunk, elbows and wrist joints.

[0024] As a preferred technical solution, in step S1, the process of acquiring the user's voice and converting it into original input text includes:

[0025] Obtain the interactive language and scenario selected by the user, and use the preset model to convert the voice into original input text.

[0026] Another aspect of the present invention provides an electronic device, comprising: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned humanoid robot multimodal interaction method based on a large language model.

[0027] Another aspect of the present invention provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned humanoid robot multimodal interaction method based on a large language model.

[0028] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0029] (1) Real-time speech recognition with high accuracy and low latency: The present invention uses speech recognition to efficiently recognize and process the user's voice commands, achieving high-accuracy and low-latency speech recognition functions. By detecting whether the user's voice command is a wake-up command and dynamically managing the speech recognition model, the intelligence level and user experience are improved.

[0030] (2) Achieve natural and smooth human-computer interaction: It is suitable for multi-language and multi-scenario interaction, improves the adaptability and usability of the system, and uses a large language model to ensure the logic of the conversation content while improving the fluency of the conversation. Through speech synthesis, efficient conversion from text to speech is achieved, allowing the robot to interact with the user naturally, and output speech synchronously when performing actions, enhancing the authenticity of the interaction and the user's immersion.

[0031] (3) High degree of matching between speech and action: The present invention uses a large language model to parse semantics, accurately selects the robot target action sequence that matches the conversation content, and achieves efficient action mapping. It reduces the complexity of the action while maintaining the integrity of the action, ensuring that the robot's action is smooth and stable.

[0032] (4) Strong real-time performance: The present invention utilizes motion mapping to output key point target poses, and uses a pre-trained multi-layer perceptron (MLP) network to quickly solve the transformation of the robot's base coordinate system pose to the joint coordinate system, thereby reducing the complex kinematic modeling process in the traditional inverse kinematics algorithm and improving real-time performance and computational efficiency. It is suitable for real-time control systems, reduces computational delays, and improves the fluency and accuracy of robot operations.

[0033] (5) High control smoothness: Multi-threading is used to control multiple joint motors to perform the same action simultaneously, which improves the coordination of a single action. Interpolation algorithm is used between adjacent target postures to ensure the smoothness of motor movement and improve the stability of the movement of each joint motor. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Flow chart of a multimodal interaction method of a humanoid robot based on a large language model in an embodiment. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] Example 1

[0037] In response to the problems existing in the aforementioned prior art, this embodiment provides a humanoid robot multimodal interaction method based on a large language model, which not only has efficient speech recognition and natural language processing capabilities in multiple languages ​​and scenarios, but also realizes actions matching semantics to enhance the robot's multimodal expression in complex interactive tasks.

[0038] See also Figure 1 , the method comprises the following steps:

[0039] Step S1, speech recognition. It includes:

[0040] It uses speech recognition algorithms to identify user voice commands, processes voice input in real time, and provides high-precision and low-latency speech recognition services.

[0041] When the user inputs a voice command, the intelligent voice recognition module will detect whether the voice is the wake-up command of the robot; if the voice is detected as the wake-up command of the robot (such as: Orient), the voice recognition function will be turned on. Preferably, the user can set the wake-up command of the robot.

[0042] Step S2: interaction that complies with human conversation logic. This includes:

[0043] Based on the user's pre-selected interaction language and scenario, speech is converted into text through intelligent models (Whisper, DeepSpeech, etc.).

[0044] The first converted text may contain homophones, traditional Chinese characters, etc., and needs to be input into the large language model for further processing. The large language model will analyze the preset dialogue scene and context information, generate text that matches the current situation, and generate appropriate response content based on the text.

[0045] The text is converted into speech through intelligent speech generation models. Based on these models, users can choose from a variety of real voices as the robot's voice, and can also train their own voice as the robot's voice, which is output synchronously when the robot performs actions.

[0046] Step S3, matching action mapping of the conversation content. This includes:

[0047] According to the semantic analysis of the interaction content by the large language model, the continuous actions that match the semantics, i.e., the action sequence, are selected from the pre-trained "action mapping model". The purpose of action mapping is to map the generated humanoid actions to the humanoid robot. The following methods are used to solve the problem of size difference between the virtual human model and the actual robot:

[0048] Step S401, select ten key marking points from the digital human body model, and use the kinematic chain of the robot's upper limbs to normalize the positions of these marking points in the coordinate system of the humanoid robot.

[0049] Step S402, the normalized marker positions are processed by a pre-trained multi-layer perceptron (MLP) network to generate joint angle outputs for driving the robot. The MLP network is specially trained to solve the inverse kinematics problem and learn the mapping relationship from the marker positions to the joint angles by minimizing the loss function in Cartesian space.

[0050] In the process of training the MLP network, the Cartesian space loss function is used as the main optimization objective. This strategy effectively reduces the error accumulation caused by the redundant degrees of freedom of the robot.

[0051] Step S403, taking into account the joint angle error, in order to further optimize the calculation process of the joint angle, a local search method based on a genetic algorithm is implemented. First, the initial joint angle is calculated by the inverse kinematics (IK) solver, and then the three-step genetic algorithm is applied to optimize the trunk, elbow and wrist joints respectively until the preset error threshold is reached. Among them, the initial joint angle calculated by the inverse kinematics solver is used as the starting point of the genetic algorithm optimization to alleviate the morphological differences between the human body model and the robot, and provide a better initial value for the subsequent optimization process.

[0052] Specifically, the genetic algorithm optimization object is the joint angle output by the MLP network in step S402, and the optimized loss function L C Including position loss L pos and rotation loss L rot Two parts. Position loss L pos The body position p is calculated n With predicted position The smooth L1 distance between the rotation loss L rot Then use the SMQL method to calculate the quaternion q i,j With predicted quaternion The distance between them, where quaternions are used to represent rotations in three-dimensional space.

[0053] By minimizing the loss function L in Cartesian space C , more accurate joint angles can be obtained, making the robot's movements closer to human expressive movements.

[0054] Step S404, controlling the motor motion of each joint of the upper body of the humanoid robot. This includes:

[0055] Classify the collaborative control robot's upper body motors into arm motors, head motors and waist motors.

[0056] A method for synchronous motion control of motors in the upper body of a humanoid robot based on multi-threading and interpolation algorithms includes realizing collaborative control of multiple motors through multi-threading technology, calculating the target angle change of each motor using a differential interpolation algorithm, and gradually approaching the target posture through iterative optimization.

[0057] In summary, this method has the following characteristics:

[0058] (1) Real-time speech recognition with high accuracy and low latency: The present invention adopts advanced speech recognition algorithms to efficiently recognize and process user voice commands, thus achieving high-accuracy and low-latency speech recognition functions. This technology supports users to set personalized wake-up commands, and dynamically manages speech recognition models by detecting whether the user's voice command is a wake-up command, thereby improving the system's intelligence level and user experience.

[0059] (2) Natural and smooth human-computer interaction: It provides multi-language and multi-scenario interaction methods. Users can choose the interaction language according to their needs, which improves the adaptability and usability of the system. The intelligent speech-text model and large language model are used to ensure the logic of the conversation content and improve the fluency of the conversation. Through advanced speech synthesis technology, efficient conversion from text to speech is achieved, allowing the robot to interact with users naturally, and output speech synchronously when performing actions, which enhances the authenticity of the interaction and the user's immersion.

[0060] (3) Human-machine action mapping model: Dynamically matches action sequences based on the conversation content to provide a multi-dimensional interactive experience. By parsing semantics through a large language model, the robot's target action sequence that matches the conversation content is accurately selected to achieve efficient action mapping. Furthermore, the extracted "transition" action reduces the complexity of the action while maintaining the integrity of the action, ensuring smooth and stable robot movements.

[0061] (4) Upper body kinematics algorithm: The action mapping module is used to output the target pose of key points, and a pre-trained multi-layer perceptron (MLP) network is used to quickly solve the transformation from the robot's base coordinate system pose to the joint coordinate system. This reduces the complex kinematic modeling process in the traditional inverse kinematics algorithm and improves real-time performance and computational efficiency. It is suitable for real-time control systems, reduces computational delays, and improves the smoothness and accuracy of robot operations.

[0062] (5) Motor synchronous smooth control: Multithreading technology is used to control multiple joint motors to perform the same action simultaneously, improving the coordination of a single action. Interpolation algorithms are used between adjacent target poses to ensure the smoothness of motor movement and improve the stability of each joint motor movement. Classification management of motors improves the modularity of the system and makes maintenance and expansion more convenient.

[0063] Example 2

[0064] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the humanoid robot multimodal interaction method based on a large language model as described in Example 1.

[0065] Example 3

[0066] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the humanoid robot multimodal interaction method based on a large language model as described in Example 1.

[0067] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multimodal interaction method for humanoid robots based on a large language model, It is characterized in that The steps include: Step S1, in response to receiving a user output voice command and matching the wake-up instruction preset by the humanoid robot, obtaining the user's voice and converting it into original input text; Step S2, based on the original input text, using a large language model to generate text matching the current context, and for output text corresponding to the text matching the current context, converting the output text into speech and then outputting it; Step S3, by parsing the interaction content of the large language model, selecting a continuous humanoid action sequence with the highest matching degree from a plurality of preset candidate humanoid action sequences; Step S4, taking into account the size difference between the virtual human body and the actual robot, as well as the joint angle error, the generated humanoid action sequence is mapped to the humanoid robot to achieve multimodal interaction.

2. A humanoid robot multimodal interaction method based on a large language model according to claim 1, It is characterized in that The step S2 includes the following sub-steps: Step S201, based on the original input text, the preset interaction scenario and the context information, using a large language model to generate text matching the current context; Step S202: Based on the text matching the current context, a large language model is used to generate corresponding output text, and the output text is converted into speech for output.

3. The humanoid robot multimodal interaction method based on a large language model according to claim 1, It is characterized in that The step S4 comprises the following steps: Step S401, obtaining a plurality of key marking points selected from the digital human body model, and normalizing the key marking points to the coordinate system of the humanoid robot using a robot kinematic chain; Step S402, for the normalized key markers, use a pre-trained multi-layer perceptron network to obtain the output target joint angle; Step S403, taking the target joint angle as the optimization object, minimizing the position loss and the rotation loss as the goal, and optimizing the target joint angles of multiple parts by using a genetic algorithm to obtain the final target joint angle; Step S404, controlling each part in a multi-threaded manner, and controlling the actuator of each part to approach the final target joint angle through differential interpolation.

4. The humanoid robot multimodal interaction method based on a large language model according to claim 3, It is characterized in that During the training process of the multi-layer perceptron network, minimizing the Cartesian space loss is taken as the optimization goal.

5. The humanoid robot multimodal interaction method based on a large language model according to claim 3, It is characterized in that In step S403, the initial joint angles of multiple parts are obtained by inverse kinematics solution, and the target joint angles are optimized by genetic algorithm until a preset error threshold is reached.

6. The humanoid robot multimodal interaction method based on a large language model according to claim 3, It is characterized in that The position loss is the smoothed L1 distance between the body position and the predicted position, and the rotation loss is the distance between the quaternion calculated using the SMQL method and the predicted quaternion.

7. The humanoid robot multimodal interaction method based on a large language model according to claim 3, It is characterized in that The parts include the trunk, elbows and wrist joints.

8. The humanoid robot multimodal interaction method based on a large language model according to claim 1, It is characterized in that In step S1, the process of acquiring the user's voice and converting it into original input text includes: Obtain the interactive language and scenario selected by the user, and use the preset model to convert the voice into original input text.

9. An electronic device, It is characterized in that include: One or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the humanoid robot multimodal interaction method based on a large language model as described in any one of claims 1-8.

10. A computer-readable storage medium, It is characterized in that It includes one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing the humanoid robot multimodal interaction method based on a large language model as described in any one of claims 1-8.

Images