A language interactive robot capable of serving a user

Abstract

The application discloses a language interactive robot capable of serving users, comprising: a rack; a laser radar sensor located above the rack; an RGB-D camera located on one side of the rack, the laser radar sensor and the RGB-D camera being capable of environment sensing, enabling the user to customize the working range of the robot; a WIFI module located on the other side of the rack opposite to the RGB-D camera; a GPT model module for interacting with the user, comprising a microphone for receiving the voice of the user; a speaker for playing the voice of the robot; an ASR voice-to-text module for converting the voice of the user into text as input data; a TTS text-to-speech module for converting the text generated by the robot into the voice of the robot for the speaker to play; a processor and a memory, wherein the memory has computer program instructions stored thereon. The robot can realize intelligent and efficient voice interaction with the user, and can perform more accurate mobile navigation.

Description

Technical Field

[0001] This application generally relates to the technical field of intelligent service robots, and more particularly to a language-interactive robot capable of serving users. Background Technology

[0002] With the increasing aging of the population, the demand for services for the elderly is becoming more and more urgent. In the field of robotics, since the early 1990s, a wide range of projects have focused on developing robotic applications for "elderly care in place," thereby expanding the scope of such applications from general healthcare to home care. However, even after thirty years of activity and numerous prototypes, no commercially available robot can support independent elderly care at home. This situation is mainly due to the inherent complexity of the tasks that robots must perform, the unpredictability of the home environment, and the unstructured interaction with users. Low levels of intelligence have always been a bottleneck in the industry. Despite years of development of intelligent service robots in my country, the shortcomings in natural language processing technology have remained a serious limitation. Therefore, how to equip robots with a certain level of dialogue understanding and develop more efficient robot mobility control systems are problems that urgently need to be solved by those skilled in the art. Therefore, our goal is to design an intelligent service robot that: 1. can understand human language and behavior; 2. can complete specific tasks according to human needs; 3. can self-optimize and conform to green and environmentally friendly principles, providing services to the elderly through natural voice interaction. Summary of the Invention

[0003] In view of the above-mentioned technical problems, this disclosure proposes a language-interactive robot capable of serving users, characterized in that the robot includes: a frame; a LiDAR sensor located above the frame; an RGB-D (Red Green Blue-Depth) camera located on one side of the frame, wherein the LiDAR sensor and the RGB-D camera are capable of environmental sensing, allowing the user to customize the robot's working range; a WIFI module located on the other side of the frame opposite the RGB-D camera, wherein the robot can wirelessly communicate with a remote server via the WIFI module; a GPT (Generative Pretrained Transformer) model module for interacting with the user; the GPT model module includes a microphone for receiving the user's speech; a speaker for playing the robot's speech; an ASR (Automatic Speech Recognition) speech-to-text module for converting the received user's speech into text as input data; and a TTS (Text-to-Speech) module. Speech (Text-to-Speech) module, the TTS module is used to convert text generated by the robot into the robot's speech for playback by the speaker; processor and memory, wherein the microphone, the speaker, the ASR speech-to-text module, the TTS module, and the processor and memory are located together inside the rack, and wherein the memory stores computer program instructions, which, when executed by the processor, implement a method for serving a user, the method comprising the following steps: S1, receiving a voice command from the user through the microphone; S2, the user's... S3. Voice commands are processed by the ASR speech-to-text module to convert them into text; S4. The converted text is sent to ChatGPT on the cloud server. The response generated by ChatGPT is sent back to the robot, then converted into speech by the TTS text-to-speech module and played to the user through the speaker; S5. The LiDAR sensor and the RGB-D camera are used to acquire three-dimensional image information of the activity area; S6. Based on the three-dimensional image information, the Gmapping algorithm is used to obtain environmental information; S7. Based on the environmental information, the RRT (Rapidly-exploring Random Tree) algorithm is used to search the path space and determine the movement path; S8. The robot moves along the movement path to the target position.And S8, using the YOLO7 algorithm to perform feature recognition on the object at the target location.

[0004] In a preferred embodiment, the robot communicates wirelessly with a remote wireless server via the WIFI module using the SSH (Secure Shell) network protocol.

[0005] In a preferred embodiment, the robot further includes a robotic arm mounted on the upper part of the frame for grasping objects.

[0006] In a preferred embodiment, in S4, a three-dimensional map of the activity area is constructed using the acquired three-dimensional image information through the Gmapping algorithm.

[0007] In a preferred embodiment, the processor is a JETSON-NANO-DEV-KIT-A artificial intelligence main controller.

[0008] In a preferred embodiment, a drive motor is also included, which is controlled by a motor driver.

[0009] In a preferred embodiment, the motor driver outputs a PRM (Parameters) signal via the CPIO pin to control the speed of the drive motor, and controls the rotational direction of the drive motor by the duty cycle of the PRM signal.

[0010] In a preferred embodiment, in S6, the lidar sensor is used to determine whether there is an obstruction in the active area. If the obstruction exists, it is determined whether the size of the obstruction is greater than a threshold, and the obstruction greater than the threshold is identified as an obstacle.

[0011] In a preferred embodiment, the method further includes: S9, after the robot reaches the target location, requesting the user to confirm via the microphone whether the target location has been reached or whether the item at the target location has been obtained.

[0012] In a preferred embodiment, a display is also included, which shows the user the dialogue text between the user and the robot.

[0013] In a preferred embodiment, as the usage time increases, the robot can learn the user's usage habits, including but not limited to user commands corresponding to usage time and location.

[0014] In a preferred embodiment, the method further includes: S10, based on the user's usage habits learned by the robot, when the robot reaches the corresponding position and has not received any user instructions, issuing a prompt to the user based on the user's learned usage habits and instructions, and interacting with the user.

[0015] Compared with existing technologies, the beneficial effects of this disclosure are as follows: it can overcome the long-standing limitations of natural language processing technology, and by accessing pre-trained language models, it can achieve more intelligent and efficient voice interaction between users and robots. With the increase in usage, the robot will become more intelligent by learning user habits, and the robot will have more accurate mobile navigation, enabling it to move and operate more accurately in the actual environment according to user instructions, and complete the corresponding tasks. Attached Figure Description

[0016] The novel features of this application are specifically set forth in the appended claims. A better understanding of the features and advantages of this application will be gained by referring to the following detailed description and accompanying drawings, which illustrate illustrative embodiments in which the principles of this application are utilized. The drawings are for illustrative purposes only and should not be considered as limiting the scope of this application. Furthermore, the same reference numerals denote the same elements throughout the drawings, in which:

[0017] Figure 1 A schematic diagram of a language-interactive robot capable of serving a user, according to an exemplary embodiment of the present disclosure, is shown.

[0018] Figure 2 A flowchart is shown illustrating a method by which a robot performs services for a user according to an exemplary embodiment of the present disclosure;

[0019] Figure 3 A schematic diagram of a three-dimensional map constructed by a robot according to an exemplary embodiment of the present disclosure is shown;

[0020] Figure 4 A schematic diagram illustrating remote server control of a robot according to an exemplary embodiment of this disclosure is shown; and

[0021] Figure 5 A schematic diagram illustrating the training of a robot according to an exemplary embodiment of the present disclosure is shown.

[0022] Explanation of reference numerals in the attached diagram: 1. LiDAR sensor, 2. RGB-D camera, 3. WIFI module, 4. Microphone, 5. Speaker, 6. Drive motor, 7. Processor, 8. Rack. Detailed Implementation

[0023] Exemplary embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this disclosure and to fully convey the scope of this disclosure to those skilled in the art. Nothing in the following detailed description is intended to suggest that any particular component, feature, or step is essential to this application. Those skilled in the art will understand that various features or steps may be substituted for or combined with each other without departing from the scope of this disclosure.

[0024] Figure 1A schematic diagram of a language-interactive robot capable of serving a user is shown according to an exemplary embodiment of this disclosure. This disclosure proposes a language-interactive robot (i.e., a service robot ServiceGPT) capable of serving a user (e.g., navigation, object retrieval, chat, etc.), characterized in that the robot may include: a LiDAR sensor 1 and an RGB-D camera 2, a Wi-Fi module 3, a microphone 4, a speaker 5, an ASR speech-to-text module, a TTS text-to-speech module, a processor 7, and a memory, as well as a rack 8. The LiDAR sensor 1 may be located above the rack 8. The RGB-D camera 2 may be located on one side of the rack 8. The LiDAR sensor 1 and the RGB-D camera 2 are capable of environmental sensing, allowing the user to customize the robot's working range and autonomously navigate the robot in indoor environments to avoid obstacles. In a preferred embodiment, the LiDAR sensor 1 may be a SLAMTEC RPLIDAR A1 radar, and the RGB-D camera 2 may be a Jetson Nano IMX219 camera. In some other embodiments, the LiDAR sensor 1 and the RGB-D camera 2 may be any other type of radar sensor and camera that a person skilled in the art deems appropriate. The robot can wirelessly communicate with a remote server via the WIFI module 3. The WIFI module 3 may include multiple antennas. The WIFI module 3 may be located on the other side of the rack 8 opposite the RGB-D camera 2. Preferably, the WIFI module 3 may include two antennas, both located on one side of the robot. In a preferred embodiment, the robot can wirelessly communicate with a remote server via the WIFI module 3 using the SSH network protocol. For example, an operator can remotely control the robot from a remote control room via a wireless remote server, such as performing navigation, viewing the robot's surrounding environment in real-time via the RGB-D camera 2 on a remote server's display, and performing cloud training for the robot's generative pre-trained natural language interaction model. Furthermore, the SSH network protocol allows the robot to remotely connect to the server and quickly handle unexpected events, ensuring high communication stability. The robot may also include a microphone 4, which can be used to receive user voice, for example, inputting the received user voice into an ASR speech-to-text module for further processing. In a preferred embodiment, the microphone 4 may be a dual MEMS (Micro-Electro-Mechanical System) silicon microphone. In some other embodiments, the microphone 4 may be any other type of microphone that a person skilled in the art deems appropriate. The robot may also include a speaker 5, which can be used to play the robot's voice.The robot may further include an ASR speech-to-text module, which can be used to convert received user speech into text as input data. The robot may also include a TTS text-to-speech module, which can be used to convert text generated by the robot into the robot's speech for playback by the speaker 5. For example, the TTS text-to-speech module can be used to convert text processed by the processor (i.e., text output by a fine-tuned and improved supervised tuning model) into the robot's speech for playback by the speaker 5. Typically, the ASR speech-to-text module, the TTS text-to-speech module, and the processor can be integrated together. For example, the ASR speech-to-text module, the TTS text-to-speech module, the processor 7, and the memory can be integrated into the same substrate inside the rack 8. This substrate can be located in the middle of the robot. Based on traditional robot voice interaction, according to embodiments of the present invention, dialogue management modules (e.g., ASR speech-to-text module and TTS text-to-speech module) are connected to a pre-trained language model to improve the robot's intelligence and provide users with a better language interaction experience. The robot may also include a GPT model module, which can be used to interact with the user. In some embodiments, since the GPT model requires a large amount of computing resources and time, Microsoft's Deepspeed chat platform can be used to train the GPT model of this invention after building the framework and investing in a large-scale dialogue dataset. Combining the GPT model with the service robot of this invention can provide a more efficient, convenient and comfortable service experience. For example, the GPT model module of this invention can also be optimized with cloud feedback, that is, based on a large database, the GPT model module can be further trained using its own data, and continuously fed back and corrected during operation to improve the personalized experience for users. The robot of this invention can also realize real-time user feedback and model updates, and can upload real-time collected data to the cloud for real-time analysis and decision-making to obtain more accurate feedback and guidance, while supporting key links such as data collection, model training, distributed learning and model updates. In a preferred embodiment, the robot may also include a drive motor 6. The drive motor may be located at the lower part of the robot, and the aforementioned base plate may be placed above the drive motor 6. Wheels may be attached to both ends of the drive motor 6. When the drive motor 6 rotates, it can drive the wheels to rotate so that the robot can move. The drive motor 6 can be controlled by a motor driver. In a preferred embodiment, the motor driver can control the speed of the drive motor 6 by outputting a PRM signal through the CPIO pin, and can control the rotational direction of the drive motor 6 by the duty cycle of the PRM signal. In a preferred embodiment, the robot may also include a display.The display can show the user the dialogue text between the user and the robot. In some embodiments, the display can be a CRT display, LCD display, LED display, 3D display, plasma display, or any other type of display deemed suitable by those skilled in the art. The robot may also include a processor 7 and a memory, wherein the memory stores computer program instructions that, when executed by the processor, implement methods for the robot to perform services for the user. In a preferred embodiment, the processor 7 can be a JETSON-NANO-DEV-KIT-A AI main controller. In a preferred embodiment, the robot of this application can use a ROS robot system, which can employ a cerebellum-cerebrum design, with the cerebrum using a JETSON-NANO-DEV-KIT-A AI main controller for AI-related processing, and the cerebellum using a Raspberry Pi RP2040 microcontroller for processing other than AI. In some other embodiments, the processor 7 can be any other type of processor deemed suitable by those skilled in the art. In some embodiments, the robot may further include an IMU (Inertial Measurement Unit) attitude sensor driven by a DCM (Direct Current Machine) and employing PID (Proportional Integral Derivative) speed control to achieve a wheel tachometer output. In some embodiments, the robot also includes a robotic arm mounted on the upper part of the frame for grasping objects.

[0025] Figure 2 A flowchart illustrating a method for a robot to perform services to a user according to an exemplary embodiment of this disclosure is shown. This disclosure also proposes a method for a robot to perform services to a user, which may include the following S1-S8. In S1, a user's voice command can be received via the microphone 4. In S2, the user's voice command can be processed via the ASR speech-to-text module to convert the voice command into text. The ASR system converts the user's voice input into text form so that a computer can understand and process it. In S3, the converted text can be sent to a language model on a cloud server, which may be ChatGPT (or a similar language model). The response generated by the language model is transmitted back to the robot, then converted into speech via a TTS text-to-speech module and played to the user through the speaker. In S4, three-dimensional image information of the activity area can be acquired using the LiDAR sensor and the RGB-D camera. Figure 3A schematic diagram of a 3D map constructed by a robot according to an exemplary embodiment of this disclosure is shown. In S5, environmental information can be acquired using a Gmapping algorithm based on the 3D image information. In a preferred embodiment, a 3D map of the activity area can be constructed using the acquired 3D image information through a real-time 3D Gmapping SLAM (Simultaneous Localization and Mapping) algorithm. In a preferred embodiment, the LiDAR sensor 1 can be used to determine whether there are obstructions in the activity area; if obstructions exist, it is determined whether the size of the obstruction is greater than a threshold, and obstructions greater than the threshold are identified as obstacles. In S6, a path space can be searched and a movement path determined using an RRT algorithm based on the environmental information. In a preferred embodiment, the LiDAR sensor 1 can be used to determine whether there are obstructions in the activity area; if obstructions exist, it is determined whether the size of the obstruction is greater than a threshold, and obstructions greater than the threshold are identified as obstacles. In S7, the robot can move along the movement path to the target location. In S8, the YOLOv7 algorithm can be used to perform feature recognition on objects at the target location. In a preferred embodiment, the method may further include: S9, after the robot reaches the target location, requesting the user to confirm via the microphone 4 whether the target location has been reached or whether the item at the target location has been retrieved. In a preferred embodiment, as usage time increases, the robot can learn the user's usage habits, including but not limited to user commands corresponding to usage time and location. In a preferred embodiment, the method further includes: S10, based on the user's usage habits learned by the robot, when the robot reaches the corresponding location and has not received a user command, issuing a prompt to the user based on the learned user usage habits and commands, and interacting with the user.

[0026] Figure 4A schematic diagram illustrating remote server control of a robot according to an exemplary embodiment of this disclosure is shown. In a preferred embodiment, the LiDAR sensor 2 and RGB-D camera 3 use a real-time 3D GmappingSLAM algorithm to construct a 3D environment map during motion and estimate the robot's position and orientation on the map for autonomous navigation. In another preferred embodiment, the LiDAR sensor 1 and RGB-D camera 2 sense 3D scene and 2D image information of the environment to accurately identify, locate, and track objects. The robot can activate the camera to acquire images, compress the images, and transmit them to a remote server via Wi-Fi module 3. The remote server can then receive these compressed images and process them accordingly. For example, the robot can wirelessly communicate with the remote server using the SSH network protocol, allowing operators to remotely control the robot using the wireless remote server and view the robot's surrounding environment in real time via a display.

[0027] Figure 5A schematic diagram illustrating the training of a robot according to an exemplary embodiment of this disclosure is shown. For example, the input data can first be labeled, and then the labeled data 11 can be input into a pre-trained model 12 for training to obtain a supervised fine-tuning model 13 (i.e., an SFT (Supervised Fine-Tuning) model). Due to the complexity and uncertainty of real-world environments, learning directly from reward signals can become very difficult. Therefore, reinforcement learning using human feedback has become an effective method. Reinforcement learning from human feedback uses human feedback as an additional signal to guide the agent's learning, thereby helping us generate our own high-quality generative natural language interaction models. For example, training a natural language interaction model requires a large amount of computational resources and time. For this purpose, the present invention uses an existing platform, i.e., the pre-trained model is a model pre-trained by an existing platform, and it is pre-installed in the robot of the present invention to process the input data received by the ASR speech-to-text module and output the processed or generated results to the TTS text-to-speech module for playback by the speaker 5. After establishing the framework and deploying a large-scale dialogue dataset, the platform enables end-to-end RLHF (Reinforcement Learning from Human Feedback) training of natural language interaction models, utilizing human feedback for reinforcement learning. The supervised tuning model obtained after training can include an exponential moving average 14, an actor model 15, and a frozen reference model 16. Combining generative natural language interaction models with service robots can provide a more efficient, convenient, and comfortable service experience. Secondly, the output dataset of the supervised tuning model can be scored (i.e., paired good / bad answers 17), and then the scored output dataset can be input into a pre-trained model 12 for training to obtain an RW (Reward) model 19, which differs from the pre-trained model 12. In some implementations, in addition to obtaining an independent reward model, a critic model 20 can also be obtained. Thirdly, the supervised tuning model can be further fine-tuned and improved based on the reward feedback of the reward model 19. Reinforcement learning can typically be applied to tune SFT models by optimizing the training reward model. For example, based on the reward feedback of the reward model, the critic model, the actor model, or the frozen reference model, the supervised tuning model can be further fine-tuned and improved by inputting data 21 generated using the actor model, or pre-training data 24, or the actor's pre-training target 23 into the existing platform (i.e., DeepSpeed ​​22), using the Proximal Policy Optimization (PPO) algorithm (which is performed by DeepSpeed ​​22).The fine-tuned and improved supervised tuning model can output the results and output the text of the results to the TTS text-to-speech module to convert the text of the results into the robot's speech for playback by the speaker 5.

[0028] Currently, most service robots can only provide voice broadcasts or text interactions, with a low level of intelligence. To enhance the convenience and applicability of the design, natural language interaction between users is an essential technology, and voice interaction technology needs to be highly intelligent, accurately recognize user intent, and have strong adaptability. Compared with existing technologies, the language model involved in this application can achieve end-to-end RLHF training, that is, reinforcement learning using human feedback. The robot disclosed in this application can overcome the long-standing limitations of natural language processing technology. By accessing a pre-trained language model, it can achieve more intelligent and efficient voice interaction between users and the robot. Furthermore, the robot possesses more precise mobile navigation, can more accurately perceive 3D scenes and 2D image information in the real environment, can perform large-scale complex calculations and AI reasoning, and can more accurately move and operate autonomously in the real environment according to user instructions to complete corresponding tasks.

[0029] It should be understood that the systems and / or methods provided in the various embodiments of this invention can be combined, modified, and / or altered to form new technical solutions. Without inventive effort, these technical solutions should also be included within the scope of protection claimed by this invention.

[0030] Numerous specific examples are provided in the embodiments described herein, and it should be understood that these examples are merely for illustrating the embodiments of the invention in detail and are not intended to limit the invention. Embodiments of the invention can be practiced without these specific examples. In some embodiments, methods, structures, and / or techniques well-known to those skilled in the art are not shown in detail so as not to obscure the understanding of the invention.

[0031] While preferred embodiments of the invention have been shown and described herein, it will be readily understood by those skilled in the art that these embodiments are provided by way of example only. Various changes, modifications, and substitutions will now occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein are optionally used to practice the invention. The scope of the invention is intended to be defined by the claims, and thereby covers the methods and structures within the scope of these claims and their equivalents.

Claims

1. A language-interactive robot capable of serving users, characterized in that, The robot includes: frame; A lidar sensor, the lidar sensor being located above the frame; An RGB-D camera is located on one side of the frame. The LiDAR sensor and the RGB-D camera are capable of environmental sensing, allowing the user to customize the robot's working range. A WIFI module is located on the other side of the rack opposite the RGB-D camera, wherein the robot can wirelessly communicate with a remote server through the WIFI module; The GPT model module is used to interact with the user; The GPT model module includes a microphone, which is used to receive the user's voice. A speaker, used to play the robot's voice; The ASR speech-to-text module is used to convert the received speech of the user into text as input data. The TTS text-to-speech module is used to convert the text generated by the robot into the robot's speech for playback by the speaker. As the robot is used over time, it can learn the user's habits, including but not limited to user commands corresponding to usage time and location; and A processor and memory, wherein the microphone, the speaker, the ASR speech-to-text module, the TTS text-to-speech module, and the processor and memory are located together within the rack, and wherein the memory stores computer program instructions that, when executed by the processor, implement a method for serving a user, the method comprising: S1, Receive the user's voice command through the microphone; S2, the user's voice command is processed by the ASR speech-to-text module to convert the voice command into text; S3, the converted text is sent to ChatGPT of the GPT model module on the cloud server, the response generated by ChatGPT is transmitted back to the robot, and then converted into speech by the TTS text-to-speech module and played to the user through the speaker; S4, use the lidar sensor and the RGB-D camera to acquire three-dimensional image information of the active area; S5, Based on the three-dimensional image information, the Gmapping algorithm is used to obtain environmental information; S6. Based on the environmental information, use the RRT algorithm to search the path space and determine the movement path; S7, the robot moves along the movement path to the target position; S8, use the YOLOv7 algorithm to identify the features of the items at the target location; and S10, based on the user's usage habits learned by the robot, when the robot reaches the corresponding position and does not receive the user's instruction, the robot issues a prompt to the user based on the learned user's usage habits and instructions, and interacts with the user. The training of the GPT model module includes optimizing the supervised optimization model by optimizing the training reward model. The supervised optimization model is obtained by training the pre-trained model with labeled data, and the reward model is obtained by training the pre-trained model with the scoring output dataset.

2. The robot according to claim 1, characterized in that, The robot communicates wirelessly with a remote wireless server via the WIFI module using the SSH network protocol.

3. The robot according to claim 1, characterized in that, It also includes a robotic arm mounted on the upper part of the frame for grasping items.

4. The robot according to claim 1, characterized in that, In S4, the acquired 3D image information is used to construct a 3D map of the activity area using the Gmapping algorithm.

5. The robot according to claim 1, characterized in that, The processor is the JETSON-NANO-DEV-KIT-A artificial intelligence main controller.

6. The robot according to claim 1, characterized in that, It also includes a drive motor, which is controlled by a motor driver.

7. The robot according to claim 6, characterized in that, The motor driver outputs a PRM signal through the CPI0 pin to control the speed of the drive motor, and controls the rotational direction of the drive motor through the duty cycle of the PRM signal.

8. The robot according to claim 1, characterized in that, In S6, the lidar sensor is used to determine whether there is an obstruction in the active area. If there is an obstruction, it is determined whether the size of the obstruction is greater than a threshold, and the obstruction that is greater than the threshold is identified as an obstacle.

9. The robot according to claim 1, characterized in that, The method further includes: S9, after the robot reaches the target location, requesting the user to confirm via the microphone whether the target location has been reached or whether the item at the target location has been obtained.

10. The robot according to claim 1, characterized in that, It also includes a display that shows the user the text of the conversation between the user and the robot.

Images