Indoor mobile service robot interaction task execution method and device, storage medium and indoor mobile service robot system

Abstract

The invention relates to an indoor mobile service robot interaction task execution method and device, a storage medium and an indoor mobile service robot system. The method comprises the following steps: acquiring original visual data and original spatial data, respectively processing the original visual data and the original spatial data, acquiring an original audio signal, and converting the original audio signal into a text signal to obtain a dynamic interactive input instruction; analyzing the dynamic interaction input instruction by utilizing a preset large language model to obtain a meta-action sequence corresponding to the to-be-processed task; according to the meta-action sequence, executing each meta-action in the to-be-processed task in sequence, and scheduling the corresponding processed visual data and spatial data when each meta-action is executed; when the execution result of the meta-action is confirmed, the current task execution state is broadcasted through voice, and task interaction is completed. Compared with the prior art, the method has the advantages of complex and wide task processing, high dynamic adaptability, natural interaction, high task execution success rate of an indoor service scene and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of service robots, and in particular to an indoor mobile service robot interactive task execution method, device and storage medium, and an indoor mobile service robot system. Background Art

[0002] With the continuous development of artificial intelligence and robotics, the application of indoor mobile service robots in home, medical, hotel and other scenarios has gradually increased. However, existing indoor service robots still face the following problems during task execution:

[0003] First, although traditional robots are equipped with multiple sensors, they process data in a fragmented manner, making it difficult to effectively integrate multimodal information such as vision, language, depth, and lidar to complete the understanding of task instructions and data scheduling. Second, task instruction parsing is not intelligent enough: existing robot systems mostly use predefined rules to process input instructions, and lack the ability to parse the intent and details of complex and long text instructions, resulting in a limited range of task execution types. Third, weak adaptability to dynamic environments: In indoor scenes, changes in dynamic objects and obstacles prevent robots from updating task plans in a timely manner, affecting the interactive experience and task completion rate. Finally, lack of language interaction capabilities: Traditional service robots' language interactions are mostly limited to fixed templates, making it difficult to achieve natural and flexible voice interactions.

[0004] Therefore, how to improve the indoor service robot's ability to process multimodal information and adapt to dynamic environments, and enhance the robot's intelligence in understanding and executing tasks, has become a problem that needs to be solved in this field. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an indoor mobile service robot interactive task execution method, device and storage medium, and an indoor mobile service robot system, which integrates the scheduling of multimodal perception data and dynamic environment adaptation, and can effectively improve the intelligence level of task understanding and execution.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, there is provided a method for executing interactive tasks of an indoor mobile service robot driven by a large model, comprising the following steps: acquiring original visual data and original spatial data and processing them respectively, acquiring original audio signals and converting them into text signals, and obtaining dynamic interactive input instructions; parsing the dynamic interactive input instructions using a preset large language model to obtain a meta-action sequence corresponding to the task to be processed; executing each meta-action in the task to be processed in sequence according to the meta-action sequence, and each meta-action dispatching the corresponding processed visual data and spatial data during execution; when the execution result of the meta-action is confirmed, voice broadcasting the current task execution status to complete the task interaction.

[0008] As a preferred technical solution, the original visual data is obtained and processed. The specific process includes: using an RGB_D camera to obtain the depth and color visual information of the dynamic environment, completing image correction and transformation, and obtaining a depth image and an RGB image; using a YOLOv7 target detector to process the RGB image to obtain object categories, object details, and pedestrian postures; using a binocular vision estimation method to obtain depth information from the depth image and solve the three-dimensional spatial coordinates of the target operation position.

[0009] As a preferred technical solution, the original spatial data is obtained and processed. The specific process includes: building an interference-free static scene model, and using lidar and SLAM methods to extract static obstacle information and robot positioning, and building a grid obstacle map.

[0010] As a preferred technical solution, the original audio signal is obtained and converted into a text signal. The specific process includes: using a preset microphone device to obtain the audio signal; and using an automatic speech recognition method to convert the audio signal into a text signal.

[0011] As a preferred technical solution, the automatic speech recognition method is implemented based on a deep learning model.

[0012] As a preferred technical solution, the meta-action sequence includes multiple meta-actions, each meta-action includes an action type, an implementation object and implementation details. The action type is classified according to the robot's sensors and operating capabilities, the implementation object is determined according to actual task requirements, and the implementation details are used to describe the specific execution method of the task.

[0013] As a preferred technical solution, when the execution result of the meta-action is confirmed, the current task execution status is reported by voice. The specific process includes: real-time monitoring and data collection of the actual execution process of the current meta-action; judging whether the current meta-action is completed or there is a deviation based on the preset task completion index; using the preset large language model to analyze and summarize the data collected during the execution process and the task completion status, and generate corresponding interactive feedback results; using the text-to-speech method to convert the feedback results into natural language, and report them by voice.

[0014] According to a second aspect of the present invention, there is provided a large model-driven indoor mobile service robot interactive task execution device, comprising a memory, a processor, and a program stored in the memory, wherein the processor implements the described method when executing the program.

[0015] According to a third aspect of the present invention, there is provided a storage medium having a program stored thereon, wherein the program implements the method described above when executed.

[0016] According to a fourth aspect provided by the present invention, there is provided an indoor mobile service robot system, the system comprising a multimodal perception module, an instruction parsing module, a navigation and execution module, and an interaction and feedback module: the multimodal perception module is used to acquire original visual data and original spatial data and process them respectively, acquire original audio signals and convert them into text signals, and obtain dynamic interaction input instructions; the instruction parsing module is used to parse the dynamic interaction input instructions using a preset large language model to obtain a meta-action sequence corresponding to the task to be processed; the navigation and execution module is used to execute each meta-action in the task to be processed in sequence according to the meta-action sequence, and each meta-action dispatches the corresponding processed visual data and spatial data when executed; the interaction and feedback module is used to voice broadcast the current task execution status after the execution result of the meta-action is confirmed, so as to complete the task interaction.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. Improve the intelligent level of task execution: By combining a large language model with a multimodal perception method, the present invention can efficiently parse and understand complex natural language instructions and generate precise meta-action sequences that can be executed by the robot. Compared with traditional robots that can only process simple instructions, the robot system using the present invention can understand and execute complex, situational and logical task instructions. For example, the robot can understand the complex instruction of "please find Xiao Ming in the living room and take two apples from the kitchen to him" and decompose it into multiple executable subtasks;

[0019] 2. Strengthen the ability to adapt to dynamic environments: The present invention uses an RGB_D camera, a YOLOv7 target detector, and a binocular vision estimation method to obtain and process raw visual data, uses a laser radar and SLAM method to extract static obstacle information and robot positioning, constructs a grid obstacle map, completes the acquisition of raw spatial data and processes it, and can quickly adapt to changes in the indoor environment through real-time environmental perception and dynamic path planning. When new obstacles or dynamic changes appear in the environment, the robot using the present invention can update the path in real time and adjust the task execution strategy, which makes the robot have strong adaptability and robustness in complex and changing indoor environments, greatly improving the success rate of task execution;

[0020] 3. Enhanced voice interaction experience: The present invention uses a large model-driven automatic speech recognition (ASR) model and a text-to-speech (TTS) method. The robot using the present invention can interact with the user in a natural and smooth voice manner, rather than being limited to template-based feedback. This can improve the user experience. The user can not only directly issue tasks through voice commands, but also obtain feedback information on task execution in real time. The addition of voice interaction enables the robot to be more intelligently integrated into home and work scenarios, providing more humane services.

[0021] 4. Optimize task execution efficiency and accuracy: For non-sequential task expressions, the robot using the present invention can more flexibly schedule and arrange the task execution order, more accurately perceive and locate the target, and adjust the target point order and operation progress according to real-time environmental information, thereby being able to efficiently perform tasks, reduce error rates and resource waste. For example, the robot can avoid repeated movements, optimize the grasping path, shorten the task execution time, and improve work efficiency;

[0022] 5. Expanded application scenarios: The present invention has strong versatility and can adapt to various indoor environments, such as homes, hospitals, hotels, etc. With the diversification of task requirements and the continuous changes in the environment, the robot using the present invention can meet the complex needs in different scenarios by intelligently adjusting the task execution strategy. This enables the robot to be widely used in various fields and promotes the popularization of intelligent service robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The present invention provides a schematic diagram of the overall process of the method;

[0024] Figure 2 A schematic diagram of a home scene layout provided in Example 2 of the present invention;

[0025] Figure 3 A schematic diagram of a meta-action generation process of an instruction parsing module provided in Embodiment 2 of the present invention;

[0026] Figure 4 A schematic diagram of the result of a task executing a detection meta-action provided in Example 2 of the present invention;

[0027] Figure 5 Schematic diagram of a robot performing some tasks provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0028] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0029] Embodiment 1:

[0030] The large model-driven indoor mobile service robot interactive task execution method provided by the present invention combines a large language model with a multimodal perception method to perform dynamic navigation and operation tasks, aiming to improve the service robot's task understanding, planning and execution capabilities in a dynamic environment. The method first obtains the original visual data and the original spatial data and processes them separately, obtains the original audio signal and converts it into a text signal, and obtains the dynamic interactive input instruction; secondly, the dynamic interactive input instruction is parsed using a preset large language model to obtain the meta-action sequence corresponding to the task to be processed; then, according to the meta-action sequence, each meta-action in the task to be processed is executed in sequence, and each meta-action schedules the corresponding processed visual data and spatial data during execution; finally, when the execution result of the meta-action is confirmed, the current task execution status is announced by voice to complete the task interaction.

[0031] This embodiment provides one implementation of the above method, and the implementation process is as follows: Figure 1 As shown, the specific execution process is as follows:

[0032] Step S1, use an RGB_D camera to obtain depth and color visual information of a dynamic environment, complete image correction and transformation, and obtain a depth image and an RGB image.

[0033] Specifically, an RGB-D camera capable of perceiving visual spatial information is used, which can capture color and depth visual information in a dynamically changing environment. The acquired images are further processed to complete image correction and image enhancement operations. Image correction processing is used to correct camera distortion, especially radial and tangential distortion, to ensure the accuracy of the geometric structure of the image. At the same time, parallax correction is performed using depth information to reduce artifacts and distortion in the image. Image enhancement optimizes the illumination contrast of the image through methods such as histogram equalization and adaptive contrast enhancement to adapt to different indoor lighting environments. In low-light or high dynamic range environments, a noise reduction algorithm using non-local mean filtering is used to improve image clarity. Through these processes, the quality of RGB-D camera visual information can be improved in complex environments to meet the needs of robot perception and decision-making.

[0034] Step S2, use the YOLOv7 target detector to process the RGB image to obtain the object category, object details and pedestrian posture, so as to understand the location information of dynamic objects in the environment and the spatial relationship between objects. Optionally, use a pre-built dataset of common objects in the home environment to prepare the YOLOv7 target detector for training to improve the visual recognition ability of operable objects. The YOLOv7 target detector is used to detect the object category, the color segmenter obtains the color features of the object, the human posture detection locates the human position and posture features, and the gesture recognition is used to wake up the robot for special tasks, so as to comprehensively obtain the location information and spatial state relationship of dynamic objects and pedestrians in the environment.

[0035] Specifically, the processed camera RGB data and the categories of common objects in the home environment are used to construct a visual perception dataset for the construction of the visual perception submodule. For the object category detection part, the YOLOv7 target detection algorithm and the calibrated object dataset are used to train the target detection model. The training adopts a supervised deep learning method, and iterative optimization is performed through a large amount of labeled data to improve the object detection accuracy and recall rate. The YOLOv7 algorithm quickly detects the location and category of objects through a convolutional neural network (CNN) model, and can maintain high detection accuracy while processing in real time. During the training process, data enhancement methods (such as rotation, scaling, cropping, and color jittering) are also used to expand the dataset to improve the generalization ability and robustness of the model. Through this training, the robot can accurately identify and locate various operable objects in the home environment, and judge their spatial positions and relative relationships. Based on these visual perception capabilities, the robot can efficiently perform tasks such as automatic grasping, obstacle avoidance, and object classification. In addition, the color features of objects extracted by the color segmenter are combined with the human posture detection method to further enhance the robot's ability to identify pedestrians and objects, thereby achieving more complex interactive tasks, such as gesture recognition to wake up the robot or respond according to the user's behavior. Ultimately, the combination of these methods enables the robot to obtain and update the location information of surrounding objects and pedestrians in real time in a dynamic environment, so as to perform efficient tasks more accurately.

[0036] Step S3, using a binocular vision estimation method to obtain depth information from the depth image, and solving the three-dimensional spatial coordinates of the target operation position. Further, after using the binocular vision estimation method to obtain the depth information, the hand-eye calibration is used to complete the solution of the three-dimensional spatial coordinates from the camera to the target operation position, providing the robot with accurate operation position coordinates, ensuring high-precision spatial perception capabilities when performing tasks.

[0037] Specifically, the three-dimensional spatial coordinates of the higher-precision depth image processed in step S1 are solved by the binocular vision estimation method and the hand-eye calibration method to provide accurate operation position coordinates for the robot manipulator. The hand-eye calibration method is used to calibrate the conversion relationship between the camera coordinate system and the robot manipulator coordinate system, thereby ensuring that the machine vision system can accurately dock with the movement of the manipulator, and ensuring that the depth image acquired by the visual sensor can be converted into the operation coordinate system of the robot manipulator, so that the visual system can correctly guide the manipulator to perform precise operations in three-dimensional space. By combining the binocular vision estimation method and the hand-eye calibration method, the robot can achieve high-precision spatial perception, accurately judge the three-dimensional position of the object, and provide accurate operation target position information for the operation of the manipulator. This ability is essential for performing complex operation tasks, such as object grasping, fine assembly, and object sorting. The robot can accurately adjust the motion trajectory of the manipulator according to the three-dimensional depth map and spatial coordinates generated in real time to complete the task.

[0038] Step S4, construct an interference-free static scene model, and use lidar and SLAM (Simultaneous Localization and Mapping) methods to extract static obstacle information and robot positioning in the robot deployment environment, and construct a grid obstacle map for defining navigation nodes and navigable areas, thereby providing basic environmental data for robot navigation.

[0039] Specifically, laser radar and other equipment are used to build environmental maps in preset indoor scenes. For the construction of environmental maps, this embodiment deploys the Cartograhper algorithm on the AGV robot chassis based on the Sick Tim single-line laser radar to build a 2D grid map and realize positioning during navigation. In the pre-built map and scene, a global planner is combined with a local planner for obstacles, and the obstacle information scanned by the laser radar is stored in the local obstacle map. Navigable topological points are selected around common furniture, room entrances, operable countertops, etc. to ensure that the robot can accurately understand the static obstacle information in the global space when performing tasks, and plan a suitable navigation route according to the A* algorithm.

[0040] Steps S1 to S3 complete the acquisition and processing of raw visual data and raw spatial data. The processed visual data (such as the object position and category extracted from the RGB image, and the three-dimensional spatial coordinates of the target operation position extracted from the depth image) and the processed spatial data (such as the grid obstacle map) can be called by the corresponding meta-action when executing the task to be processed later.

[0041] Step S5, using a preset microphone device to obtain an audio signal, and using a preset automatic speech recognition model to convert the audio signal into a text signal to obtain a dynamic interactive input instruction. The microphone device obtains an audio signal by capturing the spoken instruction, and uses the automatic speech recognition model to convert the audio signal into text information, thereby converting the audio modality into a text modality that can be understood by the large model.

[0042] Exemplarily, the robot is equipped with a multi-microphone array to obtain verbal interactive instructions from humans in real time. Through multi-microphone array technology, the robot can clearly capture audio signals in different directions and use spatial sound source localization methods to determine the source location of the instructions. After the acquired audio signal is processed by noise suppression and echo cancellation, it is converted into text instructions using an automatic speech recognition (Automatic Speech Recognition) model. The ASR model effectively improves the accuracy of speech recognition, especially in noisy environments, through deep learning models such as convolutional neural networks (CNN) and recurrent neural networks (RNN). Finally, the converted text instructions are sent to the large model for task parsing and execution to ensure that the robot can complete accurate task operations according to verbal instructions.

[0043] Step S6, using a preset large language model to parse the dynamic interactive input instructions, and obtain a meta-action sequence corresponding to the task to be processed.

[0044] This step applies a large language model to the interactive instructions that have been converted into text, understands and organizes the complex text instructions based on the prompt words, generates a meta-action sequence containing three attributes: action, goal, and details, and converts the abstract long text instructions into robot executable code parameters, mapping them into meta-action sequences. Specifically:

[0045] The received text instructions, as well as the text description words of the environment information and the robot status are input into the large language model as prompt content for understanding and integration, and the generated text instructions are disassembled into a fine-grained meta-action sequence. Each meta-action consists of three elements: action type, implementation object (i.e., target) and implementation details. Among them, the action type is classified according to the robot's sensors and operating capabilities, mainly including: listening, broadcasting, observation, grasping and navigation. The implementation object is determined according to the actual task requirements, which may be an object, place or person. The implementation details further describe the specific execution method of the task.

[0046] Step S7, according to the meta-action sequence, execute each meta-action in the task to be processed in sequence, and each meta-action schedules the corresponding processed visual data and spatial data when executing. Specifically, according to the generated meta-action sequence, the robot executes each meta-action in the task with timing characteristics that needs to schedule multiple modal data in sequence, ensuring that the task is carried out in a predetermined order and within a time limit, ensuring the efficient execution of the task and the correctness of the path. Among them, the multi-modal data that can be called include the item category, item details and pedestrian posture obtained in step S2, the three-dimensional space coordinates obtained in step S3, and the static obstacle information obtained in step S4.

[0047] Each meta-action sequence has a timing characteristic to ensure the continuity and consistency of task execution and return the meta-action completion status or result. Specifically, each meta-action corresponds to an operation of the robot, such as moving, grabbing, placing, etc., and each meta-action has a timing characteristic, that is, it has a clear start and end time, execution order and dependency. Therefore, the execution of each meta-action needs to refer to the task timing, that is, some actions may need to wait for the previous action to be completed before they can be started. By executing the meta-action queue strictly arranged by the large model in time, the robot can efficiently and safely complete the scheduled tasks in complex environments, such as object handling, environmental cleaning, etc., while ensuring the continuity and consistency of task execution. During the execution process, the robot continuously evaluates the completion of each action through the state feedback mechanism, and dynamically adjusts the next operation according to the actual results. If a meta-action fails to complete as expected, the robot will recalculate and insert a noise action into the task queue to move left and right to adjust the posture, so as to adjust the task meta-action execution environment, improve the success rate, and ensure that the task goal is finally achieved.

[0048] Step S8: When the execution result of the meta-action is confirmed, the current task execution status is announced by voice to complete the task interaction. Optionally, when the meta-action execution result is returned or interaction with the user is required, the current task execution status is described by a text-to-speech method, a voice announcement is generated, and the task interaction process is completed. Specifically:

[0049] The robot detects and reports the results of task execution. First, it uses sensors (such as RGB-D cameras, lidar, force feedback sensors, etc.) to monitor the actual execution process of the current meta-action, i.e. the current task, in real time and collect data to ensure accurate feedback of the task status; it determines whether the current meta-action, i.e. the current task is completed or has deviations, or needs to re-execute certain operations based on the preset task completion indicators; it uses a large language model to analyze and summarize the data collected during the execution process and the task completion status, and generates corresponding interactive feedback results; when the task execution results are confirmed, the robot uses the TTS method to convert the feedback results into natural language, and broadcasts them through the built-in speaker.

[0050] In this process, the speech synthesis method based on deep neural network (DNN) can generate more natural and fluent speech output. By synthesizing text into speech, the robot can not only clearly inform the user of the results of task execution, but also provide further suggestions or adjust operations according to user needs. The feedback interaction method combining TTS method with large model dynamically adjusts the tone, speed and speech emotion according to context and sentiment analysis, making the robot report closer to real conversation, making the human-computer interaction process smoother and more effective, and improving the user experience.

[0051] Furthermore, this embodiment also provides a large model-driven indoor mobile service robot interactive task execution device, including a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements one or more steps of the aforementioned method, and the specific implementation process will not be repeated here.

[0052] According to a third aspect of the present invention, there is provided a storage medium on which a program is stored. When the program is executed, one or more steps of the aforementioned method are implemented, and the specific implementation process will not be repeated here.

[0053] Embodiment 2:

[0054] This embodiment provides an indoor mobile service robot system, which includes a multimodal perception module, an instruction parsing module, a navigation and execution module, and an interaction and feedback module. Among them: the multimodal perception module is used to execute steps S1 to S5 of the method in Example 1, aiming to obtain original visual, audio and spatial data through sensors such as RGB-D cameras, microphones and laser radars, and provide dynamic and static information of the environment; the instruction parsing module is used to execute step S6 of the method in Example 1, aiming to combine a large language model to parse and abstract text instructions and generate a sequence of meta-actions executable by the robot; the navigation and execution module is used to execute steps S4 and S7 of the method in Example 1, aiming to use the SLAM method to build an environmental map, combine target detection and three-dimensional space positioning methods, plan navigation paths and gradually execute tasks; the interaction and feedback module is used to execute step S8 of the method in Example 1, aiming to support voice interaction during task execution and broadcast the task status.

[0055] (1) Implementation of multimodal perception module

[0056] By using a variety of sensors to capture Figure 2 The perceptual information of the real home environment is arranged, and the type of environmental information to be used by the parameter call, such as object-level visual detection information from the camera and pixel-level spatial position relationship information.

[0057] (1.1) RGB-D camera data processing: Image data containing RGB and depth information is collected. The RGB image undergoes image correction, and the depth information undergoes camera correction and noise processing. The hand-eye relationship is calibrated to complete the perspective conversion from the camera coordinate system to the robot coordinate system, obtain pixel-level spatial relationship information, and then complete the acquisition of the robot arm's grasping position and the three-dimensional coordinate solution of the camera operation target.

[0058] (1.2) Audio input processing: Get user voice commands through the microphone, reduce audio noise, perform silent recognition, identify the command release status, and when it is recognized that the command is being released, start recording and get the recorded audio commands. Use the speech-to-text method (implemented by the automatic speech recognition model) to convert the human audio modality commands into text commands that can be understood by the large language model;

[0059] (1.3) LiDAR and SLAM mapping: LiDAR is used to detect environmental obstacle information and form the robot's perception of the environmental accessibility, which includes both static and dynamic content. For generating static environmental obstacle information such as room location walls, the global map and navigation nodes of the scene are constructed in advance through SLAM technology. For dynamic obstacles such as pedestrians, movable trash cans, seats, etc., the real-time scanning results of LiDAR are used to process the accessible information of the obstacles.

[0060] (2) Implementation of instruction parsing module

[0061] The implementation of the instruction parsing module is as follows Figure 3 As shown, by designing background prompt words containing environmental information and executable operations of the robot, the large language model is guided to process the text instructions, decompose them according to the scene context, and generate a meta-action code sequence containing actions, goals, and details.

[0062] The use environment information includes: important environmental features that may appear in the scene, and the task objectives that the large model needs to complete. The operation information includes: the action interface that the robot can implement, and the environmental information that the robot can obtain

[0063] a) Partial display of environmental prompt words:

[0064] You are a robot in house. Your task is to analyze instructions from user and disassemble them to a list sequence of executable triples.

[0065] You can see objects{self.objs}in front of you.You are in{self.room}now.

[0066] b) The prompt words that can set the robot's meta-action parameters are shown as follows:

[0067] You have the following actions in meta_actions_tuple that you can use to achieve this task:

[0068] {meta_actions_tuple},

[0069] Positions containing "lambda" describe replaceable keywords based on instructions.

[0070] c) Provide few-shot prompts for the tasks that the robot needs to complete:

[0071] #Examples:

[0072] Example 1

[0073] command:Could you please bring me the left most object from the sink

[0074] Action list:[MetaAction("move","sink",None),MetaAction("findobject","object","leftmost"),MetaAction("take","object","leftmost"),MetaAction("move","return",None),MetaAction("give","object","me")]

[0075] Example 2

[0076] command:Tell me what's the biggest cleaning stuff on the sink

[0077] action_list[("move","sink",None),("findobject","cleaning stuff","biggest"),("say","The biggest cleaning stuff is","biggest"),("move","return",None)]

[0078] In response to the different types of action tasks that may appear in the tasks, the large model is prompted to split the basic actions into a combination of multiple meta-actions, and different types of data need to be called accordingly. For example, following someone requires calling the two meta-actions of "recognition" and "movement", grabbing an object requires calling the two meta-actions of "recognition" and "grab", and conversation requires calling the two meta-actions of "listening" and "speaking".

[0079] Action types include: observation, grasping, navigation, etc.;

[0080] The target is an object, place, or person;

[0081] The details include specific implementation methods and parameter settings.

[0082] According to the common home environment command categories, six basic meta-actions, namely "listen", "speak", "move", "detect", "clamp" and "relax", are designed respectively. Based on these six basic actions, complex action requirements for other common home scenes are combined: "search", "follow", "grasp", "take", "identify", "collect", "pick up", "place", "talk", "judge status", etc. Different types of operation targets and different levels of detail parameters are designed. For example, the target types of the complex action "identification" can be divided into identifying objects and people, and detailed parameters such as features such as color, size, number, name, etc.

[0083] Table 1 shows the currently available meta-action parameters.

[0084] Table 1 Currently available meta-action parameters

[0085]

[0086]

[0087] Based on the above design, for long commands, such as "Please find Xiao Ming in the bedroom and bring him two apples from the living room", a multilingual text-to-speech sub-module is implemented in actual scenarios, and the input command language is not limited.

[0088] First, the large model needs to understand the correct order of actions implied by the instructions and the two tasks to be accomplished:

[0089] First: find two apples in the living room and bring them to Xiao Ming;

[0090] Second: Secondly, the large model splits the action goals of the two non-meta-actions into a combination of meta-action codes that can be implemented by the robot:

[0091] [("move","location","living_room"),("find","apple",None),("grasp","apple",None),("find","apple",None),("grasp","apple",None),("move","location","bedroom"),("find","person","Xiao Ming"),("take","apple",2)]

[0092] The main reason why the big model can complete complex thinking with prerequisites is to realize CoT and instruct the big model to generate the displayed thinking process first and then generate the target result. The following is a partial display of the prompt words for multi-step thinking:

[0093] #Output Response Json Format:

[0094] "command":original task instruction,

[0095] "reasoning":justify why the next action is important to solve thetask,

[0096] "action_list":a list of tuple_list choose from meta_actions_tuple

[0097] Here is the command you are going to parse:{command}.

[0098] now think step by step!

[0099] Finally, the generated meta-action triplet parameter sequence is initialized and passed into the meta-action class MetaAction as execution parameters to be mapped to generate executable code.

[0100] (3) Implementation of navigation and execution modules

[0101] The navigation and execution module is the core part of the indoor mobile service robot to complete the task. It is responsible for parsing the action operation code to be executed from the task instruction and generating the actual robot behavior. The implementation of this module mainly relies on target detection, path planning, task execution and real-time feedback technologies. The specific implementation details of the five basic meta-actions are as follows:

[0102] (3.1) “Detection”:

[0103] Object detection is the first step of the navigation and execution module, which involves accurately identifying the objects that the robot needs to operate from the environment, such as objects, people, or places. In this system, the object detection that processes object category information uses the YOLOv7 algorithm, which performs well in speed and accuracy and is suitable for real-time processing of complex indoor environment data in robot edge computing scenarios.

[0104] The YOLOv7 algorithm uses a convolutional neural network (CNN) to detect and classify objects, which can efficiently identify objects and calibrate their positions in images. At the same time, YOLOv7 can also generate a customized object detection model through training, supporting the adaptation of object changes in dynamic environments.

[0105] (3.2) “Mobile”:

[0106] After receiving the instruction, the robot usually needs to reach the target location in the scene before it can perform subsequent operations. Therefore, the robot needs to plan the path to ensure that it can reach the target location efficiently and safely and perform the specified task. The path planning uses the SLAM method to build and update the environment map.

[0107] SLAM map construction and optimization Through real-time perception of the environment by LiDAR and RGB-D cameras, the robot can build a global environment map and perform self-localization at the same time. In this process, the SLAM method can handle obstacles and unknown environmental changes in a dynamic environment, ensuring the stability and accuracy of path planning.

[0108] Path planning algorithm: This system uses a path planning method based on A* or Dijkstra algorithm to optimize the target path, reduce detours around obstacles, and ensure the balance between the shortest path and the safe path. The robot decomposes the action into multiple small navigation tasks (such as "move to a certain location", "bypass obstacles", "locate to a specified location") according to the parsed instructions, and makes dynamic adjustments based on real-time environmental data.

[0109] (3.3) “Grab” and “Place”:

[0110] In the task, if it involves grasping and placing objects, the robot needs to perform precise operations to ensure that the objects are grasped and placed correctly. In order to accurately obtain the position of the target in three-dimensional space, the end effector is used to operate the target object. It is necessary to first obtain the space transformation matrix from the robot camera coordinates to the robot's end effector through the three-dimensional hand-eye calibration of the robot arm, and then further complete the mapping from the camera pixel point to the robot joint position coordinates. The specific steps are as follows:

[0111] (3.3.1) Object detection and 3D coordinate acquisition: The target object is obtained through the item-level return information of the "detection" meta-action, and its pixel position in the image is calculated. Next, the 3D coordinates (X, Y, Z) of the object are obtained through binocular vision technology.

[0112] (3.3.2) Coordinate transformation: The three-dimensional coordinates of the object are transformed from the camera coordinate system to the robot coordinate system using the transformation matrix obtained by hand-eye calibration. This transformation process involves the calculation of the rotation matrix and translation vector to ensure that the spatial position of the target object can be accurately mapped to the control space of the robot joint.

[0113] After obtaining the position of the target object in the robot coordinate system, the next step is to use the inverse kinematics algorithm to calculate the position of the end effector of the robot arm (the coordinates of the grasping point), and reversely solve the angles of each joint (joint space coordinates) through the robot's kinematic model. According to the working range and structure of the robot arm, the system can plan the optimal grasping path and complete precise grasping and placement actions.

[0114] (3.4) “Listening” and “Speaking” (Voice Interaction):

[0115] In order to achieve more humane interaction, the robot also has voice input and output capabilities, and can have natural voice conversations with users.

[0116] (3.4.1) ASR submodule: The microphone collects the user's voice commands, and ASR can convert them into text. The system uses a deep learning model to reduce noise and process the voice to improve recognition accuracy, especially in noisy environments.

[0117] (3.4.2) TTS (Text-to-Speech) submodule: During the task execution, the robot can use the TTS method to feedback the execution progress and status to the user in real time in the form of voice broadcast. For example, "The task has been completed, please give the item to Xiao Ming."

[0118] (4) Implementation of interaction and feedback modules

[0119] During execution, the robot will dynamically adjust the task status based on sensor information and user feedback to ensure that the task proceeds as expected.

[0120] (4.1) Real-time environmental perception and feedback: The robot will continuously monitor the surrounding environment and obtain real-time information such as obstacles and changes in the position of objects. If there are environmental changes during the execution of the task (such as the appearance of new obstacles or the displacement of the target position), the robot will immediately adjust the path and execution strategy.

[0121] (4.2) Task progress feedback: The robot will provide the user with task execution progress feedback based on the execution status of each meta-action. For example, the robot will inform the user that the robot is currently in the "target identification" or "path planning" stage to ensure that the user is aware of the progress of the task.

[0122] (4.3) Failure and attempt: When a meta-action fails to execute through the above processing, a noise action is added to the task sequence, that is, moving on the spot and adjusting the posture and then performing the common sense of the task action sequence. When the task is successfully executed, the meta-action that has been successfully executed is kicked out from the task execution sequence.

[0123] Figure 4shows one of the results when the task performs the detection meta-action; Figure 5 The robot is shown performing a portion of the tasks.

[0124] In summary, the present invention significantly improves the task understanding, execution efficiency and adaptability of indoor mobile service robots through the combination of multimodal perception, dynamic navigation and execution, intelligent command parsing of a large language model, and real-time feedback mechanism, and can provide solid methodological support for the practical application of robots in dynamic and complex environments.

[0125] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for executing interactive tasks of an indoor mobile service robot driven by a large model, characterized in that: The following steps are involved: The original visual data and the original spatial data are obtained and processed respectively, the original audio signal is obtained and converted into a text signal, and a dynamic interactive input instruction is obtained; Parsing the dynamic interactive input command using a preset large language model to obtain a meta-action sequence corresponding to the task to be processed; According to the meta-action sequence, each meta-action in the task to be processed is executed in sequence, and each meta-action schedules the corresponding processed visual data and spatial data when executed; When the execution result of the meta-action is confirmed, the voice broadcasts the current task execution status to complete the task interaction.

2. The method for executing interactive tasks of an indoor mobile service robot driven by a large model according to claim 1, characterized in that: Obtaining raw visual data and processing it. The specific process includes: Use RGB_D camera to obtain depth and color visual information of dynamic environment, complete image correction and transformation, and obtain depth image and RGB image; Processing the RGB image using a YOLOv7 target detector to obtain object categories, object details, and pedestrian postures; A binocular vision estimation method is used to obtain depth information from the depth image and calculate the three-dimensional spatial coordinates of the target operation position.

3. The method for executing interactive tasks of an indoor mobile service robot driven by a large model according to claim 1, characterized in that: Obtain the original spatial data and process it. The specific process includes: Construct an interference-free static scene model, use lidar and SLAM methods to extract static obstacle information and robot positioning, and construct a grid obstacle map.

4. The method for executing interactive tasks of an indoor mobile service robot driven by a large model according to claim 1, characterized in that: The original audio signal is obtained and converted into a text signal. The specific process includes: Use the preset microphone device to obtain audio signals; The audio signal is converted into a text signal using an automatic speech recognition method.

5. The method for executing interactive tasks of an indoor mobile service robot driven by a large model according to claim 4, characterized in that: The automatic speech recognition method is implemented based on a deep learning model.

6. The method for executing interactive tasks of an indoor mobile service robot driven by a large model according to claim 1, characterized in that: The meta-action sequence includes multiple meta-actions, each meta-action includes an action type, an implementation object and implementation details. The action type is classified according to the robot's sensors and operating capabilities, the implementation object is determined according to actual task requirements, and the implementation details are used to describe the specific execution method of the task.

7. The method for executing interactive tasks of an indoor mobile service robot driven by a large model according to claim 1, characterized in that: When the execution result of the meta-action is confirmed, the voice broadcasts the current task execution status. The specific process includes: Real-time monitoring and data collection of the actual execution process of the current meta-action; Determine whether the current meta-action is completed or has deviations based on the preset task completion indicators; Use the preset large language model to analyze and summarize the data collected during the execution and the task completion status, and generate corresponding interactive feedback results; The feedback result is converted into natural language using a text-to-speech method and then voiced.

8. A large model driven indoor mobile service robot interactive task execution device, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

9. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 7 is implemented.

10. An indoor mobile service robot system, characterized in that: The system includes a multimodal perception module, a command parsing module, a navigation and execution module, and an interaction and feedback module: The multimodal perception module is used to obtain original visual data and original spatial data and process them respectively, obtain original audio signals and convert them into text signals, and obtain dynamic interactive input instructions; The instruction parsing module is used to parse the dynamic interactive input instruction using a preset large language model to obtain a meta-action sequence corresponding to the task to be processed; The navigation and execution module is used to execute each meta-action in the task to be processed in sequence according to the meta-action sequence, and each meta-action schedules the corresponding processed visual data and spatial data when executing; The interaction and feedback module is used to voice broadcast the current task execution status after the execution result of the meta-action is confirmed, thereby completing the task interaction.

Images