A control method, system, robot and medium for embodied intelligent robot

Abstract

This invention provides a control method, system, robot, and medium for an embodied intelligent robot. The control method includes: acquiring interaction commands after the embodied intelligent robot enters an activated state; determining whether the embodied intelligent robot has entered a task planning phase based on the interaction commands; acquiring a tool list if the embodied intelligent robot has entered the task planning phase; performing task planning on the embodied intelligent robot based on the interaction commands and the tool list to obtain a feasible action sequence; and having the embodied intelligent robot execute the interaction commands according to the action sequence. This invention solves the technical problem that embodied intelligent robots sometimes fail to effectively utilize existing tools in the environment, thus making originally feasible tasks unplannable.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a control method for an embodied intelligent robot triggered based on a large model, a control system for an embodied intelligent robot triggered based on a large model, an embodied intelligent robot and a readable storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, especially the breakthroughs in the field of deep learning, large models (such as the GPT series, BERT, etc.) have received widespread attention due to their powerful language understanding and generation capabilities. Among them, embodied intelligence (Embodied AI) is considered to be the next wave of artificial intelligence development, both a new technological breakthrough and a new economic growth point. Embodied intelligence generally refers to the ability to perceive and recognize in the physical world, interact with the environment, and achieve autonomous learning and decision-making. In particular, when it is able to perceive information in multiple modalities (such as vision, text, voice, etc.), it can be called multimodal embodied intelligence.

[0003] However, in actual use, there is a problem: the embodied intelligent robot does not effectively utilize the existing tools in the environment, causing the originally feasible tasks to become unplannable. Summary of the invention

[0004] The present invention solves the technical problem that an embodied intelligent robot fails to effectively utilize existing tools in the environment, thereby causing an originally feasible task to become unplannable.

[0005] To solve the above problems, the present invention provides a control method for an embodied intelligent robot triggered based on a large model, comprising: obtaining interaction instructions after the embodied intelligent robot enters an activated state; judging whether the embodied intelligent robot enters a task planning stage according to the interaction instructions; obtaining a tool list when it is judged that the embodied intelligent robot enters the task planning stage; performing task planning for the embodied intelligent robot according to the interaction instructions and the tool list to obtain a feasible action sequence; and the embodied intelligent robot executes the interaction instructions according to the action sequence.

[0006] Compared with the existing technology, the technical effects achieved by adopting this technical solution are as follows: when the embodied intelligent robot enters the task planning stage, a tool list is obtained; the embodied intelligent robot is task planned according to the interactive instructions and the tool list to obtain a feasible action sequence, so that the embodied intelligent robot can make full use of the tools in the current environment for task planning, making the success rate of task planning higher and increasing the planning success rate; the use of environmental tools can enhance the reasoning success rate, task completion capability and environmental alignment level of the embodied intelligent robot.

[0007] In one example of the present invention, the interaction instructions include: executable task instructions and non-executable task instructions; based on the interaction instructions, determining whether the embodied intelligent robot enters the task planning stage, including: when the interaction instructions are executable task instructions, determining that the embodied intelligent robot enters the task planning stage; and / or when the interaction instructions are non-executable task instructions, determining that the embodied intelligent robot does not enter the task planning stage, and the embodied intelligent robot ends the interaction after responding to the non-executable task instruction.

[0008] Compared with the prior art, the technical effect achieved by adopting this technical solution is as follows: after the embodied intelligent robot enters the activated state, it will accept the user's interactive instructions. If the planning module large model does not receive an executable task instruction, the interaction is terminated, and then the interactive instruction is obtained after the embodied intelligent robot enters the activated state; if the planning module large model receives an executable task instruction and is triggered, the tool list is obtained when it is judged that the embodied intelligent robot enters the task planning stage. Whether to enter the task planning stage is directly determined by the type of interactive instruction, which saves the overall control time.

[0009] In one example of the present invention, when it is determined that the embodied intelligent robot has entered the task planning stage, a tool list is obtained, including: initializing the tool list; detecting whether there are objects related to the interactive instructions in the environment; if there are objects, using the objects as available tools and adding the available tools to the tool list.

[0010] Compared with the existing technology, the technical effect achieved by adopting this technical solution is as follows: when the embodied intelligent robot enters the task planning stage, the embodied intelligent robot rotates 360 degrees to observe the surrounding environment, finds and recognizes tools (that is, objects) that are highly relevant to the current task (that is, interactive instructions), and inputs these tools and their attributes as prompts (that is, prompts), together with given tasks, completed execution actions, etc., into the planning module large model, which infers the feasible action sequence of the task, thereby aligning the task planning with the current physical world.

[0011] In one example of the present invention, task planning is performed for the embodied intelligent robot based on interaction instructions and a tool list to obtain a feasible action sequence, including: performing task planning for the embodied intelligent robot based on interaction instructions and available tools; determining whether there is a feasible plan in the task planning process; if so, obtaining an action sequence based on the feasible plan; and / or if not, updating the interaction instructions.

[0012] Compared with the prior art, the technical effect achieved by adopting this technical solution is as follows: these available tools and their attributes are used as prompts, together with the given task, completed execution actions, etc., and the planning module large model infers a feasible action sequence for the task: if the planning module returns a feasible action sequence, the action sequence is sent to the execution module; if the planning module cannot plan a feasible action sequence based on the current information (that is, the available tools and their attributes) (that is, there is no feasible plan in the task planning process), it returns to the parent step to re-perform the task interaction; if the planning module infers that the given task has been completed based on the current information, it returns to the parent step to identify new interaction instructions.

[0013] In one example of the present invention, after the embodied intelligent robot enters an activated state and before obtaining interactive instructions, the control method also includes: obtaining environmental perception data when the embodied intelligent robot is in a standby state; judging whether the perception module large model is triggered based on the environmental perception data; when the perception module large model is triggered, the embodied intelligent robot enters an activated state from the standby state; wherein the environmental perception data includes: voice data, image data.

[0014] Compared with the existing technology, the technical effect achieved by adopting this technical solution is as follows: the embodied intelligent robot enters the activated state only after the wake-up condition of the large model of the perception module is triggered, thereby achieving the effect of all-weather standby and reducing energy consumption. At the same time, it makes full use of the obtained environmental perception data for interaction and decision-making wake-up, thereby improving the interactive intelligence of the embodied intelligent robot.

[0015] In one example of the present invention, the embodied intelligent robot executes an interactive instruction according to an action sequence, including: the embodied intelligent robot executes a first action in the action sequence, and detects whether a risk condition is triggered during the execution of the first action.

[0016] Compared with the prior art, the technical effect achieved by adopting this technical solution is as follows: after obtaining the action sequence, the present application refers to the idea of ​​rolling optimization in model predictive control, and only executes the first action in the action sequence. After executing the first action obtained by planning, the embodied intelligent robot will return to the planning module to re-reason and plan a new feasible action sequence based on the interaction result of the first action in the environment, thereby realizing closed-loop control and increasing the robustness of the control system. While executing the first action, the execution module large model is also used to monitor the environment in real time, so that potential safety risks can be perceived in time, and autonomous emergency braking can be performed in time to terminate the current first action. Potential safety risks can be avoided through the autonomous emergency braking mechanism, thereby improving the safety performance of the embodied intelligent robot.

[0017] In one example of the present invention, after the embodied intelligent robot executes the first action in the action sequence and detects whether a risk condition is triggered during the execution of the first action, the control method also includes: stopping the first action if the risk condition is triggered; and / or executing and completing the first action if the risk condition is not triggered.

[0018] Compared with the prior art, the technical effect achieved by adopting this technical solution is as follows: while executing the first action, the execution module large model is also used to monitor the environment in real time: if the execution module large model triggers a risk condition, emergency braking is immediately performed to stop the current first action, and then the interaction instruction is obtained, requesting the user to update the task (that is, re-obtaining the interaction instruction); if the current first action is successfully completed, task planning of the embodied intelligent robot is performed according to the interaction instruction and the tool list to obtain a feasible action sequence, and the task is re-planned based on the updated information; the present application can perceive potential safety risks in a timely manner, and terminate the current first action by autonomous emergency braking in a timely manner.

[0019] On the other hand, an embodiment of the present invention also provides a control system of an embodied intelligent robot triggered by a large model, which is used to execute the control method of an embodied intelligent robot triggered by a large model as in any of the above embodiments, including: a first acquisition module, the first acquisition module is used to acquire interaction instructions after the embodied intelligent robot enters an activated state; a judgment module, the judgment module is used to judge whether the embodied intelligent robot enters the task planning stage according to the interaction instructions; a second acquisition module, the second acquisition module is used to acquire a tool list when it is judged that the embodied intelligent robot enters the task planning stage; a planning module, the planning module is used to perform task planning for the embodied intelligent robot according to the interaction instructions and the tool list to obtain a feasible action sequence; an execution module, the execution module is used for the embodied intelligent robot to execute the interaction instructions according to the action sequence.

[0020] Compared with the prior art, the technical effect achieved by adopting this technical solution is as follows: the control system of the embodied intelligent robot triggered based on a large model in this embodiment is used to implement the control method of the embodied intelligent robot triggered based on a large model as in any embodiment of the present invention, and therefore it has all the beneficial effects of the control method of the embodied intelligent robot triggered based on a large model as in any embodiment of the present invention, which will not be repeated here.

[0021] On the other hand, an embodiment of the present invention further provides an embodied intelligent robot, which includes: a processor, a memory, and a program or instruction stored in the memory and executable on the processor, which, when executed by the processor, implements the steps of the control method of the embodied intelligent robot based on large model triggering as in any of the above embodiments.

[0022] Compared with the prior art, the technical effect achieved by adopting this technical solution is as follows: the embodied intelligent robot in this embodiment runs the control method of the embodied intelligent robot based on large model triggering as in any embodiment of the present invention, so it has all the beneficial effects of the control method of the embodied intelligent robot based on large model triggering as in any embodiment of the present invention, which will not be repeated here.

[0023] On the other hand, an embodiment of the present invention further provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the control method of an embodied intelligent robot triggered by a large model as in any of the above embodiments are implemented.

[0024] Compared with the prior art, the technical effect achieved by adopting this technical solution is as follows: the readable storage medium in this embodiment is used to store the control method of an embodied intelligent robot triggered based on a large model as in any embodiment of the present invention, so it has all the beneficial effects of the control method of an embodied intelligent robot triggered based on a large model as in any embodiment of the present invention, which will not be repeated here.

[0025] After adopting the technical solution of the present invention, the following technical effects can be achieved: (1) Enable the embodied intelligent robot to make full use of the tools in the current environment to plan tasks, so that the success rate of task planning is higher and the planning success rate is increased; the use of environmental tools can enhance the reasoning success rate, task completion ability and environmental alignment level of the embodied intelligent robot; (2) Whether to enter the task planning phase is directly determined by the type of interactive instructions, saving overall control time; (3) The embodied intelligent robot enters the activated state only after the large model of the perception module is triggered to wake up, thereby achieving the effect of 24-hour standby and reducing energy consumption. At the same time, it makes full use of the acquired environmental perception data for interaction and decision-making wake-up, thereby improving the interactive intelligence of the embodied intelligent robot. (4) After obtaining the action sequence, the present application refers to the idea of ​​rolling optimization in model predictive control and only executes the first action in the action sequence. After executing the first action obtained by planning, the embodied intelligent robot will return to the planning module to re-reason and plan a new feasible action sequence based on the interaction result of the first action in the environment, thereby realizing closed-loop control and increasing the robustness of the control system; (5) While executing the first action, the execution module large model is used to monitor the environment in real time, so that potential safety risks can be perceived in time, and autonomous emergency braking can be performed in time to terminate the current first action. Potential safety risks can be avoided through the autonomous emergency braking mechanism, thereby improving the safety performance of the embodied intelligent robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flow chart of a control method of an embodied intelligent robot based on large model triggering provided in Embodiment 1 of the present invention; Figure 2 A detailed flow chart of a control method of an embodied intelligent robot based on large model triggering provided in Embodiment 1 of the present invention; Figure 3 This is a detailed flow chart of voice / visual awakening of a multimodal large model in Embodiment 1 of the present invention; Figure 4 for Figure 1 Detailed flow chart of step S300; Figure 5 for Figure 1 Detailed flow chart of step S500; Figure 6 A schematic block diagram of the structure of a control system of an embodied intelligent robot triggered by a large model provided in the second embodiment of the present invention; Figure 7 A block diagram of a composition of an embodied intelligent robot provided in Embodiment 3 of the present invention; Figure 8 A schematic diagram of the structure of a readable storage medium provided in Embodiment 4 of the present invention.

[0027] Description of reference numerals: 100-control system of embodied intelligent robot triggered by large model; 101-first acquisition module; 102-judgment module; 103-second acquisition module; 104-planning module; 105-execution module; 200-embodied intelligent robot; 210-memory; 211-computer program; 220-processor; 300-readable storage medium; 310-computer executable instructions. DETAILED DESCRIPTION

[0028] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] [Example 1] See also Figure 1 , which is a flow chart of a control method of an embodied intelligent robot based on large model triggering provided by the first embodiment of the present invention, combined with Figure 2-Figure 5 , the control method of the embodied intelligent robot triggered by the large model includes: Step S100: After the embodied intelligent robot enters an activated state, an interaction instruction is obtained; Step S200: judging whether the embodied intelligent robot has entered the task planning stage according to the interactive instruction; Step S300: when it is determined that the embodied intelligent robot has entered the task planning phase, obtaining a tool list; Step S400: performing task planning for the embodied intelligent robot according to the interactive instructions and the tool list to obtain a feasible action sequence; Step S500: The embodied intelligent robot executes the interaction instruction according to the action sequence.

[0030] In a specific embodiment, in the prior art, the planning stage lacks an activation mechanism based on scene tool cognition: although the large model has the ability to think in chains to plan tasks, it fails to fully utilize the tools available in the scene to assist reasoning and decision-making. Therefore, the embodied intelligent robot may fail to effectively utilize the existing tools in the environment, making it impossible to plan tasks that were originally feasible.

[0031] Specifically, this application uses a humanoid robot as an embodied intelligence carrier and conditional triggering of multiple large models as a guarantee for embodied intelligence to achieve real-time, safe and reliable interaction with the environment, providing a new control method for embodied intelligent robots. When the embodied intelligent robot enters the task planning stage, a tool list is obtained; the embodied intelligent robot is task planned according to the interactive instructions and the tool list to obtain a feasible action sequence, so that the embodied intelligent robot can make full use of the tools in the current environment for task planning, making the success rate of task planning higher and increasing the planning success rate; the use of environmental tools can enhance the reasoning success rate, task completion ability and environmental alignment level of the embodied intelligent robot.

[0032] Furthermore, the interactive instructions include: executable task instructions and non-executable task instructions; step S200 includes: Step S210: when the interactive instruction is an executable task instruction, determining that the embodied intelligent robot enters the task planning stage; and / or Step S220: When the interaction instruction is a non-executable task instruction, it is determined that the embodied intelligent robot does not enter the task planning stage, and the embodied intelligent robot responds to the non-executable task instruction and ends the interaction.

[0033] Specifically, after the embodied intelligent robot enters the activated state, it will accept the user's interactive instructions. If the planning module model does not receive an executable task instruction, the interaction ends and then executes step S100; if the planning module model receives an executable task instruction and is triggered, step S300 is executed. Whether to enter the task planning stage is directly determined by the type of interactive instruction, which saves the overall control time.

[0034] When the embodied intelligent robot enters the activated state, it will respond to the user's real-time dialogue interaction. If the content of the interaction is a daily conversation of non-executable tasks (that is, when the interaction instruction is a non-executable task instruction), such as asking "Please introduce the intelligent robot", the embodied intelligent robot calls the open source large model "Qwen2.5" to answer, and returns to the standby state after the interaction ends.

[0035] For example, if the content of the user interaction is to give a specific executable task (that is, when the interactive instruction is an executable task instruction), such as "Please help me clean up the paper scraps on the ground", the embodied intelligent robot enters step S300 and uses the planning module to perform complex task decomposition reasoning and planning.

[0036] For further information, see Figure 4 , step S300 includes: Step S310: Initialize tool list; Step S320: Detect whether there is an object related to the interaction instruction in the environment; Step S330: If the object exists, the object is used as an available tool and the available tool is added to the tool list.

[0037] Specifically, when the embodied intelligent robot enters the task planning stage, the embodied intelligent robot rotates 360 degrees to observe the surrounding environment, finds and recognizes tools (that is, objects) that are highly relevant to the current task (which can also be interactive instructions), and inputs these tools and their attributes as prompts (that is, prompts), together with the given task, completed execution actions, etc., to the planning module large model, which infers the feasible action sequence of the task, so that the task planning is aligned with the current physical world. If there is no feasible plan (that is, a feasible action sequence), step S200 is executed to request the user to update the task; if a feasible plan is obtained, step S400 is executed; if the task has been completed, step S200 is executed, and the embodied intelligent robot returns to the interactive waiting stage.

[0038] For example, when the given task is "please help me clean up the paper scraps on the floor", the embodied intelligent robot will first observe the environment, trigger the corresponding large model to identify objects that can be used for cleaning, and activate these objects as usable tools; when the given task is "wipe the table", if there is a "rag" in the environment, it will be activated as a usable tool. The large model here is a visual language large model that has been fine-tuned based on common tools.

[0039] If no object related to the task is identified, the tool list is an empty list tool_list = []; if an object related to the task is identified, the activated tool list is in the form of tool_list = [{"Tool 1": "Broom", "Attribute": "Can be used for sweeping the floor"}, {"Tool 2": "Dustpan", "Attribute": "Can be used to store small amounts of garbage"}].

[0040] The prompt for inputting the planning module large model for planning reasoning has the following form: "You are an embodied intelligent robot. Given a task {task}, a known tool list {tool_list}, and the following actions {actions} have been executed, please give a plan to complete the task {task}. If the task has been completed, output <Task completed>. Now start planning:" Among them, {·} is a placeholder, {task} is the interactive task given by the user (that is, the interactive instruction), {tool_list} is the tool list identified in the environment, {actions} is the action executed in step S500, and the initial action is an empty list.

[0041] Furthermore, step S400 includes: Step S410: performing task planning for the embodied intelligent robot according to the interactive instructions and available tools; Step S420: determining whether there is a feasible plan in the task planning process; Step S421: If yes, obtain an action sequence according to the feasible plan; and / or Step S422: If not, update the interaction instruction.

[0042] Specifically, these available tools and their attributes are used as prompts, together with the given task, completed execution actions, etc., and input into the planning module large model, and the planning module large model infers a feasible action sequence for the task; if the planning module returns a feasible action sequence, the action sequence is sent to the execution module; if the planning module cannot plan a feasible action sequence based on the current information (that is, the available tools and their attributes) (that is, there is no feasible plan in the task planning process), it returns to step S200 to re-perform task interaction; if the planning module infers that the given task has been completed based on the current information, it returns to step S200 to identify new task interactions.

[0043] For further information, see Figure 3 , before step S100, the control method further includes: When the embodied intelligent robot is in a standby state, obtaining environmental perception data; Determine whether the perception module large model is triggered based on the environmental perception data; When the perception module large model is triggered, the embodied intelligent robot enters the active state from the standby state; Among them, environmental perception data includes: voice data and image data.

[0044] Specifically, in the prior art, the perception module large model lacks an awakening mechanism based on scene interaction understanding: although the perception module large model can fully obtain multimodal information such as voice and images from the physical environment, it fails to make full use of the acquired perception information (that is, environmental perception data) for interaction and decision-making awakening, so the interactive intelligence effect is not good.

[0045] The embodied intelligent robot in the present application is usually in a standby state, and uses the perception module large model to continuously perceive the environment, generate environmental perception data based on the current physical environment and perform real-time processing, and judge whether the perception module large model is triggered according to the environmental perception data; only after the perception module large model is triggered to wake up the condition, the embodied intelligent robot enters the activation state, thereby achieving the effect of all-weather standby and reducing energy consumption, and at the same time making full use of the obtained environmental perception data for interaction and decision-making wake-up, improving the interactive intelligence of the embodied intelligent robot. If the perception module large model is triggered, step S100 is executed, the embodied intelligent robot is activated, and starts to actively interact with the user, requesting and waiting for the user to issue an executable task instruction; if the perception module large model is not triggered, it is repeated to obtain environmental perception data when the embodied intelligent robot is in a standby state, and judge whether the perception module large model is triggered according to the environmental perception data. Among them, the wake-up conditions at least include: voice wake-up and visual wake-up, both of which are triggered by the perception module large model; the perception module large model includes multiple multimodal large models, at least including: voice large model and visual large model; environmental perception data at least includes: voice data and image data.

[0046] Voice wake-up is based on the open source voice model such as "xiaoyun" to fine-tune the LoRA technology for the specific wake-up word "Joyson Robot". Visual wake-up is based on the open source visual model "yolo-world" to fine-tune the LoRA technology for the specific wake-up label "eyes thatare watching you".

[0047] The voice data is used to determine whether to trigger voice wake-up. The voice wake-up is set by fine-tuning the voice big model to set a custom voice wake-up word, such as "intelligent robot". When the voice data in the environmental perception data is recognized to contain "intelligent robot", the voice big model triggers voice wake-up and controls the embodied intelligent robot to enter the activated state. The image data is used to determine whether to trigger visual wake-up. The visual wake-up is set by fine-tuning the visual big model to set a custom visual wake-up label, such as "a person who looks at you". When the image data in the environmental perception data is recognized to contain "a person who looks at you", the visual big model triggers visual wake-up and controls the embodied intelligent robot to enter the activated state.

[0048] Preferably, the wake-up conditions correspond to the multimodal large model and the environmental perception data one by one, and if one of the wake-up conditions is met, the embodied intelligent robot can be activated, increasing the activation success rate. The two wake-up conditions do not need to be met at the same time. You can first determine whether the voice wake-up is triggered. If the voice wake-up is triggered, the embodied intelligent robot is directly activated to start real-time interaction; if the voice wake-up is not triggered, then determine whether the visual wake-up is triggered. If the visual wake-up is triggered, the embodied intelligent robot is activated to start real-time interaction; otherwise (that is, neither the voice wake-up nor the visual wake-up is triggered), continue to perform environmental perception and data processing, and re-monitor whether the corresponding multimodal large model is triggered. When the embodied intelligent robot is in standby mode, it is always in the loop detection waiting wake-up mode. The standby wake-up mode can reduce the energy consumption of embodied intelligent robots such as humanoid robots.

[0049] For further information, see Figure 5 , step S500 includes: Step S510: The embodied intelligent robot executes the first action in the action sequence, and detects whether a risk condition is triggered during the execution of the first action.

[0050] Specifically, in the prior art, the execution phase after task planning lacks an autonomous emergency braking mechanism based on scenario risk perception: although the execution module can smoothly execute the subtasks and action sequences generated by the planning module, it fails to fully consider the changes in the environment since the planning phase, especially when unexpected risks occur in the environment (for example, the current action will touch a person who suddenly appears). The embodied intelligent robot may cause potential safety risks because the execution logic does not have an autonomous emergency braking mechanism.

[0051] After obtaining the action sequence, the present application refers to the idea of ​​rolling optimization in model predictive control and only executes the first action in the action sequence. After executing the first action obtained by planning, the embodied intelligent robot will return to the planning module to re-reason and plan a new feasible action sequence based on the interaction result of the first action in the environment, thereby realizing closed-loop control and increasing the robustness of the control system. While executing the first action, the execution module large model is also used to monitor the environment in real time, so that potential safety risks can be perceived in time, and autonomous emergency braking can be performed in time to terminate the current first action. Potential safety risks can be avoided through the autonomous emergency braking mechanism, thereby improving the safety performance of the embodied intelligent robot.

[0052] This emergency brake is also triggered by the big model, which is achieved by fine-tuning the visual big model to set relevant risk brake labels, such as adding the label "the robotic arm is about to touch a person"; when the perception module big model recognizes the label "the robotic arm is about to touch a person", the risk condition is triggered. For example, the execution module big model will analyze in real time whether the robotic arm of the embodied intelligent robot will touch a person. Once the probability of the bounding box corresponding to the label "the robotic arm touches a person" is greater than the set threshold, it will automatically trigger the emergency brake and trigger the risk condition, thereby improving the safety factor of the embodied intelligent robot. Among them, the big model here is a visual language big model that has been fine-tuned according to common risk warning labels.

[0053] Furthermore, step S510 includes: Step S511: When the risk condition is triggered, stop the first action; and / or Step S512: When the risk condition is not triggered, execute and complete the first action.

[0054] Specifically, while executing the first action, the execution module large model is also used to monitor the environment in real time: if the execution module large model triggers a risk condition, emergency braking is immediately performed to stop the current first action, and then step S100 is executed to request the user to update the task (that is, to re-acquire the interaction instruction); if the current first action is successfully completed, step S400 is executed to re-plan the task based on the updated information; potential safety risks can be perceived in a timely manner, and emergency braking can be performed autonomously in a timely manner to terminate the current first action.

[0055] [Example 2] See also Figure 6The present embodiment also provides a control system 100 of an embodied intelligent robot triggered based on a large model, comprising: a first acquisition module 101, the first acquisition module 101 is used to acquire interaction instructions after the embodied intelligent robot enters an activated state; a judgment module 102, the judgment module 102 is used to judge whether the embodied intelligent robot enters a task planning stage according to the interaction instructions; a second acquisition module 103, the second acquisition module 103 is used to acquire a tool list when it is judged that the embodied intelligent robot enters the task planning stage; a planning module 104, the planning module 104 is used to perform task planning for the embodied intelligent robot according to the interaction instructions and the tool list to obtain a feasible action sequence; an execution module 105, the execution module 105 is used for the embodied intelligent robot to execute the interaction instructions according to the action sequence.

[0056] In a specific embodiment, the first acquisition module 101, judgment module 102, second acquisition module 103, planning module 104 and execution module 105 of the control system 100 of the embodied intelligent robot triggered by a large model cooperate to implement the control method of the embodied intelligent robot triggered by a large model as in the first embodiment above, which will not be repeated here.

[0057] [Example 3] See also Figure 7 The present embodiment provides a schematic structural diagram of an embodied intelligent robot 200. The embodied intelligent robot 200 includes, for example, a processor 220 and a memory 210 electrically connected to the processor 220. The memory 210 stores a computer program 211. The processor 220 loads the computer program 211 to implement the control method of the embodied intelligent robot based on large model triggering as in the first embodiment.

[0058] [Example 4] See also Figure 8 This embodiment also provides a readable storage medium 300, which stores computer executable instructions 310. When the computer executable instructions 310 are read and executed by the processor 220, the embodied intelligent robot 200 where the readable storage medium 300 is located is controlled to implement the control method of the embodied intelligent robot based on large model triggering as in the first embodiment.

[0059] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only illustrative. For example, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0060] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A control method of an embodied intelligent robot based on large model triggering, characterized in that: The control method comprises: After the embodied intelligent robot enters an activated state, obtaining an interaction instruction; According to the interactive instruction, determining whether the embodied intelligent robot enters a task planning phase; When it is determined that the embodied intelligent robot enters the task planning phase, obtaining a tool list; Performing task planning for the embodied intelligent robot according to the interaction instructions and the tool list to obtain a feasible action sequence; The embodied intelligent robot executes the interaction instruction according to the action sequence.

2. The control method according to claim 1, characterized in that: The interactive instructions include: executable task instructions and non-executable task instructions; The step of judging whether the embodied intelligent robot enters a task planning phase according to the interaction instruction includes: In the case where the interaction instruction is the executable task instruction, determining that the embodied intelligent robot enters the task planning stage; and / or In the case where the interaction instruction is the non-executable task instruction, it is determined that the embodied intelligent robot does not enter the task planning stage, and the embodied intelligent robot ends the interaction after responding to the non-executable task instruction.

3. The control method according to claim 1, characterized in that: When it is determined that the embodied intelligent robot enters the task planning stage, obtaining a tool list includes: Initializing the tool list; Detecting whether there is an object related to the interaction instruction in the environment; If the object exists, the object is used as an available tool, and the available tool is added to the tool list.

4. The control method according to claim 3, characterized in that: The performing task planning for the embodied intelligent robot according to the interaction instruction and the tool list to obtain a feasible action sequence includes: Performing task planning for the embodied intelligent robot according to the interactive instructions and the available tools; Determine whether there is a feasible plan during the task planning process; If yes, then obtaining the action sequence according to the feasible plan; and / or If not, the interaction instruction is updated.

5. The control method according to claim 1, characterized in that: After the embodied intelligent robot enters the activated state and before acquiring the interaction instruction, the control method further includes: When the embodied intelligent robot is in a standby state, acquiring environmental perception data; Determining whether the perception module large model is triggered according to the environmental perception data; When the perception module large model is triggered, the embodied intelligent robot enters the activation state from the standby state; The environmental perception data includes: voice data and image data.

6. The control method according to claim 1, characterized in that: The embodied intelligent robot executes the interaction instruction according to the action sequence, including: The embodied intelligent robot executes a first action in the action sequence and detects whether a risk condition is triggered during the execution of the first action.

7. The control method according to claim 6, characterized in that: After the embodied intelligent robot executes the first action in the action sequence and detects whether a risk condition is triggered during the execution of the first action, the control method further includes: In case the risk condition is triggered, stopping the first action; and / or When the risk condition is not triggered, the first action is executed and completed.

8. A control system for an embodied intelligent robot triggered by a large model, characterized in that: The control system is used to execute the control method of the embodied intelligent robot based on large model triggering according to any one of claims 1 to 7, and the control system includes: A first acquisition module, the first acquisition module is used to acquire an interaction instruction after the embodied intelligent robot enters an activated state; A judgment module, the judgment module is used to judge whether the embodied intelligent robot enters the task planning stage according to the interaction instruction; A second acquisition module, the second acquisition module is used to acquire a tool list when it is determined that the embodied intelligent robot enters the task planning stage; A planning module, the planning module is used to perform task planning for the embodied intelligent robot according to the interaction instructions and the tool list to obtain a feasible action sequence; An execution module is used by the embodied intelligent robot to execute the interaction instruction according to the action sequence.

9. An embodied intelligent robot, characterized in that: The embodied intelligent robot comprises: a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the control method of the embodied intelligent robot based on large model triggering are implemented as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, which, when executed by a processor, implement the steps of the control method of an embodied intelligent robot based on large model triggering as described in any one of claims 1 to 7.

Images