Robot control method and device based on decision state machine and computer equipment

Abstract

The present disclosure provides a robot control method and device based on a decision state machine and a computer device, relates to the technical field of robots and computers, in particular to the field of artificial intelligence, visual language large models, agents, robot control, and the method comprises the following steps: receiving an initial task instruction to enter a first state, determining a first atomic skill sequence of an execution instruction; in response to determining the first atomic skill sequence, entering a second state, and controlling the robot to execute the instruction; in response to determining that the number of failure times of a first execution result of the robot is greater than or equal to a preset threshold, entering a third state, providing context information to a visual language large model agent to obtain a recommended task instruction and convert it into a second atomic skill sequence; in response to obtaining the second atomic skill sequence, entering a fourth state, controlling the robot to execute the recommended task instruction to obtain a second execution result; and in response to the second execution result being successful, entering a fifth state, which can help the robot to efficiently execute the task.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of robots and computer technology, in particular to the fields of artificial intelligence, visual language large model (VLM), agent, robot control, and the like, and specifically to a robot control method and device based on a decision state machine, a computer device, a computer readable storage medium and a computer program product, and a robot. BACKGROUND

[0002] In the field of robots and automation, the core goal is to achieve intelligent work in unstructured environments. Traditional control architectures rely on predefined rules and deterministic motion planning algorithms, which are efficient in structured scenarios, but when faced with object occlusion, environmental mutations, or task ambiguity, etc. "long tail situations", the system has poor generalization, low robustness and high maintenance cost, as it cannot exhaust all failure response logic. On the other hand, end-to-end embodied intelligence technologies represented by visual language large models have strong semantic understanding and generalization potential, but their high data and computing cost, reasoning delay, and "black box" characteristics make them difficult to directly meet the stringent requirements of industrial robots for real-time and safety, and there is a risk of unstable output in zero-shot scenarios.

[0003] Therefore, there is an urgent need for an intelligent solution that can combine the determinism of traditional control with the advanced cognitive ability of large models, to bridge the "semantic gap" between high-level semantic reasoning and low-level physical execution, ensuring accurate execution of atomic skills while giving robots the ability to adapt to complex unknown scenarios. SUMMARY

[0004] The present disclosure provides a robot control method and device based on a decision state machine, a computer device, a computer readable storage medium and a computer program product, and a robot.

[0005] According to an aspect of the present disclosure, a robot control method based on a decision state machine is provided, including: receiving an initial task instruction to enter a first state, wherein in the first state, a first atomic skill sequence for the robot to execute the initial task instruction is determined, wherein the first atomic skill sequence includes at least one first atomic skill in a first predetermined order, each first atomic skill being associated with a corresponding action unit of the robot for executing the initial task instruction; in response to determining the first atomic skill sequence, entering a second state, wherein in the second state, the robot is controlled to execute the initial task instruction based on the first atomic skill sequence; in response to determining that a first execution result of the robot executing the initial task instruction is a failure and a number of failures is greater than or equal to a preset threshold, entering a third state, wherein in the third state, context information associated with the failure scenario is provided to a visual language large model agent to obtain a suggested task instruction provided by the visual language large model agent; and the suggested task instruction is converted into a second atomic skill sequence, wherein the second atomic skill sequence includes at least one second atomic skill in a second predetermined order, each second atomic skill being associated with a corresponding action unit of the robot for executing the suggested task instruction; in response to obtaining the second atomic skill sequence, entering a fourth state, wherein in the fourth state, the robot is controlled to execute the suggested task instruction based on the second atomic skill sequence to obtain a second execution result; and in response to determining that the second execution result of the robot executing the suggested task instruction is a success, entering a fifth state, the fifth state indicating that the robot completes the task.

[0006] In some embodiments, in the first state, determining the first atomic skill sequence for the robot to execute the initial task instruction includes: selecting at least one first atomic skill matching the initial task instruction in the first predetermined order from an atomic skill library; and for each first atomic skill, determining an execution parameter of the first atomic skill based on real-time acquired environment perception data, wherein the environment perception data is used to structurally represent a task environment of the robot.

[0007] In some embodiments, in the third state, the context information associated with the failure scenario is provided to the visual language large model agent to obtain the suggested task instruction provided by the visual language large model agent, including: determining a failure type corresponding to the failure scenario; generating a prompt word for the failure type, wherein the context information associated with the failure scenario is fused in the prompt word, the prompt word including at least a first prompt word part and a second prompt word part, the first prompt word part being used to indicate analysis of the failure scenario, and the second prompt word part being used to indicate providing a solution strategy for the failure scenario; and sending the prompt word to the visual language large model agent.

[0008] In some embodiments, generating the prompt word for the failure type comprises: determining, in a preset failure label mapping table, whether there is a failure diagnosis label corresponding to the failure type, wherein the failure label mapping table is used to indicate the correspondence between the failure type and the failure diagnosis label, the failure diagnosis label has a corresponding prompt word template, and the prompt word template includes a placeholder for fusing context information; in response to determining that there is a failure diagnosis label, determining a context variable corresponding to the placeholder from the context information; and filling the context variable into the prompt word template to generate the prompt word.

[0009] In some embodiments, converting the suggestion task instruction into the second atomic skill sequence comprises: performing natural language parsing on the suggestion task instruction to determine at least one action unit, each action unit including an extracted verb and an entity associated with the verb; for each action unit, determining an identifier corresponding to the entity and a parameter corresponding to an action represented by the verb; and based on a preset action mapping rule library, determining at least one symbolic function having a second predetermined order according to the action, the parameter, and the identifier of each action unit, the second atomic skill sequence including the at least one symbolic function, wherein each action unit is mapped to one or more symbolic functions.

[0010] In some embodiments, the robot control method based on the decision state machine further comprises: in response to determining that the first execution result of the robot executing the initial task instruction is failure and the number of failures is less than a preset threshold, entering a sixth state, wherein in the sixth state, the robot is controlled to execute a predefined local recovery strategy for solving the failure.

[0011] In some embodiments, in response to the suggestion task instruction corresponding to a plurality of action units, the second execution result includes sub-execution results obtained by the robot executing the plurality of action units respectively, and the robot control method based on the decision state machine further comprises: in response to determining that the sub-execution result of the robot executing a current action unit is success and the current action unit is not the last one in the suggestion task instruction, entering a seventh state, wherein in the seventh state, updated context information associated with the current scene is provided to the visual language large model agent to obtain a sub-suggestion task instruction for executing a next action unit provided by the visual language large model agent.

[0012] According to an aspect of the present disclosure, a robot is provided, which is controlled to perform a task according to the robot control method based on the decision state machine as described above.

[0013] According to an aspect of the present disclosure, there is provided a decision state machine based robot control apparatus, comprising: a first state determining module configured to receive an initial task instruction to enter a first state, wherein a first sequence determining module is comprised in the first state determining module and configured to determine a first atomic skill sequence for the robot to execute the initial task instruction, wherein the first atomic skill sequence comprises at least one first atomic skill having a first predetermined order, each first atomic skill being associated with a corresponding action unit of the robot for executing the initial task instruction; a second state determining module configured to enter a second state in response to determining the first atomic skill sequence, wherein an initial task execution module is comprised in the second state determining module and configured to control the robot to execute the initial task instruction based on the first atomic skill sequence; a third state determining module configured to enter a third state in response to determining that a first execution result of the robot executing the initial task instruction is a failure and a number of failures is greater than or equal to a preset threshold, wherein a suggestion task determining module and a second sequence determining module are comprised in the third state determining module, the suggestion task determining module is configured to provide context information associated with the failure scenario to a visual language large model agent to obtain a suggestion task instruction provided by the visual language large model agent, and the second sequence determining module is configured to convert the suggestion task instruction into a second atomic skill sequence, wherein the second atomic skill sequence comprises at least one second atomic skill having a second predetermined order, each second atomic skill being associated with a corresponding action unit of the robot for executing the suggestion task instruction; a fourth state determining module configured to enter a fourth state in response to obtaining the second atomic skill sequence, wherein a suggestion task execution module is comprised in the fourth state determining module and configured to control the robot to execute the suggestion task instruction based on the second atomic skill sequence to obtain a second execution result; and a fifth state determining module configured to enter a fifth state in response to determining that the second execution result of the robot executing the suggestion task instruction is a success, the fifth state indicating that the robot completes the task.

[0014] According to another aspect of the present disclosure, there is provided a computer device, comprising: at least one processor; and a memory having a computer program stored thereon, the computer program, when executed by the at least one processor, causing the at least one processor to perform the method provided above.

[0015] According to another aspect of the present disclosure, there is provided a computer readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, causing the processor to perform the method provided above.

[0016] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, the computer program, when executed by a processor, causing the processor to perform the method provided above.

[0017] According to one or more embodiments of the present disclosure, by introducing a decision state machine, traditional robot control technology can be combined with a visual language large model agent with reasoning capability, so as to ensure that the robot can efficiently perform tasks while also being able to cope with complex unknown situations.

[0018] These and other aspects of the present disclosure will become apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description serve to explain exemplary implementations of the application. The illustrated embodiments are exemplary only and not limiting of the scope of the present disclosure. In all the drawings, like reference numerals refer to like parts throughout the various figures. The key elements of the drawings are as follows:

[0020] Figure 1 is a module schematic diagram illustrating a decision state machine-based robot control system of an exemplary embodiment.

[0021] Figure 2 is a flowchart illustrating a decision state machine-based robot control method according to an exemplary embodiment.

[0022] Figure 3 is a flowchart illustrating a suggestion task instruction conversion method in a third state according to an exemplary embodiment.

[0023] Figure 4 is a schematic diagram illustrating a prompt word template corresponding to a failed grasp according to an exemplary embodiment.

[0024] Figure 5 is a schematic diagram illustrating conversion of a suggestion task instruction into a second atomic skill sequence according to an exemplary embodiment.

[0025] Figure 6 is a schematic diagram illustrating a decision state machine-based control flow according to an exemplary embodiment.

[0026] Figure 7 is a schematic block diagram illustrating a decision state machine-based robot control apparatus according to an exemplary embodiment.

[0027] Figure 8 is a block diagram illustrating an exemplary computer device that can be applied to an exemplary embodiment. DETAILED DESCRIPTION

[0028] Exemplary embodiments of the present disclosure are described herein below with reference to the accompanying drawings, in which various details of embodiments of the present disclosure are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, to those skilled in the art that the embodiments described herein are intended to be exemplary only and that various changes and modifications can be made thereto without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

[0029] In the present disclosure, the terms "first", "second", and the like are used to describe various elements only for the purpose of distinguishing one element from another, and the terms are not intended to limit the positions, sequence, or importance of the elements. In some examples, a first element and a second element can refer to the same instance at different times. In some cases, based on the context of the description, they can also refer to different instances.

[0030] The terms used in the description of various described examples in the present disclosure are only for the purpose of describing particular examples and are not intended to be limiting. Unless the context clearly indicates otherwise, the number of a particular element is intended to be one or more. As used herein, the term "plurality" means two or more, and the term "based on" is to be interpreted as "based, at least in part, on." In addition, the terms "and / or" and "at least one of" encompass any and all possible combinations of the listed items.

[0031] In the related art, the robot control route based on the traditional model and algorithm has high execution certainty in a specific, structured scene, but its inherent limitations will cause many problems when facing open, dynamic, or non-preset situations.

[0032] First, the entire decision-making process of the system relies on pre-written rules and logic. When encountering unexpected situations, such as objects being partially obscured, irregular poses, sudden changes in environmental light, or ambiguities in task instructions themselves, the performance of the system will decrease dramatically or even fail completely. Existing coping strategies mostly enumerate possible failure modes in advance in the rule engine and write corresponding recovery programs, but this method has several fundamental flaws: first, the intelligence level of the system is inherently limited by the developers' ability to foresee all unknown situations that may arise, and it cannot adapt to the infinite richness of the real world; second, the way of writing specific code for each specific failure makes the system difficult to extend and maintain, and lacks true adaptability and generalization ability. When the system fails, it usually does not have the ability to understand the failure reason at a deep semantic level, so it cannot make truly intelligent coping decisions, often falling into the dilemma of repeated ineffective operations or direct error reporting and shutdown.

[0033] With the breakthrough of the foundation large model, a series of new models with visual language and action reasoning ability also emerged. However, when these frontier models are put into real, open and highly demanding robot application scenarios for testing, their limitations become apparent. The performance of these models is highly dependent on the field covered by their training data. For scenes and objects that frequently appear in the data set, they can show a high success rate, but once they face zero-shot situations outside the training distribution, their performance becomes extremely unstable. The model will produce illogical action sequences, fail to accurately understand subtle spatial instructions, or get stuck in a low-efficiency dilemma of repeated trial and error. Therefore, although these models endow robots with intelligence, if they are directly used as the core decision controller of the robot system, they will face serious challenges in real-time performance, interpretability, safety and task success rate.

[0034] To this end, the embodiments of the present disclosure propose a more effective robot control scheme. By introducing a decision state machine, the robot is set to complete corresponding tasks in different states, and the visual language large model agent is embedded into the control loop of the robot system instead of replacing the traditional control chain. When the robot cannot handle abnormal situations, it enters the large model consultation state, providing the current global context to the visual language large model agent. This agent uses its powerful cross-modal understanding and common sense reasoning ability to analyze the abnormal situation in depth and give a solution suggestion. Finally, the solution suggestion is mapped into the robot's executable bottom-level skills, enabling the robot to have problem-solving ability in complex real-world environments.

[0035] Therefore, the method of the embodiments of the present disclosure deeply integrates reliable and determinate robot bottom-level skills with visual language large model agents with general semantic understanding and reasoning ability by using a decision state machine. This not only ensures the efficiency and safety of basic execution, but also reliably completes tasks when facing complex situations, significantly improving the task success rate and robustness of robots in unstructured environments.

[0036] Figure 1 is a schematic diagram of a robot control system module based on a decision state machine illustrating an example embodiment.

[0037] As Figure 1 shown, the system can include a perception module, an atomic skill library execution module, a decision state machine module, and a visual language large model agent module.

[0038] The workflow of the system begins with the perception module, which can acquire environmental data in real time through various sensors of the robot, and use the built-in, optimized lightweight computer vision model to preliminarily analyze the scene and extract basic structured information such as object pose, passable area, etc. These information, together with the raw visual images, are passed to the decision state machine module.

[0039] The decision state machine module is the core of the system. In a normal case, it directly calls the atomic skill execution module to execute the atomic skills to perform the task according to the current task target, and feeds back the result to the decision state machine module. Among them, the atomic skill library execution module pre-encapsulates highly optimized basic action instructions for the robot. When the system encounters a situation that cannot be handled according to the preset rules, the decision state machine will actively request the visual language large model agent module, provide the current global context to the visual language large model agent module to obtain a structured strategy, then map the strategy to the basic action instructions executable by the robot, and finally call the atomic skill library execution module to execute the basic action instructions.

[0040] Therefore, through the cooperation between the above four modules, an adaptive robot system with efficiency and intelligence, local closure and global openness is constructed, which provides a feasible technical path for the reliable and autonomous work of the robot in a complex real unstructured environment.

[0041] Exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0042] Figure 2 is a flowchart illustrating a robot control method based on a decision state machine according to an exemplary embodiment.

[0043] In an embodiment of the present disclosure, the decision state machine is a mechanism based on a finite state machine, which is responsible for the control of global scheduling and intelligent decision. Therefore, the decision state machine is not a simple sequential flow controller, but an intelligent scheduling engine with state maintenance, context management and event-driven capabilities, which can define all possible states of the robot during task execution, the transition conditions between states and the actions to be performed when in a particular state.

[0044] As shown in Figure 2 The robot control method based on the decision state machine includes steps S201 to S205.

[0045] In step S201, an initial task instruction is received to enter a first state. In the first state, a first atomic skill sequence for the robot to execute the initial task instruction is determined. The first atomic skill sequence includes at least one first atomic skill with a first predetermined order, each first atomic skill being associated with a corresponding action unit of the robot for executing the initial task instruction.

[0046] In an example, the task execution flow can start from an idle waiting state, when a task instruction, i.e. the initial task instruction in the embodiment (such as "pick up the red mug and place it in the designated area"), is received, the decision state machine first transitions to a task initialization state, i.e. the first state in the embodiment. In the task initialization state, the perception module as described above can be called to comprehensively scan the current scene, identify and locate the relevant objects, and at the same time, pre-process and parameter bind the atomic skill sequence required for the task, i.e. determine the first atomic skill sequence for the robot to execute the initial task instruction in the embodiment.

[0047] In an example, the atomic skill can be a bottom-level action function pre-encapsulated in the system library and constructed based on traditional kinematics and dynamics algorithms, such as a target object picking function, a path navigation function, etc., which constitute the basic behavior ability of the robot. In the first state, the first atomic skill required by the task can be determined according to the received initial task instruction, and the order between these first atomic skills can be determined. For example, the initial instruction is "pick up the red mug and place it in the designated area", because the actions of picking up, navigating, and placing are required to complete the instruction, therefore the atomic skills required can be determined as the target object picking function, the path navigation function, and the target object placing function. In actual operation, the behavior of the robot needs to conform to the physical logic, i.e. it needs to move to the vicinity of the red mug first, then pick up the mug, and then move to the designated area to place the mug, therefore there is a certain order between these actions, i.e. the first predetermined order. After determining the order, the required bottom-level action functions are combined in the order, and the parameters related to the target object obtained from the environment are bound to the functions, thereby constituting the first atomic skill sequence executable by the robot. Still taking the initial task instruction "pick up the red mug and place it in the designated area" as an example, after determining the action order and binding the parameters, the first atomic skill sequence includes navigating to the red mug area, picking up the red mug, navigating to the designated area, and placing the red mug.

[0048] In step S202, in response to determining the first atomic skill sequence, the second state is entered. In the second state, the robot is controlled to execute the initial task instruction based on the first atomic skill sequence.

[0049] In an example, when the task initialization state is completed, i.e., in response to determining the first atomic skill sequence, the state machine can enter an atomic skill execution state, i.e., the second state in the embodiment. The state is a substantial physical operation stage entered after the completion of task planning, and its responsibility is to convert instructions at the logical level into mechanical movements at the physical level. The atomic skill execution state can be the main loop state. In the atomic skill execution state, the robot sequentially executes the first atomic skill sequence, such as performing atomic skills such as "navigate to the front of the target", "perform pre-grabbing posture", "perform grabbing", etc. After each atomic skill is executed, success or failure feedback can be received.

[0050] In step S203, in response to determining that the first execution result of the robot executing the initial task instruction is failure and the number of failures is greater than or equal to a preset threshold, the third state is entered.

[0051] In an example, the first execution result can refer to real-time result feedback received from the bottom execution unit when the robot sequentially executes the first atomic skill sequence, such as movement success, grabbing slip, motion obstruction, or navigation timeout, etc. When the first execution result is failure, the built-in failure counter is first updated. Then the current cumulative number of failures is compared with the preset threshold, and once it is detected that the number of failures reaches or exceeds the threshold, the state transition is triggered immediately, and the third state is entered to start the advanced cognitive process. For example, the preset threshold is 3 times, and the result feedback of the robot executing the grabbing action is grabbing failure, and the previous two times are also grabbing failure, then the value of the current failure counter is 3, which reaches the preset threshold, and enters the third state immediately.

[0052] Figure 3 is a flowchart illustrating a suggestion task instruction conversion method in the third state according to an example embodiment.

[0053] As Figure 3 shown, in the third state, steps S2031 and S2032 are sub-steps of step S203.

[0054] In sub-step S2031, the context information associated with the failure scenario is provided to the visual language large model agent to obtain the suggested task instruction provided by the visual language large model agent.

[0055] In the example, a failure scenario refers to a specific situation where the robot's task execution is interrupted due to dynamic changes in the environment or continuous failure of atomic skills (such as grasping failure or navigation obstruction). Contextual information can include environmental data, historical execution logs, and task objective descriptions for the failure scenario, used to accurately reconstruct the problem background. When consecutive task execution failures are detected (such as grasping operations failing three times exceeding a preset threshold), the current environmental image, historical execution logs, and task objective descriptions obtained from the perception module are provided to the visual language large-scale model agent. The visual language large-scale model agent can leverage its cross-modal reasoning capabilities to perform deep diagnosis of complex scenarios, analyze the causes of failure, and thus provide strategy suggestions, i.e., suggested task instructions, for that failure scenario.

[0056] In sub-step S2032, the proposed task instructions are converted into a second atomic skill sequence. The second atomic skill sequence includes at least one second atomic skill having a second predetermined order. Each second atomic skill is associated with a corresponding action unit used by the robot to execute the proposed task instructions.

[0057] In the example, since the suggested task instruction is a solution expressed in natural language, output by the visual language large model agent after cognitive reasoning about the current failure scenario, such as "remove the blue book from the red mug first," its essence is semantic-level behavior planning. Therefore, it needs to be converted into a series of basic actions that the robot can execute. This means determining the second atomic skills for executing the suggested task instruction, as well as the order in which these second atomic skills are executed. Again, using the suggested task instruction "remove the blue book from the red mug first" as an example, the robot doesn't know the operation corresponding to "remove." Therefore, "remove the blue book" needs to be converted into executable second atomic skills: a function for grasping the target object, a path navigation function, and a function for placing the target object. Then, following a predetermined second order of navigation-grabbing-navigation-placement, parameters related to the blue book obtained from the environment are bound to the functions to form the second atomic sequence. The second atomic skill sequence then includes navigating to the blue book area, grasping the blue book, navigating to another designated area, and placing the blue book.

[0058] Back to Figure 2 In step S204, in response to obtaining the second atomic skill sequence, the robot enters the fourth state. In the fourth state, the robot executes the suggested task instruction based on the second atomic skill sequence to obtain the second execution result.

[0059] In an example, when the suggested task instruction given by the visual language large model agent is converted into a basic operation executable by the robot, that is, in response to determining the second atomic skill sequence, the state machine can enter the atomic skill execution state again, that is, the fourth state in this embodiment. In this atomic skill execution state, the robot is controlled to execute the second atomic skill sequence in sequence, for example, to execute instructions such as "navigate to the blue book area", "pick up the blue book", etc. After execution, success or failure feedback can be received.

[0060] In step S205, in response to determining that the second execution result of the robot executing the suggested task instruction is successful, the fifth state is entered, which indicates that the robot has completed the task.

[0061] In an example, the second execution result can refer to real-time task result feedback when the robot executes the suggested task instruction, for example, that the red mug has been placed in the specified area. When the second execution result is successful, the fifth state is entered for resource release and result reporting. The fifth state is the task end state, indicating the successful closed loop of the current work process.

[0062] In an example, if the second execution result is also a failure, the state machine enters the "task failure" safety state to ensure that the robot is in a controllable stop posture.

[0063] Therefore, by introducing the decision state machine, atomic skills are directly called in regular tasks, and the visual language large model agent is enabled and generates strategy suggestions only when the task fails. On the one hand, the efficiency of regular tasks can be guaranteed, and the delay and safety risks caused by direct control of end-to-end large models can be avoided; on the other hand, through the on-demand consultation mechanism of the visual language large model agent, the robot is given high-level cognitive ability to cope with unknown environmental changes. At the same time, the generated strategy suggestions are accurately mapped to atomic skill sequences, so that the output results of the visual language large model agent are safely and efficiently landed as specific actions of the robot. This "perception-failure-visual language large model agent analysis-strategy generation-re-mapping-re-execution" collaborative closed-loop control logic not only breaks through the generalization bottleneck of traditional rule systems, but also solves the problem of large model landing, significantly improving the robustness and task success rate of the robot in complex unstructured scenes.

[0064] In some embodiments, in the first state, determining the first atomic skill sequence for the robot to execute the initial task instruction can include: selecting at least one first atomic skill matching the initial task instruction from the atomic skill library in a first predetermined order; and for each first atomic skill, determining the execution parameter of the first atomic skill based on real-time acquired environment perception data, wherein the environment perception data is used to structurally represent the task environment of the robot.

[0065] In an example, the atomic skill library can refer to a collection of underlying action functions pre-packaged with standard calling interfaces, such as "point-to-point movement", "precise grasping", etc. which are built based on kinematics and dynamics algorithms; the first predetermined order can refer to a time sequence that conforms to the logic of robot operation, for example, navigation to the target position must be performed before the grasping action. After receiving the initial task instruction, the first atomic skill capable of executing the instruction, i.e. the specific function function, is retrieved and matched from the atomic skill library according to the content in the instruction. Then these functions are assembled into an ordered execution chain according to the operation sequence. For example, the initial task instruction is "press the red button", then the required first atomic skill: the arm movement function, the path navigation function can be matched from the atomic skill library. Then according to the operation logic, the path navigation function and the arm movement function are sequentially assembled into an ordered execution chain.

[0066] In an example, during the execution of the function function, the robot also needs to determine the parameters for executing the function function, which can be obtained by calculation using the environmental information obtained in real time by the perception module in the environment. The perception module can be an efficient and accurate environment information extraction front end, which can include a camera, and can also include sensors such as laser radar, etc. These devices are rigidly fixed to the robot body, forming the "eyes" of the system. The "environmental perception data" can refer to structured information that can quantitatively describe the state of the physical world, which is generated after being collected by sensors and processed by computer vision algorithms. For example, it specifically includes the position coordinates and rotation angle of the target object, the three-dimensional size bounding box and obstacle distribution information, etc. Through these environmental perception data, the execution parameters suitable for the current function function can be calculated.

[0067] In an example, after receiving the initial task instruction “press the red button”, the camera-captured environment image can be input into a target detection model (e.g., YOLO model) that has been fine-tuned for this purpose. The model has been previously trained on a dataset of custom images that meticulously capture the states of all target objects (e.g., different colored mugs, different shaped buttons, etc.) in the demonstration scene under various lighting, angles, and partial occlusion conditions, ensuring that the model can identify these specific objects with high confidence. The output of the target detection model is a precise two-dimensional pixel coordinate bounding box of the “red button”. The two-dimensional pixel coordinate bounding box is then input into a segmentation base model, which generates a pixel-level precision segmentation mask of the “red button”. This mask is combined with the point cloud data provided by the depth camera, and through camera intrinsic parameter coordinate transformation, the three-dimensional spatial position data and rotation angle of the “red button” in the robot base coordinate system are calculated. Finally, the position data and rotation angle are filled into the parameters of the path navigation function and the mechanical arm movement function, thus forming a first atomic skill sequence executable by the robot.

[0068] Therefore, by sequentially selecting atomic skills that match the initial task instruction from the atomic skill library, and dynamically obtaining the execution parameters required by the atomic skills from the environment perception data, routine tasks can be efficiently executed, and the basic execution efficiency is maintained.

[0069] In some embodiments, in the third state, providing the context information associated with the failure scenario to the visual language large model agent to obtain the suggested task instruction provided by the visual language large model agent can include: determining a failure type corresponding to the failure scenario; generating a prompt word for the failure type, wherein the context information associated with the failure scenario is fused in the prompt word, and the prompt word at least includes a first prompt word part and a second prompt word part, the first prompt word part is used to instruct to analyze the failure scenario, and the second prompt word part is used to instruct to provide a solution strategy for the failure scenario; and sending the prompt word to the visual language large model agent.

[0070] In an example, the failure type can refer to a labeled definition of the specific reason that causes the current operation of the robot to be interrupted or unable to achieve the goal, such as “grasping failure”, “path planning timeout”, etc., which is used to quickly classify the specific scenario of task failure. Since the visual language large model agent needs to generate a recoverable strategy executable under the current failure condition, there is a corresponding prompt word for each failure type, rather than relying on a general prompt word to cause invalid output.

[0071] In an example, the context information can include real-time environmental data (e.g., scene image) of the failure scenario, historical execution logs, and a task goal description (e.g., "grab the blue cup") for structurally representing the failure background. Therefore, providing the context information to the visual language large model agent can enable the visual language large model agent to accurately diagnose the root cause of the failure and generate an executable recovery strategy for the current failure scenario.

[0072] In an example, the prompt words are a set of dynamically generated structured text instructions, not just simple questions, but a complex instruction set containing role setting, task background, and thought chain guidance. In some prompt words, the visual language large model agent can be guided to analyze the failure cause, i.e., the first prompt word part. For example, in a grabbing failure scenario, the analysis requirement can be set as "possible reasons for failure: occlusion, posture, instability". In another part of the prompt words, the visual language large model agent can be guided to provide an executable solution strategy, i.e., the second prompt word part. For example, also in the grabbing failure scenario, the output requirement can be set as "based on the analysis, propose a specific and executable recovery strategy. Format requirement: the strategy must be a natural language description of the action sequence".

[0073] Therefore, by using this prompt word construction mechanism for different failure types and guiding the visual language large model agent to analyze the failure cause and output the coping strategy in the prompt words, the attention of the visual language large model agent can be focused on problem diagnosis and executable strategy generation, improving the quality and reliability of its output.

[0074] In some embodiments, generating the prompt word for the failure type can include determining whether there is a failure diagnosis label corresponding to the failure type in a preset failure label mapping table, wherein the failure label mapping table is used to indicate the correspondence between the failure type and the failure diagnosis label, the failure diagnosis label has a corresponding prompt word template, and the prompt word template includes a placeholder for fusing context information; in response to determining that there is a failure diagnosis label, determining a context variable corresponding to the placeholder from the context information; and filling the context variable into the prompt word template to generate the prompt word.

[0075] In an example, the failure label mapping table can refer to a structured database pre-set in the system memory, which defines the correspondence between the failure type output by the underlying system and the failure diagnosis label associated with a specific prompt template. For example, the failure type is path planning timeout, and the failure diagnosis label is navigation blockage; for another example, the failure type is grasping sensor abnormality, and the failure diagnosis label is grasping failure. The failure diagnosis label can be an identifier for indexing, which points to a pre-set prompt template. The prompt template is a structured text framework, which contains pre-set role settings, fixed analysis instructions (i.e., the first prompt part and the second prompt part as described above), and placeholders for dynamically inserting data, which are used to indicate the positions where context information variables need to be inserted.

[0076] In an example, by associating the failure type to a specific prompt template through the failure label mapping table, the placeholders in the template automatically match the corresponding variables in the context information, realizing accurate filling of variable values. For example, when the task of "grasping a red mug" fails and the number of failures exceeds a threshold, the failure diagnosis label is grasping failure, the prompt template corresponding to the failure diagnosis label is called, and then "red mug" is filled into the placeholder representing "target", the record of grasping failure is filled into the placeholder representing "history log", and the image of grasping failure is filled into the placeholder representing "attachment". Thus, the prompt for the failure of the task of "grasping a red mug" can be finally generated.

[0077] In an example, when none of the pre-set failure diagnosis labels matches, a general diagnosis template can be used. The general template can guide the visual language large model to perform problem diagnosis, starting from scratch to analyze the image, the history log, and the target, and to perform creative reasoning. It can also use common sense, vertical domain knowledge, and cross-modal reasoning to make up for the shortcomings of pre-set rules, and identify deep failure root causes that traditional programs cannot understand.

[0078] Therefore, by this method of generating prompts by filling templates for different failure types, prompts that accurately describe the current failure scenario can be generated, which is a basic step for the visual language large model to generate accurate strategies.

[0079] Figure 4 is a schematic diagram illustrating a prompt template corresponding to grasping failure according to an example embodiment.

[0080] As Figure 4 shown, Figure 4The prompt word template corresponding to the failed grasping in the task instruction includes role setting, task target, data to be analyzed, analysis requirements, and format requirements. The role setting can guide the visual language large model agent to output content related to robot motion planning; the task target can determine the entity that needs to be grasped currently; providing the data to be analyzed can enable the visual language large model agent to analyze in combination with the context of the grasping failure scene; the analysis requirements can guide the visual language large model agent to analyze the reason for the grasping failure and output an executable recovery strategy; and the format requirements can enable the output strategy to be a natural language description of an action sequence. For example, when the task of “grasping a red mug” fails and the number of failures exceeds a threshold, “red mug” can be filled into [target], the image of the failed grasping can be filled into [appendix], and the record of the failed grasping can be filled into [history log]. Thus, the prompt word for the grasping task failure can be generated.

[0081] In some embodiments, converting the recommended task instruction into the second atomic skill sequence can include: performing natural language parsing on the recommended task instruction to determine at least one action unit, each action unit including an extracted verb and an entity associated with the verb; for each action unit, determining an identifier corresponding to the entity and a parameter corresponding to the action represented by the verb; and based on a preset action mapping rule library, determining at least one symbolic function having a second predetermined order according to the action, the parameter, and the identifier of each action unit, the second atomic skill sequence including the at least one symbolic function, wherein each action unit is mapped to one or more symbolic functions.

[0082] In an example, since the recommended task instruction is in natural language, it is necessary to perform natural language parsing on the recommended task instruction to decompose it into one or more basic action units. An action unit is composed of a verb representing a specific action intent and an entity as an action object. For example, the recommended task is “move the backpack in front of the table”, which can be parsed as “move the backpack” through natural language parsing.

[0083] In an example, for the entity in each action unit, it is necessary to cross-compare it with the list of real-time object names output by the perception module to confirm the corresponding object of the entity in the robot system and obtain its identifier. For example, in the action unit of “move the backpack”, through cross-comparison, it is confirmed that the entity “backpack” corresponds to the “backpack” object in the robot system, and the identifier of the “backpack” object is obtained.

[0084] In an example, the parameter can refer to a specific numerical value required when performing the action represented by the verb. For example, in the action unit of “move the backpack”, the position of the backpack, the position where the backpack is placed, and the grasping pose need to be obtained.

[0085] In an example, the action mapping rule library can refer to a pre-set rule library defining the correspondence between natural language semantic actions and robot underlying execution code, which contains conversion rules from abstract verbs such as "move", "remove" to specific symbolic functions. The symbolic function can refer to a standardized program interface that can be directly called for execution.

[0086] In an example, according to the verb in each action unit, one or more symbolic functions corresponding to the verb in each action unit can be determined in sequence according to the matching rules in the action mapping rule library, and then the identifiers of the parameters and entities required when performing the action are provided to the symbolic functions, and finally the symbolic functions are combined into a second atomic skill sequence in a second predetermined order that conforms to the physical operation logic. For example, the decomposed action unit is "move backpack", and the identifier of the backpack, the position of the backpack, the grabbing posture of the backpack, and the position of the backpack are determined. According to the action mapping library, the symbolic functions required to complete these action units are matched, i.e., the target object grabbing function and the target object placing function. The identifier of the backpack, the position of the backpack, and the grabbing posture of the backpack are provided to the target object grabbing function to make the robot grab the backpack, and the identifier of the backpack and the position of the backpack are provided to the target object placing function to make the robot place the backpack at the specified position.

[0087] Therefore, by converting high-level semantics into robot executable sequences through action mapping rules, a safety firewall is established between the visual language large model agent and the robot underlying control, which can ensure that each action is precisely controlled and improve the reliability of task execution.

[0088] Figure 5 is a schematic diagram illustrating the conversion of the recommended task instruction into a second atomic skill sequence according to an example embodiment.

[0089] As Figure 5As shown, the suggested task instruction given by the visual language large model agent is "remove the blue book first, then pick up the red mug". First, the suggested task instruction is parsed in natural language, and two action units are decomposed, namely "remove the blue book" and "pick up the red mug". Then the parameters required by the two action units "remove" and "pick up" are determined, namely the identifier of the blue book "blue_book", the position where it needs to be placed "staging_area", and the identifier of the red mug "red_mug_body". Then the matching rules in the action mapping rule library are used to determine the symbolic functions corresponding to the two action units "remove" and "pick up", wherein "remove" corresponds to "pick up function" and "place function", and "pick up" corresponds to "pick up function", and the parameters are bound to the functions. Finally, an executable second atomic skill sequence is generated, that is, the blue book pick up function "execute_manipulation("pick","blue_book")", the blue book place function "execute_manipulation("place","staging_area")", and the red mug pick up function "execute_manipulation("pick","red_mug_body")" as shown in the figure.

[0090] In some embodiments, the robot control method based on the decision state machine can further include, in response to determining that the first execution result of the robot executing the initial task instruction is failure and the number of failures is less than a preset threshold, entering a sixth state, wherein in the sixth state, the robot is controlled to execute a predefined local recovery strategy for solving the failure.

[0091] In an example, when the first execution result of the robot executing the initial task instruction is failure, the visual language large model is not immediately sought for help, but a built-in failure counter is updated, and the value of the failure counter is increased by 1 each time there is a failure. Then the value of the current failure counter is compared with the preset threshold, and if the value of the current failure counter is less than the preset threshold, the "local recovery attempt" state, i.e. the sixth state in this embodiment, is entered. In this state, the system attempts to execute a series of predefined simple recovery strategies, such as switching to a backup pick-up point of the same object, retrying after a small amount of position adjustment, or waiting for a short time for environmental changes. These strategies are fast and low-consumption, and are designed to solve some common and minor interference problems.

[0092] Therefore, by setting the sixth state, when the number of failures is less than the preset threshold, the predefined local recovery strategy is directly called to quickly correct occasional failures. This effectively avoids frequent triggering of the large model consultation process with high algorithmic power consumption due to minor failures, and maximizes the execution efficiency of the robot operation.

[0093] In some embodiments, in response to the suggestion task instruction corresponding to multiple action units, the second execution result includes sub-execution results obtained by the robot executing the multiple action units respectively, and the robot control method based on the decision state machine further includes: in response to determining that the sub-execution result of the robot executing the current action unit is successful and the current action unit is not the last one in the suggestion task instruction, entering a seventh state, wherein in the seventh state, the updated context information associated with the current scene is provided to the visual language large model agent to obtain a sub-suggestion task instruction provided by the visual language large model agent for executing the next action unit.

[0094] In an example, the suggestion task instruction can contain a natural language solution of multi-step logic, for example, the suggestion task instruction is “try to remove the blue book first, and then grab the red mug”, which logically corresponds to multiple ordered action units, i.e., the two action units of “remove the blue book” and “grab the red mug”, and each execution of an action unit will obtain a corresponding sub-execution result.

[0095] In an example, when the executed action unit is not the last one and its sub-execution result is successful, a state transition to the execution optimization strategy state, i.e., the seventh state in this embodiment, is automatically triggered. In the seventh state, the perception module can be activated to take a new image of the current scene, and the history log of successful execution and the new image are sent to the visual language large model agent as new context information to ask the large model for the next strategy, thereby obtaining a sub-suggestion task instruction for executing the next action unit. After receiving the task instruction, the robot is controlled to continue executing the task instruction.

[0096] In an example, still taking the suggestion task instruction “try to remove the blue book first, and then grab the red mug” as an example, the current action unit “remove the blue book” has been successfully completed and is not the last one, then the seventh state is entered, the history log of successful execution and the new scene image are sent to the visual language large model agent as new context information, the visual language large model agent gives a sub-suggestion instruction of “continue to grab the red mug”, and finally the robot is controlled to continue to complete the operation of grabbing the red mug.

[0097] Therefore, by adopting this “execution-observation-reconfirmation” feedback mechanism, a complex task instruction can be decomposed into multiple sub-action units, and after the completion of each action unit, a cognitive evaluation is immediately followed. This mechanism can ensure that the strategy of each sub-suggestion task matches the current environment, providing reliable execution support for complex tasks.

[0098] In some embodiments, a robot is also provided, which is controlled to perform a task according to the decision state machine based robot control method as described above.

[0099] In an example, a robot can refer to an intelligent device capable of accepting task instructions and executing them. The robot is internally integrated with a decision state machine controller, which automatically manages the task flow according to pre-set states, and controls the robot to perform tasks based on the decision state machine based robot control method.

[0100] Figure 6 is a schematic diagram illustrating a control flow based on a decision state machine according to another exemplary embodiment.

[0101] As Figure 6 shown, in the decision state machine, the control flow starts with the input of a task instruction. The instruction is passed to the state and context manager of the decision state machine, thereby starting a complete autonomous job cycle. First, the environment information is obtained through a perception module, which is composed of sensors and computer vision models, and can convert raw sensor data into structured data representation containing object poses, detection boxes, spatial relationships, etc., providing real-time perception basis for subsequent decision-making.

[0102] After obtaining the environment perception data, the decision state machine enters the core task execution phase. It first attempts to call the basic capabilities in the atomic skill library to complete the task, which encapsulates strictly verified atomic skills such as grasping, moving, obstacle avoidance, etc. These skills are based on traditional motion planning and control algorithms, ensuring the efficiency and determinacy of execution. When the feedback of skill execution is successful, it enters the task completion state; however, when the feedback is execution failure, a failure handling mechanism can be started.

[0103] The failure handling mechanism first updates the failure counter and records the scene context when the failure occurs in detail, forming a complete "failure snapshot". If the number of failures is within the pre-set threshold, the system will try to re-execute the original atomic skill sequence; once the number of failures exceeds the threshold, it indicates that the conventional method cannot solve the problem, at which time the system will automatically trigger the visual language large model agent consultation process. The process will assemble a prompt word containing the current context information and send it to the visual language large model agent.

[0104] As an advanced cognitive engine, the visual language big data agent, upon receiving a request for assistance, leverages its powerful visual language understanding and reasoning capabilities to conduct in-depth analysis of failed scenarios. It can identify root causes such as the robotic arm itself obstructing the target, the object being in a difficult-to-grasp posture, or the presence of unforeseen obstacles. Based on this analysis, the visual language big data agent can generate structured recovery strategies, such as "adjusting the viewing angle to reassess the scene" or "removing the obstruction before attempting to grasp."

[0105] These structured strategies, once returned to the decision state machine, are parsed and transformed by a specialized policy mapper. The policy mapper intelligently translates and decomposes the natural language-described strategies output by the visual-language large-scale model agent into ordered combinations of one or more atomic skills from the atomic skill library. For example, "remove an obstruction" is mapped to a series of specific navigation, grasping, and placing operations. The newly generated atomic skill sequences are then fed into the execution phase, forming a complete intelligent closed loop of "perception-failure-visual-language large-scale model agent analysis-policy generation-remapping-re-execution".

[0106] Embodiments of this disclosure also provide a robot control device based on a decision state machine.

[0107] Figure 7 This is a schematic block diagram illustrating a robot control device 700 based on a decision state machine according to an exemplary embodiment.

[0108] In some embodiments, the robot control device 700 may include a first state determination module 701, a second state determination module 702, a third state determination module 703, a fourth state determination module 704, and a fifth state determination module 705.

[0109] The first state determination module 701 is configured to receive an initial task instruction to enter a first state. The first state determination module 701 includes a first sequence determination module 7011, which is configured to determine a first atomic skill sequence for the robot to execute the initial task instruction. The first atomic skill sequence includes at least one first atomic skill having a first predetermined order, and each first atomic skill is associated with a corresponding action unit for the robot to execute the initial task instruction.

[0110] The second state determination module 702 is configured to enter a second state in response to determining the first atomic skill sequence. The second state determination module 702 includes an initial task execution module 7021, which is configured to control the robot to execute an initial task instruction based on the first atomic skill sequence.

[0111] The third state determination module 703 is configured to enter a third state in response to determining that the first execution result of the robot executing the initial task instruction is failure and the number of failures is greater than or equal to a preset threshold, wherein the third state determination module 703 comprises a suggested task determination module 7031 and a second sequence determination module 7032, the suggested task determination module 7031 is configured to provide context information associated with the failure scene to the visual language large model agent to obtain a suggested task instruction provided by the visual language large model agent, and the second sequence determination module 7032 is configured to convert the suggested task instruction into a second atomic skill sequence, wherein the second atomic skill sequence comprises at least one second atomic skill having a second predetermined order, and each second atomic skill is associated with a corresponding action unit of the robot for executing the suggested task instruction.

[0112] The fourth state determination module 704 is configured to enter a fourth state in response to obtaining the second atomic skill sequence, wherein the fourth state determination module 704 comprises a suggested task execution module 7041 configured to control the robot to execute the suggested task instruction based on the second atomic skill sequence to obtain a second execution result.

[0113] The fifth state determination module 705 is configured to enter a fifth state in response to determining that the second execution result of the robot executing the suggested task instruction is success, and the fifth state indicates that the robot completes the task.

[0114] In some embodiments, the first sequence determination module 7011 can comprise a first skill selection module 7011a and an execution parameter determination module 7011b.

[0115] The first skill selection module 7011a can be configured to select at least one first atomic skill matching the initial task instruction from the atomic skill library in a first predetermined order.

[0116] The execution parameter determination module 7011b can be configured to determine, for each first atomic skill, an execution parameter of the first atomic skill based on real-time acquired environment perception data, wherein the environment perception data is used to structurally represent the task environment of the robot.

[0117] In some embodiments, the suggested task determination module 7031 can comprise a failure type determination module 7031a, a prompt word generation module 7031b, and a prompt word sending module 7031c.

[0118] The failure type determination module 7031a can be configured to determine a failure type corresponding to the failure scene.

[0119] The prompt word generation module 7031b can be configured to generate a prompt word for the failure type, wherein context information associated with the failure scenario is fused in the prompt word, the prompt word at least including a first prompt word part and a second prompt word part, the first prompt word part being used to instruct to analyze the failure scenario, and the second prompt word part being used to instruct to provide a solution strategy for the failure scenario.

[0120] The prompt word sending module 7031c can be configured to send the prompt word to the visual language large model agent.

[0121] In some embodiments, the prompt word generation module 7031b can include a failure label determination module 7031b-1, a context determination module 7031b-2, and a context filling module 7031b-3.

[0122] The failure label determination module 7031b-1 can be configured to determine whether there is a failure diagnosis label corresponding to the failure type in a preset failure label mapping table, wherein the failure label mapping table is used to indicate the correspondence between the failure type and the failure diagnosis label, the failure diagnosis label has a corresponding prompt word template, and the prompt word template includes a placeholder for fusing context information.

[0123] The context determination module 7031b-2 can be configured to determine a context variable corresponding to the placeholder from the context information in response to determining that there is a failure diagnosis label.

[0124] The context filling module 7031b-3 can be configured to fill the context variable into the prompt word template to generate the prompt word.

[0125] In some embodiments, the second sequence determination module 7032 can include an action unit determination module 7032a, an action parameter determination module 7032b, and a function determination module 7032c.

[0126] The action unit determination module 7032a can be configured to perform natural language parsing on the suggestion task instruction to determine at least one action unit, each action unit including an extracted verb and an entity associated with the verb.

[0127] The action parameter determination module 7032b can be configured to determine, for each action unit, an identifier corresponding to the entity and a parameter corresponding to the action represented by the verb.

[0128] The function determination module 7032c can be configured to determine, based on a preset action mapping rule library, at least one symbolic function having a second predetermined order according to the action, the parameter, and the identifier of each action unit, the second atomic skill sequence including the at least one symbolic function, wherein each action unit is mapped to one or more symbolic functions.

[0129] In some embodiments, the robot control apparatus 700 can further include a sixth state determining module 706.

[0130] The sixth state determining module 706 can be configured to enter a sixth state in response to determining that the first execution result of the robot executing the initial task instruction is a failure and the number of failures is less than a preset threshold, wherein a local strategy execution module 7061 is included in the sixth state determining module 706 and is configured to control the robot to execute a predefined local recovery strategy for resolving the failure.

[0131] In some embodiments, in response to the suggested task instruction corresponding to a plurality of action units, the second execution result includes sub-execution results obtained by the robot executing the plurality of action units respectively, wherein the robot control apparatus 700 further includes a seventh state determining module 707.

[0132] The seventh state determining module 707 can be configured to enter a seventh state in response to determining that the sub-execution result of the robot executing the current action unit is a success and the current action unit is not the last one in the suggested task instruction, wherein a sub-task determining module 7071 is included in the seventh state determining module 707 and is configured to provide updated context information associated with the current scene to the visual language large model agent to obtain a sub-suggested task instruction for executing the next action unit provided by the visual language large model agent.

[0133] Although specific functions are discussed above with reference to specific modules, it should be noted that the functions of the various modules discussed herein can be split among multiple modules and / or at least some of the functions of multiple modules can be combined into a single module. A specific module performing an action as discussed herein includes that specific module itself performing the action, or alternatively that specific module invoking or otherwise accessing another component or module that performs the action (or that combines with the specific module to perform the action). Thus, a specific module performing an action can include the specific module itself performing the action and / or another module invoked or otherwise accessed by the specific module that performs the action.

[0134] It should also be understood that the above regarding Figure 7The described modules can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuit. For example, these modules can be implemented together in a System on Chip (SoC). An SoC may include an integrated circuit chip (which includes one or more components in a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0135] According to one aspect of this disclosure, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0136] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0137] According to one aspect of this disclosure, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0138] In the following text, combined with Figure 8 Illustrative examples describing such computer devices, non-transitory computer-readable storage media, and computer program products.

[0139] Figure 8 An example configuration of a computer device 800 that can be used to implement the methods described herein is shown.

[0140] Computer device 800 can be a variety of different types of devices. Examples of computer device 800 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablets, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on.

[0141] The computer device 800 can include at least one processor 802, memory 804, communication interface(s) 806, display device 808, other I / O devices 810, and one or more mass storage devices 812, which can communicate with one another via a system bus 814 or other appropriate connection.

[0142] The processor 802 can be a single processing unit or a plurality of processing units, all of which can include single or multiple computing units or multiple cores. The processor 802 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and / or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor 802 can be configured to fetch and execute computer-readable instructions stored in the memory 804, the mass storage device 812, or any other computer-readable medium, such as program code for an operating system 816, program code for applications 818, program code for other programs 820, and the like.

[0143] The memory 804 and the mass storage device 812 are examples of computer-readable storage media for storing instructions which are executed by the processor 802 to implement the various functionalities described above. For example, the memory 804 can generally include both volatile memory and nonvolatile memory (e.g., RAM, ROM, etc.). Further, the mass storage device 812 can generally include hard disk drives, solid state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CD, DVD), storage arrays, network attached storage, storage area networks, etc. Both the memory 804 and the mass storage device 812 can be collectively referred to herein as the memory or computer-readable storage media, and can be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that can be executed by the processor 802 as a particular machine configured to implement the operations and functionalities described in the examples herein.

[0144] A plurality of programs can be stored on the mass storage device 812. These programs include an operating system 816, one or more application programs 818, other programs 820, and program data 822, and they can be loaded into the memory 804 for execution. Examples of such application programs or program modules can include computer program logic (e.g., computer program code or instructions) for the methods described herein and / or for additional embodiments described herein.

[0145] Although the various examples described herein can make reference to computer-executable instructions, software, programs, applications, modules, logic, etc., it is to be understood that the example computer programs, applications, modules, etc. can be written in any of numerous programming languages or high-level, low-level, object-oriented, visual, compiled, or interpreted programming languages, including C, C++, Java, Visual Basic, Python, JavaScript, Flash, etc. It will be appreciated that a variety of programming Figure 8The operating system 816, the application programs 818, the other programs 820 and the program data 822, or portions thereof, can also be downloaded via computer readable media for use within computer device 800. Computer readable media, as used herein, includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired computer program code in the form of computer readable instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, or twisted pair, then the coaxial cable, fiber optic cable, or twisted pair are included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, Blu-ray disc, and floppy disk where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer readable media.

[0146] In some examples, a display device 808 such as a monitor can be included for displaying information and images to a user. Other I / O devices 810 can be devices that receive various inputs from a user and provide various outputs to the user, and can include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, etc.

[0147] One or more communication interfaces 806 are used to exchange data with other devices, such as over a network, a direct connection, etc. Such communication interfaces can be one or more of: any type of network interface (e.g., network interface cards (NICs)), wired or wireless (such as IEEE 802.11 wireless LAN (WLAN)) wireless interfaces, Worldwide Interoperability for Microwave Access (Wi-MAX) interfaces, Ethernet interfaces, Universal Serial Bus (USB) interfaces, cellular network interfaces, Bluetooth interfaces, near field communication (NFC) interfaces, etc. The communication interfaces 806 can facilitate communications within a variety of network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. The communication interfaces 806 can also provide for communication with external storage devices (not shown), such as storage arrays, network attached storage, storage area networks, etc. TM

[0148] In some examples, a display device 808 such as a monitor can be included for displaying information and images to a user. Other I / O devices 810 can be devices that receive various inputs from a user and provide various outputs to the user, and can include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, etc.

[0149] ​The techniques described herein can be supported by these various configurations of the computer device 800 and are not limited to the specific examples carried through the disclosure. For example, functionalities can also be implemented through other computing systems that are in "the cloud" via remote processing or computation. The cloud includes and / or comprises one or more servers, as well as, possibly, database and software applications. The cloud can also mediate access for computer device 800 to services, including via application programming interfaces (APIs).

Claims

1. A robot control method based on a decision state machine, characterized in that, The method includes: Receive an initial task instruction to enter a first state, wherein, in the first state, a first atomic skill sequence for the robot to execute the initial task instruction is determined, wherein the first atomic skill sequence includes at least one first atomic skill having a first predetermined order, and each first atomic skill is associated with a corresponding action unit of the robot for executing the initial task instruction; In response to determining the first atomic skill sequence, the robot enters a second state, wherein, in the second state, the robot is controlled to execute the initial task instruction based on the first atomic skill sequence; In response to determining that the robot's execution of the initial task instruction results in failure and the number of failures is greater than or equal to a preset threshold, the robot enters a third state, wherein, in the third state, Contextual information associated with the failure scenario is provided to the visual language large model agent to obtain suggested task instructions from the visual language large model agent; and The suggested task instruction is converted into a second atomic skill sequence, wherein the second atomic skill sequence includes at least one second atomic skill having a second predetermined order, and each second atomic skill is associated with a corresponding action unit of the robot for executing the suggested task instruction; In response to obtaining the second atomic skill sequence, the robot enters a fourth state, wherein, in the fourth state, the robot is controlled to execute the suggested task instruction based on the second atomic skill sequence to obtain a second execution result; and In response to determining that the second execution result of the robot executing the suggested task instruction is successful, the robot enters a fifth state, which instructs the robot to complete the task.

2. The method according to claim 1, characterized in that, In the first state, determining the first atomic skill sequence for the robot to execute the initial task instruction includes: Select at least one first atomic skill from the atomic skill library in the first predetermined order that matches the initial task instruction; and For each first atomic skill, the execution parameters of the first atomic skill are determined based on real-time acquired environmental perception data, wherein the environmental perception data is used to perform structured data representation of the robot's task environment.

3. The method according to claim 2, characterized in that, In the third state, contextual information associated with the failure scenario is provided to the visual language large model agent to obtain suggested task instructions provided by the visual language large model agent, including: Determine the failure type corresponding to the failure scenario; Generate a prompt word for the failure type, wherein the contextual information associated with the failure scenario is incorporated into the prompt word, and the prompt word includes at least a first prompt word portion and a second prompt word portion, wherein the first prompt word portion is used to indicate analysis of the failure scenario, and the second prompt word portion is used to indicate providing a solution strategy for the failure scenario; and The prompt word is sent to the visual language big data agent.

4. The method according to claim 3, characterized in that, The generation of prompt words for the failure type includes: Determine whether there is a failure diagnosis label corresponding to the failure type in a preset failure label mapping table, wherein the failure label mapping table is used to indicate the correspondence between the failure type and the failure diagnosis label, and the failure diagnosis label has a corresponding prompt word template, the prompt word template including a placeholder for integrating the context information; In response to determining the existence of the failure diagnostic label, a context variable corresponding to the placeholder is determined from the context information; and The context variables are populated into the prompt word template to generate the prompt word.

5. The method according to claim 1, characterized in that, The step of converting the suggested task instruction into a second atomic skill sequence includes: The suggested task instructions are parsed using natural language to determine at least one action unit, each action unit including an extracted verb and an entity associated with the verb; For each action unit, determine the identifier corresponding to the entity, and the parameters corresponding to the action represented by the verb; and Based on a preset action mapping rule base, at least one symbolic function with a second predetermined order is determined according to the action, parameters and identifier corresponding to each action unit. The second atomic skill sequence includes the at least one symbolic function, wherein each action unit is mapped to one or more of the symbolic functions.

6. The method according to claim 1, characterized in that, The method further includes: In response to determining that the first execution result of the robot executing the initial task instruction is a failure and the number of failures is less than the preset threshold, the robot enters a sixth state, wherein, in the sixth state, the robot is controlled to execute a predefined local recovery strategy to resolve the failure.

7. The method according to claim 1, characterized in that, In response to the suggested task instruction corresponding to multiple action units, the second execution result includes the sub-execution results obtained by the robot executing each of the multiple action units, wherein the method further includes: In response to determining that the robot has successfully executed the sub-execution result of the current action unit and that the current action unit is not the last of the suggested task instructions, the robot enters a seventh state. In the seventh state, updated context information associated with the current scene is provided to the visual language big model agent to obtain a sub-suggested task instruction provided by the visual language big model agent for executing the next action unit.

8. A robot, characterized in that, The robot is controlled to perform a task according to any one of claims 1 to 7.

9. A robot control device based on a decision state machine, characterized in that, The device includes: A first state determination module is configured to receive an initial task instruction to enter a first state. The first state determination module includes a first sequence determination module configured to determine a first atomic skill sequence for the robot to execute the initial task instruction. The first atomic skill sequence includes at least one first atomic skill having a first predetermined order, and each first atomic skill is associated with a corresponding action unit of the robot for executing the initial task instruction. The second state determination module is configured to enter a second state in response to determining the first atomic skill sequence, wherein the second state determination module includes an initial task execution module configured to control the robot to execute the initial task instruction based on the first atomic skill sequence; The third state determination module is configured to enter a third state in response to determining that the robot's execution of the initial task instruction results in failure and the number of failures is greater than or equal to a preset threshold. The third state determination module includes a suggested task determination module and a second sequence determination module. The task suggestion determination module is configured to provide contextual information associated with the failure scenario to the visual language large model agent to obtain a task suggestion instruction provided by the visual language large model agent; and The second sequence determination module is configured to convert the suggested task instruction into a second atomic skill sequence, wherein the second atomic skill sequence includes at least one second atomic skill having a second predetermined order, and each second atomic skill is associated with a corresponding action unit of the robot for performing the suggested task instruction; A fourth state determination module is configured to enter a fourth state in response to obtaining the second atomic skill sequence, wherein the fourth state determination module includes a suggested task execution module configured to control the robot to execute the suggested task instruction based on the second atomic skill sequence to obtain a second execution result; and The fifth state determination module is configured to enter a fifth state in response to determining that the second execution result of the robot executing the suggested task instruction is successful, the fifth state indicating that the robot has completed the task.

10. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-7.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-7.

Images