Robot task planning method based on visual language model and temporal logic regulation

Abstract

The application relates to the technical field of artificial intelligence, and is a robot task planning method and system based on a visual language model and time sequence logic regulation, which improves regulation satisfaction and controllability of long-range multi-constraint task planning. The method steps are as follows: a planning context is constructed and multi-modal joint understanding is performed; a task state proposition truth value set is constructed, natural language task requirements are converted into corresponding time sequence logic regulation, and are compiled into a time sequence logic constraint state machine used for representing a task logic advancing state, and a constraint state is updated according to the task state proposition truth value set; a to-be-checked word element is selected to form a to-be-checked action word element sequence, and a complete structured candidate action set is determined; a mapping relationship among the candidate action, the planning state and the time sequence logic constraint state is established; prospective checking of the candidate action is performed based on the time sequence logic constraint state machine, visual language model constraint decoding is performed, a complete structured robot action is generated, and a structured robot task plan is formed.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a robot task planning method and system based on visual language models and temporal logic reduction. Background Technology

[0002] Task planning is a crucial technical step for robots to generate continuous action sequences based on user objectives, environmental conditions, and their own capabilities. As robot applications evolve from single-step operations to long-range, complex tasks, user instructions typically include multiple requirements, such as target states, action sequences, execution prerequisites, and safety constraints. For example, a robot may not only need to grasp and place a specified object but also ensure that certain operations are executed before others and that it avoids specific objects or areas throughout the task. Therefore, robot task planning requires understanding natural language and the visual scene to control whether the action sequence continuously satisfies the task specification.

[0003] Currently, robot task planning is increasingly employing large language models or visual language models to jointly understand user commands and scene information, generating action suggestions, skill call sequences, or structured task plans. These methods leverage the semantic understanding and open generative capabilities of the underlying models to map natural language tasks into robot actions, thereby improving the robot's adaptability to complex tasks and open scenarios to some extent.

[0004] However, visual language models typically generate task plans using autoregressive methods based on lexical probabilities, with action selection primarily determined by semantic associations and generation preferences learned from the training data. In long-range tasks with multiple constraints, while the model may generate actions semantically relevant to the task objective, it struggles to consistently guarantee that each action satisfies preconditions, sequential relationships, and safety restrictions. For example, the model might generate the target action prematurely before the preconditions are completed, might overlook prohibited objects, or might generate actions inconsistent with the current robot state. As the action sequence grows, these local errors can accumulate throughout the subsequent planning process, resulting in semantically reasonable but logically unexecutable or unsafe planning outcomes.

[0005] Existing robot planning methods based on temporal logic reduction can represent task objectives, sequential relationships, and safety conditions as formalized reductions, and determine whether the task trajectory satisfies the reductions through automata, graph search, or policy solving. These methods improve the expressiveness of task constraints and the verifiability of planning results, but they typically operate on predefined state spaces, action sets, or already formed planned trajectories. Temporal logic reduction is more often used as a constraint during the search process or as a verification basis after planning is completed, and has not yet established a direct connection with the word-by-word generation of actions by visual language models. Therefore, it is difficult to eliminate generation branches that violate the reductions in time before the model officially outputs the action.

[0006] Furthermore, existing finite-state constraint decoding techniques for large models typically restrict the lexical, grammatical, or data format of the model output based on the matching relationship between text prefixes and preset grammar. While these techniques can determine whether the next lexical unit can still form correctly formatted text, they do not further establish the connection between lexical units, robot actions, and the task state after action execution. Therefore, it is difficult to determine whether correctly formatted action text satisfies the current robot state, action preconditions, and task timing specifications.

[0007] In summary, existing technologies lack a planning method for long-term, multi-constraint tasks of robots that enables temporal logic reduction to establish a direct link with the actions generated by task planning, and to perform forward-looking checks and judgments on robot actions and task state consequences. Summary of the Invention

[0008] This invention proposes a robot task planning method and system based on visual language models and temporal logic reduction. It aims to solve the problem that in the existing technology, when visual language models are used for robot long-term multi-constraint task planning, task reduction is difficult to directly apply to the task planning generation process. This may lead to the model generating planning content that is inconsistent with the scene state, robot capabilities, action preconditions or task timing requirements, and cause constraint omissions and error accumulation in the multi-step generation process.

[0009] In a first aspect, the present invention provides a robot task planning method based on a visual language model and temporal logic reduction, comprising the following steps:

[0010] S1. Based on natural language task instructions, robot initial environment observation information and robot capability description, the task objective, operation object, spatial relationship, execution conditions and safety requirements are identified through a visual language model and jointly encoded to construct a planning context; the planning context is subjected to multimodal joint understanding to obtain the task semantic representation and initial planning state;

[0011] S2. Based on the task semantic representation and initial planning state, determine the task state propositions that can reflect the task objectives, execution order, preconditions and safety constraints, construct the truth set of task state propositions, and convert the natural language task requirements into the corresponding temporal logic specifications.

[0012] S3. The sequential logic specification is reduced and compiled into a sequential logic constraint state machine to represent the progress state of the task logic, and its initial state, accepting state, violation state and state transition relationship are determined. The constraint state machine updates the constraint state according to the set of truth values ​​of the task state propositions corresponding to the robot's planning state, providing a basis for the prospective check of subsequent candidate actions.

[0013] S4. Based on the current planning context and the action lexicon prefixes already generated by the current structured action, the visual language model generates the robot's next structured action word by word in an autoregressive manner, obtaining the original output probability of each lexicon in the visual language model vocabulary; selects the lexicon that has not yet been masked and has the highest original output probability as the lexicon to be checked, and appends the lexicon to be checked to the current action lexicon prefix to form the action lexicon sequence to be checked, further determining the complete set of structured candidate actions that the action lexicon sequence to be checked can form;

[0014] S5. Based on the complete structured candidate action set, establish the mapping relationship between candidate actions, planning states, and temporal logic constraint states. The mapping relationship includes action action function and state labeling function. The action action function is used to predict the subsequent planning state formed after the candidate action is executed. The state labeling function is used to extract the true task state propositions from the subsequent planning states and form a set of truth values ​​of task state propositions that the temporal logic constraint state machine can read, thereby establishing the mapping relationship between the visual language model word generation space, robot action space, planning state space, and constraint state space.

[0015] S6. Input the set of truth values ​​of the task state proposition obtained in step S5 into the temporal logic constraint state machine constructed in step S3, and combine it with the current constraint state to calculate the next constraint state that may be reached after the candidate action is executed, so as to perform a prospective check on the task logic state that may be reached after the candidate action is executed.

[0016] The prospective check is used to determine, based on the next constraint state, whether the candidate action will cause the task specification to be violated or cause the subsequent task trajectory to lose the possibility of reaching the acceptance state before the candidate action is officially output by the visual language model.

[0017] S7. The temporal logic constraint state machine performs iterative constraint decoding on the vocabulary output of the current word position based on the prospective inspection results of the candidate action, before the visual language model officially outputs the next word, sets the original output probability of the word to be blocked to an unselectable state, and redetermines the next word within the range of other words.

[0018] The process involves word-by-word repetition of candidate actions, prospective checks, and constraint decoding until a complete structured robot action is generated.

[0019] S8. Add the generated complete structured robot motion to the current task planning sequence, and formally update the robot planning state and constraint state according to the generated complete structured robot motion, and generate a structured robot task plan.

[0020] Secondly, the present invention also provides a robot task planning system based on a visual language model and a temporal logic reduction, including at least one processor and a memory communicatively connected to the at least one processor; wherein, the memory stores program instructions that are executed by the at least one processor, and when the program instructions are executed by the at least one processor, the at least one processor performs steps S1-S8 of the above-described robot task planning method.

[0021] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0022] 1. Improve the consistency between constraint judgment and robot task state.

[0023] This invention establishes a mapping relationship between visual language model lexical units, structured candidate actions, predicted planning states, and task state propositions. This enables the temporal logic constraint state machine to judge the robot's state based on the possible consequences of candidate actions, rather than simply judging the text format or action expression. Therefore, constraint checking can directly correspond to task facts such as the operation object, target position, grasping state, action prerequisites, and safety state. This reduces planning results where the text expression is correct but the action logic is unreasonable, and improves the consistency between the task plan and the scene state, robot capabilities, and task requirements.

[0024] 2. Reduce violations and error accumulation in long-range multi-constraint programming.

[0025] This invention utilizes a temporal logic constraint state machine to perform a forward-looking check on the complete candidate actions that may be formed by the generated lexical units before their formal output. It also continuously records the task specification satisfaction progress corresponding to the generated action sequences. This allows the system to promptly block generation branches that violate action order, execution prerequisites, or safety constraints before these actions are written into the task plan. Consequently, it reduces constraint omissions, error propagation, and the resulting planning rollbacks and regenerations during multi-step planning, improving the safety, stability, and reliability of long-term multi-constraint task plan generation.

[0026] 3. Improve planning controllability while retaining the ability to generate visual language models.

[0027] This invention applies a constraint mask to the original output probability of the current lexical position in the visual language model based on the prospective check results of candidate actions. Lexical units that can only form illegal or invalid actions are set to an unselectable state, while maintaining the relative generation preference of the remaining legal lexical units. Thus, without changing the parameter structure and autoregressive generation mechanism of the visual language model, the model can generate robot actions within a reduced and satisfyable generation space, thereby balancing the reduction satisfaction of task planning, generation flexibility, and deployment compatibility. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall process of the robot task planning method based on visual language model and temporal logic reduction in an embodiment of the present invention. Detailed Implementation

[0029] This invention provides a robot task planning method and system based on a visual language model and a temporal logic constraint state machine. It transforms natural language tasks into temporal logic specifications and constructs a temporal logic constraint state machine to represent the task progression state. This constraint state machine is then integrated into the word-by-word generation process of the visual language model in real time. When the visual language model generates action planning sequences word-by-word, the temporal logic constraint state machine performs a prospective check on the potential robot action branches formed by the generated words. For words whose generated actions would all lead to a violation state, the system sets their original output probability to zero, effectively blocking generation branches that can only lead to a violation state. For words that can still form actions that satisfy the specification, their original output probability preference is retained. Thus, this invention introduces a constraint mask based on a temporal logic constraint state machine between the vocabulary output and word selection of the visual language model. Without changing the model parameter structure and autoregressive generation mechanism, it enables the visual language model to generate robot task plans within a specification-satisfiable generation space according to its original output probability preference, improving the specification satisfaction and generation controllability of long-term multi-constraint task plans.

[0030] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0031] Example

[0032] This embodiment provides a robot task planning method based on visual language models and temporal logic reduction, such as... Figure 1 As shown, the specific steps include:

[0033] S1. Constructing a planning context and performing multimodal joint understanding: Based on natural language task instructions, robot initial environment observation information, and robot capability description, the task objective, operation object, spatial relationship, execution conditions, and safety requirements are identified through a visual language model and jointly encoded to construct a planning context; multimodal joint understanding of the planning context is performed to obtain the task semantic representation and initial planning state.

[0034] Specifically, the visual language model processes user-input natural language task instructions through both text encoding channels and visual encoding channels. Initial environmental observation information of the robot Description of robot capabilities Perform joint encoding and form a planning context within the model. .

[0035] The natural language task instructions The instructions are natural language commands for robot operation tasks, which include one or more task types such as object grasping, object movement, object classification, object stacking, push-pull tasks, and opening-closing tasks. The natural language task commands further include constraint information such as task objectives, execution order, preconditions for actions, condition triggering relationships, priority requirements, safety maintenance conditions, and prohibited areas.

[0036] The robot's initial environmental observation information This includes color images, depth information, and structured environmental data input to the object detection model.

[0037] The robot's capability description Includes a set of robot action primitives The end effector type and status, available workspace, load capacity, and motion constraints are specified. The set of robot motion primitives is also included. This is used to represent the basic action types that can be invoked by the robot during the task planning process. The basic action types include at least one or more of the following: movement, grasping, releasing, lifting, posture adjustment, waiting, and holding. Each action primitive can be represented in a parameterized form, such as the grasping action. Movement Release action ,in Indicates the object being operated on. Indicates the target location, container, surface, or area.

[0038] Visual language models for planning context Perform multimodal joint understanding to obtain the task semantic representation. Compared with the initial planning state .

[0039] The task semantic representation Semantic information related to task planning includes task objects, target locations, available actions, task objectives, sequence relationships, preconditions, safety conditions, and prohibition conditions.

[0040] The initial planning state Used to represent the state facts related to planning at the start of a task, including object location, spatial relationships, grasping status, end effector status, obstacles, risky objects, and restricted areas.

[0041] S2. Construct task state propositions and generate temporal logic specifications: Based on the task semantic representation and initial planning state, determine task state propositions that can reflect task objectives, execution order, preconditions and safety constraints, construct a set of truth values ​​for task state propositions, and convert the natural language task requirements into corresponding temporal logic specifications.

[0042] Specifically, it includes the following steps:

[0043] S21. Based on the task semantic representation, initial planning state, and robot capability description, construct a set of truth values ​​for task state propositions.

[0044] The visual language model is based on the task semantic representation obtained in step S1. Initial planning status and the set of robot action primitives This involves identifying the target facts, process facts, and safety facts that require logical judgment in the current task, and constructing a truth set for task state propositions based on these facts. , denoted as:

[0045] ;

[0046] Set of truth values ​​for task state propositions Truth value of each task state proposition in Each corresponds to a basic task fact that can be determined as true or false based on the robot's planning state. The basic task facts include one or more of the following: object location, grasping state, action occurrence state, action completion state, goal achievement state, action prerequisites, safe state, and risk state.

[0047] The truth value of a task state proposition can be expressed in predicate form:

[0048] ;

[0049] in, These represent location predicates, state predicates, action progress predicates, conditional predicates, or risk predicates related to task planning. It represents parameters such as object, location, container, surface, action, risk object, or restricted area.

[0050] In this embodiment, based on the persistence mode of the described task facts, task state propositions are divided into persistent state propositions and action completion propositions. Persistent state propositions are used to describe task states that remain true throughout a planning process, for example, This indicates that the robot is holding an apple; the action completion proposition is used to describe that a specific action or subtask has been completed, for example, This indicates that the lemon has been placed on the plate.

[0051] S22. Based on the constructed set of truth values ​​of task state propositions, construct a temporal logic reduction using logical operators and temporal operators.

[0052] Specifically, the visual language model is based on the set of truth values ​​for propositions related to the task state. By combining task objectives, the sequence of actions, the prerequisites for action execution, safety conditions, prohibition conditions, and condition triggering relationships, a temporal logic reduction is constructed using logical operators and temporal operators. Sequential logic reduction It is used to describe the target achievement requirements, task progression sequence, action execution prerequisites, and safety limitations of a robot in a limited task trajectory.

[0053] Furthermore, the system reduces the generated timing logic specification. Perform syntactic validity and semantic consistency checks, and reduce the temporal logic specification if it fails the checks. The system performs regeneration, rule correction, or manual confirmation to ensure that the propositions, objects, and logical relationships in the temporal logic specification correspond to the current task.

[0054] In a preferred embodiment, the timing logic specification It is represented by a finite trajectory linear time-series logic formula, and its basic syntax includes logical NOT. Logic and , logic or The next moment (Next), until (Until), Finally, and Globally Persistent Operators such as (Globally). For example, the goal of "finally placing the apple inside the red bowl" can be represented as: The prerequisite for the action "grab the apple, but make sure the lemon is in the plate first" can be represented as:

[0055] .

[0056] S3. Construct a temporal logic constraint state machine: Compile the temporal logic specification into a temporal logic constraint state machine to represent the progress of task logic (i.e., the progress of task constraint satisfaction), and determine its initial state, accepting state, violation state, and state transition relationship; The constraint state machine updates the constraint state according to the set of truth values ​​of the task state propositions corresponding to the robot's planning state, providing a basis for the prospective check of subsequent candidate actions (i.e., specification satisfiability judgment).

[0057] In this embodiment, the timing logic constraint state machine is used to determine whether the generated action sequence satisfies the task specification and whether there is still a possibility of completing the task.

[0058] Specifically, from the perspective of temporal logic reduction The compilation of a sequential logic constraint state machine can be achieved by calling an automaton compilation tool, by converting it using preset state construction rules, or by a combination of the above methods.

[0059] In a preferred embodiment, the system compiles the linear temporal logic formula of the finite trajectory into a deterministic finite state automaton, denoted as . ;

[0060] in, This represents a deterministic finite automaton. Represents the set of constraint states. This indicates that the alphabet should be entered. Represents the state transition function. The initial state of the representation This represents the set of accepting states.

[0061] The set of constraint states This is used to represent the progression stage of the task trajectory prefix corresponding to a generated complete action sequence in the temporal logic specification. The progression stage includes states such as the task objective not yet achieved, some prerequisite objectives achieved, all task objectives satisfied, and the current task trajectory irreversibly violating the task specification.

[0062] The input alphabet The set of truth values ​​of the task state propositions obtained from step S2 The truth value assignment is denoted as ,

[0063] in Represents the set of truth values ​​for the propositions concerning the task state. The power set of the alphabet. Input each symbol from the alphabet. This represents a subset of truth values ​​for task state propositions that hold true under a given robot planning state.

[0064] The state transition function This is used to calculate the next constraint state based on the current constraint state and the set of truth values ​​of the input task state propositions, denoted as:

[0065] ;

[0066] ;

[0067] in, Indicates the current constraint state. This indicates the next constraint state obtained after reading the current constraint state.

[0068] The initial state of the form This represents the logical state of a deterministic finite state automaton before it has read the robot's initial planned state.

[0069] The set of acceptance states This is used to indicate that the constraint state has been satisfied by the sequential logic reduction. When the deterministic finite state automaton enters any accepting state... When, it means that the finite task trajectory corresponding to the currently generated complete action sequence has satisfied the temporal logic specification.

[0070] Furthermore, for deterministic finite state automata, we define the set of trapped states:

[0071] ;symbol" "Indicates that it is contained in;

[0072] When the constrained state enters the deadlock state set When this occurs, it indicates that the current planned path has violated the task specification, and the corresponding violation cannot be eliminated by subsequent actions. At this point, for any input symbol... All satisfy:

[0073] ;

[0074] Therefore, once a deterministic finite automaton enters a deadlock state, it cannot re-enter a reduced satisfyable path regardless of the subsequent input.

[0075] Define a function that accepts state reachability:

[0076] ;

[0077] when When, it indicates that the current task trajectory is still on a path that can be satisfied by the reduction; when When this occurs, it means that continuing to generate subsequent actions from the current constraint state is no longer possible to bring the task trajectory to the set of accepting states.

[0078] S4. Obtain the real-time vocabulary output of the visual language model and form candidate actions: Based on the current planning context and the action word prefixes already generated by the current structured action, the visual language model generates the robot's next structured action word by word in an autoregressive manner, obtaining the original output probability of each word in the visual language model's vocabulary; select the word that has not yet been masked and has the highest original output probability as the word to be checked, and append the word to be checked to the current action word prefix to form the sequence of action words to be checked, further determining the complete set of structured candidate actions that the sequence of action words to be checked can form.

[0079] Suppose the action prefixes generated by the current structured action have been... for:

[0080] ;

[0081] in, Indicates the position of word generation in the current structured action. This indicates the first generated and confirmed action in the current structured action. There are lexical elements; when no lexical element for the current action has been generated yet, it is an empty prefix, denoted as . .

[0082] In the current structured action, the first At each lexical generation location, the visual language model obtains the original output probability of each lexical in the vocabulary based on the current planning context and the already generated lexical prefixes:

[0083] ;

[0084] in, A vocabulary representing a visual language model. Indicates word elements The original output probability at the current generation position.

[0085] The system prioritizes selecting the words that are not currently blocked and have the highest probability of being output in the original output. As the term to be checked:

[0086] ;

[0087] in, Indicates the check round at the current word position; Indicates preceding The set of words that have been masked during round checking initially satisfies ,symbol Represents the empty set; Indicates from the vocabulary list Remove the set of words that have been blocked. The remaining set of lexical units after the initial check represents the set of lexical units that are currently allowed to continue participating in the check, where the symbol " " represents the difference between sets; This indicates selecting the word whose original output probability is maximized from the set of words currently allowed to continue participating in the inspection.

[0088] For the word to be checked The system appends it to the prefix of the current action verb, forming a sequence of action verbs to be checked. :

[0089] ;

[0090] in, This indicates a word appending operation.

[0091] Based on the action primitives, structured action grammar, and parameter forms of action predicates in the robot's capability description, as well as the object, position, and region parameters in the current task, the system determines the complete set of structured candidate actions that the sequence of action lexies to be examined can form. :

[0092] ;

[0093] in, Indicates the prefix relationship between word sequences. Represents a complete structured action The corresponding lexical sequence.

[0094] When the sequence of action lexes to be checked has not yet formed a complete action, the system completes the action according to the parameter type required by the action predicate, and obtains one or more complete structured candidate actions.

[0095] For example, at the first lexical position of the first action, the visual language model outputs the lexical with the highest probability as... The system completes the structured action candidate into possible full structured action candidates based on the structured action rules and objects in the scene:

[0096] ;

[0097] in, This means grabbing an apple. It means to grab a lemon. It means to grab a knife. It means to grab the cup.

[0098] Due to the current word element Since the target of the operation has not yet been uniquely determined, the system considers all complete structured actions that match the prefix of this word as candidate actions.

[0099] In actual word segmentation, the action predicate, object name, and position name can be composed of one or more sub-word elements. The system determines candidate actions based on the prefix relationship of the complete word element sequence, rather than requiring each action parameter to correspond to only one word element.

[0100] In other words, in this step, the system selects the word with the highest original output probability in the visual language model vocabulary as the word to be checked. Based on the currently generated action word sequence, the word to be checked, the robot action expression rules, and task parameters, it determines one or more complete structured candidate actions that the word with the highest original output probability may continue to form.

[0101] S5. Establish the mapping relationship between candidate actions and temporal logic constraint states: Based on the complete structured candidate action set obtained in step S4, establish the mapping relationship between candidate actions, planning states, and temporal logic constraint states; the mapping relationship includes action action function and state labeling function. The action action function is used to predict the subsequent planning state formed after the candidate action is executed; the state labeling function is used to extract the valid task state propositions from the subsequent planning state and form a set of truth values ​​of task state propositions that the temporal logic constraint state machine can read, thereby establishing the mapping relationship between the visual language model word generation space, robot action space, planning state space, and constraint state space.

[0102] Specifically, this step includes:

[0103] S51. Predict the planning state after the candidate action is executed by using the action action function.

[0104] Action function Defined as:

[0105] ;

[0106] in, Represents the robot's planning state space. Represents the set of robot action primitives, symbolized by " "Indicates the function definition relationship, symbol " " " represents the Cartesian product, symbol " "" indicates a mapping relationship. The action function is used to describe the impact of the candidate action on the task-related state, including one or more changes in object position, grasping state, action completion state, safe state, and risk state.

[0107] For the complete set of structured candidate actions output in step S4 any candidate action in Calculate any candidate action The corresponding predicted successor planning state for:

[0108] ;

[0109] in, Indicates the first The current planning status of each planning step. This represents the action function.

[0110] The action function is represented by the action preconditions and action effect rules. For any candidate action... Define candidate actions The basic execution conditions are Define the state fact that is added or becomes true after the action is executed as follows: The state fact that is deleted or becomes false after the action is executed is defined as follows: If the current planning state meets the preconditions for the action:

[0111] ; where the symbol " " indicates a satisfied relationship;

[0112] The predicted successor planning state corresponding to the candidate action can then be represented as:

[0113] ;

[0114] in, Indicates the current planning status Delete candidate actions A state fact that no longer holds true after execution, symbolized by " " represents the set union operation.

[0115] For example, for candidate actions Its preconditions for action may include:

[0116] ;

[0117] Its positive effects can include: ;

[0118] Its negative effects can include: ;

[0119] in, This indicates that the end effector currently does not hold an operation object. This indicates that Apple is currently within the reach of the end effector, which can perform grasping operations, under the current planning state. This indicates that the end effector holds the apple. This indicates that the end effector has completed its grasping action on the apple. This indicates the location of the apple before it was captured. This indicates the location of the apple. The state of affairs.

[0120] Step S52: Generate the state machine input corresponding to the candidate action through the state marking function.

[0121] State marking function Defined as:

[0122] ;

[0123] in, Represents the set of truth values ​​for the propositions concerning task state. The power set, symbol " "Indicates the function definition relationship, symbol " "" indicates a mapping relationship. The state labeling function is used to convert planning information such as object position, grasping state, action completion state, safety state, and risk state in the robot task into logical inputs that can be recognized by the constraint state machine.

[0124] For candidate actions The system calculates the corresponding predicted successor planning state:

[0125] ;

[0126] in, Indicates the assumption that candidate actions will be performed. The set of truth values ​​for subsequent task state propositions; Let be the state labeling function, representing the set of truth values ​​for task state propositions that hold true under the corresponding planning state. The specific task state propositions included in the set of truth values ​​are determined by the definition of the proposition meanings and the action effect rules in step S2.

[0127] For example, for candidate actions The action function predicts that after the candidate action is executed, the robot will be in a state holding the apple, the end effector will no longer be idle, and the apple will leave its original position. The state labeling function, based on the predicted subsequent planning state, can obtain:

[0128] .

[0129] in, Indicates the current planning status. Under the assumption that candidate actions will be executed Then, the predicted successor planning state is obtained from the action function, and the set of truth values ​​of the task state proposition is obtained by mapping through the state labeling function; This indicates that the end effector holds the apple. This indicates that the end effector has completed its grasping action on the apple.

[0130] S6. Prospective checking of candidate actions based on the state machine constrained by timing logic.

[0131] The set of truth values ​​of task state propositions corresponding to each candidate action obtained in step S5 is input into the temporal logic constraint state machine constructed in step S3. Combined with the current constraint state, the next constraint state that may be reached after the candidate action is executed is calculated, thereby performing a prospective check on the possible task logic state that may be reached after the candidate action is executed. This prospective check is used to determine, before the candidate action is formally output by the visual language model, whether the corresponding candidate action will lead to a violation of the task specification or cause the subsequent task trajectory to lose the possibility of reaching the accepting state, based on the next constraint state.

[0132] S61. Determine the temporal logic state basis for the current planning step.

[0133] Let the visual language model be in the first... The complete sequence of actions that has been generated and confirmed before each planning step. for:

[0134] ;

[0135] in For the first Each planning step generates and confirms a complete structured robot motion.

[0136] In the first planning step, the system is based on the initial planning state. Through the state flag function Obtain the set of truth values ​​for the initial task state proposition. Specifically: ;

[0137] The set of truth values ​​of the initial task state proposition Input the sequential logic constraint state machine to obtain the initial constraint states actually used for task planning. for:

[0138] ;

[0139] in This represents the initial state in the form of the sequential logic constraint state machine obtained by sequential logic reduction compilation.

[0140] In the In each planning step, the system progresses from the initial planning state. Depart, following the complete sequence of actions The actions are generated sequentially by calling the action function one after another to obtain the current planning state. :

[0141] ;

[0142] in, This indicates that the action function follows the generated complete action sequence. The continuous action process is specifically represented as follows:

[0143] ;

[0144] in, Indicates the first The planning state at the beginning of each planning step. Indicates the first Each planning step generates and confirms a complete structured robot motion. Indicates preceding After the first complete structured robot action, the... The planning state at the start of each planning step.

[0145] In sync with the planning state update, the system obtains the set of truth values ​​for the task state propositions corresponding to each planning state through a state marking function. : ;

[0146] And update the constraint state according to the order in which the actions are generated in the complete action sequence: ;

[0147] in For the first The constraint states corresponding to the start of each planning step Indicates the first The planning state is formed after a complete structured robot action is generated and confirmed. The set of truth values ​​for the corresponding task state propositions.

[0148] In the During the online generation of each planning step, the system uses the current planning state... and current constraint state This serves as the state basis for prospective examination of candidate actions.

[0149] S62. Determine the specification satisfiability of candidate actions.

[0150] The system inputs the set of truth values ​​of the task state propositions corresponding to the candidate actions into the temporal logic constraint state machine constructed in step S3. The temporal logic constraint state machine then adjusts the state according to the current constraint state. Calculate the next constraint state :

[0151] ;

[0152] Based on the above next constraint state, the sequential logic constraint state machine determines whether its corresponding candidate action satisfies the following two conditions:

[0153] ;

[0154] as well as: ;

[0155] in, This represents the set of deadlocked states in a sequential logic constrained state machine. This represents the reachability function for the accepted states. The first condition determines whether a candidate action would cause the current task trajectory to directly enter an unrecoverable violation state; the second condition determines whether, after executing the candidate action, there is still a way to reach the set of accepted states through subsequent actions. Possible paths.

[0156] When both conditions are met, the sequential logic constraint state machine will select candidate actions. The action is determined to be satisfyable by the specification: When the next constraint state enters a deadlock state, or when it is impossible to reach the set of acceptable states from the next constraint state, the sequential logic constraint state machine determines the candidate action as an unretainable action: ;in This indicates the result of the specification satisfiability judgment for the candidate action.

[0157] S7. Perform visual language model constraint decoding.

[0158] Based on the prospective check results of the candidate actions obtained in step S6, the temporal logic constraint state machine performs iterative constraint decoding on the vocabulary output at the current word position before the visual language model officially outputs the next word. It sets the original output probability of the word to be blocked to an unselectable state and redetermines the next word within the remaining word range. This constraint decoding process only excludes generation branches that cannot form actions that can be satisfied by the reduction, without changing the relative generation preference of unblocked words.

[0159] The system repeats candidate action formation, prospective checking, and constraint decoding word by word until a complete structured robot action is generated.

[0160] When the word to be checked If at least one candidate action in the corresponding complete structured candidate action set passes the prospective check, it means that the term to be checked can still be extended to form a complete robot action that can be satisfied by the specification, and the system retains the term to be checked; when the term to be checked cannot form a valid structured action, or all its corresponding candidate actions violate the task specification, the corresponding term to be checked is determined as a term to be blocked, and the blocking set is updated.

[0161] Specifically, when the word to be inspected When at least one complete candidate action that can be satisfied by the specification can be formed, the system determines the corresponding lexical to be checked as the next lexical at the current generation position:

[0162] ;

[0163] in Indicates the first The word generation positions are the words that are ultimately selected through prospective checks.

[0164] The visual language model calculates the original output probability of the next word position based on the updated action word sequence and follows the original autoregressive generation method, and then proceeds to the generation process of the next word position.

[0165] When the word to be checked If all candidate actions fail the temporal logic reduction check, the system will identify the corresponding term to be checked as a term to be blocked and update the blocking set.

[0166] ;

[0167] in Indicates preceding The set of words that have been blocked during round checking. Indicates preceding The set of words that are blocked during round checking.

[0168] The system constructs a vocabulary-level constraint mask based on the updated mask set. :

[0169] ;

[0170] symbol" "Indicates not belonging, symbol " "Indicates belonging to".

[0171] The vocabulary-level constraint mask is applied to the original output probability of the visual language model at the current word position in the vocabulary. Obtain the constrained lexical probabilities:

[0172] ;

[0173] Subsequently, normalization is performed on the constrained lexical probabilities:

[0174] ;

[0175] in, This refers to any word in the vocabulary used to perform normalization. This indicates that after several rounds (in this embodiment, the [number]th round)... Normalized lexical probabilities after round constraint processing.

[0176] Indicates the first The planning step, the first The generation position of the lexical unit, after the th ... After processing and updating the constraint mask, the tokens are checked in turn. The corresponding constrained lexical probabilities are determined by the original output probabilities of the visual language model and the corresponding constraint mask. This indicates any word in the vocabulary. According to The constrained lexical probabilities are obtained in the same way.

[0177] For lexical units that can only lead to expressions of illegal actions, their constrained lexical unit probability is zero; for lexical units that may still lead to actions that can be satisfied by the specification, their relative generation preference is still determined by the original output probability of the visual language model.

[0178] The visual language model selects the next lexical unit based on the normalized lexical unit probabilities after constraint processing and appends it to the current action prefix. Subsequently, the system re-executes steps S4 to S7 until a complete structured robot action is formed. .

[0179] During the current generation of complete structured robot actions, the current planning state and current constraint state If this remains unchanged, the sequential logic constraint state machine only calculates the predicted state corresponding to each action branch.

[0180] S8. Update planning status and complete task plan generation: Add the generated complete structured robot motion to the current task planning sequence, and formally update the robot planning status and constraint status according to the generated complete structured robot motion, and generate a structured robot task plan.

[0181] The structured robot task plan can be further converted into action primitives, robot skill instructions, or trajectory control instructions that can be invoked by the robot control system.

[0182] Specifically, the complete structured robot motion output in step S7 Add to the already generated task planning sequence :

[0183] ;

[0184] According to the corresponding complete structured robot actions The system updates the planning status sequentially. Constraint state As input for the next planning step.

[0185] When the updated constraint state has not yet entered the set of acceptable states, but is still on the path that can be satisfied by the specification, the system returns to step S4 and continues to generate the next complete structured robot action.

[0186] When the updated constraint state enters the set of acceptable states and the task objective has been met, the system terminates the task planning process and outputs a complete structured robot task plan. :

[0187] .

[0188] in, Indicates the first Each planning step generates and confirms a complete structured robot motion; This represents the planning step number of the action in the task planning sequence, satisfying... ; This represents the total number of complete structured robot actions generated and confirmed in order to complete the current task.

[0189] The structured robot task plan can be further converted into action primitives, robot skill instructions, or trajectory control instructions that can be invoked by the robot control system.

[0190] Through the above technical solution, this embodiment establishes a planning mechanism in which a visual language model proposes action generation preferences, and a temporal logic constraint state machine eliminates illegal generated branches in real time. The visual language model still forms action content word by word in the original way, while the temporal logic constraint state machine checks the possible robot state consequences before the word is officially output, thereby improving the specification satisfaction of long-term multi-constraint robot task plans while retaining the model's open generation capabilities.

[0191] Preferably, by constructing supervised data that is strictly aligned between natural language task instructions, task reasoning information, temporal logic reduction, and high-efficiency planning trajectory, the parameters of the visual language model are jointly fine-tuned, enabling it to simultaneously learn the transformation relationship from task semantics to temporal logic reduction, as well as the generation relationship from task constraints to high-efficiency action sequences.

[0192] Through the aforementioned joint fine-tuning, on the one hand, the visual language model's ability to recognize semantic information such as task objectives, action sequences, execution prerequisites, and safety constraints can be improved, reducing the probability of proposition omissions, logical relationship errors, and object correspondence errors during the generation of temporal logic specifications. This improves the accuracy of converting natural language task instructions into temporal logic specifications, thus making it more suitable for the robot task planning method of this embodiment. On the other hand, the visual language model can learn efficiency features such as action steps, execution paths, task time, and redundant operations from planning samples that meet task constraints. This allows the model to form a preference for generating high-efficiency action sequences in its original generation distribution, thereby optimizing the length of the action sequences.

[0193] The fine-tuned visual language model possesses both accurate temporal logic reduction capabilities and a preference for generating low-cost planning sequences at the internal generation distribution level. In the subsequent constraint decoding process, the temporal logic constraint state machine is responsible for defining the legal generation space. The fine-tuned visual language model then prioritizes planning branches with fewer action steps, shorter paths, fewer redundant operations, or lower overall execution costs within this legal generation space.

[0194] In other words, the parameter fine-tuning of the visual language model is mainly used to improve the generation quality of the temporal logic specification and optimize the generation priority among legal candidate actions. The temporal logic constraint state machine and the visual language model generation space constraint decoding mechanism described in steps S3 to S7 are still used to exclude action generation branches that violate the task specification. Thus, while keeping the planning results constrained by the temporal logic specification, invalid action attempts and redundant planning are reduced, improving the generation efficiency and execution efficiency of the task plan.

[0195] Specifically, fine-tuning the parameters of the visual language model includes the following steps:

[0196] S101. Construct a supervised dataset for model fine-tuning.

[0197] To implement the efficiency preference learning mechanism, a supervised dataset for model fine-tuning is first constructed. This supervised dataset includes four types of information: natural language task instructions, thought chain reasoning text, temporal logic reduction, and task planning trajectories, with strict alignment achieved among these four types of information within the same training samples.

[0198] Preferably, each training sample can be represented as:

[0199] ;

[0200] in, Indicates the first Natural language task instructions and corresponding environmental context information for each sample. Indicates the relationship with the first The thought chain reasoning text corresponding to the natural language task instructions for each sample. This represents the temporal logic reduction obtained from task semantic parsing. This represents an expert-planned trajectory that satisfies the temporal logic limitation and has high efficiency characteristics.

[0201] Furthermore, the expert-planned trajectory To select the optimal planning sequence from the legal planning space that satisfies the temporal logic specification, it must satisfy at least one or more of the following efficiency conditions: fewer action steps, shorter movement path, fewer redundant operations, and lower overall execution cost.

[0202] Preferably, the supervised dataset is organized in a conversational supervised format, enabling the model to simultaneously learn the correspondence between natural language task understanding, reasoning process generation, temporal logic formula expression, and optimal execution plan generation.

[0203] S102. Define the efficiency preference learning objective function and set the fine-tuning parameters.

[0204] Based on the supervised dataset, the model is trained using autoregressive supervised training. The training objective employs an autoregressive cross-entropy loss function to minimize the negative log-likelihood of the expert-planned trajectory. The loss function for the training objective can be expressed as:

[0205] ;

[0206] in, This represents the total number of lexical units contained in the target output sequence corresponding to the current supervised training sample; The parameter is The visual language model provides the conditional generation probability, which is the probability generated given the natural language task instructions and environmental context information. Mind Chain Reasoning Text Timing logic specification and the The target output prefix before each output position Under the given conditions, the model generates the target output lexical units. The probability of.

[0207] The target output unit It includes at least reasoning fragments from thought chain reasoning texts, logical expression units from temporal logic formulas, and action sequence units from expert planning trajectories. This allows the training process to simultaneously optimize the model's ability to jointly generate reasoning processes, reduced expressions, and efficient execution plans.

[0208] The expert-planned trajectory is a highly efficient sequence of actions that satisfies logical constraints. Therefore, the optimization process of the above loss function can make the model's policy distribution approximate the expert policy, so that given the task semantics and environmental state, it tends to assign a higher generation probability to the more efficient candidate actions.

[0209] During the model training phase, a visual language model is selected as the base planning model, and its planning capabilities are enhanced using a parameter-efficient fine-tuning method. This method introduces low-rank trainable parameters into at least some attention layers, projection layers, or feedforward layers of the base planning model, while freezing the remaining basic parameters, thereby achieving capability transfer to the planning domain at a lower training cost.

[0210] S103. Employ a mask loss strategy to enhance reasoning and planning learning.

[0211] To prevent the model from weakening its learning effect on the logical reasoning and planning processes by simply mechanically completing the input instructions, this invention further adopts a mask-based loss calculation strategy.

[0212] Specifically, the training only targets thought chain reasoning texts. Timing logic specification and expert planning trajectory The gradient is calculated for the corresponding output part, while the natural language task instructions for the input are... The input context is masked and not included in the loss calculation.

[0213] Furthermore, a mask function can be defined:

[0214] ;

[0215] in This indicates that the current supervised training sample is at the th ... The target word corresponding to each sequence position. Then the mask loss function can be expressed as:

[0216] ;

[0217] in This indicates that the current supervised training sample is at the th ... The target word sequence preceding each sequence position.

[0218] By employing the mask loss strategy described above, the model can be forced to focus its learning on the path of generating reasoning processes, generating reduced representations, and generating efficient actions, rather than merely completing shallow parts at the input text level. This enhances the model's endogenous modeling ability for action-level planning and reasoning.

[0219] S104. A stable training strategy is adopted to achieve smooth transfer.

[0220] To prevent catastrophic forgetting during fine-tuning and to ensure the stability of model training, this invention further employs a stable training strategy during the training process.

[0221] Specifically, the AdamW (Adaptive Moment Estimation) optimizer is used to optimize the low-rank trainable parameters, and a cosine annealing learning rate scheduling strategy is combined to control the learning rate changes during training. Furthermore, the stable training strategy may include one or more of the following: gradient pruning, weight decay, learning rate preheating, early stopping on the validation set, mixed precision training, and phased training plan.

[0222] S105, Deploy the fine-tuning model.

[0223] After fine-tuning the parameters, the system generates accuracy and trajectory planning efficiency indices based on the temporal logic reduction on the validation set, selects model parameters that meet preset requirements, and obtains the optimized visual language model:

[0224] ;

[0225] in, Represents the basic visual language model, This represents the preferred low-rank incremental parameters obtained through training steps S101 to S104; (symbol) This indicates that low-rank incremental parameters are loaded into the base visual language model, or fused with the corresponding base parameters of the base visual language model.

[0226] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A robot task planning method based on visual language models and temporal logic reduction, characterized in that, Includes the following steps: S1. Based on natural language task instructions, robot initial environment observation information and robot capability description, the task objectives, operation objects, spatial relationships, execution conditions and safety requirements are identified through a visual language model and jointly encoded to construct a planning context; Multimodal joint understanding of the planning context yields a task semantic representation and an initial planning state; S2. Based on the task semantic representation and initial planning state, determine the task state propositions that can reflect the task objectives, execution order, preconditions and safety constraints, construct the truth set of task state propositions, and convert the natural language task requirements into the corresponding temporal logic specifications. S3. The sequential logic specification is reduced and compiled into a sequential logic constraint state machine to represent the progress state of the task logic, and its initial state, accepting state, violation state and state transition relationship are determined. The constraint state machine updates the constraint state according to the set of truth values ​​of the task state propositions corresponding to the robot's planning state, providing a basis for the prospective check of subsequent candidate actions. S4. Based on the current planning context and the action word prefixes already generated by the current structured action, the visual language model generates the robot's next structured action word by word in an autoregressive manner, thus obtaining the original output probability of each word in the visual language model's vocabulary. Select the word that is not currently blocked and has the highest original output probability as the word to be checked, and append the word to be checked to the prefix of the current action word to form a sequence of action words to be checked, and further determine the complete set of structured candidate actions that the sequence of action words to be checked can form; S5. Based on the complete structured candidate action set, establish the mapping relationship between candidate actions, planning states, and temporal logic constraint states. The mapping relationship includes action action function and state labeling function. The action action function is used to predict the subsequent planning state formed after the candidate action is executed. The state labeling function is used to extract the true task state propositions from the subsequent planning states and form a set of truth values ​​of task state propositions that the temporal logic constraint state machine can read, thereby establishing the mapping relationship between the visual language model word generation space, robot action space, planning state space, and constraint state space. S6. Input the set of truth values ​​of the task state proposition obtained in step S5 into the temporal logic constraint state machine constructed in step S3, and combine it with the current constraint state to calculate the next constraint state that may be reached after the candidate action is executed, so as to perform a prospective check on the task logic state that may be reached after the candidate action is executed. The prospective check is used to determine, based on the next constraint state, whether the candidate action will cause the task specification to be violated or cause the subsequent task trajectory to lose the possibility of reaching the acceptance state before the candidate action is officially output by the visual language model. S7. The temporal logic constraint state machine performs iterative constraint decoding on the vocabulary output of the current word position based on the prospective inspection results of the candidate action, before the visual language model officially outputs the next word, sets the original output probability of the word to be blocked to an unselectable state, and redetermines the next word within the range of other words. The process involves word-by-word repetition of candidate actions, prospective checks, and constraint decoding until a complete structured robot action is generated. S8. Add the generated complete structured robot motion to the current task planning sequence, and formally update the robot planning state and constraint state according to the generated complete structured robot motion, and generate a structured robot task plan.

2. The robot task planning method according to claim 1, characterized in that, Step S2 includes: S21. Based on the task semantic representation, initial planning state and robot capability description, determine the target facts, process facts and safety facts that need to be logically judged in the current task, and construct the truth value set of task state propositions. S22. Based on the constructed set of truth values ​​of task state propositions, and in combination with task objectives, action sequence relationships, action execution prerequisites, safety conditions, prohibition conditions, and condition triggering relationships, construct a temporal logic reduction using logical operators and temporal operators. The timing logic specification is used to describe the robot's goal achievement requirements, task progression sequence, action execution prerequisites, and safety constraints throughout the process in a finite task trajectory.

3. The robot task planning method according to claim 1, characterized in that, The temporal logic specification uses a finite trajectory linear temporal logic formula; step S3 compiles the finite trajectory linear temporal logic formula into a deterministic finite state automaton, denoted as . ; in, This represents a deterministic finite automaton; Constraint State Set Used to represent the advancement stage of the task trajectory prefix corresponding to the generated complete action sequence in the temporal logic specification; the advancement stage includes the task objective not yet achieved, some prerequisite objectives achieved, all task objectives satisfied, and the current task trajectory irreversibly violating the task specification; Input alphabet The set of truth values ​​of the task state propositions obtained from step S2 The truth value assignment is denoted as ,in Represents the set of truth values ​​for the propositions concerning task state. The power set; each input symbol in the input alphabet This represents a subset of truth values ​​for task state propositions that hold true under a given robot planning state. State transition function Used to calculate the next constraint state based on the current constraint state and the set of truth values ​​of the input task state propositions; Formal initial state This represents the logical state of a deterministic finite state automaton before it has read the robot's initial planning state. Acceptance State Set Used to indicate that the constraint state has been satisfied by the sequential logic specification; when the deterministic finite state automaton enters any accepting state. When, it means that the finite task trajectory corresponding to the currently generated complete action sequence has satisfied the temporal logic specification.

4. The robot task planning method according to claim 3, characterized in that, For deterministic finite state automata, define the set of deadlock states: ;symbol" "Indicates that it is contained in; When the constrained state enters the deadlock state set When this occurs, it indicates that the current planned path has violated the task specifications, and the corresponding violation cannot be eliminated through subsequent actions; Define a function that accepts state reachability: 。 5. The robot task planning method according to claim 1, characterized in that, In step S4, assume that the action word prefixes generated by the current structured action have been... for: ； in, Indicates the position of word generation in the current structured action. This indicates the first generated and confirmed action in the current structured action. Each word element; In the current structured action, the first At each lexical generation location, the visual language model obtains the original output probability of each lexical in the vocabulary based on the current planning context and the lexical prefixes that have already been generated; Select the word that is not currently blocked and whose original output probability reaches the maximum value. , as the word element to be checked; For the word to be checked This is appended to the current action word prefix to form the action word sequence to be checked; Based on the action primitives, structured action syntax, parameter forms of action predicates in the robot's capability description, as well as the object, position, and region parameters in the current task, determine the complete set of structured candidate actions that the sequence of action lexes to be examined can form. When the sequence of action lexes to be checked does not yet constitute a complete action, it is completed according to the parameter type required by the action predicate to obtain one or more complete structured candidate actions.

6. The robot task planning method according to claim 1, characterized in that, Step S5 includes: S51. Predict the planning state after the candidate action is executed by using the action action function; Action function Defined as: ; in, Represents the robot's planning state space. Represents the set of robot action primitives, symbolized by " "Indicates the function definition relationship, symbol" "Represents the Cartesian product, symbol " " indicates a mapping relationship; Calculate any candidate action from the complete structured candidate action set. The corresponding predicted successor planning state for: ； in, Indicates the first The current planning status of each planning step. This represents the action function; Define candidate actions The basic execution conditions are Define the state fact that is added or becomes true after the action is executed as follows: Define the fact that a state is deleted or becomes false after an action is performed as follows: If the current planning state meets the preconditions for the action: ; where the symbol " "Indicates a satisfied relationship; The predicted successor planning state corresponding to the candidate action is represented as follows: ； in, Indicates the current planning status Delete candidate actions A state fact that no longer holds true after execution, symbolized by " " represents the set union operation; S52. Generate the state machine inputs corresponding to candidate actions through the state labeling function; State marking function Defined as: ; in, Represents the set of truth values ​​for the propositions concerning task state. The power set; For candidate actions The corresponding predicted successor planning state is calculated as follows: ； in, Indicates the assumption that candidate actions will be performed. The set of truth values ​​for subsequent task state propositions; Let be the state labeling function, representing the set of truth values ​​of task state propositions that hold true under the corresponding planning state.

7. The robot task planning method according to claim 1, characterized in that, Step S6 includes: S61. Determine the temporal logic state basis for the current planning step; Let the visual language model be in the first... The complete sequence of actions that has been generated and confirmed before each planning step. for: ； in For the first Each planning step generates and confirms a complete structured robot motion; In the first planning step, based on the initial planning state, the set of truth values ​​of the initial task state propositions is obtained through the state labeling function; the set of truth values ​​of the initial task state propositions is input into the sequential logic constraint state machine to obtain the initial constraint state actually used for task planning; In the In each planning step, starting from the initial planning state, the action action functions are called consecutively according to the action generation order in the complete action sequence to obtain the current planning state. ; Synchronizing with the planning state update, the set of truth values ​​of task state propositions corresponding to each planning state is obtained through the state labeling function; and the constraint state is updated according to the action generation order in the complete action sequence. In the During the online generation of each planning step, based on the current planning state and current constraint state As the state basis for prospective examination of candidate actions; S62. Determine the specification satisfiability of candidate actions; Input the set of truth values ​​of the task state propositions corresponding to the candidate actions into the temporal logic constraint state machine constructed in step S3. The temporal logic constraint state machine then adjusts the state according to the current constraint state. Calculate the next constraint state ; The sequential logic constraint state machine determines whether the corresponding candidate action satisfies the following two conditions based on the next constraint state: ； as well as: ; in, This represents the set of deadlocked states in a sequential logic constrained state machine. Represents the reachability function of the accepted state; when the constrained state enters the set of trapped states. When this occurs, it indicates that the current planned path has violated the task specifications, and the corresponding violation cannot be eliminated through subsequent actions; When both conditions are met, the sequential logic constraint state machine will select candidate actions. If a candidate action is determined to be a satisfiable action according to the specification, or if the next constraint state enters a deadlock state, or if the set of acceptable states cannot be reached from the next constraint state, the sequential logic constraint state machine will determine the candidate action as an unretainable action.

8. The robot task planning method according to claim 1, characterized in that, In step S7, if at least one candidate action in the complete set of structured candidate actions corresponding to the word to be checked passes the prospective check, the word to be checked is retained; if the word to be checked cannot form a valid structured action, or all its corresponding candidate actions violate the task specification, the corresponding word to be checked is determined as a word to be blocked, and the blocking set is updated. Construct a vocabulary-level constraint mask based on the updated mask set; The vocabulary-level constraint mask is applied to the original output probability of the visual language model at the current word position in the vocabulary to obtain the constrained word probability. Normalize the constrained lexical probabilities to obtain the normalized lexical probabilities after several rounds of constraint processing; The next lexical term is selected based on the normalized lexical probability after constraint processing, and it is appended to the current action prefix. Then, repeat steps S4-S7 until a complete structured robot motion is formed.

9. The robot task planning method according to claim 1, characterized in that, Natural language task instructions are natural language instructions for robot operation tasks, which include one or more of the following task types: object grasping, object movement, object classification, object stacking, push-pull tasks, and opening-closing tasks. The robot's initial environmental observation information includes color images, depth information, and structured environmental data input from the target detection model; The robot capability description includes the robot action primitive set, end effector type and state, available workspace, load capacity, and motion constraints; the robot action primitive set is used to represent the basic action types that the robot can call in the task planning process, and the basic action types include at least one or more of the following: movement action, grasping action, releasing action, lifting action, posture adjustment action, waiting action, and holding action; Task semantic representation includes task object, target location, available actions, task objective, sequence relationship, preconditions, safety conditions, and prohibition conditions; The initial planning state is used to represent the state facts related to planning at the start of the task, including object location, spatial relationships, grasping state, end effector state, obstacles, risk objects, and restricted areas.

10. A robot task planning system based on visual language models and temporal logic reduction, characterized in that, It includes at least one processor and a memory connected to the at least one processor; The memory stores program instructions that are executed by the at least one processor. When the program instructions are executed by the at least one processor, the at least one processor performs steps S1-S8 of the robot task planning method of claim 1.

Images