Llm-wm dynamic scheduling method and device based on world model confidence

Abstract

The application discloses a kind of LLM-WM dynamic scheduling method and device based on world model confidence, first acquire scene context and input world model to carry out multi-step rehearsal, according to the prediction evidence to calculate comprehensive confidence, then combine threshold cluster to determine fast road or slow road;Under fast road branch, the first action control instruction is generated by world model and drives executor to execute, under slow road branch, scene card is generated and structured re-planning result is output by calling large language model, then it is parsed into the second action control instruction executable by executor;Meanwhile, the execution feedback is fed back to the world model, and is used for confidence calibration, threshold updating and distillation sample pool construction, to improve the scheduling stability and closed-loop adaptive ability under complex scene.

Description

Technical Field

[0001] This invention belongs to the fields of embodied intelligence, world model prediction, large language model decision-making and agent dynamic scheduling technology, specifically involving an LLM-WM dynamic scheduling method and device based on world model confidence. Background Technology

[0002] In embodied intelligence systems, world models are typically used for short-term rehearsals and local action selection based on the current scene context, while large language models are typically used to understand complex task intentions, interpret multi-condition constraints, and generate structured replanning results. The two are complementary in their capabilities.

[0003] Existing systems often lack a unified confidence-based routing framework between fast path prediction and slow path replanning. When local observations are insufficient, obstacles change frequently, or the execution entity's resources are strained, the world model may provide unstable action suggestions, while the output of the large language model is difficult to directly map into executable instructions for the execution entity.

[0004] Furthermore, if the slow path results remain only at the natural language level, or lack execution feedback, it is difficult to form a continuous closed loop between the fast path output, slow path output, and execution feedback. When encountering similar low-confidence scenarios, the slow path will still be repeatedly triggered, or the fast path will be incorrectly selected in high-risk scenarios.

[0005] Therefore, there is a need for a closed-loop method and apparatus that can perform dynamic routing based on the confidence level of the world model, directly convert the slow path replanning results into execution control instructions, and feed the execution feedback back to the world model. Summary of the Invention

[0006] The purpose of this invention is to provide an LLM-WM dynamic scheduling method and apparatus based on world model confidence, so as to solve the problem that embodied intelligent systems have difficulty simultaneously taking into account real-time performance, low-confidence risk identification, fast and slow path context continuity, and long-term feedback evolution when coordinating fast path prediction of the world model and slow path replanning of the large language model.

[0007] According to a first aspect of the embodiments of this application, an LLM-WM dynamic scheduling method based on world model confidence is provided, comprising: Obtain the scenario context of the current task scenario, which includes the task objective, environmental observation data, execution entity state, and security constraints; The scene context is input into the world model for multi-step pre-playing, and the world model outputs a set of candidate actions, future state trajectories, and prediction evidence. Calculate the overall confidence level based on the aforementioned predictive evidence; The overall confidence level is compared with the target route threshold to obtain the dynamic routing result; When the dynamic routing result points to a fast path, a first action control instruction is generated based on the candidate action set and the corresponding future state trajectory, and the executor is driven to execute according to the first action control instruction; When the dynamic routing result points to the slow path, a scene card is generated based on the scene context, the candidate action set, the future state trajectory, and the prediction evidence. The large language model is called to output the structured replanning result. The structured replanning result is parsed into a second action control instruction that can be executed by the execution body control interface, and the execution body is driven to execute according to the second action control instruction. The execution feedback generated after the executor executes the action control instruction is obtained; when the dynamic routing result points to a fast path, the first action control instruction and the execution feedback are injected back into the world model; when the dynamic routing result points to a slow path, the structured replanning result and the execution feedback are injected back into the world model; and the execution feedback is written into the event chain, the threshold updater, and the distillation sample pool.

[0008] According to a second aspect of the embodiments of this application, an LLM-WM dynamic scheduling device based on world model confidence is provided, comprising: The data acquisition module is used to acquire the scene context of the current task scenario, which includes the task objective, environmental observation data, execution entity status, and security constraints. The world model pre-playing module is used to input the scene context into the world model for multi-step pre-playing, and the world model outputs a set of candidate actions, future state trajectories and prediction evidence. The confidence measurement module is used to calculate the overall confidence level based on the prediction evidence. The dynamic routing module is used to compare the overall confidence level with the target routing threshold to obtain the dynamic routing result; The fast path execution module is used to generate a first action control instruction based on the candidate action set and the corresponding future state trajectory when the dynamic routing result points to a fast path, and drive the executor to execute according to the first action control instruction; The slow path replanning module is used to generate a scene card based on the scene context, the candidate action set, the future state trajectory and the prediction evidence when the dynamic routing result points to the slow path, call the large language model to output the structured replanning result, parse the structured replanning result into a second action control instruction that can be executed by the execution body control interface, and drive the execution body to execute according to the second action control instruction. The feedback evolution module is used to obtain the execution feedback generated after the executor executes the action control instruction; when the dynamic routing result points to a fast path, the first action control instruction and the execution feedback are injected back into the world model; when the dynamic routing result points to a slow path, the structured replanning result and the execution feedback are injected back into the world model; and the execution feedback is written into the event chain, the threshold updater, and the distillation sample pool.

[0009] According to a third aspect of the embodiments of this application, an electronic device is provided, including a processor and a memory, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the above-described method.

[0010] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, which stores computer instructions that, when executed by a processor, implement the above-described method.

[0011] Compared with the prior art, the present invention has at least the following beneficial effects: This application uses a world model to predict evidence to calculate the overall confidence level, and generates dynamic routing results based on the overall confidence level and threshold clusters. Therefore, it overcomes the problem that fixed calling strategies or manual threshold configurations are difficult to adapt to changes in the scenario, thereby improving the accuracy of fast and slow path switching.

[0012] This application generates a scene card containing prediction evidence, slow path triggering reasons, and safety constraints when a slow path is triggered, thus overcoming the problem that the fast path state is difficult to pass to a large language model, thereby improving the continuity between slow path replanning and fast path historical state.

[0013] This application parses the structured reprogramming results into action control instructions that can be executed by the execution control interface, thus overcoming the problem that free text output is difficult to execute directly and securely, thereby improving the parsability and security of the embodied execution process.

[0014] This application injects the structured reprogramming results and execution feedback back into the world model and writes them into the event chain, threshold updater, and distillation sample pool. This overcomes the problem of slow path experience being difficult to accumulate, thereby reducing the probability of repeated slow path triggering in similar low-confidence scenarios. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] Figure 1 The flowchart of an LLM-WM dynamic scheduling method based on world model confidence is provided for an embodiment of the present invention.

[0017] Figure 2The diagram below illustrates the structure of an LLM-WM dynamic scheduling device based on world model confidence, as provided in this embodiment of the invention.

[0018] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0020] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0021] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0022] The following example illustrates a composite task of "item delivery and on-route inspection" performed by an autonomous mobile device deployed in an indoor park. The actuator includes a camera, distance sensor, positioning unit, battery management unit, motion control unit, communication interface, and processor. A world model is deployed on the edge controller, and a large language model is deployed on the edge inference node. The world model is used for multi-step state pre-planning based on the current scene context, while the large language model is used to generate structured replanning results when slow-path triggering occurs. However, this is not limited to this; it can also be applied to composite task scenarios such as goods handling and inventory counting in a warehouse environment, drug delivery and ward inspection in a hospital environment, material transfer and equipment inspection in an industrial plant, and unmanned vehicle delivery and security patrol in an outdoor park. Correspondingly, the actuator can be an intelligent device with environmental perception, state acquisition, motion control, communication, and processing capabilities, such as a mobile robot, unmanned transport vehicle, or drone. In the above scenarios, a scenario context can be constructed based on the task objective, environmental observation data, executor status, and security constraints. The world model performs multi-step pre-playing and calculates the comprehensive confidence level. When the slow path is triggered, the large language model is called to generate structured replanning results, thereby realizing dynamic scheduling between fast path execution and slow path replanning.

[0023] Figure 1 A flowchart of an LLM-WM dynamic scheduling method based on world model confidence is provided for embodiments of the present invention. The method may include the following sub-steps: S1: Obtain the scenario context of the current task scenario, which includes the task objective, environmental observation data, execution entity state, and security constraints; this step includes the following sub-steps: S11: Obtain task objectives and task phase information from the task management system.

[0024] Specifically, at the current decision-making moment Get the scene context of the current task scene. The scene context Including mission objectives Task Phase Information Environmental observation data Execution status and safety constraints , denoted as: .

[0025] Obtain task objectives from the task management system. and task phase information In indoor park delivery inspection scenarios, the task objective is... This includes delivering items to target room A302, collecting inspection images while passing through corridor C, the task deadline, and the set of inspection points to be passed; task phase information. This indicates that the current task is in at least one stage: awaiting departure, en route for delivery, nearing delivery, inspection and data collection, or returning and docking. Thus, the world model can distinguish action preferences under different task stages, such as "priority delivery," "priority obstacle avoidance while waiting," and "collecting inspection images along the way," during subsequent simulations.

[0026] S12: Acquire environmental observation data from sensors, including visual observation data, distance observation data, pose observation data, velocity observation data, and obstacle status data.

[0027] Specifically, the system acquires images of the corridor, access control area, and the surrounding area of ​​the target room through cameras as visual observation data; it acquires a set of distance points for pedestrians, temporary objects, walls, and door frames through distance sensors as distance observation data; it acquires the pose of the actuator through the positioning unit as pose observation data; it acquires the current driving speed through the motion control unit as speed observation data; and it identifies pedestrians, mobile carts, temporary obstacles, and the opening and closing status of access control based on multiple consecutive frames of visual observation data and distance observation data, forming obstacle status data.

[0028] S13: Obtain the execution entity status from the execution entity control interface. The execution entity status includes remaining power, load status, communication quality, current location, and health status.

[0029] Specifically, the system reads the remaining battery power output from the battery management unit, the load status output from the cargo sensor, the communication quality output from the communication interface, the current position output from the positioning unit, and the health status corresponding to the self-test results of the motion control unit and sensors from the actuator control interface. In this embodiment, the load status is used to determine whether the actuator is still carrying items to be delivered, the communication quality is used to determine whether the edge inference node can be stably invoked, and the health status is used to determine whether the actuator is suitable to continue performing the delivery inspection task.

[0030] S14: Obtain safety constraints, including restricted areas, minimum safe distances, mission time limits, and safe stopping points.

[0031] Specifically, the system obtains the restricted areas marked on the indoor park map, the minimum safe distance to pedestrians and obstacles, the task time limits for delivery and inspection tasks, and safe docking points along corridors, near charging points, and near service counters. The restricted areas include densely populated areas, construction-closed areas, and access control areas where robots are prohibited from entering. The safe docking points provide feasible parking locations in situations such as slow-path replanning, communication failures, or low battery conditions.

[0032] S15: Encapsulate the task objectives, task phase information, environmental observation data, execution entity status, and security constraints into a scene context using a unified timestamp.

[0033] Specifically, the task objectives Task Phase Information Environmental observation data Execution status and safety constraints Encapsulated into scene context according to a unified timestamp This allows the world model to simultaneously obtain information such as "where to go, what stage of the task is currently being performed, the surrounding environment, the state of the executor, and which actions cannot be violated" at the same decision-making moment. Through unified encapsulation, it avoids the world model generating candidate actions solely based on local sensor data, thereby reducing the risk of inconsistencies between candidate actions and delivery goals, inspection requirements, or safety constraints.

[0034] S2: Input the scene context into the world model for multi-step pre-playing, and the world model outputs a set of candidate actions, future state trajectories, and prediction evidence; this step includes the following sub-steps: S21: Input the task objective, environmental observation data, execution state, and security constraints from the scenario context into the world model.

[0035] Specifically, the World Model (WM) is derived from the scene context. Read the task target Environmental observation data Execution status and safety constraints and combined with task phase information Status coding is performed. During delivery, the world model focuses on coding the location of the target room A302, the current distance to pedestrians in the corridor, the current position of the actuator, the status of the cargo, and the minimum safe distance. During the inspection and data collection phase, the world model further focuses on the inspection points, camera orientation, image quality in corridor C area, and available locations for stopping and shooting.

[0036] S22: The world model, based on the task objective, environmental observation data, executor state, and security constraints, performs a preliminary simulation of state changes within several future decision steps, generating a set of candidate actions.

[0037] Specifically, the World Model (WM) simulates changes in the position of the executor, obstacles, power levels, and task progress over several future decision steps based on the current hidden state and security constraints, and generates a set of candidate actions. Candidate action set Actions can include continuing forward, slowing down and waiting, bypassing temporary obstacles, pulling over, proceeding to the inspection point to collect images, and requesting slow-path replanning. For indoor park delivery inspection tasks, candidate actions are not simply waypoint selections, but rather action plans that simultaneously consider delivery deadlines, inspection data collection requirements, pedestrian avoidance, and safe stopping conditions.

[0038] S23: For each candidate action in the candidate action set, output the corresponding future state trajectory, and generate prediction evidence based on the future state trajectory.

[0039] Specifically, for the candidate action set For each candidate action, the World Model (WM) pre-determines the actor's position, velocity, distance to pedestrians or obstacles, whether it reaches the target room, whether it passes through inspection points, and whether it triggers safety constraints within several decision steps, thus obtaining the corresponding future state trajectory. The WM further generates prediction evidence based on the stability between these future state trajectories, the consistency of outputs from different prediction heads, and the satisfaction of task and safety constraints by the candidate actions. The predictive evidence is used to support subsequent comprehensive confidence calculations, ensuring that dynamic scheduling is not based solely on a single predicted path, but rather on the stability and risk performance of multiple candidate actions in future states.

[0040] S3: Calculate the overall confidence level based on the predicted evidence; this step includes the following sub-steps: S31: Extract from the prediction evidence the prediction distribution entropy for characterizing the dispersion of candidate state distribution, the trajectory divergence for characterizing the divergence of future state trajectories, the prediction head divergence for characterizing the differences in the outputs of multiple prediction heads, and the task constraint conflict degree for characterizing the degree to which candidate actions violate task constraints and security constraints.

[0041] Specifically, based on the predictive evidence output by the World Model (WM) Extract the predicted distribution entropy Trajectory divergence Predicting head divergence Conflict degree with task constraints In one alternative implementation, the indicators are calculated according to the following formula: In this embodiment, if the direction of pedestrian movement is unstable, the opening and closing status of the access control is uncertain, or the position of temporary obstacles changes significantly, the candidate state distribution will be more dispersed, and the predicted distribution entropy will be higher. Increase; if the future positions corresponding to candidate actions such as continuing forward, detouring, and waiting differ significantly, then the trajectory divergence will increase. Increase; if multiple prediction heads give different results regarding whether the executor can safely pass through corridor C, then the prediction head divergence degree is increased. Increase the task constraint conflict degree if the candidate action might lead to exceeding the task time limit, entering a restricted area, falling below the minimum safe distance, or insufficient battery power to complete the docking. Increase. By drawing on predictive evidence. Extracting the above indicators allows us to characterize the uncertainty of the world model prediction from four aspects: state distribution, trajectory change, model output consistency, and constraint conflict.

[0042] S32: Normalize the predicted distribution entropy, trajectory divergence, prediction head divergence, and task constraint conflict degree to obtain the corresponding normalized indices.

[0043] Specifically, these are the predicted distribution entropy. Trajectory divergence Predicting head divergence Conflict degree with task constraints Maintain the sliding window quantile estimator. For any index to be normalized... , for , , and Any term in the above is normalized according to the following formula: In the formula, Indicators The lower quantile boundary within the current sliding window; Indicators The high quantile boundary within the current sliding window; This represents a preset positive number used to prevent the denominator from being zero; This means that the calculation result will be limited to a numerical range of 0 to 1; Indicators The normalized result is obtained. From this, the normalized predicted distribution entropy is obtained. Normalized trajectory divergence Normalized forecast head divergence Conflict degree with normalized task constraints If subsequent execution results show a deviation between the overall confidence level and the actual execution results—for example, a safe stop occurs after the high-confidence expressway is executed, or the original expressway action can still be successfully executed after the low-confidence slowway is replanned—then the low-quantile and high-quantile boundaries of the corresponding indicators are updated according to the deviation to calibrate the normalized mapping relationship.

[0044] S33: The comprehensive uncertainty is obtained by weighted fusion based on the normalization indices and their corresponding weights.

[0045] Specifically, predict the distribution entropy based on normalization. Normalized trajectory divergence Normalized forecast head divergence Conflict degree with normalized task constraints The weights and their corresponding weights are weighted and fused according to the following formula to obtain the comprehensive uncertainty. : In the formula, , , and These represent the weights of the four normalized indicators at the current decision moment, with each weight being no less than 0 and the sum of all weights being 1. In the indoor park delivery inspection scenario, when the executor approaches a densely populated corridor or access control area, the weights corresponding to task constraint conflict degree and predictor head divergence degree are increased; when the executor is in an open corridor and the obstacle state is stable, the weights of the corresponding conservative indicators are decreased, thereby avoiding unnecessary triggering of slow path replanning.

[0046] S34: Determine the comprehensive confidence level based on the comprehensive uncertainty.

[0047] Specifically, based on the comprehensive uncertainty The overall confidence level at the current decision moment is calculated according to the following formula. : In the formula, This represents the overall uncertainty at the current decision-making moment; This indicates the overall confidence level at the current decision-making moment; Indicates will The value is limited to the range of 0 to 1. The overall confidence level... This is used to represent the reliability of the world model's predictions of the current set of candidate actions and their future trajectories. Higher overall uncertainty corresponds to lower overall confidence, and vice versa. The scheduler can also update the quantile boundaries used in the normalization process and the weights corresponding to each normalization index based on historical execution results such as successful delivery, trajectory deviation, safe docking, and manual takeover in the event chain, thereby calibrating the overall confidence. By continuously aligning the overall confidence with the actual execution results, the risk of the world model consistently overestimating or underestimating prediction reliability in scenarios with similar corridor congestion, access control changes, or temporary obstacles can be reduced.

[0048] S4: Compare the overall confidence level with the target routing threshold to obtain the dynamic routing result; this step includes the following sub-steps: S41: Determine the target routing threshold based on task type, environmental complexity, execution status, and security level.

[0049] Specifically, the scheduler determines the target routing threshold based on the current task scenario. In an indoor campus delivery and inspection scenario, task types include delivery tasks, inspection tasks, and composite tasks involving delivery and on-route inspection; environmental complexity is determined based on corridor congestion, the number of obstacles, changes in obstacle movement, and access control status; the execution entity status is determined based on remaining battery power, load status, communication quality, current location, and health status; and the safety level is determined based on the distance to restricted areas, compliance with minimum safety distance requirements, personnel density, and available safe parking spots. The target routing threshold at the current decision moment is denoted as... The half-width of the gray area corresponding to the target routing threshold is denoted as In one implementation, the dynamic routing result Determined according to the following formula: In the formula, Fast represents the dynamic routing result pointing to the fast path; Slow represents the dynamic routing result pointing to the slow path; Review represents entering the gray area for review. Indicates the target route threshold; This indicates the half-width of the gray area. If the current scenario is a densely populated corridor, an unstable access control system, or a low-power execution unit, the target route threshold is increased to make it easier for the system to enter slow path replanning. If the current scenario is an open corridor, a stable obstacle system, and a good execution unit status, the target route threshold is decreased to reduce unnecessary slow path calls.

[0050] S42: When the overall confidence level is higher than the target route threshold, a dynamic route result pointing to the fast path is generated.

[0051] Specifically, the scheduler will integrate the confidence levels. Compare with the target route threshold. When the overall confidence level... When the gray zone bound exceeds the target routing threshold, it indicates that the world model's predictions of the candidate action set and its future state trajectories are relatively stable, and the scheduler generates dynamic routing results pointing to the fast path. At this point, the system prioritizes generating the first action control command based on the candidate action set output by the world model and the corresponding future state trajectory. For example, if there is sufficient distance between pedestrians ahead in the corridor, the access control remains open, and the path to the target room A302 is stable, the control actuator continues to move forward or passes through the current corridor area at the planned speed. This avoids calling large language models when the prediction reliability is high, shortening the response time at the current decision moment.

[0052] S43: When the overall confidence level is lower than the target route threshold, generate a dynamic route result pointing to the slow path.

[0053] Specifically, the scheduler will integrate the confidence levels. Compare with the target route threshold. When the overall confidence level... When the gray zone boundary falls below the target routing threshold, it indicates that the world model's prediction of future states is unreliable, and the scheduler generates a dynamic routing result pointing to the slow path. At this point, the system generates a scene card based on the scene context, candidate action set, future state trajectory, and prediction evidence, and calls a large language model for structured replanning. For example, when the pedestrian trajectory ahead is unstable, temporary obstacles block the passage, or the access control status changes frequently, the system enters the slow path to obtain replanning results such as waiting, detouring, stopping, or manual confirmation. This avoids directly executing high-risk candidate actions when the prediction reliability is insufficient.

[0054] S44: When the overall confidence level is within the gray zone of the target routing threshold, the prediction evidence and overall confidence level are recalculated based on the most recent several frames of environmental observation data, and a dynamic routing result is generated based on the recalculated overall confidence level.

[0055] Specifically, when the overall confidence level When the target route threshold is within the gray zone, the scheduler enters gray zone verification. During gray zone verification, the scheduler reads the most recent frames of environmental observation data, including continuous corridor images, distance observation data, pose observation data, velocity observation data, and obstacle state data. This data, along with the current task objective, the execution entity's state, and safety constraints, is then re-encapsulated into a verification scenario context. The world model performs a multi-step re-enactment based on this verification scenario context, regenerating prediction evidence and calculating the overall confidence score. If the recalculated overall confidence score is higher than the upper bound of the gray zone, a dynamic route pointing to the fast path is generated; if the recalculated overall confidence score is lower than the lower bound of the gray zone, a dynamic route pointing to the slow path is generated. Gray zone verification reduces the frequency of execution entities switching between fast and slow paths at critical confidence levels.

[0056] S5: When the dynamic routing result points to a fast path, a first action control instruction is generated based on the candidate action set and the corresponding future state trajectory, and the executor is driven to execute according to the first action control instruction; this step includes the following sub-steps: S51: Based on the future state trajectory corresponding to each candidate action in the candidate action set, determine whether each candidate action satisfies the task objective and safety constraints, and determine the target candidate action from the candidate actions that satisfy the task objective and safety constraints.

[0057] Specifically, when the dynamic routing result points to a fast path, it indicates that the world model has high predictive reliability for the current set of candidate actions and their future state trajectories, and the scheduler enters the fast path action screening. In the indoor park delivery inspection scenario, candidate actions can include continuing forward, slowing down and waiting, detouring, pulling over, and passing through inspection points to collect images. Based on the future state trajectory corresponding to each candidate action, the scheduler determines whether the executor can move towards the target room A302, whether it can pass through the inspection point in corridor C, whether it maintains a minimum safe distance from pedestrians and obstacles, whether it avoids restricted areas, and whether it meets the task time limit. If there are no candidate actions that meet the task objectives and safety constraints, the fast path execution is stopped and slow path replanning is triggered; if there are candidate actions that meet the task objectives and safety constraints, the target candidate action is determined based on the progress of the task, safety risks, energy consumption, and trajectory deviation. This avoids executing candidate actions that do not meet the task objectives or safety constraints simply because the overall confidence level is high.

[0058] S52: Based on the target candidate action and its corresponding future state trajectory, generate local path constraints, maximum speed constraints, minimum safe distance constraints and stopping condition constraints.

[0059] Specifically, the scheduler converts the future trajectory corresponding to the target candidate action into local path constraints, and determines the maximum speed constraint based on corridor width, pedestrian distance, access control status, and the state of the load on the execution unit; it determines the minimum safe distance constraint based on the minimum safe distance in the safety constraints; and it determines stopping condition constraints based on conditions such as obstacles being too close, access control being closed, communication failures, insufficient power, or deviation from the planned trajectory. In delivery and on-route inspection tasks, local path constraints may also include maintaining a suitable driving posture for image acquisition when passing through the inspection point in corridor C. By converting the target candidate action into a clear path, speed, safe distance, and stopping condition, a defined boundary can be provided for subsequently generating the first action control command executable by the execution unit control interface.

[0060] S53: Generate a first motion control command based on the local path constraint, maximum speed constraint, minimum safe distance constraint and stopping condition constraint, and send the first motion control command to the execution body control interface to drive the execution body to execute according to the first motion control command.

[0061] Specifically, under the expressway branch, the scheduler does not invoke the large language model, but directly generates the first action control command based on local path constraints, maximum speed constraints, minimum safe distance constraints, and stopping condition constraints. When the world model determines that the main corridor's traffic status is stable and continuing forward can simultaneously satisfy the delivery target and inspection point passing requirements, the first action control command can include proceeding along the main corridor, maintaining a maximum speed not exceeding a preset speed, keeping a minimum safe distance from pedestrians, triggering image acquisition when passing through corridor C, and stopping when the distance to an obstacle ahead is below a stopping threshold. After receiving the first action control command, the actuator control interface converts it into chassis speed, steering angle, and sensor acquisition commands to drive the actuator to perform the corresponding actions. This allows the world model output to be converted into directly executable control commands while maintaining a low decision-making delay under the expressway branch.

[0062] S6: When the dynamic routing result points to a slow path, a scene card is generated based on the scene context, the candidate action set, the future state trajectory, and the prediction evidence. The large language model is then invoked to output the structured replanning result. This structured replanning result is parsed into a second action control instruction executable by the execution body control interface, and the execution body is driven to execute according to the second action control instruction. This step includes the following sub-steps: S61: Extract the task objective, environmental observation data, execution entity state, and security constraints from the scenario context to obtain scenario description information.

[0063] Specifically, when the dynamic routing result points to a slow path, it indicates that the world model's prediction reliability for the current set of candidate actions and their future state trajectories is insufficient. The scheduler extracts task objectives, environmental observation data, executor states, and safety constraints from the scene context to obtain scene description information. In the indoor park delivery inspection scenario, the scheduler extracts information such as the target room A302, inspection requirements for corridor C, current corridor image, distance to pedestrians ahead, location of temporary obstacles, access control status, current position of the executor, remaining battery power, load status, communication quality, restricted areas, minimum safe distance, and safe stopping point. Task stage information can be written into the scene description information as auxiliary information for the task objective, indicating that the executor is currently in the delivery process, nearing delivery, or in the inspection and data collection stage. This enables the large language model to provide a scene description consistent with the current task stage and safety status.

[0064] S62: Based on the candidate action set and the future state trajectory, determine the candidate action information and trajectory prediction information for replanning.

[0065] Specifically, the scheduler organizes the candidate action set output by the world model and the corresponding future state trajectory of each candidate action into candidate action information and trajectory prediction information. For example, candidate action information includes continuing forward, waiting, detouring, stopping, and requesting manual confirmation; trajectory prediction information includes the expected position, expected speed, distance changes to pedestrians or obstacles, whether it passes through inspection points, and whether it may trigger safety constraints for each candidate action within several decision steps in the future. This enables the large language model to replan using the candidate actions and pre-planning results already generated by the world model, avoiding the need to reconstruct action plans that are out of touch with the current scene.

[0066] S63: Determine the cause of slow path triggering based on the predicted evidence and the dynamic routing results.

[0067] Specifically, the scheduler determines the slow path triggering cause based on the prediction distribution entropy, trajectory divergence, predictor divergence, and task constraint conflict degree in the prediction evidence. If the pedestrian trajectory ahead is unstable, a temporary obstacle blocks the main passage, multiple predictors give inconsistent results for the detour route, or the candidate action carries the risk of entering a restricted area, falling below the minimum safe distance, or exceeding the task time limit, then the corresponding cause is written into the slow path triggering cause. By clearly defining the slow path triggering cause, the large language model can replan based on the specific factors leading to low confidence.

[0068] S64: Generate the scene card based on the scene description information, candidate action information, trajectory prediction information, prediction evidence, and slow path triggering reasons.

[0069] Specifically, the scheduler organizes the scene description information corresponding to the current task, environment, and executor state, along with candidate action information, trajectory prediction information, prediction evidence, and slow-path triggering reasons, into a scene card. This scene card provides structured replanning input to the large language model. In an indoor park delivery inspection scenario, the scene card might specify: the executor is delivering items to target room A302, currently located in the second-floor main corridor, with temporary debris and pedestrians ahead; the world model has significant discrepancies in predicting trajectories for continuing forward and detouring through the side corridor; the construction area is a restricted zone; the executor must maintain a minimum safe distance and can choose to wait, detour, reduce speed, return to a safe stopping point, or request manual confirmation. In one implementation, the scene card also includes field constraints required for the large language model to output structured replanning results. By generating structured scene cards, the continuity between the fast-path pre-planning state and the slow-path replanning input can be maintained.

[0070] S65: Input the scene card into the large language model, and the large language model outputs the structured replanning result.

[0071] Specifically, the scheduler inputs the scene card into the large language model and calls the large language model to perform structured replanning. The large language model outputs the structured replanning result based on the scene card. The structured replanning result can include action sequences, waiting strategies, detour strategies, safe stopping strategies, backtracking conditions, and confidence explanation information. For example, the large language model can output "Wait at the current safe position for a preset time. If the pedestrian leaves the main passage, proceed at low speed; if the temporary obstacle still blocks the main passage, detour via the side corridor; if the access control is closed or the safe distance is insufficient, proceed to the nearest safe stopping point and report the anomaly." By limiting the structured output fields, the risk of free text being difficult to parse into executable control instructions can be reduced.

[0072] S66: Verify the structured replanning result, and after the verification is passed, parse the structured replanning result into the second action control instruction to drive the executor to execute according to the second action control instruction.

[0073] Specifically, the scheduler performs field integrity checks, safety constraint checks, and instruction executability checks on the structured replanning results. If the structured replanning results violate restricted area constraints, minimum safe distance constraints, or power constraints, the structured replanning results are not executed directly; instead, a safe docking instruction is generated. If the checks pass, the structured replanning results are parsed into a second action control instruction executable by the execution body control interface. This second action control instruction includes an executable path, speed limit, waiting time, inspection and data collection trigger point, safe stop condition, and rollback action. The execution body control interface drives the execution body to execute according to the second action control instruction. This avoids directly sending the unverified large language model output to the execution body.

[0074] S7: Obtain the execution feedback generated after the executor executes the action control instruction; when the dynamic routing result points to a fast path, inject the first action control instruction and the execution feedback back into the world model; when the dynamic routing result points to a slow path, inject the structured replanning result and the execution feedback back into the world model; and write the execution feedback into the event chain, threshold updater, and distillation sample pool; this step includes the following sub-steps: S71: Obtain the execution feedback generated after the executor executes the first action control instruction or the second action control instruction.

[0075] Specifically, after the executor executes the first or second action control command, the system obtains execution feedback. This feedback includes information such as whether the execution was successful, actual execution time, energy consumption data, actual trajectory deviation, distance to the nearest obstacle, safe stopping status, manual intervention status, and whether the inspection image was successfully acquired. In a delivery inspection scenario, if the executor successfully passes through corridor C according to the fast-path command and completes image acquisition, the execution feedback records a successful fast-path execution; if the executor stops due to a pedestrian suddenly approaching, or cannot pass due to a closed access gate, the execution feedback records the corresponding abnormal reason. By obtaining actual execution feedback, a real execution basis can be provided for world model calibration, target route threshold updates, and slow-path sample quality assessment.

[0076] S72: When the dynamic routing result points to a fast path, the first action control command and the execution feedback are injected back into the world model.

[0077] Specifically, when the dynamic routing result points to a fast path, the system feeds back the first action control command, the actual trajectory of the executor, the actual speed, the actual distance to obstacles, the task progress, and the triggering status of safety constraints into the world model. If the future state trajectory corresponding to the first action control command is consistent with the actual execution trajectory, the predictive reliability of the world model for this type of candidate action in similar scenarios is enhanced; if stopping, yaw, or insufficient safety distance occurs during actual execution, the corresponding deviation is used as the basis for subsequent calibration of the world model. This enables the world model to perform subsequent optimization based on the difference between the fast path prediction result and the actual execution result.

[0078] S73: When the dynamic routing result points to a slow path, the structured replanning result and the execution feedback are injected back into the world model.

[0079] Specifically, when the dynamic routing result points to a slow path, the system injects the structured replanning result, second action control command, and execution feedback from the large language model back into the world model. If the waiting, detour, or safe stop strategy given by the large language model is successfully executed, the world model can use the slow path replanning result as a reference action in similar low-confidence scenarios; if the replanning result fails or triggers manual intervention, the reason for the failure is injected back into the world model as negative sample information. This allows the slow path replanning experience to be transformed into positive or negative sample information needed for subsequent optimization of the world model.

[0080] S74: Write the execution feedback, overall confidence level, and dynamic routing results into the event chain, and have the threshold updater adjust the target routing threshold based on the execution feedback.

[0081] Specifically, the system uses execution feedback for event logging and target routing threshold updates. The event chain records the current scene context, overall confidence level, dynamic routing results, action control instructions, execution feedback, and exception reasons. The threshold updater adjusts the target routing threshold based on the execution feedback. In one implementation, the target routing threshold is updated according to the following formula: In the formula, Indicates the current target route threshold; Indicates the updated target route threshold; Indicates the threshold update step size; This indicates the threshold adjustment amount determined based on the current execution feedback; and These represent the lower and upper limits of the target route threshold, respectively. If the overall confidence level is high but the express route execution fails, a safe stop occurs, or manual takeover takes place, then... Set to a positive value to increase the target route threshold in similar scenarios, making it easier to trigger slow paths subsequently; if the fast path executes successfully and the actual trajectory deviation is small, then... Setting it to a negative value lowers the target route threshold in similar scenarios, reducing unnecessary slow-path calls. By limiting the upper and lower bounds of the target route threshold, it's possible to prevent the target route threshold from exceeding its effective range during continuous updates.

[0082] S75: Perform quality filtering on the slow path samples containing the structured replanning results and the execution feedback, and write the slow path samples that meet the quality filtering conditions into the distillation sample pool.

[0083] Specifically, for slow-path scenarios, the system constructs slow-path samples from the scenario context, prediction evidence, scenario cards, structured replanning results, second action control commands, and execution feedback. If a slow-path sample meets quality filtering conditions such as successful task execution, no violation of safety constraints, no manual intervention, actual trajectory deviation below a preset deviation threshold, and valid inspection and collection results, then the slow-path sample is written into the distillation sample pool. The distillation sample pool is used for subsequent lightweight adaptation of the world model, enabling the world model to output more stable candidate actions, future state trajectories, and prediction evidence when encountering similar scenarios such as corridor congestion, access control changes, or temporary obstacles. By performing quality filtering on slow-path samples, the system avoids using replanning results that have failed execution or violated safety constraints for world model adaptation.

[0084] Corresponding to the aforementioned embodiments of the LLM-WM dynamic scheduling method based on world model confidence, this application also provides embodiments of an LLM-WM dynamic scheduling device based on world model confidence.

[0085] Figure 2This is a block diagram of an LLM-WM dynamic scheduling device based on world model confidence, according to an exemplary embodiment. (Refer to...) Figure 2 The device includes: Data acquisition module 1 is used to acquire the scene context of the current task scenario, which includes the task objective, environmental observation data, execution entity status, and security constraints. World model pre-playing module 2 is used to input the scene context into the world model for multi-step pre-playing, and the world model outputs a set of candidate actions, future state trajectories and prediction evidence. Confidence measurement module 3 is used to calculate the overall confidence level based on the prediction evidence; Dynamic routing module 4 is used to compare the comprehensive confidence level with the target routing threshold to obtain the dynamic routing result; The fast path execution module 5 is used to generate a first action control instruction based on the candidate action set and the corresponding future state trajectory when the dynamic routing result points to a fast path, and drive the executor to execute according to the first action control instruction; The slow path replanning module 6 is used to generate a scene card based on the scene context, the candidate action set, the future state trajectory and the prediction evidence when the dynamic routing result points to the slow path, call the large language model to output the structured replanning result, parse the structured replanning result into a second action control instruction that can be executed by the execution body control interface, and drive the execution body to execute according to the second action control instruction. Feedback evolution module 7 is used to obtain the execution feedback generated after the executor executes the action control instruction; when the dynamic routing result points to a fast path, the first action control instruction and the execution feedback are injected back into the world model; when the dynamic routing result points to a slow path, the structured replanning result and the execution feedback are injected back into the world model; and the execution feedback is written into the event chain, threshold updater, and distillation sample pool.

[0086] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0087] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0088] Accordingly, this application also provides an information display device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to: Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the LLM-WM dynamic scheduling method based on world model confidence as described above. Figure 3 The diagram shown is a hardware structure diagram of any data processing-capable device where an LLM-WM dynamic scheduling device based on world model confidence is located, according to an embodiment of the present invention. (Except for...) Figure 3 In addition to the processor, network interface, memory, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0089] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the LLM-WM dynamic scheduling method based on world model confidence as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0090] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0091] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A dynamic scheduling method for LLM-WM based on world model confidence, characterized in that, include: Obtain the scenario context of the current task scenario, which includes the task objective, environmental observation data, execution entity state, and security constraints; The scene context is input into the world model for multi-step pre-playing, and the world model outputs a set of candidate actions, future state trajectories, and prediction evidence. Calculate the overall confidence level based on the aforementioned predictive evidence; The overall confidence level is compared with the target route threshold to obtain the dynamic routing result; When the dynamic routing result points to a fast path, a first action control instruction is generated based on the candidate action set and the corresponding future state trajectory, and the executor is driven to execute according to the first action control instruction; When the dynamic routing result points to the slow path, a scene card is generated based on the scene context, the candidate action set, the future state trajectory, and the prediction evidence. The large language model is called to output the structured replanning result. The structured replanning result is parsed into a second action control instruction that can be executed by the execution body control interface, and the execution body is driven to execute according to the second action control instruction. Obtain the execution feedback generated after the executor executes the action control command; When the dynamic routing result points to a fast path, the first action control instruction and the execution feedback are injected back into the world model; when the dynamic routing result points to a slow path, the structured replanning result and the execution feedback are injected back into the world model; and the execution feedback is written into the event chain, the threshold updater, and the distillation sample pool.

2. The method according to claim 1, characterized in that, Obtain the scene context of the current task scene, including: Obtain task objectives and task phase information from the task management system; Environmental observation data is acquired from sensors, including visual observation data, distance observation data, pose observation data, velocity observation data, and obstacle status data. The actuator status is obtained from the actuator control interface, and the actuator status includes remaining power, load status, communication quality, current location, and health status. Obtain safety constraints, including restricted areas, minimum safe distances, mission time limits, and safe stopping points; The task objectives, task phase information, environmental observation data, execution entity status, and security constraints are encapsulated into a scene context using a unified timestamp.

3. The method according to claim 1, characterized in that, The scene context is input into the world model for multi-step pre-playing, and the world model outputs a set of candidate actions, future state trajectories, and prediction evidence, including: Input the task objective, environmental observation data, execution entity state, and security constraints from the scenario context into the world model; The world model, based on the task objective, environmental observation data, executor state, and security constraints, pre-simulates the state changes within several future decision steps and generates a set of candidate actions. For each candidate action in the candidate action set, output the corresponding future state trajectory, and generate prediction evidence based on the future state trajectory.

4. The method according to claim 1, characterized in that, The overall confidence level is calculated based on the predictive evidence, including: From the prediction evidence, we extract the prediction distribution entropy to characterize the dispersion of candidate state distribution, the trajectory divergence to characterize the divergence of future state trajectory, the prediction head divergence to characterize the differences between the outputs of multiple prediction heads, and the task constraint conflict degree to characterize the degree to which candidate actions violate task constraints and safety constraints. The predicted distribution entropy, trajectory divergence, predicted head divergence, and task constraint conflict degree are normalized respectively to obtain the corresponding normalized indices. The comprehensive uncertainty is obtained by weighting and fusing the normalized indices and their corresponding weights. The comprehensive confidence level is determined based on the comprehensive uncertainty.

5. The method according to claim 4, characterized in that, The predicted distribution entropy, trajectory divergence, predicted head divergence, and task constraint conflict degree are normalized to obtain corresponding normalized indices, including: Maintain sliding window quantile estimators for the predicted distribution entropy, trajectory divergence, predicted head divergence, and task constraint conflict, respectively. Based on the sliding window quantile estimators, the predicted distribution entropy, trajectory divergence, predicted head divergence, and task constraint conflict are mapped to a unified numerical range to obtain the corresponding normalized indices.

6. The method according to claim 1, characterized in that, The overall confidence level is compared with the target route threshold to obtain the dynamic routing result, including: The target routing threshold is determined based on the task type, environmental complexity, execution status, and security level. When the overall confidence level is higher than the target route threshold, a dynamic route result pointing to the fast path is generated; When the overall confidence level is lower than the target route threshold, a dynamic route result pointing to the slow path is generated; When the overall confidence level is within the gray zone of the target routing threshold, the prediction evidence and overall confidence level are recalculated based on the most recent several frames of environmental observation data, and dynamic routing results are generated based on the recalculated overall confidence level.

7. The method according to claim 1, characterized in that, Generate a first action control instruction based on the candidate action set and the corresponding future state trajectory, and drive the executor to execute the first action control instruction, including: Based on the future state trajectory corresponding to each candidate action in the candidate action set, determine whether each candidate action satisfies the task objective and safety constraints, and determine the target candidate action from the candidate actions that satisfy the task objective and safety constraints; Based on the target candidate action and its corresponding future state trajectory, generate local path constraints, maximum speed constraints, minimum safe distance constraints and stopping condition constraints; A first action control command is generated based on the local path constraint, maximum speed constraint, minimum safe distance constraint, and stopping condition constraint, and the first action control command is sent to the execution body control interface to drive the execution body to execute according to the first action control command.

8. The method according to claim 1, characterized in that, When the dynamic routing result points to a slow path, a scene card is generated based on the scene context, the candidate action set, the predicted evidence of the future state trajectory, and the large language model is invoked to output the structured replanning result. The structured replanning result is parsed into a second action control instruction executable by the execution body control interface, and the execution body is driven to execute according to the second action control instruction, including: The task objective, environmental observation data, execution entity state, and security constraints are extracted from the scene context to obtain scene description information; Based on the set of candidate actions and the future state trajectory, determine the candidate action information and trajectory prediction information for replanning; Based on the predicted evidence and the dynamic routing results, the cause of the slow path triggering is determined; The scene card is generated based on the scene description information, candidate action information, trajectory prediction information, prediction evidence, and slow path triggering reasons. The scene card is input into the large language model, and the large language model outputs the structured replanning result; The structured replanning result is verified, and after the verification is passed, the structured replanning result is parsed into the second action control instruction to drive the executor to execute according to the second action control instruction.

9. The method according to claim 1, characterized in that, Obtain the execution feedback generated after the executor executes the action control command; when the dynamic routing result points to a fast path, inject the first action control command and the execution feedback back into the world model; when the dynamic routing result points to a slow path, inject the structured replanning result and the execution feedback back into the world model. The execution feedback is then written into the event chain, threshold updater, and distillation sample pool, including: Obtain the execution feedback generated after the executor executes the first action control instruction or the second action control instruction; When the dynamic routing result points to a fast path, the first action control command and the execution feedback are injected back into the world model; When the dynamic routing result points to a slow path, the structured replanning result and the execution feedback are injected back into the world model; The execution feedback, overall confidence level, and dynamic routing results are written into the event chain, and the threshold updater adjusts the target routing threshold based on the execution feedback. Slow-path samples containing the structured replanning results and the execution feedback are quality filtered, and slow-path samples that meet the quality filtering conditions are written into the distillation sample pool.

10. A dynamic scheduling device based on world model confidence, characterized in that, include: The data acquisition module is used to acquire the scene context of the current task scenario, which includes the task objective, environmental observation data, execution entity status, and security constraints. The world model pre-playing module is used to input the scene context into the world model for multi-step pre-playing, and the world model outputs a set of candidate actions, future state trajectories and prediction evidence. The confidence measurement module is used to calculate the overall confidence level based on the prediction evidence. The dynamic routing module is used to compare the overall confidence level with the target routing threshold to obtain the dynamic routing result; The fast path execution module is used to generate a first action control instruction based on the candidate action set and the corresponding future state trajectory when the dynamic routing result points to a fast path, and drive the executor to execute according to the first action control instruction; The slow path replanning module is used to generate a scene card based on the scene context, the candidate action set, the future state trajectory and the prediction evidence when the dynamic routing result points to the slow path, call the large language model to output the structured replanning result, parse the structured replanning result into a second action control instruction that can be executed by the execution body control interface, and drive the execution body to execute according to the second action control instruction. The feedback evolution module is used to obtain the execution feedback generated after the executor executes the action control command; When the dynamic routing result points to a fast path, the first action control instruction and the execution feedback are injected back into the world model; when the dynamic routing result points to a slow path, the structured replanning result and the execution feedback are injected back into the world model; and the execution feedback is written into the event chain, the threshold updater, and the distillation sample pool.

11. An electronic device comprising a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the method of any one of claims 1 to 9.

12. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method of any one of claims 1 to 9.

Images