Physical Constraint Hardening Method for VLA Models for Power Equipment Operation

By introducing dynamic mind maps and optical flow dynamics priors into the VLA model, combined with physical constraints, the problem of the lack of physical feasibility of the VLA model in complex power environments is solved, realizing the synergistic improvement of high-level planning and low-level execution, and enhancing the robustness and security of the system.

CN123069870APending Publication Date: 2026-11-10CHINA ELECTRONICS RELIABILITY AND ENVIRONMENTAL TESTING INSTITUTE ((THE FIFTH INSTITUTE OF ELECTRONICS MINISTRY OF INDUSTRY AND INFORMATION TECHNOLOGY) (CHINA SAIBAO LABORATORY)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611203423.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-10
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing VLA models lack explicit modeling of physical feasibility and contact constraints in complex power environments, which makes them prone to unsafe behaviors such as boundary crossing, collision, and slippage under distributed external disturbances and dynamic changes, making them difficult to deploy reliably in power equipment operation.

Method used

By constructing dynamic mind maps, introducing optical flow dynamics priors and physical information constraints, a complete technical framework is formed from constraint expression and prediction to constraint internalization. Combined with group-based relative strategy optimization and physical loss design, a synergistic improvement between high-level planning and low-level execution is achieved.

Benefits of technology

It significantly improves the robustness and safety of VLA models in complex environments, ensuring that operations are within the physically feasible domain, reducing the risks of boundary crossing, collision and slippage, and achieving trustworthy decision-making and safe execution.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069870A_ABST
    Figure CN123069870A_ABST
Patent Text Reader

Abstract

This invention discloses a method for hardening the physical constraints of a VLA model for power equipment operation, comprising: generating multiple candidate high-level plans based on natural language operation instructions, current visual observations, and dynamic mind maps; optimizing the selection probability of the high-level candidate plans using a group relative strategy optimization formula; generating corresponding candidate action sequences based on the candidate high-level plans and their potential representations, inputting them into an action conditional world model, predicting the future visual and motion states corresponding to each candidate high-level plan, and selecting the candidate high-level plan with the highest evaluation score as the optimal candidate high-level plan; inputting the current visual observations, natural language operation instructions, and potential representations of the optimal candidate high-level plan into a low-level VLA model to obtain the actual continuous control actions. This invention enables reliable decision-making and safe execution of VLA models in power equipment operation by deeply embedding physical prior knowledge and dynamic constraints into the reasoning, prediction, and execution of the VLA model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of embodied intelligence and robot control, specifically relating to a method for strengthening the physical constraints of VLA (Vision Language Action) models for power equipment operation. Background Technology

[0002] As the level of automation in power systems increases, the application scenarios of robotic arms or intelligent inspection robots equipped with VLA models are gradually expanding from structured operations of power transmission and transformation equipment to unstructured, dynamic, and complex scenarios such as distribution automation, intelligent inspection, and maintenance, including high-voltage switchgear operation, transformer reading and tightening, and cable joint insertion and removal. These VLA systems typically use visual observation images captured by cameras and natural language task commands (such as "close the disconnect switch") as input. After encoding by a Vision Language Model (VLM), the output is directly mapped to continuous control actions (joint angular velocity or end-effector pose increments), forming a perception-action closed loop.

[0003] In these scenarios, equipment operation involves high-voltage equipment, complex topologies, and uncertain object movements. Environmental disturbances and safety requirements pose significant challenges to the physical reliability and decision robustness of VLA systems. However, existing VLA models generally lack explicit modeling of physical feasibility and contact constraints in complex power environments. This leads to unsafe behaviors such as boundary crossings, collisions, and slippage under distributed external disturbances and dynamic changes, severely restricting their reliable deployment in power equipment operation.

[0004] Currently, most mainstream VLA systems for power equipment operation are based on an end-to-end supervised learning paradigm, directly mapping visual observations and verbal commands into control actions. While these methods can achieve a perception-action closed loop, they are essentially data-driven black-box fitting, lacking an inherent understanding of physical laws and the constraints of power equipment operation. For example, existing models are prone to outputting physically infeasible operation sequences under complex topologies, occlusion, or atypical equipment conditions, manifesting as high-frequency jitter, action overstepping boundaries, or end-point collisions, which may ultimately lead to operation failure or equipment damage.

[0005] To introduce an understanding of physical laws, some studies have attempted to combine robot physics simulation with reinforcement learning, such as generating physically labeled data through simulation or using domain randomization for policy transfer. However, simulation and real-world power field conditions differ fundamentally: modeling contact dynamics, friction characteristics, equipment flexibility, and environmental constraints cannot perfectly match reality, meaning that strategies safe in simulation may still fail unpredictably in the field. Furthermore, these methods typically use physical feasibility as an implicit training objective, lacking interpretable constraint verification during the inference phase, making it difficult to construct credible operational safety boundaries.

[0006] In recent years, world models have provided a new approach to environmental dynamics modeling. Predicting potential future states through cyclic state spaces can aid in long-term task planning. However, these models lack explicit constraints on physical dynamics (such as continuous motion and contact consistency), making them prone to "physical illusions"—predicted scenarios that violate real-world physical laws, such as equipment contact errors or motion trajectories penetrating objects. When such predictions are used to generate strategies for power equipment operation, they introduce accumulated errors and decision drift, seriously threatening operational safety.

[0007] Meanwhile, large language models and vision-language models have shown potential in high-level task planning and semantic reasoning. Some methods attempt to decompose tasks and plan sub-goals through CoT, but there is a semantic gap between CoT reasoning and continuous action control, and there is a lack of ability to verify the physical feasibility of power equipment operation. That is, the model cannot evaluate whether the operation trajectory meets the equipment safety constraints, nor can it dynamically correct the planning based on physical feedback, resulting in a disconnect between high-level reasoning and actual operation. Summary of the Invention

[0008] The purpose of this invention is to provide a method for strengthening the physical constraints of VLA models for power equipment operation, a computer device, a computer-readable storage medium, and a computer program product. This method can achieve reliable decision-making and safe execution of VLA models in power equipment operation by deeply embedding physical prior knowledge and dynamic constraints into the entire chain of reasoning, prediction, and execution of VLA models.

[0009] To achieve the above objectives, one aspect of the present invention provides a method for hardening the physical constraints of a VLA model for power equipment operation, wherein the VLA model includes a high-level planning model, a planning projector, a world model of action conditions, and a low-level VLA model, and the method includes: Step S1: Obtain natural language operation instructions, current visual observations, and the historical execution status of the current task; construct the current dynamic mind map; the high-level planning model generates multiple candidate high-level plans based on the natural language operation instructions, current visual observations, and dynamic mind map; optimize the selection probability of each high-level candidate plan through the group relative strategy optimization formula; and use the planning projector to convert each candidate high-level plan into a potential plan representation. Step S2: Based on the candidate high-level plans and their potential representations, for each candidate high-level plan, generate a corresponding candidate action sequence based on the current visual observation, natural language operation instructions and potential representations of the plan, and input it into the action conditional world model to predict the future visual state and motion state corresponding to each candidate high-level plan. Based on the task progress, physical risk and physical illusion of the future state, a comprehensive evaluation score is formed, and the candidate high-level plan with the highest evaluation score is selected as the optimal candidate high-level plan. Step S3 introduces non-penetration and collision constraints, action timing smoothing constraints, and friction consistency constraints to form the overall training loss. The low-level VLA model is updated based on the overall training loss. The current visual observation, natural language operation instructions, and plan potential representation of the optimal candidate high-level plan are input into the low-level VLA model to obtain the actual continuous control actions.

[0010] Another aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0011] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0012] Another aspect of the present invention provides a computer program product including a computer program that, when executed by a processor, implements the steps of the method described above.

[0013] According to the present invention, the method for physical constraint hardening of VLA model for power equipment operation, computer equipment, computer-readable storage medium and computer program product can realize reliable decision-making and safe execution of VLA model in power equipment operation by deeply embedding physical prior knowledge and dynamic constraints into the entire chain of reasoning, prediction and execution of VLA model. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a flowchart of a method for physical constraint hardening of a VLA model for power equipment operation according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the principle of a VLA model physical constraint hardening method for power equipment operation according to an embodiment of the present invention. Figure 3 This is a structural diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0016] One embodiment of the present invention provides a method for physical constraint hardening of VLA models for power equipment operation. Addressing the common problems of lack of physical feasibility and decision drift in VLA models during long-range interactive tasks, a complete technical framework is designed, encompassing constraint expression, constraint prediction, and constraint internalization. This breaks through the limitations of traditional VLA models that rely solely on visual-linguistic representation and behavioral supervision for end-to-end fitting. It systematically combines the explicit reasoning ability of dynamic mind maps, the physical perception ability of optical flow dynamics, and the learning ability of differentiable physical constraints to form a closed-loop control mechanism for physical constraint hardening that is "expressible, predictable, and learnable."

[0017] This invention's method, based on three core dimensions—constraint auditing, forward prediction, and internalized learning—utilizes dynamic mind maps for explicit modeling and reflective reinforcement of high-level planning. This ensures the reasoning process continuously maintains the structured relationships between sub-goals, macro-actions, and constraints. Furthermore, it employs Group Relative Policy Optimization (GRPO) to select the best candidate high-level action sequences within each group. Building upon this foundation, the invention further constructs a representation-motion dual-flow world model, introducing optical flow dynamics priors as explicit intermediate representations. This allows the model to learn the temporal motion trajectories of pixels while generating future images, using the geometric continuity constraint of optical flow to suppress "physical illusions" that violate physical laws. Simultaneously, by designing a joint loss that integrates physical information constraints such as non-penetration / collision, action smoothing, and friction consistency, the invention internalizes physical feasibility and safety boundaries into the inherent preferences of the policy. This forms a complete closed loop from the three levels of constraint expression, prediction, and learning, achieving a synergistic improvement in long-range robust decision-making and safe execution in complex real-world environments.

[0018] In this embodiment of the invention, the VLA model includes a high-level planning model, a planning projector, a motion-conditional world model, and a low-level visual-language-action (VLA) model. The high-level planning model generates candidate high-level plans consisting of sub-objectives and macro-actions; the planning projector converts the candidate high-level plans into a potential representation that the low-level visual-language-action model can process; the motion-conditional world model predicts the future visual state and pixel motion field after executing candidate actions; and the low-level visual-language-action model generates continuous control actions for the robotic arm.

[0019] like Figure 1 As shown, the VLA model physical constraint hardening method for power equipment operation according to an embodiment of the present invention includes steps S1 to S3. The following is in conjunction with... Figure 2 The schematic diagram illustrates each step of the method in this embodiment of the invention in detail.

[0020] Step S1: Constructing a reflection and reinforcement framework based on dynamic mind maps. Obtaining user natural language operation commands. Robots at all times Visual observations collected and the historical execution status of the current task. A dynamic mind map is constructed and updated based on the natural language operation instructions, visual observations, and historical execution states. Dynamic Mind Map The explicit task planning structure includes sub-objective nodes, macro-action nodes, and constraint nodes. Sub-objective nodes represent phased task objectives during the operation of power equipment; macro-action nodes represent at least one operation among approach, alignment, grasping, rotation, insertion / removal, and release; and constraint nodes represent at least one constraint among operation sequence, safety distance, prohibited intrusion zones, equipment state conditions, and contact requirements. Directed edges in the graph represent the sequential dependencies between sub-objectives, the realization relationships between macro-actions and sub-objectives, and the effects of constraints on corresponding macro-actions. The high-level planning model generates multiple candidate high-level plans based on natural language operation instructions, current visual observations, and a dynamic mind map. ,in The parameter is High-level planning model, Indicates the number of candidate high-level plans. Indicates the first A candidate high-level program is composed of sub-objectives or macro-actions. During the training of the high-level programming model, candidate rewards are calculated based on the task progress, security constraint satisfaction, and program consistency of the candidate programs. The relative advantages of the GRPO group were then constructed. Subsequently, a planning projector was used to convert each candidate high-rise plan into a potential representation of the plan.

[0021] Preferably, step S1 specifically includes: Step S11: Dynamic mind map construction and task status update, obtaining user natural language operation instructions. Robots at all times Visual observations collected and historical mission execution status The language encoder and visual encoder are used respectively to obtain language features and visual features, and combined with the dynamic mind map of the previous time step. Update to get the current dynamic mind map ,in Indicates the current time Dynamic mind maps; The parameter is Mind map update module; Represents a language encoder; This represents the visual encoder. Nodes describe sub-goals, macro-actions, and constraints, while edges characterize the dependencies between sub-goals and the interactions of constraints. Sub-goal nodes represent phased goals such as "confirming the target device," "locating the operating handle," and "confirming the operation is in place." Macro-action nodes represent operations such as approaching, aligning, gripping, rotating, inserting, or releasing. Constraint nodes represent operation sequence, safety distances, prohibited intrusion areas, equipment status conditions, and contact requirements. Directed edges can be used to characterize the sequential dependencies between sub-goals, the realization relationships between macro-actions and sub-goals, or the interaction relationships between constraints and corresponding macro-actions. Based on current observations and historical execution results, the status flags and confidence levels of each node are updated. Node states include at least one of the following: not started, currently executing, completed, and execution error. Node confidence levels represent the degree to which the current visual observation supports the corresponding task state.

[0022] Step S12: Candidate high-level planning generation and GRPO optimization, using natural language operation instructions. Current visual observation The dynamic mind map generated in step S11 Input a high-level planning model and sample under the same planning state. Candidate high-level planning ,in The parameter is High-level planning model; This indicates the number of candidate high-level plans; Indicates the first There are 10 candidate high-level plans; each candidate high-level plan consists of sub-objectives or parameterized macro-actions arranged in execution order. For the 10th... For each candidate high-level plan, the overall return is calculated based on the progress of the mission, the satisfaction of security constraints, and the consistency of the plan: , in Indicates the first The overall return on investment for each candidate high-level development plan; This indicates the degree to which the candidate plan advances the current task; This indicates the degree of risk that the candidate plan violates operational sequence, safety distance, prohibited intrusion areas, or equipment status constraints. This indicates the degree of consistency between the candidate plan and natural language instructions, current observations, historical execution status, and dynamic mind map; , and These are the weighting coefficients for the corresponding evaluation items. Calculate the mean and standard deviation of the returns for candidate plans in the same group. , and construct the first The relative advantages of the candidate high-level planning groups Next, we use the GRPO optimization formula:

[0023] Optimize the selection probability of each high-level plan, among which... The parameter is The relative strategy optimization formula is used to update the parameters of the high-rise planning model based on the relative merits of each candidate high-rise plan under the same planning state. , , Indicates that the current training parameters are New high-rise planning model, This indicates that the parameters before this round of GRPO parameter updates were: The old high-rise planning model, Indicates the first The first candidate plan One macro action; This indicates a macro action that was generated before this macro action; This indicates that the parameters are before and after the GRPO optimization method update. The new high-rise planning model and parameters are The probability ratio of generating corresponding macro actions by the old high-rise planning model; This indicates that the strategy updates the clipping factor; This represents the KL divergence between the current high-level planning model and the high-level planning model used for reference. Indicates the weights of the KL constraints; Indicates the first The number of macro actions contained in the plan. By maximizing This increases the probability of generating positive dominant candidate programs and decreases the probability of generating negative dominant candidate programs, making the high-level planning model more inclined to generate candidate high-level programs with high task progress, low safety risks, and strong planning consistency.

[0024] Step S13: Generation of potential representations for candidate plans. For each generated high-level candidate plan, extract the features of the candidate plan. Features of dynamic mind maps ,in Indicates a candidate planning encoder; Indicates the first Characteristics of the candidate high-level plans; Represents a graph encoder; This represents the overall characteristics of the dynamic mind map. The planning projector combines candidate planning features with mind map features and maps them to a fixed-dimensional latent space. ,in The parameter is The planning projector; Indicates feature concatenation operation; Indicates the first The potential representation of the plan corresponds to each candidate high-level plan. This potential representation encodes the current execution objective, macro-action sequence, and constraint preferences in the candidate plan, and serves as the conditional input for the VLA model to generate candidate continuous action sequences. The final result is a set of potential representations of the plans. .

[0025] Step S2: Construction of a physical illusion mitigation scheme based on optical flow dynamics prior constraints. Based on the candidate high-level plans and their potential representations, for each candidate high-level plan, the VLA model generates a corresponding candidate continuous action sequence based on the current visual observation, natural language operation instructions, and the potential representation of the plan. Then, the current visual observation and candidate actions are input into the action conditional world model. The visual representation prediction branch predicts the future visual state, and the motion field prediction branch predicts the optical flow field from the current moment to the next moment. To ensure that the world model prediction results conform to the laws of continuous motion, the following overall training loss function is constructed. ,in, Indicates the loss in future image reconstruction. This represents the optical flow consistency loss between the predicted optical flow and the reference optical flow. This represents the difference between the image obtained after geometric transformation of the current image based on predicted optical flow and the predicted future image. and These represent the weight coefficients of the corresponding loss terms. During training, the optical flow consistency loss and the optical flow-guided geometric transformation loss together constitute the dynamic consistency supervision signal, used to suppress physical illusions such as the robotic arm penetrating equipment, sudden changes in object position, and inconsistencies between the direction of motion and the visual direction of motion. The motion-conditional world model predicts the future visual and motion states corresponding to each candidate solution (candidate high-level planning), and forms a comprehensive evaluation score for the candidate solution based on the progress of the task, physical risk, and the degree of physical illusion. Select the candidate solution with the highest evaluation score. ,in, This indicates the number of the selected optimal candidate solution.

[0026] Preferably, step S2 specifically includes: Step S21: Construct a representation-motion dual-stream world model, receiving the candidate planning set output from step S1. and its planned potential representation set Regarding the first The candidate high-level planning will be based on current visual observation. Natural language operation instructions and potential representations of the plan Input a low-level VLA model to generate corresponding candidate continuous action sequences. in The parameter is Low-level VLA model; Indicates the time frame for forward forecasting; Indicates the first The candidate program corresponds to a sequence of consecutive actions. The action-conditional world model first encodes the current visual observation as a latent state. And predict the potential state at the next moment based on the candidate actions. in Indicates a visual encoder; Indicates the current potential visual state; The parameter is The action-conditional state transition module divides the predicted latent state into visual representation features and motion features. ,in, Indicates visual representation features, Motion characteristics are represented. These are obtained through both the visual decoder and the optical flow decoder. and in Represents the predicted visual image for the next moment; This represents the predicted optical flow field from the current frame to the next frame; and These represent the visual representation decoder and the optical flow decoder, respectively. By recursively executing the above state transition process according to the candidate action sequence, the future visual state sequence and optical flow field sequence corresponding to each candidate scheme are obtained. This dual-flow structure expands the world model prediction target from a single visual image to a joint prediction of visual representation and pixel motion field.

[0027] Step S22: Constructing the dynamic consistency loss and training the world model based on real neighboring visual observations (visual frames). and The reference optical flow is obtained using a pre-defined optical flow estimation method. ,in, This represents the reference optical flow estimated from real adjacent frames.

[0028] Constructing future image reconstruction loss:

[0029] in, This represents the L1 norm.

[0030] Constructing optical flow consistency loss:

[0031] Further differentiable resampling of the current visual image based on the predicted optical flow:

[0032] And construct an optical flow-guided geometric transformation loss:

[0033] in Indicates the loss in future image reconstruction; This represents the difference between the predicted optical flow and the reference optical flow; Represents a differentiable geometric transformation based on optical flow; This represents the future image obtained based on the current image and the predicted optical flow transformation. This represents the difference between the image directly predicted by the world model and the image transformed by optical flow.

[0034] Optical flow uniformity loss and geometric transformation loss are defined as dynamic uniformity monitoring signals. The overall training loss of the world model is ,in and These represent the weighting coefficients for optical flow uniformity loss and geometric transformation loss, respectively.

[0035] The overall training loss is used to update the visual encoder, action conditional state transition module, visual representation prediction branch, and motion field prediction branch, so that the predicted future image can be interpreted from the current image according to the predicted motion field, thereby suppressing physical illusions such as sudden changes in object position, robotic arms penetrating equipment, and inconsistencies between the direction of action and the direction of visual motion.

[0036] Step S23: Optical flow anomaly assessment and optimal candidate scheme selection, for two-dimensional predicted optical flow field ,in and Let the pixel displacements in the horizontal and vertical directions be represented respectively, and their divergence be calculated. Curl To detect non-physically drastic changes in optical flow over time, in one implementation, the first... Candidate schemes are defined by optical flow anomaly scores.

[0037] in Indicates the first The degree of optical flow anomaly in each candidate scheme; and These represent the weights of the divergence and curl changes, respectively. Indicates the time range for future predictions.

[0038] Based on the overall progress of the mission in the future state, planning consistency, physical risks, and the degree of optical flow anomalies, an evaluation score for candidate solutions is constructed:

[0039] in Indicates the first The comprehensive evaluation score of each candidate scheme; This indicates the degree to which the predicted future state contributes to the progress of the task; This indicates the degree of consistency between the prediction results and the operational instructions and dynamic mind map; This indicates a predicted collision, boundary crossing, or other physical risk in a future state. Indicates the degree of non-physical anomaly in the optical flow field; , , and These are the weighting coefficients for the corresponding evaluation items. The candidate solution with the highest overall evaluation score is selected. .

[0040] Step S3: Design of VLA loss based on physical information constraints. Based on the obtained optimal candidate high-level plan and its potential representation, the current visual observations are... Natural language operation instructions The optimal plan latent representation is input into the VLA model to generate continuous control actions that are actually executed. During the learning process of the corresponding control strategy, the VLA model introduces three types of basic physical constraints based on the behavioral cloning principal objective. Non-penetration and collision constraints, based on simplified environmental geometry modeling, construct a collision loss using the signed distance between the end effector or key link and the environment to suppress out-of-bounds and penetration behaviors. Action temporal smoothing constraints suppress high-frequency control jitter caused by perceived noise by using the L2 norm of the action difference between adjacent time steps. Friction consistency constraints construct soft constraints based on the Coulomb friction model, ensuring that actions conform to the friction cone feasibility condition by constraining the ratio of the tangential to normal components of the contact force, thus enabling the strategy to possess physical consistency in complex contact scenarios. This forms the overall training loss. ,in This represents the behavior cloning loss, used to constrain the model's predicted actions to closely approximate the expert-demonstrated actions. Indicates non-penetration and collision loss, used to suppress collisions and geometric penetration between the robotic arm and equipment or the environment; This represents the loss of smoothness in action timing, used to suppress abrupt changes in action between adjacent time steps and high-frequency control jitter; This represents the friction consistency loss, used to constrain the contact force to meet the feasibility condition of the friction cone. , and These represent the weighting coefficients corresponding to the three types of physical constraint losses.

[0041] Preferably, step S3 specifically includes: Step S31: Construct a VLA joint learning framework that integrates physical consistency constraints. Receive the optimal candidate plan and its potential plan representation determined in Step S2. Encode the current visual observation, natural language operation instructions, and potential plan representation separately, and then input them into the VLA action model. ,in The parameter is VLA motion model; Indicates time The output continuous control action; the continuous control action includes at least one of joint velocity, joint position increment, end position increment, end attitude increment and gripper opening / closing amount.

[0042] Introducing lightweight physical networks According to the robot's current state and candidate actions Predict the next state and related physical quantities:

[0043] in The parameter is Lightweight physical networks; This indicates the predicted next robot-environment state; This represents the predicted minimum signed distance between the robot and its environment. Indicates the tangential component of the predicted contact force; This represents the normal component of the predicted contact force. This lightweight physics network provides differentiable physical quantities for collision constraints and friction consistency constraints, enabling the corresponding losses to backpropagate to the lower-level VLA model.

[0044] Step S32: Construct the joint loss of behavior cloning and physical constraints, and predict actions based on the model. Demonstrating the movements with experts Construct behavioral cloning loss Non-penetration and collision losses are constructed based on the signed distance between the robot's end effector or key link and the environmental surface. ,in This indicates that the robot has separated from the environmental surface; Indicates contact; This indicates geometric penetration. An action temporal smoothing loss is constructed based on the differences in actions at adjacent time points. ,in, This represents the control action at the previous moment. The friction consistency loss is constructed based on the Coulomb friction model:

[0045] in Indicates the tangential component of the contact force; Indicates the normal component of the contact force; This represents the coefficient of friction of the contact surfaces; the first term is used to constrain the tangential force from exceeding the friction cone; the second term is used to constrain the normal contact force direction to be effective.

[0046] The overall training loss of the low-level VLA action model is:

[0047] in Indicates the collision loss weight; Indicates the weight of motion smoothing loss; This represents the weight of frictional consistency loss.

[0048] Through the aforementioned joint loss, while maintaining the accuracy of expert action imitation, the low-level VLA action model learns to avoid geometric penetration, suppress control jitter, and maintain contact stability.

[0049] Step S33: Use gradient descent-based optimization methods to update the parameters of the low-level VLA action model and the lightweight physics network based on the overall training loss. ,

[0050] in This represents the learning rate of the low-level VLA action model; This represents the learning rate of a lightweight physical network. and Relative to the parameter and parameters The gradient.

[0051] After training is complete, the latent representation of the plan selected in step S2 is... The current visual observation and natural language operation commands are input into the low-level VLA action model to obtain the actual continuous control actions:

[0052] The actual continuous control actions are sent to the robot's underlying controller to drive the robot to perform operations such as approaching, aligning, grasping, rotating, plugging or releasing electrical equipment.

[0053] In summary, the physical constraint hardening method for VLA models oriented towards power equipment operation presented in this invention provides a reliable hardening method for visual language action models that integrates dynamic mind maps, optical flow dynamics priors, and physical information constraints. This method achieves seamless integration from high-level constraint expression to low-level physical execution, overcoming the bottlenecks of traditional VLA models in physical feasibility modeling, long-range prediction stability, and safety boundary verifiability. Its core advantage lies in the deep integration of constraint expression, constraint prediction, and constraint internalization, thereby significantly improving the interpretability, physical consistency, and safety reliability of system decisions.

[0054] This invention designs a complete technical framework from constraint auditing and physical prediction to loss internalization, significantly improving the robustness and generalization ability of VLA models in complex real-world environments. Traditional VLA models are prone to outputting physically infeasible actions under external perturbations and dynamic changes, leading to unsafe behaviors such as boundary crossing, collisions, and slippage. This invention ensures that high-level planning always remains within the physically feasible domain through explicit constraint expression using dynamic mind maps; it suppresses physical illusions in the world model through optical flow dynamics priors, improving the physical consistency of long-range predictions; and it internalizes physical feasibility and safety into the policy's intrinsic preferences through differentiable physical constraint losses, enabling the policy to maintain physically feasible control outputs even under external perceptual perturbations such as occlusion, illumination changes, and background interference.

[0055] This invention constructs a "representable-predictable-learnable" physical constraint-reinforced closed loop at the system level, enhancing the robustness of VLA models in the face of long-range interactions and dynamic disturbances. In unstructured real-world scenarios, simple end-to-end fitting or reactive control is insufficient to handle uncertainties such as target motion, occlusion, and interference. This method utilizes dynamic mind maps for reflective reinforcement, enabling the inference process to continuously update constraints and adaptively trigger replanning; it employs an optical flow dynamics world model for physically consistent forward prediction, allowing the policy to proactively plan based on future states that conform to physical laws; and it internalizes the safety boundary into policy preferences through physical information constraint loss, ensuring that the execution process always meets physical constraints such as collision avoidance, smooth action, and frictional feasibility. Simultaneously, through a parallel asynchronous inference execution mechanism and anomaly detection gating, it improves real-time performance and robustness while ensuring safety and consistency, ensuring the reliable and coherent completion of complex long-range tasks.

[0056] Taking the task of a robotic arm performing the operation of "closing a designated disconnecting switch" on power equipment as an example, the implementation process and technical effects of the method in this embodiment of the invention are described. The robot is positioned in the switchgear operating area and uses a camera to collect visual observations of the switchgear, the disconnecting switch handle, surrounding obstacles, and the end effector of the robotic arm. It also receives the user's natural language command to "close the designated disconnecting switch." The visual observations, the natural language command, and the historical execution status of the current task are collectively input into the visual language action system of this invention.

[0057] First, the high-level planning model constructs a dynamic mind map based on natural language instructions and current visual observations. The sub-target nodes in this dynamic mind map sequentially include "confirm target switch," "locate operating handle," "adjust end effector posture," "approach and grasp handle," "rotate handle to the closed position," and "confirm switch operation in place." Macro-action nodes include identification, alignment, approach, grasp, rotation, and release. Constraint nodes include ensuring target device identity is consistent, maintaining a safe distance between the robotic arm and the cabinet, correct handle rotation direction, and confirming device status after operation. Directed edges establish the sequential dependencies between sub-targets and the effects of each constraint on the corresponding macro-actions.

[0058] Under the constraints of the updated dynamic mind map, the high-level planning model generates multiple candidate high-level plans. For example, one candidate plan directly executes the approach, grasp, and rotation operations; another candidate plan sequentially executes device identification, handle positioning, end effector attitude adjustment, slow approach, stable grasp, rotation in the specified direction, and status confirmation. Candidate rewards are calculated based on the task progress, safety constraint satisfaction, and planning consistency of each candidate plan. The high-level planning model is then optimized using GRPO (Gradual Progressive Objectives) to increase the generation probability of candidate plans containing complete operation steps and with high safety constraint satisfaction. This process enables the high-level planning model to avoid unreasonable planning such as direct operation without device identification, grasping before attitude adjustment is completed, and premature termination before the operation is fully performed. In this embodiment of the invention, the high-level planning part maintains sub-goals, macro-actions, and constraint relationships through a dynamic mind map and performs intra-group comparisons of candidate plans based on task progress, safety constraints, and consistency signals.

[0059] Subsequently, the planning projector converts each candidate high-level plan into a corresponding potential plan representation, and the low-level visual language action model generates candidate continuous action sequences based on the potential plan representation. The action conditional world model performs look-ahead predictions on different candidate action sequences, outputting the visual state and optical flow field of the robotic arm, disconnect switch handle, and surrounding equipment at future moments. By using future image reconstruction loss, optical flow consistency loss, and optical flow-guided geometric transformation loss, the motion of the robotic arm and handle in the predicted image is constrained to satisfy continuous geometric change relationships, avoiding physical illusions such as the robotic arm end passing through the cabinet, the handle rotating on its own without contact, the direction of motion inconsistent with the control action, and sudden jumps in the position of the target object. This invention uses a representation-motion dual-flow world model to simultaneously predict future visual representations and pixel motion fields, and utilizes optical flow consistency and geometric transformation constraints to improve the physical rationality of the prediction results.

[0060] This invention further evaluates each candidate scheme based on the predicted progress, collision risk, and optical flow anomaly. For example, if a candidate action sequence causes the robotic arm's end effector to approach or enter the cabinet surface during the prediction process, or if the predicted optical flow exhibits a drastic change inconsistent with the actual action direction, the evaluation score of that candidate scheme is reduced. Conversely, if another candidate action sequence enables the robotic arm to approach the handle along a continuous trajectory while maintaining a safe distance from the cabinet, the evaluation score of the corresponding candidate scheme is increased. Thus, the candidate plan with the highest comprehensive evaluation score and its potential representation are selected as the execution conditions for low-level action generation.

[0061] Finally, the low-level visual language motion model generates continuous control actions such as joint velocity, end-effector position increment, end-effector posture increment, and gripper opening / closing amount based on the selected plan latent representation. During motion model training, non-penetration and collision losses constrain the end-effector and key links of the robotic arm to prevent them from entering cabinets or other obstacles; motion temporal smoothing losses reduce abrupt changes between adjacent control actions, avoiding high-frequency jitter during approach and rotation; and friction consistency losses constrain the tangential and normal forces at the contact point between the gripper and the handle to satisfy the friction cone condition, preventing slippage during gripping and rotation. This invention incorporates three types of physical constraints—non-penetration and collision, motion smoothing, and friction consistency—into the training objectives of the low-level VLA motion model, making physical safety requirements an intrinsic preference of the motion strategy.

[0062] In this embodiment, the robotic arm performs the isolating switch closing task in the sequence of "equipment confirmation—handle positioning—posture adjustment—approach gripping—stable rotation—state confirmation". Compared to methods that rely solely on visual language features and behavioral cloning for end-to-end motion generation, this invention can reduce omissions in operation steps and constraint conflicts in the high-level planning stage, reduce prediction results that do not conform to continuous motion laws in the future state prediction stage, and reduce the risk of robotic arm colliding with the cabinet, control motion jitter, and handle gripping slippage in the low-level motion execution stage. This enables the robotic arm to complete the isolating switch closing operation with continuous, stable movements that conform to equipment operation constraints.

[0063] As can be seen, this invention achieves effective connection between high-level planning, future state prediction and low-level continuous control in power equipment operation tasks, making high-level operation steps interpretable, making the future results of candidate actions have good motion consistency, and making the final control action meet the requirements of collision avoidance, action smoothing and friction feasibility, thereby improving the physical credibility and execution safety of the visual language action system in complex power equipment operation scenarios.

[0064] This invention integrates dynamic mind maps, optical flow dynamics priors, and differentiable physical information constraints to deeply evaluate and substantially improve the physical reliability and decision robustness of VLA models in complex power environments.

[0065] In the operational scenarios of target power equipment, the VLA model carries the risk of lacking physical feasibility due to its end-to-end black-box fitting characteristics. Attackers or environmental disturbances may introduce occlusion, lighting changes, or background interference in visual observation, inducing the model to output physically infeasible actions such as penetrating the equipment surface, high-frequency jitter, or end-point collisions. This invention constructs an explicit dynamic mind map to structurally model sub-objectives, macro-actions, and physical constraints, and uses GRPO to perform intra-group optimization of candidate high-level action sequences, ensuring that high-level planning always remains within the physically feasible domain. This explicit constraint expression mechanism not only suppresses unrealistic planning detached from physical reality caused by purely semantic reasoning, but also focuses on the logical consistency between high-level intent and low-level physical execution, thereby continuously maintaining the operational safety boundary during long-term task progression.

[0066] In operational conditions characterized by incomplete perception and rapid environmental changes, this invention constructs a representation-motion dual-flow world model, introducing optical flow dynamics priors as explicit intermediate representations to evaluate whether the future states generated by the model conform to the laws of rigid body motion. By calculating the difference between the predicted optical flow field and the actual optical flow, as well as the geometric transformation error based on the motion field, this method can proactively detect and suppress physical illusions in the world model, ensuring that the strategy is based on a forward-looking plan for future states that conform to physical laws. At the decision-making level, this invention designs a joint loss that integrates physical information constraints such as non-penetration / collision, smooth motion, and friction consistency, internalizing the safety boundary into the intrinsic preference of the policy network. This invention calculates the physical compliance level of the tested strategy by quantifying the model's ability to balance meeting task progress objectives and complying with physical constraints. When online verification reveals a collision risk or friction infeasibility in predicting the next state, the lightweight physical network provided by this invention can trigger a safety degradation mechanism in real time and strengthen physical constraints based on the accumulation of abnormal data, thereby maintaining operational safety and reliability under dynamically changing power field conditions.

[0067] This invention is applicable not only to routine operations in power scenarios such as power transmission and transformation equipment inspection and distribution automation, but also to other industrial operation scenarios with extremely high physical safety requirements. By deeply embedding physical priors and dynamic constraints into the entire chain of reasoning, prediction, and execution of the VLA model, this invention can significantly enhance the safety, reliability, and physical consistency of embodied intelligent system planning and execution in complex industrial environments, fundamentally solving the problem of unsafe behaviors such as boundary crossing, collision, and slippage caused by the lack of built-in physical understanding in traditional VLA models.

[0068] In addition to the physical hardening scheme based on explicit reasoning using dynamic mind graphs and prior constraints of optical flow dynamics as described in this invention, there are several alternative implementation methods that can achieve the same or similar objectives. These methods can harden the physical credibility and security boundaries of the VLA model under different implementation conditions and hardware environments. In the high-level planning and constraint expression stage, besides using explicit dynamic mind graphs to describe sub-goals, macro-actions, and constraints with nodes and edges to characterize dependencies, directed acyclic graphs or hierarchical task network structures can also be used to maintain task logic and physical constraints. Through incremental update mechanisms, visual observations and language instructions are continuously integrated, and compact anchor point summaries are generated for consistency verification. Thus, high-level planning reflection hardening and adaptive replanning can be achieved without relying on a specific graph structure form.

[0069] In the stages of physical illusion mitigation and prospective prediction, in addition to constructing a representation-motion dual-flow world model and introducing a dense optical flow field as an explicit motion representation, scene flow or keypoint-based motion offset vector fields can be used as alternatives to motion flow paths. The geometric rationality of the generated image is constrained by optical flow consistency loss and optical flow-guided geometric transformation loss. Correspondingly, in addition to constructing gating signals based on the divergence and curl of the predicted optical flow field, the physical anomaly detection mechanism can also utilize scene flow divergence thresholds or motion field Jacobian matrix eigenvalue anomaly criteria to monitor and trigger anomaly handling in real time during the inference stage.

[0070] In the stages of physical constraint internalization and policy learning, in addition to introducing non-penetration / collision penalties based on signed distance functions, action temporal smoothing constraints, and friction cone soft constraints based on the Coulomb friction model on the basis of the behavior cloning principal objective, differentiable collision penalties based on the bounding box minimum separation vector can also be used, or the action smoothing term can be extended to a composite temporal constraint that includes jerk suppression. Alternatively, friction constraints can be replaced with friction cone projection residuals based on the maximum dissipation principle. Furthermore, the application location of physical constraints can also be moved forward from the policy network output to the latent space of the world model or the feature layer, and equivalent constraints can be achieved by applying physical feasibility regularization terms in the feature space.

[0071] During the online deployment and closed-loop verification phases, in addition to using lightweight physical networks to predict the next state in real time and verify joint limits, velocity boundaries, and collision distances, online safety filters based on artificial potential fields or control barrier functions can be employed to correct the actions output by the strategy in real time. Safety degradation strategies, besides reverting to the previous safe action, can also employ slight retraction along the negative velocity direction or switching to a preset zero-velocity / retreat action to ensure that a safe state is maintained when a physical violation is detected. All of the above alternative solutions can achieve the core objective of this invention to varying degrees: strengthening the closed loop through expressible, predictable, and learnable physical constraints to improve the long-range robust decision-making and safe execution capabilities of the VLA model in power equipment operation.

[0072] Embodiments of the present invention also provide a computer device, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores operating parameter data for various components. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements the steps of the method according to embodiments of the present invention.

[0073] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0074] Embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method of the embodiments of the present invention.

[0075] Embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the method of the embodiments of the present invention.

[0076] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for hardening the physical constraints of a VLA model for power equipment operation, characterized in that, The VLA model includes a high-level planning model, a planning projector, a world model of action conditions, and a low-level VLA model; the method includes: Step S1: Obtain natural language operation instructions, current visual observations, and the historical execution status of the current task; construct the current dynamic mind map; the high-level planning model generates multiple candidate high-level plans based on the natural language operation instructions, current visual observations, and dynamic mind map; optimize the selection probability of each high-level candidate plan through the group relative strategy optimization formula; and use the planning projector to convert each candidate high-level plan into a potential plan representation. Step S2: Based on the candidate high-level plans and their potential representations, for each candidate high-level plan, generate a corresponding candidate action sequence based on the current visual observation, natural language operation instructions and potential representations of the plan, and input it into the action conditional world model to predict the future visual state and motion state corresponding to each candidate high-level plan. Based on the task progress, physical risk and physical illusion of the future state, a comprehensive evaluation score is formed, and the candidate high-level plan with the highest evaluation score is selected as the optimal candidate high-level plan. Step S3 introduces non-penetration and collision constraints, action timing smoothing constraints, and friction consistency constraints to form the overall training loss. The low-level VLA model is updated based on the overall training loss. The current visual observation, natural language operation instructions, and plan potential representation of the optimal candidate high-level plan are input into the low-level VLA model to obtain the actual continuous control actions.

2. The method as described in claim 1, characterized in that, Step S1 includes: Step S11: Obtain natural language operation instructions, current visual observation, and the historical execution state of the current task. Use the language encoder and visual encoder to obtain language features and visual features respectively, and combine them with the dynamic mind map of the previous moment to update the current dynamic mind map. Step S12: Input the natural language operation instructions, current visual observation and dynamic mind map into the high-level planning model, sample multiple candidate high-level plans, calculate the comprehensive reward for each candidate high-level plan based on the progress of the task, the satisfaction of safety constraints and the consistency of the plan, construct the group relative advantage of each candidate high-level plan, and select the candidate high-level plan that maximizes the group relative strategy optimization formula. Step S13: For each candidate high-level plan, extract candidate plan features and dynamic mind map features respectively, combine the candidate plan features and dynamic mind map features and map them to a fixed-dimensional latent space to obtain the potential representation of the plan corresponding to the candidate high-level plan.

3. The method as described in claim 2, characterized in that, The dynamic mind map includes sub-goal nodes, macro-action nodes, and constraint nodes. Sub-goal nodes represent the phased task objectives in the operation of power equipment. Macro-action nodes represent at least one operation among approach, alignment, grasping, rotation, insertion / removal, and release. Constraint nodes represent at least one constraint among operation sequence, safety distance, prohibited intrusion zone, equipment status conditions, and contact requirements. Directed edges represent the sequential dependencies between sub-goals, the realization relationships between macro-actions and sub-goals, and the effects of constraints on corresponding macro-actions.

4. The method as described in claim 3, characterized in that, In step S12, the overall return is calculated as follows: , in, Indicates the first The overall return on investment for each candidate high-level plan Indicates the first The progress of each candidate high-level plan in advancing current tasks. Indicates the first The degree of risk of each candidate high-level plan violating constraints. Indicates the first The degree of consistency between the candidate high-level plans and natural language instructions, current visual observations, historical execution status, and dynamic mind maps. , and They are respectively , and Weighting coefficients; Calculate the mean return of candidate plans in the same group. and standard deviation : , , Construct the first The relative advantages of the candidate high-level planning groups : 。 5. The method as described in claim 4, characterized in that, The group relative strategy optimization formula is: in, The parameter is The group relative strategy optimization formula, Indicates the number of candidate high-level plans. Indicates the first The number of macro-level actions included in the candidate high-level plans. This indicates that the strategy updates the clipping factor. This represents the KL divergence between the current high-level planning model and the high-level planning model used for reference. Indicates the weights of the KL constraints. This indicates that the parameters are before and after the GRPO optimization method update. The new high-rise planning model and parameters are The probability ratio of generating corresponding macro actions by the old high-rise planning model; in, Indicates that the current training parameters are New high-rise planning model, This indicates that the parameters before this round of GRPO parameter updates were: The old high-rise planning model, Indicates the first The first candidate high-level plan A macro action, This indicates a macro action that was generated before this macro action. Represents natural language operation commands. Indicates time Visual observation, A dynamic mind map representing time t.

6. The method according to any one of claims 1-5, characterized in that, Step S2 includes: Step S21: Input the current visual observation, natural language operation instructions, and plan potential representation into the low-level VLA model to generate the corresponding candidate action sequence. Use the action conditional world model to encode the current visual observation into a potential state, and predict the potential state at the next moment based on the candidate actions. Divide the predicted potential state into visual representation features and motion features. Predict the visual image at the next moment and the optical flow field from the current frame to the next frame through the visual decoder and optical flow decoder, respectively. By recursively executing according to the candidate action sequence, obtain the future visual state sequence and optical flow field sequence corresponding to each candidate high-level plan. Step S22: Based on the adjacent current visual observation and the next visual observation, obtain the reference optical flow using the optical flow estimation method; construct the future image reconstruction loss based on the predicted next visual image and the next visual observation; construct the optical flow consistency loss based on the predicted optical flow and the reference optical flow; perform differentiable resampling on the current visual observation based on the predicted optical flow; and construct the geometric transformation loss based on the differentiable resampling of the predicted next visual image and the current visual observation to obtain the overall training loss of the world model. Step S23: Calculate the divergence and curl of the predicted optical flow field, define an optical flow anomaly score for each candidate high-rise plan, and construct an evaluation score for candidate schemes by comprehensively considering the task progress, planning consistency, physical risk, and optical flow anomaly degree in the future state, and select the candidate high-rise plan with the highest evaluation score.

7. The method as described in claim 6, characterized in that, World model overall training loss for: , in, Indicates the loss in future image reconstruction. Indicates optical flow uniformity loss, Represents the geometric transformation loss. and These represent the weighting coefficients for optical flow uniformity loss and geometric transformation loss, respectively. in, Indicates the current time Visual observation, Indicates the next moment Visual observation, This represents the predicted visual image for the next moment. This represents a future image obtained based on current visual observations and predicted optical flow transformations. This represents the predicted optical flow field from the current frame to the next frame. Indicates the reference optical flow. This represents a differentiable geometric transformation based on optical flow.

8. The method as described in claim 7, characterized in that, The following are the evaluation scores for candidate solutions: in, Indicates the first The comprehensive evaluation score of each candidate high-rise building plan. This indicates the degree to which the predicted future state contributes to the progress of the task. This indicates the degree of consistency between the prediction results, operational instructions, and the dynamic mind map. This indicates the physical risk in the predicted future state. Indicates the degree of non-physical anomaly in the optical flow field. , , and They are respectively , , and Weighting coefficients; ,in, Indicates the first The degree of optical flow anomaly in the candidate high-rise planning and These represent the weights for the changes in divergence and curl, respectively. Indicates the time range for future forecasts. and These represent the divergence and curl of the predicted optical flow field, respectively.

9. The method according to any one of claims 1-5, characterized in that, Step S3 includes: Step S31: Receive the optimal candidate high-level plan and its potential plan representation. Encode the current visual observation, natural language operation instructions and the potential plan representation respectively and input them into the VLA model. Introduce a lightweight physical network and predict the next state based on the robot's current state and candidate actions. Step S32: Based on the predicted action and the expert demonstration action, construct the behavior cloning loss, construct the non-penetration and collision loss based on the signed distance between the robot and the environmental surface, construct the action temporal smoothing loss based on the action difference between adjacent time moments, construct the friction consistency loss based on the Coulomb friction model, and obtain the overall training loss of the low-level VLA model. Step S33: Using a gradient descent-type optimization method, the parameters of the low-level VLA model and the lightweight physical network are updated according to the overall training loss. After training is completed, the potential representation of the optimal candidate high-level plan, the current visual observation, and the natural language operation instructions are input into the low-level VLA model to obtain the actual continuous control actions. The actual continuous control actions are sent to the robot's bottom-level controller to drive the robot to complete the operation of the power equipment.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-9.