Embodied intelligent behavior generation and strategy optimization method based on visual perception driving

Abstract

The application provides a somatic intelligent behavior generation and strategy optimization method based on visual perception driving, relates to the technical field of somatic intelligent control and robot perception decision, constructs a unified behavior decision field, jointly evaluates visual implementability, intervention necessity, target consistency, environmental risk, environmental uncertainty and somatic accessibility, generates an optimal behavior sequence containing cognitive detection actions and entity operation actions, and implements directional optimization update according to a deviation source after execution, so as to improve the task success rate, execution safety and strategy stability in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligent control and robot perception and decision-making technology, specifically to a method for generating embodied intelligent behavior and optimizing strategies based on visual perception. Background Technology

[0002] Existing embodied intelligence technologies mainly fall into three processing routes. The first route first uses object detection, semantic segmentation, depth estimation, or visual encoding models to obtain scene features, and then inputs the scene features into a behavior cloning model, reinforcement learning model, or imitation learning model to output actions. The second route first locates areas that can be grasped, pushed, opened, or pressed, and then the planner and controller generate actions. The third route uses a world model or hierarchical planner to establish a task sequence, and then the low-level controller implements trajectory tracking.

[0003] While the aforementioned technical approaches can accomplish specified tasks in some standardized scenarios, they still have significant shortcomings in semi-occluded environments, dynamic interference environments, confined operating environments, and environments with high security constraints. First, visual results are typically only used as state input and cannot directly characterize whether an action has sufficient visual basis at the current moment, current position, and current physical conditions. Second, the action planning process usually prioritizes target salience and geometric reachability, making it difficult to determine whether the current action has a decisive role in task progress. Third, visual understanding, action planning, control execution, and policy updates are usually connected via a serial interface; once execution fails, the system struggles to distinguish whether the problem stems from errors in visual understanding, incorrect intervention sequence, incorrect embodied constraint estimation, or action execution errors. Fourth, existing solutions typically do not incorporate the recovery costs after failure into the main action generation process, leading the system to easily select high-reward but highly vulnerable actions in real-world environments. Summary of the Invention

[0004] Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a method for generating embodied intelligent behaviors and optimizing strategies based on visual perception, thus solving the problems of existing technologies.

[0005] Technical solution To achieve the above objectives, the present invention provides the following technical solution: a method for generating and optimizing embodied intelligent behavior based on visual perception, comprising the following steps: collecting environmental visual data, depth or geometric data, body state data, and task target data of the embodied intelligent agent at the current moment; extracting an object set, an interaction surface set, an occlusion relationship set, and a risk distribution set based on the environmental visual data and depth or geometric data; constructing a unified behavior decision field based on the object set, interaction surface set, occlusion relationship set, risk distribution set, body state data, and task target data, wherein the unified behavior decision field simultaneously characterizes the visual feasibility and intervention of candidate actions. Necessity, goal consistency, environmental risk, environmental uncertainty, and embodied accessibility are considered. A comprehensive priority is generated for the candidate action set based on the unified behavioral decision field. When the candidate action with the highest comprehensive priority is a cognitive detection action, the cognitive detection action is executed, and environmental visual data, depth or geometric data, and body state data are updated. When the candidate action with the highest comprehensive priority is a physical operation action, the physical operation action is executed, and the execution result is obtained. The execution deviation is calculated based on the execution result, and the source of the execution deviation is determined. The visual feasibility assessment parameters, intervention necessity assessment parameters, embodied accessibility assessment parameters, and action generation parameters are updated in a targeted manner according to the source of the execution deviation.

[0006] Preferably, the visual feasibility is determined as follows: an object attribute vector is established for each candidate object or candidate interaction region; an action semantic encoding vector is established for each candidate action; the matching value between the object attribute vector and the action semantic encoding vector is calculated; and the visual feasibility evaluation results corresponding to the candidate interaction position, candidate action, and prediction time window are obtained by combining the distance term from the candidate interaction position to the representative interaction center of the object and the deviation term between the current prediction time window and the optimal interaction time window of the target object.

[0007] Preferably, the necessity of intervention is determined by: predicting the future task state after performing the candidate action on the target object and the future task state when the candidate action is not performed on the target object; calculating the task value corresponding to the two future task states respectively; and determining the necessity of the candidate action for advancing the current task based on the difference between the two task values.

[0008] Preferably, the candidate action set consists of grasping actions, pushing actions, placing actions, delivering actions, circling actions, viewpoint adjustment actions, tentative contact actions, and short pause actions; the system reorders the candidate action set according to the degree of occlusion, the degree of action ambiguity, and the prediction variance; when the degree of occlusion, the degree of action ambiguity, or the prediction variance exceeds the corresponding threshold, the overall priority of viewpoint adjustment actions, tentative contact actions, and short pause actions is increased.

[0009] Preferably, the physical accessibility is jointly determined by the degree of kinematic inaccessibility, joint load, energy consumption load, body stability margin, and line-of-sight maintenance margin; the line-of-sight maintenance margin is used to characterize the degree to which the target object is within the effective field of view of the sensor after the candidate action is performed.

[0010] Preferably, when generating the optimal behavior sequence based on the unified behavior decision field, both an action switching penalty term and an irreversible penalty term are introduced; the irreversible penalty term is determined by the relationship between the recovery cost of restoring the next state to a state where the task can continue to be executed safely and the preset recovery budget; the optimal behavior sequence is generated according to the principle of high overall benefit, few action switching, and low recovery cost.

[0011] Preferably, the execution deviation is decomposed into visual understanding deviation, intervention necessity judgment deviation, embodied accessibility judgment deviation, and action execution deviation; when the visual understanding deviation accounts for the highest proportion, the visual feasibility assessment parameters are updated; when the intervention necessity judgment deviation accounts for the highest proportion, the counterfactual prediction parameters and action ranking parameters are updated; when the embodied accessibility judgment deviation accounts for the highest proportion, the embodied constraint mapping parameters are updated; when the action execution deviation accounts for the highest proportion, the action decoding parameters and control parameters are updated.

[0012] Preferably, the system corresponding to the visual perception-driven embodied intelligent behavior generation and strategy optimization method includes: a visual information acquisition module for acquiring environmental visual data, depth or geometric data; a scene parsing module for generating an object set, an interaction surface set, an occlusion relationship set, and a risk distribution set; a body state acquisition module for acquiring chassis state, robotic arm state, end effector state, and energy state; a unified behavior decision field construction module for generating a unified behavior decision field based on the object set, interaction surface set, occlusion relationship set, risk distribution set, body state data, and task target data; an optimal behavior generation module for generating a comprehensive priority of the candidate action set based on the unified behavior decision field and outputting the optimal behavior sequence; a motion control and execution module for executing the optimal behavior sequence; an execution deviation analysis module for determining the source of execution deviation based on the execution result; and a directional optimization update module for updating relevant parameters in the unified behavior decision field construction module and the optimal behavior generation module according to the source of execution deviation.

[0013] Preferably, the system further includes an experience distillation module; the experience distillation module is used to compress the successful behavior sequences, failure recovery paths and corresponding execution deviations obtained during multiple task executions into a strategy prototype library, and to call the initial behavior sequence according to the similarity between the current scene interaction state and the strategy prototype library when a new task arrives.

[0014] Beneficial effects This invention provides a method for generating embodied intelligent behavior and optimizing strategies based on visual perception. It has the following beneficial effects: 1. This invention revolves around a unified behavioral decision-making field, organizing visual feasibility assessment, intervention necessity assessment, goal consistency assessment, risk assessment, environmental uncertainty assessment, and embodied accessibility assessment. This ensures that visual perception results are no longer passively fed into the subsequent planner as state input, but rather directly characterize, during the behavior generation stage, whether the current visual evidence is sufficient to support action execution, whether the current action plays a crucial role in task advancement, and whether the current entity truly possesses the conditions for execution. Therefore, the system can simultaneously eliminate candidate actions with insufficient visual evidence, excessive risk, or physical inaccessibility before action generation, reducing the problems of repetitive replanning, action rollback, and control jitter caused by the traditional sequential approach of first identifying, then planning, and then remediating during execution.

[0015] 2. This invention integrates perspective adjustment actions, tentative contact actions, and short pause actions with grasping actions, pushing actions, delivering actions, placing actions, and circling actions into the same candidate action set, and ranks all types of actions on the same scale through a unified behavior decision field. When the environment has occlusion, reflection, blurred boundaries, dynamic interference, or the target state is unstable, the system prioritizes exploratory actions that can reduce uncertainty before proceeding to the formal operation action, thus making evidence gathering before execution an algorithm-inherent behavior generation result. Compared with existing methods that rely solely on low-level emergency stops or re-identification after execution, this technical solution can significantly reduce the probability of false grasps, false collisions, blind operations, and high-risk contacts.

[0016] 3. This invention introduces embodied reachability assessment, line-of-sight margin assessment, and recovery cost constraints into a unified behavior decision-making field. This requires candidate actions to not only meet geometric reachability requirements but also to satisfy conditions such as posture stability, manageable joint load, executable control links, and the target remaining within the effective field of view after the action is executed. Based on this technical solution, the system can prioritize behavior paths that allow for rapid reversal after failure, continuous observation during execution, and seamless transition to subsequent actions in scenarios such as narrow passages, partially obscured grasping, collaborative mobile operations, and high safety requirements. This enhances continuous operation capabilities and system safety in real-world environments.

[0017] 4. This invention establishes an execution deviation analysis and targeted optimization update mechanism, subdividing execution errors into visual understanding deviation, intervention necessity judgment deviation, embodied reachability judgment deviation, and action execution deviation. It updates the corresponding module parameters according to the source of the deviation, rather than indiscriminately transmitting data back to the entire system. This technical solution can accurately locate the cause of failure, avoiding the parameter drift, low learning efficiency, and slow scene adaptation problems that occur in traditional end-to-end updates, thereby improving online incremental learning efficiency and cross-scene transfer capabilities.

[0018] 5. This invention presents a complete technical closed loop from data acquisition, data cleaning, time synchronization, sample construction, offline training, online deployment, execution feedback to incremental updates, and provides rules for sensor configuration range, thread partitioning, control cycle, upper limit of candidate objects, number of candidate interaction points, and safety threshold settings. Because this solution not only explains the decision-making principle but also the data source, data usage process, training and deployment logic, and hardware / software interface relationships, it can be directly applied to scenarios such as mobile operation robots, ward companion robots, warehouse sorting robots, industrial collaborative robots, and inspection robots, and has a clear engineering implementation path. Attached Figure Description

[0019] Figure 1 This is a system architecture diagram of the embodied intelligent behavior generation and strategy optimization algorithm based on visual perception driven by the present invention.

[0020] Figure 2 This is a schematic diagram illustrating the construction principle of the unified behavior decision field of this invention.

[0021] Figure 3 This is a flowchart of behavior generation based on a unified behavior decision field, as presented in this invention.

[0022] Figure 4 This is a schematic diagram illustrating the deviation-oriented update mechanism of the present invention.

[0023] Figure 5 This is the execution closed-loop diagram of the present invention applied to a mobile operating robot.

[0024] Figure 6 This is a closed-loop diagram showing the data sources, model training, and online deployment of this invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Specific Implementation Example 1: The visual perception-driven embodied intelligent behavior generation and strategy optimization method does not treat vision, planning, control, and learning as independent modules, but rather models them around a unified behavioral decision field. This unified behavioral decision field takes candidate interaction locations, action primitives, body states, and prediction time windows as inputs, and uses visual feasibility, intervention necessity, goal consistency, environmental risk, environmental uncertainty, and embodied accessibility as core evaluation dimensions to uniformly rank and jointly constrain candidate actions.

[0027] Within each decision-making cycle, the system first collects environmental visual data, depth or geometric data, body state data, and task target data. Then, it analyzes the object set, interaction surface set, occlusion relationship set, and risk distribution set. Subsequently, the unified behavior decision field construction module comprehensively calculates the visual feasibility, intervention necessity, target consistency, environmental risk, environmental uncertainty, and embodied accessibility of candidate actions to obtain the comprehensive priority of each candidate action. The candidate actions include both physical operation actions and cognitive detection actions. When there are occlusions, insufficient viewing angles, reflections, dynamic interference, or blurred boundaries in the environment, the system prioritizes the execution of cognitive detection actions to supplement environmental evidence. When the formal operation conditions are met, the system then executes grasping, pushing, placing, delivering, or bypassing actions. After the action is executed, the system determines the source of the error based on the deviation between the predicted result before execution and the actual result after execution, and updates the corresponding parameters accordingly.

[0028] Compared with existing serial technical approaches, this invention describes embodied behavior generation as an optimal behavior sequence generation problem in a unified behavior decision field, thereby allowing visual evidence, task value, risk constraints, ontological capabilities, and recovery costs to be considered simultaneously within the same computational framework.

[0029] Formula (1) Unified Behavioral Decision Field: ; in, denoted as the comprehensive decision value of the candidate actions at the t-th decision time; t represents the discrete decision time index; x represents the candidate interaction position, represented by three-dimensional coordinates or a combination of pixel and depth coordinates; q represents the current body state vector of the embodied agent, which consists of chassis position, chassis attitude, robotic arm joint angle, robotic arm joint velocity, end effector opening / closing state, load state, and energy state; u represents the action primitive; τ represents the prediction time window. , , , and These represent the weight coefficients for the visual feasibility item, intervention necessity item, goal consistency item, environmental risk item, and environmental uncertainty item, respectively. Indicates the visual feasibility assessment value; Indicates the assessment value of the necessity of intervention; This represents the target consistency assessment value; Indicates the environmental risk assessment value; This represents the assessment value of environmental uncertainty. The symbol "+" indicates the gain superposition relationship, the symbol "-" indicates the penalty relationship, and the symbol "·" indicates the multiplicative modulation of the aforementioned comprehensive value by the embodied reachability modulation term.

[0030] Formula (2) Visual feasibility assessment: ; in, Σ represents the visual feasibility at the t-th decision time, given the candidate interaction position x, candidate action u, and prediction time window τ; Σ represents the summation of all candidate objects or candidate interaction regions; i represents the index of the candidate object or candidate interaction region. This represents the total number of candidate objects or candidate interaction regions at the t-th decision time. σ represents the spatial influence coefficient of the i-th candidate object on position x; σ represents the normalized activation function, used to map intermediate scores to bounded evaluation values; Let represent the object attribute vector of the i-th candidate object at the t-th decision time. The object attribute vector includes material attributes, boundary attributes, surface normal attributes, texture attributes, contact stability attributes, and local occlusion attributes. The superscript T indicates the transpose operation. e(u) represents the action semantic encoding vector corresponding to action u. The action semantic encoding vector is used to characterize the action's requirements for contact area, force direction, and spatial posture. Indicates the distance penalty coefficient; This represents the representative interaction center between position x and the i-th object. The distance between them; This represents the representative interaction center of the object; Indicates the time window penalty coefficient; This represents the optimal interaction time window between the current prediction time window τ and the i-th object. The absolute deviation between them.

[0031] Formula (3) Intervention Necessity Assessment: ; in, V(·) represents the degree of necessity for performing action u at position x to advance the task; V(·) represents the value evaluation function for future task states; This represents the predicted future task state at time t+τ, given that action u is performed on the i-th object. This represents an external intervention event that performs action u on the i-th object; This represents the predicted future task state at time t+τ without performing action u on the i-th object. This indicates that the external intervention event will not be implemented; the difference in square brackets represents the difference in the future mission value between "implementing the action" and "not implementing the action"; the larger the difference, the more critical the action is to the current mission.

[0032] Formula (4) Objective Consistency Assessment: ; in, This represents the degree of consistency between action u and the task objective after execution at position x; cos represents the cosine similarity operation; g represents the task objective vector, which is jointly encoded by the target object category, target position, target pose, target process constraints, and target safety constraints. This represents the predicted future task situation vector after performing action u at position x; the greater the cosine similarity, the more consistent the future task situation is with the target requirements.

[0033] Formula (5) Environmental Risk Assessment: ; in, ∫ represents the cumulative environmental risk corresponding to position x performing action u within time window τ; ∫ represents the integral accumulation over the spatiotemporal neighborhood; Let x represent the spatiotemporal neighborhood centered at position x and covering the prediction time window τ; y represents the integral variable. Indicates the collision risk weight; This represents the collision risk density at position y at the t-th decision time; Indicates the risk weight of dangerous contact force; This represents the hazardous contact force risk density at location y; Indicates the risk weight of slippage or fall; This indicates the density of the risk of slippage or fall at position y; Indicates the risk weight of human-machine proximity; This indicates the risk density at location y where there is a dangerous proximity to a human body or a sensitive subject. This represents the integral variable relating the risk distribution within the spatiotemporal neighborhood.

[0034] Formula (6) Environmental uncertainty assessment: ; in, This represents the uncertainty penalty value for the candidate action; Indicates the weight of the occlusion penalty; This represents the degree of occlusion at position x at the t-th decision time; H(·) represents the action ambiguity penalty weight; H(·) represents the entropy function. Let x represent the posterior choice distribution of action u given position x; This represents the weights that penalize the model's prediction variance. Let x represent the prediction covariance matrix given position x and action u. This represents the Frobenius norm, used to characterize the overall magnitude of the prediction covariance matrix.

[0035] Formula (7) Embodied Accessibility Assessment: ; in, The modulated value represents the embodied reachability; exp represents the exponential function. , , and These represent the weighting coefficients for kinematic inaccessibility, joint load and energy consumption load, body stability margin, and visual retention margin, respectively. This indicates the degree of kinematic inaccessibility of performing an action u at position x while in body state q; This represents the joint load and energy consumption load when performing action u under body state q. Indicates the body's stability margin; This indicates the line-of-sight margin when the body state q is close to position x; "+" indicates a gain that increases the embodied accessibility, and "-" indicates a penalty that decreases the embodied accessibility.

[0036] Formula (8) Optimal behavior sequence generation objective: ; in, Represents the optimal sequence of actions; This indicates selecting the action sequence with the largest objective function from the set of candidate action sequences; π represents the candidate action sequence. Represents the set of candidate behavior sequences; This indicates that the accumulation is performed over the planning time domain from 0 to T; T represents the length of the planning time domain; x(t), q(t), u(t), and τ(t) represent the interaction position, body state, action primitive, and prediction time window at time t, respectively; The first derivative of bodily state with respect to time; The motion metric matrix represents the body state q(t); This indicates the penalty weight for effort exerted during exercise; Indicates the penalty for switching actions; This indicates that the penalty weight for the action is being switched. Indicates an irreversible penalty; dt represents the irreversible penalty weight; dt represents the time integral variable.

[0037] Formula (9) execution deviation analysis: ; in This represents the overall execution deviation in the t-th decision-making cycle; and These represent the weighting coefficients for visual observation bias, field summary bias, target state bias, and distribution bias, respectively. Denotes the norm 1; This represents the actual visual observation results after execution; Indicates the execution of an action Previous predictions of visual observations at the next moment; Represents the square norm 2; This represents the actual field summary vector after execution; This represents the field summary vector predicted before execution; This represents the key state vector of the actual task after execution; This represents the key state vector of the task predicted before execution. Indicates the Kullback-Leibler divergence; This represents the posterior distribution of the prediction before execution; This represents the true posterior distribution obtained after re-estimation following execution.

[0038] Formula (10) for updating directional parameters: ; in, Indicates the parameter update amount; This represents the set of trainable parameters in the unified behavior decision field construction module and the behavior generation module. Indicates the learning rate; and These represent the update weights for visual understanding bias, intervention necessity judgment bias, embodied accessibility judgment bias, and action execution bias, respectively. Represents the parameter set Calculating the gradient; This indicates a visual comprehension bias; This indicates a bias in the judgment of the necessity of intervention; This indicates a bias in the assessment of embodied accessibility; This indicates a deviation in the execution of the action.

[0039] Formula (11) Overall optimization objective: ; in, Represents the total loss function; Represents the mathematical expectation; Indicates the reward discount factor; Indicates the immediate task reward at time t; Indicates the weight of the execution deviation item; Indicates the weight of the irreversible penalty term; Indicates the weight of the penalty for switching actions; Indicates the weight of risk items; and These represent the interaction position, action primitive, and prediction time window at time t, respectively.

[0040] like Figure 1 As shown, the system of the present invention generally includes a visual information acquisition module, a multimodal visual analysis module, a scene object and interaction area extraction module, a visual feasibility assessment module, an intervention necessity assessment module, a goal consistency and risk assessment module, an environmental uncertainty assessment module, a robot body state acquisition module, an embodied reachability assessment module, a unified behavior decision field construction module, an optimal behavior generation module, a cognitive detection action generation module, an entity operation action generation module, a motion control and execution module, an execution result feedback acquisition module, an execution deviation analysis module, and a directional optimization and update module.

[0041] The visual information acquisition module acquires RGB images, depth maps, point clouds, or video streams; the multimodal visual analysis module performs object detection, instance segmentation, depth reconstruction, boundary extraction, and motion estimation; the scene object and interaction region extraction module outputs target objects, obstacle objects, accessible surfaces, passable areas, occluded areas, and danger zones; the visual feasibility assessment module generates attribute vectors for each candidate object and calculates the local feasibility of each action; the intervention necessity assessment module calculates the difference in future task value between "performing the action" and "not performing the action" based on a counterfactual predictor; and the goal consistency and risk assessment module calculates the degree of consistency between the post-action result and the task goal, and superimposes collision, contact force, slippage, fall, and human factors. The system assesses the proximity risk of the machine; the environmental uncertainty assessment module calculates the degree of occlusion, the degree of action ambiguity, and the model prediction variance; the body state acquisition module collects data from the chassis odometer, robotic arm joint angles, joint speeds, end effector opening and closing, load, and battery status; the embodied reachability assessment module determines whether an action is truly executable based on kinematic solution results, torque margin, stability margin, and field of view coverage results; the unified behavior decision field construction module maps the above assessment values ​​to the same priority space; the optimal behavior generation module outputs the behavior sequence at the current moment according to priority; the motion control and execution module executes the behavior sequence; the execution deviation analysis module compares the predicted results before execution with the actual results after execution; and the directional optimization update module updates relevant parameters according to the source of deviation.

[0042] like Figure 2 As shown, the unified behavioral decision field is composed of visual feasibility information, intervention necessity information, goal consistency information, risk information, environmental uncertainty information, and embodied accessibility information. The system does not directly perform local filtering of candidate actions; instead, it first maps various types of information into the unified behavioral decision field, so that each candidate interaction position and each candidate action corresponds to a comparable comprehensive decision value.

[0043] In actual calculations, visual feasibility information answers questions such as "Is the current view clear? Is the contact area complete? Is the current time window appropriate?"; intervention necessity information answers "Is the current action critical to task progress?"; goal consistency information answers "Will the action bring us closer to the task objective?"; risk information answers "Will the action result in collisions, instability, or dangerous approaches?"; environmental uncertainty information answers "Are occlusions, reflections, dynamic interferences, and boundary blurring sufficient to affect the reliability of the decision?"; and embodied accessibility information answers "Is the current body capable of performing the action, and can it maintain stable observation and continue execution afterward?" After these pieces of information are jointly ordered within a unified behavioral decision-making field, the system can naturally make a decision between "see clearly before operating" and "directly implement the formal operation."

[0044] like Figure 3 As shown, the behavior generation process of the present invention includes inputting environmental observations and task information, constructing a candidate interaction point set, generating a candidate action set, calculating the priority of candidate actions, detecting high-uncertainty candidate actions, introducing cognitive detection actions, reordering candidate actions, generating the optimal behavior sequence, executing actions, updating environmental observations, and determining the task completion status.

[0045] The candidate interaction point set is constructed using the following rules: for each target object, edge points, normal stable points, contact reachable points, and line-of-sight points are extracted; for the movement channel, chassis passage points, detour turning points, and obstacle avoidance buffer points are extracted; for the scene to be explored, camera view adjustment points and trial contact points are extracted. The candidate action set is constructed using discrete action primitives: for grasping actions, the end-effector contact point, gripping direction, and approach path are extracted; for pushing actions, the force application point, force direction, and force amplitude are extracted; for placing actions, the target placement point, placement posture, and release sequence are extracted; for delivering actions, the delivery direction, end-effector hovering position, and safe approach distance are extracted; for circling actions, the chassis trajectory and turning sequence are extracted; for view adjustment actions, the camera posture change and chassis micro-movement are extracted; for trial contact actions, the contact depth and upper limit of contact force are extracted; and for short pause actions, the waiting time is extracted. After each round of action execution, the system re-collects environmental information and recalculates the priority of candidate actions.

[0046] like Figure 4 As shown, this invention does not adopt a single feedback method of task success or failure, but compares the predicted results before execution with the actual results after execution item by item, forming four types of biases: visual understanding bias, intervention necessity judgment bias, embodied accessibility judgment bias, and action execution bias.

[0047] Visual understanding biases include target boundary misalignment, incorrect occlusion assessment, missed risk areas, and incorrect surface normal estimation; biases in intervention necessity assessment include incorrect sequence of preliminary actions, incorrect prioritization of non-critical objects, and incorrect neglect of critical obstructions; biases in embodied accessibility assessment include incorrect workspace assessment, incorrect joint torque estimation, incorrect chassis stability estimation, and incorrect line-of-sight estimation; and biases in action execution include excessive contact force, trajectory deviation, timing deviation, and release deviation. The directional optimization update module updates corresponding parameters based on the percentage of bias, avoiding indiscriminate updates across the entire network.

[0048] like Figure 5 As shown, in the application scenario of mobile operation robots, environmental visual input, task target input, and robot body state input are jointly entered into the unified behavior decision field construction module, and respectively affect the chassis motion decision, robotic arm motion decision, and cognitive detection motion decision.

[0049] The chassis motion decision-making system generates forward, backward, lateral, turning, and circling actions; the robotic arm motion decision-making system generates extension, retraction, lifting, pressing down, rotating, gripping, and releasing actions; and the cognitive detection motion decision-making system generates perspective adjustment, tentative contact, and brief pause actions. The system prioritizes these three types of actions within the same priority space, ensuring that chassis repositioning, robotic arm contact, and detection actions are invoked in the order most advantageous to the task, thus forming a complete closed loop of perception, decision-making, execution, and feedback.

[0050] like Figure 6 As shown, the data sources, model training, and online deployment of this invention revolve around a closed-loop process: multi-source data acquisition, calibration and time synchronization, sample construction and dataset management, label generation and quality review, model training, unified behavioral decision field parameter initialization, online deployment inference, behavior execution and result recording, execution deviation analysis, and incremental updates, forming a complete implementation path from data entering the system to continuous model optimization.

[0051] In terms of data sources, the system simultaneously receives airborne vision data, depth data, point cloud data, robot body state data, end effector force data, task instruction data, and historical execution log data. Airborne vision data is acquired by RGB cameras, depth cameras, and wide-angle auxiliary cameras; robot body state data is acquired by chassis encoders, robotic arm joint encoders, inertial measurement units, and end effector state interfaces; task instruction data comes from a host scheduling system, hospital nursing task system, warehouse task management system, or a human teaching terminal; historical execution log data is automatically recorded by the system during online operation, including the input state, candidate interaction points, candidate actions, decision results, execution results, execution deviations, and recovery paths for each action.

[0052] During data usage, the system first performs timestamp alignment, intrinsic and extrinsic parameter calibration, and coordinate system transformation on data from different sensors, unifying visual coordinates, depth coordinates, point cloud coordinates, robot base coordinates, and end-effector coordinates to the same reference coordinate system. After calibration, the system performs distortion correction on image data, hole filling and filtering on depth data, voxel downsampling and outlier removal on point cloud data, and interpolation synchronization on ontology state data. It then generates sample units for training and inference. Each sample unit contains at least environmental observations, body state, task objective, object instance, interaction area, candidate interaction points, candidate actions, execution result, deviation label, and recovery label, supporting visual feasibility assessment, intervention necessity assessment, embodied accessibility assessment, and behavior ranking training.

[0053] During label generation and quality review, the system labels object boundaries, accessible areas, occlusion relationships, risk areas, successful action states, failed action states, recovery path types, and the effectiveness of detection actions. For offline-collected teaching data, the teaching terminal records the operation sequence, contact positions, and target state changes; for online running data, the execution log automatically generates labels for the state difference before and after the action, collision events, emergency stop events, joint over-limit events, and recovery events. To ensure the quality of training samples, the system filters out or corrects label conflict samples, temporally misaligned samples, sensor frame missing samples, and control failure samples, and manages them hierarchically according to task category, environment type, and ontology platform.

[0054] The core algorithm of this invention comprises five collaborative computational units. First, the visual analysis unit performs object detection, instance segmentation, surface normal estimation, occlusion relationship recognition, and dynamic change recognition on environmental images, depth maps, and point clouds, outputting object sets, interaction region sets, and risk region sets. Second, the counterfactual prediction and value assessment unit predicts the future task state under two conditions: executing candidate actions and not executing candidate actions, and compares the impact of the two states on task completion to determine the necessity of intervention for the current action. Third, the embodied accessibility and risk assessment unit jointly assesses the kinematic accessibility, joint burden, energy consumption burden, stability margin, line-of-sight margin, and collision and instability risks during execution of candidate actions. Fourth, the unified behavior decision field construction unit integrates the aforementioned results into a comprehensive decision value in the same priority space, and based on this, completes the reordering of candidate actions and the generation of the optimal behavior sequence. Fifth, the execution deviation analysis and incremental update unit updates relevant parameters in a targeted manner based on the difference between the predicted results before execution and the actual results after execution.

[0055] In terms of technical logic, this invention makes decisions in the following order: first verifying the sufficiency of evidence, then verifying the necessity of the action, then verifying the feasibility of the ontology, and finally generating a sequence of behaviors. Specifically, in each decision cycle, the system first generates a set of candidate interaction points, then generates a set of corresponding candidate actions, and then performs a comprehensive evaluation of all candidate actions. When the environmental uncertainty of a candidate action exceeds a threshold or its prediction variance exceeds a threshold, the system does not directly execute the action. Instead, it includes perspective adjustment actions, tentative contact actions, or short pause actions in the candidate set and reorders them. Only when the visual evidence is sufficient, the risk is controlled, and the embodied accessibility requirements are met does the system enter the formal operation phase. This technical logic ensures that the system will not blindly operate due to a single visual salience in complex environments.

[0056] In terms of actual training and deployment, the offline training phase employs a model construction method that separates training, validation, and test sets. The visual information acquisition module uses an RGB camera with a resolution of 1280x720 and a depth camera, with a sampling frequency set to 15 to 30 frames per second. The side length of the point cloud downsampling voxels is set to 1 to 2 centimeters. The maximum number of candidate objects is set to 32. 12 to 32 candidate interaction points are extracted for each object. The recalculation cycle of the unified behavior decision field is set to 50 to 100 milliseconds. The chassis control cycle is set to 10 to 20 milliseconds. The robotic arm control cycle is set to 5 to 10 milliseconds. When the predicted collision probability, joint over-limit probability, or contact force exceeds a preset threshold, the system immediately switches to detection, retraction, or emergency stop actions. In the online deployment phase, the software system is divided into perception threads, decision threads, control threads, and update threads. The threads exchange time-synchronized data through shared memory or a message bus.

[0057] In terms of overall application, this invention is not limited to a single robot scenario. In ward care scenarios, task instructions come from nursing terminals, and the system is responsible for delivery, retrieval, and obstacle avoidance collaboration. In warehousing scenarios, task instructions come from the warehouse management system, and the system is responsible for container extraction, transfer, and stacking adjustment. In industrial collaboration scenarios, task instructions come from the manufacturing execution system, and the system is responsible for material loading, handling, workstation switching, and risk isolation. In inspection scenarios, task instructions come from the inspection planning system, and the system is responsible for close observation, on / off operation, and anomaly verification. Regardless of the application object, this invention always completes behavior generation and strategy optimization through the same data flow, the same unified behavior decision field construction logic, and the same execution deviation-oriented update logic, thus achieving the level of practical application.

[0058] Specific application example two: In a scenario where a ward companion robot is tasked with delivering a water cup and assisting with medication retrieval, the system first collects visual and depth data of the bedside table, water cup, patient's hand, IV stand, bed rails, and other bedside clutter. It then extracts the water cup contact area, the patient's receiving area, the IV tube interference area, and the bed rail restriction area. The visual feasibility assessment module determines that the cup handle area is unsuitable as an initial grasping point due to partial occlusion and missing boundaries. While the upper part of the cup is not the optimal final holding area, it offers higher visual integrity and embodied accessibility from the current perspective. The intervention necessity assessment module further determines that continuing to extend the hand forward is not the most advantageous action before the patient's hand has formed a stable receiving posture; a brief pause and adjustment of the perspective can significantly improve the success rate of subsequent delivery actions.

[0059] Based on this, the system generates a sequence of actions: "grasp the upper part of the cup, adjust the end effector posture, confirm the patient's receiving posture, execute delivery, monitor contact status, and complete release." If the patient's hand posture changes during delivery, the system recalculates the unified action decision field and inserts a viewpoint adjustment action or a short pause action into the current action sequence. During deployment testing, the update cycle of the chassis and robotic arm was set to 100 milliseconds, the upper limit of the end effector gripping force was set to 20 Newtons, the hovering distance of the delivery end effector was set to 5 to 8 centimeters, and the patient's hand stability detection window was set to 0.6 seconds. According to the above process, the system can stably complete the cup retrieval and delivery tasks in scenarios with partial occlusion, high dynamic interference, and high safety constraints.

[0060] Specific application example three: In scenarios where warehouse mobile robots perform target box extraction and transfer tasks, the system collects visual and depth data of the target box, adjacent boxes, shelf uprights, shelf panels, and aisle boundaries within the shelf area. It also extracts the accessible edges of the target box, the areas obscured by adjacent boxes, the traversable area of ​​the chassis, and the operable space of the robotic arm. The unified behavior decision field first determines that although the target box is visually visible, the front extraction path is limited by adjacent boxes. Subsequently, the intervention necessity assessment module determines that prioritizing the removal of obscured boxes or first horizontally moving the chassis to reconstruct the aisle is more beneficial for task progress than directly extracting the target box.

[0061] If the target container boundary exhibits high uncertainty due to reflection or pressure, the system prioritizes angle adjustment or low-intensity probing contact to re-estimate the true boundary and stress state. The system ultimately generates a sequence of actions: "chassis lateral movement, angle adjustment, probing contact, removal of obstructing containers, extension of the robotic arm to extract the target container, transfer to the conveyor belt, placement and release." During deployment, the chassis lateral movement step is set to 3-10 cm, the probing contact depth to 2-4 mm, the maximum probing contact force to 5 Newtons, and the robotic arm extraction speed to 5-15 cm / s. Based on this implementation, the system can improve target extraction success rate and reduce jamming risk under conditions of reflective packaging, dense stacking, and dynamic channel changes.

[0062] It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising a reference structure" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0063] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating embodied intelligent behavior and optimizing strategies based on visual perception, characterized in that, The process includes the following steps: collecting environmental visual data, depth or geometric data, body state data, and task target data of the embodied intelligent agent at the current moment; extracting object sets, interaction surface sets, occlusion relationship sets, and risk distribution sets based on the environmental visual data and depth or geometric data; constructing a unified behavioral decision field based on the object sets, interaction surface sets, occlusion relationship sets, risk distribution sets, body state data, and task target data, wherein the unified behavioral decision field simultaneously characterizes the visual feasibility, intervention necessity, target consistency, environmental risk, environmental uncertainty, and embodied accessibility of candidate actions; generating a comprehensive priority corresponding to the candidate action set based on the unified behavioral decision field; when the candidate action with the highest comprehensive priority belongs to a cognitive detection action, executing the cognitive detection action and updating the environmental visual data, depth or geometric data, and body state data; when the candidate action with the highest comprehensive priority belongs to a physical operation action, executing the physical operation action and obtaining the execution result; calculating the execution deviation based on the execution result and determining the source of the execution deviation; and updating the visual feasibility assessment parameters, intervention necessity assessment parameters, embodied accessibility assessment parameters, and action generation parameters in a targeted manner according to the source of the execution deviation.

2. The method for generating embodied intelligent behavior and optimizing strategies based on visual perception as described in claim 1, characterized in that, The visual feasibility is determined by the following method: establishing an object attribute vector for each candidate object or candidate interaction area; establishing an action semantic encoding vector for each candidate action; and calculating the matching value between the object attribute vector and the action semantic encoding vector. By combining the distance term from the candidate interaction location to the representative interaction center of the object and the deviation term between the current prediction time window and the optimal interaction time window of the target object, the visual feasibility evaluation results corresponding to the candidate interaction location, candidate action and prediction time window are obtained.

3. The method for generating embodied intelligent behavior and optimizing strategies based on visual perception as described in claim 1, characterized in that, The necessity of intervention is determined by: predicting the future task state after performing the candidate action on the target object and the future task state when the candidate action is not performed on the target object; and calculating the task value corresponding to the two future task states respectively. The necessity of the candidate action for advancing the current task is determined based on the difference between the two types of task values.

4. The method for generating embodied intelligent behavior and optimizing strategies based on visual perception as described in claim 1, characterized in that, The candidate action set consists of grabbing actions, pushing actions, placing actions, delivering actions, circling actions, viewpoint adjustment actions, probing contact actions, and short pause actions. The system reorders the candidate action set based on the degree of occlusion, the degree of action ambiguity, and the prediction variance. When the degree of occlusion, the degree of action ambiguity, or the prediction variance exceeds the corresponding threshold, the overall priority of viewpoint adjustment actions, probing contact actions, and short pause actions is increased.

5. The method for generating embodied intelligent behavior and optimizing strategies based on visual perception as described in claim 1, characterized in that, The physical accessibility is determined by kinematic inaccessibility, joint load, energy load, body stability margin, and line-of-sight maintenance margin; the line-of-sight maintenance margin is used to characterize the degree to which the target object is within the sensor's effective field of view after the candidate action is performed.

6. The method for generating embodied intelligent behavior and optimizing strategies based on visual perception as described in claim 1, characterized in that, When generating the optimal behavior sequence based on the unified behavior decision field, both action switching penalty and irreversible penalty are introduced. The irreversible penalty is determined by the relationship between the recovery cost of restoring the next state to a state where the task can continue to be executed safely and the preset recovery budget. The optimal behavior sequence is generated according to the principle of high overall benefit, few action switching and low recovery cost.

7. The method for generating embodied intelligent behavior and optimizing strategies based on visual perception as described in claim 1, characterized in that, The execution bias is broken down into visual understanding bias, intervention necessity judgment bias, embodied accessibility judgment bias, and action execution bias. When the visual understanding bias accounts for the highest proportion, the visual feasibility assessment parameters are updated. When the intervention necessity judgment bias accounts for the highest proportion, the counterfactual prediction parameters and action ranking parameters are updated. When the embodied accessibility judgment bias accounts for the highest proportion, the embodied constraint mapping parameters are updated. When the action execution bias accounts for the highest proportion, the action decoding parameters and control parameters are updated.

8. The system corresponding to the visual perception-driven embodied intelligent behavior generation and strategy optimization method according to any one of claims 1-7, characterized in that, include: The visual information acquisition module is used to acquire environmental visual data, depth, or geometric data. The scene analysis module is used to generate a set of objects, a set of interactive surfaces, a set of occlusion relationships, and a set of risk distributions. The body state acquisition module is used to collect chassis state, robotic arm state, end effector state, and energy state; the unified behavior decision field construction module is used to generate a unified behavior decision field based on object set, interaction surface set, occlusion relationship set, risk distribution set, body state data, and task target data; the optimal behavior generation module is used to generate a comprehensive priority of candidate action set based on the unified behavior decision field and output the optimal behavior sequence. The motion control and execution module is used to execute the optimal sequence of actions; The execution deviation analysis module is used to determine the source of execution deviation based on the execution results; The targeted optimization update module is used to update the relevant parameters in the unified behavior decision field construction module and the optimal behavior generation module according to the source of the execution deviation.

9. The embodied intelligent behavior generation and strategy optimization system based on visual perception-driven according to claim 8, characterized in that, The system also includes an experience distillation module; the experience distillation module is used to compress the successful behavior sequences, failure recovery paths and corresponding execution deviations obtained during multiple task executions into a strategy prototype library, and to call the initial behavior sequence according to the similarity between the current scene interaction state and the strategy prototype library when a new task arrives.

Images