Robot object operation method based on cooperation of multi-action network and decision network

Abstract

The invention discloses a robot object operation method based on cooperation of a multi-action network and a decision network, and the method comprises the steps: constructing a multi-vision priori database based on operability prompt and motion flow, and the multi-vision priori database comprises mechanical arm operation track data and annotation information; establishing a unified visual perception module based on stream matching, and combining the learning operability prompt and the prediction ability of the motion stream; a hierarchical control strategy combining a multi-action expert module and a decision-making expert module is constructed, the action expert module is responsible for specific skills, and the decision-making expert module dynamically selects the expert module; hardware control and a sensing-action closed loop are realized based on a Franka mechanical arm and a Realsense camera. According to the method, skill decoupling and multiplexing are realized, the operation robustness, generalization and training efficiency are improved, and the method is suitable for robot object operation in a complex dynamic environment.

Description

Technical Field

[0001] This invention belongs to the technical field of artificial intelligence and robotics, specifically relating to a robot object manipulation method based on the collaboration of multi-action networks and decision networks. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, most mainstream robotic object manipulation methods are currently based on a "single action model" strategy. This involves inputting visual perception data into a motion decision model to generate the robotic arm's movements end-to-end, thereby controlling the robotic arm to complete the corresponding object manipulation. However, existing methods suffer from the following two key technical bottlenecks:

[0003] Object manipulation tasks typically involve multiple stages; for example, manipulating a drawer requires grasping the handle before pulling it open. These tasks demand that robots possess various skills such as grasping, placing, pushing, rotating, and positioning. However, existing methods often rely on a single model to achieve end-to-end mapping of perceived actions, leading to blurred functional boundaries between skills, difficulty in distinguishing semantic features of operations, and a tendency to cause policy confusion, thus limiting the flexible combination and generalization capabilities of skills.

[0004] Existing techniques do not fully consider the differentiated role of various visual prior information at different stages of object manipulation, resulting in models lacking sensitivity to task context and prior guidance capabilities. This not only weakens the model's ability to identify semantic features at each stage but also leads to low training efficiency and difficulty in achieving robust decision-making and interpretable behavior in multi-stage tasks.

[0005] In summary, existing robot object manipulation methods suffer from significant shortcomings in terms of robustness, generalization, and training efficiency, making it difficult to meet the practical needs of generalization to different objects in complex dynamic environments, adaptability to different placements, and rapid deployment. Therefore, there is an urgent need for a novel object manipulation framework that can effectively integrate multi-source visual priors, decouple action strategies and decision-making processes, and support modular learning and adaptive selection, in order to improve the intelligence and practicality of robots in real-world scenarios. Summary of the Invention

[0006] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a robot object manipulation method based on the collaboration of multi-action networks and decision networks. By introducing a modular architecture, complex object manipulation tasks are decomposed into multiple reusable action skill modules. At the same time, a multi-visual feature perception module is introduced and combined with a decision module with context awareness capabilities, thus realizing a unified and scalable robot object operating system.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a robot object manipulation method based on the collaboration of a multi-action network and a decision network, comprising the following steps:

[0009] A multi-vision prior database based on operability cues and motion flow is constructed; the multi-vision prior database includes robotic arm operation trajectory data and annotation information;

[0010] A unified visual perception module based on flow matching is constructed to jointly learn the ability to provide actionable cues and predict motion flow from input RGB-D images and point cloud data. The unified visual perception module constructs a continuous conditional vector field to gradually transform random noise into a target distribution, thereby achieving refined modeling of the functional areas and dynamic behaviors of object surfaces.

[0011] A hierarchical control strategy combining multiple action expert modules and decision expert modules is constructed. The overall operational capability is decoupled into multiple functionally independent action expert modules and a high-level decision expert module. Each action expert module is responsible for a set type of operational skill, integrates an action generation network optimized for that set operational skill, receives operability prompts and motion flow information as input, and outputs local action instructions. The decision expert module receives the current task objective, environmental observation, and task stage information, dynamically judges the current operation stage, and adaptively selects the most suitable action expert module.

[0012] A robotic arm is used as the actuator, and an RGB-D camera is used as the scene perception device. At the underlying control level, the robotic arm runs closed-loop control through an open-source interface, and Kalman filtering is used to denoise the pose estimation. In the perception-action closed loop, the RGB-D data collected by the RGB-D camera is input into the unified visual perception module to generate an operability prompt map and a 3D motion flow field. The decision expert module selects the optimal action expert module outputting the target end velocity or pose increment, which is then converted into a reference path that the robotic arm can execute after being processed by the trajectory generation module and then handed over to the underlying controller for execution.

[0013] As a preferred technical solution, the annotation of the operability prompt is achieved by automatically calculating the distance field based on the geometric structure of the object's functional components in the simulation environment to generate a pixel-level affordance map. The annotation of the motion flow is achieved by combining the kinematic model and the robot arm trajectory, deriving the three-dimensional motion vector through forward kinematic simulation and normalizing it.

[0014] As a preferred technical solution, the annotation information includes RGB-D images and point clouds from the third-person perspective and the robotic arm wrist perspective, the robotic arm operation trajectory, operability prompts for the functional semantics of each region on the object surface marked point by point, motion flow annotations that calculate and normalize the three-dimensional instantaneous motion direction and velocity of the object components, and task stage division annotations.

[0015] As a preferred technical solution, the unified visual perception module takes RGB-D images and point clouds as input, combines task language instructions as conditional guidance, and predicts the normalized motion direction and operability map of each spatial point through a neural network to form a unified multi-visual prior representation.

[0016] During the training phase, supervised learning is performed using finely labeled affordance maps and motion flow fields from the database. A flow matching loss function is used to optimize the smoothness and physical consistency of the vector field, thereby achieving joint learning of affordance and motion flow. During the inference phase, dense, operable heatmaps and three-dimensional motion flow fields are generated in real time.

[0017] As a preferred technical solution, the training and inference process of the hierarchical control strategy is as follows:

[0018] (1) During the training phase, based on the operation trajectory and task phase annotation in the database, a phase-aware adaptive loss function is introduced to conduct differentiated training for each action expert module. The weight of the loss function is dynamically adjusted according to the current phase, while maintaining the action learning of each action expert module in non-key task phases.

[0019] (2) The decision expert module is trained end-to-end through supervised learning, with the current observation state as the input. and user instructions The output is the gating probability distribution of each action expert module. :

[0020] ;

[0021] in Let N represent the decision network, where N is the number of action expert modules. The training objective is to minimize the cross-entropy loss between the predicted choice and the given task stage label, ensuring that the decision logic is consistent with the actual operation sequence.

[0022] (3) During the reasoning process, a hard selection mechanism is used to select the action expert module with the highest probability, satisfying the following conditions: Only activate the first one. The motion expert module generates control commands.

[0023] As a preferred technical solution, the step of dynamically adjusting the weights of the loss function according to the current stage specifically includes:

[0024] The task phase is divided into "approach phase" and "operation phase". When in the "approach phase", the weight of the loss function coefficient of the training grasping expert module is increased; when entering the "operation phase", the weight of the loss function coefficient of the pushing and rotating expert modules is increased.

[0025] As a preferred technical solution, the robotic arm is the Franka Emika Panda robotic arm, which has 7 degrees of freedom high-precision joint control capability and torque feedback function, and the RGB-D camera is the Realsense L515 RGB-D camera.

[0026] Secondly, the present invention provides a robot object operating system based on the collaboration of multi-action network and decision network, which is applied to the robot object operation method based on the collaboration of multi-action network and decision network, including a database construction module, a joint learning module, a control strategy construction module and an execution module;

[0027] The database construction module is used to construct a multi-vision prior database based on operability cues and motion flow; the multi-vision prior database includes robotic arm operation trajectory data and annotation information;

[0028] The joint learning module is used to construct a unified visual perception module based on flow matching, which jointly learns the ability to provide operability cues and predict motion flow from the input RGB-D images and point cloud data. The unified visual perception module constructs a continuous conditional vector field to gradually transform random noise into a target distribution, thereby achieving refined modeling of the functional areas and dynamic behaviors of the object surface.

[0029] The control strategy construction module is used to construct a hierarchical control strategy that combines multiple action expert modules and decision expert modules. It decouples the overall operational capability into multiple functionally independent action expert modules and a high-level decision expert module. Each action expert module is responsible for a set type of operational skill, integrates an action generation network optimized for that set operational skill, receives operability prompts and motion flow information as input, and outputs local action instructions. The decision expert module receives the current task objective, environmental observation, and task stage information, dynamically judges the current operation stage, and adaptively selects the most suitable action expert module.

[0030] The execution module uses a robotic arm as the actuator and an RGB-D camera as the scene perception device. At the underlying control level, the robotic arm runs closed-loop control through an open-source interface, and Kalman filtering is used to denoise the pose estimation. In the perception-action closed loop, the RGB-D data collected by the RGB-D camera is input into the unified visual perception module to generate an operability prompt map and a three-dimensional motion flow field. The decision expert module selects the optimal action expert module outputting the target end velocity or pose increment, which is then converted into a reference path executable by the robotic arm by the trajectory generation module and handed over to the underlying controller for execution.

[0031] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0032] At least one processor; and,

[0033] A memory communicatively connected to the at least one processor; wherein,

[0034] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to perform the robot object manipulation method based on the collaboration of multi-action networks and decision networks.

[0035] Fourthly, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the robot object manipulation method based on the collaboration of a multi-action network and a decision network.

[0036] The robot object manipulation method proposed in this invention, based on the collaboration of multi-action networks and decision networks, enhances the features of different motion experts at different stages by combining the different characteristics of affordance and motion flow. Through the core technical approach of a "control strategy combining multi-action expert modules and decision experts," it effectively solves key technical problems in existing methods such as strong skill coupling, semantic feature ambiguity, strategy confusion, and low training efficiency, bringing significant technical progress and practical benefits. Specific technical effects are as follows:

[0037] (1) This invention achieves modularization and specialization of skills by decoupling the overall operational capability into multiple functionally independent motion expert modules (such as grasping, pushing, rotating, etc.). Each motion expert module focuses on a specific operation type and generates high-precision movements under the guidance of a unified visual prior (operability cues and motion flow), significantly improving the execution accuracy and robustness of each skill. Finally, the decision expert module rationally schedules each motion expert module, achieving complete robustness and automated operation.

[0038] (2) This invention significantly enhances the reusability of motion skills by constructing a modular multi-action expert architecture. Each motion expert module (such as grasping, pushing, rotating, etc.) independently encapsulates specific operational capabilities and is trained under unified operability prompts and motion flow prior guidance, enabling it to generalize across objects and scenes. When facing new tasks or new environments, there is no need to retrain the entire system; only existing expert modules need to be reused and a few experts added or fine-tuned as needed to complete the functional expansion, greatly reducing the model migration and deployment costs. At the same time, each motion expert can be trained in parallel based on visual priors with rich feature information and an adaptive loss function with stage awareness, avoiding gradient conflicts and training instability caused by multi-task coupling in traditional end-to-end models, and significantly improving training efficiency.

[0039] (3) This invention achieves semi-automatic construction of affordance and motion flow through a "simulation generation + physical model-driven annotation" approach. In the simulation environment, the distance field is automatically calculated based on the geometric structure of the object's functional components to generate a pixel-level affordance map; combined with the kinematic model and the robotic arm trajectory, the three-dimensional motion vector is derived and normalized through forward kinematic simulation to generate a physically reasonable motion flow field. This method eliminates the need for large-scale manual annotation, achieving large-scale, high-precision, and cross-category consistent automatic generation of visual priors. This approach significantly reduces data construction costs, provides high-quality supervision signals for model pre-training, and effectively improves the generalization ability of the visual perception module and the performance of downstream tasks.

[0040] (4) This invention selectively introduces corresponding visual prior information by matching the functional requirements of different action experts, thereby achieving "perception-action" alignment and enhancing the sensitivity and understanding of key operational features of each expert. For example, grasping experts mainly rely on operability cue maps, while rotation or pushing experts pay more attention to the direction and structure of the motion flow field. This targeted prior input mechanism enables each expert to focus on the perceptual features most relevant to their task, significantly improving their sensitivity and understanding of key operational areas. This design effectively reduces information redundancy and interference in the model learning process, alleviates gradient conflicts in multi-task training, thereby accelerating convergence speed and improving training efficiency and stability. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of a robot object manipulation method based on the collaboration of a multi-action network and a decision network, according to an embodiment of the present invention.

[0043] Figure 2 This is a system framework diagram of the robot object manipulation method based on the collaboration of multi-action network and decision network according to an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0045] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0046] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0047] like Figure 1 As shown, this embodiment provides a robot object manipulation method based on the collaboration of multi-action networks and decision networks. By introducing a modular architecture, complex object manipulation tasks are decomposed into multiple reusable action skill modules. Simultaneously, a multi-visual feature perception module is introduced, combined with a decision module possessing context awareness capabilities, to achieve a unified and scalable robot object operating system. This invention achieves precise and robust control over object manipulation tasks, improving model training efficiency. The specific implementation process and technical solution are described in detail below:

[0048] S1. Construct a multi-vision prior database based on affordance and motion flow; the multi-vision prior database includes robotic arm operation trajectory data and annotation information.

[0049] This embodiment first addresses the need to reduce model training time and improve operational generalization by designing and constructing a multi-vision prior-guided robot operation database that integrates affordance cues and motion flow. Its core lies in proposing an efficient data construction paradigm of "pre-training with a large amount of simulation data + fine-tuning with a small amount of real data." This method utilizes the annotation information of the GAPartNet dataset to achieve patterned and automated generation of robotic arm operation tasks and synthesis of training data on the Isaac-Gym simulation platform. Specifically, GAPartNet provides rich geometric perception and semantic segmentation annotations, including the graspable parts of objects, functional parts, and the topological relationships between parts. This information is used to automatically identify key interaction points of target objects in the simulation environment and generate diverse operation trajectories that conform to physical laws. Combined with the efficient parallel simulation capabilities of Isaac-Gym, the system can generate large amounts of demonstration data with accurate state annotations and action labels in batches for pre-training general strategies with object perception capabilities. Through large-scale pre-training, the model possesses sufficient knowledge, thus allowing it to operate in a real environment using only a small amount of real-world data. This approach fully leverages the richness and ease of annotation of data in the simulation environment to automatically generate large-scale data and perform large-scale pre-training. Then, it fine-tunes the domain transfer using a small amount of real data, significantly reducing data acquisition costs and improving the model's generalization ability and deployment efficiency in real-world scenarios. This provides high-quality, scalable data support for complex object manipulation tasks.

[0050] Furthermore, each robotic arm movement trajectory in the database contains detailed data and annotation information, including:

[0051] (1) Complete RGB-D images and point clouds of the third-person perspective and the wrist perspective of the robotic arm, as well as the robotic arm operation trajectory;

[0052] (2) For each scene and object, fine-grained affordance annotation is used to annotate the functional semantics of each area on the object surface in a point-by-point manner, and to accurately indicate the key action areas of operations such as grasping, pushing, and rotating.

[0053] (3) For each scene and object, the fine motion flow annotation is used to calculate the three-dimensional instantaneous motion direction and velocity of the object parts during the operation process, and the normalization process is used to ensure cross-category consistency.

[0054] (4) Detailed task phase division and labeling.

[0055] The detailed robot operation trajectory data and refined annotations mentioned above provide precise guidance for the robotic arm's object manipulation process.

[0056] Understandably, this embodiment constructs a multimodal prior database for robot operation tasks, the core of which lies in the simultaneous annotation of object maneuverability cues (affordance) and 3D motion flow. This database is automatically generated through a combination of large-scale simulation pre-training and fine-tuning with a small amount of real data, and includes RGB-D images, point clouds, robotic arm trajectories, point-by-point affordance labels, normalized motion flow vector fields, and task stage division information. Affordance is automatically calculated from the geometric structure of the object's functional components, while motion flow is derived through forward kinematics simulation. This database provides structured and physically consistent supervision signals for model training, significantly improving generalization capabilities across objects and scenes.

[0057] S2. Construct a unified visual perception module based on flow matching to jointly learn the predictive capabilities of affordance and motion flow from the input RGB images and point cloud data. The unified visual perception module constructs a continuous conditional vector field to gradually transform random noise into a target distribution, thereby achieving refined modeling of the functional areas and dynamic behaviors of the object's surface.

[0058] Furthermore, the unified visual perception module (AFG network) takes RGB-D images and point clouds as input, and combines task language instructions as conditional guidance. It predicts the normalized motion flow and affordance map of each spatial point through a neural network, forming a unified multi-visual prior representation. The unified visual perception module uses task language instructions as conditional input to construct a continuous conditional vector field. Through a gradual transformation from noise to target distribution, it achieves joint modeling of object functional semantics and dynamic behavior. The adopted flow matching loss function ensures the smoothness and physical rationality of the output field, generating dense, structured multi-visual prior information. This provides high-quality guidance signals for downstream action experts, effectively improving the response accuracy and robustness of the perception-action closed loop. The flow matching loss function is as follows:

[0059] ;

[0060] ;

[0061] in This represents the Unified Visual Perception Module (AFG) network. and This represents the affordance and motion flow after adding noise. and This represents the current affordance and motion flow truth values, where t∈[0, 1] represents the noisy time step. This represents images and point clouds obtained from observations.

[0062] Furthermore, during the training phase, the unified visual perception module utilizes finely labeled affordance maps and motion flow fields from the database for supervised learning, employing a flow matching loss function to optimize the smoothness and physical consistency of the vector field. During the inference phase, the model can generate dense maneuverability heatmaps and 3D motion flow fields in real time, providing structured visual prior support for downstream decision-making and action modules. This method achieves unified modeling of various visual prior information, enhancing the model's understanding of object functional semantics and dynamic behavior, and improving the robot's robustness and generalization performance in complex scenarios.

[0063] S3, a control strategy combining multiple action expert modules and decision experts, is used to achieve efficient and robust execution of complex object manipulation tasks. This control strategy adopts a modular architecture, decoupling the overall operational capability into multiple functionally independent action expert modules and a high-level decision expert module. Each action expert module is specifically responsible for a particular type of operation skill, such as grasping, pushing, pulling, rotating, or placing. It integrates an optimized motion generation network for that skill and accepts multi-visual prior information, such as operability cues and motion flow, as input, outputting high-precision local action commands. The decision expert module acts as a scheduling hub, receiving information about the current task objective, environmental observations, and task stage. By analyzing visual priors and contextual states, it dynamically determines the current operation stage (approach stage or operation stage) and adaptively selects the most suitable action expert module for invocation.

[0064] Furthermore, the training and inference processes of the control strategy include:

[0065] S31. During the training phase, based on the complete operation trajectories in the database and their detailed task stage annotations (such as "approach stage" and "operation stage"), a stage-aware adaptive loss function is introduced to perform differentiated training on each motion expert module. In the early stages of training, all motion experts learn the actions of each stage equally, aiming to ensure each expert learns basic motion knowledge. Once the training steps reach a threshold, each motion expert focuses on learning specific motion knowledge to enhance their expertise. Specifically, for each motion expert, the weight of their corresponding loss function is dynamically adjusted according to the current stage to strengthen their learning intensity in critical stages. For example, when in the "approach stage," the weight of the loss function coefficient for grasping experts is increased; when entering the "operation stage," the weight of the loss function coefficient for pushing, rotating, and other experts is increased. Simultaneously, the motion learning of each motion expert model is maintained in non-critical task stages to address situations where decision-making experts make mistakes, improving the overall robustness of the model.

[0066] S32. The decision expert module is trained end-to-end using supervised learning, and its input is the current observation state. and user instructions The output is the gating probability distribution of each motion expert. :

[0067] ;

[0068] in Let N represent the decision network, where N is the number of experts. The training objective is to minimize the cross-entropy loss between the predicted choice and the labeling of a given task stage, ensuring that the decision logic is consistent with the actual operation sequence.

[0069] S33. During the reasoning process, the system integrates environmental observations and user commands for real-time prediction: First, the unified visual perception module generates an operability map A and a motion flow field F; the decision expert module outputs the activation probabilities P of each expert; finally, a hard selection mechanism is used to select the action expert module with the highest probability.

[0070] ;

[0071] Only activate the first one The motion expert module generates control commands to avoid motion overlap, improving the determinism and stability of execution. This strategy, through functional decoupling, stage-aware decision-making, and hard-selection mechanisms, achieves skill reusability, decision interpretability, and control robustness, significantly improving the robot's generalization ability and practical deployment efficiency in complex dynamic environments.

[0072] Furthermore, the control strategy in this embodiment decomposes complex operational tasks into multiple functionally independent "action expert" modules (such as grasping, pushing, pulling, rotating, and placing) and a unified "decision expert" module. This strategy separates action generation from task decision-making, achieving high reusability of skill modules and interpretability of control logic. The decision expert dynamically selects the most suitable action expert to execute the operation based on the current environmental state and task objective, avoiding gradient conflicts and behavior aliasing problems in multi-task learning common in traditional single-policy networks, significantly improving the system's task adaptability and execution stability.

[0073] It is understood that the multiple action expert modules in this embodiment constitute a prior-driven action expert enhancement mechanism. Based on the dependence of different action types on visual information, it achieves directional guidance and dynamic allocation of multiple visual priors. For example, the grasping expert primarily relies on affordance maps to identify graspable areas, while the pushing / rotating expert focuses on using motionflow fields to predict the object's motion direction and trajectory. This mechanism, through perception-action alignment design, allows each expert module to focus on the semantic features most relevant to its skill, effectively suppressing interference from irrelevant information, improving the accuracy and robustness of action generation, while simultaneously reducing model training difficulty and accelerating convergence speed.

[0074] S4. Hardware configuration and control implementation of the robot object operating system;

[0075] The robot object operating system hardware configuration in this embodiment includes the Franka Emika Panda robotic arm, which uses its built-in parallel gripper for grasping, and a Realsense L515 RGB-D camera for scene perception. The Franka robotic arm, as the main actuator, possesses 7-DOF high-precision joint control and torque feedback capabilities, supporting flexible end-effector pose adjustment and safe human-machine interaction. The Realsense L515 RGB-D camera is mounted on the upper side of the robotic arm or at a fixed viewing angle to acquire RGB images and high-resolution depth information of the operating scene in real time, generating dense point cloud data to provide an input source for the unified visual perception module.

[0076] The system control employs a hierarchical collaborative strategy: at the lowest control level, the Franka robotic arm runs closed-loop control through the open-source franka_ros interface, with end-effector motion driven by the target pose output by our proposed multi-expert decision-making model strategy. Simultaneously, Kalman filtering is used to denoise the pose estimation, improving motion smoothness and response accuracy. The affordance and flow priors output by the perception module are input to the decision expert and motion expert modules, forming a closed-loop control chain of "perception → decision → action → execution," ensuring high stability and strong adaptability of the system in real-world environments.

[0077] In the perception-action closed loop, RGB-D data collected by Realsense L515 is input in real time to the unified vision perception module based on flow matching, which infers online to generate an operability prompt map and a three-dimensional motion flow field; the decision expert module combines the task instructions and the current stage information to select the optimal action expert and output the target end velocity or pose increment; the instruction is converted into a reference path that the robotic arm can execute through the trajectory generation module and handed over to the underlying controller to complete the physical execution.

[0078] In summary, this invention, by combining the high-precision execution capability of the Franka robotic arm with the high-quality multimodal perception fusion of the Realsense L515, and integrating smooth trajectory planning, filtering optimization, and closed-loop visual guidance mechanisms, constructs a robot object operating system that is accurate in perception, stable in control, efficient in response, and combines multi-motion experts and decision experts. This significantly improves the robustness, adaptability, and deployment practicality of the robot in complex operational tasks.

[0079] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0080] Based on the same idea as the robot object manipulation method based on the collaboration of multi-action networks and decision networks in the above embodiments, the present invention also provides a robot object operating system based on the collaboration of multi-action networks and decision networks. This system can be used to execute the above-described robot object manipulation method based on the collaboration of multi-action networks and decision networks. For ease of explanation, the structural diagram of the embodiment of the robot object operating system based on the collaboration of multi-action networks and decision networks only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0081] Please see Figure 2 In another embodiment of this application, a robot object operating system 100 based on the collaboration of a multi-action network and a decision network is provided. The system includes a database construction module 101, a joint learning module 102, a control strategy construction module 103, and an execution module 104.

[0082] The database construction module 101 is used to construct a multi-vision prior database based on operability cues and motion flow; the multi-vision prior database includes robotic arm operation trajectory data and annotation information.

[0083] The joint learning module 102 is used to construct a unified visual perception module based on flow matching, which jointly learns the ability to provide operability cues and predict motion flow from the input RGB-D image and point cloud data. The unified visual perception module constructs a continuous conditional vector field to gradually transform random noise into a target distribution, thereby achieving refined modeling of the functional areas and dynamic behaviors of the object surface.

[0084] The control strategy construction module 103 is used to construct a hierarchical control strategy that combines multiple action expert modules and decision expert modules. It decouples the overall operation capability into multiple functionally independent action expert modules and a high-level decision expert module. Each action expert module is responsible for a set operation skill, integrates an action generation network optimized for that set operation skill, receives operability prompts and motion flow information as input, and outputs local action instructions. The decision expert module receives the current task objective, environmental observation, and task stage information, dynamically judges the current operation stage, and adaptively selects the most suitable action expert module.

[0085] The execution module 104 is used to employ a robotic arm as the execution mechanism and an RGB-D camera as the scene perception device. At the underlying control level, the robotic arm runs closed-loop control through an open-source interface, and combines Kalman filtering to denoise the pose estimation. In the perception-action closed loop, the RGB-D data collected by the RGB-D camera is input into the unified visual perception module to generate an operability prompt map and a three-dimensional motion flow field. The decision expert module selects the optimal action expert to output the target end velocity or pose increment, which is then converted into a reference path that the robotic arm can execute by the trajectory generation module and then handed over to the underlying controller for execution.

[0086] It should be noted that the robot object operating system based on the collaboration of multi-action network and decision network of the present invention corresponds one-to-one with the robot object operation method based on the collaboration of multi-action network and decision network of the present invention. The technical features and beneficial effects described in the embodiments of the robot object operation method based on the collaboration of multi-action network and decision network are applicable to the embodiments of robot object operation based on the collaboration of multi-action network and decision network. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0087] Furthermore, in the above embodiments of the robot object operating system based on the collaboration of multi-action networks and decision networks, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or for the convenience of software implementation. That is, the internal structure of the robot object operating system based on the collaboration of multi-action networks and decision networks is divided into different program modules to complete all or part of the functions described above.

[0088] Please see Figure 3 In one embodiment, an electronic device is provided for implementing a robot object manipulation method based on the collaboration of a multi-action network and a decision network. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a robot object manipulation program 203 based on the collaboration of a multi-action network and a decision network.

[0089] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal and external storage units of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of a robot object manipulation program 203 based on multi-action network and decision network collaboration, but also to temporarily store data that has been output or will be output.

[0090] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.

[0091] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0092] The robot object manipulation program 203 based on the collaboration of a multi-action network and a decision network, stored in the first memory 202 of the electronic device 200, is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0093] A multi-vision prior database based on operability cues and motion flow is constructed; the multi-vision prior database includes robotic arm operation trajectory data and annotation information;

[0094] A unified visual perception module based on flow matching is constructed to jointly learn the ability to provide actionable cues and predict motion flow from input RGB-D images and point cloud data. The unified visual perception module constructs a continuous conditional vector field to gradually transform random noise into a target distribution, thereby achieving refined modeling of the functional areas and dynamic behaviors of object surfaces.

[0095] A hierarchical control strategy combining multiple action expert modules and decision expert modules is constructed. The overall operational capability is decoupled into multiple functionally independent action expert modules and a high-level decision expert module. Each action expert module is responsible for a set type of operational skill, integrates an action generation network optimized for that set operational skill, receives operability prompts and motion flow information as input, and outputs local action instructions. The decision expert module receives the current task objective, environmental observation, and task stage information, dynamically judges the current operation stage, and adaptively selects the most suitable action expert module.

[0096] A robotic arm is used as the actuator, and an RGB-D camera is used as the scene perception device. At the underlying control level, the robotic arm runs closed-loop control through an open-source interface, and Kalman filtering is used to denoise the pose estimation. In the perception-action closed loop, the RGB-D data collected by the RGB-D camera is input into the unified visual perception module to generate an operability prompt map and a 3D motion flow field. The decision expert module selects the optimal action expert to output the target end velocity or pose increment, which is then converted into a reference path that the robotic arm can execute after being processed by the trajectory generation module and then handed over to the underlying controller for execution.

[0097] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0098] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0099] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A robot object manipulation method based on the collaboration of multi-action networks and decision networks, characterized in that, Includes the following steps: A multi-vision prior database based on operability cues and motion flow is constructed; the multi-vision prior database includes robotic arm operation trajectory data and annotation information; A unified visual perception module based on flow matching is constructed to jointly learn the ability to provide actionable cues and predict motion flow from input RGB-D images and point cloud data. The unified visual perception module constructs a continuous conditional vector field to gradually transform random noise into a target distribution, thereby achieving refined modeling of the functional areas and dynamic behaviors of object surfaces. A hierarchical control strategy combining multiple action expert modules and decision expert modules is constructed. The overall operational capability is decoupled into multiple functionally independent action expert modules and a high-level decision expert module. Each action expert module is responsible for a set type of operational skill, integrates an action generation network optimized for that set operational skill, receives operability prompts and motion flow information as input, and outputs local action instructions. The decision expert module receives the current task objective, environmental observation, and task stage information, dynamically judges the current operation stage, and adaptively selects the most suitable action expert module. A robotic arm is used as the actuator, and an RGB-D camera is used as the scene perception device. At the underlying control level, the robotic arm runs closed-loop control through an open-source interface, and Kalman filtering is used to denoise the pose estimation. In the perception-action closed loop, the RGB-D data collected by the RGB-D camera is input into the unified visual perception module to generate an operability prompt map and a 3D motion flow field. The decision expert module selects the optimal action expert module outputting the target end velocity or pose increment, which is then converted into a reference path that the robotic arm can execute after being processed by the trajectory generation module and then handed over to the underlying controller for execution.

2. The robot object manipulation method based on the collaboration of multi-action networks and decision networks according to claim 1, characterized in that, The annotation of the operability prompts is achieved by automatically calculating the distance field based on the geometric structure of the object's functional components in the simulation environment to generate a pixel-level affordance map. The annotation of the motion flow is achieved by combining the kinematic model and the robot arm trajectory, deriving the three-dimensional motion vector through forward kinematic simulation and normalizing it.

3. The robot object manipulation method based on the collaboration of multi-action networks and decision networks according to claim 1, characterized in that, The annotation information includes RGB-D images and point clouds from the third-person perspective and the robotic arm wrist perspective, the robotic arm's operating trajectory, operability prompts for the functional semantics of each region on the object's surface marked point by point, motion flow annotations that calculate and normalize the three-dimensional instantaneous motion direction and velocity of the object's components, and task phase division annotations.

4. The robot object manipulation method based on the collaboration of multi-action networks and decision networks according to claim 1, characterized in that, The unified visual perception module takes RGB-D images and point clouds as input, and combines task language instructions as conditional guidance. It predicts the normalized motion direction and operability map of each spatial point through a neural network, forming a unified multi-visual prior representation. During the training phase, supervised learning is performed using finely labeled affordance maps and motion flow fields from the database. A flow matching loss function is used to optimize the smoothness and physical consistency of the vector field, thereby achieving joint learning of affordance and motion flow. During the inference phase, dense, operable heatmaps and three-dimensional motion flow fields are generated in real time.

5. The robot object manipulation method based on the collaboration of multi-action networks and decision networks according to claim 1, characterized in that, The training and reasoning process of the hierarchical control strategy is as follows: (1) During the training phase, based on the operation trajectory and task phase annotation in the database, a phase-aware adaptive loss function is introduced to conduct differentiated training for each action expert module. The weight of the loss function is dynamically adjusted according to the current phase, while maintaining the action learning of each action expert module in non-key task phases. (2) The decision expert module is trained end-to-end through supervised learning, with the current observation state as the input. and user instructions The output is the gating probability distribution of each action expert module. : ； in Let N represent the decision network, where N is the number of action expert modules. The training objective is to minimize the cross-entropy loss between the predicted choice and the given task stage label, ensuring that the decision logic is consistent with the actual operation sequence. (3) During the reasoning process, a hard selection mechanism is used to select the action expert module with the highest probability, satisfying the following conditions: Only activate the first one. The motion expert module generates control commands.

6. The robot object manipulation method based on the collaboration of multi-action networks and decision networks according to claim 5, characterized in that, The specific steps for dynamically adjusting the loss function weights based on the current stage are as follows: The task phase is divided into "approach phase" and "operation phase". When in the "approach phase", the weight of the loss function coefficient of the training grasping expert module is increased; when entering the "operation phase", the weight of the loss function coefficient of the pushing and rotating expert modules is increased.

7. The robot object manipulation method based on the collaboration of multi-action networks and decision networks according to claim 1, characterized in that, The robotic arm is a Franka Emika Panda robotic arm, which has 7 degrees of freedom high-precision joint control and torque feedback function, and the RGB-D camera is a Realsense L515 RGB-D camera.

8. A robot object operating system based on the collaboration of multi-action networks and decision networks, characterized in that, The robot object manipulation method based on the collaboration of multi-action network and decision network, as described in any one of claims 1-7, includes a database construction module, a joint learning module, a control strategy construction module, and an execution module. The database construction module is used to construct a multi-vision prior database based on operability cues and motion flow; the multi-vision prior database includes robotic arm operation trajectory data and annotation information; The joint learning module is used to construct a unified visual perception module based on flow matching, which jointly learns the ability to provide operability cues and predict motion flow from the input RGB-D images and point cloud data. The unified visual perception module constructs a continuous conditional vector field to gradually transform random noise into a target distribution, thereby achieving refined modeling of the functional areas and dynamic behaviors of the object surface. The control strategy construction module is used to construct a hierarchical control strategy that combines multiple action expert modules and decision expert modules. It decouples the overall operational capability into multiple functionally independent action expert modules and a high-level decision expert module. Each action expert module is responsible for a set type of operational skill, integrates an action generation network optimized for that set operational skill, receives operability prompts and motion flow information as input, and outputs local action instructions. The decision expert module receives the current task objective, environmental observation, and task stage information, dynamically judges the current operation stage, and adaptively selects the most suitable action expert module. The execution module uses a robotic arm as the actuator and an RGB-D camera as the scene perception device. At the underlying control level, the robotic arm runs closed-loop control through an open-source interface, and Kalman filtering is used to denoise the pose estimation. In the perception-action closed loop, the RGB-D data collected by the RGB-D camera is input into the unified visual perception module to generate an operability prompt map and a three-dimensional motion flow field. The decision expert module selects the optimal action expert module outputting the target end velocity or pose increment, which is then converted into a reference path executable by the robotic arm by the trajectory generation module and handed over to the underlying controller for execution.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to perform the robot object manipulation method based on the collaboration of a multi-action network and a decision network as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the robot object manipulation method based on the collaboration of multi-action network and decision network as described in any one of claims 1-7.

Images