Automatic synthesis method and apparatus for availability data for dual-arm robot operation

Abstract

This invention proposes an automatic method and apparatus for synthesizing availability data for dual-arm robot operations. The method includes: identifying an initial frame and a contact frame from time-series data containing the operation process of the mechanical dual arms; the initial frame corresponds to the moment when the dual arms are not in contact with the target object; the contact frame corresponds to the starting moment when the dual arms make contact with the target object; obtaining a segmentation mask for the target object based on the initial frame; obtaining a segmentation mask for the end effector of the dual arms based on the contact frame; determining static contact availability information between the dual arms and the target object based on the target object segmentation mask and the end effector segmentation mask; generating dynamic motion availability information by tracking at least one feature point associated with the static contact availability information based on the contact frame and subsequent image sequences; and integrating the static contact and dynamic motion availability information to form an availability data annotation file. A corresponding apparatus is also proposed based on this method. This invention achieves large-scale, highly diverse dual-arm availability data synthesis.

Description

Technical Field

[0001] This invention belongs to the technical field of combining robotics and computer vision, and specifically relates to a method and system for automatic synthesis of availability data for dual-arm robot operation. Background Technology

[0002] With the development of embodied intelligence, robots are increasingly capable of learning complex operational tasks. Availability, the potential attributes of an object that support specific interactive actions (such as grasping, pushing, and pulling), has become a key intermediate representation connecting environmental perception and action generation. Learning high-quality availability representations is crucial for improving a robot's operational generalization ability and task understanding.

[0003] Existing affordance learning methods primarily focus on single-arm robot manipulation scenarios. For example, the VRB model learns visual affordance representations from large-scale human interaction videos to predict possible human interaction positions and methods in a scene, and uses this as a multifunctional intermediate representation for robot learning, supporting paradigms such as imitation learning, exploration, and reinforcement learning. The HRP model automatically mines affordance labels such as contact points, hand poses, and target objects from human videos to fine-tune pre-trained visual encoders, improving the generalization ability of downstream robot operations. The RT-Affordance method uses the end effector poses of key stages as affordance representations for conditional policy learning, improving the inference performance of visual language action models in manipulation tasks. The UAD framework utilizes unsupervised distillation of object affordance knowledge from visual language models into task conditional models to achieve generalization to novel objects and instructions. While these methods have made significant progress in single-arm manipulation scenarios, they mainly rely on human single-hand or single-arm demonstration videos, neglecting the complex coordination mechanisms of two-arm collaboration. In bi-arm egomania scenarios, bi-arm collaborative tasks (such as coordinating container manipulation, jointly carrying large objects, or fine assembly) require simultaneous consideration of the relative posture, contact allocation, and motion synchronization of the two arms. However, there is currently a lack of large-scale, highly diverse available annotation datasets and automated synthesis methods for bi-arm collaboration.

[0004] Furthermore, most publicly available 2HandedAfforders are limited to extracting static contact points or interaction heatmaps, neglecting the future motion trajectory of objects after interaction (such as displacement or rotation caused by pushing or pulling). In embodied manipulation, the dynamic motion of objects is an indispensable part of availability. Static availability cannot provide a complete prediction of the consequences of actions, limiting the robot's ability to anticipate and plan the results of operations.

[0005] Therefore, there is an urgent need for a method that can automatically and accurately synthesize dual-arm availability data containing static contact and dynamic motion information to support the further development of dual-arm embodied intelligence. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes an automatic availability data synthesis method and system for dual-arm robot operations. High-quality availability data, including pixel-level static contact areas and the future trajectory of the object, is automatically generated from the mechanical dual-arm operation sequence without manual annotation.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention proposes an automatic synthesis method for availability data for dual-arm robot operation, comprising the following steps: The initial frame and the contact frame are identified from the timing data containing the operation process of the mechanical dual arms; the initial frame corresponds to the moment when the dual arms are not in contact with the target object; the contact frame corresponds to the starting moment when the dual arms make contact with the target object. The segmentation mask of the target object is obtained based on the initial frame; the segmentation mask of the dual-arm end effector is obtained based on the contact frame; Based on the segmentation mask of the target object and the segmentation mask of the dual-arm end effector, determine the static contact availability information between the dual arms and the target object; Based on the contact frame and subsequent image sequence, dynamic motion availability information is generated by tracking at least one feature point associated with the static contact availability information. The static contact availability information and the dynamic motion availability information are integrated to form an availability data annotation file.

[0008] Furthermore, the time-series data includes RGB video frames and robotic arm motion trajectory data.

[0009] Furthermore, the initial frame and contact frame are identified from the timing data containing the mechanical dual-arm operation process, specifically: Use the first frame image corresponding to the start time of the time series data as the initial frame; Analyze the motion trajectory data of the robotic arm and detect changes in the closed state of the end effectors of the two arms; The video frame corresponding to the moment when the two end effectors first reach a fully closed state simultaneously is taken as the contact frame.

[0010] Furthermore, a segmentation mask for the target object is obtained based on the initial frame; a segmentation mask for the dual-arm end effector is obtained based on the contact frame, specifically as follows: An open-vocabulary object detection model is used to detect objects in the initial frame to obtain bounding boxes of the target objects; based on the bounding boxes, an instance segmentation model is used to generate a segmentation mask for the target objects. An open vocabulary object detection model is used to detect the dual-arm end effectors in the contact frame, and the bounding boxes of the left and right end effectors are obtained. Based on the bounding boxes, an instance segmentation model is used to generate the segmentation masks of the left and right end effectors respectively.

[0011] Furthermore, the open vocabulary target detection model adopts the Grounding DINO model; the instance segmentation model adopts the Segment Anything Model 2 model.

[0012] Furthermore, based on the segmentation mask of the target object and the segmentation mask of the dual-arm end effector, the static contact availability information between the dual arms and the target object is determined, specifically as follows: Calculate the first intersection region between the segmentation mask of the target object and the segmentation mask of the left end effector; Calculate the second intersection region between the segmentation mask of the target object and the segmentation mask of the right end effector; The union of the first and second intersection regions is taken as the static contact availability region.

[0013] Furthermore, the method also includes: copying the static contact availability information determined by the contact frame to all image frames preceding the contact frame, as static availability annotation information for each frame.

[0014] Furthermore, dynamic motion availability information is generated by tracking at least one feature point associated with the static contact availability information, specifically as follows: A predetermined number of feature points are uniformly sampled within the static contact availability area of ​​the contact frame; The feature points are input into the video point tracking model to perform dense tracking on the image sequence after the contact frame; Obtain the pixel coordinates of each feature point in each subsequent frame to form a set of motion trajectories.

[0015] Furthermore, the static contact availability information and the dynamic motion availability information are integrated to form an availability data annotation file, specifically as follows: Establish a time-series index structure corresponding to the RGB video frame sequence in the time-series data; The static contact availability information of each RGB video frame is stored in the form of a mask and associated with the frame index of the corresponding frame. The feature point trajectory coordinates in the dynamic motion availability information of each RGB video frame after the contact frame are associated with the frame index of the corresponding frame and the static contact availability information of that frame. Generate an availability data annotation file, which includes at least: RGB video frame index, static contact availability mask data corresponding to each frame, and a set of feature point trajectory coordinates for each frame after the contact frame. The annotation file is associated with and stored with the RGB video frame sequence to form a complete availability dataset.

[0016] Secondly, the present invention also proposes an automatic availability data synthesis device for dual-arm robot operation, comprising at least one processor and a memory, wherein the memory stores a computer program, and the computer program, when executed by the at least one processor, implements the automatic availability data synthesis method for dual-arm robot operation.

[0017] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects: This invention proposes an automatic method and apparatus for synthesizing availability data for dual-arm robot operations, comprising the following steps: identifying an initial frame and a contact frame from time-series data containing the mechanical dual-arm operation process; the initial frame corresponds to the moment when the dual arms are not in contact with the target object; the contact frame corresponds to the starting moment when the dual arms make contact with the target object; obtaining a segmentation mask of the target object based on the initial frame; obtaining a segmentation mask of the dual-arm end effector based on the contact frame; determining static contact availability information between the dual arms and the target object based on the segmentation mask of the target object and the segmentation mask of the dual-arm end effector; generating dynamic motion availability information based on the contact frame and subsequent image sequences by tracking at least one feature point associated with the static contact availability information; and integrating the static contact availability information and the dynamic motion availability information to form an availability data annotation file. Based on this method, this invention also proposes an automatic availability data synthesis apparatus for dual-arm robot operations. This invention uses video and sequence data of mechanical dual-arm operations acquired in simulated or real environments, and utilizes a combination of a visual basic model and a point tracking model to efficiently generate high-quality annotated data containing static contact availability and dynamic motion availability. This method effectively overcomes the shortcomings of existing technologies, such as the lack of collaborative dual-arm availability datasets and the limitation of annotation to static contact. It achieves large-scale, highly diverse dual-arm availability data synthesis with high annotation accuracy and strong scalability.

[0018] This invention achieves zero-sample, pixel-level accurate segmentation of target objects and robotic arm end effectors by automatically extracting keyframes and combining advanced open vocabulary detection (Grounding DINO model) and instance segmentation model (SAM 2 model). This enables the automatic generation of high-quality dual-arm static contact availability annotations, solving the problems of existing technologies that rely on manual annotation and lack dual-arm collaboration data.

[0019] This invention not only accurately calculates the static contact area between the arms and the object, but also innovatively introduces a video point tracking model (such as the CoTracker-3 model) to generate the object's future trajectory based on feature point tracking within the contact area, thereby synthesizing dynamic motion availability information that includes a complete prediction of the action's consequences. This overcomes the fundamental deficiency of existing methods that only focus on static contact and ignore dynamic motion consequences.

[0020] This invention can adapt to various novel objects and different shapes of robotic arm end effectors without requiring retraining for specific objects. Combined with simulator or real-device data acquisition, it can efficiently and in parallel generate massive amounts of diverse dual-arm manipulation capability data, greatly increasing the scale of the dataset and the scope of tasks covered, providing a crucial data foundation for the pre-training and generalization of dual-arm embodied intelligent models. Attached Figure Description

[0021] Figure 1 This is a flowchart of the automatic synthesis method for availability data for dual-arm robot operation proposed in Embodiment 1 of the present invention; Figure 2 This is an implementation architecture diagram of the automatic synthesis method for availability data for dual-arm robot operation proposed in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of an automatic availability data synthesis device for dual-arm robot operation proposed in Embodiment 2 of the present invention. Detailed Implementation

[0022] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0023] Example 1 The automatic synthesis method of availability data for dual-arm robot operation proposed in Embodiment 1 of this invention is used to solve the problems of lack of dual-arm collaboration availability datasets and limited annotation information to static contact in the prior art.

[0024] Figure 1 This is a flowchart of the automatic synthesis method for availability data for dual-arm robot operation proposed in Embodiment 1 of the present invention; Figure 2 This is an implementation architecture diagram of the automatic synthesis method for availability data for dual-arm robot operation proposed in Embodiment 1 of the present invention; combined with Figure 1 and Figure 2 The process of implementing this invention will be explained together.

[0025] In step S1, timing data of the mechanical dual-arm operation process is acquired; Timing data of a robotic dual-arm performing a task (such as coordinating the pushing of an object) is collected from a robot simulation environment or a real robot experimental platform. The timing data includes at least: a sequence of RGB video frames arranged by timestamps, and synchronously recorded motion trajectory data such as the joint angles of the robotic dual arms, the pose of the end effector, and the opening and closing states of the gripper.

[0026] In step S2, an initial frame and a contact frame are identified from the timing data containing the operation process of the mechanical dual arms; the initial frame corresponds to the moment when the dual arms are not in contact with the target object; the contact frame corresponds to the starting moment when the dual arms make contact with the target object. The first frame of the time series data is taken as the initial frame; at this time, the robotic arm has not yet come into contact with the target object, and the target object is fully visible in the scene without any obstruction.

[0027] Analyze the motion trajectory data of the robotic arm and detect the changes in the closed state of the end effectors of the two arms; take the video frame corresponding to the moment when the end effectors of the two arms first reach the fully closed state simultaneously as the contact frame.

[0028] When the closing state values ​​of the left and right end effectors (such as grippers) simultaneously reach the preset full closure threshold for the first time, it is determined that the arms have established stable contact with the target object. The video frame corresponding to this moment is marked as the contact frame. The contact frame is the dividing point for subsequent calculations.

[0029] In step S3, the segmentation mask of the target object is obtained based on the initial frame; the segmentation mask of the dual-arm end effector is obtained based on the contact frame; An open-vocabulary object detection model is used to detect objects in the initial frame to obtain bounding boxes of the target objects; based on the bounding boxes, an instance segmentation model is used to generate a segmentation mask for the target objects. An open vocabulary object detection model is used to detect the dual-arm end effectors in the contact frame, and the bounding boxes of the left and right end effectors are obtained. Based on the bounding boxes, an instance segmentation model is used to generate the segmentation masks of the left and right end effectors respectively.

[0030] Among them, the open vocabulary target detection model adopts the Grounding DINO model; the instance segmentation model adopts the Segment Anything Model 2 model.

[0031] For the initial frame, the object name (e.g., "wooden block") from the task description is used as a text prompt input to the Grounding DINO model to detect the bounding box of the target object. This bounding box is then input into the SAM 2 model to obtain a high-precision object semantic segmentation mask (Mask_obj).

[0032] For contact frames, the text prompts "robotic gripper" or "end effector" are used, and the bounding boxes of the left and right end effectors are detected using Grounding DINO. Then, the left gripper segmentation mask (Mask_gripper_left) and the right gripper segmentation mask (Mask_gripper_right) are generated using the SAM 2 model.

[0033] If it is a single-arm operation task, this operation will obtain the semantic segmentation mask of the corresponding left or right arm.

[0034] This step fully utilizes the zero-shot open-vocabulary detection capability of the Grounding DINO model and the fine segmentation capability of the SAM 2 model to perform semantic segmentation on objects and robotic arms, avoiding the complexity of traditional manual annotation methods, and achieving natural language generalization for the detection and segmentation of novel object and robotic arm shapes.

[0035] In step S4, the static contact availability information between the dual arms and the target object is determined based on the segmentation mask of the target object and the segmentation mask of the dual-arm end effector; the detailed process is as follows: Calculate the first intersection region between the segmentation mask of the target object and the segmentation mask of the left end effector; Calculate the second intersection region between the segmentation mask of the target object and the segmentation mask of the right end effector; The union of the first and second intersection regions is taken as the static contact availability region.

[0036] ; in, Indicates the area of ​​static contact availability; Represents a semantic segmentation mask; This represents the semantic segmentation mask for the left arm gripper. This represents the semantic segmentation mask for the right arm gripper.

[0037] The static contact availability region accurately characterizes the static contact availability (i.e., graspable / accessible functional parts) in bi-arm collaboration, overcoming the limitation of existing methods that can only roughly estimate contact heatmaps from human videos, and achieving pixel-level accuracy in bi-arm contact availability labeling.

[0038] The present invention also includes: copying the static contact availability information determined by the contact frame to all image frames preceding the contact frame, as static availability annotation information for each frame.

[0039] Since the object is not obstructed by the robotic arm in all video frames before contact, and the object is stationary or slightly moving, its appearance is basically the same. Therefore, the static contact availability information obtained in step S3 is directly copied to all frames before the contact moment as static availability labels. This inheritance mechanism significantly reduces computational overhead while ensuring temporal consistency.

[0040] In step S5, based on the contact frame and subsequent image sequence, dynamic motion availability information is generated by tracking at least one feature point associated with the static contact availability information; specifically: A preset number of feature points are uniformly sampled within the static contact availability area of ​​the contact frame; the feature points are input into a video point tracking model to perform dense tracking on the image sequence after the contact frame; the pixel coordinates of each feature point in each subsequent frame are obtained to form a set of motion trajectories.

[0041] The video point tracking model uses the CoTracker-3 model.

[0042] The sequence is divided into two segments: before and after contact, using the contact frame as the boundary. For the video frame after contact, 10 keypoints are uniformly and randomly sampled within the static contact availability region. The contact frame and its sampled points are used as initialization cues and input into the CoTracker-3 model to perform dense point tracking on the entire post-contact video subsequence, obtaining the pixel coordinate trajectories of these 10 points in each subsequent frame. This set of trajectory points is defined as the dynamic motion availability annotation of the sequence, representing the future motion trend of the object under dual-arm manipulation.

[0043] The scope of protection of this invention is not limited to the specific values ​​listed in Example 1, and those skilled in the art can make reasonable selections based on the actual situation.

[0044] As the most advanced video point tracking model currently available, the CoTracker-3 model can maintain stable tracking for a long time under complex occlusion, rapid motion, and non-rigid deformation, thus providing a reliable representation of the motion consequences for dynamic availability and completely solving the fundamental defect of existing technologies that cannot capture the motion of objects after interaction.

[0045] In step S6, the static contact availability information and the dynamic motion availability information are integrated to form an availability data annotation file. The specific process includes: Establish a temporal index structure corresponding to the RGB video frame sequence in the temporal data; store the static contact availability information of each RGB video frame in mask form as an association with the frame index of the corresponding frame; associate the feature point trajectory coordinates in the dynamic motion availability information of each RGB video frame after the contact frame with the frame index of the corresponding frame and the static contact availability information of that frame; generate an availability data annotation file, which at least includes: RGB video frame index, static contact availability mask data corresponding to each frame, and a set of feature point trajectory coordinates corresponding to each frame after the contact frame; associate the annotation file with the RGB video frame sequence to form a complete availability dataset.

[0046] The static contact affordance mask (labeled separately for the left and right arms) and dynamic motion affordance trajectory points (only in the frame after contact) of each frame are integrated into a structured annotation file, which is then stored in pairs with the original RGB frames to form a complete two-arm affordance dataset. This dataset can be directly used for downstream two-arm policy pre-training, imitation learning, or reinforcement learning.

[0047] The automatic synthesis method for availability data for dual-arm robot operations proposed in Embodiment 1 of this invention uses video and sequence data of dual-arm robot operations collected in simulated or real environments. It efficiently generates high-quality labeled data containing both static contact and dynamic motion availability by combining a visual base model with a point tracking model. This method effectively overcomes the shortcomings of existing technologies, such as the lack of dual-arm collaborative availability datasets and the limitation of annotation to static contact. It achieves large-scale, highly diverse dual-arm availability data synthesis with high annotation accuracy and strong scalability.

[0048] Example 2 The present invention also proposes a device, Figure 3 This is a schematic diagram of an automatic availability data synthesis device for dual-arm robot operation proposed in Embodiment 2 of the present invention.

[0049] At the hardware level, electronic device 300 includes a processor 310, and optionally, an internal bus 320, a network interface 330, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or it may also include non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its functions. The processor 310, network interface 330, and memory can be interconnected via an internal bus 320. This internal bus 320 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in this diagram, but this does not imply that there is only one bus or one type of bus. The memory is used to store programs. Specifically, the program can include program code, which includes computer operation instructions. The memory can include main memory 340 and non-volatile memory 350, and provides instructions and data to the processor 310. Processor 310 reads the corresponding computer program from non-volatile memory 350 into memory 340 and then runs it, forming a device for locating the target user at the logical level. Processor 310 executes the program stored in memory and specifically performs the following: In step S1, timing data of the mechanical dual-arm operation process is acquired; In step S2, an initial frame and a contact frame are identified from the timing data containing the operation process of the mechanical dual arms; the initial frame corresponds to the moment when the dual arms are not in contact with the target object; the contact frame corresponds to the starting moment when the dual arms make contact with the target object. In step S3, the segmentation mask of the target object is obtained based on the initial frame; the segmentation mask of the dual-arm end effector is obtained based on the contact frame; In step S4, the static contact availability information between the dual arms and the target object is determined based on the segmentation mask of the target object and the segmentation mask of the dual-arm end effector. In step S5, based on the contact frame and subsequent image sequence, dynamic motion availability information is generated by tracking at least one feature point associated with the static contact availability information. In step S6, the static contact availability information and the dynamic motion availability information are integrated to form an availability data annotation file.

[0050] Figure 1 It can be applied to processor 810, or implemented by processor 810. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the processor or by instructions in the form of software. The processor mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0051] Example 3 The present invention also proposes a readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the following method steps: In step S1, timing data of the mechanical dual-arm operation process is acquired; In step S2, an initial frame and a contact frame are identified from the timing data containing the operation process of the mechanical dual arms; the initial frame corresponds to the moment when the dual arms are not in contact with the target object; the contact frame corresponds to the starting moment when the dual arms make contact with the target object. In step S3, the segmentation mask of the target object is obtained based on the initial frame; the segmentation mask of the dual-arm end effector is obtained based on the contact frame; In step S4, the static contact availability information between the dual arms and the target object is determined based on the segmentation mask of the target object and the segmentation mask of the dual-arm end effector. In step S5, based on the contact frame and subsequent image sequence, dynamic motion availability information is generated by tracking at least one feature point associated with the static contact availability information. In step S6, the static contact availability information and the dynamic motion availability information are integrated to form an availability data annotation file.

[0052] Embodiment 3 of this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory that stores a computer program, which can be executed by a processor to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0053] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks. Alternatively, if the integrated units of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0054] The automatic availability data synthesis device for dual-arm robot operation provided in Embodiment 2 of this application, and the description of the relevant part of the storage medium for automatic availability data synthesis for dual-arm robot operation in Embodiment 3 of this application, can be found in the detailed description of the corresponding part of the automatic availability data synthesis method for dual-arm robot operation provided in Embodiment 1 of this application, and will not be repeated here.

[0055] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0056] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for automatic synthesis of availability data for dual-arm robot operation, characterized in that, Includes the following steps: Identify the initial frame and contact frame from the timing data containing the operation process of the robotic arms; The initial frame corresponds to the moment when the arms are not in contact with the target object; the contact frame corresponds to the starting moment when the arms make contact with the target object. The segmentation mask of the target object is obtained based on the initial frame; The segmentation mask of the dual-arm end effector is obtained based on the contact frame; Based on the segmentation mask of the target object and the segmentation mask of the dual-arm end effector, determine the static contact availability information between the dual arms and the target object; Based on the contact frame and subsequent image sequence, dynamic motion availability information is generated by tracking at least one feature point associated with the static contact availability information. The static contact availability information and the dynamic motion availability information are integrated to form an availability data annotation file.

2. The method according to claim 1, characterized in that, The time-series data includes RGB video frames and robotic arm motion trajectory data.

3. The method according to claim 2, characterized in that, The initial frame and contact frame are identified from the timing data containing the operation process of the robotic arms, specifically: Use the first frame image corresponding to the start time of the time series data as the initial frame; Analyze the motion trajectory data of the robotic arm and detect changes in the closed state of the end effectors of the two arms; The video frame corresponding to the moment when the two end effectors first reach a fully closed state simultaneously is taken as the contact frame.

4. The method according to claim 1, characterized in that, The segmentation mask of the target object is obtained based on the initial frame; the segmentation mask of the dual-arm end effector is obtained based on the contact frame, specifically as follows: An open-vocabulary object detection model is used to detect objects in the initial frame to obtain bounding boxes of the target objects; based on the bounding boxes, an instance segmentation model is used to generate a segmentation mask for the target objects. An open vocabulary object detection model is used to detect the dual-arm end effectors in the contact frame, and the bounding boxes of the left and right end effectors are obtained. Based on the bounding boxes, an instance segmentation model is used to generate the segmentation masks of the left and right end effectors respectively.

5. The method according to claim 4, characterized in that, The open vocabulary target detection model adopts the Grounding DINO model; the instance segmentation model adopts the SAM2 model.

6. The method according to claim 1, characterized in that, Based on the segmentation mask of the target object and the segmentation mask of the dual-arm end effector, the static contact availability information between the dual arms and the target object is determined, specifically as follows: Calculate the first intersection region between the segmentation mask of the target object and the segmentation mask of the left end effector; Calculate the second intersection region between the segmentation mask of the target object and the segmentation mask of the right end effector; The union of the first and second intersection regions is taken as the static contact availability region.

7. The method according to claim 1, characterized in that, The method further includes: copying the static contact availability information determined by the contact frame to all image frames preceding the contact frame, as static availability annotation information for each frame.

8. The method according to claim 1, characterized in that, Dynamic motion availability information is generated by tracking at least one feature point associated with the static contact availability information, specifically as follows: A predetermined number of feature points are uniformly sampled within the static contact availability area of ​​the contact frame; The feature points are input into the video point tracking model to perform dense tracking on the image sequence after the contact frame; Obtain the pixel coordinates of each feature point in each subsequent frame to form a set of motion trajectories.

9. The method according to claim 2, characterized in that, The static contact availability information and the dynamic motion availability information are integrated to form an availability data annotation file, specifically as follows: Establish a time-series index structure corresponding to the RGB video frame sequence in the time-series data; The static contact availability information of each RGB video frame is stored in the form of a mask and associated with the frame index of the same RGB video frame. The feature point trajectory coordinates in the dynamic motion availability information of each RGB video frame after the contact frame are associated with the frame index and static contact availability information of the same RGB video frame. Generate an availability data annotation file, which includes at least: RGB video frame index, static contact availability mask data corresponding to each frame, and a set of feature point trajectory coordinates for each frame after the contact frame. The annotation file is associated with and stored with the RGB video frame sequence to form a complete availability dataset.

10. An automatic data synthesis device for dual-arm robot operation, comprising at least one processor and a memory, the memory storing a computer program, characterized in that, When the computer program is executed by the at least one processor, it implements the automatic synthesis method of availability data for dual-arm robot operation as described in any one of claims 1 to 9.

Images