A method, device and medium for obstacle avoidance of a mechanical arm for transparent experimental vessels

Abstract

The application discloses a kind of mechanical arm obstacle avoidance methods, equipment and medium to transparent experimental vessel, belong to robot motion planning field, method includes: step 1, in execution process, the mobile multi-view image of current scene is collected by RGB camera, and input pre-trained transparent experimental vessel instance segmentation model, obtain the pixel-level instance mask of each transparent experimental vessel, calculate image plane geometric centroid, and the observation ray of camera center to transparent experimental vessel center is established in combination with camera internal parameter and pose, after collecting a plurality of rays of predetermined quantity, the intersection point of a plurality of rays is calculated using least square method, as vessel three-dimensional centroid;Step 2, construct conservative obstacle bounding volume with three-dimensional centroid as center, and set side length or length-width-height according to category priori size, identification error upper bound and safety margin;Conservative obstacle bounding volume is written into mechanical arm obstacle map, collision detection and trajectory planning are carried out by calling motion planning module, complete online safety obstacle avoidance.

Description

Technical Field

[0001] This invention relates to the field of robot visual perception and motion planning technology, and in particular to a method for obstacle avoidance by a robotic arm over transparent experimental containers. Background Technology

[0002] In chemical laboratory settings, robotic arms performing tasks such as grasping, handling, and pouring often need to avoid widely distributed, fragile transparent containers such as beakers, conical flasks, and test tubes. Therefore, it is necessary to first identify the location and extent of these transparent containers and then generate obstacle avoidance constraints accordingly. Tong Ling et al. [see Reference 1] improved the contour recognition of transparent and reflective targets using a joint learning method of boundary detection and segmentation. However, this method primarily addresses 2D segmentation of transparent targets and cannot directly provide the 3D obstacle constraints required for robotic arm obstacle avoidance. Furthermore, this method mainly focuses on optimizing segmentation accuracy and does not further address instance-level separation, spatial positioning, and online obstacle avoidance of multiple transparent containers in laboratory settings. Wang Yi Ru et al. [see Reference 2] used a multi-view perception approach to improve the spatial perception capability of transparent objects, but their focus remains on the perception of the transparent objects themselves. This typically requires complex multi-view geometric reasoning or 3D reconstruction processes. In laboratory scenarios where robotic arms need to respond in real time, this approach suffers from complex computational processes, high deployment costs, and difficulty in directly converting these into motion planning obstacle avoidance constraints.

[0003] In view of this, the present invention is hereby proposed.

[0004] [Reference 1] Tong Ling, Qian Kun, Jing Xingshuo. LBSNet: LightweightJoint Boundary Detection and Semantic Segmentation for Transparent and Reflective Objects. IEEE Robotics and Automation Letters, 2024.

[0005] [Reference 2] Wang Yi Ru, Zhao Yuchi, Xu Haoping, Eppel Sagi, Aspuru-Guzik Alan, Shkurti Florian, Garg Animesh. MvTrans: Multi-view Perception ofTransparent Objects. arXiv, 2023. Summary of the Invention

[0006] The purpose of this invention is to provide a method, device, and medium for obstacle avoidance by a robotic arm for transparent laboratory vessels, which can achieve accurate identification, spatial position estimation, and safe online obstacle avoidance by the robotic arm, thereby solving the above-mentioned technical problems existing in the prior art.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A method for obstacle avoidance by a robotic arm over transparent laboratory containers, used in a laboratory robot system, comprising:

[0009] Step 1: During the task execution of the robotic arm of the laboratory robot system, the RGB camera of the laboratory robot system acquires moving multi-view images of the current scene. All acquired images are sequentially input into the pre-trained transparent experimental vessel instance segmentation model to obtain the pixel-level instance mask of each transparent experimental vessel. Based on the pixel-level instance mask of each transparent experimental vessel, the planar geometric centroid of all images is calculated. Combined with the intrinsic parameters and pose of the RGB camera, the observation ray from the camera center to the center of the transparent experimental vessel is established. After acquiring a predetermined number of observation rays as the RGB camera moves, the intersection point of the multiple observation rays is calculated using the least squares method, which serves as the three-dimensional centroid of the transparent experimental vessel.

[0010] Step 2: Construct a conservative obstacle enclosure centered on the three-dimensional centroid, and set the side length or length, width and height dimensions of the conservative obstacle enclosure centered on the prior size of the transparent experimental vessel category, the upper bound of the recognition error and the safety margin; write the conservative obstacle enclosure centered on the obstacle map of the robotic arm, call the motion planning module of the laboratory robot system to perform collision detection and trajectory planning based on the obstacle map of the robotic arm, and complete the online safe obstacle avoidance of the robotic arm for the transparent experimental vessel.

[0011] A processing apparatus, comprising:

[0012] At least one memory for storing one or more programs;

[0013] At least one processor is capable of executing one or more programs stored in the memory, such that when the one or more programs are executed by the processor, the processor can implement the method of the present invention.

[0014] A readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the methods described in this invention.

[0015] Compared with the prior art, the obstacle avoidance method, equipment, and medium for transparent experimental vessels provided by this invention have the following advantages:

[0016] By combining edge-aware instance segmentation, multi-view centroid triangulation, and conservative obstacle bounding volume modeling with a trained transparent experimental vessel instance segmentation model, the advantages of transparent vessel boundary recognition are retained, and three-dimensional obstacle avoidance constraints usable by the robotic arm can be directly generated with lower computational complexity. This is more suitable for the online safe obstacle avoidance of the robotic arm on transparent experimental vessels in real laboratory scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a robotic arm obstacle avoidance method for transparent experimental vessels provided in an embodiment of the present invention.

[0019] Figure 2 This is an overall flowchart of the obstacle avoidance method for a robotic arm on a transparent experimental vessel provided in an embodiment of the present invention.

[0020] Figure 3 This is a flowchart illustrating the obstacle avoidance method for a robotic arm over a transparent experimental vessel, as provided in an embodiment of the present invention.

[0021] Figure 4 This is a flowchart of the training phase of the obstacle avoidance method for a robotic arm on a transparent experimental vessel provided in an embodiment of the present invention.

[0022] Figure 5 This is a diagram illustrating the architecture of a transparent experimental vessel instance segmentation model for the obstacle avoidance method provided in this embodiment of the invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them, and do not constitute a limitation on the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0024] First, the following explanations are provided for the terms that may be used in this article:

[0025] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0026] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0027] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0028] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0029] The terms “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “up,” “down,” “front,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” indicate the current orientation or positional relationship, and are only for the convenience and simplification of description, and do not explicitly or implicitly suggest that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this document.

[0030] The technical solution provided by this invention will be described in detail below. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they shall be performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention whose manufacturers are not specified are all conventional products that can be purchased commercially.

[0031] like Figure 1 , Figure 2 and Figure 3As shown, this invention provides a method for obstacle avoidance by a robotic arm over transparent experimental containers, used in a laboratory robot system. The laboratory robot system includes: a robotic arm, an RGB camera, a camera calibration parameter acquisition module, a camera pose acquisition module, and a motion planning module; wherein the RGB camera is mounted at the end effector of the robotic arm and includes:

[0032] Step 1: During the task execution of the robotic arm of the laboratory robot system, the RGB camera of the laboratory robot system acquires moving multi-view images of the current scene. All acquired images are sequentially input into the pre-trained transparent experimental vessel instance segmentation model to obtain the pixel-level instance mask of each transparent experimental vessel. Based on the pixel-level instance mask of each transparent experimental vessel, the planar geometric centroid of all images is calculated. Combined with the intrinsic parameters and pose of the RGB camera, the observation ray from the camera center to the center of the transparent experimental vessel is established. After acquiring a predetermined number of observation rays as the RGB camera moves, the intersection point of the multiple observation rays is calculated using the least squares method, which serves as the three-dimensional centroid of the transparent experimental vessel.

[0033] Step 2: Construct a conservative obstacle enclosure centered on the three-dimensional centroid, and set the side length or length, width and height dimensions of the conservative obstacle enclosure centered on the prior size of the transparent experimental vessel category, the upper bound of the recognition error and the safety margin; write the conservative obstacle enclosure centered on the obstacle map of the robotic arm, call the motion planning module of the laboratory robot system to perform collision detection and trajectory planning based on the obstacle map of the robotic arm, and complete the online safe obstacle avoidance of the robotic arm for the transparent experimental vessel.

[0034] See Figure 4 Preferably, in the above method, the transparent experimental vessel instance segmentation model is pre-trained in the following manner, including:

[0035] Step S1: Acquire images of transparent laboratory vessels in a real laboratory environment, perform instance-level mask annotation on the transparent laboratory vessels in the images, generate edge supervision labels based on the instance mask boundaries, and divide the annotated transparent laboratory vessel images into training set, validation set, and test set;

[0036] Step S2: Construct and train a transparent experimental vessel instance segmentation model: Using a real-time instance segmentation network as the baseline network, extract multi-scale feature maps from the backbone network of the real-time instance segmentation network to construct a multi-scale edge detection branch. Perform channel compression, spatial alignment, and upsampling fusion on the extracted multi-scale feature maps to obtain fused edge features. Input the fused edge features into the real-time instance segmentation network as an edge-guided fusion structure. Set an edge map generation module and a SimAM attention module in the real-time instance segmentation network to obtain an edge-aware instance segmentation model. Train the edge-aware instance segmentation model using instance segmentation loss and edge supervision loss to obtain the transparent experimental vessel instance segmentation model.

[0037] Preferably, in step S1 of the above method, images of transparent laboratory vessels in a real laboratory environment are acquired in the following manner; instance-level mask annotations are performed on the transparent laboratory vessels in the images; edge supervision labels are generated based on the instance mask boundaries; and the annotated transparent laboratory vessel images are divided into training set, validation set, and test set, including:

[0038] Step S11: Use an RGB camera to record video of transparent experimental vessels in a real laboratory environment. During the acquisition process, change the distance between the RGB camera and the transparent experimental vessels and the shooting angle to obtain a video dataset. Perform frame extraction processing on the obtained video dataset, and then filter out frames with similarity higher than the preset value to obtain a transparent experimental vessel image dataset.

[0039] Step S12: Perform pixel-level instance mask annotation on the transparent experimental vessels in the transparent experimental vessel images of the acquired transparent experimental vessel image dataset, and generate edge supervision labels for each transparent experimental vessel image based on the pixel-level instance mask boundaries.

[0040] Step S13: Divide the labeled transparent experimental vessel image dataset into training set, validation set and test set in a ratio of 10:1:1; Before training, perform size normalization and data augmentation processing on the transparent experimental vessel images in the training set, validation set and test set.

[0041] Preferably, in step S11 of the above method, the RGB camera is a depth camera with an inertial measurement unit, such as an Intel RealSense D435i camera;

[0042] The transparent experimental apparatus in steps S11 and S12 includes one or more of the following: beaker, test tube, conical flask, volumetric flask, dropper, and glass rod;

[0043] In step S13, normalizing the size of the transparent experimental vessel images of the training set, validation set, and test set before training means adjusting the size of the transparent experimental vessel images to 640×640.

[0044] The data augmentation methods used include random flipping, brightness perturbation, and contrast perturbation.

[0045] Preferably, in step S2 of the above method, a real-time instance segmentation network is used as the baseline network. Multi-scale feature maps are extracted from the backbone network of the real-time instance segmentation network to construct a multi-scale edge detection branch. The extracted multi-scale feature maps are subjected to channel compression, spatial alignment, and upsampling fusion to obtain fused edge features. These fused edge features are input into the real-time instance segmentation network as an edge-guided fusion structure. An edge map generation module and a SimAM attention module are set in the real-time instance segmentation network to obtain an edge-aware instance segmentation model. The edge-aware instance segmentation model is trained using instance segmentation loss and edge supervision loss to obtain a transparent experimental vessel instance segmentation model, including:

[0046] Step S21: Select the real-time instance segmentation network YOLOv5-Seg as the baseline network. Extract the P2, P3, P4 and P5 feature maps from the backbone network of the real-time instance segmentation network to construct the multi-scale edge detection branch of the real-time instance segmentation network. Perform channel compression, spatial alignment and upsampling fusion on the P2, P3, P4 and P5 feature maps to obtain fused edge features. Input the fused edge features into the neck network of the real-time instance segmentation network as an edge-guided fusion structure. Enhance the boundary region features of the transparent experimental vessel and suppress background interference through this edge-guided fusion structure.

[0047] Step S23: After the multi-scale edge detection branch of the real-time instance segmentation network, an edge map generation module is set up to output an edge confidence map and use edge supervision loss to constrain the boundary of the transparent experimental vessel during the training phase.

[0048] Step S24: Set the SimAM attention module after the feature extraction module of the backbone network of the real-time instance segmentation network to enhance the feature separability between the transparent experimental vessel and the background, and obtain the edge-aware instance segmentation model.

[0049] Step S25: The weighted sum of instance segmentation loss and edge supervision loss is used as the training objective of the edge-aware instance segmentation model. The edge-aware instance segmentation model is trained until a predetermined training epoch is reached, at which point the transparent experimental vessel instance segmentation model is obtained. Here, training is considered complete after 400 epochs.

[0050] Preferably, in the above method, the training platform for the edge-aware instance segmentation model is PyTorch, the optimizer is Adam, the initial learning rate is 1×10^-3, and the batch size is 16.

[0051] Preferably, in step 1 of the above method, during the task execution by the robotic arm of the laboratory robot system, the following steps are taken: The RGB camera of the laboratory robot system acquires moving multi-view images of the current scene, and all acquired images are sequentially input into a pre-trained transparent experimental vessel instance segmentation model to obtain a pixel-level instance mask for each transparent experimental vessel. Based on the pixel-level instance mask of each transparent experimental vessel, the planar geometric centroid of all images is calculated. Combined with the intrinsic parameters and pose of the RGB camera, an observation ray from the camera center to the center of the transparent experimental vessel is established. After acquiring a predetermined number of rays as the RGB camera moves, the intersection point of the multiple rays is calculated using the least squares method, which serves as the three-dimensional centroid of the transparent experimental vessel. This includes:

[0052] Step 11: Perform hand-eye calibration on the RGB camera. When the RGB camera is mounted on the end of the robotic arm and moves with it, the camera pose is published through the TF tree.

[0053] Step 12: During the task execution of the robotic arm, acquire the current scene image and input the current scene image into the pre-trained transparent experimental vessel instance segmentation model, and output the category label, detection box and pixel-level instance mask for each transparent experimental vessel;

[0054] Step 13: Based on the category information, location proximity, mask overlap and temporal continuity of the transparent experimental vessel, perform correlation tracking on the segmentation results of instances belonging to the same transparent experimental vessel under continuous viewpoints, so that the same transparent experimental vessel under different viewpoints corresponds.

[0055] Step 14: Create an instance mask for the transparent experimental vessel from the i-th viewpoint. Calculate the two-dimensional geometric centroid using the following formula. ,for:

[0056] ;

[0057] in, These are the two-dimensional coordinates of the center of mass of a transparent object; These are the horizontal and vertical coordinates in the image; It is an instance mask of the transparent experimental vessel from the i-th viewpoint; It is the centroid of a two-dimensional geometry;

[0058] The unit line-of-sight direction in the camera coordinate system is obtained by back-projecting the two-dimensional geometric centroid onto the RGB camera intrinsic parameter matrix K. The unit line of sight direction satisfies:

[0059] ;

[0060] in, It represents the unit viewing direction in the camera coordinate system; K is the RGB camera intrinsic parameter matrix. These are the two-dimensional coordinates of the center of mass of a transparent object;

[0061] Based on the camera rotation matrix corresponding to the i-th viewpoint The unit's line of sight direction Transform to the robot arm's base coordinate system to obtain the direction of the observed ray. The direction of the observed ray satisfy:

[0062] ;

[0063] in, It is to observe the direction of the ray; It is the camera rotation matrix.

[0064] When the cumulative number of observation angles for the same transparent experimental vessel reaches the minimum number of solutions. At that time, multiple sets of observation rays corresponding to the transparent experimental vessel were collected. , The three-dimensional centroid of the transparent experimental vessel was determined using the least-squares multi-ray intersection method with multiple sets of observation rays. The three-dimensional centroid satisfies:

[0065] ;

[0066] in, It is the three-dimensional centroid of the transparent experimental vessel; It is the identity matrix; It is to observe the direction of the ray; is the camera center; p is a point in the 3D space to be optimized.

[0067] The three-dimensional centroid of the transparent experimental vessel obtained by the solution. The result serves as an estimate of the spatial position of the transparent experimental vessel within the workspace of the robotic arm, and the three-dimensional centroid is updated online after obtaining new perspective observations.

[0068] Preferably, in step 2 of the above method, a conservative obstacle enclosure is constructed with the three-dimensional centroid as the center, and the side length or length, width, and height dimensions of the conservative obstacle enclosure are set according to the prior size of the transparent experimental vessel category, the upper bound of the recognition error, and the safety margin; the conservative obstacle enclosure is written into the obstacle map of the robotic arm, and the motion planning module of the laboratory robot system is called to perform collision detection and trajectory planning based on the obstacle map of the robotic arm, thereby completing the online safe obstacle avoidance of the robotic arm for the transparent experimental vessel, including:

[0069] Step 21, using the three-dimensional centroid obtained in Step 1 Construct a conservative barrier enclosure around the center, wherein the conservative barrier enclosure is an axis-aligned cube or an axis-aligned cuboid;

[0070] Step 22: Set the side length or length, width and height dimensions of the conservative barrier enclosure according to the prior size of the transparent experimental vessel category, the upper limit of the identification error and the safety margin, so that the conservative barrier enclosure covers the space occupied by the transparent experimental vessel.

[0071] Step 23: Write the conservative obstacle enclosure into the robotic arm obstacle map, call the motion planning module to perform collision detection and trajectory planning based on the robotic arm obstacle map, and trigger trajectory replanning when the obstacle map is updated during trajectory execution;

[0072] Step 24: The robotic arm continues to repeat steps 3 to 4 during the movement until the target operation task is completed.

[0073] This invention also provides a processing apparatus, comprising:

[0074] At least one memory for storing one or more programs;

[0075] At least one processor is capable of executing one or more programs stored in the memory, such that when the one or more programs are executed by the processor, the processor can implement the methods described above.

[0076] The present invention further provides a readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0077] Compared to existing technologies that primarily focus on 2D segmentation of transparent targets and struggle to directly provide the 3D constraints required for obstacle avoidance by robotic arms, or that emphasize 3D perception of transparent objects but suffer from complex computational processes and insufficient real-time performance, the obstacle avoidance method of this invention enhances the representation of boundary features of transparent experimental vessels by introducing edge detection branches, edge-guided fusion structures, and SimAM attention modules into the transparent experimental vessel instance segmentation model. This improves instance recognition performance under conditions of refraction, reflection, and weak texture. Simultaneously, this invention employs multi-view triangulation using the geometric centroid of the instance mask and constructs a conservative obstacle bounding volume using the 3D centroid, enabling the direct conversion of 2D recognition results into 3D obstacle avoidance constraints usable by robotic arms. Therefore, compared to existing methods that only output 2D segmentation results, this invention adds spatial localization and online obstacle avoidance capabilities; compared to existing methods that rely on depth completion or complex 3D reconstruction, it reduces dependence on depth quality and the reconstruction process, making it more suitable for online applications in laboratory transparent vessel scenarios.

[0078] When validated on a real-world dataset of transparent laboratory vessels, the method of this invention achieved an instance segmentation performance of 82.2 mAP50:95, a boundary F-score of 97.80, and an F-score of 98.21 for slender target boundaries, with a single-frame inference latency of approximately 7.1 ms. In a real robotic arm experiment, the average error of 3D centroid estimation was approximately 38.0 mm, and it succeeded in 14 out of 15 obstacle avoidance experiments of varying difficulty, achieving a success rate of 93.3%. These results demonstrate that the present invention achieves relatively stable technical performance in terms of transparent laboratory vessel recognition accuracy, boundary quality, spatial localization, and online obstacle avoidance.

[0079] To more clearly demonstrate the technical solution and its effects provided by the present invention, the following detailed description of the solution provided by the embodiments of the present invention is provided with reference to specific examples.

[0080] Example 1

[0081] This embodiment provides a robotic arm obstacle avoidance method for transparent experimental vessels, which is a robotic arm obstacle avoidance method based on edge-aware instance segmentation and multi-view triangulation. This method is applicable to laboratory robot systems, which include a robotic arm, an RGB camera, a camera calibration parameter acquisition module, a camera pose acquisition module, and a motion planning module; wherein the RGB camera is mounted at the end effector of the robotic arm. Figure 2 The left-hand diagram illustrates a laboratory obstacle avoidance scenario involving transparent containers: an RGB camera is mounted at the end of a robotic arm, and transparent containers such as graduated cylinders, test tubes, and conical flasks are placed on the experimental table. The robotic arm needs to move from an initial pose to a target pose, identifying the positions of the transparent containers on the table during the movement, and avoiding collisions to complete the task. The method includes a training phase and an application phase. The training phase is used to obtain a segmentation model of transparent container instances, and the application phase is used for the identification, localization, and online obstacle avoidance of the transparent containers.

[0082] Among them, using a camera to acquire images, using a segmentation model to identify targets, and calling a motion planning module to perform trajectory planning are common features in existing technologies. The difference between this invention and existing technologies is that: an edge detection branch, an edge-guided fusion structure, and a SimAM attention module are introduced into the instance segmentation network to improve the boundary recognition effect of transparent vessels; multi-view triangulation is performed using the geometric centroid of the instance mask to obtain the three-dimensional centroid of the transparent vessel; a conservative obstacle bounding body is constructed based on the three-dimensional centroid and used for online obstacle avoidance by the robotic arm.

[0083] See Figures 1 to 5 The obstacle avoidance method for the robotic arm includes the following steps:

[0084] (I) Training Phase:

[0085] Step S1: Acquire images of transparent experimental vessels in a real laboratory environment and perform instance-level mask annotation on the transparent experimental vessels in the images; generate edge supervision labels based on the instance mask boundaries, and divide the annotated images into training set, validation set and test set.

[0086] Step S2: Construct and train an edge-aware instance segmentation model. Using a real-time instance segmentation network as the baseline network, multi-scale feature maps are extracted from the backbone network to construct a multi-scale edge detection branch. Channel compression, spatial alignment, and upsampling fusion are performed on the multi-scale feature maps. An edge map generation module, an edge-guided fusion structure, and a SimAM attention module are set in the model. The model is trained using instance segmentation loss and edge supervision loss to obtain the transparent experimental vessel instance segmentation model.

[0087] Further, in step S1 above: images of transparent experimental vessels in a real laboratory environment are acquired, and instance-level mask annotations are performed on the transparent experimental vessels in the images; edge supervision labels are generated based on the instance mask boundaries, and the annotated images are divided into training set, validation set, and test set. The validation set is used to evaluate model performance during training, help adjust hyperparameters, monitor for overfitting, and select the best-performing model; the test set is used after all model training and hyperparameter tuning are completed to finally evaluate the model's generalization ability.

[0088] Specifically, it includes the following sub-steps S11 to S13:

[0089] Step S11: Record video of transparent laboratory glassware in a real laboratory environment using an Intel RealSense D435i camera. During acquisition, the distance between the camera and the glassware and the viewing angle are varied. The obtained video dataset is then processed by frame extraction, and frames with excessively high similarity are filtered out. In this embodiment, a total of 3485 RGB images were obtained, with a resolution of 640×480. The images contain 21 common types of transparent laboratory glassware, totaling 6099 instances.

[0090] Step S12: Perform pixel-level instance mask annotation on the acquired image, and generate edge supervision labels based on the instance mask boundaries. The transparent experimental apparatus includes one or more of beakers, test tubes, conical flasks, volumetric flasks, droppers, and glass rods.

[0091] Step S13: Divide the labeled images into training set, validation set and test set in a ratio of 10:1:1; Before training, perform size normalization and data augmentation on the images, where the input image size is adjusted to 640×640, and the data augmentation methods include random flipping, brightness perturbation and contrast perturbation.

[0092] Further, in step S2 above: constructing a transparent experimental vessel instance segmentation model. Using a real-time instance segmentation network as the baseline network, multi-scale feature maps are extracted from the backbone network, a multi-scale edge detection branch is constructed, and channel compression, spatial alignment, and upsampling fusion are performed on the multi-scale feature maps; an edge map generation module, an edge-guided fusion structure, and a SimAM attention module are set in the model, and the model is trained using instance segmentation loss and edge supervision loss to obtain the transparent experimental vessel instance segmentation model.

[0093] Specifically, it includes the following sub-steps S21 to S24:

[0094] Step S21: Select YOLOv5-Seg as the baseline network, construct a multi-scale edge detection branch, perform channel compression, spatial alignment and upsampling fusion on the P2, P3, P4 and P5 feature maps to obtain fused edge features, input the fused edge features into the neck network of the segmentation model, and enhance the features of the boundary region of the transparent experimental vessel and suppress background interference through the edge-guided fusion structure.

[0095] Step S22: Simultaneously, set up an edge map generation module after the edge detection branch to output an edge confidence map, and use edge supervision loss to constrain the boundary of the transparent experimental vessel during the training phase.

[0096] Step S23: Set up a SimAM attention module after the feature extraction module of the backbone network to enhance the feature separability between the transparent experimental vessel and the background.

[0097] Step S24: The edge-aware instance segmentation model is trained using a weighted sum of instance segmentation loss and edge supervision loss as the model training objective. In this embodiment, the training platform is PyTorch, the optimizer is Adam, the initial learning rate is 1×10^-3, and the batch size is 16. After training, the transparent experimental vessel instance segmentation model is obtained.

[0098] (II) Application Phase:

[0099] like Figure 1 , Figure 2 As shown, it includes the following steps:

[0100] Step 1: During the robot arm's task execution, multi-view images of the current scene are acquired using an RGB camera. These images are then input into the transparent experimental vessel instance segmentation model obtained in Step S2 to obtain a pixel-level instance mask for each transparent experimental vessel. Based on the instance mask of each transparent experimental vessel, the geometric centroid of the image plane is calculated. Combined with camera intrinsic parameters and camera pose, an observation ray from the camera center to the target object center at the current pose is established. After acquiring a sufficient number of rays as the camera moves, the intersection point of the multiple rays is calculated using the least squares method, which serves as the three-dimensional centroid of the transparent experimental vessel.

[0101] Step 2: Construct a conservative obstacle enclosure centered on the three-dimensional centroid, and set the side length or length, width and height dimensions of the conservative obstacle enclosure centered on the prior size of the transparent experimental vessel category, the upper bound of the recognition error and the safety margin; write the conservative obstacle enclosure centered on the obstacle map of the robotic arm, call the motion planning module to perform collision detection and trajectory planning, thereby realizing the online safe obstacle avoidance of the robotic arm for the transparent experimental vessel.

[0102] Further, in step 1 above: during the robotic arm's task execution, a moving multi-view image of the current scene is acquired using an RGB camera, and this current scene image is input into the transparent experimental vessel instance segmentation model obtained in step 2 to obtain a pixel-level instance mask for each transparent experimental vessel. Based on the instance mask of each transparent experimental vessel, the geometric centroid of the image plane is calculated, and an observation ray from the camera center to the target object center is established using the camera intrinsic parameters and camera pose. After acquiring a sufficient number of observation rays as the camera moves, the intersection point of multiple observation rays is calculated using the least squares method, which serves as the three-dimensional centroid of the transparent experimental vessel.

[0103] Specifically, it includes the following sub-steps 11 to 14:

[0104] Step 11: Perform hand-eye calibration on the RGB camera. When the RGB camera is mounted on the end of the robotic arm and moves with it, the camera pose is published through the TF tree.

[0105] Step 12: Acquire the current scene image during the robotic arm's task execution, and input the current scene image into the transparent experimental vessel instance segmentation model obtained in Step 2, outputting the category label, detection box, and pixel-level instance mask for each transparent experimental vessel.

[0106] Step 13: Based on category information, location proximity, mask overlap, or temporal continuity, perform correlation tracking on the segmentation results of instances belonging to the same transparent experimental vessel under continuous viewpoints, thereby matching the same transparent experimental vessel under different viewpoints.

[0107] Step 14: For the first Example mask of transparent experimental apparatus from a different perspective Calculate the centroid of a two-dimensional geometry The two-dimensional geometric centroid satisfies:

[0108] ;

[0109] in, These are the two-dimensional coordinates of the center of mass of a transparent object; These are the horizontal and vertical coordinates in the image; It is an instance mask of the transparent experimental vessel from the i-th viewpoint; It is the centroid of a two-dimensional geometry;

[0110] Based on the camera intrinsic parameter matrix By back-projecting the two-dimensional geometric centroid, the unit line-of-sight direction in the camera coordinate system is obtained. The unit line of sight direction satisfies:

[0111] ;

[0112] in, It represents the unit viewing direction in the camera coordinate system; K is the RGB camera intrinsic parameter matrix. These are the two-dimensional coordinates of the center of mass of a transparent object;

[0113] According to the Camera rotation matrix corresponding to each viewpoint The unit line of sight is transformed to the robot arm's base coordinate system to obtain the observation ray. The observed ray satisfies:

[0114] ;

[0115] in, It is to observe the direction of the ray. It is the camera rotation matrix.

[0116] When the cumulative number of observation angles of the same transparent experimental vessel When the predetermined minimum number of solutions is reached, collect multiple sets of observation rays corresponding to the transparent experimental vessel. , In this embodiment, the predetermined minimum number of solvable quantities is set to be no less than 3; the three-dimensional centroid of the transparent experimental vessel is solved using the least squares multi-ray intersection method. The three-dimensional centroid satisfies:

[0117] ;

[0118] in, It is the three-dimensional centroid of the transparent experimental vessel; It is the identity matrix; It is to observe the direction of the ray; is the camera center; p is a point in the 3D space to be optimized.

[0119] The obtained three-dimensional centroid The three-dimensional centroid is estimated as the spatial position of the transparent experimental vessel within the robotic arm's workspace, and is updated online after obtaining new perspective observations. In this embodiment, tests were conducted on five scenarios consisting of 1 to 5 transparent experimental vessels. The average error of the three-dimensional centroid estimation was approximately 38.0 mm, and the maximum error was approximately 60.5 mm.

[0120] Further, in step 2 above: a conservative obstacle enclosure is constructed with the three-dimensional centroid as the center, and the side length or length, width and height dimensions of the conservative obstacle enclosure are set according to the prior size of the transparent experimental vessel category, the upper bound of the recognition error and the safety margin; the conservative obstacle enclosure is written into the obstacle map of the robotic arm, and the motion planning module is called to perform collision detection and trajectory planning, thereby realizing the online safe obstacle avoidance of the robotic arm for the transparent experimental vessel.

[0121] Specifically, this includes the following sub-steps 21 to 24:

[0122] Step 21: Using the three-dimensional centroid obtained in Step 1 Construct a conservative barrier enclosure around the center. The conservative barrier enclosure is preferably an axis-aligned cube, but it can also be an axis-aligned cuboid.

[0123] Step 22: Based on the prior dimensions of the transparent experimental vessel category, the upper bound of the identification error, and the safety margin, set the side length or length, width, and height dimensions of the conservative barrier enclosure to ensure that the conservative barrier enclosure covers the space that the transparent experimental vessel may occupy. For slender vessels, a larger safety redundancy can be set along their main direction.

[0124] Step 23: Write the conservative obstacle bounding body into the robotic arm obstacle map, and call the motion planning module to perform collision detection and trajectory planning based on the robotic arm obstacle map; when the obstacle map is updated during trajectory execution, trigger trajectory replanning.

[0125] Step 24: The robotic arm continues to repeat steps 13 to 14 during its movement until the target operation task is completed. In this embodiment, obstacle avoidance verification was performed on three types of experimental scenarios: simple, medium, and complex. Five experiments were conducted for each scenario, for a total of 15 experiments. The experimental results show that the robotic arm successfully completed 14 obstacle avoidance tasks, with an overall success rate of 93.3% and an average planning time of approximately 0.017 seconds.

[0126] It is understood that the obstacle avoidance method of the present invention is not limited to the above implementation scenarios. In actual use, the camera arrangement, instance segmentation baseline network, obstacle enclosing body form and safety margin parameters can be adjusted according to the characteristics of the laboratory scenario. Alternatively, other network structures and planning algorithms that meet real-time requirements can be adopted to achieve the same technical idea as technology develops.

[0127] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0128] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for obstacle avoidance by a robotic arm over transparent experimental containers, characterized in that, Robotic arms used in laboratory robotic systems avoid obstacles in transparent laboratory containers, including: Step 1: During the task execution of the robotic arm of the laboratory robot system, the RGB camera of the laboratory robot system acquires moving multi-view images of the current scene. All acquired images are sequentially input into the pre-trained transparent experimental vessel instance segmentation model to obtain the pixel-level instance mask of each transparent experimental vessel. The planar geometric centroid of all images is calculated based on the pixel-level instance mask of each transparent experimental vessel. The observation ray from the camera center to the center of the transparent experimental vessel is established by combining the intrinsic parameters and pose of the RGB camera. After a predetermined number of observation rays are acquired as the RGB camera moves, the intersection point of the multiple observation rays is calculated using the least squares method, which serves as the three-dimensional centroid of the transparent experimental vessel. Step 2: Construct a conservative obstacle enclosure centered on the three-dimensional centroid, and set the side length or length, width and height dimensions of the conservative obstacle enclosure centered on the prior size of the transparent experimental vessel category, the upper bound of the recognition error and the safety margin; write the conservative obstacle enclosure centered on the obstacle map of the robotic arm, call the motion planning module of the laboratory robot system to perform collision detection and trajectory planning based on the obstacle map of the robotic arm, and complete the online safe obstacle avoidance of the robotic arm for the transparent experimental vessel; The transparent experimental vessel instance segmentation model was pre-trained in the following manner, including: Step S1: Acquire images of transparent laboratory vessels in a real laboratory environment, perform instance-level mask annotation on the transparent laboratory vessels in the images, generate edge supervision labels based on the instance mask boundaries, and divide the annotated transparent laboratory vessel images into training set, validation set, and test set; Step S2: Construct and train a transparent experimental vessel instance segmentation model: Using a real-time instance segmentation network as the baseline network, extract multi-scale feature maps from the backbone network of the real-time instance segmentation network to construct a multi-scale edge detection branch. Perform channel compression, spatial alignment, and upsampling fusion on the extracted multi-scale feature maps to obtain fused edge features. Input the fused edge features into the real-time instance segmentation network as an edge-guided fusion structure. Set an edge map generation module and a SimAM attention module in the real-time instance segmentation network to obtain an edge-aware instance segmentation model. Train the edge-aware instance segmentation model using instance segmentation loss and edge supervision loss to obtain a transparent experimental vessel instance segmentation model. In step S2, the real-time instance segmentation network is used as the baseline network. Multi-scale feature maps are extracted from the backbone network of the real-time instance segmentation network to construct a multi-scale edge detection branch. The extracted multi-scale feature maps are subjected to channel compression, spatial alignment, and upsampling fusion to obtain fused edge features. These fused edge features are input into the real-time instance segmentation network as an edge-guided fusion structure. An edge map generation module and a SimAM attention module are set in the real-time instance segmentation network to obtain an edge-aware instance segmentation model. The edge-aware instance segmentation model is trained using instance segmentation loss and edge supervision loss to obtain a transparent experimental vessel instance segmentation model, including: Step S21: Select the real-time instance segmentation network YOLOv5-Seg as the baseline network. Extract the P2, P3, P4 and P5 feature maps from the backbone network of the real-time instance segmentation network to construct the multi-scale edge detection branch of the real-time instance segmentation network. Perform channel compression, spatial alignment and upsampling fusion on the P2, P3, P4 and P5 feature maps to obtain fused edge features. Input the fused edge features into the neck network of the real-time instance segmentation network as an edge-guided fusion structure. Enhance the boundary region features of the transparent experimental vessel and suppress background interference through this edge-guided fusion structure. Step S23: After the multi-scale edge detection branch of the real-time instance segmentation network, an edge map generation module is set up to output an edge confidence map and use edge supervision loss to constrain the boundary of the transparent experimental vessel during the training phase. Step S24: Set the SimAM attention module after the feature extraction module of the backbone network of the real-time instance segmentation network to enhance the feature separability between the transparent experimental vessel and the background, and obtain the edge-aware instance segmentation model. Step S25: The weighted sum of instance segmentation loss and edge supervision loss is used as the training objective of the edge-aware instance segmentation model. The edge-aware instance segmentation model is trained until a predetermined training epoch is reached, and the transparent experimental vessel instance segmentation model is obtained after the training is completed.

2. The obstacle avoidance method for a robotic arm over a transparent experimental vessel according to claim 1, characterized in that, In step S1, images of transparent experimental vessels in a real laboratory environment are acquired in the following manner, and instance-level mask annotations are performed on the transparent experimental vessels in the images. Edge supervision labels are generated based on the instance mask boundaries, and the labeled transparent experimental vessel images are divided into training, validation, and test sets, including: Step S11: Use an RGB camera to record video of transparent experimental vessels in a real laboratory environment. During the acquisition process, change the distance between the RGB camera and the transparent experimental vessels and the shooting angle to obtain a video dataset. Perform frame extraction processing on the obtained video dataset, and then filter out frames with similarity higher than the preset value to obtain a transparent experimental vessel image dataset. Step S12: Perform pixel-level instance mask annotation on the transparent experimental vessels in the transparent experimental vessel images of the acquired transparent experimental vessel image dataset, and generate edge supervision labels for each transparent experimental vessel image based on the pixel-level instance mask boundaries. Step S13: Divide the labeled transparent experimental vessel image dataset into training set, validation set and test set in a ratio of 10:1:1; Before training, perform size normalization and data augmentation processing on the transparent experimental vessel images in the training set, validation set and test set.

3. The obstacle avoidance method for a robotic arm on a transparent experimental vessel according to claim 2, characterized in that, In step S11, the RGB camera is a depth camera with an inertial measurement unit; The transparent experimental apparatus in steps S11 and S12 includes one or more of the following: beaker, test tube, conical flask, volumetric flask, dropper, and glass rod; In step S13, normalizing the size of the transparent experimental vessel images of the training set, validation set, and test set before training means adjusting the size of the transparent experimental vessel images to 640×640. The data augmentation methods used include random flipping, brightness perturbation, and contrast perturbation.

4. The obstacle avoidance method for a robotic arm on a transparent experimental vessel according to claim 1, characterized in that, The training platform for the edge-aware instance segmentation model is PyTorch, the optimizer is Adam, the initial learning rate is 1×10^-3, and the batch size is 16.

5. The obstacle avoidance method for a robotic arm over a transparent experimental vessel according to any one of claims 1 to 3, characterized in that, In step 1, during the task execution by the robotic arm of the laboratory robot system, the following method is used: The RGB camera of the laboratory robot system acquires multi-view images of the current scene, and all acquired images are sequentially input into a pre-trained transparent experimental vessel instance segmentation model to obtain a pixel-level instance mask for each transparent experimental vessel. Based on the pixel-level instance mask of each transparent experimental vessel, the planar geometric centroid of all images is calculated. Combined with the RGB camera intrinsic parameters and RGB camera pose, an observation ray from the camera center to the center of the transparent experimental vessel is established. After acquiring a predetermined number of rays as the RGB camera moves, the intersection point of the multiple rays is calculated using the least squares method, which serves as the three-dimensional centroid of the transparent experimental vessel. This includes: Step 11: Perform hand-eye calibration on the RGB camera. When the RGB camera is mounted on the end of the robotic arm and moves with it, the camera pose is published through the TF tree. Step 12: During the task execution of the robotic arm, acquire the current scene image and input the current scene image into the pre-trained transparent experimental vessel instance segmentation model, and output the category label, detection box and pixel-level instance mask for each transparent experimental vessel; Step 13: Based on the category information, location proximity, mask overlap and temporal continuity of the transparent experimental vessel, perform correlation tracking on the segmentation results of instances belonging to the same transparent experimental vessel under continuous viewpoints, so that the same transparent experimental vessel under different viewpoints corresponds. Step 14: Apply an instance mask to the transparent experimental vessel from the current viewpoint. Calculate the two-dimensional geometric centroid using the following formula. ,for: ； in, These are the two-dimensional coordinates of the centroid of the transparent experimental apparatus; These are the x and y coordinates in the image; It is the first Examples of transparent experimental apparatus masks from various perspectives; It is the centroid of a two-dimensional geometry; The unit line-of-sight direction in the camera coordinate system is obtained by back-projecting the two-dimensional geometric centroid onto the RGB camera intrinsic parameter matrix K. The unit line of sight direction satisfies: ； in, It is the unit line-of-sight direction in the camera coordinate system; It is the RGB camera intrinsic parameter matrix; These are the two-dimensional coordinates of the centroid of the transparent experimental apparatus; Based on the camera rotation matrix corresponding to the current viewpoint The unit's line of sight direction Transform to the robot arm's base coordinate system to obtain the direction of the observed ray. The direction of the observed ray satisfy: ； When the cumulative number of observation angles of the same transparent experimental vessel When the predetermined minimum number of solutions is reached, collect multiple sets of observation rays corresponding to the transparent experimental vessel. , The three-dimensional centroid of the transparent experimental vessel was determined using the least-squares multi-ray intersection method with multiple sets of observation rays. The three-dimensional centroid satisfy: ； in, It is the identity matrix; It is to observe the direction of the ray; is the camera center; p is a point in the 3D space to be optimized. The obtained three-dimensional centroid The result serves as an estimate of the spatial position of the transparent experimental vessel within the workspace of the robotic arm, and the three-dimensional centroid is updated online after obtaining new perspective observations.

6. The obstacle avoidance method for a robotic arm on a transparent experimental vessel according to claim 5, characterized in that, In step 2, a conservative obstacle enclosure is constructed centered on the three-dimensional centroid in the following manner, and the side length or length, width, and height dimensions of the conservative obstacle enclosure are set according to the prior size of the transparent experimental vessel category, the upper bound of the recognition error, and the safety margin; the conservative obstacle enclosure is written into the obstacle map of the robotic arm, and the motion planning module of the laboratory robot system is invoked to perform collision detection and trajectory planning based on the obstacle map of the robotic arm, completing the online safe obstacle avoidance of the robotic arm for the transparent experimental vessel, including: Step 21, using the three-dimensional centroid obtained in Step 1 Construct a conservative barrier enclosure around the center, wherein the conservative barrier enclosure is an axis-aligned cube or an axis-aligned cuboid; Step 22: Set the side length or length, width and height dimensions of the conservative barrier enclosure according to the prior size of the transparent experimental vessel category, the upper limit of the identification error and the safety margin, so that the conservative barrier enclosure covers the space occupied by the transparent experimental vessel. Step 23: Write the conservative obstacle enclosure into the robotic arm obstacle map, call the motion planning module to perform collision detection and trajectory planning based on the robotic arm obstacle map, and trigger trajectory replanning when the obstacle map is updated during trajectory execution; Step 24: The robotic arm continues to repeat steps 13 to 14 during the movement until the target operation task is completed.

7. A processing device, characterized in that, include: At least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, such that when the one or more programs are executed by the processor, the processor can perform the method according to any one of claims 1-6.

8. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it can implement the method described in any one of claims 1-6.

Images