Dual-robot collaborative grasping method based on drug traceability code and multi-source vision fusion

Abstract

The application provides a double-robot-arm cooperative grabbing method based on a medicine traceability code and multi-source vision fusion, relates to the technical field of control, and comprises the following steps: obtaining a material type and a surface friction coefficient by analyzing the traceability code, combining a surface geometric topology structure generated by multi-source vision, establishing a contact mechanics mapping relationship, determining an adaptive grabbing area set, calculating an envelope fitting degree to select an optimal target grabbing area, and determining a force application size to perform grabbing according to the optimal target grabbing area. The application improves the grabbing stability and safety of double robots for different material medicine containers, and reduces the risk of medicine damage.

Description

Technical Field

[0001] This invention relates to the field of control technology, and in particular to a dual-robotic arm collaborative grasping method based on drug traceability codes and multi-source vision fusion. Background Technology

[0002] With the development of the pharmaceutical industry and the advancement of intelligent manufacturing, the demand for automation in pharmaceutical logistics and distribution is increasing. Automated gripping and handling of pharmaceutical containers is a key link in pharmaceutical production lines and smart pharmacies, among which dual-robotic arm collaborative gripping technology has attracted much attention due to its high stability and strong adaptability. Traditional pharmaceutical container gripping technology mainly relies on a single vision system for target recognition and positioning, and executes fixed-pattern gripping actions through preset gripping strategies.

[0003] In recent years, drug traceability codes have been widely used in the entire lifecycle management of drugs as an important means of drug supervision. The information contained in these codes can not only be used for drug traceability but also provide important physical characteristic parameters for intelligent grasping systems. Meanwhile, the development of multi-source vision technology has made it possible to obtain comprehensive geometric information of drug containers. Combining drug traceability code information with multi-source vision perception to achieve precise collaborative grasping by robotic arms is currently a research hotspot in the field of pharmaceutical automation.

[0004] Existing technologies lack the ability to accurately perceive the material properties of pharmaceutical containers. They typically rely on empirical or standardized parameters for gripping control, which is difficult to adapt to the gripping needs of containers made of different materials. This can easily lead to problems such as excessive gripping causing container damage or unstable gripping causing the container to slip.

[0005] Secondly, traditional grasping methods mainly rely on a single vision system to obtain target information. For pharmaceutical containers with complex shapes or smooth surfaces, it is difficult to accurately obtain their complete geometric features, resulting in improper selection of grasping positions and a low grasping success rate, which is especially evident for containers with complex curved surfaces. Summary of the Invention

[0006] This invention provides a dual-robotic arm collaborative grasping method based on drug traceability codes and multi-source vision fusion, which can solve the problems in the prior art.

[0007] A first aspect of this invention provides a dual-robotic arm collaborative grasping method based on drug traceability codes and multi-source visual fusion, comprising:

[0008] The system reads the traceability code information on the medicine container and collects image data of the medicine container from multiple vision sensors; it parses the traceability code information to obtain the material type and surface friction coefficient of the medicine container; it performs depth estimation and semantic segmentation on the image data to generate the surface geometric topology of the medicine container.

[0009] A contact mechanics mapping relationship is established based on the material type and the surface friction coefficient. This relationship describes the proportional constraints of the normal and tangential forces required at different gripping positions. The curvature distribution of each surface region of the drug container is calculated based on the surface geometry. The curvature distribution is then combined with the contact mechanics mapping relationship to determine the set of suitable gripping regions.

[0010] Calculate the envelope fit when the end effectors of the two robotic arms come into contact with each region in the set of adapted gripping regions, and select the pair of regions with the largest sum of envelope fit as the target gripping regions of the two robotic arms.

[0011] The magnitude of the force applied by the two robotic arms is determined based on the contact mechanics mapping relationship corresponding to the target grasping area, and the two robotic arms are driven to move to the target grasping area and perform grasping according to the determined magnitude of the force applied.

[0012] The process of performing depth estimation and semantic segmentation on the image data to generate the surface geometric topology of the drug container includes:

[0013] Depth estimation is performed on the image data to generate a depth image of the medicine container, and the depth image records the distance values ​​of each pixel on the surface of the medicine container relative to the visual sensor;

[0014] The image data is semantically segmented, dividing the pixel regions in the image data into a container body region and a background region, wherein the container body region corresponds to the visible surface of the drug container;

[0015] The depth image is spatially aligned with the main body region of the container, and the depth value distribution within the main body region of the container is extracted. Based on the depth value distribution, a three-dimensional surface point set of the drug container is constructed. The spatial coordinates of each point in the three-dimensional surface point set are jointly determined by the two-dimensional image coordinates of the corresponding pixel and the depth value.

[0016] A topological analysis is performed on the three-dimensional surface point set to identify the connected regions and boundary contours on the surface of the drug container. The connected regions are formed by point clusters whose spatial distance is less than a connectivity threshold. The boundary contours are formed by the sequence of intersection points between the connected regions and the background region. The surface geometric topology of the drug container is generated based on the connected regions and the boundary contours.

[0017] Establishing a contact mechanics mapping relationship based on the material type and the surface friction coefficient includes:

[0018] The elastic modulus of the medicine container is determined based on the material type, and the elastic modulus characterizes the stiffness of the medicine container under stress.

[0019] The anti-slip critical condition is calculated based on the surface friction coefficient. The anti-slip critical condition specifies the maximum tangential force that the surface of the drug container can withstand under a given normal force. The ratio of the maximum tangential force to the normal force is equal to the product of the surface friction coefficient and a preset safety factor.

[0020] A contact mechanics mapping relationship is constructed by combining the elastic modulus and the anti-slip critical condition. The contact mechanics mapping relationship establishes a proportional constraint function of normal force and tangential force for different positions on the surface of the drug container. The proportional constraint function adjusts the anti-slip critical condition with the elastic modulus as a weighting factor.

[0021] The contact mechanics mapping relationship establishes proportional constraint functions for normal and tangential forces at different locations on the surface of the drug container, including:

[0022] The surface of the drug container is divided into multiple discrete location units, and each discrete location unit corresponds to a local region in the surface geometric topology.

[0023] Extract the curvature value and normal vector direction of each discrete position unit in the surface geometry; express the upper limit of the tangential force of each discrete position unit as a function of the product of the normal force, the surface friction coefficient, and the curvature value;

[0024] The proportional constraint function decomposes the force vector applied to the discrete position unit into normal and tangential components through the normal vector direction;

[0025] The proportional constraint functions corresponding to all discrete position units are summarized to form the contact mechanics mapping relationship.

[0026] Calculate the envelope fit when the end effectors of the two robotic arms contact each region in the set of adapted gripping regions, and select the pair of regions with the largest sum of envelope fit as the target gripping regions for the dual robotic arms, including:

[0027] Obtain the gripping configuration parameters of the end effectors of the two robotic arms, wherein the gripping configuration parameters include the opening and closing distance range and the normal distribution of the contact surface of the end effectors;

[0028] A virtual envelope surface is constructed for each region in the set of adaptive grasping regions. The virtual envelope surface is formed by extending the surface of the region outward along its normal vector direction to form a three-dimensional envelope space. The contact surface of the end effector is projected onto the virtual envelope surface under the constraints of the grasping configuration parameters. The area of ​​the overlapping region between the projected contact surface and the virtual envelope surface is calculated. The ratio of the overlapping region area to the total area of ​​the virtual envelope surface is used as the envelope fit degree.

[0029] The target grasping areas of the two robotic arms are required to satisfy a relative positional relationship such that the resultant force applied by the two robotic arms passes through the center of mass of the medicine container. The pairing scheme is traversed through the set of suitable grasping areas that satisfy the relative positional relationship.

[0030] For each region pairing scheme, the sum of the envelope fit of the two regions it contains is calculated, and the pair of regions with the largest sum of envelope fit is selected as the target grasping area of ​​the dual robotic arms.

[0031] Based on the contact mechanics mapping relationship corresponding to the target grasping area, the magnitude of the applied force of the two robotic arms is determined, and the two robotic arms are driven to move to the target grasping area and perform grasping according to the determined applied force magnitude, including:

[0032] The mechanical parameters corresponding to the target grasping area are extracted from the contact mechanics mapping relationship, and the mechanical parameters include the contact angle;

[0033] Based on the aforementioned mechanical parameters, a set of mechanical equilibrium equations for the dual robotic arms is established. Then, based on these equations and the gravitational component of the drug container, the reference values ​​for the tangential forces applied by the two robotic arms are determined.

[0034] The motion trajectory of the two robotic arms from their current positions to the target grasping area is planned. The motion trajectory is synchronized in the time domain so that the end effectors of the two robotic arms arrive at their respective target grasping areas at the same time. During the planning of the motion trajectory, the collision risk between the two robotic arms is detected. When a collision risk is detected, the motion trajectory is adjusted to avoid collisions and maintain time synchronization constraints. The two robotic arms are driven to move along the motion trajectory and perform grasping according to the tangential force reference value.

[0035] Based on the aforementioned mechanical parameters, a set of mechanical equilibrium equations for the two robotic arms is established. The reference values ​​for the tangential forces applied by the two robotic arms are then calculated based on these mechanical equilibrium equations and the gravitational component of the drug container.

[0036] The set of mechanical equilibrium equations includes force equilibrium equations and torque equilibrium equations. The force equilibrium equations constrain the sum of the resultant force of the normal force applied by the two robotic arms through the contact angle to the target grasping area and the normal component of the gravity component to be zero. The torque equilibrium equations constrain the sum of the torques generated by the normal force applied by the two robotic arms through the contact angle to the target grasping area on the center of gravity of the drug container to be zero.

[0037] Solving the mechanical equilibrium equations determines the required tangential force for grasping the medicine container. When the required tangential force does not exceed the preset upper limit of the tangential force, the required tangential force is determined as the reference value of the tangential force. When the required tangential force exceeds the preset upper limit of the tangential force, the upper limit of the tangential force is determined as the reference value of the tangential force.

[0038] A second aspect of the present invention provides a dual-robotic arm collaborative grasping system based on drug traceability codes and multi-source vision fusion, comprising:

[0039] The first unit is used to read the traceability code information on the medicine container and collect image data of the medicine container obtained by multiple vision sensors; parse the traceability code information to obtain the material type and surface friction coefficient of the medicine container; perform depth estimation and semantic segmentation on the image data to generate the surface geometric topology of the medicine container;

[0040] The second unit is used to establish a contact mechanics mapping relationship based on the material type and the surface friction coefficient. The contact mechanics mapping relationship describes the proportional constraint of the normal force and tangential force to be applied at different gripping positions. The curvature distribution of each surface region of the drug container is calculated according to the surface geometric topology. The curvature distribution is combined with the contact mechanics mapping relationship to determine the set of suitable gripping regions.

[0041] The third unit is used to calculate the envelope fit degree when the end effectors of the two robotic arms contact each region in the set of adapted gripping regions, and select the pair of regions with the largest sum of envelope fit degrees as the target gripping regions of the two robotic arms.

[0042] The fourth unit is used to determine the magnitude of the applied force of the two robotic arms according to the contact mechanics mapping relationship corresponding to the target grasping area, drive the two robotic arms to move to the target grasping area and perform grasping according to the determined magnitude of the applied force.

[0043] A third aspect of the present invention,

[0044] An electronic device is provided, comprising:

[0045] processor;

[0046] Memory used to store processor-executable instructions;

[0047] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0048] Fourth aspect of the embodiments of the present invention,

[0049] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0050] The beneficial effects of this application are as follows:

[0051] By reading and parsing the traceability code information on the medicine container to obtain the material type and surface friction coefficient, and combining the surface geometric topology obtained by multi-source visual image data processing, a comprehensive perception of the physical characteristics of the medicine container is achieved, providing basic data support for accurate grasping and solving the technical problem that traditional methods cannot adjust the grasping strategy for medicine containers of different materials.

[0052] By combining the surface curvature distribution with the contact mechanics mapping relationship, a set of suitable gripping areas was determined, which solved the limitation of traditional methods that rely solely on geometric features to select gripping points while ignoring material properties, and improved the gripping adaptability to irregularly shaped pharmaceutical containers.

[0053] By selecting the optimal gripping area pairing based on the envelope fit evaluation index and combining the contact mechanics mapping relationship to precisely control the force applied by the two robotic arms, the coordinated gripping of the two robotic arms was realized, which significantly improved the success rate and stability of gripping various drug containers. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating the collaborative grasping method of dual robotic arms based on drug traceability codes and multi-source vision fusion according to an embodiment of the present invention.

[0055] Figure 2 A schematic diagram of the process for driving two robotic arms to move to the target grasping area. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0058] refer to Figure 1 and Figure 2 The present invention provides a dual-robotic arm collaborative grasping method based on drug traceability codes and multi-source visual fusion, comprising:

[0059] The system reads the traceability code information on the medicine container and collects image data of the medicine container from multiple vision sensors; it parses the traceability code information to obtain the material type and surface friction coefficient of the medicine container; it performs depth estimation and semantic segmentation on the image data to generate the surface geometric topology of the medicine container.

[0060] A contact mechanics mapping relationship is established based on the material type and the surface friction coefficient. This relationship describes the proportional constraints of the normal and tangential forces required at different gripping positions. The curvature distribution of each surface region of the drug container is calculated based on the surface geometry. The curvature distribution is then combined with the contact mechanics mapping relationship to determine the set of suitable gripping regions.

[0061] Calculate the envelope fit when the end effectors of the two robotic arms come into contact with each region in the set of adapted gripping regions, and select the pair of regions with the largest sum of envelope fit as the target gripping regions of the two robotic arms.

[0062] The magnitude of the force applied by the two robotic arms is determined based on the contact mechanics mapping relationship corresponding to the target grasping area, and the two robotic arms are driven to move to the target grasping area and perform grasping according to the determined magnitude of the force applied.

[0063] In one optional implementation, performing depth estimation and semantic segmentation on the image data to generate the surface geometric topology of the drug container includes:

[0064] Depth estimation is performed on the image data to generate a depth image of the medicine container, and the depth image records the distance values ​​of each pixel on the surface of the medicine container relative to the visual sensor;

[0065] The image data is semantically segmented, dividing the pixel regions in the image data into a container body region and a background region, wherein the container body region corresponds to the visible surface of the drug container;

[0066] The depth image is spatially aligned with the main body region of the container, and the depth value distribution within the main body region of the container is extracted. Based on the depth value distribution, a three-dimensional surface point set of the drug container is constructed. The spatial coordinates of each point in the three-dimensional surface point set are jointly determined by the two-dimensional image coordinates of the corresponding pixel and the depth value.

[0067] A topological analysis is performed on the three-dimensional surface point set to identify the connected regions and boundary contours on the surface of the drug container. The connected regions are formed by point clusters whose spatial distance is less than a connectivity threshold. The boundary contours are formed by the sequence of intersection points between the connected regions and the background region. The surface geometric topology of the drug container is generated based on the connected regions and the boundary contours.

[0068] Depth estimation is performed on the acquired image data to generate a depth image of the medicine container. Specifically, multi-view images of the medicine container are acquired using binocular stereo vision or structured light technology, and a depth map is generated through disparity calculation. For binocular vision technology, feature points are matched between images captured by two cameras, and the depth value corresponding to each pixel is calculated based on the pixel offset of the matching points and camera parameters. In practical applications, a local region block matching algorithm can be used to select a matching window of an appropriate size (e.g., 9×9 pixels), calculate the normalized cross-correlation value of pixels within the window to determine the optimal matching position, and then derive the depth value. For depth discontinuities in edge regions, adaptive window technology is used for optimization to improve the accuracy of depth boundaries. The final generated depth image records the distance values ​​of each pixel on the surface of the medicine container relative to the vision sensor, and the depth values ​​are usually expressed in millimeters.

[0069] Semantic segmentation is performed on image data, dividing pixel regions into container bodies and background regions. Semantic segmentation is implemented using a deep convolutional neural network, with an encoder and a decoder. The encoder consists of multiple convolutional and pooling layers to extract multi-scale features from the image; the decoder recovers the spatial resolution of the feature maps through upsampling or deconvolution operations and outputs the class prediction result for each pixel. To improve segmentation accuracy, a residual connection structure is used to fuse the feature maps from the encoding stage with the corresponding feature maps from the decoding stage. During network training, a labeled dataset containing both the medicine container and the background is used, and the cross-entropy loss function guides the network to learn its ability to distinguish between the container and the background. The segmentation results are represented by a binary mask, where regions with a pixel value of 1 correspond to the visible surface of the medicine container, and regions with a pixel value of 0 represent the background.

[0070] The depth image is spatially aligned with the container's main body region. Specifically, depth values ​​within the container's main body region are extracted using a masking operation. For pixels with a value of 1 in the semantically segmented container main body region mask, the depth value at the corresponding location in the depth image is retained, while the depth values ​​at other locations are set to invalid values. To eliminate noise, median filtering is applied to the extracted depth value distribution, with a filtering window size of 5×5 pixels, effectively suppressing interference from abnormal depth values. Subsequently, a 3D surface point set of the drug container is constructed based on the aligned depth value distribution. For each valid pixel (u, v) in the image and its corresponding depth value d, the 3D coordinates (X, Y, Z) of that pixel in the camera coordinate system are calculated based on the camera intrinsic parameter matrix K (containing focal length and optical center coordinates), where Z equals the depth value d, and X and Y are obtained through backprojection of the pixel coordinates. Thus, the spatial coordinates of each 3D point are jointly determined by the corresponding pixel's 2D image coordinates and depth value, forming a 3D point cloud representing the surface of the drug container.

[0071] Topological analysis is performed on a set of 3D surface points to identify connected regions and boundary contours on the surface of pharmaceutical containers. A distance-based clustering method is used to group points with a spatial distance less than a connectivity threshold (set to 2 mm) into the same connected region. Specifically, a region growing algorithm is used, starting from any unprocessed point, recursively adding its surrounding points with a distance less than the connectivity threshold to the current connected region until no new points meeting the criteria can be found. For each connected region, its boundary points are extracted, i.e., the set of points directly adjacent to the background region. The criterion for determining boundary points is that each point has at least one neighboring point belonging to the background region. All boundary points are sorted according to spatial continuity to form a closed boundary contour. To ensure the smoothness of the contour, B-spline interpolation is applied to the extracted boundary point sequence to generate a continuous and smooth curve representation.

[0072] Based on the identified connected regions and boundary contours, the surface geometric topology of the pharmaceutical container is generated. For each connected region, its surface geometric features are calculated, including surface area, curvature distribution, and principal orientation. For regularly shaped pharmaceutical containers, such as cylindrical bottles, their axial and radial directions are determined through principal component analysis, and an accurate geometric model is fitted. For irregularly shaped containers, the point cloud representation is preserved, and a surface model is generated by reconstructing a triangular mesh. By integrating the geometric features of the connected regions and the boundary contour information, a complete surface geometric topology of the pharmaceutical container is finally constructed, which accurately reflects the shape characteristics and surface undulations of the container.

[0073] In practical applications, the geometric topology of container surfaces can be used for subsequent tasks such as defect detection, label identification, and capacity estimation. For example, by analyzing abnormal changes in surface curvature, dents or deformations on the container surface can be detected; by analyzing the shape of the boundary contour, different types of pharmaceutical containers can be distinguished; and by calculating the surface area, the specifications and capacity of the container can be estimated.

[0074] In one optional implementation, establishing a contact mechanics mapping relationship based on the material type and the surface friction coefficient includes:

[0075] The elastic modulus of the medicine container is determined based on the material type, and the elastic modulus characterizes the stiffness of the medicine container under stress.

[0076] The anti-slip critical condition is calculated based on the surface friction coefficient. The anti-slip critical condition specifies the maximum tangential force that the surface of the drug container can withstand under a given normal force. The ratio of the maximum tangential force to the normal force is equal to the product of the surface friction coefficient and a preset safety factor.

[0077] A contact mechanics mapping relationship is constructed by combining the elastic modulus and the anti-slip critical condition. The contact mechanics mapping relationship establishes a proportional constraint function of normal force and tangential force for different positions on the surface of the drug container. The proportional constraint function adjusts the anti-slip critical condition with the elastic modulus as a weighting factor.

[0078] In the automated handling of pharmaceutical containers, accurately establishing the contact force mapping relationship is crucial for the robotic arm to grasp and manipulate the containers. This mapping relationship guides the robotic arm to apply appropriate force, ensuring a firm grip without damaging the container.

[0079] The elastic modulus of a medicine container is determined by its material type. The elastic modulus is a physical quantity that characterizes the stiffness of a medicine container under stress; different materials have different elastic modulus values. In practical applications, common medicine container materials include glass, plastic, and metal. For example, the elastic modulus of a medicine bottle made of borosilicate glass is approximately 60 to 70 gigapascals; for a polyethylene plastic container, the elastic modulus is approximately 0.8 to 1.2 gigapascals; while the elastic modulus of a stainless steel container can reach around 200 gigapascals.

[0080] Material type identification can be accomplished through spectral analysis or a pre-entered database. In one embodiment, after the material is determined, the corresponding elastic modulus parameter is retrieved from a material physical property database.

[0081] Secondly, the critical condition for anti-slip is calculated based on the surface friction coefficient. The surface friction coefficient can be obtained through experimental measurement, reflecting the frictional characteristics between the surface of the medicine container and the contact surface of the gripping device. During the measurement process, the inclined plane method can be used, where the medicine container is placed on an adjustable-angle plane, and the tilt angle is gradually increased until the container begins to slide. The friction coefficient is then calculated by measuring this critical angle.

[0082] The calculation of the anti-slip critical condition is based on classical tribology theory, which defines the maximum tangential force that the surface of a pharmaceutical container can withstand under a given normal force. This critical condition can be expressed as: the ratio of the maximum tangential force to the normal force is equal to the product of the surface friction coefficient and the preset safety factor. The preset safety factor is typically taken in the range of 0.6 to 0.8 to provide an additional safety margin to prevent slippage caused by external vibration or disturbance during actual operation.

[0083] For example, assuming the coefficient of friction between the glass medicine bottle and the silicone gripping device is 0.4, and a safety factor of 0.7 is selected, the critical condition for anti-slip can be determined as follows: the maximum tangential force should not exceed 28% of the normal force. This means that in the actual gripping process, if the normal force is 10 Newtons, the tangential force should be controlled below 2.8 Newtons to ensure stable gripping without slippage.

[0084] Third, a contact mechanics mapping relationship is constructed by combining the elastic modulus and the anti-slip critical condition. This mapping relationship establishes a proportional constraint function for the normal force and tangential force at different locations on the surface of the drug container. This function uses the elastic modulus as a weighting factor to adjust the anti-slip critical condition, thereby forming a more accurate mechanical control model.

[0085] When constructing the mapping relationship, the surface of the drug container is first divided into several contact areas, each of which may have different geometric features and stress characteristics. For each contact area, based on the curvature, location, and other characteristics of that area, combined with the material's elastic modulus, the degree of deformation under a given normal force is calculated. The greater the degree of deformation, the larger the actual contact area, and the greater the frictional force; therefore, the basic friction coefficient needs to be corrected.

[0086] Specifically, for materials with a low modulus of elasticity (such as some soft plastics), a larger deformation occurs under the same normal force, increasing the actual contact area and thus improving the equivalent coefficient of friction. In this case, the safety factor in the anti-slip critical condition can be appropriately increased. Conversely, for materials with a high modulus of elasticity (such as glass or metals), the deformation is smaller, and the actual contact area is close to the theoretical contact area, so a coefficient of friction closer to the measured value can be used.

[0087] In one embodiment, the proportional constraint function can be constructed by first calculating the basic anti-slip ratio (the product of the friction coefficient and the safety factor), and then introducing a correction factor based on the elastic modulus. The correction factor decreases as the elastic modulus increases, reflecting the need for stricter control of tangential forces in rigid materials. For example, for plastic containers (low elastic modulus), the correction factor might be 1.2; for glass containers (medium elastic modulus), it might be 1.0; and for metal containers (high elastic modulus), it might be 0.85.

[0088] The contact mechanics mapping relationship constructed in this way can provide customized gripping force control strategies for medicine containers of different materials and shapes, enabling safe and stable automated operation. This mapping relationship can be integrated into the robot control system to adjust the gripping force of the robotic arm in real time, preventing damage or slippage of the medicine containers.

[0089] During application, the mechanical system first identifies the material of the medicine container, queries the corresponding elastic modulus, measures or queries the surface friction coefficient, then calculates the appropriate gripping force range based on the constructed mapping relationship, and finally executes precisely controlled gripping actions to ensure the safety and reliability of the medicine container handling process.

[0090] In one optional implementation, the contact mechanics mapping relationship establishes a proportional constraint function between normal and tangential forces at different locations on the surface of the drug container, including:

[0091] The surface of the drug container is divided into multiple discrete location units, and each discrete location unit corresponds to a local region in the surface geometric topology.

[0092] Extract the curvature value and normal vector direction of each discrete position unit in the surface geometry; express the upper limit of the tangential force of each discrete position unit as a function of the product of the normal force, the surface friction coefficient, and the curvature value;

[0093] The proportional constraint function decomposes the force vector applied to the discrete position unit into normal and tangential components through the normal vector direction;

[0094] The proportional constraint functions corresponding to all discrete position units are summarized to form the contact mechanics mapping relationship.

[0095] In establishing the contact mechanics mapping relationship on the surface of a pharmaceutical container, the surface first needs to be discretized, dividing the continuous surface into a finite number of discrete positional units. For this purpose, a mesh generation algorithm, such as a quadrilateral mesh or a triangular mesh, can be used, with the appropriate mesh density determined based on the complexity of the container surface. For areas with significant curvature changes, such as the neck or bottom transition region of the container, the mesh density can be appropriately increased to improve computational accuracy.

[0096] After the surface of the pharmaceutical container is segmented, local geometric features are extracted for each discrete location unit. Specifically, the curvature value of each location unit is calculated using differential geometry methods. For any surface point, the curvature characteristics of that point can be characterized by calculating its principal curvatures κ1 and κ2. The principal curvatures can be calculated based on the second derivative information of the surface in the neighborhood of that point, by constructing a local coordinate system and solving the eigenvalue problem of the surface equation. For axisymmetric objects like pharmaceutical containers, the surface equation can be expressed in cylindrical coordinates, simplifying the curvature calculation process.

[0097] Simultaneously, determining the direction of the normal vector for each discrete location element is crucial. The normal vector is typically defined as a unit vector perpendicular to the surface. For parameterized surfaces, it can be obtained by the cross product of the partial derivative vectors of the surface parameterization equations and then normalized. In practical calculations, discrete differential geometry methods can be used to approximate the calculation of the normal vector through the positional relationships of adjacent mesh elements.

[0098] Based on the obtained curvature values ​​and normal vector directions, a constraint relationship between the tangential and normal forces is established for each discrete position element. According to tribological theory, when an object is in a state of static friction, there is a proportional relationship between the tangential and normal forces, meaning the frictional force does not exceed the maximum static frictional force. Considering the influence of surface curvature, the upper limit of the tangential force can be expressed as:

[0099] For discrete position element i, the relationship between its upper limit of tangential force F_ti_max and normal force F_ni can be expressed as follows: the upper limit of tangential force equals the normal force multiplied by the friction coefficient and the curvature correction factor. The curvature correction factor can be designed as a function of curvature κ to reflect the influence of curvature on the contact force distribution. For convex curvature regions, the upper limit of tangential force will decrease accordingly; for concave curvature regions, the upper limit of tangential force may increase.

[0100] In practical applications, when an external force is applied to the surface of a container, the force vector needs to be decomposed into normal and tangential components. For a discrete position element i, if its normal vector is n_i and the external force is F_i, then the normal component F_ni can be expressed as the dot product of F_i and n_i multiplied by n_i; the tangential component F_ti is the difference between F_i and the normal component.

[0101] To ensure contact stability, the tangential force on each discrete position element must not exceed its upper limit, i.e., |F_ti| ≤ F_ti_max. This constraint constitutes the core of the proportional constraint function.

[0102] In practical engineering applications, the aforementioned constraint functions can be used for planning the gripping operation of medicine containers. For example, for a cylindrical glass bottle, its surface can be divided into multiple discrete positional units. The curvature of the straight cylindrical part of the bottle is relatively small, while the curvature of the transition area between the neck and the bottom is larger. Based on the established mapping relationship, when the robot grips the bottle, it will apply a larger clamping force to the straight cylindrical part of the bottle, while reducing the clamping force in the transition area with larger curvature, in order to avoid damage to the container caused by local stress concentration.

[0103] By summing the proportional constraint functions corresponding to all discrete position units, a contact mechanics mapping relationship covering the entire surface of the medicine container is formed. This mapping relationship can be represented as a set of functions or a constraint matrix, describing the force decomposition and constraint conditions at each point on the container surface. In practical applications, this mapping relationship can be integrated into the robot control system to guide the robotic arm in rationally distributing gripping forces when operating the medicine container, ensuring operational safety and stability.

[0104] By establishing this precise contact mechanics mapping relationship, damage to pharmaceutical containers caused by improper contact forces on automated production lines can be effectively avoided, improving production efficiency and product integrity. This method is particularly advantageous for fragile glass containers or complex-shaped special containers, as it can adaptively adjust the contact force distribution based on the local geometry of the container surface.

[0105] Once a complete contact mechanics mapping relationship is established, it can be applied to subsequent operation control of pharmaceutical containers. When operations such as gripping, rotating, or moving the container are required, the control system can calculate the magnitude and direction of the force required at each contact point based on this mapping relationship, ensuring that the container is not damaged during operation, while also guaranteeing the stability and reliability of the operation.

[0106] In one optional implementation, calculating the envelope fit when the end effectors of the two robotic arms contact each region in the set of adapted gripping regions, and selecting the pair of regions with the largest sum of envelope fit as the target gripping regions for the dual robotic arms includes:

[0107] Obtain the gripping configuration parameters of the end effectors of the two robotic arms, wherein the gripping configuration parameters include the opening and closing distance range and the normal distribution of the contact surface of the end effectors;

[0108] A virtual envelope surface is constructed for each region in the set of adaptive grasping regions. The virtual envelope surface is formed by extending the surface of the region outward along its normal vector direction to form a three-dimensional envelope space. The contact surface of the end effector is projected onto the virtual envelope surface under the constraints of the grasping configuration parameters. The area of ​​the overlapping region between the projected contact surface and the virtual envelope surface is calculated. The ratio of the overlapping region area to the total area of ​​the virtual envelope surface is used as the envelope fit degree.

[0109] The target grasping areas of the two robotic arms are required to satisfy a relative positional relationship such that the resultant force applied by the two robotic arms passes through the center of mass of the medicine container. The pairing scheme is traversed through the set of suitable grasping areas that satisfy the relative positional relationship.

[0110] For each region pairing scheme, the sum of the envelope fit of the two regions it contains is calculated, and the pair of regions with the largest sum of envelope fit is selected as the target grasping area of ​​the dual robotic arms.

[0111] In dual-arm gripping operations, selecting the most suitable gripping area is a crucial issue. This solution proposes a gripping area selection method based on envelope fit, which selects the optimal gripping position by calculating the degree of matching between the robotic arm end effector and the gripping area.

[0112] First, the gripping configuration parameters of the end effectors of the two robotic arms are obtained. These parameters describe the physical characteristics and operational capabilities of the end effectors. The gripping configuration parameters mainly include the opening and closing distance range and the normal distribution of the contact surface. The opening and closing distance range defines the maximum distance the actuator can open and the minimum distance it can close; for example, for a parallel gripper, the possible opening and closing distance range is 0 to 10 centimeters. The normal distribution of the contact surface describes the orientation characteristics of the actuator's contact surface, usually represented as a set of unit normal vectors. These normal vectors collectively define the direction in which the actuator can effectively contact the target object.

[0113] For each region in the set of adaptive grasping regions, a virtual envelope surface needs to be constructed to evaluate its compatibility with the robotic arm's end effector. The virtual envelope surface is a three-dimensional envelope space formed by extending the region's surface outwards along its normal vector direction. The specific construction process is as follows: First, discrete point cloud data of the region's surface is acquired, and the normal vector of each point is calculated; then, each point is extended outwards along its normal vector direction by a predetermined distance (e.g., 5 mm), generating a new set of points; finally, these points are connected into a closed virtual envelope surface using triangulation or mesh reconstruction algorithms.

[0114] When calculating envelope fit, the contact surface of the end effector is projected onto a virtual envelope surface under the constraints of the gripping configuration parameters. Specifically, points on the actuator contact surface are transformed into the coordinate system of the virtual envelope surface according to the current gripping posture, and then the intersection area between the projected points and the virtual envelope surface is calculated. The overlapping area refers to the area of ​​the portion where the actuator contact surface projection intersects with the virtual envelope surface, which can be obtained through geometric calculations or discrete sampling methods. Envelope fit is defined as the ratio of the overlapping area to the total area of ​​the virtual envelope surface; this value ranges from 0 to 1, with a higher value indicating a higher degree of fit.

[0115] To ensure gripping stability, the target gripping areas of the two robotic arms must satisfy a specific relative positional relationship, such that the resultant force applied by the two robotic arms passes through the center of mass of the medicine container. This requires first determining the position of the center of mass of the medicine container, which can be obtained through calculations using the geometric center or mass distribution of the 3D model. Then, the pairing of regions in the adaptive gripping area set is traversed, and each pair of regions is checked to see if it satisfies the force balance condition: the line connecting the two regions must pass through or be close to the center of mass of the container, and the normal vectors of the two regions must be approximately opposite or able to form a stable gripping force.

[0116] For each pairing scheme that satisfies the relative positional relationship, the sum of their envelope fit is calculated. For example, if the envelope fit of region A with the first robotic arm is 0.75, and the envelope fit of region B with the second robotic arm is 0.80, then the total envelope fit of this pairing scheme is 1.55. By comparing the total envelope fit of all valid pairing schemes, the pair of regions with the largest sum of fit is selected as the target grasping area for the dual robotic arms.

[0117] In practical applications, the kinematic constraints and accessibility of the robotic arm may also need to be considered. Therefore, before finalizing the target grasping area, accessibility verification can be performed: for each candidate region pairing, the trajectory of the robotic arm from its current position to the target position is calculated, and a valid, collision-free path is checked. Only region pairings that satisfy the kinematic constraints and can be safely reached are considered valid solutions.

[0118] This method demonstrates good stability and adaptability in practical applications. For example, in an automated pharmaceutical sorting system, dual robotic arms need to grasp pharmaceutical containers of various shapes. By calculating the envelope fit and selecting the optimal grasping area, the system can adaptively grasp containers of different shapes, such as cylindrical medicine bottles and rectangular medicine boxes, significantly improving the grasping success rate.

[0119] In summary, the grasping region selection method based on envelope fit provides an efficient decision-making strategy for collaborative grasping tasks of dual robotic arms by quantifying the matching degree between the end effector and the grasping region and combining it with the principle of mechanical balance. This method can significantly improve the stability and success rate of robotic arm operations.

[0120] In one optional implementation, determining the magnitude of the applied force of the two robotic arms based on the contact mechanics mapping relationship corresponding to the target grasping area, and driving the two robotic arms to move to the target grasping area and perform grasping according to the determined applied force magnitude includes:

[0121] The mechanical parameters corresponding to the target grasping area are extracted from the contact mechanics mapping relationship, and the mechanical parameters include the contact angle;

[0122] Based on the aforementioned mechanical parameters, a set of mechanical equilibrium equations for the dual robotic arms is established. Then, based on these equations and the gravitational component of the drug container, the reference values ​​for the tangential forces applied by the two robotic arms are determined.

[0123] The motion trajectory of the two robotic arms from their current positions to the target grasping area is planned. The motion trajectory is synchronized in the time domain so that the end effectors of the two robotic arms arrive at their respective target grasping areas at the same time. During the planning of the motion trajectory, the collision risk between the two robotic arms is detected. When a collision risk is detected, the motion trajectory is adjusted to avoid collisions and maintain time synchronization constraints. The two robotic arms are driven to move along the motion trajectory and perform grasping according to the tangential force reference value.

[0124] The mechanical parameters corresponding to the target grasping area are extracted from the contact mechanics mapping relationship. Based on a pre-established mesh model of the drug container surface, the surface is discretized into multiple candidate grasping areas, and each area stores its corresponding mechanical parameter information. These mechanical parameters mainly include the contact angle, which is the angle between the end effector of the robotic arm and the tangent direction of the drug container surface. The contact angle can be obtained by calculating the relationship between the container surface normal vector and the orientation vector of the end effector of the robotic arm. For example, for a cylindrical medicine bottle, when the contact point is located on the side of the bottle, the contact angle is approximately 90 degrees; if the contact point is on the bottom or cap of the bottle, the contact angle may be close to 0 degrees or 180 degrees. In addition, the mechanical parameters also include the friction coefficient corresponding to the container surface material, the local curvature of the container, etc., which together constitute a complete contact mechanics mapping relationship.

[0125] Based on the extracted mechanical parameters, a set of mechanical equilibrium equations for the dual robotic arms is established. Considering that the medicine container is subjected to gravity in the vertical direction, and the dual robotic arms need to provide sufficient clamping force in the horizontal plane to maintain the stability of the container, the force equilibrium equations are as follows: For the first robotic arm, the normal force and tangential force it applies need to be balanced with the force applied by the second robotic arm and the weight of the container. Specifically, let the normal force applied by the first robotic arm be F1n, and the tangential force be F1t; let the normal force applied by the second robotic arm be F2n, and the tangential force be F2t; and let the weight of the container be G. According to the principle of mechanical equilibrium, F1n needs to be balanced with F2n in the horizontal direction, and the resultant force of F1t and F2t in the vertical direction needs to balance the weight of the container G. At the same time, considering the influence of the contact angle, the actual direction of the applied force will change. Therefore, the contact angles θ1 and θ2 need to be introduced as variables in the equations to decompose the components of each force in different directions.

[0126] Based on the established set of mechanical equilibrium equations and the gravitational component of the medicine container, the reference values ​​for the tangential force applied by the two robotic arms are determined. The gravitational component is calculated based on the container's mass and gravitational acceleration. By solving the equations, the required tangential force from the two robotic arms can be obtained, ensuring that the container will not slip or be damaged by excessive pressure during the grasping process. The reference values ​​for the tangential force will vary depending on the mass and shape of the medicine container. For example, a larger tangential force may be required for heavier glass medicine bottles, while a smaller force is needed for lightweight plastic containers to avoid deformation due to compression.

[0127] Next, the motion trajectories of the two robotic arms from their current positions to the target grasping area are planned. The trajectory planning employs a fifth-order polynomial interpolation method to ensure smooth velocity and acceleration characteristics for both arms. To ensure that the end effectors of both robotic arms arrive at their respective target grasping areas simultaneously, synchronization and coordination are performed in the time domain. Specifically, the motion time of the two robotic arms is normalized to ensure that they complete their respective path percentages in the same amount of time. For example, when the first robotic arm completes 50% of its path, the second robotic arm also completes exactly 50% of its path, thus ensuring that both robotic arms arrive at the target position synchronously and avoiding grasping failure caused by one robotic arm arriving too early while the other is not yet in position.

[0128] During trajectory planning, collision risk between the two robotic arms is detected in real time. A collision detection model for the robotic arms is constructed using a spherical hierarchical bounding box algorithm. The robotic arm configuration at each time point on the trajectory is predicted through forward kinematics calculations, and the minimum distance between the two robotic arms is calculated. When the detected minimum distance is less than a preset safety threshold, a collision risk is considered to exist. The collision risk detection frequency is set to 100Hz to ensure timely detection of potential collisions.

[0129] When a collision risk is detected, the motion trajectory is adjusted for collision avoidance while maintaining time synchronization constraints. Collision avoidance adjustment employs an artificial potential field method, adding a repulsive force to the original trajectory to guide the robotic arm away from the collision area. The adjusted trajectory requires re-synchronization to ensure both robotic arms reach the target position simultaneously. During collision avoidance adjustment, the overall shape of the trajectory remains unchanged, with only minor adjustments made locally to maintain motion stability.

[0130] Finally, the two robotic arms are driven to move along the planned motion trajectory and perform a grasping operation according to the calculated tangential force reference value. During the grasping process, the actual contact force is monitored in real time by force sensors, and the position and force of the end effector of the robotic arm are adjusted through an impedance control algorithm to ensure that the actual applied force is consistent with the calculated reference value. After the grasping is completed, the robotic arms stably lift the medicine container and move it to the designated position according to the subsequent task requirements.

[0131] The above method enables a dual-arm robotic system to collaboratively grasp medicine containers based on contact mechanics mapping, ensuring the stability and safety of the grasping process while avoiding collisions between the robotic arms, thus improving the success rate and operational efficiency of grasping medicine containers.

[0132] In one optional implementation, a set of mechanical equilibrium equations for the two robotic arms is established based on the mechanical parameters. The tangential force reference values ​​for the two robotic arms are then calculated based on the set of mechanical equilibrium equations and the gravitational component of the drug container, including:

[0133] The set of mechanical equilibrium equations includes force equilibrium equations and torque equilibrium equations. The force equilibrium equations constrain the sum of the resultant force of the normal force applied by the two robotic arms through the contact angle to the target grasping area and the normal component of the gravity component to be zero. The torque equilibrium equations constrain the sum of the torques generated by the normal force applied by the two robotic arms through the contact angle to the target grasping area on the center of gravity of the drug container to be zero.

[0134] Solving the mechanical equilibrium equations determines the required tangential force for grasping the medicine container. When the required tangential force does not exceed the preset upper limit of the tangential force, the required tangential force is determined as the reference value of the tangential force. When the required tangential force exceeds the preset upper limit of the tangential force, the upper limit of the tangential force is determined as the reference value of the tangential force.

[0135] In the application of dual robotic arms to grasp medicine containers, in order to ensure the stability and safety of the grasping operation, it is necessary to establish a set of mechanical equilibrium equations and solve for the reference value of tangential force based on the gravitational component of the medicine container.

[0136] First, the mechanical parameters of the dual robotic arms are obtained, including the contact angles between the two arms and the medicine container, the target grasping area, and parameters such as the weight and center of gravity of the medicine container. The contact angle directly affects the direction and effect of the force applied by the robotic arms, and is usually obtained through a vision system or preset configuration. The target grasping area refers to the area where the robotic arms actually contact the medicine container, which can be determined through 3D modeling or sensor measurement.

[0137] Based on the acquired mechanical parameters, a set of mechanical equilibrium equations was established for the dual robotic arms to grasp a medicine container. This set of equations consists of two parts: force equilibrium equations and torque equilibrium equations, used to ensure that the medicine container is in static equilibrium during the grasping process.

[0138] The force balance equation is established based on Newton's first law, constraining the sum of the resultant force of the normal forces applied by the two robotic arms through the contact angle in the target grasping area and the normal component of gravity to be zero. Specifically, assuming that the two robotic arms apply normal forces F1 and F2 at the contact point respectively, and the weight of the medicine container is G, the normal force balance equation can be expressed as:

[0139] F1 cosθ1+ F2 cosθ2 + Gsinα = 0;

[0140] Where θ1 and θ2 are the angles between the force applied by the two robotic arms and the normal, respectively, and α is the angle between the direction of gravity and the horizontal plane. When the medicine container is placed vertically, α is 90 degrees, and gravity is distributed entirely along the normal.

[0141] Simultaneously, a torque balance equation is established to constrain the sum of the torques generated by the normal forces applied by the two robotic arms through the contact angle on the center of gravity of the medicine container to be zero. Assuming the distances from the contact points of the two robotic arms to the center of gravity of the medicine container are r1 and r2 respectively, and the angles between the lines connecting the contact points to the center of gravity and the normal forces are β1 and β2 respectively, the torque balance equation can be expressed as:

[0142] F1·r1·sinβ1 + F2·r2·sinβ2 = 0;

[0143] By solving the force balance equation and the moment balance equation simultaneously, the normal forces F1 and F2 that the two robotic arms need to apply can be obtained.

[0144] Based on the determined normal force, calculate the tangential force to satisfy the frictional requirements. According to Coulomb's law of friction, to prevent the medicine container from slipping during grasping, the required tangential force should satisfy:

[0145] Ft ≥ μ·Fn;

[0146] Where Ft is the tangential force, Fn is the normal force, and μ is the friction coefficient between the contact surfaces. By solving this inequality, the required value of the tangential force can be obtained.

[0147] When the calculated tangential force requirement does not exceed the preset tangential force upper limit, this requirement value is directly determined as the tangential force reference value. The preset tangential force upper limit is usually determined by the working parameters and safety factors of the robotic arm, and can be comprehensively determined based on the maximum load capacity of the robotic arm, the material characteristics of the medicine container, and the safety factor.

[0148] When the calculated tangential force requirement exceeds the preset tangential force upper limit, the tangential force upper limit is determined as the tangential force benchmark value. In this case, it is necessary to adjust the gripping strategy, such as changing the contact angle, increasing the number of contact points, or selecting a clamping material with a higher coefficient of friction, to ensure that the drug container can still be firmly gripped without exceeding the upper limit of the tangential force.

[0149] In practical applications, force sensors can monitor the force applied by the robotic arm in real time and compare it with the calculated tangential force reference value, so as to dynamically adjust the control parameters of the robotic arm and ensure the stability and safety of the grasping process.

[0150] The above method allows for the scientific calculation of the required tangential force benchmark for gripping medicine containers using dual robotic arms, based on the physical characteristics and position of the containers. This effectively improves the success rate of gripping and operational safety, while preventing deformation or damage to the containers due to excessive force. This method is particularly suitable for gripping precision medicine containers, such as the handling and assembly of fragile containers like glass bottles and plastic ampoules.

[0151] This invention relates to a dual-robotic arm collaborative grasping system based on drug traceability codes and multi-source vision fusion, comprising:

[0152] The first unit is used to read the traceability code information on the medicine container and collect image data of the medicine container obtained by multiple vision sensors; parse the traceability code information to obtain the material type and surface friction coefficient of the medicine container; perform depth estimation and semantic segmentation on the image data to generate the surface geometric topology of the medicine container;

[0153] The second unit is used to establish a contact mechanics mapping relationship based on the material type and the surface friction coefficient. The contact mechanics mapping relationship describes the proportional constraint of the normal force and tangential force to be applied at different gripping positions. The curvature distribution of each surface region of the drug container is calculated according to the surface geometric topology. The curvature distribution is combined with the contact mechanics mapping relationship to determine the set of suitable gripping regions.

[0154] The third unit is used to calculate the envelope fit degree when the end effectors of the two robotic arms contact each region in the set of adapted gripping regions, and select the pair of regions with the largest sum of envelope fit degrees as the target gripping regions of the two robotic arms.

[0155] The fourth unit is used to determine the magnitude of the applied force of the two robotic arms according to the contact mechanics mapping relationship corresponding to the target grasping area, drive the two robotic arms to move to the target grasping area and perform grasping according to the determined magnitude of the applied force.

[0156] A third aspect of the present invention provides an electronic device, comprising:

[0157] processor;

[0158] Memory used to store processor-executable instructions;

[0159] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0160] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0161] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dual-robotic arm collaborative grasping method based on drug traceability codes and multi-source vision fusion, characterized in that, include: The system reads the traceability code information on the medicine container and collects image data of the medicine container from multiple visual sensors. The traceability code information is analyzed to obtain the material type and surface friction coefficient of the drug container; Depth estimation and semantic segmentation are performed on the image data to generate the surface geometric topology of the drug container; Establishing a contact mechanics mapping relationship based on the material type and the surface friction coefficient includes: The elastic modulus of the medicine container is determined based on the material type, and the elastic modulus characterizes the stiffness of the medicine container under stress. The anti-slip critical condition is calculated based on the surface friction coefficient. The anti-slip critical condition specifies the maximum tangential force that the surface of the drug container can withstand under a given normal force. The ratio of the maximum tangential force to the normal force is equal to the product of the surface friction coefficient and a preset safety factor. A contact mechanics mapping relationship is constructed by combining the elastic modulus and the anti-slip critical condition. This mapping relationship establishes a proportional constraint function for the normal and tangential forces at different locations on the surface of the drug container. The proportional constraint function uses the elastic modulus as a weighting factor to adjust the anti-slip critical condition. The contact mechanics mapping relationship describes the proportional constraints of the normal and tangential forces required at different gripping positions. The curvature distribution of each surface region of the drug container is calculated based on the surface geometry and topology. The curvature distribution is then combined with the contact mechanics mapping relationship to determine a set of suitable gripping regions. Calculate the envelope fit when the end effectors of the two robotic arms contact each region in the set of adapted gripping regions, and select the pair of regions with the largest sum of envelope fit as the target gripping regions for the dual robotic arms, including: Obtain the gripping configuration parameters of the end effectors of the two robotic arms, wherein the gripping configuration parameters include the opening and closing distance range and the normal distribution of the contact surface of the end effectors; A virtual envelope surface is constructed for each region in the set of adaptive grasping regions. The virtual envelope surface is formed by extending the surface of the region outward along its normal vector direction to form a three-dimensional envelope space. The contact surface of the end effector is projected onto the virtual envelope surface under the constraints of the grasping configuration parameters. The area of ​​the overlapping region between the projected contact surface and the virtual envelope surface is calculated. The ratio of the overlapping region area to the total area of ​​the virtual envelope surface is used as the envelope fit degree. The target grasping areas of the two robotic arms are required to satisfy a relative positional relationship such that the resultant force applied by the two robotic arms passes through the center of mass of the medicine container. The pairing scheme is traversed through the set of suitable grasping areas that satisfy the relative positional relationship. For each region pairing scheme, the sum of the envelope fit of the two regions it contains is calculated, and the pair of regions with the largest sum of envelope fit is selected as the target grasping area of ​​the dual robotic arms. The magnitude of the force applied by the two robotic arms is determined based on the contact mechanics mapping relationship corresponding to the target grasping area, and the two robotic arms are driven to move to the target grasping area and perform grasping according to the determined magnitude of the force applied.

2. The method according to claim 1, characterized in that, The process of performing depth estimation and semantic segmentation on the image data to generate the surface geometric topology of the drug container includes: Depth estimation is performed on the image data to generate a depth image of the medicine container, and the depth image records the distance values ​​of each pixel on the surface of the medicine container relative to the visual sensor; The image data is semantically segmented, dividing the pixel regions in the image data into a container body region and a background region, wherein the container body region corresponds to the visible surface of the drug container; The depth image is spatially aligned with the main body region of the container, and the depth value distribution within the main body region of the container is extracted. Based on the depth value distribution, a three-dimensional surface point set of the drug container is constructed. The spatial coordinates of each point in the three-dimensional surface point set are jointly determined by the two-dimensional image coordinates of the corresponding pixel and the depth value. A topological analysis is performed on the three-dimensional surface point set to identify the connected regions and boundary contours on the surface of the drug container. The connected regions are formed by point clusters whose spatial distance is less than a connectivity threshold. The boundary contours are formed by the sequence of intersection points between the connected regions and the background region. The surface geometric topology of the drug container is generated based on the connected regions and the boundary contours.

3. The method according to claim 1, characterized in that, The contact mechanics mapping relationship establishes proportional constraint functions for normal and tangential forces at different locations on the surface of the drug container, including: The surface of the drug container is divided into multiple discrete location units, and each discrete location unit corresponds to a local region in the surface geometric topology. Extract the curvature value and normal vector direction of each discrete location unit in the surface geometry; The upper limit of the tangential force for each discrete position unit is expressed as a function of the product of the normal force, the surface friction coefficient, and the curvature value. The proportional constraint function decomposes the force vector applied to the discrete position unit into normal and tangential components through the normal vector direction; The proportional constraint functions corresponding to all discrete position units are summarized to form the contact mechanics mapping relationship.

4. The method according to claim 1, characterized in that, Based on the contact mechanics mapping relationship corresponding to the target grasping area, the magnitude of the applied force of the two robotic arms is determined, and the two robotic arms are driven to move to the target grasping area and perform grasping according to the determined applied force magnitude, including: The mechanical parameters corresponding to the target grasping area are extracted from the contact mechanics mapping relationship, and the mechanical parameters include the contact angle; Based on the aforementioned mechanical parameters, a set of mechanical equilibrium equations for the dual robotic arms is established. Then, based on these equations and the gravitational component of the drug container, the reference values ​​for the tangential forces applied by the two robotic arms are determined. The motion trajectory of the two robotic arms from their current positions to the target grasping area is planned. The motion trajectory is synchronized in the time domain so that the end effectors of the two robotic arms arrive at their respective target grasping areas at the same time. During the planning of the motion trajectory, the collision risk between the two robotic arms is detected. When a collision risk is detected, the motion trajectory is adjusted to avoid collisions and maintain time synchronization constraints. The two robotic arms are driven to move along the motion trajectory and perform grasping according to the tangential force reference value.

5. The method according to claim 4, characterized in that, Based on the aforementioned mechanical parameters, a set of mechanical equilibrium equations for the two robotic arms is established. The reference values ​​for the tangential forces applied by the two robotic arms are then calculated based on these mechanical equilibrium equations and the gravitational component of the drug container. The set of mechanical equilibrium equations includes force equilibrium equations and torque equilibrium equations. The force equilibrium equations constrain the sum of the resultant force of the normal force applied by the two robotic arms through the contact angle to the target grasping area and the normal component of the gravity component to be zero. The torque equilibrium equations constrain the sum of the torques generated by the normal force applied by the two robotic arms through the contact angle to the target grasping area on the center of gravity of the drug container to be zero. Solving the mechanical equilibrium equations determines the required tangential force for grasping the medicine container. When the required tangential force does not exceed the preset upper limit of the tangential force, the required tangential force is determined as the reference value of the tangential force. When the required tangential force exceeds the preset upper limit of the tangential force, the upper limit of the tangential force is determined as the reference value of the tangential force.

6. A dual-robotic arm collaborative grasping system based on drug traceability codes and multi-source vision fusion, used to implement the method as described in any one of claims 1-5, characterized in that, include: The first unit is used to read the traceability code information on the medicine container and collect image data of the medicine container obtained by multiple vision sensors; The traceability code information is analyzed to obtain the material type and surface friction coefficient of the drug container; Depth estimation and semantic segmentation are performed on the image data to generate the surface geometric topology of the drug container; The second unit is used to establish a contact mechanics mapping relationship based on the material type and the surface friction coefficient. The contact mechanics mapping relationship describes the proportional constraint of the normal force and tangential force to be applied at different gripping positions. The curvature distribution of each surface region of the drug container is calculated according to the surface geometric topology. The curvature distribution is combined with the contact mechanics mapping relationship to determine the set of suitable gripping regions. The third unit is used to calculate the envelope fit degree when the end effectors of the two robotic arms contact each region in the set of adapted gripping regions, and select the pair of regions with the largest sum of envelope fit degrees as the target gripping regions of the two robotic arms. The fourth unit is used to determine the magnitude of the applied force of the two robotic arms according to the contact mechanics mapping relationship corresponding to the target grasping area, drive the two robotic arms to move to the target grasping area and perform grasping according to the determined magnitude of the applied force.

7. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images