Self-adaptive medicine grabbing system and method based on multi-mode perception
By using a multimodal perception-based adaptive drug grasping system, and leveraging the YOLO11 algorithm with SAHI slice-assisted reasoning and a dynamic routing strategy, the challenges of small target detection and high-frequency real-time control in automated drug grasping are solved, achieving high-precision and high-frequency drug grasping.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV
- Filing Date
- 2026-02-02
- Publication Date
- 2026-05-12
AI Technical Summary
Existing automated drug-grabbing systems suffer from problems such as poor detection of small targets, high data acquisition costs, and large control delays when faced with highly complex and uncertain drug environments, making it difficult to achieve a balance between high-frequency real-time control and high accuracy.
An adaptive drug-grabbing system employing multimodal perception combines a visual perception and counting module, a strategy decision-making and routing module, and a dual-channel embodied execution module. It utilizes the YOLO11 algorithm with SAHI slice-assisted reasoning for high-precision detection and dynamically selects either the ACT fine-grained operation submodule or the π0.5 generalization operation submodule to achieve a balance between high-frequency real-time control and generalization capability.
It significantly improves the recall rate and positioning accuracy of small-sized drugs, solves the problem of small target detection, reduces data acquisition costs, and achieves a balance between high-frequency real-time control and generalization ability, adapting to various drug types and complex scenarios.
Smart Images
Figure CN122008205A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent robots and industrial automation technology, and in particular discloses an adaptive drug grasping system and method based on multimodal perception. Background Technology
[0002] With the rapid development of smart healthcare and automated logistics, automated drug sorting and retrieval have become key links in improving the efficiency of medical services. However, drug retrieval scenarios are highly complex and uncertain: drug packaging sizes vary greatly (from tiny ampoules to large pillboxes), are placed haphazardly, and there are serious problems with obstruction.
[0003] Traditional automated grasping solutions typically rely on conventional computer vision algorithms combined with fixed motion planning, making them ill-suited for unstructured environments. In recent years, while deep learning-based grasping detection algorithms (such as GraspNet) have improved grasping success rates, their detection performance is often poor when dealing with extremely small targets (such as scattered pills or small vials) due to resolution limitations.
[0004] In terms of control strategies, end-to-end imitation learning is gradually becoming the mainstream. Among them, large-scale vision-language-action (VLA) models (such as the π0.5 series) have demonstrated strong semantic understanding and generalization capabilities, and can handle objects that have never been seen before. However, such large-scale models have two significant drawbacks: first, they are difficult to "cold start," meaning that they require massive amounts of data for fine-tuning for specific vertical domains (such as handling special drugs), resulting in extremely high data acquisition costs; second, they have large inference latency, making it difficult to meet the high-frequency closed-loop control requirements of industrial robots above 50Hz, leading to sluggish or jittery grasping movements.
[0005] On the other hand, while the ACT (Action Chunking with Transformers) algorithm, which is based on action chunking, performs well in few-shot learning and high-frequency control, its generalization ability is not as good as the VLA large model when faced with extremely complex scenes with a large number of objects and complex semantics.
[0006] Therefore, there is an urgent need for a fusion-type grasping algorithm that can combine the advantages of small target detection and dynamically switch between "high generalization" and "high accuracy" according to the complexity of the scene, in order to solve the pain points of difficult identification, lack of data and slow control in the current automated drug grasping. Summary of the Invention
[0007] Based on this, the present invention provides an adaptive drug grasping system and method based on multimodal perception, so as to realize the semantic breadth of the large model and the operational depth of the special model, and solve the contradiction between the difficulty of "cold start" and high-frequency real-time control in drug grasping.
[0008] To achieve the above objectives, the present invention provides a multimodal fusion efficient drug grasping system based on imitation learning, including a visual perception and counting module, a strategy decision and routing module, a dual-channel embodied execution module, and an end-point control execution module.
[0009] The visual perception and counting module acquires RGB images of scenes containing pharmaceuticals using cameras installed at the robot's end effector or in the environment. It then combines this with the YOLO11 object detection algorithm, which uses SAHI slice-assisted reasoning, to perform high-precision identification and localization of pharmaceuticals within the scene, and outputs the total number of pharmaceuticals in the scene. In addition to the bounding box information of various drugs; the strategy decision and routing module is connected to the visual perception and counting module to receive the total number of drugs. and compare it with a preset scene complexity threshold. The comparison dynamically selects and activates one of the sub-modules in the dual-channel embodied execution module based on the comparison results; the dual-channel embodied execution module includes the ACT fine-tuning sub-module configured in parallel and... The generalized operation submodule generates smooth joint motion sequences and performs semantic understanding and long-range planning to generate motion trajectories in the form of flow matching; and the end-effector control execution module receives motion commands output by the dual-channel embodied execution module to drive the robotic arm and gripper to complete the task of grasping and placing medicines.
[0010] SAHI (Slicing Aided Hyper Inference) is an enhanced inference framework designed to address the "false negatives" problem in small object detection. During deep learning convolution, small objects often lose feature information due to image downsampling, making them unrecognizable. SAHI employs a "sliding window" mechanism to implement a "divide and conquer" strategy: it slices the high-resolution original image into multiple local slices with a set overlap rate, preserving the relative pixel resolution of small objects in their local viewpoints. The system then uses an AI model to perform independent inference on each slice, detecting objects within the local field of view. Finally, a coordinate mapping algorithm restores all local detection boxes to the global coordinate system, and non-maximum suppression (NMS) removes overlapping boxes. This mechanism significantly improves the recall rate (the proportion of successfully detecting all true objects) and precision for small objects without altering the original model structure.
[0011] YOLO11 is an advanced iteration of the YOLO (You Only Look Once) algorithm series, boasting both high inference speed and high detection accuracy. Its architecture is specifically optimized to solve real-time perception challenges in industrial scenarios. As a single-stage detector, YOLO11 eliminates the cumbersome candidate region generation process of traditional two-stage algorithms, directly regressing object categories and locations from images, significantly reducing computational latency. This makes it particularly suitable for edge computing scenarios—that is, on local terminals with limited computing power, such as robot controllers and drones, it can complete millisecond-level visual processing without relying on remote cloud servers, meeting the stringent requirements of automated production lines for low power consumption and high-frequency response.
[0012] At the core of its network structure, YOLO11 introduces an innovative C3k2 feature extraction module and the SPPF (Spatial Pyramid Pooling - Fast) multi-scale fusion mechanism. The C3k2 module, through its optimized residual structure, extracts more discriminative deep semantic features from complex backgrounds. The SPPF module, through pooling operations at different scales, enables the model to simultaneously "see large" and "see small" objects, ensuring accurate capture of everything from large pillboxes to tiny scattered pills. Furthermore, this model transcends the limitations of single-detection methods by integrating instance segmentation capabilities: it not only selects objects with bounding boxes but also generates pixel-level masks to precisely delineate the edges of irregular objects, which is crucial for handling the capture of stacked or occluded medications. Finally, YOLO11 provides end-to-end support from model training to deployment and export, and can be easily converted to TensorRT or ONNX formats, enabling efficient transformation of algorithms from laboratory prototypes to industrial applications.
[0013] Optionally, the visual perception and counting module uses a detection strategy combining YOLO11 and SAHI, and the steps include:
[0014] Original high-resolution image Divided into multiple overlapping slices The size of each slice is ;
[0015] Each slice is independently input into the YOLO11 detection network for inference. The YOLO11 detection network uses the C3k2 module as the feature extraction backbone and combines it with the SPPF module to enhance the multi-scale feature fusion capability.
[0016] The detection results of all slices are mapped back to the original image coordinate system, and the non-maximum suppression (NMS) algorithm is used to remove redundant boxes, and the final set of detection boxes is calculated. ;
[0017] If two prediction boxes intersection ratio If the value is greater than the preset threshold, the prediction box with higher confidence will be retained.
[0018] The final output total number of drugs For set The number of elements in the middle.
[0019] Optionally, the coordinate mapping and fusion process involves mapping the coordinates of the detection boxes in the sub-image. Map back to the original coordinate system:
[0020]
[0021] in For the first The top-left offset of each slice.
[0022] Optionally, the judgment logic of the strategy decision and routing module is as follows:
[0023] Set scene complexity threshold This threshold is determined based on the congestion of the robot's operating space and the computational resource consumption of a single planning operation.
[0024] when When the scenario is identified as a "low-density fine-grained operation scenario", the system routes to the ACT fine-grained operation submodule to take advantage of its high precision and low latency characteristics under a small number of expert demonstrations.
[0025] when When the scene is identified as a "high-density complex semantic scene", the system routes the data to... The generalization operation submodule leverages its semantic understanding and generalized crawling capabilities, which are pre-trained on large-scale Internet data.
[0026] Optionally, the ACT fine-tuning submodule in When activated, it generates smooth joint motion sequences based on conditional variational autoencoders and motion segmentation mechanisms for grasping tasks with small sample sizes and high precision.
[0027] Optionally, the ACT fine-grained operation submodule uses a Transformer-based CVAE architecture, and its action generation process includes:
[0028] During the training phase, the objective function is optimized, which includes reconstruction loss and KL divergence loss:
[0029]
[0030] in, For the future Step action sequence, For current observations, As latent variables, For encoder, Given a prior Gaussian distribution, These are the weighting coefficients;
[0031] In the reasoning phase, set up (Mean), the decoder is based on the current observation and Predict the future in one go Step action block ;
[0032] The action is smoothed using a temporal ensemble strategy, and the action predictions for overlapping time steps are weighted and averaged.
[0033]
[0034] in As a smoothing factor, This is the time offset of the action block, ensuring a high-frequency control response of over 50Hz.
[0035] Optionally, ACT uses a conditional variational autoencoder (CVAE) to model expert actions, with the training loss function including reconstruction loss and KL divergence loss.
[0036]
[0037] in, For the future Step action block, For current observations, This is a style variable.
[0038] Optionally, the The generalization operation submodule is in When activated, based on a vision-language-action multimodal large model architecture, it performs semantic understanding and long-range planning for complex backgrounds and multi-object stacking scenarios, generating action trajectories in the form of flow matching.
[0039] Optionally, the The generalization operation submodule uses a vision-language-action heterogeneous architecture, including:
[0040] Scene features were extracted using the SigLIP visual encoder and then cross-attention fused with natural language instructions such as "grab the amoxicillin box".
[0041] Using flow matching as the action decoder, continuous action trajectories are generated by iteratively denoising Gaussian noise, with the optimization objective being to minimize the vector field regression loss.
[0042]
[0043] in For noise distribution, For the target action distribution, The velocity field predicted by the model;
[0044] This module outputs high-level planning actions with semantic alignment, used to handle complex scenes with multiple object occlusions and unstructured placements.
[0045] To achieve the above objectives, this invention provides a multimodal fusion-based efficient drug acquisition method based on imitation learning, comprising the following steps:
[0046] S1: Collect image data of the drug sorting area through visual sensing devices;
[0047] S2: Using SAHI slice-assisted inference technology, the image is input into the YOLO11 network for small-target drug detection, and the output is the category, location, and total quantity of the drugs. ;
[0048] S3: The total number of drugs detected With preset threshold Perform a comparison;
[0049] S4: If Activate the ACT algorithm branch: Read the weights of the ACT model that have been pre-trained based on a small amount of expert demonstration data, combine them with the current observations, predict the joint angle sequence over a period of time in the future through the action block mechanism, and apply time integration smoothing.
[0050] S5: If Activate the π0.5 algorithm branch: Load the pre-trained VLA large model, input visual images and corresponding language instructions, and generate a grasping trajectory adapted to complex environments through the stream matching decoder;
[0051] S6: Send the motion commands generated by S4 or S5 to the robot controller to perform drug grabbing and placement, and monitor the execution status in real time until the task is completed.
[0052] Compared with existing technologies, the multimodal fusion-based efficient drug grasping system based on imitation learning provided by this invention has at least the following technical advantages:
[0053] Firstly, by integrating SAHI and YOLO11, this invention can effectively identify tiny drug packages that are difficult to detect by traditional algorithms, providing accurate perceptual input for the grasping strategy and solving the problem of detecting small target drugs.
[0054] Secondly, for specific high-difficulty drug retrieval tasks, this invention utilizes ACT's extremely high data efficiency, requiring only a small number of expert demonstrations for rapid deployment, without the need to retrain a massive VLA model for each new drug, thus solving the "cold start" problem of scarce expert data;
[0055] Thirdly, through a dynamic threshold switching strategy, the system uses ACT to achieve millisecond-level precise reflex control similar to the cerebellum in simple scenarios, and uses π0.5 to achieve deep semantic planning similar to the brain in complex scenarios. This achieves a balance between high-frequency real-time control and generalization ability, taking into account both operational robustness and task adaptability. Attached Figure Description
[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0057] Figure 1 is a schematic diagram of the overall architecture of a multimodal fusion efficient drug capture system based on imitation learning, which is applicable to multiple scenarios and multiple samples, as proposed in this invention.
[0058] Figure 2 is a flowchart of the strategy decision-making and routing logic based on drug quantity threshold in this invention.
[0059] Figure 3 is a schematic diagram of the principle of YOLO11 combined with SAHI slices to assist reasoning in the visual perception and counting module.
[0060] Figure 4 is a schematic diagram of the algorithm architecture of the ACT fine operation submodule.
[0061] Figure 5 is a schematic diagram of the VLA network architecture of the π0.5 generalization operation submodule. Detailed Implementation
[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0063] This invention belongs to the field of industrial robot and artificial intelligence technology, specifically relating to a multimodal fusion efficient drug grasping system and method based on imitation learning, applicable to multiple scenarios and multiple samples. It is particularly suitable for application scenarios with high requirements for small target recognition accuracy and scenario adaptability, such as automated sorting in hospital pharmacies and quality inspection and grasping in pharmaceutical factory assembly lines.
[0064] To address the technical challenges of difficult identification, data scarcity, and slow control in current automated drug grasping technologies, as mentioned in the background, this invention proposes a multimodal fusion-based efficient drug grasping system and method applicable to multiple scenarios and samples. Its core lies in constructing a heterogeneous and complementary control architecture. It utilizes YOLO11+SAHI to achieve high-precision perception of minute drug particles, leverages the "breadth" of π0.5 to handle diverse drug types and complex scenarios, and utilizes the "depth" of ACT to solve the execution challenges of extremely limited samples and high precision requirements.
[0065] In practice, this invention provides a multimodal fusion efficient drug grasping system based on imitation learning, applicable to multiple scenarios and multiple samples, including: a visual perception and counting module, a strategy decision and routing module, a dual-channel embodied execution module, and an end-point control execution module.
[0066] The visual perception and counting module uses YOLO11 as its base model and incorporates SAHI (Slicing-Aided Hyper Inference) technology. Addressing the small size of pharmaceutical items, SAHI slices the high-resolution image using a sliding window, performs inference on each slice separately, and then merges the results, significantly improving the recall rate for small targets. This module outputs the number of pharmaceutical items within the field of view in real time. .
[0067] The strategy decision and routing module uses "scene density" as the decision variable. When Less than or equal to the set threshold When the system is in a "low entropy" state, it focuses on the precision and speed of operations; when... When the value exceeds the threshold, the system considers it to be in a "high entropy" state and focuses on semantic understanding and obstacle avoidance planning.
[0068] The dual-channel embodied execution module contains two core algorithm branches:
[0069] The ACT branch (for scenarios with few samples and few targets) uses CVAE (Conditional Variational Autoencoder) to model the teaching trajectories of human experts. ACT introduces an action chunking mechanism to predict the future in a single inference. Step (e.g.) The system generates a sequence of actions and uses temporal ensemble (TIA) technology to perform a weighted average of overlapping action blocks. This mechanism not only smooths the trajectory but also eliminates robotic arm jitter caused by network inference latency, achieving real-time control at frequencies above 50Hz, and achieving a very high success rate with only a few dozen teaching data points.
[0070] π0.5 Branch (for multi-sample, complex scenarios): This branch uses a VLA architecture based on SigLIP and PaliGemma, combined with a flow matching generation strategy. Leveraging prior knowledge pre-trained on large-scale internet data, it can understand complex natural language instructions (such as "put the red medicine box on the left") and handle long-range planning problems involving stacked objects, thus compensating for ACT's shortcomings in generalizing complex semantics.
[0071] This invention provides a multimodal fusion-based efficient drug acquisition method based on imitation learning, characterized by the following steps:
[0072] S1: Collect image data of the drug sorting area through visual sensing devices;
[0073] S2: Using SAHI slice-assisted inference technology, the image is input into the YOLO11 network for small-target drug detection, and the output is the category, location, and total quantity of the drugs. ;
[0074] S3: The total number of drugs detected With preset threshold Perform a comparison;
[0075] S4: If Activate the ACT algorithm branch: Read the weights of the ACT model that have been pre-trained based on a small amount of expert demonstration data, combine them with the current observations, predict the joint angle sequence over a period of time in the future through the action block mechanism, and apply time integration smoothing.
[0076] S5: If Activate the π0.5 algorithm branch: Load the pre-trained VLA large model, input visual images and corresponding language instructions, and generate a grasping trajectory adapted to complex environments through the stream matching decoder;
[0077] S6: Send the motion commands generated by S4 or S5 to the robot controller to perform drug grabbing and placement, and monitor the execution status in real time until the task is completed.
[0078] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0079] Example 1: System Overall Architecture and Working Principle
[0080] Figure 1 is a schematic diagram of the overall architecture of a multimodal fusion efficient drug capture system based on imitation learning, which is applicable to multiple scenarios and multiple samples, as proposed in this invention. It shows the data flow of visual perception, policy routing, dual-channel execution, and end control.
[0081] As shown in Figure 1, this invention constructs a heterogeneous and complementary drug grasping control architecture. The system mainly consists of a visual perception layer, a decision routing layer, and a heterogeneous execution layer. The core logic of the system design lies in: utilizing the "breadth" of π0.5 to handle the diverse types of drugs and unstructured instructions 333, while utilizing the "depth" of ACT to solve the execution challenges of extremely small sample sizes and extremely high accuracy requirements 4.
[0082] In terms of physical deployment, the robotic arm's end effector is equipped with an RGB-D camera, and the host computer deploys a YOLO11 model accelerated by TensorRT, an ACT inference engine, and a quantized π0.5 large model. The system's workflow is triggered by visual perception, ultimately outputting 6-DOF (6-DoF) end effector pose and gripper opening and closing commands.
[0083] Example 2: High-precision visual perception based on YOLO11+SAHI
[0084] Figure 3 is a schematic diagram of the principle of YOLO11 combined with SAHI slice-assisted reasoning in the visual perception and counting module, which shows the process of high-resolution image slicing, independent reasoning and result mapping fusion.
[0085] As shown in Figure 3, in response to the problem that drug packaging is extremely small (such as a 2ml vial) and difficult to detect under a large field of view, this embodiment introduces SAHI (Slicing Aided Hyper Inference) technology based on YOLO11.
[0086] The specific implementation steps are as follows:
[0087] Image slicing: Let the original input image be... SAHI uses a sliding window strategy, setting the slice size to... The overlap rate is (For example, 20%) Divided into Sub-image .
[0088] Independent reasoning: For each slice Independent input YOLO11 detection network. YOLO11 uses the C3k2 backbone network and SPPF module to effectively extract features of small targets.
[0089] Coordinate mapping and fusion: Mapping the coordinates of the detection boxes in the sub-image Map back to the original coordinate system:
[0090]
[0091] in For the first The top-left offset of each slice.
[0092] Non-maximum suppression (NMS): Filters out redundant bounding boxes generated in overlapping regions, and finally outputs the total number of deduplicated drugs. And location information for various medicines.
[0093] Example 3: Policy Routing Mechanism Based on Scene Density
[0094] Figure 2 is a flowchart of the strategy decision-making and routing logic based on the drug quantity threshold in this invention, illustrating how the system dynamically switches between the ACT fine-grained operation mode and the π0.5 generalized operation mode.
[0095] As shown in Figure 2, this invention designs a dynamic routing strategy to balance computational cost and operational accuracy. A scenario complexity threshold is set. (For example This threshold is determined based on the congestion level of the robotic arm's workspace and the time required for a single planning operation.
[0096] Path A (High-density scene): When When the scene is identified as an "unstructured complex scenario," the system activates the π0.5 generalization operation submodule. At this point, the drugs in the scene are heavily piled up, requiring the semantic understanding capabilities of a large model for "cleaning" and data extraction.
[0097] Path B (low-density scene): When When the scene is identified as a "precision operation scenario," the system activates the ACT precision operation submodule. In this scenario, medicines are sparsely distributed, requiring high-precision, low-latency pickup to avoid collisions or drops.
[0098] Example 4: Implementation of the ACT Fine-grained Operation Submodule
[0099] Figure 4 is a schematic diagram of the algorithm architecture of the ACT fine-tuning submodule, showing the training process of the CVAE encoder and decoder and the time integration smoothing mechanism in the inference stage.
[0100] As shown in Figure 4, the ACT (Action Chunking with Transformers) submodule is used to handle small sample tasks with extremely high requirements for accuracy and real-time performance. Its core lies in solving the error accumulation problem in imitation learning and using action chunking to achieve smooth control.
[0101] CVAE Modeling: ACT uses a Conditional Variational Autoencoder (CVAE) to model expert actions. The training loss function includes reconstruction loss and KL divergence loss.
[0102]
[0103] in, For the future Step action block, For current observations, This is a style variable.
[0104] Action segmentation and reasoning: In the reasoning phase, set... (Prior mean), the decoder is based on the current observation Predict the future in one go Step target joint position sequence This approach reduces the effective horizon of the task. This significantly reduces the cumulative error by a factor of 10.
[0105] Temporal Ensembling: To eliminate action jumps at block transitions, as shown on the right side of Figure 4, the system performs a weighted average of the predicted actions for overlapping time steps.
[0106]
[0107] Where $m$ is the smoothing coefficient. This mechanism ensures that the system can output control commands with extremely high smoothness at a frequency of 50Hz, making it particularly suitable for the stable grasping of fragile medicines.
[0108] Example 5: Implementation of the π0.5 generalization operation submodule
[0109] Figure 5 is a schematic diagram of the VLA network architecture of the π0.5 generalization operation submodule, illustrating the visual encoder, language instruction input, and action generation process based on flow matching.
[0110] As shown in Figure 5, the π0.5 submodule is based on the Vision-Language-Action (VLA) architecture and aims to solve complex semantic understanding and long-term planning problems.
[0111] Heterogeneous data training: The training data for π0.5 not only includes robot operation data, but also integrates multimodal internet data (Web Data), high-level semantic subtask prediction, and cross-embodiment data from other robots.12 This hybrid training gives the model extremely strong zero-shot generalization ability, enabling it to recognize unseen drug packaging.
[0112] Hierarchical Inference Architecture: π0.5 uses a hierarchical inference mechanism. First, the model predicts high-level semantic subtasks (such as "πck up the blue box") based on the current image and task instructions; then, it predicts the underlying action sequence based on this subtask.
[0113] Flow Matching Action Generation: Unlike the deterministic decoding of ACT, π0.5 uses flow matching to generate continuous action trajectories. The model learns a vector field $v_t$ and maps Gaussian noise to an action distribution by solving ordinary differential equations (ODEs).
[0114]
[0115] This generative approach enables the model to generate more reasonable and diverse grasping paths when faced with multimodal distributions (such as drugs having multiple grasping postures).
[0116] Example 6: Demonstration of a Typical Workflow
[0117] Taking automated dispensing in hospital pharmacies as an example:
[0118] Scene initialization: A conveyor belt delivers a box of mixed medicines, which is detected by the YOLO11+SAHI module after scanning. (Greater than the threshold of 5).
[0119] Generalized Cleanup (π0.5 Intervention): The system routes to the π0.5 module. The host computer issues the command "Distribute the medicine boxes". Based on its semantic understanding capabilities, π0.5 plans a series of tossing and grasping actions to separate the stacked medicine boxes.
[0120] Status Switching: After several operations, four specific boxes of amoxicillin capsules remain in the field of view and need to be precisely placed into the narrow slots of the dispensing machine. At this point... (less than or equal to the threshold of 5).
[0121] Fine-grasp execution (ACT intervention): The system automatically switches to the ACT module. ACT loads weights finely tuned for the "card slot insertion" task (requiring only about 50 teaching data points15), and utilizes its time integration mechanism to complete the final grasping and insertion actions with extremely high precision and stability, ensuring no stuttering or damage.
[0122] The adaptive drug grasping system and method based on multimodal perception provided by this invention addresses the problems of large differences in target size, disordered placement, and difficulty in acquiring expert data in drug sorting scenarios. This invention constructs a closed-loop control system of "visual perception-strategy routing-heterogeneous execution". The system first uses the YOLO11 algorithm with integrated SAHI slicing inference technology to solve the problem of high-precision detection and counting of tiny drugs; then, based on the density threshold of the number of drugs in the scene, it dynamically routes to the dual-channel embodied execution module: in the fine operation scenario with few samples and low density, the ACT algorithm based on action chunking and temporal ensemble is activated to achieve smooth control above 50Hz with its high data efficiency; in the complex semantic scenario with multiple targets and high density, the π0.5 vision-language-action (VLA) large model based on flow matching is activated to handle unstructured instructions and long-range planning with its cross-scenario generalization ability. This invention effectively integrates the semantic breadth of a large model with the operational depth of a specialized model, resolving the contradiction between the difficulty of "cold start" and high-frequency real-time control in drug grasping.
[0123] This document provides a detailed description and uses specific examples to illustrate the principles and implementation methods of the present invention. The above embodiments are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
Claims
1. A multimodal fusion-based efficient drug grasping system based on imitation learning, characterized in that, include: The visual perception and counting module acquires RGB images of the scene containing the drugs using cameras installed at the robot's end effector or in the environment. It then combines this with the YOLO11 object detection algorithm, which uses SAHI slice-assisted reasoning, to perform high-precision identification and localization of the drugs in the scene, and outputs the total number of drugs within the scene. And the bounding box information for various drugs; The strategy decision-making and routing module is connected to the visual perception and counting module and receives the total number of drugs. and compare it with a preset scene complexity threshold. The comparison is performed, and one of the sub-modules in the dual-channel embodied execution module is dynamically selected and activated based on the comparison results. The dual-channel embodied execution module includes a parallel ACT fine-grained operation submodule and The generalization operation submodule generates smooth joint motion sequences, performs semantic understanding and long-range planning, and generates motion trajectories in the form of flow matching; and The end effector module receives motion commands from the dual-channel embodied execution module and drives the robotic arm and grippers to complete the task of grasping and placing medicines.
2. The multimodal fusion-based high-efficiency drug grasping system based on imitation learning according to claim 1, characterized in that, The visual perception and counting module uses a detection strategy combining YOLO11 and SAHI, and the steps include: Original high-resolution image Divided into multiple overlapping slices The size of each slice is ; Each slice is independently input into the YOLO11 detection network for inference. The YOLO11 detection network uses the C3k2 module as the feature extraction backbone and combines it with the SPPF module to enhance the multi-scale feature fusion capability. The detection results of all slices are mapped back to the original image coordinate system, and the non-maximum suppression (NMS) algorithm is used to remove redundant boxes, and the final set of detection boxes is calculated. ; If two prediction boxes intersection ratio If the value is greater than the preset threshold, the prediction box with higher confidence will be retained. The final output total number of drugs For set The number of elements in the middle.
3. The multimodal fusion-based high-efficiency drug grasping system based on imitation learning according to claim 1, characterized in that, The process of coordinate mapping and fusion involves mapping the coordinates of the detection boxes in the sub-image. Map back to the original coordinate system: ； in For the first The top-left offset of each slice.
4. The multimodal fusion-based high-efficiency drug grasping system based on imitation learning according to claim 1, characterized in that, The decision-making and routing module's judgment logic is as follows: Set scene complexity threshold This threshold is determined based on the congestion of the robot's operating space and the computational resource consumption of a single planning operation. when When the scenario is identified as a "low-density fine-grained operation scenario", the system routes to the ACT fine-grained operation submodule to take advantage of its high precision and low latency characteristics under a small number of expert demonstrations. when When the scene is identified as a "high-density complex semantic scene", the system routes the data to... The generalization operation submodule leverages its semantic understanding and generalized crawling capabilities, which are pre-trained on large-scale Internet data.
5. The multimodal fusion-based high-efficiency drug grasping system based on imitation learning according to claim 1, characterized in that, The ACT fine-operation submodule in When activated, it generates smooth joint motion sequences based on conditional variational autoencoders and motion segmentation mechanisms for grasping tasks with small sample sizes and high precision.
6. The multimodal fusion efficient drug grasping system based on imitation learning according to claim 5, characterized in that, The ACT fine-grained manipulation submodule uses a Transformer-based CVAE architecture, and its motion generation process includes: During the training phase, the objective function is optimized, which includes reconstruction loss and KL divergence loss: ； in, For the future Step action sequence, For current observations, As latent variables, For encoder, Given a prior Gaussian distribution, These are the weighting coefficients; In the reasoning phase, set up (Mean), the decoder is based on the current observation and Predict the future in one go Step action block ; The action is smoothed using a temporal ensemble strategy, and the action predictions for overlapping time steps are weighted and averaged. ； in As a smoothing factor, This is the time offset of the action block, ensuring a high-frequency control response of over 50Hz.
7. The multimodal fusion-based high-efficiency drug grasping system based on imitation learning according to claim 6, characterized in that, ACT uses a Conditional Variational Autoencoder (CVAE) to model expert actions, and the training loss function includes reconstruction loss and KL divergence loss. ； in, For the future Step action block, For current observations, This is a style variable.
8. The multimodal fusion-based high-efficiency drug grasping system based on imitation learning according to claim 1, characterized in that, The The generalization operation submodule is in When activated, based on a vision-language-action multimodal large model architecture, it performs semantic understanding and long-range planning for complex backgrounds and multi-object stacking scenarios, generating action trajectories in the form of flow matching.
9. The multimodal fusion efficient drug grasping system based on imitation learning according to claim 8, characterized in that, The The generalized operation submodule uses a vision-language-action heterogeneous architecture, including: Scene features are extracted using the SigLIP visual encoder and then cross-attention fused with natural language instructions such as "grab the amoxicillin box". Using flow matching as the action decoder, continuous action trajectories are generated by iteratively denoising Gaussian noise, with the optimization objective being to minimize the vector field regression loss. ； in For noise distribution, For the target action distribution, The velocity field predicted by the model; This module outputs high-level planning actions with semantic alignment, used to handle complex scenes with multiple object occlusions and unstructured placements.
10. A multimodal fusion-based efficient drug grasping method based on imitation learning, characterized in that, Includes the following steps: S1: Collect image data of the drug sorting area through visual sensing devices; S2: Using SAHI slice-assisted inference technology, the image is input into the YOLO11 network for small-target drug detection, and the output is the category, location, and total quantity of the drugs. ; S3: The total number of drugs detected With preset threshold Perform a comparison; S4: If Activate the ACT algorithm branch: Read the weights of the ACT model that have been pre-trained based on a small amount of expert demonstration data, combine them with the current observations, predict the joint angle sequence over a period of time in the future through the action block mechanism, and apply time integration smoothing. S5: If Activate the π0.5 algorithm branch: Load the pre-trained VLA large model, input visual images and corresponding language instructions, and generate a grasping trajectory adapted to complex environments through the stream matching decoder; S6: Send the motion commands generated by S4 or S5 to the robot controller to perform drug grabbing and placement, and monitor the execution status in real time until the task is completed.