A reagent management method and system based on multi-modal visual perception

Abstract

The application provides a reagent management method and system based on multi-modal visual perception, and relates to the field of automated laboratory technology. The method uses a multi-modal visual unit to collect reagent data, obtains reagent attributes and physical dimensions through target detection and character recognition, calculates optimal storage coordinates in combination with chemical safety isolation rules, controls a mechanical arm to complete reagent storage and update inventory, collects real-time poses of reagents after receiving a request for use, generates a grabbing instruction through coordinate system transformation and inverse kinematics operation, and completes reagent transfer. Finally, the weight closed-loop feedback dynamically adjusts the parameters of the precision pump to realize high-precision quantitative liquid taking and automatic pump stopping. The application realizes the full-process automation of reagent storage and use, reduces manual operation and errors, reduces labor costs, builds an integrated intelligent management and control system, and significantly improves the automation and safety level of laboratory reagent management.

Description

Technical Field

[0001] This invention mainly relates to the field of automated laboratory technology, specifically to a reagent management method and system based on multimodal visual perception. Background Technology

[0002] In the daily management and quantitative dispensing of laboratory reagents, manual operation remains the mainstream method in the industry. Staff are required to manually complete tasks such as reagent registration, sorting, and record-keeping. This process is not only cumbersome and labor-intensive, but also makes it difficult to strictly adhere to chemical segregation protocols, potentially leading to safety hazards due to mixed storage. In the reagent dispensing stage, relying on manual hand-held instruments for pipetting and weighing makes the accuracy susceptible to variations in operator skill and visual errors, failing to meet the high-precision dispensing requirements of trace reagents.

[0003] Currently, some laboratories have introduced basic automated reagent cabinets and simple pipetting devices. These devices mostly rely on physical barcodes and RFID tags for reagent entry and exit identification and on pre-set fixed coordinates for reagent transfer, only achieving basic item storage and retrieval functions. Such automation solutions have limited sensing capabilities, unable to autonomously identify reagent types, expiration dates, or dynamically adjust storage positions based on reagent physicochemical properties; the transfer mechanism operates along a preset trajectory, making reagent bottles susceptible to collision and breakage even with slight deviations. Furthermore, conventional automated pipetting devices often employ fixed-parameter open-loop control modes, without dynamic adjustments based on real-time weighing data. This makes them highly susceptible to reagent viscosity and environmental conditions, resulting in significant quantitative dispensing errors. Overall, their automation and intelligence levels are low, failing to meet the unmanned, high-precision, and high-safety operational requirements of modern smart laboratories. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a reagent management method and system based on multimodal visual perception, which addresses the shortcomings of the prior art.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A reagent management method based on multimodal visual perception, comprising the following steps: Visual perception data of reagents to be stored is collected by a multimodal vision acquisition unit deployed in the reagent storage system. The visual perception data is processed by target detection and character recognition to obtain the structured attribute data and physical size data of the reagents to be put into the warehouse; Based on the structured attribute data, physical size data, and pre-configured chemical safety isolation rules, calculate the optimal storage coordinates of the reagent to be stored in the reagent storage system coordinate system. The multi-degree-of-freedom robotic arm is controlled to move the reagents to be stored to the optimal storage location based on the optimal storage coordinates, and the reagent location status in the inventory database is updated synchronously. In response to a retrieval request, the real-time pose data of the target reagent is acquired by the vision acquisition unit mounted on the end of the robotic arm, and the real-time pose data is mapped to the target grasping pose in the coordinate system of the robotic arm base based on the pre-calibrated coordinate system transformation parameters. Inverse kinematics calculations are performed with the target grasping pose as the solution objective to generate grasping control commands. The grasping control commands are used to control the robotic arm to perform flexible grasping actions and transport the target reagent to the weighing station. After the target reagent is placed at the weighing station, a liquid dispensing control command is generated. The liquid dispensing control command is used to control the precision pump to perform a liquid dispensing operation. The driving parameters of the liquid dispensing control command are dynamically adjusted through a closed-loop control algorithm using real-time weight feedback data from the weighing station as the adjustment input. Once the liquid dispensing weight reaches the target threshold, a cutoff control command is generated. The cutoff control command is used to control the precision pump to perform a cutoff operation.

[0006] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: A reagent management system based on multimodal visual perception, comprising: The data acquisition module is used to acquire visual perception data of reagents to be stored through a multimodal vision acquisition unit deployed in the reagent storage system. The data processing module is used to perform target detection and character recognition processing on the visual perception data to obtain the structured attribute data and physical size data of the reagents to be put into the warehouse. The coordinate calculation module is used to calculate the optimal storage coordinates of the reagent to be stored in the reagent storage system coordinate system based on the structured attribute data, physical size data and pre-configured chemical safety isolation rules. The handling execution module is used to control the multi-degree-of-freedom robotic arm to move the reagents to be put into storage to the optimal storage location based on the optimal storage coordinates, and to synchronously update the reagent location status in the inventory database. The pose acquisition module is used to respond to the retrieval request by acquiring the real-time pose data of the target reagent through the vision acquisition unit mounted on the end of the robotic arm, and mapping the real-time pose data to the target grasping pose in the coordinate system of the robotic arm base based on the pre-calibrated coordinate system transformation parameters. The grasping control module is used to perform inverse kinematics calculations with the target grasping pose as the solution target, and generate grasping control instructions. The grasping control instructions are used to control the robotic arm to perform flexible grasping actions and transport the target reagent to the weighing station. The liquid dispensing control module is used to generate a liquid dispensing control command after the target reagent is placed at the weighing station. The liquid dispensing control command is used to control the precision pump to perform a liquid dispensing operation. The module uses the real-time weight feedback data of the weighing station as the adjustment input and dynamically adjusts the driving parameters of the liquid dispensing control command through a closed-loop control algorithm until the liquid dispensing weight reaches the target threshold. Then, a cutoff control command is generated to control the precision pump to perform a cutoff operation.

[0007] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a reagent management system based on multimodal visual perception, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the reagent management method based on multimodal visual perception as described above.

[0008] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the reagent management method based on multimodal visual perception as described above.

[0009] The beneficial effects of this invention are as follows: by sequentially completing visual data acquisition, image and text parsing, optimal storage coordinate calculation, reagent transfer and storage, target pose conversion, robotic arm grasping and handling, and closed-loop quantitative liquid dispensing, it can fully realize the automated operation of reagents from storage management to precise dispensing, get rid of the operation mode of full manual intervention, and thus effectively reduce the error rate and labor costs caused by manual operation. At the same time, it establishes an integrated intelligent reagent management and control system, and comprehensively improves the overall automation level of laboratory reagent management and dispensing operations. Attached Figure Description

[0010] Figure 1 A flowchart of a reagent management method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the coordinate mapping relationship between the camera and the end effector of the robotic arm, provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the overall process of adaptive grasping and closed-loop quantitative liquid dispensing provided in an embodiment of the present invention; Figure 4 This is a functional block diagram of the reagent management system provided in an embodiment of the present invention; Figure 5 This is a hardware architecture diagram of the reagent management system provided in an embodiment of the present invention; Figure 6 This is a diagram illustrating the hierarchical data flow and control architecture of the reagent management system provided in an embodiment of the present invention. Detailed Implementation

[0011] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0012] Example 1: As Figure 1 As shown, this embodiment of the invention provides a reagent management method based on multimodal visual perception, including the following steps: S1. Visual perception data of reagents to be stored are collected by a multimodal vision acquisition unit deployed in the reagent storage system; S2. Perform target detection and character recognition processing on the visual perception data to obtain the structured attribute data and physical size data of the reagent to be put into the warehouse; S3. Based on the structured attribute data, physical size data, and pre-configured chemical safety isolation rules, calculate the optimal storage coordinates of the reagent to be stored in the reagent storage system coordinate system. S4. Control the multi-degree-of-freedom robotic arm to move the reagent to be stored to the optimal storage location based on the optimal storage coordinates, and simultaneously update the reagent location status in the inventory database. S5. In response to the retrieval request, the real-time pose data of the target reagent is acquired through the vision acquisition unit mounted on the end of the robotic arm, and the real-time pose data is mapped to the target grasping pose in the coordinate system of the robotic arm base based on the pre-calibrated coordinate system transformation parameters. S6. Perform inverse kinematics calculations with the target grasping pose as the solution target to generate grasping control instructions. The grasping control instructions are used to control the robotic arm to perform flexible grasping actions and transport the target reagent to the weighing station. S7. After the target reagent is placed at the weighing station, a liquid dispensing control command is generated. The liquid dispensing control command is used to control the precision pump to perform a liquid dispensing operation. The driving parameters of the liquid dispensing control command are dynamically adjusted through a closed-loop control algorithm using the real-time weight feedback data of the weighing station as the adjustment input until the liquid dispensing weight reaches the target threshold. Then, a cutoff control command is generated. The cutoff control command is used to control the precision pump to perform a cutoff operation.

[0013] In the above embodiments, by sequentially completing visual data acquisition, image and text parsing, optimal storage coordinate calculation, reagent transfer and storage, target pose conversion, robotic arm grasping and handling, and closed-loop quantitative liquid dispensing, the entire process of reagent storage management and precise dispensing can be automated, eliminating the need for manual intervention throughout the entire process. This effectively reduces the error rate and labor costs caused by manual operation, while also establishing an integrated intelligent reagent management and control system to comprehensively improve the overall automation level of laboratory reagent management and dispensing operations.

[0014] Preferably, S1, visual perception data of the reagents to be stored is acquired through a multimodal vision acquisition unit deployed in the reagent storage system, including: S11. Control the multi-degree-of-freedom robotic arm to move the end effector to the preset scanning pose, and trigger the RGB-D camera in the multimodal vision acquisition unit to acquire the initial color image and initial depth image of the reagent to be put into storage. S12. Perform adaptive histogram equalization on the initial color image, and extract the contour features of the reagent label region based on the improved Canny edge detection operator, and calculate the minimum bounding rectangle of the contour features. S13. Calculate the three-dimensional spatial coordinates of the reagent to be stored in the camera coordinate system based on the pixel coordinates of the minimum bounding rectangle in the initial color image and the depth value corresponding to the initial depth image. S14. If the confidence level of the three-dimensional spatial coordinates is lower than a preset threshold, a rescan control command is generated to control the robotic arm to drive the multimodal vision acquisition unit to perform relative displacement sampling along a preset spiral trajectory until visual perception data that meets the confidence level requirements is obtained. Wherein, the radial displacement increment of the spiral trajectory With rotation angle increment Satisfying Relationship: , in, This is the sequence number of the current sampled frame. This is the radial step size coefficient. is the angular velocity coefficient, and , .

[0015] In the above embodiments, the robotic arm is controlled to work with an RGB-D camera to acquire image and depth data, and edge features are extracted and three-dimensional coordinates are calculated from the images. At the same time, when the data confidence is insufficient, multi-frame rescanning sampling is performed according to a specific spiral trajectory. This can stably acquire complete visual information of reagents in scenarios with complex lighting and limited viewing angles, effectively avoiding the problem of inaccurate recognition of single-frame images, thereby improving the reliability and environmental adaptability of visual data acquisition, and providing accurate and effective raw data sources for subsequent data processing.

[0016] Based on the above data acquisition process, before the robotic arm moves to the scanning pose, it calculates the pre-approach point outside the shelf by combining the warehouse location coordinates in the inventory record. A safe distance of 80mm to 150mm is reserved along the outer normal of the shelf before reaching this point, and then the camera starts working. The RGB-D camera acquires images and depth point cloud data at a high frame rate and transmits them to the scheduling center in real time via gigabit Ethernet or a high-speed data bus. A Gaussian filtering noise reduction step is added after image acquisition to effectively eliminate image noise caused by dust and corrosive gases in a chemical environment. In this embodiment, the confidence threshold for three-dimensional spatial coordinate recognition is set to 0.8, and the spiral trajectory takes the center point of the target warehouse as the origin of the local coordinate system. The trajectory equation is: , , in, For the rescan angle parameters, The coefficient represents the height variation. The system limits the spiral rescan to a maximum of three times. If the fusion confidence score is still below 0.85 after multiple rescans, the corresponding storage location will be marked as a visual anomaly, a maintenance prompt will be pushed to the management terminal, and the system will automatically jump to the next task.

[0017] As an optional implementation, the RGB-D camera can be replaced with a binocular stereo camera or a structured light camera, or a monocular camera paired with an ultrasonic ranging sensor can be used to complete 3D information acquisition. In addition to the deployment of the vision acquisition unit with the "eye in the hand" at the end of the robotic arm, it can also adopt an "eye outside the hand" mode, fixing multiple sets of industrial cameras on the top or sides of the shelf, and achieving data acquisition through multi-view image fusion. For high-risk reagent storage areas with chemical fumes or strong light reflections, camera exposure parameters can be adjusted to further improve the stability of visual data acquisition under complex working conditions.

[0018] Preferably, S2, the visual perception data is processed by target detection and character recognition to obtain the structured attribute data and physical size data of the reagent to be stored, including: S21. Target detection is performed on the visual perception data, and the bounding box and label region contour features of the reagent to be put into the warehouse are output. The label region of interest is then cropped based on the contour features. S22. Perform grayscale conversion, adaptive histogram equalization contrast enhancement, Gaussian filtering noise reduction and perspective correction on the region of interest of the label in sequence to obtain a standardized label image; S23. Perform text line detection on the standardized label image, output a text detection box, and perform optical character recognition processing on the character sequence in the text detection box to obtain the original text sequence containing the drug name, specifications, batch number and expiration date. S24. Based on the predefined keyword library and regular expression matching rules, the original text sequence is parsed into the corresponding field types to generate structured attribute data; S25. Based on the pixel size of the bounding box and the pre-calibrated pixel equivalent, calculate the bottle height and cross-sectional diameter of the reagent to be stored, and generate physical size data; S26. If the recognition confidence of the key field is lower than the preset threshold, a rescan mechanism is triggered, and weighted voting fusion is performed based on the recognition results of multiple frames. The confidence score of the fusion result is then calculated. Satisfying Relationship: , in, For the first Confidence level for frame image recognition For the first Frame image sharpness weighting, n The total number of image frames involved in the fusion, when the confidence score of the fusion result is... If the value is greater than or equal to a preset threshold, the structured attribute data is confirmed to be valid.

[0019] In the above embodiments, by performing target detection, image preprocessing, text recognition and field parsing on visual data, and calculating reagent physical parameters in combination with pixel size, and by using a weighted fusion method of multi-frame recognition results to correct low-confidence recognition content, it is possible to accurately extract various information such as reagent name, specifications, batch number, and size, reduce recognition errors caused by label reflection and image distortion, and thus ensure the accuracy of reagent attribute data and physical size data, providing reliable data support for storage location allocation and safety management.

[0020] Specifically, the OCR module is used to convert the text in the bottle label image into structured reagent attributes that the system can access. First, the ROI of the bottle label is cropped based on the bounding box output by YOLOv5. Then, the ROI image is subjected to grayscale conversion, contrast enhancement, noise reduction, tilt correction, and perspective correction to reduce recognition errors caused by reflections, label tilt, and bottle surface deformation. After image preprocessing, the OCR module first locates the text region in the label and outputs a text detection box, then recognizes the characters within each text box to obtain a character sequence. The field parsing unit then uses a keyword library, regular expression matching rules, and context rules to categorize the recognition results into fields such as drug name, specification / concentration, production batch number, expiration date, hazardous material category, and storage. For example, when keywords such as "expiration date," "EXP," and "shelf life" are recognized, the adjacent dates are parsed as the expiration date; when keywords such as "batch number," "Lot," and "Batch" are recognized, the adjacent alphanumeric combinations are used as the production batch number; when keywords such as "store away from light," "strong acid," "flammable," and "oxidizer" are recognized, they are mapped to the corresponding storage constraint labels. The system ultimately binds the reagent bottle location, size, and structured text attributes into reagent object records for dynamic storage allocation and robotic arm grasping and scheduling. When the OCR confidence of a key field falls below a preset threshold of 0.85, the system re-acquires images or performs multi-angle re-shooting, and then performs voting fusion on the results of multiple frames to reduce the impact of single-frame misidentification on the stored attributes.

[0021] As an optional implementation, the target detection model used in this solution can be replaced with other models in the YOLO series, Faster R-CNN, DETR, and other similar visual detection algorithms. The optical character recognition module can also use a text recognition algorithm based on an attention mechanism, adapting to different computing power devices and application scenarios without changing the overall processing flow. Simultaneously, the entire data processing flow can be linked with the front-end multimodal visual acquisition module, dynamically supplementing image data sources based on visual rescan results, forming a closed-loop logic of acquisition, processing, and re-inspection, comprehensively improving the stability of data processing under complex laboratory conditions.

[0022] Preferably, step S3 involves calculating the optimal storage coordinates of the reagent to be stored in the reagent storage system coordinate system based on the structured attribute data, physical size data, and pre-configured chemical safety isolation rules, including: S31. Traverse the candidate storage locations in the reagent storage system and filter out candidate storage locations that do not meet environmental constraints or are within the reach of the robotic arm based on the hazardous properties and physical size data of the reagents to be stored. S32. Let the coordinates of the candidate storage bits that pass the filter be... Calculate the Euclidean distance between the candidate storage location and the already stored incompatible reagent storage locations: , S33, if the Euclidean distance If so, then the candidate storage bit is removed, where, This refers to the minimum isolation distance determined based on the pre-configured chemical safety isolation rules. S34. Calculate a comprehensive evaluation function for candidate storage sites that have passed the chemical safety isolation verification: , in, The path cost for the robotic arm to return from the candidate storage location to the weighing station. The cost of matching candidate storage space volume with physical size data. Penalties are based on the level of chemical hazard. Historical frequency of reagent use for each category , , , These are preset weighting coefficients; S35. Select the comprehensive evaluation function The smallest candidate storage location is identified, and the coordinates of the smallest candidate storage location are determined as the optimal storage coordinates for the reagents to be stored.

[0023] In the above embodiments, by combining reagent properties and physical dimensions to screen available storage locations, relying on Euclidean distance to verify the safe isolation distance of chemicals, and then using a comprehensive evaluation function to screen the optimal storage location, it is possible to strictly follow the chemical storage specifications to avoid the risk of mixed storage. At the same time, it takes into account the movement path of the robotic arm, spatial matching degree and reagent usage frequency to complete intelligent allocation, thereby improving the safety of reagent storage, space utilization and the overall efficiency of subsequent retrieval operations.

[0024] In this embodiment, during the storage location screening and evaluation stage, multiple constraints and logical refinements are implemented based on the actual application scenario of the reagents and the hardware operating characteristics. The specific implementation is as follows: 1. Candidate storage bit pre-filtering details: The system reads the real-time storage location status database to obtain the 3D coordinates, physical dimensions, occupancy status, adjacent reagent types, environmental conditions, and robotic arm reachability of all candidate storage locations. Combined with the structured attributes obtained from reagent analysis, a multi-dimensional pre-filter is prioritized: candidate storage locations are eliminated if they are already occupied, have internal space dimensions smaller than the outer diameter / height of the reagent bottle, are outside the reachability range of the robotic arm, or have mismatched environmental conditions such as temperature, humidity, light protection, and ventilation. For example, light-sensitive reagents are directly eliminated from storage locations without light-shielding structures, and reagents requiring low-temperature storage are eliminated from ordinary room-temperature shelf locations.

[0025] 2. Implementation instructions for chemical safety isolation rules: Pre-configured chemical compatibility rules tables set corresponding minimum isolation distances for incompatible reagent combinations such as strong acids and strong bases, oxidants and reducing agents, flammable substances and strong oxidants, and volatile corrosive reagents. After completing the basic filtering, the coordinates of all incompatible reagent storage locations that have been stored are traversed, and distance verification is performed one by one. If the distance does not meet the standard, the corresponding candidate storage location is directly removed to avoid safety accidents caused by mixing different types of reagents in physical space.

[0026] Finally, the storage location with the lowest overall score was selected as the optimal storage coordinate. If, after traversing all candidate storage locations, no location meets all the constraints, the system will not force the storage operation. Instead, it will mark the reagent as requiring manual confirmation or expansion of the safe storage location, push an exception message to the upper management terminal, and automatically jump to the next task.

[0027] After determining the optimal storage coordinates, the system synchronously binds and enters all information such as reagent number, name, specifications, hazardous attributes, physical dimensions, and storage coordinates into the inventory database, and updates the storage location occupancy status and adjacent reagent matching relationships in real time, providing complete data support for subsequent reagent retrieval, secondary storage, safety verification, and task scheduling.

[0028] Figure 2 This is a schematic diagram illustrating the coordinate mapping relationship between the camera and the end effector of the robotic arm, provided as an embodiment of the present invention. Figure 2 As shown, this system is equipped with a base coordinate system. Robotic arm end-effector coordinate system Camera coordinate system and the target coordinate system of the reagent bottle Coordinate mapping between different coordinate systems is accomplished using homogeneous transformation matrices.

[0029] Where, transformation matrix The pose of the robotic arm's end effector relative to the base is obtained by solving the forward kinematics of the robotic arm; the transformation matrix is ​​used. The hand-eye calibration is pre-calibrated and solved to characterize the fixed pose of the end-effector camera relative to the robotic arm's end effector; the transformation matrix... The pose of the target reagent bottle relative to the camera coordinate system is obtained by solving point cloud data collected by a depth camera.

[0030] By performing the cascaded operations of the above three sets of transformation matrices, the pose of the target reagent bottle relative to the coordinate system of the robotic arm base can be calculated. Then, by substituting the values ​​into the inverse kinematics of the robotic arm, the rotation angles of each joint are solved, ultimately achieving high-precision positioning and gripping of the reagent bottle by the gripper.

[0031] Preferably, S4, controlling the multi-degree-of-freedom robotic arm to move the reagent to be stored to the optimal storage location based on the optimal storage coordinates, includes: S41. Based on the optimal storage coordinates, construct a collision-free path for the robotic arm from the initial pose to the target storage pose; S42. A time-optimal trajectory planning algorithm is used to generate a motion trajectory in joint space on the collision-free path. The motion trajectory satisfies the following constraints: , in, , and The first The velocity, acceleration, and jump of each joint , and These are the preset maximum speed, maximum acceleration, and maximum jump thresholds, respectively; S43. Based on the generated motion trajectory, generate the handling control command to control the multi-degree-of-freedom robotic arm to transport the reagent to be stored to the optimal storage location.

[0032] In the above embodiments, by pre-planning the collision-free passage path of the robotic arm and setting constraints on the speed, acceleration, and jump of each joint to generate the optimal motion trajectory in time, the impact and vibration during the movement of the robotic arm can be effectively limited, avoiding collisions between the robotic arm and the shelves or reagents, while also reducing the shaking of chemical liquids in the container. Thus, while ensuring transfer efficiency, the stability of the reagent handling process and the overall operational safety factor are greatly improved.

[0033] Building upon the aforementioned basic approach, this method further refines the engineering aspects of trajectory planning and motion control, as detailed below: 1. Safety strategy before path planning: Before the robotic arm officially starts its movement, the system, based on the target storage location coordinates, shelf structure, reagent bottle dimensions, and gripper physical parameters, sets a safe pre-approach point 80mm~150mm outside the target storage location. The robotic arm first moves to this outer observation point before performing the subsequent storage action, thus preventing the gripper and robotic arm linkage from directly extending into the shelf and scraping or colliding with shelf partitions or adjacent reagent bottles.

[0034] 2. Trajectory parameter adaptive configuration rules: The speed, acceleration, and jump thresholds of each joint are not fixed values. The system dynamically adjusts them based on real-time operating conditions: a safety reduction factor is calculated based on the total weight of the reagent bottle, the liquid level inside the bottle, and the hazard level of the reagent. The larger the reagent weight, the higher the liquid level, or the more corrosive, volatile, or hazardous the reagent, the smaller the reduction factor becomes, further reducing the joint motion parameters and lowering the risk of violent liquid sloshing and spillage during start-up, shutdown, and turning.

[0035] 3. Trajectory interpolation implementation method: Specifically, for the first Each joint, from the starting joint angle To the target joint angle The motion process is represented by normalized path parameters. s The function below ,in When using fifth-order polynomial interpolation, , to It is determined by the initial joint angle, the target joint angle, and the start and end velocities and acceleration continuity conditions.

[0036] When using B-spline curves, the first The trajectory of each joint can be represented as follows: .in, for B-spline basis functions For the first Each joint corresponds to a control point. After path parameterization, the system does not execute at a constant speed directly, but instead adds velocity, acceleration, and jerk constraints to the path and optimizes the execution time.

[0037] 4. Trajectory Iteration Verification Mechanism: The system discretizes the complete motion trajectory into multiple sampling points, and verifies the velocity, acceleration, and jump values ​​of each joint at the corresponding moment. If any sampling point exceeds the preset constraint threshold, the system automatically extends the motion duration, adjusts the interpolation control points, and regenerates the trajectory until the entire path meets the constraint requirements before issuing the transport control command to execute the action.

[0038] 5. Logic for completing the inbound process: After the robotic arm completes the placement of reagents, resets the gripper, and leaves the shelf area, the system immediately updates the inventory database, marks the current storage location as occupied, and records the movement path data corresponding to the reagent. This provides a reference for subsequent reagent retrieval and secondary storage tasks at the same location, improving overall scheduling efficiency.

[0039] Preferably, S5, in response to a retrieval request, real-time pose data of the target reagent is acquired through a vision acquisition unit mounted on the end effector of the robotic arm, and the real-time pose data is mapped to a target grasping pose in the robotic arm base coordinate system based on pre-calibrated coordinate system transformation parameters, including: S51. Acquire two-dimensional and depth images of the target reagent through the vision acquisition unit mounted on the end of the robotic arm, extract the contour features of the target reagent based on the improved Canny edge detection operator, and calculate the minimum bounding rectangle of the contour features. S52. Based on the pixel coordinates of the minimum bounding rectangle in the two-dimensional image and the depth value corresponding to the depth image, calculate the real-time three-dimensional coordinates and attitude angles of the target reagent in the visual acquisition unit coordinate system, and generate the real-time pose data. S53. Based on the pre-calibrated hand-eye transformation matrix and tool calibration matrix, and combined with the current joint angle data of the robotic arm, the real-time pose data is mapped to the target grasping pose in the robotic arm base coordinate system, wherein the mapping relationship satisfies the following homogeneous coordinate transformation formula: , in, To obtain the homogeneous transformation matrix of the target's pose in the robot arm's base coordinate system. This is the pose matrix of the robot arm's end flange in the base coordinate system, calculated using forward kinematics based on the current joint angle data. The hand-eye transformation matrix of the vision acquisition unit relative to the end flange. This represents the pose matrix of the target reagent relative to the visual acquisition unit. S54. If the confidence level of the real-time pose data is lower than a preset threshold, a rescan control command is generated to control the robotic arm to drive the vision acquisition unit to perform relative displacement sampling along a preset spiral trajectory until real-time pose data that meets the confidence level requirements is obtained.

[0040] In the above embodiments, the reagent contour is extracted by edge detection and the three-dimensional pose is calculated. Then, the data conversion between multiple coordinate systems is completed by relying on the homogeneous coordinate transformation formula. At the same time, spiral rescan sampling is started for low confidence data, which can accurately convert the local pose information acquired by vision into a global pose that can be recognized by the robotic arm. This effectively adapts to complex working conditions such as reagent placement offset and screen occlusion, thereby continuously improving the accuracy of target grasping and positioning and the system's environmental adaptability.

[0041] This step is further refined and supplemented based on actual engineering application scenarios and the overall system collaboration logic: 1. Diverse adaptation solutions for vision devices: The vision acquisition unit mounted on the end effector of the robotic arm is flexibly selectable. In addition to conventional RGB-D cameras, it can be replaced with binocular stereo cameras or structured light cameras. The deployment method can also be changed to adopt an "eye-on-hand" architecture, where multiple industrial cameras are fixed in the shelf area. This multi-view image fusion method is used to acquire 2D images and depth information of reagents, adapting to different laboratory layouts and workspace conditions. For high-risk reagent storage areas with chemical fumes, strong light reflections, or uneven lighting, the system can adaptively adjust camera exposure, gain, and other parameters to further improve the stability of contour extraction and pose calculation under complex conditions.

[0042] 2. Approach pose planning strategy: Before initiating visual data acquisition, the robotic arm calculates a safe pre-approach point outside the shelf by combining the coordinates of the target reagent's location, the shelf's shape, the reagent bottle's size, and the field of view of the end-effector camera. The robotic arm first moves to this point before acquiring image and depth data, avoiding direct insertion of the robotic arm and camera into the shelf and preventing scraping, while ensuring the camera has a sufficient effective field of view and reducing the impact of occlusion on the sampling results.

[0043] 3. Supplementary Explanation of Rescan Trajectory When the robotic arm drives the vision acquisition unit to perform helical rescan sampling, a helical motion trajectory is constructed with the center point of the target reagent as the local coordinate origin. The trajectory equation is: , , , For the rescan angle parameters, The height variation coefficient is used. The system limits the number of rescans, performing a maximum of three helical samplings. If the confidence level of the real-time pose data still fails to meet the standard after multiple samplings, the system determines that there is a visual anomaly at the storage location, automatically records the anomaly log, pushes a maintenance reminder to the management terminal, and jumps to the next task to avoid system crashes.

[0044] 4. Supplementary Explanation of Rescan Trajectory After completing the pose calculation for a single frame, the system can combine the sampling results of multiple consecutive frames for filtering to filter pose jump data caused by slight shaking of the robotic arm and environmental interference, thereby further improving the accuracy of real-time pose data and providing a reliable data source for subsequent coordinate system transformation and grasping pose calculation.

[0045] Preferably, S6, performing inverse kinematics calculations with the target grasping pose as the solution objective to generate grasping control commands, including: S61. Obtain the homogeneous transformation matrix corresponding to the target's pose. To solve for the objective, a pose error function is constructed that includes both position and attitude errors: , in, and These are the current position vector and attitude vector of the robotic arm's end effector, respectively. and These are the position vector and attitude vector of the target capture pose, respectively. These are the attitude error weighting coefficients; S62. The inverse kinematics iterative solution is performed using the damped least squares method, and the joint angle vector is updated using the following iterative formula: , in, For the first The joint angle vector of the next iteration. For Jacobian matrices, This is the pose synthesis error vector. The damping coefficient is... It is the identity matrix; S63. During the iteration process, joint limit verification and collision detection are performed synchronously. For the candidate solutions that pass the verification, the solution with the smallest change in the current joint angle vector and which satisfies the collision constraint is selected as the target joint angle vector, and the grasping control command is generated.

[0046] In the above embodiments, by constructing a comprehensive error function that integrates position and attitude, and using the damped least squares method to iteratively solve the joint angles of the robotic arm, while completing limit and collision screening during the iteration process, the singular configuration problem that occurs when solving inverse kinematics can be effectively avoided, and safe and reasonable joint motion parameters can be selected, thereby ensuring that the robotic arm's grasping action is executed smoothly and reliably.

[0047] Based on the above inverse kinematics solution and grasping control, further detailed explanations are provided, taking into account the on-site operating conditions, the robotic arm's operating characteristics, and reagent protection requirements: 1. Error function weight adaptation strategy: Attitude error weighting coefficient Supports dynamic configuration as needed. For reagent containers that are long and slender, have a high center of gravity, and are prone to tipping, the size can be appropriately increased. When selecting values, priority should be given to ensuring the accuracy of the grasping posture; for conventional reagent bottles that are short, wide, and highly stable, the value can be appropriately reduced. Prioritize positional accuracy to ensure the safety of gripping different reagent bottles.

[0048] 2. Dynamic adjustment mechanism of damping coefficient: Damping coefficient during iterative solution process It is not a fixed value. The system automatically increases the value as the robotic arm approaches the region of unusual configuration. This effectively suppresses solution oscillations and avoids sudden changes in joint velocity; when the robotic arm is in a normal workspace, the speed is appropriately reduced. This improves the iteration convergence speed and achieves a balance between solution stability and computational efficiency.

[0049] 3. Supplement to the logic for selecting the best solution from multiple candidates: Inverse kinematics solutions often yield multiple candidate solutions for joint angles that satisfy the constraints. In addition to selecting the solution with the smallest change in the current joint angle, the system also performs a secondary screening based on the robot arm's motion energy consumption and the smoothness of the subsequent handling path, prioritizing joint combinations with smoother motion trajectories and lower energy consumption to reduce reagent shaking caused by the robot arm's start-stop and turning.

[0050] 4. Flexible grasping linkage control: After generating the target joint angle and gripping control commands, the system synchronously activates the flexible end gripper. Based on the previously identified reagent bottle shape and weight data, the system adaptively adjusts the gripping force and opening / closing range of the gripper, employing a flexible gripping method to ensure a firm grip without slipping, while avoiding damage to the bottle body or label caused by rigid gripping. This method is suitable for reagent containers made of different materials such as glass and plastic.

[0051] 5. Real-time monitoring of the grabbing process: Throughout the entire grasping process, the system continuously monitors the reagent in real time, combining visual data and gripper pressure sensor data. If any abnormalities such as reagent slippage or displacement are detected, the grasping action is immediately paused, and the pose acquisition and inverse kinematics solution process is restarted to perform a secondary positioning and grasping, preventing safety accidents such as reagent drop or collision damage.

[0052] Preferably, S63, during the iteration process, joint limit verification and collision detection are performed simultaneously, including: The robot arm's joint angles are calculated based on the current candidate solutions. If any joint angle exceeds the preset joint angle range, the current candidate solution is discarded. Based on the current candidate solution, the pose of the robotic arm link is calculated using forward kinematics, and the pose is then compared with a pre-constructed obstacle geometry model for distance detection. The obstacle geometry model includes at least shelf partitions and adjacent reagent bottles. If the shortest distance between the robotic arm link and the obstacle geometry is less than a preset safety threshold, interference is determined and the current candidate solution is eliminated.

[0053] In the above embodiments, by performing range verification on the joint angles obtained by solving, and combining the positive kinematics results with the obstacle model to make distance judgments, invalid solutions that may cause joint over-limit and mechanism interference can be identified and eliminated in a timely manner, thereby avoiding failures such as robotic arm jamming, collision with shelves or reagent bottles from the source, and further enhancing the safety and operational stability of robotic arm grasping operations.

[0054] To further enhance the engineering practicality and environmental adaptability of collision detection and limit calibration, the following detailed solutions are provided based on actual operational scenarios: 1. Obstacle model construction and dynamic update mechanism: The system pre-models static obstacles such as laboratory shelves, cabinets, and fixed fixtures in 3D and imports them into the model library. Simultaneously, for dynamic obstacles like reagent bottles, it updates the obstacle geometry model in real time based on the reagent shape and placement obtained from prior visual recognition. Before each grabbing task, it automatically refreshes the surrounding reagent distribution data to ensure that the collision detection criteria remain consistent with the actual environment, preventing detection failures due to reagent position changes.

[0055] 2. Tiered security threshold settings: The system classifies safety distance thresholds into multiple levels based on the risk level of the work area: higher safety thresholds are used in areas with dense reagents and around fragile glass reagents to allow sufficient space for avoidance; the thresholds can be appropriately lowered in open work areas to improve the flexibility of the robotic arm while ensuring safety. Safety thresholds can be manually configured or automatically adjusted by the system according to the reagent material and hazard level.

[0056] 3. Scope of full-area collision screening: Collision detection does not only check the end effector of the robotic arm, but also conducts distance detection on all moving parts such as all links, joint housings, and grippers in sequence to achieve full-domain interference detection and eliminate hidden safety issues such as local links scraping against partitions or squeezing adjacent reagents.

[0057] 4. Exception handling and logging: When a candidate solution is rejected due to joint over-limit or collision risk, the system will not terminate the task directly. Instead, it will automatically mark the solution as invalid and continue to traverse other candidate joint angle combinations. If multiple solutions fail the verification, the system determines that there is a blind spot in the current grasping pose, immediately pauses the action, generates an error log, and pushes an alarm message to the upper terminal, waiting for manual intervention or replanning the grasping pose.

[0058] 5. Pre-collision prediction based on motion trajectory: In addition to static collision detection of the target joints, the system also performs dynamic pre-simulation of the complete motion trajectory of the robotic arm from its current position to the target position, performs distance verification at each trajectory point, and identifies dynamic interference risks that may occur during the motion in advance, achieving dual protection of "points and trajectories".

[0059] Preferably, in step S7, after the target reagent is placed at the weighing station, a liquid dispensing control command is generated. This command controls the precision pump to perform the liquid dispensing operation. Using real-time weight feedback data from the weighing station as the adjustment input, the driving parameters of the liquid dispensing control command are dynamically adjusted through a closed-loop control algorithm until the dispensing weight reaches the target threshold. Then, a cutoff control command is generated, including: S71. Calculate the target sampling weight Real-time weight feedback data from the weighing station Weight deviation between: , Set the trigger threshold for intercept control commands The trigger threshold for: , in, This refers to the residual fluid inertia in the pump pipe. To determine the resolution of the weighing station. This represents the allowable error ratio; , in, and These are the maximum and minimum duty cycles of the liquid dispensing control command, respectively. When the weight deviation Less than or equal to the trigger threshold At that time, a truncation control command is generated.

[0060] In the above embodiments, by calculating the sampling weight deviation in real time, setting a reasonable cutoff threshold in combination with parameters such as fluid inertia and equipment resolution, and dynamically adjusting the duty cycle of the drive signal according to the deviation, the pump operation status can be smoothly controlled according to the liquid extraction progress, offsetting the liquid extraction error caused by fluid inertia and reagent characteristics, thereby achieving high-precision quantitative liquid extraction and effectively improving the accuracy of micro-reagent extraction.

[0061] Based on actual laboratory conditions, the characteristics of different reagents, and system coordination logic, this step is further explained with an engineering-oriented approach: 1. Adaptive parameter adjustment for reagent properties: The system can adaptively correct the inertia based on the previously identified reagent type, viscosity, surface tension, and other physicochemical properties. Error ratio Key parameters such as these are adjusted. For high-viscosity reagents that easily adhere to the pipe walls, the inertia compensation value is appropriately increased; for low-flow-rate reagents, the pump drive rate is simultaneously slowed down to avoid problems such as excessive liquid dispensing and excessive residue in the tubing, thus adapting to the dispensing needs of different types of reagents.

[0062] 2. Multi-stage variable speed liquid dispensing control logic: The system adopts a hierarchical control strategy of "rapid liquid extraction - slow approach". When the weight deviation is large, a higher PWM duty cycle is used to improve the liquid extraction efficiency; as the actual weight approaches the target value, the duty cycle is gradually reduced and the pump speed is slowed down. Combined with a preset cutoff threshold, the pump is stopped precisely, balancing liquid extraction efficiency and micro-sampling accuracy.

[0063] 3. Multi-device closed-loop collaborative architecture: This step employs a closed-loop collaborative architecture using a precision pump and a high-precision electronic balance. The balance transmits weight data back to the control center at a high frequency in real time, forming a real-time feedback loop. Unlike the traditional open-loop constant-rate liquid sampling mode, this architecture can offset the sampling errors caused by changes in ambient temperature and reagent state in real time, making it particularly suitable for high-precision sampling scenarios at the microgram and microliter levels.

[0064] 4. Post-cutoff compensation and pipeline treatment: After the cutoff operation is performed, the system will add a short-delay pressure stabilization phase to wait for the residual fluid in the pipeline to completely recede before re-verifying the final sample weight. If there is a slight deviation, a micro-liquid replenishment operation can be initiated. At the same time, for volatile and easily crystallizing reagents, a simple pipeline purging will be automatically performed after the liquid is collected to prevent reagent residue from clogging the pipeline or causing deterioration.

[0065] 5. Anomaly Monitoring and Emergency Response: The system monitors the weight data change trend in real time throughout the liquid collection process. If any abnormal situation occurs, such as sudden weight change or no change in data for a long time, the system will immediately stop the pump and determine that the problem is caused by pipeline blockage, reagent depletion, balance failure, etc. Simultaneously, the abnormal log is recorded and alarm information is pushed to the upper terminal for manual investigation and handling.

[0066] 6. Data archiving linkage: After a single liquid sampling operation is completed, the system automatically binds data such as target weight, actual sample weight, operating parameters, and operation time to the corresponding reagent ledger, improving the data recording of the entire process and facilitating subsequent experimental traceability, reagent usage statistics, and ledger management.

[0067] Figure 3 This is a schematic diagram of the overall process of adaptive grasping and closed-loop quantitative liquid dispensing provided in an embodiment of the present invention; as shown below. Figure 3 As shown, the adaptive grasping and closed-loop quantitative liquid dispensing method of the present invention has the following specific execution steps: T1. Capture images using the global camera and extract drug attribute information using YOLOv5+OCR; T2. Calculate and allocate the optimal three-dimensional storage location based on drug attributes; T3. Trigger the liquid retrieval order, the robotic arm completes coarse positioning, and at the same time activates the end-effector Eye-in-Hand depth camera; T4. Perform coordinate system homogeneous transformation and inverse kinematics solution to achieve precise grasping of reagent bottles; T5. Introduce the Jerk constraint, plan the time-optimal smooth trajectory, and complete the reagent bottle handling; T6. Start the precision peristaltic pump to pump liquid, and use a high-precision electronic balance to collect weight data in real time at high frequency; T7. Determine if the weight deviation approaches zero; If the result is negative, the pump's PWM duty cycle is dynamically reduced using the PID algorithm, and the process returns to the real-time weight acquisition step. If the determination result is yes, the pump hard brake is triggered, the robotic arm is controlled to return the reagent bottle, and the entire liquid separation process ends.

[0068] Example 2: Figure 4 As shown, this embodiment of the invention also provides a reagent management system based on multimodal visual perception, including: The data acquisition module is used to acquire visual perception data of reagents to be stored through a multimodal vision acquisition unit deployed in the reagent storage system. The data processing module is used to perform target detection and character recognition processing on the visual perception data to obtain the structured attribute data and physical size data of the reagents to be put into the warehouse. The coordinate calculation module is used to calculate the optimal storage coordinates of the reagent to be stored in the reagent storage system coordinate system based on the structured attribute data, physical size data and pre-configured chemical safety isolation rules. The handling execution module is used to control the multi-degree-of-freedom robotic arm to move the reagents to be put into storage to the optimal storage location based on the optimal storage coordinates, and to synchronously update the reagent location status in the inventory database. The pose acquisition module is used to respond to the retrieval request by acquiring the real-time pose data of the target reagent through the vision acquisition unit mounted on the end of the robotic arm, and mapping the real-time pose data to the target grasping pose in the coordinate system of the robotic arm base based on the pre-calibrated coordinate system transformation parameters. The grasping control module is used to perform inverse kinematics calculations with the target grasping pose as the solution target, and generate grasping control instructions. The grasping control instructions are used to control the robotic arm to perform flexible grasping actions and transport the target reagent to the weighing station. The liquid dispensing control module is used to generate a liquid dispensing control command after the target reagent is placed at the weighing station. The liquid dispensing control command is used to control the precision pump to perform a liquid dispensing operation. The module uses the real-time weight feedback data of the weighing station as the adjustment input and dynamically adjusts the driving parameters of the liquid dispensing control command through a closed-loop control algorithm until the liquid dispensing weight reaches the target threshold. Then, a cutoff control command is generated to control the precision pump to perform a cutoff operation.

[0069] like Figure 5As shown, the reagent management system in this embodiment consists of a multimodal visual perception module (i.e., a multimodal visual acquisition unit, a data acquisition module, and a pose acquisition module), an intelligent warehousing and scheduling hub (i.e., a data processing module, a coordinate calculation module, and a grasping control module), an execution mechanism (i.e., a handling execution module and a liquid dispensing control module), and a closed-loop feedback module. The multimodal visual perception module includes a global monitoring camera and an eye-in-hand end-effector depth camera, responsible for acquiring video streams and point cloud data and obtaining end-effector pose. The intelligent warehousing and scheduling hub integrates YOLOv5 and an OCR recognition network, a robotic arm kinematics solver, and a 3D storage location scheduling algorithm to complete data parsing, coordinate calculation, trajectory planning, and control command generation. The execution mechanism includes a six-degree-of-freedom robotic arm, an adaptive electric gripper, and a precision peristaltic pump, performing grasping, transporting, and dispensing operations. The closed-loop feedback module relies on a high-precision electronic balance to achieve high-frequency weight feedback, forming a fully automated control system.

[0070] like Figure 6 As shown, the reagent management system in this embodiment adopts a horizontally layered data flow architecture, which can be divided into six functional modules: data acquisition module, data processing module, coordinate calculation module, handling execution module, pose acquisition module, grasping control module, and liquid dispensing control module. The two layers correspond one-to-one, fully realizing the entire process of visual perception, calculation and solution, execution and closed-loop regulation.

[0071] The architecture input includes two inputs: a global visual stream and an end-effector pose stream, which are implemented by the data acquisition module and the pose acquisition module, respectively. The global visual stream outputs raw pixel frames, and the end-effector pose stream uploads the end-effector pose data of the robotic arm in real time, providing the raw perception data source for this system. The AI ​​feature extraction layer is equipped with YOLOv5 and OCR algorithms, and is complemented by a preprocessing layer to complete data regularization. Together, they correspond to the data processing module to perform target detection and text recognition on image frames, and output structured data such as detection boxes (BBoxes) and drug text attributes. The logic calculation layer is divided into a multi-dimensional coordinate transformation matrix solution unit and a Jerk constraint trajectory planning unit, which correspond to the coordinate calculation module and the grasping control module, respectively: the coordinate transformation unit solves the coordinate system mapping matrix and solves the pose of the target relative to the robot arm base; the trajectory planning unit introduces jump constraints to generate a smooth optimal motion trajectory and outputs the joint target angle and motion control commands; The execution and feedback closed-loop area on the right includes a multi-degree-of-freedom actuator and a high-frequency feedback sensor, which are respectively connected to the transport execution module and the liquid dispensing control module: the actuator receives motion commands to complete the reagent grabbing and transfer operation; during the quantitative dispensing stage, the liquid dispensing control module drives the precision pump to work, while the high-frequency feedback sensor transmits real-time weight and contact torque data, generates task correction and closed-loop adjustment commands and sends them back to the logic computing layer to form a complete closed-loop control system.

[0072] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0073] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0074] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0075] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0076] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A reagent management method based on multi-modal visual perception, characterized in that, Includes the following steps: Visual perception data of reagents to be stored is collected by a multimodal vision acquisition unit deployed in the reagent storage system. The visual perception data is processed by target detection and character recognition to obtain the structured attribute data and physical size data of the reagents to be put into the warehouse; Based on the structured attribute data, physical size data, and pre-configured chemical safety isolation rules, calculate the optimal storage coordinates of the reagent to be stored in the reagent storage system coordinate system. The multi-degree-of-freedom robotic arm is controlled to move the reagents to be stored to the optimal storage location based on the optimal storage coordinates, and the reagent location status in the inventory database is updated synchronously. In response to a retrieval request, the real-time pose data of the target reagent is acquired by the vision acquisition unit mounted on the end of the robotic arm, and the real-time pose data is mapped to the target grasping pose in the coordinate system of the robotic arm base based on the pre-calibrated coordinate system transformation parameters. Inverse kinematics calculations are performed with the target grasping pose as the solution objective to generate grasping control commands. The grasping control commands are used to control the robotic arm to perform flexible grasping actions and transport the target reagent to the weighing station. After the target reagent is placed at the weighing station, a liquid dispensing control command is generated. The liquid dispensing control command is used to control the precision pump to perform a liquid dispensing operation. The driving parameters of the liquid dispensing control command are dynamically adjusted through a closed-loop control algorithm using real-time weight feedback data from the weighing station as the adjustment input. Once the liquid dispensing weight reaches the target threshold, a cutoff control command is generated. The cutoff control command is used to control the precision pump to perform a cutoff operation.

2. The reagent management method based on multimodal visual perception according to claim 1, characterized in that, Visual perception data of reagents to be stored is collected by a multimodal vision acquisition unit deployed in the reagent storage system, including: The multi-degree-of-freedom robotic arm is controlled to move the end effector to the preset scanning pose, and the RGB-D camera in the multimodal vision acquisition unit is triggered to acquire the initial color image and initial depth image of the reagent to be put into storage. The initial color image is subjected to adaptive histogram equalization, and the contour features of the reagent label region are extracted based on the improved Canny edge detection operator. The minimum bounding rectangle of the contour features is then calculated. Based on the pixel coordinates of the minimum bounding rectangle in the initial color image and the depth value corresponding to the initial depth image, calculate the three-dimensional spatial coordinates of the reagent to be stored in the camera coordinate system. If the confidence level of the three-dimensional spatial coordinates is lower than a preset threshold, a rescan control command is generated to control the robotic arm to drive the multimodal vision acquisition unit to perform relative displacement sampling along a preset spiral trajectory until visual perception data that meets the confidence level requirements is obtained. Wherein, the radial displacement increment of the spiral trajectory With rotation angle increment Satisfying Relationship: ， in, This is the sequence number of the current sampled frame. This is the radial step size coefficient. is the angular velocity coefficient, and , .

3. The reagent management method based on multimodal visual perception according to claim 1, characterized in that, The visual perception data is processed by target detection and character recognition to obtain the structured attribute data and physical size data of the reagents to be stored, including: The visual perception data is used to perform target detection, output the bounding box and the outline features of the label area of ​​the reagent to be put into the warehouse, and the label interest region is obtained by cropping based on the outline features. The region of interest of the label is sequentially processed with grayscale conversion, adaptive histogram equalization contrast enhancement, Gaussian filtering noise reduction and perspective correction to obtain a standardized label image. Text line detection is performed on the standardized label image to output a text detection box, and optical character recognition processing is performed on the character sequence within the text detection box to obtain the original text sequence containing the drug name, specifications, batch number and expiration date. Based on a predefined keyword library and regular expression matching rules, the original text sequence is parsed into corresponding field types to generate structured attribute data; Based on the pixel size of the bounding box and the pre-calibrated pixel equivalent, the bottle height and cross-sectional diameter of the reagent to be stored are calculated to generate physical size data; If the recognition confidence of the key field is lower than the preset threshold, a rescan mechanism is triggered, and a weighted voting fusion is performed based on the recognition results of multiple frames. The confidence score of the fusion result is then calculated. Satisfying Relationship: ， in, For the first Confidence level for frame image recognition For the first Frame image sharpness weighting, n The total number of image frames involved in the fusion, when the confidence score of the fusion result is... If the value is greater than or equal to a preset threshold, the structured attribute data is confirmed to be valid.

4. The reagent management method based on multimodal visual perception according to claim 1, characterized in that, Based on the structured attribute data, physical size data, and pre-configured chemical safety isolation rules, the optimal storage coordinates of the reagent to be stored in the reagent storage system coordinate system are calculated, including: Traverse the candidate storage locations within the reagent storage system, and filter out candidate storage locations that do not meet environmental constraints or are within the reach of the robotic arm based on the hazardous properties and physical size data of the reagents to be stored. Let the coordinates of the candidate storage bits that pass the filtering be... Calculate the Euclidean distance between the candidate storage location and the already stored incompatible reagent storage locations: ， If the Euclidean distance If so, then the candidate storage bit is removed, where, This refers to the minimum isolation distance determined based on the pre-configured chemical safety isolation rules. Calculate a comprehensive evaluation function for candidate storage sites that pass the chemical safety isolation verification: ， in, The path cost for the robotic arm to return from the candidate storage location to the weighing station. The cost of matching candidate storage space volume with physical size data. Penalties are based on the level of chemical hazard. Historical frequency of reagent use for each category , , , These are preset weighting coefficients; Select a comprehensive evaluation function The smallest candidate storage location is identified, and the coordinates of the smallest candidate storage location are determined as the optimal storage coordinates for the reagents to be stored.

5. The reagent management method based on multimodal visual perception according to claim 1, characterized in that, Controlling a multi-degree-of-freedom robotic arm to move reagents to the optimal storage location based on the optimal storage coordinates includes: Based on the optimal storage coordinates, a collision-free path is constructed for the robotic arm from the initial pose to the target storage pose; A time-optimal trajectory planning algorithm is used to generate motion trajectories in joint space on the collision-free path, and the motion trajectories satisfy the following constraints: ， in, , and The first The velocity, acceleration, and jump of each joint , and These are the preset maximum speed, maximum acceleration, and maximum jump thresholds, respectively; Based on the generated motion trajectory, the handling control command is generated to control the multi-degree-of-freedom robotic arm to transport the reagents to be stored to the optimal storage location.

6. The reagent management method based on multimodal visual perception according to claim 1, characterized in that, In response to a retrieval request, the real-time pose data of the target reagent is acquired through a vision acquisition unit mounted on the end effector of the robotic arm. Based on pre-calibrated coordinate system transformation parameters, the real-time pose data is mapped to a target grasping pose in the robotic arm's base coordinate system, including: Two-dimensional and depth images of the target reagent are acquired by a vision acquisition unit mounted on the end of the robotic arm, and the contour features of the target reagent are extracted based on the improved Canny edge detection operator, and the minimum bounding rectangle of the contour features is calculated. Based on the pixel coordinates of the minimum bounding rectangle in the two-dimensional image and the depth value corresponding to the depth image, the real-time three-dimensional coordinates and attitude angles of the target reagent in the visual acquisition unit coordinate system are calculated to generate the real-time pose data. Based on the pre-calibrated hand-eye transformation matrix and tool calibration matrix, and combined with the current joint angle data of the robotic arm, the real-time pose data is mapped to the target grasping pose in the robotic arm base coordinate system, wherein the mapping relationship satisfies the following homogeneous coordinate transformation formula: ， in, To obtain the homogeneous transformation matrix of the target's pose in the robot arm's base coordinate system. This is the pose matrix of the robot arm's end flange in the base coordinate system, calculated using forward kinematics based on the current joint angle data. The hand-eye transformation matrix of the vision acquisition unit relative to the end flange. This represents the pose matrix of the target reagent relative to the visual acquisition unit. If the confidence level of the real-time pose data is lower than a preset threshold, a rescan control command is generated to control the robotic arm to drive the vision acquisition unit to perform relative displacement sampling along a preset spiral trajectory until real-time pose data that meets the confidence level requirements is obtained.

7. The reagent management method based on multimodal visual perception according to claim 6, characterized in that, Inverse kinematics calculations are performed using the target grasping pose as the solution objective to generate grasping control commands, including: The homogeneous transformation matrix corresponding to the target grasp pose. To solve for the objective, a pose error function is constructed that includes both position error and attitude error: ， in, and These are the current position vector and attitude vector of the robotic arm's end effector, respectively. and These are the position vector and attitude vector of the target capture pose, respectively. These are the attitude error weighting coefficients; The inverse kinematics are solved iteratively using the damped least squares method, and the joint angle vectors are updated using the following iterative formula: ， in, For the first The joint angle vector of the next iteration. For Jacobian matrices, This is the pose synthesis error vector. The damping coefficient is... It is the identity matrix; During the iteration process, joint limit verification and collision detection are performed simultaneously. For candidate solutions that pass the verification, the solution with the smallest change in the current joint angle vector and which satisfies the collision constraint is selected as the target joint angle vector, and the grasping control command is generated.

8. The reagent management method based on multimodal visual perception according to claim 7, characterized in that, During the iteration process, joint limit verification and collision detection are performed simultaneously, including: The robot arm's joint angles are calculated based on the current candidate solutions. If any joint angle exceeds the preset joint angle range, the current candidate solution is discarded. Based on the current candidate solution, the pose of the robotic arm link is calculated using forward kinematics, and the pose is then compared with a pre-constructed obstacle geometry model for distance detection. The obstacle geometry model includes at least shelf partitions and adjacent reagent bottles. If the shortest distance between the robotic arm link and the obstacle geometry is less than a preset safety threshold, interference is determined and the current candidate solution is eliminated.

9. The reagent management method based on multimodal visual perception according to claim 1, characterized in that, After the target reagent is placed at the weighing station, a liquid dispensing control command is generated. This command controls the precision pump to perform the liquid dispensing operation. Using real-time weight feedback data from the weighing station as the adjustment input, a closed-loop control algorithm dynamically adjusts the driving parameters of the liquid dispensing control command until the dispensing weight reaches the target threshold. Then, a cutoff control command is generated, including: Calculate the target sample weight Real-time weight feedback data from the weighing station Weight deviation between: ， Set the trigger threshold for intercept control commands The trigger threshold for: ， in, This refers to the residual fluid inertia in the pump pipe. To determine the resolution of the weighing station. This represents the allowable error ratio; ， in, and These are the maximum and minimum duty cycles of the liquid dispensing control command, respectively. When the weight deviation Less than or equal to the trigger threshold At that time, a truncation control command is generated.

10. A reagent management system based on multimodal visual perception, characterized in that, include: The data acquisition module is used to acquire visual perception data of reagents to be stored through a multimodal vision acquisition unit deployed in the reagent storage system. The data processing module is used to perform target detection and character recognition processing on the visual perception data to obtain the structured attribute data and physical size data of the reagents to be put into the warehouse. The coordinate calculation module is used to calculate the optimal storage coordinates of the reagent to be stored in the reagent storage system coordinate system based on the structured attribute data, physical size data and pre-configured chemical safety isolation rules. The handling execution module is used to control the multi-degree-of-freedom robotic arm to move the reagents to be put into storage to the optimal storage location based on the optimal storage coordinates, and to synchronously update the reagent location status in the inventory database. The pose acquisition module is used to respond to the retrieval request by acquiring the real-time pose data of the target reagent through the vision acquisition unit mounted on the end of the robotic arm, and mapping the real-time pose data to the target grasping pose in the coordinate system of the robotic arm base based on the pre-calibrated coordinate system transformation parameters. The grasping control module is used to perform inverse kinematics calculations with the target grasping pose as the solution target, and generate grasping control instructions. The grasping control instructions are used to control the robotic arm to perform flexible grasping actions and transport the target reagent to the weighing station. The liquid dispensing control module is used to generate a liquid dispensing control command after the target reagent is placed at the weighing station. The liquid dispensing control command is used to control the precision pump to perform a liquid dispensing operation. The module uses the real-time weight feedback data of the weighing station as the adjustment input and dynamically adjusts the driving parameters of the liquid dispensing control command through a closed-loop control algorithm until the liquid dispensing weight reaches the target threshold. Then, a cutoff control command is generated to control the precision pump to perform a cutoff operation.

Images