A cross-modal drug recognition engine and biomimetic grasping system and method

Abstract

This application provides a cross-modal drug recognition engine and a bionic grasping system and method, applied in the field of data processing technology. The cross-modal drug recognition engine forms a closed-loop chain from multimodal data acquisition, temporal alignment, constraint modeling, conflict identification, credibility reconstruction to result output and historical record updates. The collaborative work of these modules solves the problems of disconnect between the recognition and execution systems, inability to use historical information to verify current data, and inability to form a data closed loop in existing technologies. This provides comprehensive assurance for the overall operational efficiency and data traceability of intelligent drug storage systems. The bionic grasping method solves the problems of setting a single threshold for a single dimension, inability to distinguish the severity of deviations, and uniform processing of all deviations in existing technologies. It enables the bionic hand to perform targeted adjustments based on actual deviations, significantly improving the grasping success rate and operational safety.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a cross-modal drug recognition engine and a biomimetic grasping system and method. Background Technology

[0002] With the continuous expansion of the pharmaceutical distribution scale, the traditional warehousing management model relying on manual sorting can no longer meet the requirements of efficiency and safety. For example, in the management of high-risk products such as narcotic drugs, the error rate of manual operation may lead to serious medical safety accidents. To address this, the industry has proposed an intelligent drug grasping system based on cross-modal recognition, aiming to achieve unmanned and precise operations in the pharmaceutical warehousing process and reduce the safety risks brought about by human intervention.

[0003] In existing technologies, during the recognition phase, visual images of the target drug and surrounding drugs are acquired and processed to output information about the target drug and the location coordinates of neighboring drugs. During the grasping phase, the system receives information from the recognition module to generate grasping parameters and controls the bionic hand to perform the grasping action.

[0004] However, in existing technologies, cross-modal drug recognition engines typically only perform simple data concatenation or single-modal priority (such as image priority or code priority) after acquiring multimodal data such as images, codes, and storage locations. Therefore, when there are conflicts between different modal data (for example, the image recognition result is drug A, the code scan result is drug B, or the current location does not match the storage location of the drug in the historical operation record), existing engines cannot identify the conflict type (whether it is an image anomaly, a code anomaly, or a location anomaly), nor can they replace and reconstruct the conflicting parts based on other modal data. They can only output low-confidence recognition results or directly report errors, causing downstream crawling... The system cannot obtain reliable recognition results. During the grasping process, because the existing grasping methods only set a single deviation threshold for a single dimension (such as positional deviation) and only judge whether the threshold is exceeded, when the grasping execution status information (such as the actual pose of the drug, the actual position of the path, and the actual value of the clamping force) deviates from the preset value, the existing methods cannot simultaneously compare the offset of the three dimensions of pose, path, and clamping force, nor can they distinguish the severity of the deviation (whether it is a slight touch of the allowable range boundary or a serious exceedance of the safety range). This results in all deviations being processed in a uniform manner, making it difficult for the bionic hand to perform targeted adjustment operations based on the actual deviation. Summary of the Invention

[0005] To achieve the above objectives, this application adopts the following technical solution: This application provides a cross-modal drug recognition engine, which may include: The data acquisition module is used to acquire input data of the target drug, which includes at least image data, encoded data, and storage location data. The storage module is used to store historical operation records; The constraint module is used to construct a first correspondence between the image data and the encoded data, a second correspondence between the encoded data and the warehouse location data, and a third correspondence between the warehouse location data and the historical operation records based on the input data and the historical operation records, and to encapsulate the first correspondence, the second correspondence, and the third correspondence into a cross-modal association constraint chain; The consistency verification and reconstruction module is used to perform consistency verification on the cross-modal association constraint chain, obtain the verification result, generate a conflict state identifier when the verification result is inconsistent, generate an identification result based on the cross-modal association constraint chain, and reconstruct the credibility of the identification result based on the conflict state identifier to obtain a credible identification result. The output module is used to output the reliable identification result and update the historical operation record.

[0006] A biomimetic grasping method, utilizing the aforementioned cross-modal drug recognition engine, may include: Obtain a reliable identification result, and generate an initial grasping control command based on the reliable identification result; The initial grasping control command is sent to the bionic hand to control the bionic hand to perform a bionic grasping operation on the target drug. During the process of the bionic hand performing a bionic grasping operation on the target drug, the grasping execution status information is acquired in real time, and the grasping execution status information is compared with the preset grasping constraints to obtain the comparison result; An execution instruction is generated based on the comparison results and sent to the bionic hand to control the bionic hand to perform a graded response grasping operation on the target drug. After the bionic hand grasps the target medicine, the bionic hand pose information and the medicine placement coordinate information are obtained; Based on the bionic hand pose information, the reliable recognition result, and the drug placement coordinate information, a placement control command is generated; The placement control command is sent to the bionic hand, which controls the bionic hand to place the target drug in the target area.

[0007] A biomimetic grasping system, the system may include: Cross-modal drug recognition engine, instruction generation module, bionic hand execution module, state detection module, comparison result determination module, hierarchical response module, and placement control module; The cross-modal drug recognition engine is used to acquire input data of the target drug and output a reliable recognition result; The instruction generation module is used to obtain the reliable identification result output by the cross-modal drug recognition engine, and generate an initial grasping control instruction based on the reliable identification result; The bionic hand execution module is used to send the initial grasping control command to the bionic hand and control the bionic hand to perform a bionic grasping operation on the target drug; The state detection module is used to acquire grasping execution status information in real time during the process of the bionic hand performing a bionic grasping operation on the target drug; The comparison result determination module is used to compare the crawling execution status information with the preset crawling constraints to obtain the comparison result; The graded response module is used to generate an execution instruction based on the comparison result and send the execution instruction to the bionic hand to control the bionic hand to perform a graded response grasping operation on the target drug. The placement control module is used to acquire the bionic hand pose information and drug placement coordinate information after the bionic hand grasps the target drug, generate a placement control command based on the bionic hand pose information, the reliable recognition result and the drug placement coordinate information, and send the placement control command to the bionic hand to control the bionic hand to place the target drug in the target area.

[0008] As can be seen from the above technical solution, this application has the following beneficial effects: 1. This application constructs three correspondences between image data and coded data, coded data and warehouse location data, and warehouse location data and historical operation records, and encapsulates these three correspondences into a cross-modal association constraint chain, thereby achieving mutual verification and constraint modeling of multimodal data. Based on this, by performing item-by-item consistency checks on the cross-modal association constraint chain, when the check result is inconsistent, the conflict type (abnormal appearance identification information, abnormal coded identification information, or abnormal warehouse location data) can be accurately located. Based on the conflict type, the system uses reliable data from the other two correspondences to replace and reconstruct the abnormal parts in the identification result, enabling the system to output high-confidence identification results even when single-modal data is abnormal. This mechanism solves the problems in existing technologies where the source of conflict cannot be identified and other modal data cannot be used for correction when multimodal data conflicts occur, significantly improving the reliability of drug identification and providing accurate and reliable identification basis for downstream grasping systems.

[0009] 2. This application aligns multimodal data along the time dimension using a unified timestamp through a data acquisition module, maintains traceable historical operation records through a storage module, constructs a mutual verification constraint network between images, codes, locations, and historical records through a constraint module, accurately locates conflict sources through a consistency verification and reconstruction module and uses other modal data for replacement and reconstruction, and outputs the credible identification results to the downstream system and synchronously updates the historical operation records through an output module. This forms a closed-loop system covering the entire chain from multimodal data acquisition, time-series alignment, constraint modeling, conflict identification, credibility reconstruction to result output and historical record updates. The collaborative work of these modules solves the problems of disconnect between the identification system and the execution system, the inability to use historical information to verify current data, and the inability to form a data closed loop in existing technologies, providing comprehensive assurance for the overall operational efficiency and data traceability of intelligent pharmaceutical warehousing systems.

[0010] 3. In the process of performing bionic grasping operations, this application acquires the actual pose information, path position information, and clamping force information of the target drug in real time. The pose offset, path offset, and clamping force deviation are compared with preset grasping constraints to obtain pose offset results, path offset results, and clamping force deviation results, respectively. Based on the combination of these three offset results, the grasping execution status is determined to be normal, slightly deviated, or severely deviated. Corresponding execution instructions are generated based on the comparison results: execution continues when normal; adjustment instructions are generated for targeted fine-tuning when there is a slight deviation; and interruption and replanning instructions are generated when there is a severe deviation. This mechanism, by simultaneously monitoring the offsets in the three dimensions of pose, path, and clamping force, and classifying the comparison results into three levels (less than, equal to, and greater than), achieves multi-dimensional, refined monitoring and graded response of the grasping process. This solves the problems of existing technologies that only set a single threshold for a single dimension, cannot distinguish the severity of deviations, and use uniform processing for all deviations. It enables the bionic hand to perform targeted adjustments based on the actual deviation, significantly improving the grasping success rate and operational safety.

[0011] 4. This application transforms high-level drug identification information into low-level motion control commands executable by the bionic hand by acquiring reliable identification results and generating initial grasping control commands based on these results. During the grasping operation, the application acquires grasping execution status information in real time and performs multi-dimensional deviation comparison and graded response. After successful grasping, the application acquires the bionic hand's pose information and drug placement coordinate information, generating placement control commands to control the bionic hand to accurately place the target drug in the target area, forming a complete closed loop from identification, grasping, monitoring, graded response to placement. Specifically, for slight deviations, adjustment commands are generated to fine-tune the dimensions touching the boundary; for severe deviations, interruption and replanning commands are generated and the grasping operation is re-executed. This mechanism achieves adaptive closed-loop control of the grasping process, enabling the bionic hand to autonomously adjust its grasping strategy based on real-time state deviations. This solves the problem in existing technologies where the grasping process cannot be dynamically adjusted according to actual deviations, and grasping fails as soon as a deviation occurs, thus improving the adaptive capability and operational reliability of the intelligent drug storage system.

[0012] 5. This application establishes a biomimetic grasping system that forms a complete closed loop from drug identification, grasping execution, multi-dimensional status monitoring, hierarchical response to precise placement, thereby achieving adaptive control of the drug grasping process. This solves the problems in existing technologies that cannot distinguish the severity of deviations and use a uniform processing method for all deviations, thus improving the grasping success rate and operational reliability. Attached Figure Description

[0013] The present application will be further described below with reference to the accompanying drawings.

[0014] Figure 1 Example diagram of a cross-modal drug recognition engine provided in this application; Figure 2 A flowchart of a biomimetic grasping method provided in this application; Figure 3 An example diagram of a biomimetic grasping system provided in this application. Detailed Implementation

[0015] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are used to distinguish different objects, not to limit a specific order.

[0016] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0017] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first: Cross-modal drug identification technology refers to the technique of integrating multi-source data, such as visual images, barcodes or QR codes, and warehouse location information, to comprehensively identify and locate target drugs. Image data is used to extract the drug's appearance features (such as shape, color, and text labels), encoded data is used to read the drug's unique identification information (such as drug code and batch number), and location data is used to confirm the drug's spatial coordinates on warehouse shelves. After collection, the aforementioned multi-modal data is typically processed through data stitching or feature fusion to output the drug identification result.

[0018] Bionic grasping technology refers to automated execution technology that simulates the grasping action of a human hand. It typically includes grasping path planning, gripping force control, and posture adjustment. Based on the drug's position and shape information output by the recognition module, the system generates grasping control commands, controlling the bionic hand to move along a preset path to the target position, grasp the drug with a preset gripping force, and place the drug in the target area according to the planned path. During the grasping process, the system monitors the grasping execution status information in real time and compares the monitored values ​​with preset values ​​to determine whether the grasping process is executing normally.

[0019] Research has revealed that in existing technologies, cross-modal drug recognition engines typically perform only simple data concatenation or single-modal priority (such as image priority or code priority) after acquiring multimodal data such as images, codes, and storage locations. Therefore, when conflicts exist between different modalities (e.g., image recognition result is drug A, code scanning result is drug B, or the current location does not match the storage location of the drug in historical operation records), existing engines cannot identify the conflict type (image anomaly, code anomaly, or location anomaly) and cannot replace and reconstruct the conflicting parts based on other modal data. They can only output low-confidence recognition results or directly report errors, causing problems for downstream applications. The grasping system cannot obtain reliable recognition results. During the grasping process, because the existing grasping methods only set a single deviation threshold for a single dimension (such as positional deviation) and only judge whether the threshold is exceeded, when the grasping execution status information (such as the actual pose of the drug, the actual position of the path, and the actual value of the clamping force) deviates from the preset value, the existing methods cannot simultaneously compare the offset of the three dimensions of pose, path, and clamping force, nor can they distinguish the severity of the deviation (whether it is a slight touch of the allowable range boundary or a serious exceedance of the safety range). This results in all deviations being processed in a uniform manner, making it difficult for the bionic hand to perform targeted adjustment operations based on the actual deviation.

[0020] Example 1: To address the above problems, this application provides a cross-modal drug recognition engine, including a data acquisition module, a storage module, a constraint module, a consistency verification and reconstruction module, and an output module. Please refer to [link to example]. Figure 1 .

[0021] The data acquisition module is used to acquire input data for the target drug. The input data includes at least image data, coding data, and warehouse location data.

[0022] The storage module is used to store historical operation records.

[0023] In this embodiment, the data acquisition module is communicatively connected to multi-source acquisition devices deployed in the pharmaceutical storage area. Image data is obtained by capturing two-dimensional images of the target drug and surrounding drugs using an industrial camera at a preset shooting station; encoded data is obtained by reading the barcode on the surface of the target drug using a barcode scanner, and the encoded data contains the drug's unique identification information; storage location data is obtained through the location coordinate information recorded in the storage management system, which is stored in the form of three-dimensional spatial coordinates of shelf number, layer number, and column number. The data acquisition module associates the acquired image data, encoded data, and storage location data according to a unified timestamp to form input data bound to the current operation task, and transmits the input data to the constraint module for subsequent processing.

[0024] The storage module is deployed as a database on a local server to store historical operation records. These records include at least the input data collected for each drug identification task, the cross-modal association constraint chain, the verification results, the reliable identification results, and the operation timestamp. Each historical operation record is indexed using the drug's unique identifier as the primary key. The storage module also provides historical storage location data for the corresponding drug in the historical operation records when the constraint module constructs the third-party correspondence, supporting consistency verification between the storage location data in the current input data and the historical storage location.

[0025] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant terms is given first: Input data refers to the data set that is collected and packaged by the data acquisition module and then transmitted to the constraint module. It includes at least image data, coded data, and warehouse location data, and is accompanied by a unified timestamp.

[0026] Image data refers to two-dimensional images of the target drug and surrounding drugs acquired through visual acquisition equipment such as industrial cameras, used to extract the appearance features of the drug.

[0027] Encoded data refers to the barcode information on the surface of a target drug obtained through a barcode scanner, which contains the drug's unique identification information.

[0028] Warehouse location data refers to the spatial coordinate information of the target drug on the warehouse shelf, represented in the form of three-dimensional coordinates of shelf number, layer number, and column number, and is used to locate the physical storage location of the drug.

[0029] A unified timestamp refers to the same time stamp attached to image data, coded data, and warehouse location data collected in the same operation task by the data acquisition module, which is used to ensure the alignment of multimodal data in the time dimension.

[0030] Historical operation records refer to historical drug identification task data stored in the database, including input data collected in each task, cross-modal association constraint chains, verification results, reliable identification results, and operation timestamps, which are used to support the consistency verification between the current storage location data and the historical storage location.

[0031] Unique drug identification information refers to the information obtained from the coded data that is used to uniquely identify a drug, usually a drug code or drug traceability code.

[0032] Historical warehouse location data refers to the warehouse location data corresponding to a certain drug in a historical identification task, extracted from historical operation records, and used for consistency comparison with the current warehouse location data.

[0033] For example, when the system initiates a drug identification task, the data acquisition module first triggers an industrial camera to photograph the location of the target drug, acquiring a color image containing the target drug and surrounding drugs as image data; simultaneously, it triggers a barcode scanner to scan the barcode on the surface of the target drug, parsing it to obtain the drug's unique identification information as encoding data; and it queries the warehouse management system for the shelf number, layer number, and column number of the target drug's current location as warehouse location data. The data acquisition module appends the task number and a unified timestamp to the above three types of data, encapsulates them as input data, and transmits the input data to the constraint module.

[0034] The storage module loads the historical operation record database during system initialization. When the constraint module needs to construct a third correspondence, the storage module retrieves the historical storage location data of the most recent successful identification of the drug from the historical operation records based on the unique drug identification information in the current input data, and provides it to the constraint module for consistency comparison with the current storage location data.

[0035] It should be noted that the data acquisition module aligns three types of heterogeneous data—image data, encoded data, and warehouse location data—in the time dimension using a unified timestamp, providing a temporally consistent input foundation for subsequently constructing cross-modal association constraint chains. The storage module, by maintaining traceable historical operation records, enables the system to verify the rationality of current warehouse location data using historical warehouse location data, providing historical data support for conflict detection and credibility reconstruction of the identification results. Through the collaboration of these two modules, the system can ensure the temporal alignment of multimodal data and the traceability of historical information from the source, laying a data foundation for improving the reliability and accuracy of drug identification.

[0036] The constraint module is used to construct a first correspondence between image data and encoded data, a second correspondence between encoded data and warehouse location data, and a third correspondence between warehouse location data and historical operation records based on input data and historical operation records, and encapsulates the first, second, and third correspondences into a cross-modal association constraint chain.

[0037] In this embodiment, the constraint module receives input data from the data acquisition module and reads historical operation records from the storage module. The constraint module first preprocesses the image data, encoded data, and warehouse location data in the input data: It normalizes the image data, unifying images of different sizes and under different lighting conditions to a preset size and pixel value range; it unifies the format of the encoded data, parsing different encoding formats into a unified string format; and it standardizes the coordinates of the warehouse location data, converting location data from different coordinate systems into standard spatial coordinates with shelf number, layer number, and column number as dimensions. After preprocessing, the constraint module extracts features from the preprocessed image data, extracting visual features such as the shape outline, color distribution, and text markings of the medicine to obtain appearance identification information; it also parses the preprocessed encoded data to extract the unique identifier code of the medicine, obtaining coded identification information. Then, the constraint module associates and matches the appearance identification information and the coded identification information to establish a first correspondence between image data and coded data; it associates and matches the coded identification information with the preprocessed storage location data to establish a second correspondence between coded data and storage location data; and it associates and matches the preprocessed storage location data with the historical operation records provided by the storage module, querying the historical storage location data of the drug in the historical operation records to establish a third correspondence between storage location data and historical operation records. Finally, the constraint module encapsulates the first, second, and third correspondences in a structured manner to form a cross-modal association constraint chain, and transmits the cross-modal association constraint chain to the consistency verification and reconstruction module.

[0038] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant terms is given first: Normalization is a process of unifying image data of different sizes and under different lighting conditions into a preset size and pixel value range to eliminate differences in the image acquisition process.

[0039] Format unification processing refers to the process of parsing different encoding formats into a unified string format, which is used to eliminate differences in encoded data formats.

[0040] Coordinate standardization refers to the process of converting warehouse location data from different coordinate systems into standard spatial coordinates with shelf number, layer number, and column number as dimensions, in order to eliminate differences in location data representation.

[0041] Appearance identification information refers to the visual features of a drug, such as its shape outline, color distribution, and text markings, extracted from image data, which are used to characterize the appearance of the drug.

[0042] Encoding and identification information refers to the unique identification code of a drug obtained from the coded data, which is used to represent the identity information of the drug.

[0043] The first correspondence refers to the association and matching relationship between image data and coded data, which is used to indicate that the appearance identification information and the coded identification information should point to the same drug.

[0044] The second correspondence refers to the association and matching relationship between the coded data and the storage location data, which is used to indicate that the medicine corresponding to the coded identification information should be stored in the current storage location.

[0045] The third correspondence refers to the association and matching relationship between warehouse location data and historical operation records, which is used to indicate that the current warehouse location should be consistent with the historical storage location of the drug.

[0046] Cross-modal association constraint chains refer to a set of constraint relationships formed by structuring and encapsulating the first, second, and third correspondence relationships, which are used to characterize the mutual verification relationships between multimodal data.

[0047] For example, when the constraint module receives input data, the image data contained in the input data is a 224×224 pixel color image, the encoded data is the barcode scan result "6901234567890", and the warehouse location data is "Area A-03 Shelf-02 Layer-05 Column".

[0048] The constraint module first normalizes the image data by scaling the image to a preset size and normalizing the pixel values; it then unifies the format of the encoded data by converting the barcode scanning results into the standard string format "6901234567890"; and finally standardizes the coordinates of the warehouse location data by converting it into the format "Area A-03 Shelf-02 Layer-05 Column".

[0049] Then, the constraint module extracts features from the preprocessed image data, identifies the appearance of the drug as "white round tablets with the word 'ABC' printed on them", and obtains the appearance identification information; it also parses the preprocessed coded data and extracts the drug code "6901234567890" to obtain the coded identification information.

[0050] The constraint module associates the appearance identification information with the code identification information to establish a first correspondence (the appearance "white round tablet, ABC" corresponds to the code "6901234567890"); it associates the code identification information with the storage location data to establish a second correspondence (the code "6901234567890" corresponds to the location "Area A-03 shelf-02 layer-05 column"); it queries the historical operation records of this code from the storage module to obtain the most recent storage location of the drug as "Area A-03 shelf-02 layer-03 column", and associates the current storage location data with the historical storage location data to establish a third correspondence (the current location "Area A-03 shelf-02 layer-05 column" corresponds to the historical location "Area A-03 shelf-02 layer-03 column").

[0051] The constraint module encapsulates the above three correspondences into a cross-modal association constraint chain and transmits it to the consistency verification and reconstruction module.

[0052] It should be noted that the constraint module eliminates the differences in format and representation among the three types of heterogeneous data—images, codes, and locations—through preprocessing operations, providing a unified data foundation for subsequent association and matching. By constructing a first, second, and third correspondence, a mutually verifying constraint network is established between image data, code data, warehouse location data, and historical operation records. By encapsulating the three correspondences into a cross-modal association constraint chain, a complete verification basis is provided for subsequent consistency verification, enabling the system to identify conflict types between multimodal data and providing a traceable association path for credibility reconstruction. The above mechanisms ensure that multimodal data has formed structured constraint relationships before entering the verification stage, providing core support for improving the reliability and accuracy of drug identification.

[0053] The consistency verification and reconstruction module is used to perform consistency verification on the cross-modal association constraint chain, obtain the verification result, generate a conflict state identifier when the verification result is inconsistent, generate the identification result based on the cross-modal association constraint chain, and reconstruct the credibility of the identification result based on the conflict state identifier to obtain a credible identification result.

[0054] In this embodiment, the consistency verification and reconstruction module receives the cross-modal associated constraint chain from the constraint module. This module includes a consistency verification unit and a reconstruction unit.

[0055] The consistency verification unit first verifies the first correspondence in the cross-modal association constraint chain by comparing the appearance identification information and the code identification information in the first correspondence to determine whether they point to the same drug, thus obtaining a first result. It then verifies the second correspondence by comparing the code identification information in the second correspondence with the storage location data to determine whether the drug corresponding to the code identification should be stored in the current storage location, thus obtaining a second result. Finally, it verifies the third correspondence by comparing the storage location data in the third correspondence with the historical storage location data in the historical operation records to determine whether the current storage location is consistent with the historical storage location of the drug, thus obtaining a third result.

[0056] The consistency verification unit generates verification results according to the preset conflict determination rules: when at least one of the first, second, and third results is "different", the verification result is determined to be inconsistent; when the first, second, and third results are all "same", the verification result is determined to be consistent.

[0057] When the verification results are consistent, the reconstruction unit directly generates the recognition result based on the cross-modal association constraint chain, determines the recognition result as a reliable recognition result, and transmits it to the output module.

[0058] When the verification results are inconsistent, the reconstruction unit first determines the conflict type based on the first result, the second result, and the third result, and generates a conflict status identifier carrying the conflict type.

[0059] Specifically: when the first result is "different", the conflict type is abnormal appearance identification information; when the second result is "different", the conflict type is abnormal coding identification information; when the third result is "different", the conflict type is abnormal warehouse location data. The reconstruction unit generates the identification result based on the cross-modal association constraint chain and determines the abnormal part in the identification result according to the conflict type.

[0060] The reconstruction unit replaces the abnormal parts: when the conflict type is abnormal appearance identification information, the corresponding coded identification information and storage location data are extracted from the second correspondence and the third correspondence. The correct appearance identification information is determined based on the coded identification information and storage location data, and the abnormal appearance identification information in the identification result is replaced. When the conflict type is abnormal coding identification information, the corresponding appearance identification information and storage location data are extracted from the first correspondence and the third correspondence. The correct coding identification information is determined based on the appearance identification information and storage location data, and the abnormal coding identification information in the identification result is replaced. When the conflict type is abnormal warehouse location data, the corresponding appearance identification information and code identification information are extracted from the first correspondence and the second correspondence. The correct warehouse location data is determined based on the appearance identification information and code identification information, and the abnormal warehouse location data in the identification result is replaced.

[0061] After the replacement is completed, the reconstruction unit determines the reconstructed recognition result as a reliable recognition result and transmits it to the output module.

[0062] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant terms is given first: A consistency verification unit is a processing unit that performs consistency comparisons on the three correspondences in a cross-modal association constraint chain to determine whether the information in each correspondence matches each other.

[0063] A reconstruction unit is a processing unit that replaces and reconstructs the abnormal parts in the recognition result according to the conflict type when the verification result is inconsistent, in order to output a high-confidence recognition result.

[0064] The first result refers to the result obtained after verifying the consistency between the appearance identification information and the coding identification information in the first correspondence, which is used to determine whether the image recognition and coding scanning are consistent.

[0065] The second result refers to the result obtained after verifying the consistency between the coded identification information and the warehouse location data in the second correspondence, which is used to determine whether the coded information matches the warehouse location.

[0066] The third result refers to the result obtained after verifying the consistency between the warehouse location data and historical operation records in the third correspondence relationship, which is used to determine whether the current warehouse location is consistent with the historical storage location.

[0067] The preset conflict determination rule refers to the pre-set determination logic used to generate the verification result based on the first result, the second result, and the third result. Specifically, when at least one result is "different", the verification result is inconsistent; when all three results are "same", the verification result is consistent.

[0068] Conflict status identifier refers to the identifier information generated based on the conflict type determined by the first, second, and third results when the verification results are inconsistent. It is used to indicate whether the source of the anomaly is appearance identifier information, coding identifier information, or warehouse location data.

[0069] Conflict type refers to the type of anomaly source determined based on the specific correspondence of "differences" in the verification results, including anomalies in appearance identification information, anomalies in coding identification information, and anomalies in warehouse location data.

[0070] The identification result refers to the drug identification information generated based on the cross-modal association constraint chain, which includes appearance identification information, coding identification information and storage location data.

[0071] Reliable identification results refer to high-confidence identification results that are directly output when the verification results are consistent after consistency verification, or output after reconstruction by replacing the abnormal parts when the verification results are inconsistent.

[0072] For example, when the consistency verification and reconstruction module receives a cross-modal association constraint chain, the constraint chain includes: a first correspondence (the appearance "white round pill, ABC" corresponds to the code "6901234567890"), a second correspondence (the code "6901234567890" corresponds to the position "Area A-03 Shelf-02 Layer-05 Column"), and a third correspondence (the current position "Area A-03 Shelf-02 Layer-05 Column" corresponds to the historical position "Area A-03 Shelf-02 Layer-03 Column").

[0073] The consistency verification unit first verifies the first correspondence: the appearance identification information and the code identification information both point to the same drug, so the first result is "same"; it verifies the second correspondence: the drug corresponding to the code identification information should indeed be stored in this location, so the second result is "same"; it verifies the third correspondence: the current storage location "Area A-03 Shelf-02 Layer-05" is inconsistent with the historical storage location data "Area A-03 Shelf-02 Layer-03", so the third result is "different".

[0074] According to the preset conflict determination rules, the third result is "different", thus determining the verification result to be inconsistent. Based on the third result being "different", the reconstruction unit determines the conflict type to be warehouse location data anomaly and generates a conflict status identifier carrying "warehouse location data anomaly".

[0075] The reconstruction unit generates an identification result based on a cross-modal association constraint chain. This result contains abnormal location data "Area A-03 Shelf-02-05". Since the conflict type is abnormal warehouse location data, the reconstruction unit extracts the appearance identification information "white round pill, ABC" and the coding identification information "6901234567890" from the first and second correspondences. Based on the appearance identification information and the coding identification information, it determines that the correct warehouse location data should be "Area A-03 Shelf-02-03" in the historical operation record. The abnormal location data in the identification result is then replaced with the correct location data, resulting in a reliable identification result, which is then transmitted to the output module.

[0076] It should be noted that the consistency verification unit, by verifying each of the three correspondences, can accurately pinpoint the source of the conflict (whether it is an image anomaly, encoding anomaly, or positional anomaly), providing a basis for subsequent targeted reconstruction. The reconstruction unit, based on the conflict type, uses reliable data from the other two correspondences to replace and reconstruct the abnormal portion, achieving self-repair capability for multimodal data conflicts. This allows the system to still output high-confidence recognition results even when single-modal data is abnormal. This mechanism solves the problem in existing technologies where only low-confidence results or direct errors are output when multimodal data conflicts occur, providing a reliable recognition basis for downstream crawling systems.

[0077] The output module is used to output reliable identification results and update historical operation records.

[0078] In this embodiment, the output module receives the identified reliable results from the consistency verification and reconstruction module. The output module first encapsulates the identified reliable results into a data format that conforms to the interface protocol of the downstream grasping system, and then transmits the identified reliable results to the downstream grasping system through the communication interface to guide the bionic hand to perform grasping operations.

[0079] Meanwhile, the output module writes the input data of this recognition task, the cross-modal association constraint chain, the verification result, the reliable recognition result, and the current operation timestamp as a new historical operation record into the storage module, thus updating the historical operation record database.

[0080] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant terms is given first: The output module is used to transmit the reliable identification results to the downstream system and update the historical operation records. It is responsible for the external output of the identification results and the persistent storage of data.

[0081] Downstream grasping system refers to a system that receives and identifies reliable results and generates grasping control commands based on those results, used to control the bionic hand to perform drug grasping operations.

[0082] An operation timestamp is a time stamp that marks the current execution time of an identification task. It is used to record the time when the task occurs and serves as a time index for historical operation records.

[0083] For example, after the consistency verification and reconstruction module completes the reconstruction of the trusted identification result, it transmits the trusted identification result to the output module. The trusted identification result contains the correct appearance identification information "white round pill, ABC", the correct coding identification information "6901234567890" and the correct storage location data "Area A-03 shelf-02 layer-03 column".

[0084] The output module encapsulates the reliable identification result in the JSON format required by the downstream crawling system and sends it to the crawling system via the HTTP interface. The crawling system controls the bionic hand to move to "Area A-03 Shelf-02-03 Column" to perform the crawling operation based on the received location information.

[0085] Meanwhile, the output module combines the input data (image data, encoded data, warehouse location data), cross-modal association constraint chain, verification result (inconsistency), reliable recognition result, and the current operation timestamp "2025-03-25 14:30:25" of this recognition task into a new historical operation record, writes it to the historical operation record database of the storage module, and completes the update of the historical operation record.

[0086] It should be noted that the output module provides accurate and reliable information on the location and morphology of the drugs to the bionic hand by sending the reliable identification results to the downstream grasping system, ensuring the accuracy of the grasping operation. By writing key data from this identification task into the historical operation record, it provides traceable historical data support for constructing the third-party correspondence in subsequent identification tasks, enabling the system to verify the rationality of the current storage location data using historical storage locations, thus forming a data closed loop. These mechanisms ensure the effective transmission of identification results and the continuous accumulation of historical information, providing a guarantee for improving the overall operational efficiency and data traceability of the intelligent drug storage system.

[0087] This application establishes three correspondences between image data and coded data, coded data and warehouse location data, and warehouse location data and historical operation records. These correspondences are then encapsulated into a cross-modal association constraint chain, enabling mutual verification and constraint modeling of multimodal data. Based on this, by performing item-by-item consistency checks on the cross-modal association constraint chain, when inconsistencies are found, the conflict type (abnormal appearance identification information, abnormal coded identification information, or abnormal warehouse location data) can be accurately located. Based on the conflict type, reliable data from the other two correspondences is used to replace and reconstruct the abnormal parts of the identification results, allowing the system to output high-confidence identification results even when single-modal data is abnormal. This mechanism solves the problems in existing technologies where the source of conflict cannot be identified and corrections cannot be made using other modal data when multimodal data conflicts occur, significantly improving the reliability of drug identification and providing accurate and reliable identification data for downstream grasping systems.

[0088] This application aligns multimodal data along the time dimension using a unified timestamp through a data acquisition module, maintains traceable historical operation records through a storage module, constructs a mutual verification constraint network between images, codes, locations, and historical records through a constraint module, accurately locates conflict sources through a consistency verification and reconstruction module and replaces and reconstructs them using other modal data through a consistency verification and reconstruction module, and outputs the credible identification results to the downstream system and synchronously updates the historical operation records through an output module. This forms a closed-loop chain from multimodal data acquisition, time-series alignment, constraint modeling, conflict identification, credibility reconstruction to result output and historical record updates. The collaborative work of these modules solves the problems of disconnect between the identification system and the execution system, inability to use historical information to verify current data, and inability to form a data closed loop in existing technologies, providing comprehensive assurance for the overall operational efficiency and data traceability of intelligent pharmaceutical warehousing systems.

[0089] Example 2: To address the above problems, this application provides a biomimetic grasping method. This method utilizes a cross-modal drug recognition engine from Example 1. Please refer to [link to example 1]. Figure 2 .

[0090] S1: Obtain the trusted identification results and generate initial grasping control instructions based on the trusted identification results.

[0091] In this embodiment, a reliable identification result is obtained from a cross-modal drug identification engine. The reliable identification result includes at least the appearance identification information, coding identification information, and correct storage location data of the target drug.

[0092] The system analyzes and identifies reliable results, extracting the target drug's storage location coordinates, morphological characteristics, and unique identifier. Based on the extracted information, it generates initial grasping control commands according to a pre-defined grasping strategy. These commands include at least the target drug's three-dimensional spatial coordinates, the starting point and critical path points of the bionic hand's movement path, the target gripping force parameters of the bionic hand's gripper, and the grasping posture angle. The bionic grasping system encapsulates the generated initial grasping control commands according to the bionic hand's communication protocol, preparing them for transmission to the bionic hand.

[0093] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant terms is given first: Reliable identification results refer to high-confidence drug identification information output by the cross-modal drug identification engine, which includes at least the target drug's appearance identification information, coding identification information, and correct storage location data.

[0094] The initial grasping control command refers to the set of control commands generated based on the reliable recognition results, used to control the bionic hand to perform the first grasping action. It includes at least the three-dimensional spatial coordinates of the target drug, the bionic hand's movement path, the target clamping force parameters, and the grasping posture angle.

[0095] Drug morphological characteristics refer to the physical attributes of the target drug, such as size, shape, and surface texture, extracted from the reliable identification results. These attributes are used to determine the gripping posture and force of the bionic hand when grasping.

[0096] Preset grasping strategy refers to the pre-defined mapping rules used to convert the reliable identification results into grasping control instructions. These rules include the clamping methods corresponding to different drug forms, the movement path planning algorithm, the clamping force calculation formula, etc. The above mapping rules are conventional technical means in this field and can be pre-configured according to the morphological characteristics and grasping requirements of common drugs in drug storage scenarios.

[0097] For example, a reliable identification result is received from the cross-modal drug recognition engine. This reliable identification result includes: appearance identification information "white round tablet with 'ABC' printed on it", code identification information "6901234567890", correct storage location data "Area A-03 shelf-02 layer-03 column", and drug size information "diameter 10mm, thickness 3mm".

[0098] The bionic grasping system analyzes the reliable recognition result and extracts the three-dimensional spatial coordinates of the target drug (shelf number 03, layer number 02, column number 03, corresponding to actual physical coordinates X=1500mm, Y=800mm, Z=1200mm). The drug's morphological characteristics are circular thin slices.

[0099] According to the preset grasping strategy, for circular thin-film medicines, a two-finger concentric gripping method is adopted, with a preset gripping force of 2N and the gripping posture being that the fingers are perpendicular to the plane of the medicine. Initial grasping control commands are generated, including: target coordinates (1500, 800, 1200), movement path (starting point to path point 1 to path point 2 to target point), gripping force parameter 2N, and grasping posture angle 0°. This command is encapsulated into a binary command packet according to the bionic hand's communication protocol and awaits transmission.

[0100] It should be noted that this step, by analyzing and recognizing the drug's location and morphological information from the reliable recognition results, transforms the high-level recognition results into low-level motion control commands that the bionic hand can execute, providing precise motion parameters for subsequent grasping operations. Generating initial grasping control commands based on the reliable recognition results ensures accurate starting points, reasonable postures, and appropriate force for the grasping action, avoiding the risk of grasping failure due to unreliable recognition results.

[0101] S2 sends the initial grasping control command to the bionic hand, controlling the bionic hand to perform a bionic grasping operation on the target drug.

[0102] In this embodiment, the initial grasping control command generated by S1 is sent to the bionic hand controller via a communication interface. Upon receiving the command, the bionic hand controller parses the target drug's three-dimensional spatial coordinates, movement path, target clamping force parameters, and grasping posture angle contained in the initial grasping control command. Based on the parsed command, the bionic hand first drives the robotic arm to move along the planned movement path from the starting point to the location of the target drug. During the movement, the joint encoder provides real-time feedback of position information to ensure path accuracy. After reaching the target position, the bionic hand adjusts the gripper posture to the grasping posture angle required by the command, aligning the gripper fingers with the drug surface. The bionic hand then drives the gripper to close according to the target clamping force parameters, applying a preset clamping force to the target drug to complete the grasping action. During the grasping process, the bionic hand controller continuously monitors the clamping force feedback value. When the clamping force reaches the target value, it remains stable, confirming that the drug has been firmly grasped.

[0103] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant terms is given first: A bionic hand is an automated actuator that simulates the grasping function of a human hand. It includes at least a robotic arm, a gripper, a joint encoder, and a force sensor, and is used to perform drug grasping and placement operations.

[0104] A bionic hand controller is the control unit of a bionic hand, used to receive grasping control commands, parse command parameters, drive the robotic arm and gripper to perform actions, and collect execution status information in real time.

[0105] The target drug refers to the drug to be grasped in this operation, and its location and shape information are determined by the reliable identification results.

[0106] The initial grasping control command refers to the set of control commands generated by S1 to control the bionic hand to perform the first grasping action. It includes the three-dimensional spatial coordinates of the target drug, the movement path of the bionic hand, the target clamping force parameters, and the grasping posture angle.

[0107] For example, the initial gripping control command generated by S1 is sent to the bionic hand controller via Ethernet communication. This command includes: target coordinates (1500, 800, 1200), movement path (starting point to path point 1 to path point 2 to target point), gripping force parameter 2N, and gripping posture angle 0°. Upon receiving the command, the bionic hand controller first drives the robotic arm to move from its current position (starting point), sequentially passing through path point 1 and path point 2, to reach the target drug location (1500, 800, 1200). During the movement, the joint encoders provide real-time position information. After reaching the target position, the bionic hand adjusts the gripper posture to 0°, making the gripper fingers perpendicular to the drug plane. The bionic hand drives the gripper to close, and the force sensor monitors the gripping force value in real time. When the gripping force reaches 2N, the closing stops, and the gripping force is maintained stable, confirming that the drug has been firmly gripped.

[0108] S3. During the process of the bionic hand performing a bionic grasping operation on the target drug, the grasping execution status information is acquired in real time, and the grasping execution status information is compared with the preset grasping constraints to obtain the comparison result.

[0109] In this embodiment, during the bionic hand's bionic grasping operation on the target drug, the bionic grasping system acquires grasping execution status information in real time. The grasping execution status information includes the actual pose information of the target drug, the actual path position information, and the actual clamping force information.

[0110] The actual pose information of the target drug is compared with the preset pose information, and the spatial position deviation and attitude angle deviation between the two are calculated to obtain the pose offset. The actual position information of the path is compared with the preset grasping path information, and the spatial distance deviation between the actual path and the planned path is calculated to obtain the path offset. The actual clamping force information is compared with the preset clamping force information, and the difference between the actual clamping force and the target clamping force is calculated to obtain the clamping force deviation.

[0111] The pose offset, path offset, and clamping force deviation are compared with the preset gripping constraints. The preset gripping constraints include preset pose offset threshold range, preset path offset range, and preset clamping force deviation range.

[0112] The pose offset is compared with a preset pose offset threshold range to determine the range to which the pose offset belongs, thus obtaining the pose offset result; the path offset is compared with a preset path offset range to determine the range to which the path offset belongs, thus obtaining the path offset result; the clamping force deviation is compared with a preset clamping force deviation range to determine the range to which the clamping force deviation belongs, thus obtaining the clamping force deviation result.

[0113] When the pose offset result, path offset result, and clamping force deviation result are all less than 0, the bionic grasping system determines the grasping execution status information as normal and sets normal as the comparison result. When at least one of the pose offset result, path offset result, and clamping force deviation result is greater than 0, the bionic grasping system determines the grasping execution status information as a serious deviation and identifies the serious deviation as the comparison result. When none of the pose offset results, path offset results, and clamping force deviation results are greater than, and at least one of them is equal to, the bionic grasping system determines the grasping execution status information as a slight deviation and identifies the slight deviation as the comparison result.

[0114] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant terms is given first: Grasping execution status information refers to the set of status data collected in real time during the grasping operation of the bionic hand, which includes at least the actual pose information of the target drug, the actual position information of the path, and the actual gripping force information.

[0115] The actual position and posture information of the target drug refers to the actual spatial position and posture angle data of the target drug perceived in real time by visual sensors or force sensors during the grasping process of the bionic hand.

[0116] The actual location information of the path refers to the spatial location data of the end effector of the robotic arm, which is fed back in real time by the joint encoder during the grasping operation.

[0117] Actual clamping force information refers to the actual clamping force applied to the medicine by the gripper, which is collected in real time by the force sensor during the bionic hand's grasping process.

[0118] Preset pose information refers to the expected spatial position and attitude angle of the target drug set in the initial grasping control command, which serves as a reference benchmark for calculating pose offset.

[0119] The preset grasping path information refers to the expected movement path of the bionic hand set in the initial grasping control command, which serves as a reference benchmark for calculating the path offset.

[0120] The preset clamping force information refers to the target clamping force value set in the initial gripping control command, which serves as a reference benchmark for calculating the clamping force deviation.

[0121] Position offset refers to the spatial position and attitude angle deviation between the actual position information of the target drug and the preset position information, and is used to quantify the degree of deviation of the grasping position and attitude.

[0122] Path offset refers to the spatial distance deviation between the actual location information of the path and the preset crawling path information, and is used to quantify the degree of deviation of the crawling path.

[0123] Clamping force deviation refers to the difference between the actual clamping force information and the preset clamping force information, which is used to quantify the degree of deviation of the clamping force.

[0124] Preset grasping constraints refer to the pre-set deviation threshold conditions used to determine whether the grasping execution state is normal, including preset pose offset threshold range, preset path offset range, and preset clamping force deviation range.

[0125] The preset pose offset threshold range refers to the pre-defined numerical range used to classify pose offset levels, which includes at least three ranges: less than, equal to, and greater than.

[0126] The preset path offset range refers to the pre-defined numerical range used to classify the path offset levels, which includes at least three ranges: less than, equal to, and greater than.

[0127] The preset clamping force deviation range refers to the pre-defined numerical range used to classify the levels of clamping force deviation, which includes at least three ranges: less than, equal to, and greater than.

[0128] The pose offset result refers to the interval assignment result obtained by comparing the pose offset amount with the preset pose offset threshold interval, including less than, equal to or greater than.

[0129] The path offset result refers to the interval assignment result obtained after comparing the path offset amount with the preset path offset interval, including less than, equal to, or greater than.

[0130] The clamping force deviation result refers to the interval assignment result obtained by comparing the clamping force deviation amount with the preset clamping force deviation interval, including less than, equal to or greater than.

[0131] "Normal" refers to the gripping execution state when the pose offset result, path offset result, and clamping force deviation result are all less than the specified values, indicating that all parameters of the gripping process are within the safe allowable range.

[0132] A severe deviation refers to the grasping execution state when at least one of the pose offset result, path offset result, and clamping force deviation result is greater than a certain value, indicating that there is a serious anomaly in the grasping process that exceeds the safe range.

[0133] Slight deviation refers to the gripping execution state where none of the pose offset results, path offset results, and gripping force deviation results are greater than, and at least one of them is equal to, indicating that the gripping process has reached the safety boundary but has not exceeded it, and can be corrected through fine-tuning.

[0134] For example, during the grasping operation of the bionic hand, the bionic grasping system collects grasping execution status information in real time. Assume the preset pose offset threshold range is: less than 2mm is the normal range, 2mm to 5mm is the boundary range, and greater than 5mm is the out-of-limit range; the preset path offset range is: less than 1mm is the normal range, 1mm to 3mm is the boundary range, and greater than 3mm is the out-of-limit range; the preset clamping force deviation range is: less than 0.2N is the normal range, 0.2N to 0.5N is the boundary range, and greater than 0.5N is the out-of-limit range.

[0135] The system compares the actual pose information of the target drug collected by the system with the preset pose information and calculates the pose offset to be 1.5mm, which is within the less than range. The system compares the actual position information of the collected path with the preset grasping path information and calculates the path offset to be 1.2mm, which is within the equal to range (assuming the equal to range is defined as 1mm to 3mm). The system compares the actual clamping force information collected by the system with the preset clamping force information and calculates the clamping force deviation to be 0.1N, which is within the less than range.

[0136] The pose offset result is less than, the path offset result is equal to, and the clamping force deviation result is less than. According to the judgment rules, there is no greater than, and one of them is equal to. Therefore, the bionic grasping system determines that the grasping execution status information is slightly deviated, and the slightly deviated result is determined as the comparison result.

[0137] It should be noted that this step achieves multi-dimensional monitoring of the grasping process by simultaneously acquiring grasping execution status information in three dimensions: pose, path, and gripping force. By comparing the offsets in these three dimensions with preset threshold ranges and classifying the comparison results into three levels—less than, equal to, and greater than—it provides a refined judgment basis for subsequent graded responses. Compared with existing technologies that only set a single threshold for a single dimension, this step can simultaneously perceive deviations in the three dimensions of pose, path, and gripping force, and distinguish the severity of the deviations (normal, slight deviation, severe deviation), providing accurate input for the bionic hand to perform targeted graded control operations based on the actual deviation situation.

[0138] S4 generates an execution command based on the comparison results and sends the execution command to the bionic hand to control the bionic hand to perform a graded response grasping operation on the target drug.

[0139] In this embodiment, based on the comparison results obtained in S3, corresponding execution instructions are generated according to preset hierarchical response rules.

[0140] When the comparison result is normal, the bionic grasping system generates a continue execution instruction, which instructs the bionic hand to maintain the current grasping action and continue execution. When the comparison result shows a serious deviation, the bionic grasping system generates an interruption and replanning instruction. This instruction instructs the bionic hand to interrupt the current bionic grasping operation and triggers the replanning process. When the comparison result shows a slight deviation, the bionic grasping system generates an adjustment command, which instructs the bionic hand to perform the corresponding adjustment operation.

[0141] The generated execution instructions are sent to the bionic hand controller via the communication interface. The bionic hand controller then performs the corresponding hierarchical response operation based on the type of instruction received.

[0142] Specifically: When the comparison result is a serious deviation, the bionic hand controller executes interrupt and replanning instructions. First, it interrupts the current bionic grasping operation, causing the bionic hand to stop its current action and maintain a safe posture. Then, it acquires the termination pose information and termination gripping force information of the bionic hand after the interruption. Based on the termination pose information, termination gripping force information, and the reliable recognition result, it regenerates the first grasping control instruction. The regenerated first grasping control instruction is used as the new initial grasping control instruction, and the controller returns to S2 to re-execute the grasping operation.

[0143] When the comparison result shows a slight deviation, the bionic hand controller executes an adjustment command, extracts the result that is equal to the position offset result, path offset result, and clamping force deviation result, and generates the corresponding adjustment command.

[0144] Specifically, when the pose offset result is equal to a certain value, the pose offset is determined as the pose adjustment value, and the current grasping control command is corrected based on the pose adjustment value to obtain the corrected grasping control command. When the path offset result is equal to a certain value, the path offset is determined as the path adjustment value, and the current grasping control command is corrected based on the path adjustment value to obtain the corrected grasping control command. When the clamping force deviation result is equal to a certain value, the clamping force deviation is determined as the clamping force adjustment value, and the current grasping control command is corrected based on the clamping force adjustment value to obtain the corrected grasping control command. The bionic hand controller sends the corrected grasping control command to the bionic hand, controlling the bionic hand to perform the corresponding adjustment operation.

[0145] When the comparison result is normal, the bionic hand controller executes the continue execution instruction, and the bionic hand continues to execute the current grasping action until the grasping operation is completed.

[0146] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant terms is given first: The "continue execution instruction" refers to the execution instruction generated when the comparison result is normal. It is used to instruct the bionic hand to maintain the current grasping action and continue to execute without any intervention or adjustment.

[0147] Interruption and replanning instructions are execution instructions generated when the comparison result is a serious deviation. They are used to instruct the bionic hand to interrupt the current grasping operation, obtain termination status information, and trigger the replanning process to generate new grasping control instructions.

[0148] The adjustment instruction is an execution instruction generated when the comparison result shows a slight deviation. It is used to instruct the bionic hand to make targeted corrections to the dimensions of the touched boundary.

[0149] Termination pose information refers to the spatial position and attitude angle data of the end effector of the bionic hand when the grasping operation is interrupted, which is used as the state reference during replanning.

[0150] Termination gripping force information refers to the current gripping force applied by the gripper when the bionic hand interrupts the grasping operation, and is used as a safety reference during replanning.

[0151] The first grasping control command refers to the grasping control command regenerated based on the termination pose information, termination gripping force information, and the recognition reliability result, which is used as a new initial grasping control command for re-grasping.

[0152] The pose adjustment value refers to the value used to adjust the position and attitude parameters in the grasping control command when the pose offset result is equal to the position offset value.

[0153] The path adjustment value refers to the value used to adjust the movement path parameters in the capture control command when the path offset result is equal to the target value.

[0154] The clamping force adjustment value refers to the value of the clamping force deviation that is used as the basis for adjustment when the clamping force deviation result is equal to the value of the clamping force deviation. This value is used to correct the clamping force parameter in the gripping control command.

[0155] For example, the obtained comparison result shows a slight deviation. The bionic grasping system generates an adjustment command and extracts the path offset result as equal to 1.2mm. The bionic grasping system determines the path offset of 1.2mm as the path adjustment value, and corrects the movement path in the current grasping control command based on this path adjustment value. The critical path points in the path planning are slightly adjusted by 1.2mm in the opposite direction of the deviation, generating a corrected grasping control command. The bionic hand controller sends the corrected grasping control command to the bionic hand, controlling the bionic hand to continue performing the grasping operation along the corrected path, completing the grasping of the target drug.

[0156] If the comparison result shows a significant deviation, such as a dimension in the pose offset result being greater than (e.g., pose offset of 6mm), the bionic grasping system generates an interruption and replanning instruction. The bionic hand controller interrupts the current grasping operation and obtains the bionic hand's termination pose information and termination gripping force information (e.g., current position coordinates, current gripping force value). Based on the termination pose information, termination gripping force information, and the reliable recognition result, the first grasping control instruction is regenerated. This regenerated instruction is used as the new initial grasping control instruction, and the system returns to S2 to re-execute the grasping operation.

[0157] If the comparison result is normal, the bionic grasping system generates a continue execution instruction, and the bionic hand continues to execute the current grasping action until the grasping operation is completed.

[0158] It should be noted that this step achieves multi-level fine-grained control of the grasping process by mapping the comparison results to three hierarchical response modes: continue execution, adjust, and interrupt replanning. For normal conditions, execution efficiency is maintained; for minor deviations, only the dimensions touching the boundary are fine-tuned to avoid interrupting the entire grasping process due to small deviations; for severe deviations, timely interruption and replanning are implemented to prevent drug damage or grasping failure due to accumulated deviations. This hierarchical response mechanism solves the problem of using a uniform approach to all deviations in existing technologies, enabling the bionic hand to perform targeted adjustment operations based on the actual deviation situation, thus improving the grasping success rate and safety.

[0159] S5: After the bionic hand grasps the target drug, obtain the bionic hand's pose information and the drug's placement coordinate information.

[0160] In this embodiment, after the bionic hand successfully grasps the target drug, the bionic grasping system acquires the current pose information of the bionic hand, which includes the three-dimensional spatial coordinates and attitude angles of the bionic hand's end effector. The bionic hand pose information is acquired in real time through joint encoders and end-effector pose sensors integrated into the bionic hand controller. Simultaneously, the bionic grasping system obtains drug placement coordinate information from the warehouse management system or task instructions. This drug placement coordinate information includes the target placement location of the drug on the shelf or in the storage area, represented by three-dimensional spatial coordinates of the shelf number, layer number, and column number. The bionic grasping system associates the acquired bionic hand pose information and drug placement coordinate information as input data for the generation of subsequent placement control instructions.

[0161] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant terms is given first: Bionic hand pose information refers to the three-dimensional spatial coordinates and attitude angle data of the end effector of the bionic hand after it successfully grasps the target drug. It is used to determine the current position and orientation of the bionic hand after the grasping is completed.

[0162] Drug placement coordinate information refers to the coordinates of the target area where the target drug needs to be placed. It is represented in three-dimensional spatial coordinates as shelf number, layer number, and column number, and is used to determine the endpoint of the placement operation.

[0163] A joint encoder is an angle measurement sensor installed at each joint of a bionic hand. It is used to provide real-time feedback of joint angle data. Combined with a kinematic model, the spatial position and posture of the bionic hand's end effector can be calculated.

[0164] An end effector pose sensor is a pose measurement device installed at the end effector of a bionic hand. It is used to directly acquire the spatial position and attitude data of the end effector as a supplement or verification of the joint encoder's calculation results.

[0165] For example, after the bionic hand successfully grasps the target medicine, the bionic hand controller reads the angle values ​​of each joint through the joint encoder and calculates the current spatial coordinates (1520mm, 810mm, 1210mm) and attitude angles (yaw angle 2°, pitch angle 1°, roll angle 0°) of the bionic hand end effector based on the kinematic model of the bionic hand, as the bionic hand's pose information. Simultaneously, the bionic grasping system parses the medicine placement coordinates from the task command as "B-05 shelf-03 layer-02 column", corresponding to the actual physical coordinates (2000mm, 500mm, 800mm). The bionic grasping system associates and stores the above bionic hand pose information and medicine placement coordinate information for subsequent generation of placement control commands.

[0166] It should be noted that this step, by acquiring the bionic hand's pose information and the drug placement coordinates, provides precise starting and target coordinates for the subsequent drug placement operation. The bionic hand's pose information reflects the current state of the bionic hand after grasping, while the drug placement coordinates reflect the endpoint of the placement operation. Together, they form the basis of a complete motion planning process from the end of grasping to the start of placement, ensuring that the bionic hand can accurately place the drug in the target area and avoiding placement misalignment caused by missing or inaccurate position information.

[0167] S6 generates placement control commands based on the bionic hand pose information, reliable recognition results, and drug placement coordinate information.

[0168] In this embodiment, the bionic hand pose information and drug placement coordinate information are obtained from S5, and a reliable recognition result is obtained from the cross-modal drug recognition engine. The bionic grasping system determines the three-dimensional spatial coordinates and posture angles in the bionic hand pose information as the starting point position and starting posture of the placement operation; it determines the three-dimensional spatial coordinates corresponding to the shelf number, layer number, and column number in the drug placement coordinate information as the target point position of the placement operation; and it extracts the morphological features of the target drug from the reliable recognition result. The morphological features include the size, shape, center of gravity position, and placement posture requirements of the drug.

[0169] Based on the starting point position, starting posture, target point position, and drug morphology characteristics, a motion planning algorithm is used to generate a movement path from the starting point to the target point. The movement path includes the starting point, intermediate path points, and target point. The target posture during placement is determined based on the morphological characteristics of the drug. The target posture includes the orientation angle and rotation angle of the bionic hand end effector at the placement point. The movement path and target posture are encapsulated into placement control instructions, which at least include the trajectory point sequence of the movement path, the target posture parameters, and the release instruction after placement.

[0170] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant terms is given first: The starting point position refers to the three-dimensional spatial coordinates of the end effector of the bionic hand after it successfully grasps the medicine, which serves as the starting point for the placement operation.

[0171] The initial posture refers to the current posture angle of the end effector of the bionic hand after it has successfully grasped the medicine, which serves as the starting orientation for the placement operation.

[0172] The target point location refers to the three-dimensional spatial coordinates of the target area where the medicine needs to be placed, serving as the endpoint of the placement operation.

[0173] Morphological features refer to the size, shape, center of gravity, and placement posture requirements of the target drug extracted from the reliable identification results, which are used to determine the obstacle avoidance during the placement process and the final placement posture.

[0174] Motion planning algorithms are computational methods used to generate collision-free movement paths from a starting point to a target point. They are a common technique in the field of robot motion planning and can be pre-configured based on the spatial layout and obstacle information in a pharmaceutical warehouse scenario.

[0175] The target posture refers to the orientation and rotation angles that the bionic hand end effector needs to achieve at the placement point to ensure that the medicine can be placed in the target area in the correct orientation.

[0176] Placement control commands refer to a set of control commands generated based on the starting point position, starting attitude, target point position, movement path, and target attitude, used to control the bionic hand to perform placement operations. They include at least the trajectory point sequence of the movement path, target attitude parameters, and release commands after placement.

[0177] A trajectory point sequence refers to a series of intermediate points from the starting point to the target point, used to describe the complete path trajectory of the bionic hand's movement.

[0178] The release command refers to the control command used to control the gripper to release the medicine after the bionic hand reaches the target point and adjusts to the target posture.

[0179] For example, the bionic hand pose information obtained from S5 is: starting point position (1520mm, 810mm, 1210mm), starting posture (yaw angle 2°, pitch angle 1°, roll angle 0°); the drug placement coordinate information is obtained as: target point position (2000mm, 500mm, 800mm). From the reliable identification results, the target drug's morphological features are extracted: the drug is a circular thin sheet, 10mm in diameter, 3mm thick, with its center of gravity located at the geometric center, and the placement posture requires it to be flat (the drug surface is parallel to the shelf surface).

[0180] Based on the starting point (1520, 810, 1210) and the target point (2000, 500, 800), a motion planning algorithm is used to generate a movement path. The path trajectory point sequence includes the starting point, path point A (1700, 700, 1100), path point B (1850, 600, 950), and the target point (2000, 500, 800). Given the pharmaceutical's morphological characteristics—a circular, thin sheet requiring flat placement—the target orientation during placement is determined to be 0° yaw, 0° pitch, and 0° roll, ensuring the pharmaceutical surface is parallel to the shelf plane. The bionic grasping system encapsulates the movement path trajectory point sequence, target orientation parameters, and release command upon placement into a placement control command.

[0181] S7 sends a placement control command to the bionic hand, controlling the bionic hand to place the target drug in the target area.

[0182] In this embodiment, the placement control command generated in S6 is sent to the bionic hand controller via the communication interface. After receiving the placement control command, the bionic hand controller parses the movement path trajectory point sequence, target attitude parameters, and release command contained in the command.

[0183] The bionic hand controller drives the bionic hand to move sequentially along a path of points, from the starting point through intermediate points to the target point. During movement, the controller provides real-time position feedback via joint encoders to ensure the movement trajectory matches the commanded trajectory. Upon reaching the target point, the controller adjusts the end effector's angle based on target posture parameters, aligning the gripper correctly with the target area. After adjustment, the controller executes a release command, releasing the gripper and placing the target medication into the target area. After release, the controller drives the bionic hand back to a safe position or awaits the next command.

[0184] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the relevant terms is given first: A trajectory point sequence refers to a series of intermediate points from the starting point to the target point included in the placement control command, used to describe the complete path trajectory of the bionic hand's movement.

[0185] The target attitude parameters refer to the orientation and rotation angles that the bionic hand end effector needs to achieve at the target point, as included in the placement control command, to ensure that the medicine is placed in the correct orientation.

[0186] The release command refers to the control command included in the placement control command, which is used to control the gripper to release the medicine. It is usually executed after the bionic hand reaches the target point and adjusts to the target posture.

[0187] For example, the bionic grasping system sends the placement control command generated by S6 to the bionic hand controller via Ethernet communication. This command includes: a sequence of trajectory points (starting point (1520,810,1210) to path point A (1700,700,1100) to path point B (1850,600,950) to target point (2000,500,800)), target attitude parameters (yaw angle 0°, pitch angle 0°, roll angle 0°), and a release command.

[0188] After parsing the instructions, the bionic hand controller drives the bionic hand to move from the starting point (1520, 810, 1210), sequentially passing through path point A (1700, 700, 1100) and path point B (1850, 600, 950), finally reaching the target point (2000, 500, 800). During the movement, the joint encoder provides real-time position information to ensure the bionic hand accurately passes through each path point. Upon reaching the target point, the bionic hand controller adjusts the attitude of the bionic hand's end effector according to the target attitude parameters, from the initial attitude (yaw angle 2°, pitch angle 1°, roll angle 0°) to the target attitude (yaw angle 0°, pitch angle 0°, roll angle 0°), making the gripper parallel to the shelf plane. After the attitude adjustment is complete, the bionic hand controller executes the release command, driving the gripper to release the target drug into the target area. After the drug is released, the bionic hand controller drives the bionic hand back to a safe position, awaiting the next task instruction.

[0189] This application, during the bionic grasping operation, acquires real-time information on the actual pose, path, and clamping force of the target drug. It compares the pose offset, path offset, and clamping force deviation with preset grasping constraints to obtain pose offset, path offset, and clamping force deviation results. Based on the combination of these three offset results, it determines the grasping execution status as normal, slightly deviated, or severely deviated, and generates corresponding execution instructions based on the comparison results: continued execution in normal conditions, adjustment instructions for targeted fine-tuning in cases of slight deviation, and interruption and replanning instructions in cases of severe deviation. This mechanism, by simultaneously monitoring the offsets in pose, path, and clamping force, and classifying the comparison results into three levels (less than, equal to, and greater than), achieves multi-dimensional, refined monitoring and graded response of the grasping process. It solves the problems of existing technologies that only set a single threshold for a single dimension, cannot distinguish the severity of deviations, and apply uniform treatment to all deviations. This allows the bionic hand to perform targeted adjustments based on the actual deviation, significantly improving the grasping success rate and operational safety.

[0190] This application transforms high-level drug identification information into low-level motion control commands executable by the bionic hand by acquiring reliable identification results and generating initial grasping control commands based on these results. During the grasping operation, the application acquires grasping execution status information in real time and performs multi-dimensional deviation comparison and graded response. After successful grasping, the application acquires the bionic hand's pose information and drug placement coordinate information, generating placement control commands to precisely place the target drug in the target area, forming a complete closed loop from identification, grasping, monitoring, graded response to placement. Specifically, for minor deviations, adjustment commands are generated to fine-tune the dimensions touching the boundary; for severe deviations, interruption and replanning commands are generated, and the grasping operation is re-executed. This mechanism achieves adaptive closed-loop control of the grasping process, enabling the bionic hand to autonomously adjust its grasping strategy based on real-time state deviations. This solves the problem in existing technologies where the grasping process cannot be dynamically adjusted according to actual deviations, and grasping fails as soon as a deviation occurs, thus improving the adaptive capability and operational reliability of the intelligent drug storage system.

[0191] Example 3: This application provides a bionic grasping system. The system's module functions correspond to the specific implementation steps in Example 2, including a cross-modal drug recognition engine, an instruction generation module, a bionic hand execution module, a state detection module, a comparison result determination module, a hierarchical response module, and a placement control module. Please refer to [link to relevant documentation]. Figure 3 .

[0192] A cross-modal drug recognition engine is used to acquire input data of the target drug and output reliable recognition results; The instruction generation module is used to obtain the reliable recognition results output by the cross-modal drug recognition engine and generate initial grasping control instructions based on the reliable recognition results. The bionic hand execution module is used to send initial grasping control commands to the bionic hand, controlling the bionic hand to perform bionic grasping operations on the target drug; The status detection module is used to acquire the grasping execution status information in real time during the bionic hand's bionic grasping operation on the target drug; The comparison result determination module is used to compare the capture execution status information with the preset capture constraints to obtain the comparison result; The graded response module is used to generate execution instructions based on the comparison results and send the execution instructions to the bionic hand to control the bionic hand to perform graded response grasping operations on the target drug; The placement control module is used to obtain the bionic hand's pose information and the drug placement coordinate information after the bionic hand grasps the target drug. Based on the bionic hand's pose information, the recognition reliability result, and the drug placement coordinate information, it generates a placement control command and sends the placement control command to the bionic hand to control the bionic hand to place the target drug in the target area.

[0193] The functions of the above modules correspond to the method description in Example 2, and will not be described in detail here.

[0194] This application utilizes a biomimetic grasping system to form a complete closed loop from drug identification, grasping execution, multi-dimensional status monitoring, hierarchical response to precise placement, thereby achieving adaptive control of the drug grasping process. This solves the problems in existing technologies that cannot distinguish the severity of deviations and apply a uniform processing method to all deviations, thus improving the grasping success rate and operational reliability.

[0195] The foregoing has shown and described the basic principles, main features, and advantages of this application. Those skilled in the art should understand that this application is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this application. Various changes and modifications can be made to this application without departing from the spirit and scope thereof, and all such changes and modifications fall within the scope of this application as claimed. The scope of protection of this application is defined by the appended claims and their equivalents.

Claims

1. A cross-modal drug recognition engine, characterized in that, The recognition engine includes: The data acquisition module is used to acquire input data of the target drug, which includes at least image data, encoded data, and storage location data. The storage module is used to store historical operation records; The constraint module is used to construct a first correspondence between the image data and the encoded data, a second correspondence between the encoded data and the warehouse location data, and a third correspondence between the warehouse location data and the historical operation records based on the input data and the historical operation records, and to encapsulate the first correspondence, the second correspondence, and the third correspondence into a cross-modal association constraint chain; The consistency verification and reconstruction module is used to perform consistency verification on the cross-modal association constraint chain, obtain the verification result, generate a conflict state identifier when the verification result is inconsistent, generate an identification result based on the cross-modal association constraint chain, and reconstruct the credibility of the identification result based on the conflict state identifier to obtain a credible identification result. The output module is used to output the reliable identification result and update the historical operation record.

2. The engine according to claim 1, characterized in that, The constraint module is specifically used for: The image data is normalized, the encoded data is formatted, and the warehouse location data is coordinate standardized to eliminate representation differences between different modal data and obtain processed input data. Feature extraction is performed on the image data in the processed input data to obtain appearance identification information; The encoded data in the processed input data is parsed to obtain encoded identification information; The appearance identification information and the coded identification information are associated and matched to obtain a first correspondence relationship; The encoded identification information is associated and matched with the warehouse location data in the processed input data to obtain a second correspondence; The warehouse location data in the processed input data is associated and matched with the historical operation records to obtain a third correspondence; The first correspondence, the second correspondence, and the third correspondence are encapsulated into a cross-modal association constraint chain.

3. The engine according to claim 2, characterized in that, The consistency verification and reconstruction module includes a consistency verification unit and a reconstruction unit; The consistency verification unit is used to perform consistency verification on the appearance identification information and the encoding identification information in the first correspondence relationship to obtain a first result; The consistency of the coded identification information in the second correspondence with the warehouse location data is verified to obtain the second result; The consistency of the warehouse location data in the third correspondence relationship with the historical operation records is verified to obtain the third result; Based on the first result, the second result, and the third result, a verification result is generated according to a preset conflict determination rule; The preset conflict determination rules include: If at least one of the first result, the second result, and the third result is different, the verification result is determined to be inconsistent. When the first result, the second result, and the third result are all the same, the verification result is determined to be consistent. The reconstruction unit is used to determine the conflict type based on the first result, the second result and the third result when the verification result is inconsistent, and to generate a conflict status identifier carrying the conflict type. The identification result is generated based on the cross-modal association constraint chain, and the abnormal part in the identification result is determined according to the conflict type; The abnormal part is replaced to obtain a reliable identification result; The replacement of the abnormal portion includes: When the conflict type is abnormal appearance identification information, the corresponding coded identification information and warehouse location data are extracted from the second correspondence and the third correspondence. The correct appearance identification information is determined based on the coded identification information and warehouse location data, and the abnormal appearance identification information in the identification result is replaced. When the conflict type is abnormal coding identification information, the corresponding appearance identification information and storage location data are extracted from the first correspondence and the third correspondence. The correct coding identification information is determined based on the appearance identification information and storage location data, and the abnormal coding identification information in the identification result is replaced. When the conflict type is abnormal warehouse location data, the corresponding appearance identification information and code identification information are extracted from the first correspondence and the second correspondence. The correct warehouse location data is determined based on the appearance identification information and code identification information, and the abnormal warehouse location data in the identification result is replaced.

4. The engine according to claim 1, characterized in that, The consistency verification and reconstruction module is specifically used for: When the verification results are consistent, an identification result is generated based on the cross-modal association constraint chain, and the identification result is determined as a reliable identification result.

5. A biomimetic grasping method, comprising a cross-modal drug recognition engine according to any one of claims 1-4, characterized in that, The method includes: Obtain a reliable identification result, and generate an initial grasping control command based on the reliable identification result; The initial grasping control command is sent to the bionic hand to control the bionic hand to perform a bionic grasping operation on the target drug. During the process of the bionic hand performing a bionic grasping operation on the target drug, the grasping execution status information is acquired in real time, and the grasping execution status information is compared with the preset grasping constraints to obtain the comparison result; An execution instruction is generated based on the comparison results and sent to the bionic hand to control the bionic hand to perform a graded response grasping operation on the target drug. After the bionic hand grasps the target medicine, the bionic hand pose information and the medicine placement coordinate information are obtained; Based on the bionic hand pose information, the reliable recognition result, and the drug placement coordinate information, a placement control command is generated; The placement control command is sent to the bionic hand, which controls the bionic hand to place the target drug in the target area.

6. The method according to claim 5, characterized in that, During the process of the bionic hand performing a bionic grasping operation on the target drug, the grasping execution status information is acquired in real time, and the grasping execution status information is compared with preset grasping constraints to obtain the comparison result, including: During the process of the bionic hand performing a bionic grasping operation on the target drug, the grasping execution status information is acquired in real time. The grasping execution status information includes the actual pose information of the target drug, the actual path position information, and the actual clamping force information. The actual pose information of the target drug is compared with the preset pose information to obtain the pose offset. The actual location information of the path is compared with the preset capture path information to obtain the path offset; The actual clamping force information is compared with the preset clamping force information to obtain the clamping force deviation. The pose offset, the path offset, and the clamping force deviation are compared with the preset gripping constraints to obtain the comparison results.

7. The method according to claim 6, characterized in that, The step of comparing the pose offset, the path offset, and the clamping force deviation with preset grasping constraints to obtain the comparison result includes: The pose offset is compared with a preset pose offset threshold range to obtain the pose offset result; The path offset is compared with a preset path offset range to obtain the path offset result; The clamping force deviation is compared with the preset clamping force deviation range to obtain the clamping force deviation result; When the pose offset result, the path offset result, and the clamping force deviation result are all less than, the grasping execution status information is determined to be normal, and the normal status is determined as the comparison result. If at least one of the pose offset result, the path offset result, and the clamping force deviation result is greater than, the grasping execution status information is determined to be a serious deviation, and the serious deviation is determined as the comparison result. If none of the pose offset result, the path offset result, and the clamping force deviation result are greater than, and at least one of them is equal to, the grasping execution status information is determined to be a slight deviation, and the slight deviation is determined as the comparison result.

8. The method according to claim 7, characterized in that, The process of generating execution instructions based on the comparison results and sending the execution instructions to the bionic hand to control the bionic hand to perform a graded response grasping operation on the target drug includes: When the comparison result is normal, a continue execution instruction is generated and sent to the bionic hand to control the bionic hand to continue executing the current grasping action; When the comparison result is a serious deviation, an interruption and replanning instruction is generated and sent to the bionic hand to control the bionic hand to interrupt the current bionic grasping operation; Obtain the termination pose information and termination gripping force information of the bionic hand after the interruption; Based on the termination pose information, the termination clamping force information, and the recognition reliability result, a first grasping control command is generated. If the comparison result shows a slight deviation, an adjustment command is generated and sent to the bionic hand to control the bionic hand to perform the adjustment operation.

9. The method according to claim 8, characterized in that, When the comparison result shows a slight deviation, an adjustment command is generated and sent to the bionic hand to control the bionic hand to perform the adjustment operation, including: When the comparison result is slightly off, extract the result that is equal to the pose offset result, the path offset result, and the clamping force deviation result, generate the corresponding adjustment command, and send the adjustment command to the bionic hand to control the bionic hand to perform the corresponding adjustment operation; Wherein, when the pose offset result is equal to, the pose offset amount is determined as the pose adjustment value, and the grasping control command is modified based on the pose adjustment value to obtain the modified grasping control command. When the path offset result is equal to the path adjustment value, the path offset is determined as the path adjustment value, and the grasping control command is corrected based on the path adjustment value to obtain the corrected grasping control command. When the clamping force deviation result is equal to the value, the clamping force deviation is determined as the clamping force adjustment value, and the gripping control command is corrected based on the clamping force adjustment value to obtain the corrected gripping control command. The modified grasping control command is sent to the bionic hand, which then performs the corresponding adjustment operation.

10. A biomimetic grasping system, characterized in that, The system includes: Cross-modal drug recognition engine, instruction generation module, bionic hand execution module, state detection module, comparison result determination module, hierarchical response module, and placement control module; The cross-modal drug recognition engine is used to acquire input data of the target drug and output a reliable recognition result; The instruction generation module is used to obtain the reliable identification result output by the cross-modal drug recognition engine, and generate an initial grasping control instruction based on the reliable identification result; The bionic hand execution module is used to send the initial grasping control command to the bionic hand and control the bionic hand to perform a bionic grasping operation on the target drug; The state detection module is used to acquire grasping execution status information in real time during the process of the bionic hand performing a bionic grasping operation on the target drug; The comparison result determination module is used to compare the crawling execution status information with the preset crawling constraints to obtain the comparison result; The graded response module is used to generate an execution instruction based on the comparison result and send the execution instruction to the bionic hand to control the bionic hand to perform a graded response grasping operation on the target drug. The placement control module is used to acquire the bionic hand pose information and drug placement coordinate information after the bionic hand grasps the target drug, generate a placement control command based on the bionic hand pose information, the reliable recognition result and the drug placement coordinate information, and send the placement control command to the bionic hand to control the bionic hand to place the target drug in the target area.

Images