Electric instrument table intelligent control method based on multi-modal perception and model prediction

Abstract

The invention provides an electric instrument table intelligent control method based on multi-modal perception and model prediction, and relates to the technical field of electric instrument tables, and the method comprises the steps: obtaining high-precision environment perception data through a multi-modal sensor fusion technology, and constructing a dynamic three-dimensional map to recognize an instrument and an obstacle; the system drives the multi-degree-of-freedom mechanical arm to move efficiently through optimal path planning based on model prediction control, the operation period is remarkably shortened, the overall operation efficiency and throughput capacity are improved, meanwhile, potential collision and abnormal stress are monitored in real time in the grabbing and placing process, an intelligent obstacle avoidance and safe shutdown mechanism is started, and the safety of the robot is improved. According to the method, the robustness and safety of system operation are greatly improved, a flexible grabbing strategy is integrated for precious fragile instruments, lossless operation is achieved through real-time force feedback, high-value samples are effectively protected, finally, through machine vision verification and online optimization, the system can continuously conduct self-learning, the operation precision is continuously improved, and the success rate is continuously increased. And the self-adaptive performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electric instrument platforms, and in particular to an intelligent control method for electric instrument platforms based on multimodal perception and model prediction. Background Art

[0002] Chinese Patent No. CN111568050A discloses an electric precision analytical instrument lifting platform, belonging to the field of precision instrument analysis technology. It comprises a mobile base, a telescopic device, a guide assembly, and a tabletop. The telescopic device has a mounting end and a telescopic end. The mounting end is mounted on the mobile base, and the telescopic end is connected to the bottom of the tabletop. The guide assembly is mounted on the mobile base and passes through it. Therefore, compared with existing technologies, this invention offers the advantage of being significantly easier to operate.

[0003] The above patent documents and prior art have the following technical problems when used:

[0004] Problem 1: Existing electric instrumentation equipment generally lacks efficient path planning when performing operations. This results in lengthy and suboptimal motion paths for the robotic arm, which is particularly noticeable in complex and changing working environments. This not only increases the robotic arm's idle travel and unnecessary movements, but also significantly prolongs overall operation time.

[0005] Second, traditional electric instrument handling systems often use rigid gripping strategies, lacking the ability to perceive the material and fragility of the instrument and the ability to adjust flexibly. This poses a high risk of damage when handling these specialized instruments, especially valuable or fragile ones, which can be easily mechanically impacted or damaged.

[0006] Problem three: Traditional electric instrument equipment usually operates in a structured, preset environment. Once dynamic changes or emergencies occur in the work area, such as people walking around, accidentally dropping objects, or slight shifts in the instrument position, the equipment is often unable to perceive and respond effectively in a timely manner. At the very least, this will lead to mission interruption and reduced efficiency. At worst, it will cause the robotic arm to collide with obstacles or people, causing equipment damage or even personal injury. Summary of the Invention

[0007] Technical Solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: an intelligent control method for an electric instrument platform based on multimodal perception and model prediction, the hardware structure of the intelligent control method includes an instrument platform body, a tray, a sensor, a robotic arm, a camera, and a laser radar, characterized in that the intelligent control method includes the following steps:

[0009] Sp1: Acquire high-precision three-dimensional environmental perception data of the working area of ​​the electric instrument platform, including but not limited to instrument type, size, shape, weight, placement, and platform status. This data is achieved through the fusion of sensors, cameras, and lidar technology.

[0010] Sp2: Based on the environmental perception data obtained by Sp1, it processes and analyzes the data in real time, builds a high-precision dynamic three-dimensional map of the work area, and identifies and marks all instruments and potential obstacles;

[0011] Sp3: Based on the dynamic 3D map and recognition results constructed by Sp2 and the preset operation task requirements, it generates the optimal path planning for instrument placement and retrieval;

[0012] Sp4: Based on the optimal path planning generated by Sp3, it drives the multi-degree-of-freedom robotic arm of the electric instrument platform to perform motion control and dynamically adjusts the motion parameters of the robotic arm;

[0013] Sp5: During the instrument grasping and placement process, the real-time force feedback data obtained by Sp1 and the dynamic 3D map constructed by Sp2 are continuously used to monitor potential collisions and abnormal forces between the robot arm and the instrument, table and environment in real time;

[0014] Sp6: After the instrument is successfully placed and taken, the high-precision machine vision data obtained by Sp1 is used to accurately verify the final position and posture of the instrument, enabling online optimization and model update;

[0015] Sp7: For valuable and fragile instruments, a flexible grasping strategy is integrated into the motion control of Sp4, combined with the instrument material and structure data obtained by Sp1, to achieve flexible and non-destructive grasping of the instruments.

[0016] Preferably, in the Sp1, the sensor is a high-precision force sensor integrated on the surface of the tray, the end effector of the robotic arm and other key positions of the instrument platform body, which is used to obtain the weight distribution of the instrument when it is placed, the contact force during grasping and the environmental interaction force in real time. The camera is a high-resolution stereo vision camera installed above the instrument platform body, which is used to obtain three-dimensional image information of the working area. The laser radar is used to generate accurate point cloud data of the working area, and through multi-sensor collaborative perception, high-precision, multi-dimensional perception of all objects and their states in the working area is achieved.

[0017] Preferably, in the Sp2, the heterogeneous data obtained by sensors, cameras and lidar are synchronized in time and space and information is fused through a data fusion algorithm to generate a point cloud model and a semantic map containing precise position, posture and geometric shape. Based on the semantic map, various types of instruments to be operated in the working area and fixed or mobile obstacles in the environment are accurately identified and distinguished, and real-time tracking and posture estimation are performed on them to ensure dynamic updating and high precision of map information.

[0018] Preferably, in the Sp3, based on the high-precision dynamic three-dimensional map and recognition results constructed by Sp2, combined with the preset instrument characteristics, table load distribution, obstacle avoidance requirements, and the preset instrument placement and picking task priorities and timing requirements, a planning method based on model predictive control is used to generate a collision-free, optimal time, minimum energy consumption and smooth path three-dimensional motion trajectory of the robotic arm from the starting point to the target point, taking into account the robotic arm's own kinematics, dynamic constraints and obstacle avoidance requirements.

[0019] Preferably, in the Sp4, the multi-degree-of-freedom robotic arm has at least six or more degrees of freedom, and the motion control adopts a strategy based on force-position hybrid control. According to the real-time force feedback data obtained by Sp1, the joint torque and end force of the robotic arm are precisely controlled to achieve smooth contact and force-controlled operation with the instrument. At the same time, according to the instrument posture recognized by Sp2 and the path planning generated by Sp3, the joint speed, acceleration and target position of the robotic arm are dynamically adjusted to adapt to environmental changes and improve operation accuracy.

[0020] Preferably, in the Sp5, the preset safety distance threshold and collision force threshold are used to determine in real time whether there is a risk of contact with obstacles and instruments in the movement path of the robotic arm. When it is detected that the force and torque feedback from the force sensor exceed the safety range, the emergency stop command is immediately triggered, and the path replanning mechanism based on the dynamic three-dimensional map is started at the same time to guide the robotic arm to avoid obstacles and leave the abnormal contact area to ensure safe operation.

[0021] Preferably, in the Sp6, the multi-view images of the instrument after placement are obtained by the camera for precise verification, and the three-dimensional spatial coordinates and Euler angle posture of the instrument are accurately calculated using image processing and three-dimensional reconstruction technology. The calculation results are compared with the preset ideal placement position and posture for error analysis. When a deviation is detected between the actual placement position and posture and the planned value, the system will automatically record the deviation data and use it as feedback information to adjust and optimize the parameters of the path planning model in Sp3, so as to achieve continuous improvement and adaptive adjustment of system performance.

[0022] Preferably, in the Sp7, an instrument characteristic database is pre-established and stored, and the database contains the type, value level, fragility coefficient, maximum allowable clamping force, surface characteristics and specific grasping requirements of known instruments. When executing Sp1 to obtain environmental perception data, the identified instrument is matched with the database through the instrument identification and matching method to determine whether it is a valuable or fragile instrument. When it is determined to be a valuable or fragile instrument, the flexible grasping strategy will dynamically adjust the clamping preload, grasping speed curve and clamping action time of the end effector of the robotic arm according to the corresponding parameters in the database, and combine the real-time force feedback data provided by the sensor to ensure that the grasping torque is sufficient to stably clamp the instrument without causing any extrusion or impact to it, thereby avoiding surface scratches or internal damage to valuable or fragile instruments, and achieving precise and gentle operation.

[0023] Preferably, a tray is provided on the surface of the instrument platform body, a robotic arm is provided on the surface of the instrument platform body, sensors are provided on the surfaces of the tray and the robotic arm, a camera is provided on the surface of the instrument platform body, and a laser radar is provided on the surface of the instrument platform body.

[0024] Beneficial effects

[0025] The present invention provides an intelligent control method for an electric instrument platform based on multimodal perception and model prediction.

[0026] It has the following beneficial effects:

[0027] 1. By introducing an intelligent path planning algorithm based on model predictive control, the present invention is able to generate the optimal three-dimensional motion trajectory of the robot arm from the starting point to the target point in real time and online. This advanced planning capability means that the robot arm can intelligently avoid all static and dynamic obstacles, while optimizing multiple objectives and ensuring smooth motion. As a result, the robot arm can complete the task in the most direct and efficient way, significantly reducing unnecessary idle strokes and redundant movements. Ultimately, this enables the system to significantly shorten the single operation cycle and significantly improve the overall operating efficiency and processing throughput of the automation system.

[0028] 2. The present invention achieves accurate identification and attribute matching of instrument materials and fragility through high-precision multimodal sensor fusion technology and deep learning-based environmental cognition. On this basis, the system can intelligently activate the flexible grasping strategy, dynamically adjust the clamping preload, contact area and grasping speed curve of the robot arm end effector, and combine with real-time force sensor closed-loop control to ensure that such instruments are grasped and placed with extremely gentle and uniform force. This refined force control and flexible adjustment fundamentally eliminates the risk of instrument damage caused by mechanical shock or excessive force, significantly improves the level of protection for high-value, easily consumable samples and equipment, thereby greatly reducing material loss in the laboratory or production process and ensuring the reliability of experimental results.

[0029] 3. The present invention constructs and dynamically updates high-precision three-dimensional maps in real time, which enables the system to not only accurately identify fixed obstacles, but also detect, track and predict the movement trajectories of dynamic obstacles such as people and accidentally dropped objects in real time. When the system predicts a potential collision risk or detects a new dynamic obstacle, the intelligent obstacle avoidance mechanism will be quickly activated. Through millisecond-level path replanning, it will calculate a new collision-free path to bypass the obstacle and instruct the robotic arm to continue to perform the task smoothly and seamlessly along the new path. For sudden serious abnormalities, the multi-level linkage safety shutdown mechanism can also respond immediately to ensure the absolute safety of equipment and personnel. This highly sensitive dynamic perception and intelligent obstacle avoidance response greatly improve the operational robustness, continuity and safety of automated equipment in real and complex work scenarios, effectively avoiding task interruption, efficiency loss, potential equipment damage and personal injury, and enabling intelligent instruments to operate stably and reliably in a wider range of application environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A diagram showing the steps of the method of the present invention;

[0031] Figure 2 This is a system architecture diagram of the present invention;

[0032] Figure 3 It is the hardware structure diagram of the present invention.

[0033] Among them: 1. Instrument body; 2. Tray; 3. Sensor; 4. Robotic arm; 5. Camera; 6. LiDAR. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention. Specific embodiment one:

[0036] like Figures 1 to 3 As shown, an intelligent control method for an electric instrument platform based on multimodal perception and model prediction is provided. The hardware structure of the intelligent control method includes an instrument platform body 1, a tray 2, a sensor 3, a robotic arm 4, a camera 5, and a laser radar 6. The intelligent control method is characterized in that: the intelligent control method includes the following steps:

[0037] Sp1: Acquire three-dimensional high-precision environmental perception data of the working area of ​​the electric instrument table 1, including but not limited to instrument type, size, shape, weight, placement, and table status. Acquisition is achieved through the fusion technology of sensor 3, camera 5, and lidar 6;

[0038] In Sp1, the sensor 3 is a high-precision force sensor 3 integrated on the surface of the tray 2, the end effector of the robotic arm 4 and other key positions of the instrument platform body 1, which is used to obtain the weight distribution of the instrument when it is placed, the contact force during grasping and the environmental interaction force in real time. The camera 5 is a high-resolution stereo vision camera 5 installed above the instrument platform body 1, which is used to obtain three-dimensional image information of the working area and support the fine recognition and posture estimation of the instrument. The lidar 6 is used to generate accurate point cloud data of the working area, which can provide reliable spatial information even in poor lighting or lack of texture on the surface of the object. Through the collaborative perception of multiple sensors 3, high-precision and multi-dimensional perception of all objects and their states in the working area is achieved.

[0039] Sp2: Based on the environmental perception data obtained by Sp1, it processes and analyzes the data in real time, builds a high-precision dynamic three-dimensional map of the work area, and identifies and marks all instruments and potential obstacles;

[0040] In Sp2, the heterogeneous perception data acquired by sensor 3, camera 5, and lidar 6 are synchronized in milliseconds and feature-level information is fused through a data fusion algorithm. This fusion process utilizes the calibration model and state estimation filter of sensor 3 to ensure that the data of different sensors 3 are accurately aligned in a unified coordinate system. The fused data is input into a perception model based on a deep neural network. The model can perform real-time semantic segmentation and instance-level recognition on the data, thereby generating a high-precision dynamic three-dimensional map and semantic map containing precise position, posture, three-dimensional geometric shape, and material properties. Based on the semantic map, the system can accurately identify and distinguish various types of instruments to be operated in the work area, including but not limited to their specific models, placement directions, and internal structures. It can also identify and accurately distinguish fixed obstacles in the environment in real time, including the edges of the instrument platform 1, fixed equipment, and dynamic obstacles, including the operator's hands and mobile tools. The system uses a continuous tracking algorithm to track and estimate the pose of all identified objects in real time, ensuring that the dynamic update frequency of the map information is higher than the movement speed of the robot arm 4, thereby providing high-precision, real-time, and semantically understood environmental cognition, providing accurate input for subsequent planning;

[0041] Among them, the data processing and analysis model uses a deep learning model of a 3D convolutional neural network to process the fused point cloud and image data, and performs semantic segmentation and instance segmentation. Semantic segmentation is to classify each point and pixel in the scene into different categories such as "instrument", "tray 2", "obstacle", "background", etc. Instance segmentation is based on semantic segmentation, and further distinguishes each independent instrument individual and assigns it a unique ID. It uses an efficient inference engine and parallel computing architecture to ensure that the model can complete the processing of real-time perception data within milliseconds. The content of high-precision dynamic three-dimensional maps is not only a static three-dimensional geometric model, but also a map that contains semantic information, dynamic information and topological relationships. Geometric information is the precise three-dimensional shape, size, center of gravity, and surface normal of the instrument, and semantic information is the type of instrument, i.e. Test tubes, beakers, chips, functional properties, material properties are glass, plastic, metal, dynamic information is the real-time update of the position, speed and trajectory prediction of moving obstacles, the topological relationship is the relative position relationship between instruments, the division of the placement area on tray 2, using synchronous positioning and map construction, updating map information in real time according to new perception data, especially for changes in instrument placement or new obstacles, in the identification and marking of all instruments and potential obstacles, based on the model recognition results, a unique identifier is assigned to each identified instrument, and its precise three-dimensional pose is extracted. At the same time, fixed obstacles and dynamic obstacles are identified and distinguished, the identified instruments and obstacle information are visually marked in the dynamic three-dimensional map, and their geometric models and semantic attributes are associated for use by subsequent planning modules.

[0042] Sp3: Based on the dynamic 3D map and recognition results constructed by Sp2 and the preset operation task requirements, it generates the optimal path planning for instrument placement and retrieval;

[0043] In Sp3, based on the high-precision dynamic three-dimensional map and real-time recognition results built by Sp2, combined with the preset instrument characteristics, namely weight, center of gravity, graspable surface, table load distribution, namely the occupied area and weight of existing instruments, obstacle avoidance requirements, namely safety distance, motion space constraints, and the priority and timing requirements of preset instrument placement and picking tasks, the system uses an intelligent planning method based on model predictive control. In each control cycle, the planning method constructs and solves a multi-objective optimization problem online according to the current state of the robot arm 4 and the real-time dynamic changes of the environment. This optimization problem comprehensively considers the complex kinematic constraints, dynamic constraints and By predicting the system behavior in the future and minimizing a cost function, the planning method can generate a three-dimensional motion trajectory of the robot arm 4 from the starting point to the target point that is collision-free, has the optimal time, minimizes energy consumption, and has a smooth path. The motion trajectory not only includes the angle sequence of all joints of the robot arm 4, but also the precise path of the end effector in Cartesian space. During the trajectory generation process, the planning method also optimizes the grasping and placement postures of the end effector of the robot arm 4, ensuring that the end effector can approach and operate the instrument in a manner that best suits the instrument's geometric characteristics and task requirements, thereby achieving efficient and safe task execution.

[0044] Among them, the planning method based on model predictive control realizes the intelligent control of the rapid placement and picking of the electric instrument table body 1 by establishing the dynamic model, kinematic model, end effector model of the robot arm 4, and the dynamic three-dimensional map environment model from Sp2. The method first defines the starting and target states of the robot arm 4 according to the task, and comprehensively considers the instrument characteristics, namely the grasping area, stability, fragility, table load distribution, obstacle avoidance requirements, operation efficiency and the robot arm 4's own constraints, namely joint limits, speed, acceleration, and torque, and optimizes task priority and timing in multi-task scenarios. The planning method based on model predictive control solves the optimization problem online in each control cycle, predicts the future state, and generates a collision-free, optimal time, minimum energy consumption and smooth path three-dimensional motion trajectory. At the same time, the end effector posture is optimized to match the instrument geometry and grasping point to avoid sliding or damage. The model parameters and optimization weights are pre-optimized through a large amount of training in an offline simulation environment, so that it can cope with complex and changeable actual application scenarios and enhance its adaptive ability.

[0045] Sp4: According to the optimal path planning generated by Sp3, the multi-degree-of-freedom manipulator 4 of the electric instrument platform body 1 is driven to perform motion control. The motion control dynamically adjusts the motion parameters of the manipulator 4 based on the real-time force feedback data obtained by Sp1, the instrument posture recognized by Sp2, and the path planning generated by Sp3;

[0046] In Sp4, the multi-degree-of-freedom manipulator 4 has at least six or more degrees of freedom, and each joint is equipped with a high-precision servo motor and an absolute encoder to ensure the accuracy and repeatability of the motion. The motion control adopts a strategy based on force-position hybrid control. During the non-contact motion stage of the manipulator 4, the system mainly performs precise position tracking control to ensure that the manipulator 4 strictly follows the path planning generated by Sp3. When the end effector of the manipulator 4 contacts the instrument or table, the control strategy receives and analyzes the force feedback data provided by the force sensor 3 obtained by Sp1 in real time, and accurately controls the joint torque and end force of the manipulator 4 based on this data, thereby achieving smooth contact with the instrument, force-controlled grasping and force-controlled placement. In addition, the motion control system adaptively and dynamically adjusts the joint speed, acceleration and target position of the manipulator 4 based on the instrument posture recognized in real time by Sp2 and the dynamic path planning generated by Sp3. This dynamic adjustment capability enables the manipulator 4 to adapt to subtle changes in the environment in real time and correct potential motion deviations, thereby significantly improving the stability of the operation and the final placement and picking accuracy;

[0047] Among them, the multi-degree-of-freedom manipulator 4 has at least six or more degrees of freedom to achieve the ability to reach any position and posture in space and meet the needs of complex grasping and placement. Each joint is equipped with a high-precision servo motor and encoder, and the end effector is an adaptive gripper with a force sensor 3 to adapt to instruments of different shapes and materials. When the end effector of the manipulator 4 contacts the instrument or table, the joint torque or end force of the manipulator 4 is precisely controlled according to the real-time force feedback data obtained by Sp1. When the gripper clamps the instrument, the clamping force is detected by the force sensor 3, and the motor torque is dynamically adjusted to keep the clamping force within the preset safety range, achieving smooth contact and avoiding damage to the instrument caused by excessive clamping force. According to the real-time data of the force sensor 3, the control system It can sense the interaction force between the robot arm 4 and the environment. If the force value is too large or too small, the control system will immediately adjust the movement speed, acceleration and even path of the robot arm 4 to avoid hard collision or grasping failure. The instrument posture recognized in real time by Sp2 will be input into the motion controller as a feedback signal. The controller will dynamically adjust the end posture and grasping point of the robot arm 4 according to these deviations to ensure precise alignment. The path planning generated by Sp3 is dynamic. When the environment changes, the path will be updated in real time. The motion controller of Sp4 will respond to these updates quickly to ensure that the robot arm 4 always moves along the latest and optimal path, achieving high-precision positioning, smooth movement, impact-free grasping and placement, and can remain stable even in the presence of small disturbances or environmental uncertainties.

[0048] Sp5: During the instrument grabbing and placement process, the real-time force feedback data obtained by Sp1 and the dynamic three-dimensional map constructed by Sp2 are continuously used to monitor potential collisions and abnormal forces between the robot arm 4 and the instrument, table, and environment. Once an abnormality is detected, the intelligent obstacle avoidance and safe shutdown mechanism are immediately activated, and the abnormal information is recorded;

[0049] In Sp5, the system continuously analyzes the real-time force feedback data provided by the force sensor 3 obtained by Sp1 and the geometric relationship between the robot arm 4 and the environment in the dynamic three-dimensional map constructed by Sp2 through a high-frequency, low-latency real-time monitoring mechanism. The specific monitoring content includes: calculating the minimum distance between the robot arm 4 body, the end effector and the instrument it grasps and all obstacles in the working area in real time, and comparing it with the preset safety distance threshold to predict potential collision risks. At the same time, the system continuously monitors the end contact force and joint torque fed back by the force sensor 3. When these force and torque values ​​suddenly exceed the preset collision force threshold or abnormal force threshold, it immediately determines that a hard collision or unexpected force has occurred. Once any form of abnormality is detected, the system The system will immediately trigger a multi-level intelligent obstacle avoidance and safety shutdown mechanism: For minor potential risks, the system will initiate real-time path replanning based on a dynamic three-dimensional map, guiding the robot arm 4 to avoid obstacles or escape abnormal contact areas by the shortest path and at the fastest speed, restoring a safe state without interrupting the mission. For serious collisions that have occurred or are about to occur, the system will immediately issue an emergency stop command, cutting off the drive power supply of the robot arm 4 and applying the brakes within milliseconds to ensure the absolute safety of personnel and equipment. The occurrence time, type of abnormality, objects involved, force and position data at the time, and system response of all abnormal events will be recorded in detail to form a fault log for subsequent system analysis, troubleshooting, and optimization of safety strategies.

[0050] Among them, the collision risk prediction in the real-time monitoring mechanism includes: using a dynamic three-dimensional map to calculate in real time the minimum distance between the end effector of the robot arm 4 and the instrument it grasps and all obstacles in the environment. When the minimum distance is lower than the preset safety distance threshold, a collision warning is issued. By monitoring the end force or joint torque fed back by the force sensor 3 in real time, when these force or torque values ​​suddenly change drastically or exceed the preset collision force threshold, it is immediately determined that abnormal contact or collision has occurred. This is not limited to collisions, but also includes abnormal forces caused by unexpected sliding or deformation of the instrument during grasping, or force imbalance caused by the instrument not being completely stable during placement.

[0051] Intelligent obstacle avoidance and safe shutdown mechanism: After an abnormality is detected, a response must be made within milliseconds to ensure human and machine safety and equipment protection. When a serious collision risk is detected or an actual collision occurs, the system will immediately send an emergency stop command to the robot arm 4 driver, quickly cut off the motor power or perform emergency braking, so that the robot arm stops moving in the shortest time. For non-emergency but potential risk situations, the system will re-plan a collision-free path based on the latest dynamic three-dimensional map to guide the robot arm 4 to avoid obstacles and continue to complete the task. If a minor collision occurs or the instrument slides accidentally, the system can plan a small reverse or lateral movement to make the robot arm 4 leave the abnormal contact area to prevent further damage, and record detailed information such as the time, location, type, instruments and obstacles involved, force and position data at the time, etc. for subsequent fault diagnosis, system optimization and accident tracing.

[0052] Sp6: After the instrument is successfully placed or taken, the high-precision machine vision data obtained by Sp1 is used to accurately verify the final position and posture of the instrument. The verification results are compared with the optimal path planning generated by Sp3 to achieve online optimization and model update.

[0053] In Sp6, after the instrument placement and pick-up tasks are successfully completed, the system uses the high-precision machine vision data obtained in Sp1 to capture high-definition images and depth information of the instrument's final state in real time through camera 5 from multiple preset perspectives or intelligently selected perspectives. Subsequently, the system uses advanced image processing technology and 3D reconstruction algorithms, including but not limited to feature point matching, multi-view geometry, point cloud registration, and precise alignment based on CAD models, to accurately calculate the instrument's final 3D spatial coordinates, Euler angle posture, and precise position deviation relative to tray 2 from the captured images, and compares the calculated results with the theoretical value set by the optimal path planning generated in Sp3. High-precision error analysis and comparison are performed on the target placement position and posture. When a deviation beyond the preset tolerance range is detected between the actual placement position or posture and the planned value, the system will automatically record the deviation data, deviation type, and corresponding operation context information. These deviation data will be used as feedback information for the online learning optimization mechanism to adjust and optimize the parameters of the path planning model in Sp3. Through this closed-loop "perception-execution-verification-optimization" iterative learning process, the system can continuously learn and adjust autonomously from actual operation experience, continuously improve the accuracy, stability, and task success rate of subsequent operations, and achieve continuous improvement of adaptive performance.

[0054] Sp7: For valuable and fragile instruments, a flexible grasping strategy is integrated into the motion control of Sp4. This strategy adjusts the gripping force and contact area of ​​the end effector of Manipulator 4, combined with the instrument material and structure data obtained by Sp1, to achieve flexible and non-destructive grasping of the instrument.

[0055] In Sp7, for valuable and fragile instruments, an intelligent flexible grasping strategy is integrated into the motion control of Sp4. The activation mechanism of the strategy is: the system pre-establishes and stores a comprehensive instrument characteristic database. The database records in detail the type of known instruments, unique identification ID, economic value level, accurate fragility coefficient, maximum allowable clamping force, surface characteristics, i.e. smoothness, roughness, scratchability, geometric structure characteristics, i.e. hollow, thin-walled, with fragile accessories, and specific grasping requirements, i.e. grasping only in specific areas and avoiding heat-sensitive contact. After executing Sp1 to obtain environmental perception data, the system uses deep learning-based image recognition and point cloud matching algorithms to perform type matching and feature extraction on the identified instruments, and compares them with the instrument characteristic database. Yes, when the identified instrument is marked as "precious" or "fragile" by the database, or its "fragile coefficient" reaches the preset flexible grasping trigger threshold, the flexible grasping strategy will be automatically activated. During specific execution, the strategy will dynamically adjust the clamping preload, contact area, grasping speed curve and clamping action time of the end effector of the robot arm 4 according to the corresponding specific parameters in the database. At the same time, the strategy will monitor the force feedback data provided by the sensor 3 in real time to form a closed-loop force control to ensure that the clamping torque is sufficient to stably clamp the instrument without causing any extrusion, deformation or internal damage to it. Through this refined control, surface scratches, cracks or other mechanical impacts on precious or fragile instruments can be effectively avoided, and precise, extremely gentle and non-destructive grasping and placement can be achieved.

[0056] A tray 2 is provided on the surface of the instrument platform body 1, a robotic arm 4 is provided on the surface of the instrument platform body 1, sensors 3 are provided on the surfaces of the tray 2 and the robotic arm 4, a camera 5 is provided on the surface of the instrument platform body 1, a laser radar 6 is provided on the surface of the instrument platform body 1, and the sensor 3 integrated on the surface of the tray 2 is a high-precision force sensor 3, which forms a three-dimensional force sensor 3 array for obtaining the precise weight distribution and center of gravity position of the instrument placed on the tray 2 in real time, as well as the contact force and pressure distribution between the instrument and the surface of the tray 2 when the robotic arm 4 places the instrument, which helps to judge whether the placement is stable. The sensor 3 of the end effector of the robotic arm 4 is a torque sensor 3, which is installed between the wrist and the gripper of the robotic arm 4, and is used to perceive the three-dimensional force and torque between the end effector and the instrument in real time during the grasping and placing process. The camera 5 is a high-resolution stereo vision camera 5, which is installed above the instrument platform body 1 to provide It provides a global bird's-eye view for capturing two-dimensional images and reconstructing three-dimensional depth of the entire working area. The stereo vision camera 5 calculates depth information through left-right eye parallax, and can accurately obtain the spatial position and size of instruments and obstacles. The lidar 6 is a high-precision laser ranging device installed around the instrument platform body 1, providing a wide scanning range, and generating high-density, high-precision three-dimensional point cloud data of the working area. Compared with the visual system, the lidar 6 performs more stably in complex environments such as lighting changes and lack of texture, and can make up for the shortcomings of the visual system. It is particularly suitable for obstacle detection and environmental modeling. It performs data preprocessing operations such as denoising, calibration, and time synchronization on data from different sensors 3. It uses extended Kalman filtering and unscented Kalman filtering to fuse multi-source heterogeneous data in a unified coordinate system, improve the accuracy and robustness of perception data, and make up for the limitations of a single sensor 3.

[0057] In addition, the electric instrument platform body 1 also includes a main control unit, which is electrically connected to the sensor 3, camera 5, lidar 6, and the driver of the robotic arm 4 via a high-speed data bus and control lines. The main control unit also includes a high-performance processor, large-capacity memory, and a multi-interface communication module. The main control unit's memory stores executable program instructions and data. By executing these program instructions, the processor can implement all functions of the intelligent control method, including multimodal perception data fusion processing, high-precision dynamic three-dimensional map construction, intelligent path planning, multi-degree-of-freedom motion control of the robotic arm 4, real-time anomaly detection and safe shutdown management, precise verification of operation results and online learning optimization, and intelligent execution of flexible grasping strategies for valuable and fragile instruments, thereby achieving efficient, precise, safe, and adaptive intelligent operation of the electric instrument platform.

[0058] This method achieves high-precision perception of the environment through multi-modal sensor fusion, and constructs a dynamic three-dimensional map to identify and track instruments and dynamic obstacles in real time. It uses optimal path planning based on model predictive control to drive the efficient movement of multi-degree-of-freedom robotic arms, greatly shortening the operation cycle and significantly improving operating efficiency and processing throughput. At the same time, it has real-time abnormality monitoring and multi-level safety shutdown mechanisms, which greatly enhance the robustness and safety of the system in complex environments. In particular, for valuable and fragile instruments, the present invention integrates a flexible grasping strategy and realizes non-destructive operation through real-time force feedback. Finally, through precise verification of machine vision and online optimization mechanism, the system can continuously adaptively learn and continuously improve operation accuracy and success rate, realizing intelligent, high-precision, high-efficiency and high-reliability automated operation. Specific embodiment two:

[0060] like Figures 1 to 3 As shown, based on the content in the above specific embodiments, the following contents are further disclosed:

[0061] The core algorithms and models of this method are mainly reflected in the following aspects:

[0062] Sp1: Multimodal sensor 3 fusion technology, which not only aggregates data from different sensors 3, but also uses complex algorithms to process these heterogeneous data for complementarity, redundancy, and consistency, thereby generating more comprehensive, accurate, and robust environmental perception data than a single sensor 3;

[0063] The high-precision force sensor 3 collects the force and torque data acting on the pallet 2 or the end effector of the robotic arm 4 in real time. The original data is usually an analog voltage signal, which needs to be converted into a digital signal by a high-precision analog-to-digital converter, and then subjected to noise filtering and zero-point calibration to ensure the purity and accuracy of the data. These data are usually sampled at a high frequency of hundreds to thousands of times per second to form a time series data stream. The high-resolution stereo vision camera 5 synchronously collects RGB images and corresponding depth maps. The image data will be de-distorted to eliminate lens geometric distortion, and the depth map will be hole-filled and noise-smoothed to compensate for data loss caused by occlusion or insufficient lighting. The stereo vision system generates depth information through parallax calculation, while the structured light camera 5 obtains depth by projecting specific patterns and analyzing their deformation. These data are output at a frequency of tens of frames per second and contain pixel-level color and depth information. The lidar 6 periodically emits laser beams to the environment and receives reflections, calculates distance by measuring flight time or phase difference, and collects The data obtained is a discrete three-dimensional point cloud, each point contains X, Y, Z coordinates and reflection intensity information. The point cloud data will be processed by outlier removal and downsampling to reduce the data volume and eliminate accidental noise points in the environment. These data are generated at a scanning frequency of tens to hundreds of times per second, providing dense three-dimensional spatial structure information. Different sensors 3 have different sampling frequencies and internal clocks. In order to ensure the accuracy of data fusion, high-precision timestamp synchronization is required, which is usually achieved through a hardware-triggered synchronization mechanism to ensure that the data collected at the same time can be correctly associated. Each sensor 3 has its own local coordinate system. Before data fusion, all sensor 3 data must be converted to a unified global coordinate system through precise sensor 3 internal and external parameter calibration. This includes optical distortion calibration of camera 5, odometer calibration of lidar 6, and external parameter calibration of all sensors 3 relative to the global coordinate system. Calibration is usually completed using a checkerboard, a specific calibration plate, or an optimization-based self-calibration method;

[0064] The raw data from different sensors 3 are extracted to extract their respective features, and then fused at the feature level. The state estimation fusion adopts the extended Kalman filter and unscented Kalman filter estimation algorithms. These algorithms can use the multi-source sensor 3 data as observation values ​​to optimally estimate the system state of the manipulator 4 itself, the instrument pose, the obstacle pose, etc. They can effectively deal with the noise and uncertainty of the sensor 3, and provide a more accurate and robust state estimation result than a single sensor 3;

[0065] In Sp1, these fusion technologies are specifically used to identify the instrument type and attributes, obtain the precise position and posture of the instrument, and perceive the status of the table in real time. Specifically, the color and texture information of the camera 5 image, the point cloud shape information of the lidar 6, and the weight information of the force sensor 3 are integrated to jointly determine whether it is a "glass test tube" or a "plastic culture dish"; the depth map of stereo vision and the point cloud data of the lidar 6 are integrated to reconstruct the three-dimensional model of the instrument with high precision, and the precise three-dimensional position and orientation of the instrument in the working area are calculated through point cloud registration algorithm or feature matching; the array of force sensors 3 on the surface of the tray 2 continuously monitors the pressure distribution thereon. When a new instrument is placed or an instrument is taken, the weight distribution will change, and the system can perceive the load status and occupied area of ​​the table in real time;

[0066] The data input is the original force and tactile sensor 3 data stream, RGB image sequence, depth map sequence, and lidar 6 point cloud sequence. The output is a high-precision 3D environmental perception data packet in a unified coordinate system that has been fused and preprocessed, including: a list of identified instruments, i.e., each instrument includes type, unique ID, precise 3D pose, geometric model, real-time load distribution map of pallet 2, 3D position and velocity information of dynamic obstacles, and other environmental features.

[0067] Sp2: Mainly relies on deep learning-based perception models and dynamic map construction technology;

[0068] In the high-precision dynamic 3D map construction process, the first input data comes from the fused high-precision 3D environment perception data package of Sp1, including images, depth maps, point clouds, and semantic labels. Using the fused visual and lidar6 data, a high-precision 3D geometric model of the entire working area is constructed through a real-time 3D reconstruction algorithm. The semantic segmentation and instance segmentation models are implemented using 3D convolutional neural networks. The models need to be trained offline on a large-scale and diverse instrument and obstacle dataset to learn their geometric features, texture features, and semantic information. The training data is usually images and point clouds with precise 3D annotations. When running online, the model receives the fused 3D perception data, semantically classifies each point or voxel in the scene, and performs instance segmentation, assigning a unique instance ID to each independent instrument individual. Finally, it outputs a colored point cloud and 3D mesh model with semantic labels, where each instrument instance has its own unique label and ID. The dynamic update mechanism adopts an incremental map update strategy. When new perception data flows in, the system does not rebuild the map from scratch every time. Instead, it uses the map at the previous moment and the new perception data to perform local or global map optimization through graph optimization methods to update the position, posture, and geometric details of objects in the map. Especially for dynamic obstacles, the system will detect, track and predict their trajectories in real time, dynamically mark them as "dynamic movable obstacles" on the map, and update their possible future locations;

[0069] In target recognition and pose estimation, based on the semantic segmentation and instance segmentation results output by the deep learning model, the system can accurately identify all instruments in the working area. For each identified instrument instance, its precise 3D pose in the global coordinate system is calculated through 3D point cloud registration. Identified fixed and dynamic obstacles are clearly marked in the dynamic 3D map and the spatial area they occupy is calculated. For dynamic obstacles, the system also estimates their speed and movement trend over a period of time.

[0070] Sp2 can accurately perceive the instrument status and understand all static and dynamic objects in the working area through high-precision dynamic three-dimensional maps, providing accurate obstacle information for subsequent collision-free path planning. When the instrument on the table changes or new objects enter, the map will be quickly updated to ensure the real-time decision-making of the robot arm 4. The input data is the high-precision three-dimensional environment perception data package output by Sp1, and the output data is a high-precision dynamic three-dimensional map, which lists all identified instrument instances and all obstacles, as well as the predicted trajectory of dynamic obstacles.

[0071] Sp3: Adopts an intelligent planning method based on model predictive control, which can generate optimal actions online and on a rolling basis and handle complex dynamic constraints;

[0072] The input content is a high-level operation task, which is to move the test tube A from position X to position Y. The current environmental state output by Sp2 is the current position of the robot arm 4, the current position and target position of the instrument A, and the dynamic three-dimensional map. The system decomposes the high-level task into a series of subtasks, such as "approaching test tube A", "grabbing test tube A", "lifting test tube A", "moving to the target area", "placing test tube A", "detaching from test tube A", etc., and defines precise starting and target states for each subtask, including the position of the end effector of the robot arm 4, the opening and closing state of the gripper, and the target position of the instrument in space. The model predictive control planner mainly includes a prediction model and an optimizer. The prediction model includes the precise kinematic model, dynamic model and end effector model of the robot arm 4. In addition, it also includes an environmental model, that is, the dynamic three-dimensional map output by Sp2, which is used to predict the robot arm 4 in the future. Interaction with the environment, through the cost function to optimize trajectory tracking error, collision penalty, motion smoothness, energy consumption, time efficiency and grasping posture, the model predictive control online optimization process obtains the state of the robot arm 4 in real time, predicts the motion of dynamic obstacles, constructs and solves nonlinear optimization problems, and uses efficient nonlinear optimization algorithms to generate the optimal control sequence in each control cycle, and executes it in a rolling manner to adapt to environmental changes. The optimization process must meet multiple constraints: considering the instrument's graspable area, center of gravity, fragility level and other characteristics, ensuring that the table load is reasonably distributed during placement, avoiding over-limit or unstable areas, ensuring that the robot arm 4 and the grasping instrument maintain a safe distance from fixed and dynamic obstacles, and complying with the robot arm 4 joint angle, speed, acceleration and torque limits. In multi-task scenarios, the model predictive control is flexibly adjusted according to task priority or batch processing order to ensure safe, accurate and efficient completion of operations;

[0073] Through model predictive control, Sp3 enables the robot arm 4 to cope with complex scenarios such as multiple instruments, multiple tasks, and dynamic obstacles, and autonomously generate safe and effective paths. It not only avoids obstacles, but also takes into account multiple performance indicators such as time, energy, and stability. When the actual environment deviates from the prediction, the model predictive control can quickly re-plan to avoid rigid preset paths. The input data is the high-precision dynamic three-dimensional map, instrument posture, obstacle information, and high-level task instructions output by Sp2. The output data is the joint space motion trajectory and Cartesian space motion trajectory of the multi-degree-of-freedom robot arm 4, and also includes the end effector posture and gripper opening and closing instructions during grasping and placing.

[0074] Sp4: Multi-DOF robotic arm 4 motion control and dynamic adjustment;

[0075] Sp4 adopts a force-position hybrid control strategy to convert the abstract trajectory instructions of Sp3 into precise, force-controlled physical movements of the robot arm 4. The input includes the optimal path planning of Sp3, the real-time force feedback data of Sp1 and the real-time instrument posture of Sp2. The control strategy divides the six degrees of freedom of the robot arm 4 into position control and force control directions: position control is mainly used when moving in the air, and the clamping direction is switched to force control during contact tasks to ensure accurate position arrival and smooth force interaction. The joint controller uses PID control to adjust the servo motor torque according to the feedback of the joint encoder to accurately track the target trajectory. The force-position hybrid control logic calculates the end position deviation in the position tracking stage to generate the joint torque instruction The system intelligently determines when to enter the force control stage based on the intention of Sp3 planning and the feedback data of Sp1's real-time force sensor 3. When the end effector contacts the instrument, the reading of force sensor 3 will change, and the system will switch to force control mode accordingly. In force control mode, the controller dynamically adjusts the joint torque of the manipulator 4 according to the real-time force feedback data of force sensor 3 to achieve the preset target force. Through Sp4's real-time response to abnormal force feedback and Sp2's instrument posture deviation, the control system will fine-tune the speed, acceleration or posture of the manipulator 4 to ensure smooth operation. If Sp3 updates the path due to environmental changes, Sp4 responds smoothly to the new instruction through the trajectory smoother and interpolation algorithm.

[0076] Sp4 ensures that the robot arm 4 can move accurately according to the planned trajectory, and can make real-time adjustments based on slight changes in the environment and force feedback to improve the robustness of the operation. The input data is the target trajectory data of the robot arm 4 output by Sp3, the real-time force sensor 3 data output by Sp1, and the real-time instrument posture output by Sp2. The output data is the servo motor drive instruction, which is sent to the servo drivers of each joint of the robot arm 4 through the high-speed communication bus.

[0077] Sp5: Real-time monitoring and intelligent obstacle avoidance, safe shutdown mechanism;

[0078] Sp5 ensures operational efficiency and the safety of personnel and equipment by real-time monitoring of the force feedback data of Sp1 and the dynamic three-dimensional map of Sp2. The abnormality monitoring algorithm includes geometry-based collision detection and force-based abnormal force detection, and integrates the results of the two for risk assessment. The geometry-based collision detection adopts a collision detection algorithm to calculate the minimum Euclidean distance between the geometric model of the robot arm 4 and the geometric models of all obstacles in the dynamic three-dimensional map in real time. When this minimum distance is less than the preset safety distance threshold, the system issues a collision warning. The input is the real-time geometric model and posture of the robot arm 4 and the obstacle, and the output is a Boolean value, namely whether there is a collision and the minimum distance. The force-based abnormal force detection adopts threshold-based anomaly detection to monitor the force and torque components fed back by the force sensor 3 in real time. When the amplitude of any component or the composite force and torque suddenly exceeds the preset collision force threshold or the maximum allowable force threshold, or the force curve shows unexpected high-frequency oscillation, the system determines that abnormal force or collision has occurred. These thresholds are based on the maximum load, instrument The vulnerability of the device and the safety standards are pre-set. The input is real-time force sensor 3 data, and the output is a Boolean value, namely whether abnormal force and the magnitude and direction of the abnormal force. According to the output of the abnormal monitoring algorithm, the system evaluates the risk level. According to the risk level, Sp5 takes a graded response: when the risk is low, the collision area is marked as a no-go zone, triggering Sp3 to re-plan the obstacle avoidance trajectory to ensure a smooth transition. When the risk is medium or high or a collision has occurred, a hardware-level emergency stop signal is immediately sent to cut off the motor power or brake, and the robot arm 4 enters a safe shutdown mode. If the instrument is stuck, the system plans a small-amplitude disengagement movement to remove the abnormal contact. All abnormal events are recorded in detail in the log, including time, type, posture, force readings and response measures, to provide support for fault diagnosis, safety optimization and accident tracing. Among them, the input data is the real-time force feedback data output by Sp1, the dynamic three-dimensional map and real-time object posture output by Sp2, and the output data is collision warning signal, path replanning request, emergency stop command, disengagement command and abnormal contact data.

[0079] Sp6: Accurate verification with online optimization and model updating;

[0080] In Sp6, after the instrument is grasped or placed, the system self-correction and performance improvement are achieved through precise verification and online optimization. In the verification stage, the machine vision data of Sp1 is used to reconstruct the three-dimensional posture of the instrument through image preprocessing, feature extraction and matching, and multi-view geometry. The precise position and posture in the global coordinate system are calculated, and the dimension-by-dimension error analysis is performed with the ideal target planned by Sp3 to evaluate the position deviation and angle deviation to ensure that the instrument is placed stably without tilt. When the deviation exceeds the preset tolerance range, Sp6 records the deviation data, namely the amount, direction, instrument type, environmental conditions, and planning parameters, to the historical operation database, and adopts the error-based reverse The propagation optimization mechanism adjusts the parameters of the Sp3 path planning model, including target weight, correction factor and grasping posture parameters. The optimization can be performed periodically or event-driven. Through closed-loop iteration, the system autonomously identifies and corrects systematic errors, continuously improves operation accuracy, robustness and success rate, and realizes the adaptability and high reliability of the electric instrument platform body 1. The input data is the high-precision machine vision data obtained by Sp1 and the ideal target posture in the optimal path planning generated by Sp3. The output data is the deviation value between the actual placement posture and the target posture, as well as the deviation data and optimization instructions, which are used for online parameter adjustment and update of the path planning model in Sp3.

[0081] Sp7: Flexible grasping strategy for valuable and fragile instruments;

[0082] Sp7 ensures non-destructive and safe operation of high-value or fragile instruments through the instrument characteristic database and flexible grasping strategy. The database stores the instrument type, value level, fragility coefficient, maximum allowable clamping force, surface characteristics, grasping requirements and recommended fixture type. Sp2 uses the perception data of Sp1. The cognitive module of Sp2 will accurately identify the type and unique ID of the instrument to be operated through image recognition, point cloud matching and multimodal fusion recognition algorithm based on deep learning, query the database to obtain attributes, and the system will retrieve and obtain the detailed attributes such as the fragility coefficient, maximum allowable clamping force, surface characteristics and specific grasping requirements corresponding to the instrument. If the fragility coefficient exceeds the threshold or is marked For valuable instruments, the system determines them to be fragile / valuable instruments and activates the flexible grasping strategy. The flexible grasping strategy uses database attributes and feedback from Sp1 force sensor 3 as input, dynamically adjusts the clamping preload, grasping speed curve, clamping time, contact point and area, and Sp4 monitors the clamping force through real-time force feedback closed-loop control, makes timely adjustments to prevent over-force or sliding, and optimizes contact materials and paths based on surface characteristics to prevent scratches or damage, ensuring safe and precise operation and protecting valuable and fragile instruments. The input data is the instrument characteristic database, the environmental perception data obtained by Sp1, the real-time motion control parameters of Sp4, and the output data is the motion control index of Sp4 to ensure flexible grasping. Specific embodiment three:

[0084] like Figures 1 to 3As shown, based on the content in the above specific embodiments, the following contents are further disclosed:

[0085] The intelligent control method for rapid placement and retrieval of electric instrument platforms can be divided into multiple core software modules at the logical level. The main body of the entire intelligent control system consists of the following key software modules:

[0086] The perception fusion module is responsible for acquiring raw data from various sensors and converting it into a unified format that the system can understand. It performs time synchronization and spatial calibration of multi-source heterogeneous data to ensure that data from different sensors are accurately aligned. Subsequently, it applies a multi-sensor fusion algorithm to generate high-precision, multi-dimensional, denoised environmental perception data packets. These data contain detailed information about the instrument, table, and environment, and are the basis for all subsequent decision-making.

[0087] The Environmental Cognition Module is responsible for deeply understanding and analyzing the data output by the Perception Fusion Module. It first uses 3D reconstruction technology to construct a high-precision, dynamic 3D map of the work area. This map incorporates not only geometric information but also semantic information. Then, using a deep learning-based recognition model, it performs real-time semantic segmentation and instance-level recognition of objects in the map, accurately distinguishing and marking various instruments to be operated and their attributes, including instrument type, precise position, size, and material. The module also detects and tracks dynamic obstacles in real time and predicts their motion trajectories, ensuring the system's comprehensive and dynamic understanding of the environment.

[0088] The task planning module generates the optimal action sequence based on the task instructions input by the user and the real-time environmental information provided by the environmental cognition module. It comprehensively considers the characteristics of the instrument, the load distribution of pallet 2, the location of all obstacles, and the kinematic and dynamic constraints of the robot arm 4 itself. This module uses an intelligent planning algorithm based on model predictive control to generate an online and rolling three-dimensional motion trajectory of the robot arm 4 from the starting point to the target point without collision, with optimal time, minimum energy consumption, and a smooth path. It also optimizes the grasping or placement posture of the robot arm 4 end effector.

[0089] The motion control module is responsible for converting the abstract trajectory instructions generated by the task planning module into actual, precise physical motions of the robotic arm 4. It adopts a force-position hybrid control strategy, enabling precise position tracking during the non-contact phase. When in contact with instruments or the environment, it can achieve smooth force-controlled operation based on real-time force feedback data. This module also has dynamic adjustment capabilities. It can adjust the joint speed, acceleration, and target position of the robotic arm 4 in real time based on perceived instrument posture deviations and environmental changes to ensure operational stability and high precision.

[0090] The safety monitoring module runs through the entire operational process to ensure human-machine collaboration and instrument safety. It continuously receives and analyzes real-time force feedback data provided by the perception fusion module and the dynamic three-dimensional map constructed by the environmental cognition module. Through anomaly detection algorithms based on geometry and force, it can determine in real time whether there is a potential collision or abnormal force between the robot arm 4 and the instrument, table, or environment. Once any anomaly is detected, the module will immediately trigger a graded response mechanism, including initiating path replanning to avoid obstacles or issuing an emergency stop command when necessary. It also records all abnormal information in detail for subsequent analysis.

[0091] The performance optimization module is responsible for evaluating and continuously improving operational results. After an instrument is successfully placed or retrieved, it uses the high-precision machine vision data provided by the perception fusion module to accurately calculate the instrument's final position and attitude through image processing and 3D reconstruction technology. It then performs error analysis on the actual results compared with the ideal target set by the mission planning module. When deviations outside the tolerance range are detected, these deviations are used as feedback to adjust and optimize the parameters of the mission planning module, thereby achieving online optimization and model updates of the system, continuously improving the accuracy, robustness, and success rate of operations.

[0092] The instrument management module is responsible for storing and managing specialized information related to instruments. Its core is the instrument characteristics database, which details various known instrument attributes such as type, value level, fragility coefficient, maximum allowable gripping force, surface characteristics, and specific gripping requirements. Upon identifying an instrument, the environmental awareness module queries this database to determine whether the instrument is valuable or fragile, and to provide the necessary parameters for the flexible gripping strategy. This module is also responsible for managing historical operation data and instrument inventory information.

[0093] Human-computer interaction module: The human-computer interaction module is the interface between the system and the user. It provides an intuitive user interface that allows operators to input task instructions, monitor system operating status, view real-time perception data and operation logs, and perform parameter settings and fault diagnosis. It is also responsible for feeding back the system's operating status, warning information, and completion status to the operator.

[0094] The operation and function realization of the above software modules require the support of the following core hardware structures:

[0095] The main control unit is the core computing and control brain of the entire system. It is usually composed of a high-performance industrial-grade processor and a graphics processor that supports parallel computing. It is equipped with a large-capacity high-speed random access memory and non-volatile memory for storing the operating system, program instructions of all software modules, instrument characteristic database and historical operation data. The unit integrates multiple communication interfaces, including but not limited to industrial Ethernet interface, universal serial bus interface, serial communication interface and digital and analog input and output interfaces, for high-speed data exchange and control signal transmission with all sensors 3, robotic arm 4 drives, host computer and external networks. The main control unit is the carrier and executor of all software modules. It executes all functions and algorithms of the perception fusion module, environmental cognition module, task planning module, motion control module, safety monitoring module, performance optimization module, instrument management module and human-computer interaction module;

[0096] The sensor cluster 3 is the system's primary sensing hardware for acquiring environmental information. High-precision force sensors 3 are integrated on the surface of the tray 2, the end-effector of the robotic arm 4, and key force points on the instrument platform 1. A high-resolution stereo vision camera 5 is typically mounted above the instrument platform 1, and a high-precision lidar 6 is typically installed around the instrument platform 1. These sensors 3 are directly connected to the perception fusion module of the main control unit via their respective data cables and communication interfaces, providing the module with raw, real-time environmental perception data. The multi-degree-of-freedom robotic arm 4 system is the core actuator for the system's operations. It includes the robotic arm 4, its internally integrated high-precision servo motors and encoders, and a replaceable end-effector. Each joint actuator of the robotic arm 4 is connected to the main control unit via an industrial bus, receiving instructions from the motion control module, executing the planned motion trajectory, and the end-effector realizes the grasping and placement of instruments. The instrument platform 1 is the physical framework of the entire system, providing a stable foundation and mounting platform. The tray 2 directly carries the instruments to be operated. Some sensors 3 are integrated on its surface, providing physical workspace and support for instrument placement and robotic arm 4 operation. As part of the sensors 3, it also provides the perception fusion module with tabletop status information.

[0097] Through the logical division of the above software modules and the coordinated work of the corresponding hardware structures, this method can realize a highly integrated, intelligent and efficient electric instrument stage control system. Specific embodiment four:

[0099] like Figures 1 to 3 As shown, based on the content in the above specific embodiments, the following contents are further disclosed:

[0100] In order to further verify the feasibility of the technical solution of this application, the following case is further explained:

[0101] Use Case 1: Automated sample pre-processing in the central laboratory department of a large hospital;

[0102] Scenario Background: The central laboratory department of a large tertiary hospital processes thousands of blood, body fluid, and other samples daily. The traditional manual processing process, including sample sorting, centrifugation, decapping, pipetting, and placement on analytical instruments, is not only labor-intensive but also poses sample mix-up, aerosol contamination, and biosafety risks. To improve efficiency and safety, the laboratory department introduced an electric instrument table equipped with this intelligent control method.

[0103] A testing technician places a batch of test tube racks with tube caps of different colors on the designated sample loading area of ​​the instrument body 1. After the system is started, the high-resolution stereoscopic vision camera 5 and lidar 6 located above the instrument body 1 start working immediately. The camera 5 identifies the type of test tube by color and barcode, namely blood routine tube, biochemistry tube, and coagulation tube, while the lidar 6 scans the entire working area and fuses the data with the camera 5 to construct a high-precision dynamic three-dimensional map of all elements including all test tube racks, single test tubes, centrifuge slots, opener modules, and analyzer assembly line entrances. At the same time, the force sensor 3 array on the surface of the tray 2 senses the weight change caused by the placement of the test tube rack and confirms that the sample is in place; after the control system receives the "start processing" command, the task planning module automatically generates a task sequence based on the identified test tube type and the preset standard operating procedure, namely, the blood routine tube needs to be mixed first, and the regeneration chemical tube needs to be centrifuged first. The system adopts a planning method based on model predictive control. , generating the optimal path for the robot arm 4. The planner will calculate an optimal three-dimensional motion trajectory from grabbing the test tube to placing it in the centrifuge, then to the opener, and finally to the analyzer assembly line. This trajectory is not only collision-free, but also comprehensively considers the shortest time and lowest energy consumption, and ensures smooth movement to avoid sample oscillation caused by violent movement; the multi-degree-of-freedom robot arm 4 begins to perform the task. When approaching the target test tube, the system adopts a force-position hybrid control strategy. When moving in the air, it mainly uses high-precision position control, which is fast and accurate. When the end gripper is about to contact the test tube, the control mode switches seamlessly. For fragile glass test tubes, the system activates the flexible grasping strategy. It queries the instrument characteristic database, matches the "fragility coefficient" and "maximum allowable gripping force" of the "glass test tube", and then instructs the gripper to apply a precisely calculated and very gentle preload. The force sensor 3 on the gripper provides real-time feedback of the contact force, forming a closed-loop force control, ensuring that the gripping force can both firmly grasp and never damage the test tube.

[0104] During an operation, a technician accidentally dropped a marker pen into the working area of ​​the instrument body 1. The system safety monitoring module immediately detected this new "dynamic obstacle" through the real-time update of the dynamic three-dimensional map. Before the robot arm 4 was about to pass through the area, the system triggered the intelligent obstacle avoidance mechanism. The robot arm 4 smoothly paused outside a safety threshold from the marker pen. At the same time, the task planning module re-planned a new path to bypass the obstacle within milliseconds. The robot arm 4 then continued to perform the task along the new path. The whole process was seamless, without affecting the overall efficiency and ensuring the safety of the equipment. 4. After a test tube is placed in the centrifuge slot, camera 5 immediately captures a high-resolution image and uses 3D reconstruction technology to accurately calculate the final position and posture of the test tube. The system compares this actual result with the planned target position and discovers that due to slight wear on the slot, the actual position of the test tube has a slight deviation of 0.2 mm. The performance optimization module records this deviation data and uses it as feedback to automatically fine-tune the path planning model parameters for all subsequent placements in that specific slot. After several rounds of self-learning and iteration, the system's subsequent placement accuracy in that slot has been significantly improved, achieving adaptive adjustment.

[0105] Case effect: Through the fusion perception of multimodal sensors 3 and the construction of high-precision dynamic three-dimensional maps, the low efficiency and high error rate problems of sample sorting and placement in traditional manual operations are effectively solved. Intelligent path planning based on model predictive control ensures collision-free and high-efficiency movement of the robot arm 4 in complex working areas, and significantly reduces sample vibration caused by improper operation. The application of force-position hybrid control and flexible grasping strategy realizes smooth and non-destructive operation of various test tubes, especially when handling fragile glass test tubes, avoiding the risk of damage. Real-time safety monitoring and intelligent obstacle avoidance mechanism effectively improve the operational safety of the system and reduce the biosafety risks and accident rates caused by manual intervention. In addition, the precise verification of operation results and the online optimization mechanism enable the system to adaptively adjust and optimize performance according to actual deviations, solving the problem of accuracy degradation that may be caused by long-term operation of the equipment, and ensuring high precision and repeatability of sample processing.

[0106] Use Case 2: High-throughput screening in new drug development laboratories;

[0107] Scenario Background: A leading global pharmaceutical company needs to perform high-throughput screening of tens of thousands of compounds during the drug discovery phase. This work hinges on the precise dispensing of micro-liquids using 384-well plates, followed by cell culture and testing. Operational accuracy, reproducibility, and non-destructive handling of expensive compounds and cell lines are crucial.

[0108] At the beginning of the experiment, multiple 384-well plates, different types of reagent bottles, a pipetting workstation and an interface for a cell culture incubator were placed in the working area of ​​the electric instrument table body 1. After the system was started, the multimodal sensor 3 fusion technology began to operate. The stereo vision camera 5 was responsible for identifying the QR code on the well plate, confirming the well plate ID and experimental batch, distinguishing different colors of reagent caps, and performing preliminary positioning. The lidar 6 provided accurate three-dimensional point cloud data that was not affected by the reflection of transparent plastic, which was used to accurately construct the geometric model and spatial posture of the well plate and reagent bottle. After the data of the two were fused, the environmental cognition module constructed a dynamic three-dimensional map containing the precise position and semantic information of all objects; the researcher issued an instruction through the human-computer interaction interface: "Execute high-throughput screening experiment-Scheme S3", and the task planning module immediately parsed the instruction and carried out the task according to the requirements of Scheme S3. , generating a series of complex subtasks: grabbing the specified well plate, moving it to the bottom of the pipetting workstation, waiting for the pipetting to be completed, and then accurately placing the well plate into the incubator interface. The planner not only calculates the collision-free path of the robot arm 4, but also optimizes the posture of the end effector to ensure that the plane of the well plate and the pipette tip remain absolutely parallel when under the pipetting workstation. The 384-well plate is a thin-walled plastic product. Although its value is not high, the cells and compounds it carries are priceless. Any deformation may cause the experiment to fail. When the robot arm 4 grabs the well plate, the flexible grasping strategy is activated. The system queries the database and matches the characteristics of the "384-well plate". A large surface contact gripper is used, and according to its "maximum allowable clamping force" and real-time force feedback, a uniform and small clamping force is applied to ensure that the well plate is stable and without any deformation during rapid movement.

[0109] During a grasping process, due to a tiny drop of liquid from the pipette tip in the previous step, the surface of the well plate was slipperier than expected. When the robot arm 4 clamped the well plate and lifted it, the force sensor 3 of the end effector detected a slight sliding trend. Even though the sliding amplitude was far from visible to the naked eye, the abnormal force detection algorithm of the safety monitoring module immediately captured this anomaly. The system determined that there was a risk of "unstable grasping" and immediately triggered an emergency stop command. The robot arm 4 hovered stably in place and issued an alarm to the operator, avoiding the catastrophic consequence of the well plate falling or overturning. When the well plate is placed into the incubator interface, the alignment accuracy requirement is extremely high. After the robotic arm 4 completes the placement, the camera 5 takes images of the interface and the well plate from multiple angles. The performance optimization module calculates the final position and posture of the well plate through three-dimensional reconstruction and model matching, and compares it with the planned ideal target. The system records each slight deviation and continuously optimizes the end trajectory correction model of the robotic arm 4 through an online learning algorithm. This ensures that after long-term use, the instrument body 1 will not reduce its accuracy due to mechanical wear. Instead, it can achieve increasingly higher placement success rate and accuracy through self-learning.

[0110] Case Results: Through the fusion of high-precision multimodal sensors and environmental awareness, this system effectively addresses the difficulty of ensuring the accuracy and repeatability of micro-liquid dispensing and well plate placement in traditional high-throughput screening, which is often difficult to achieve with manual operations. Intelligent task planning and optimal path generation based on model predictive control significantly improve the automation and parallel processing capabilities of experimental processes, achieving unprecedented high-throughput processing efficiency. The application of flexible gripping strategies, especially the uniform and gentle gripping of thin-walled well plates, effectively avoids well plate deformation and damage to precious compounds and cell lines within, significantly reducing experimental failure rates and the loss of expensive reagents. Real-time abnormal force detection and an emergency stop mechanism provide immediate response to minor slip trends, effectively resolving the technical challenge of traditional automated equipment's inability to avoid catastrophic consequences under extreme conditions. This significantly ensures experimental safety and the integrity of expensive samples. Furthermore, precise verification and online optimization mechanisms enable the system to continuously and adaptively improve placement accuracy and success rate based on minor deviations, addressing the accuracy degradation that can occur with long-term operation of automated equipment and ensuring the high repeatability and reliability of data results during new drug development.

[0111] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further restrictions, an element defined by the statement "comprising a reference structure" does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0112] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent control method for an electric instrument platform based on multimodal perception and model prediction, wherein the hardware structure of the intelligent control method includes an instrument platform body (1), a tray (2), a sensor (3), a robotic arm (4), a camera (5) and a laser radar (6), and is characterized by: The intelligent control method comprises the following steps: Sp1: Acquire three-dimensional high-precision environmental perception data of the working area of ​​the electric instrument table (1), including but not limited to instrument type, size, shape, weight, placement and table status, and the acquisition is achieved through the fusion technology of sensors (3), cameras (5) and laser radar (6); Sp2: Based on the environmental perception data obtained by Sp1, it processes and analyzes the data in real time, builds a high-precision dynamic three-dimensional map of the work area, and identifies and marks all instruments and potential obstacles; Sp3: Based on the dynamic 3D map and recognition results constructed by Sp2 and the preset operation task requirements, it generates the optimal path planning for instrument placement and retrieval; Sp4: According to the optimal path planning generated by Sp3, the multi-degree-of-freedom manipulator (4) of the electric instrument platform body (1) is driven to perform motion control, and the motion parameters of the manipulator (4) are dynamically adjusted; Sp5: During the process of grasping and placing the instrument, the real-time force feedback data obtained by Sp1 and the dynamic three-dimensional map constructed by Sp2 are continuously used to monitor the potential collision and abnormal force between the robot arm (4) and the instrument, table and environment in real time; Sp6: After the instrument is successfully placed or taken, the high-precision machine vision data obtained by Sp1 is used to accurately verify the final position and posture of the instrument, enabling online optimization and model update; Sp7: For valuable and fragile instruments, a flexible grasping strategy is integrated into the motion control of Sp4, combined with the instrument material and structure data obtained by Sp1, to achieve flexible and non-destructive grasping of the instruments.

2. The intelligent control method for electric instrument platforms based on multimodal perception and model prediction according to claim 1 is characterized in that: In the Sp1, the sensor (3) is a high-precision force sensor (3) integrated on the surface of the tray (2), the end effector of the robotic arm (4) and other key positions of the instrument platform (1), which is used to obtain the weight distribution of the instrument when it is placed, the contact force during grasping and the environmental interaction force in real time. The camera (5) is a high-resolution stereo vision camera (5) installed above the instrument platform (1), which is used to obtain three-dimensional image information of the working area. The laser radar (6) is used to generate accurate point cloud data of the working area. Through the collaborative perception of multiple sensors (3), high-precision, multi-dimensional perception of all objects and their states in the working area is achieved.

3. The intelligent control method for electric instrument platforms based on multimodal perception and model prediction according to claim 1 is characterized in that: In the above-mentioned Sp2, the heterogeneous data obtained by the sensor (3), camera (5) and lidar (6) are synchronized in time and space and information is fused through a data fusion algorithm to generate a point cloud model and a semantic map containing precise position, posture and geometric shape. Based on the semantic map, various instruments to be operated in the working area and fixed or mobile obstacles in the environment are accurately identified and distinguished, and real-time tracking and posture estimation are performed on them to ensure dynamic update and high precision of map information.

4. The intelligent control method for electric instrument platforms based on multimodal perception and model prediction according to claim 1 is characterized in that: In the above-mentioned Sp3, based on the high-precision dynamic three-dimensional map and recognition results constructed by Sp2, combined with the preset instrument characteristics, table load distribution, obstacle avoidance requirements, and the priority and timing requirements of the preset instrument placement and picking tasks, a planning method based on model predictive control is used to generate a collision-free, optimal time, minimum energy consumption and smooth path three-dimensional motion trajectory of the robotic arm (4) from the starting point to the target point, taking into account the kinematics and dynamic constraints of the robotic arm (4) itself and the obstacle avoidance requirements.

5. The intelligent control method for electric instrument platforms based on multimodal perception and model prediction according to claim 1 is characterized in that: In the Sp4, the multi-degree-of-freedom manipulator (4) has at least six or more degrees of freedom, and the motion control adopts a strategy based on force-position hybrid control. According to the real-time force feedback data obtained by Sp1, the joint torque and end force of the manipulator (4) are precisely controlled to achieve smooth contact and force-controlled operation with the instrument. At the same time, according to the instrument posture recognized by Sp2 and the path planning generated by Sp3, the joint speed, acceleration and target position of the manipulator (4) are dynamically adjusted to adapt to environmental changes and improve operation accuracy.

6. The intelligent control method for electric instrument platforms based on multimodal perception and model prediction according to claim 1 is characterized in that: In the Sp5, the presence of a risk of contact with obstacles and instruments in the motion path of the robot arm (4) is determined in real time by using a preset safety distance threshold and collision force threshold. When it is detected that the force and torque fed back by the force sensor (3) exceed the safety range, an emergency stop command is immediately triggered, and a path replanning mechanism based on a dynamic three-dimensional map is simultaneously started to guide the robot arm (4) to avoid obstacles and leave the abnormal contact area, thereby ensuring operational safety.

7. The intelligent control method for electric instrument platforms based on multimodal perception and model prediction according to claim 1 is characterized in that: In the above-mentioned Sp6, the multi-view images of the instrument after placement are obtained by the camera (5) for accurate verification. The three-dimensional spatial coordinates and Euler angle posture of the instrument are accurately calculated by using image processing and three-dimensional reconstruction technology. The calculated results are compared with the preset ideal placement position and posture for error analysis. When a deviation is detected between the actual placement position and posture and the planned value, the system will automatically record the deviation data and use it as feedback information to adjust and optimize the parameters of the path planning model in Sp3, so as to achieve continuous improvement and adaptive adjustment of system performance.

8. The intelligent control method for electric instrument platforms based on multimodal perception and model prediction according to claim 1 is characterized in that: In the Sp7, an instrument characteristic database is pre-established and stored. The database contains the type, value level, fragility coefficient, maximum allowable clamping force, surface characteristics and specific grasping requirements of known instruments. When executing Sp1 to obtain environmental perception data, the identified instrument is matched with the database through the instrument identification and matching method to determine whether it is a valuable or fragile instrument. When it is determined to be a valuable or fragile instrument, the flexible grasping strategy will dynamically adjust the clamping preload, grasping speed curve and clamping action time of the end effector of the robot arm (4) according to the corresponding parameters in the database, and combine the real-time force feedback data provided by the sensor (3) to ensure that the grasping torque is sufficient to stably clamp the instrument without causing any extrusion or impact to it, avoiding surface scratches or internal damage to the valuable or fragile instrument, and achieving precise and gentle operation.

9. The hardware structure of the intelligent control method for electric instrument platforms based on multimodal perception and model prediction according to claim 1 is characterized by: A tray (2) is provided on the surface of the instrument platform body (1), a robotic arm (4) is provided on the surface of the instrument platform body (1), sensors (3) are provided on the surfaces of the tray (2) and the robotic arm (4), a camera (5) is provided on the surface of the instrument platform body (1), and a laser radar (6) is provided on the surface of the instrument platform body (1).

Images