Visual self-calibration system of automatic microneedle patch equipment and calibration method of visual self-calibration system

Abstract

The invention, which relates to the technical field of device testing, discloses a visual self-calibration system for a microneedle automatic patch device and a calibration method thereof, comprising: a data acquisition module for measuring a spatial relationship among a composite calibration target, a calibration visual camera, a laser contourgraph and a manipulator coordinate system; the compensation model building module is used for building a corresponding relation model of the grabbing condition and the pose compensation amount based on patch deformation data collected by the laser contourgraph; the drifting prediction module is used for monitoring the motor temperature and the movement mileage and establishing a drifting prediction model; the deformation pre-compensation module is used for acquiring deformation characteristics after the patch is picked up and processing and acquiring a pose pre-compensation amount by utilizing a compensation model; the deformation compensation correction module fuses the pose pre-compensation amount and the drift pre-compensation amount, generates a final compensation control instruction and drives the manipulator to complete compensation motion; under complex operation conditions, the deformation compensation model shows great adaptability and robustness.

Description

Technical Field

[0001] This invention relates to the field of device testing technology, specifically to a visual self-calibration system and calibration method for an automated microneedle patching device. Background Technology

[0002] In the modern medical and aesthetic industries, automated microneedle patching devices are widely used in areas such as drug delivery. These devices utilize microneedle technology to precisely attach drugs or other therapeutic ingredients to the target. However, due to the deformation of the microneedle patch and the influence of external factors, ensuring accurate placement and deformation compensation of the patch during operation has become a key issue in improving device performance and safety. Existing technologies typically rely on static calibration and simple machine vision algorithms to achieve device positioning and patch placement. These methods generally operate using pre-set parameters and empirical models, ensuring patch accuracy to a certain extent. In addition, some devices attempt to use sensors to monitor environmental changes for dynamic compensation. However, these technologies are mostly single-function, often unable to adapt to different operating conditions in real time, and their analysis of deformation characteristics is relatively simple and lacks specificity. While existing technologies have mitigated errors in the patch placement process to some extent, significant shortcomings remain. Key issues include a lack of in-depth analysis and dynamic compensation capabilities for patch deformation characteristics, limiting placement accuracy and consistency in practical applications. Furthermore, current technologies fail to adequately consider the impact of equipment operating conditions on operational accuracy, resulting in ineffective prediction and compensation for robot drift during prolonged use. Therefore, developing a self-calibration system capable of real-time monitoring, analysis, and compensation for deformation and drift is particularly necessary.

[0003] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a visual self-calibration system and calibration method for an automated microneedle patching device, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A visual self-calibration system for an automated microneedle patch application device, specifically comprising: The data acquisition module is used to control the robotic arm of the equipment to measure the composite calibration target fixed in the workspace and to complete the spatial relationship calibration of the coordinate system of the vision camera, the laser profilometer and the robotic arm. The end effector of the robotic arm is equipped with an actuator, a vision camera and a laser profilometer. The compensation model construction module is used to control the robot arm of the equipment to perform multiple picking and placing operations. Based on the patch deformation data collected by the laser profilometer, a model is established to correspond to the patch deformation characteristics and pose compensation amount under different main gripping conditions, which serves as the deformation compensation model. The drift prediction module establishes a drift prediction model based on the motor temperature, motion mileage, and reference position feedback monitored by the robot arm during no-load operation, describing the relationship between the drift amount of the robot arm end effector and the equipment state and time. The deformation pre-compensation module is used to obtain the deformation characteristics of the patch adsorbed on the end effector after the robot arm of the device performs the patch picking operation, and to process the patch deformation characteristics using the corresponding deformation compensation model according to the main gripping conditions of the patch picking operation to obtain the pose pre-compensation amount. The deformation compensation correction module is used to call the drift prediction model according to the current device status, obtain the drift pre-compensation amount, fuse the pose pre-compensation amount and the drift pre-compensation amount, generate the final compensation control command, and drive the robot to complete the compensation movement according to the command.

[0006] Furthermore, the specific logic for calibrating the spatial relationship between the visual camera, the laser profilometer, and the coordinate system of the robotic arm is as follows: control the movement of the robotic arm so that the visual camera and the laser profilometer at its end are aligned sequentially with a composite calibration target fixed on the worktable, the surface of which has a precise array of circular centers with known world coordinates; The specific steps include: controlling a vision camera to capture target images; obtaining the camera's intrinsic parameter matrix and initial camera extrinsic parameters relative to the target coordinate system by solving a vision camera calibration algorithm; controlling a laser profilometer to scan the target surface to obtain its measurement point cloud; calculating the transformation relationship from the laser profilometer coordinate system to the target coordinate system using a least squares fitting algorithm; controlling the device's robotic arm to repeat the above process in at least three different postures; solving the fixed transformation relationship between the camera and the laser profilometer relative to the robotic arm's end flange using a hand-eye calibration algorithm; and removing the target after calibration. The intrinsic parameter matrix includes the focal length and principal point position of the visual camera, and the initial camera extrinsic parameters specifically include the spatial position and orientation of the camera relative to the target.

[0007] Furthermore, the specific method for controlling the robotic arm of the equipment to perform multiple pick-ups and placements is as follows: set multiple sets of gripping conditions, each set of gripping conditions having different vacuum pressure values ​​or robotic arm picking postures. The specific logic for setting gripping conditions is as follows: for any vacuum pressure value, set multiple sets of robotic arm picking postures under that vacuum pressure value, combine the vacuum pressure value with the robotic arm picking postures to obtain multiple sets of gripping conditions, and summarize the gripping conditions with the same vacuum pressure value into the same type of main gripping conditions; Under various gripping conditions, the actuator at the end of the control device's robotic arm performs the action of picking up the patch sample. After each pickup, the control device's robotic arm moves the patch above a preset placement reference surface, and controls a laser profilometer to perform a full-area scan of the lower surface of the suspended patch sample, obtaining a high-density deformation point cloud of the lower surface of the patch sample, and recording the current vacuum pressure value and the robotic arm's pickup posture; the robotic arm's pickup posture specifically includes the pickup position and the robotic arm's pickup angle; after recording the data, a placement operation is performed in the current pickup state; The specific steps for performing the placement operation include: controlling the robotic arm to move the patch sample down to a preset height from the placement reference surface and pausing; cutting off the vacuum pressure value of the actuator; and using a high-speed camera to simultaneously record an analysis video from the side showing the patch sample falling freely after detaching from the actuator and contacting the placement reference surface. From the analysis video, the maximum amplitude, oscillation frequency, and damping decay time constant of the patch sample before it reaches stability are determined, and the preset height and robotic arm placement posture during placement are recorded. The robotic arm placement posture specifically includes the placement position and the robotic arm placement angle.

[0008] Furthermore, for the high-density deformed point cloud on the lower surface of the patch sample under any grasping condition, a normal vector is set to point to a reference plane on the back of the patch sample. The directed distance from the patch sample to the reference plane is calculated to form a two-dimensional deformation field, where positive values ​​represent convexity and negative values ​​represent concavity. Based on the formed two-dimensional deformation field, the patch deformation characteristics of the lower surface of the patch sample under the grasping condition are calculated. The patch deformation characteristics specifically include overall flatness, overall skewness, and overall kurtosis. The overall flatness is specifically characterized by the root mean square error of the directed distance. Based on the patch deformation characteristics, the picking and filtering coefficient under the grasping condition is calculated. The grasping conditions are then filtered based on the picking and filtering coefficient. The specific method for calculating the picking and filtering coefficient is as follows: The first evaluation coefficient is the reciprocal of the sum of the overall flatness and the preset compensation constant under the combination of vacuum pressure value and robot arm picking posture. The reciprocal of the sum of the overall skewness and the preset compensation constant under different combinations of vacuum pressure values ​​and robotic arm picking postures is used as the second evaluation coefficient. The reciprocal of the sum of the overall kurtosis and the preset compensation constant under different combinations of vacuum pressure values ​​and robotic arm picking postures is used as the third evaluation coefficient. Calculate the product of the first evaluation coefficient, the second evaluation coefficient, and the third evaluation coefficient with the setting permission coefficient, and use the sum of the three products as the picking and filtering coefficient of the vacuum pressure value and the robotic arm picking posture combination. Based on the picking and filtering coefficients, a corresponding picking and filtering coefficient prediction model is trained. The picking and filtering coefficient prediction model is constructed based on a neural network model. The set grasping conditions are used as inputs, namely, the vacuum pressure value and the picking posture of the robotic arm are used as the corresponding input values. The picking and filtering coefficients corresponding to the grasping conditions are used as labels to train the picking and filtering coefficient prediction model. The result is a picking and filtering coefficient prediction model with the grasping conditions as input and the corresponding picking and filtering coefficients as output.

[0009] Furthermore, the crawling conditions are filtered based on the picking and filtering coefficient. The specific method is as follows: the crawling conditions with a picking and filtering coefficient not greater than the picking quality threshold are marked, and the crawling conditions that are not marked are excluded. For any marked grasping condition, its corresponding preset height and robot arm placement posture are used as optimization variables. The optimization objectives are to minimize the maximum amplitude, oscillation frequency, damping decay time constant, and the difference from the target placement position. Through an optimization algorithm, the optimal preset height and optimal robot arm placement posture corresponding to the grasping condition are determined. The adjustment amount required to adjust the preset height and robot arm placement posture corresponding to the grasping condition to the optimal preset height and optimal robot arm placement posture is calculated and used as the pose compensation amount corresponding to the grasping condition. The optimization algorithm specifically refers to a multi-objective genetic algorithm. The difference in target placement position specifically refers to the distance between the midpoint of the patch sample and the midpoint of the target placement position after the patch sample is placed.

[0010] Furthermore, the specific method for training the deformation compensation model is as follows: for any main gripping condition, a corresponding deformation compensation model is constructed, and the vacuum pressure value that is consistent with the main gripping condition and is marked is extracted from all gripping conditions as the target gripping condition. The patch deformation feature of the target gripping condition is used as input, and the corresponding pose compensation amount is used as the label to train the deformation compensation model, so as to obtain a deformation compensation model that corresponds one-to-one with each main gripping condition. The specific method for determining the pose pre-compensation amount is as follows: obtain the grasping conditions of the current picking patch operation, input the grasping conditions into the completed picking and filtering coefficient prediction model, output the picking and filtering coefficient prediction value of the grasping conditions, and determine whether the current grasping conditions are not greater than the picking quality threshold based on the picking and filtering coefficient prediction value. If so, optimize and determine the corresponding pose pre-compensation amount; otherwise, re-grab. The specific optimization method is as follows: determine the corresponding deformation compensation model based on the main grasping conditions corresponding to the current patch picking operation, extract the patch deformation features of the current grasping conditions and input them into the corresponding deformation compensation model, and the deformation compensation model outputs the corresponding pose compensation amount. If there is no master gripping condition corresponding to the current pick-up and patch operation, the similarity between the current vacuum pressure value and the vacuum pressure value corresponding to the master gripping condition is calculated. The similarity of the vacuum pressure values ​​is represented by the Euclidean distance between the vacuum pressure values. The master gripping condition with the maximum similarity is determined and used as the master gripping condition corresponding to the current pick-up and patch operation.

[0011] Furthermore, the logic underlying the drift prediction model is as follows: The control device's robotic arm operates continuously at its classic working speed for several hours under no-load conditions. During this continuous operation, the temperature of the robotic arm's servo motors is collected, specifically the real-time temperatures of the X, Y, and Z axis servo motors. Simultaneously, the cumulative motion position of each axis is determined based on the grating ruler readings. The cumulative motion mileage of the robotic arm is calculated based on the cumulative motion position of each axis, and the absolute position drift of each axis at corresponding moments is collected. The specific method for calculating the cumulative motion mileage of the robotic arm is as follows: During the operation of the robotic arm, several sampling times are set. At any sampling time, the grating ruler readings of the X, Y and Z axes of the robotic arm are obtained. The grating ruler readings of the X, Y and Z axes of the robotic arm corresponding to the sampling time are accumulated to obtain the cumulative movement distance of the robotic arm at that sampling time. Based on the cumulative motion mileage of the robotic arm, the temperature of the servo motors of each axis, and the absolute position drift of each axis, a drift prediction model is established. Specifically, the absolute position drift data of each axis is used as the dependent variable, and the cumulative motion mileage of the robotic arm and the temperature of the servo motors of each axis are used as independent variables. The drift prediction model expression is constructed through multiple linear regression analysis. The method for establishing the drift prediction models for the X and Y axes of the robotic arm is the same as that for the Z axis drift prediction model. Based on the drift prediction models for the X, Y, and Z axes of the robotic arm, the drift prediction values ​​for each axis of the robotic arm are determined.

[0012] Furthermore, the logic for fusing the pose pre-compensation amount and drift pre-compensation amount is as follows: based on the deformation compensation model, the compensation values ​​and directions of the preset height and the robot's placement posture are output; the preset height is compensated by the drift prediction value of the robot's Z-axis, specifically expressed as the sum of the drift prediction value of the robot's Z-axis and the preset height; the placement position of the robot in the placement posture is compensated by the drift prediction values ​​of the robot's X-axis and Y-axis, thereby generating the final compensation control command and driving the robot to complete the compensation movement according to the command.

[0013] This invention also provides a visual self-calibration method for an automated microneedle patching device, wherein the method is used to control the aforementioned visual self-calibration system for the automated microneedle patching device, and includes: The control device robotic arm measures a composite calibration target fixed in the workspace and completes the spatial relationship calibration of the coordinate system of the vision camera, laser profilometer and the device robotic arm. The end of the device robotic arm is equipped with an actuator, a vision camera and a laser profilometer. The control device's robotic arm performs multiple pick-up and placement operations. Based on the patch deformation data collected by the laser profilometer, a model is established to correspond to the patch deformation characteristics and pose compensation amount under different main gripping conditions, serving as a deformation compensation model. The motor temperature, mileage, and reference position feedback monitored during the no-load operation of the equipment robot are used to establish a drift prediction model that describes the relationship between the drift amount of the robot's end effector and the equipment status and time. After the robotic arm of the device performs the pick-up operation, it obtains the deformation characteristics of the patch adsorbed on the end effector, and processes the patch deformation characteristics using the corresponding deformation compensation model according to the main gripping conditions of the pick-up operation to obtain the pose pre-compensation amount. The drift prediction model is invoked based on the current device status to obtain the drift pre-compensation amount. The pose pre-compensation amount and the drift pre-compensation amount are fused to generate the final compensation control command, and the robot arm is driven to complete the compensation movement according to the command.

[0014] Compared with the prior art, the beneficial effects of the present invention are: This system calibrates the spatial relationship between the vision camera, laser profilometer, and robot arm coordinate system by measuring the composite calibration target, ensuring high-precision spatial registration between key components of the system. This calibration method integrates data from multiple sensors, improving the overall calibration accuracy and reliability of the equipment, and can more comprehensively cover the spatial relationship between the robot arm actuator and the patch, ensuring high precision and consistency in the patching operation. Secondly, this solution uses a robotic arm to perform multiple pick-and-place training operations, collecting patch deformation data detected by a laser profilometer. Based on this data, a model is established to correlate patch deformation characteristics with pose compensation amounts under different gripping conditions. Unlike the simplistic handling of patch deformation in existing technologies, this solution can generate appropriate compensation amounts based on the actual deformation characteristics of the patch, further improving the placement accuracy. Especially when facing complex operating conditions, this deformation compensation model demonstrates great adaptability and robustness, significantly reducing the impact of patch deformation on the final attachment effect. This system establishes a drift prediction model that describes the relationship between the drift amount of the robot's end effector and the equipment's state and time by monitoring data from the robot's unloaded operation, including motor temperature, mileage, and reference position feedback. This model solves the accuracy drift problem that may occur when the robot operates for a long time, providing a reliable technical guarantee for the long-term stable operation of the equipment. By monitoring the equipment status in real time and predicting the drift amount, the system significantly improves the real-time performance and accuracy of drift compensation, thereby ensuring the high precision of the robot during dynamic operation. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall system structure of the present invention; Figure 2 The overall flatness-overall kurtosis fitting curve; Figure 3 The curve is fitted to the overall skewness and the selection coefficient. Figure 4 This is a schematic diagram of the overall method flow of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0017] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0018] Example: Please see Figures 1-3 The present invention provides a technical solution: A visual self-calibration system for an automated microneedle patch application device, specifically comprising: The data acquisition module is used to control the robot arm of the equipment to measure the composite calibration target fixed in the workspace and to complete the spatial relationship calibration of the coordinate system of the vision camera, the laser profilometer and the robot arm. The end effector of the robot arm is equipped with an actuator, a vision camera and a laser profilometer.

[0019] The specific logic for calibrating the spatial relationship between the coordinate system of the vision camera, the laser profilometer, and the robot arm is as follows: control the movement of the robot arm so that the vision camera and the laser profilometer at its end are aligned sequentially with a composite calibration target fixed on the worktable. The surface of the target has a precise array of circular centers with known world coordinates. The specific steps include: controlling a vision camera to capture target images; obtaining the camera's intrinsic parameter matrix and initial camera extrinsic parameters relative to the target coordinate system by solving a vision camera calibration algorithm; controlling a laser profilometer to scan the target surface to obtain its measurement point cloud; calculating the transformation relationship from the laser profilometer coordinate system to the target coordinate system using a least squares fitting algorithm; controlling the device's robotic arm to repeat the above process in at least three different postures; solving the fixed transformation relationship between the camera and the laser profilometer relative to the robotic arm's end flange using a hand-eye calibration algorithm; and removing the target after calibration. Specifically, the process involves controlling the robotic arm of the device to align its end-effector's vision camera with a specific location on the target. The vision camera captures an image of the target, ensuring that the image includes multiple centers of the target. The robotic arm is then controlled to repeat the vision camera and laser profilometer measurement process in at least three different postures. The external parameters of the camera obtained from each calibration and the transformation relationship from the laser profilometer to the target are integrated. A hand-eye calibration algorithm, such as the Tsai calibration method, is used to solve for the fixed transformation relationship between the camera and the laser profilometer relative to the end flange of the robotic arm. After calibration, the target is removed, ensuring that the working area of ​​the device is clean. The intrinsic parameter matrix includes the focal length and principal point position of the visual camera, and the initial camera extrinsic parameters specifically include the spatial position and orientation of the camera relative to the target.

[0020] The compensation model construction module is used to control the robot arm of the equipment to perform multiple picking and placing operations. Based on the patch deformation data collected by the laser profilometer, a model is established to establish the correspondence between patch deformation characteristics and pose compensation amount under different main gripping conditions, which serves as the deformation compensation model.

[0021] The specific method for controlling the robotic arm of the equipment to perform multiple pick-ups and placements is as follows: set multiple sets of gripping conditions, each set of gripping conditions having different vacuum pressure values ​​or robotic arm picking postures. The specific logic for setting gripping conditions is as follows: for any vacuum pressure value, set multiple sets of robotic arm picking postures under that vacuum pressure value, combine the vacuum pressure value with the robotic arm picking postures to obtain multiple sets of gripping conditions, and summarize the gripping conditions with the same vacuum pressure value into the same type of main gripping conditions. First, analyze the object to be grasped, including its shape, weight, material, surface roughness, etc. Determine the range of vacuum pressure that the vacuum suction cup can provide based on its shape, weight, material, surface roughness, etc. Ensure that the selected vacuum pressure value can effectively adsorb the microneedle patch without causing damage or deformation. It is generally set between 0.1 kPa and 10 kPa. For each vacuum pressure, multiple pickup postures are defined to ensure that the robot can grasp the microneedle patch from different directions. Specific pickup posture examples include: vertical gripping, where the robot grasps the microneedle patch vertically from directly above; tilted gripping, where the robot grasps the microneedle patch at a specific angle, such as 15 degrees, suitable for situations where the patch is placed at an angle or in a specific direction; and lateral gripping, where the robot contacts the microneedle patch from the side, suitable for scenarios requiring grasping from confined spaces. The combinations of different vacuum pressure values ​​with all available pickup postures should be recorded in the robot control system and prepared for experimental verification.

[0022] Under various gripping conditions, the actuator at the end of the control device's robotic arm performs the action of picking up the patch sample. After each pickup, the control device's robotic arm moves the patch above a preset placement reference surface, and controls a laser profilometer to perform a full-area scan of the lower surface of the suspended patch sample, obtaining a high-density deformation point cloud of the lower surface of the patch sample, and recording the current vacuum pressure value and the robotic arm's pickup posture; the robotic arm's pickup posture specifically includes the pickup position and the robotic arm's pickup angle; after recording the data, a placement operation is performed in the current pickup state; The specific steps for performing the placement operation include: controlling the robotic arm to move the patch sample down to a preset height from the placement reference surface and pausing; cutting off the vacuum pressure value of the actuator; and using a high-speed camera to simultaneously record an analysis video from the side showing the patch sample falling freely after detaching from the actuator and contacting the placement reference surface. From the analysis video, the maximum amplitude, oscillation frequency, and damping decay time constant of the patch sample before it reaches stability are determined, and the preset height and robotic arm placement posture during placement are recorded. The robotic arm placement posture specifically includes the placement position and the robotic arm placement angle.

[0023] The specific method for obtaining the maximum amplitude, oscillation frequency and damping decay time constant is as follows: Based on the analysis video, analyze the motion trajectory of the patch sample, and extract vibration parameters from the trajectory data, including the maximum amplitude, oscillation frequency and damping decay time constant; Ensure the high-speed camera is set to an appropriate frame rate, specifically 1000fps or higher. The specific setting can be adjusted based on the falling speed and motion characteristics to clearly capture the motion of the patch sample. Use video analysis software, such as MATLAB, Python's OpenCV, or other video processing tools, to process the data. The robotic arm is controlled to move the patch sample to a preset height, then pauses, and the vacuum suction of the actuator is cut off, allowing the patch sample to fall freely. The entire process of the patch sample falling freely is filmed from the side to ensure that the movement trajectory of the patch is clearly visible in the video. The recorded video is imported into video analysis software, and image processing techniques such as threshold segmentation and edge detection are used to identify the outline of the patch sample, extract its movement trajectory, and apply tracking algorithms such as optical flow or background subtraction to track the position of the patch sample in the video. The center coordinates of the patch sample in each frame are recorded, including the timestamp. From the motion data, the maximum deviation of the patch sample in the vertical direction (z-axis) of its placement position, i.e., the maximum amplitude, is calculated. By performing Fourier transform on the time series data, the main frequency components are identified, the oscillation frequency is determined, and the frequency corresponding to the highest peak is identified as the oscillation frequency. The damping decay time constant is obtained through the classical vibration decay model. The vibration of the patch is described by the damped vibration model. Sufficient data points are selected, and the above model is fitted using the nonlinear least squares method to extract the damping factor. The damping decay time constant is expressed as the reciprocal of the damping factor. The maximum amplitude, oscillation frequency and damping decay time constant obtained from the analysis were recorded, along with the preset height and robot arm placement posture during placement. During each round of picking and placing operations, only the vacuum pressure value of the actuator at the end of the robot arm and the picking posture of the robot arm change, which means that environmental factors remain at the same level during the test.

[0024] For any vacuum pressure value under any actuator, determine the high-density deformation point cloud on the lower surface of the patch sample corresponding to different robot arm picking postures under that vacuum pressure value. The picking angle specifically refers to the angle between the central axis of the robot arm corresponding to the actuator and the placement reference surface. For a high-density deformed point cloud on the lower surface of a patch sample under any gripping condition, a normal vector is set to point to a reference plane on the back of the patch sample. The directed distance from the patch sample to the reference plane is calculated to form a two-dimensional deformation field, where positive values ​​represent convexity and negative values ​​represent depression. Based on the formed two-dimensional deformation field, the patch deformation characteristics of the lower surface of the patch sample under the gripping condition are calculated. The patch deformation characteristics specifically include overall flatness, overall skewness, and overall kurtosis. The overall flatness is specifically characterized by the root mean square error of the directed distance. Specific methods for determining overall skewness and overall kurtosis include: obtaining high-density point cloud data of the lower surface of the microneedle patch sample through 3D scanning, laser ranging, or other methods.

[0025] For experiments with different robotic arm picking postures under the same vacuum pressure value, the point cloud acquired each time should be carried out under the same conditions to ensure data consistency. The reference plane on the back of the patch sample should be determined by the normal vector pointing to it, specifically defined by the geometric center of the sample or the design reference plane. For each point cloud point, calculate its directed distance to the reference plane. Organize the distance values ​​from the point cloud to the reference plane into a two-dimensional array for subsequent calculations. Each point corresponds to its coordinates and directed distance on the plane. The formula used for the specific calculation of the overall skewness is as follows: In the formula, For overall skewness, Let be the directed distance from the i-th point in the point cloud to the reference plane. The average directed distance from the point cloud points to the reference plane. Let N be the standard deviation of the directed distance, and N be the total number of points in the point cloud. The specific formula used to calculate the overall kurtosis is as follows: In the formula, Indicates the overall kurtosis.

[0026] The picking and screening coefficient is calculated based on the patch deformation characteristics. This coefficient is then used to screen combinations of vacuum pressure values ​​and robotic arm picking postures. The specific method for calculating the picking and screening coefficient is as follows: The first evaluation coefficient is the reciprocal of the sum of the overall flatness and the preset compensation constant under the combination of vacuum pressure value and robot arm picking posture. The reciprocal of the sum of the overall skewness and the preset compensation constant under different combinations of vacuum pressure values ​​and robotic arm picking postures is used as the second evaluation coefficient. The reciprocal of the sum of the overall kurtosis and the preset compensation constant under different combinations of vacuum pressure values ​​and robotic arm picking postures is used as the third evaluation coefficient. Calculate the products of the first evaluation coefficient, the second evaluation coefficient, and the third evaluation coefficient, and the setting permission coefficient respectively. The sum of these three products is used as the picking selection coefficient for the combination of vacuum pressure value and robotic arm picking posture. The specific formula used is as follows: In the formula, Let be the picking and filtering coefficient corresponding to the u-th robotic arm picking posture under the s-th vacuum pressure value. Let denot 's' be the overall flatness of the patch sample under the 'u'th robotic arm picking posture at the 's'th vacuum pressure value. Let be the overall skewness corresponding to the patch sample under the s-th vacuum pressure value and the u-th robotic arm picking posture. Let be the overall kurtosis of the patch sample under the s-th vacuum pressure value and the u-th robotic arm picking posture. , and These are the weighting coefficients. , This is the compensation constant; That is, the first evaluation coefficient. This is the second evaluation coefficient. This is the third evaluation coefficient; It should be noted that the selection and screening coefficients are... It is used for a comprehensive evaluation of patch deformation characteristics based on overall flatness, overall skewness and overall kurtosis. The higher the value, the better the combination is in the picking process. Overall flatness is characterized by the root mean square error of the directed distance, reflecting the smoothness of the patch's underside surface. A smaller overall flatness value indicates a smoother patch surface, which facilitates stable pickup. Therefore, overall flatness is closely related to the pickup selection coefficient. Global skewness is used to characterize the symmetry of deformation, reflecting the uniformity of the distribution of protrusions or depressions on the patch surface. A smaller skewness value indicates a more uniform deformation distribution and a more symmetrical shape on the patch surface. An increase in skewness value suggests greater asymmetry on the patch surface, leading to instability during gripping and thus reducing the pick-and-select coefficient. ; Overall kurtosis measures the sharpness of surface deformation of a patch, reflecting the concentration of surface undulations. A smaller value indicates a smoother surface deformation and less fluctuation, which helps with stable gripping. When the overall kurtosis value increases, sharper protrusions appear on the surface, which may lead to unstable gripping, negatively impacting the gripping process and thus reducing the pick-and-select coefficient. .

[0027] Overall flatness is a core indicator reflecting the smoothness of the patch surface. The flatness of the patch surface directly affects the pickup success rate; flatness is usually a key factor in whether patch pickup is successful, directly affecting the adhesion and stability. Overall kurtosis reflects the sharpness of the patch surface; higher kurtosis indicates more concentrated and sharper surface deformation. The impact of overall kurtosis is localized and usually does not directly cause pickup failure; it primarily affects the local contact quality of the patch surface. Therefore, overall flatness has the highest weight, and overall kurtosis has the lowest weight. Hence, it is set... .

[0028] Based on the picking and filtering coefficients, a corresponding picking and filtering coefficient prediction model is trained. The picking and filtering coefficient prediction model is constructed based on a neural network model. The set grasping conditions are used as inputs, namely, the vacuum pressure value and the picking posture of the robotic arm are used as the corresponding input values. The picking and filtering coefficients corresponding to the grasping conditions are used as labels to train the picking and filtering coefficient prediction model. The result is a picking and filtering coefficient prediction model with the grasping conditions as input and the corresponding picking and filtering coefficients as output.

[0029] The picking and filtering coefficient prediction model is specifically based on a convolutional neural network, which consists of an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer. The activation function in the convolutional layer is... For the fully connected layer, the number of neurons is set to 32, the initial neural network learning rate is set to 0.001, the number of training epochs is 100, the mean squared error (MSE) is used as the loss function, and an appropriate optimizer, such as the Adam optimizer, is selected to optimize the initial neural network learning rate; the number of training epochs is set to 100; the performance of the trained model is evaluated on the test set, and the MSE and other relevant metrics between the predicted results and the actual values ​​are calculated.

[0030] The vacuum pressure value and the robot arm's picking posture are screened based on the picking screening coefficient. The specific method is as follows: Under each vacuum pressure value, the picking screening coefficient corresponding to different robot arm picking postures is compared with a preset picking quality threshold. Vacuum pressure values ​​and robot arm picking posture combinations that are not greater than the picking quality threshold are marked, and unmarked vacuum pressure values ​​and robot arm picking posture combinations are excluded. The picking quality threshold is set based on the degree of damage to the patch under the picking posture combination and expert experience.

[0031] Data for placement training based on the combination of labeled vacuum pressure value and robot arm picking posture is extracted, including preset height, robot arm placement posture, and the maximum amplitude, oscillation frequency and damping decay time constant of its patch sample before it reaches stability. The crawling conditions are filtered based on the picking and filtering coefficient. The specific method is as follows: the crawling conditions with a picking and filtering coefficient not greater than the picking quality threshold are marked, and the crawling conditions that are not marked are excluded. For any marked grasping condition, its corresponding preset height and robot arm placement posture are used as optimization variables. The optimization objectives are to minimize the maximum amplitude, oscillation frequency, damping decay time constant, and the difference from the target placement position. Through an optimization algorithm, the optimal preset height and optimal robot arm placement posture corresponding to the grasping condition are determined. The adjustment amount required to adjust the preset height and robot arm placement posture corresponding to the grasping condition to the optimal preset height and optimal robot arm placement posture is calculated and used as the pose compensation amount corresponding to the grasping condition. The optimization algorithm specifically refers to a multi-objective genetic algorithm. The difference in target placement position specifically refers to the distance between the midpoint of the patch sample and the midpoint of the target placement position after the patch sample is placed.

[0032] The fitness function is set with the optimization objectives of minimizing maximum amplitude, oscillation frequency, damping decay time constant, and the difference from the target placement position, as specifically expressed as: In the formula, For the fitness function value, For maximum amplitude, The oscillation frequency is... The damping decay time constant is The difference in the target placement location refers to the difference between the center of the ideal placement location and the actual placement location. , , and Here are the weighting coefficients, where ; It should be noted that there are differences in the target placement location. This indicates the deviation between the midpoint of the actual placement position of the patch and the midpoint of the ideal placement position. The smaller the deviation, the higher the placement accuracy. Therefore, the difference in the target placement position... It is inversely proportional to the fitness function value; This indicates the maximum vibration amplitude generated by the patch during placement. The larger the amplitude, the worse the stability. This indicates the frequency of the surface mount device's vibration. A higher vibration frequency may cause instability in the device's placement. This represents the time required for patch vibration to decay to a steady state. The shorter the time, the faster and more stable the placement. Therefore, it is inversely proportional to the fitness function value.

[0033] The accuracy of the placement position directly affects whether the patch can be accurately placed at the target location, and is the most important factor in the entire pickup and placement process. The oscillation frequency reflects the intensity of the patch's vibration during placement. A high frequency may cause the patch to be difficult to stabilize quickly, affecting the accuracy and efficiency of the placement process. The oscillation frequency has a strong impact on the stability of patch placement, but its importance is lower than positional deviation. The maximum amplitude represents the amplitude of patch vibration and is another important factor affecting placement stability. A high amplitude may cause patch displacement or poor contact. Although amplitude affects placement performance, its importance is lower than positional accuracy and oscillation frequency. The damping decay time constant represents the time required for patch vibration to decay to a stable state. A large time constant may affect the placement efficiency of the patch, but compared with other indicators, the decay time has a relatively small impact on placement accuracy and stability, and may have a greater impact on efficiency. Therefore, setting a damping decay time constant is important. .

[0034] The specific method for training the deformation compensation model is as follows: For any main gripping condition, a corresponding deformation compensation model is constructed, and the vacuum pressure value that is consistent with the main gripping condition and is marked is extracted from all gripping conditions as the target gripping condition. The patch deformation feature of the target gripping condition is used as input, and the corresponding pose compensation amount is used as the label to train the deformation compensation model, so as to obtain a deformation compensation model that corresponds one-to-one with each main gripping condition. The deformation compensation model is specifically constructed using a deep learning network, specifically a fully connected neural network (FNN). The specific training process includes: taking the patch deformation features of the target grasping conditions as input, the corresponding pose compensation amount as label, setting hyperparameters such as learning rate, batch size, and number of network layers to optimize model performance, using early stopping to avoid overfitting, using the mean squared error function to evaluate the model's performance, outputting evaluation indicators to assess whether there is overfitting or underfitting, and applying the trained model to the real-time grasping process to perform pose compensation based on the real-time acquired patch deformation features.

[0035] The drift prediction module establishes a drift prediction model based on the motor temperature, mileage, and reference position feedback monitored by the robot arm during no-load operation, describing the relationship between the drift amount of the robot arm end effector and the equipment state and time.

[0036] The logic underlying the drift prediction model is as follows: The control device's robotic arm operates continuously for several hours at a classic working speed under no-load conditions. This classic working speed is determined based on the robotic arm's design parameters and technical manual, including a maximum allowable speed and a recommended speed. The recommended speed is used as the classic working speed. During continuous operation, the temperature of the robotic arm's servo motors is collected, specifically the real-time temperatures of the X, Y, and Z axis servo motors. Simultaneously, the cumulative motion position of each axis is determined based on the grating ruler readings. The cumulative motion mileage of the robotic arm is calculated based on the cumulative motion position of each axis, and the absolute position drift of each axis at corresponding moments is collected. The specific method for calculating the cumulative motion mileage of the robotic arm is as follows: During the operation of the robotic arm, several sampling times are set. At any given sampling time, the grating ruler readings of the robotic arm's X, Y, and Z axes are acquired. The corresponding grating ruler readings of the robotic arm's X, Y, and Z axes at that sampling time are accumulated to obtain the cumulative movement distance of the robotic arm at that sampling time. The specific formula used is as follows: In the formula, This represents the cumulative motion distance of the robot arm at the t-th sampling time. The X-axis grating ruler reading of the equipment's robotic arm at the t-th sampling time. The reading of the Y-axis grating ruler of the equipment's robotic arm at the t-th sampling time. The reading of the Z-axis grating ruler of the device's robot arm at the t-th sampling time is denoted as t, where t is the index of the sampling time. , This represents the total number of sampling times; It should be noted that, Indicates the first The cumulative distance traveled by the robotic arm at each sampling time. This is a measure of the total amount of motion of the robotic arm under no-load conditions; , and They were respectively in the second The changes in the grating ruler readings of the X, Y, and Z axes at each sampling time represent the displacement of each axis of the robot at that time. The data is obtained in real time through grating ruler monitoring. This formula calculates the cumulative motion mileage by summing the absolute values ​​of displacement at each sampling moment. By summing the displacement changes of each axis, the total amount of motion of the robot arm during the entire operation can be accurately reflected. Using absolute values ​​ensures that displacement is correctly counted in the cumulative mileage regardless of the direction of motion, preventing positive and negative displacements from canceling each other out.

[0037] When a motor is operating, temperature changes are closely related to its performance and drift behavior. Excessive temperature may lead to a decrease in motor performance or material expansion, thereby affecting the positioning accuracy and stability of the robot. By comprehensively considering the cumulative motion of the three axes, the motion state of the robot can be fully evaluated. The motion of each axis may produce different drifts due to factors such as thermal expansion and friction. Therefore, calculating the combined motion mileage of the three axes can better reflect the state of the entire system.

[0038] Based on the cumulative motion mileage of the robotic arm, the temperature of the servo motors of each axis, and the absolute position drift of each axis, a drift prediction model is established. Specifically, the absolute position drift data of each axis is used as the dependent variable, and the cumulative motion mileage of the robotic arm and the temperature of the servo motors of each axis are used as independent variables. Through multiple linear regression analysis, the drift prediction model expression is constructed as follows: In the formula, For the first The predicted drift value of the robot arm's Z-axis at each sampling time. , and The regression coefficient for the temperature rise of the servo motor. , and The temperature changes of the servo motors on the X, Y, and Z axes are respectively. The regression coefficient for cumulative exercise mileage. This is a constant term, specifically representing the initial drift offset of the robot's Z-axis. The method for establishing the drift prediction models for the X and Y axes of the robotic arm is the same as that for the Z axis drift prediction model. Based on the drift prediction models for the X, Y, and Z axes of the robotic arm, the drift prediction values ​​for each axis of the robotic arm are determined.

[0039] It should be noted that temperature change is an important independent variable for drift prediction. During operation, the servo motor of the robot arm will cause thermal expansion or thermal drift of the mechanical structure due to temperature rise. Cumulative motion mileage represents the total distance the robot arm travels from the start of operation to the t-th sampling time, and is used to reflect the impact of the robot arm's cumulative motion on drift. The method employs multiple linear regression, where temperature change and drift typically exhibit an approximately linear relationship: the thermal expansion of mechanical structures generally satisfies the linear thermal expansion formula; the relationship between cumulative mileage and drift can be described by linear approximation, especially over short periods or under specific conditions; the simplicity and interpretability of the linear model also facilitate engineering applications.

[0040] The deformation pre-compensation module is used to obtain the deformation characteristics of the patch adsorbed on the end effector after the robot arm of the device performs the patch picking operation, and to process the patch deformation characteristics using the corresponding deformation compensation model according to the main gripping conditions of the patch picking operation to obtain the pose pre-compensation amount.

[0041] The specific method for determining the pose pre-compensation amount is as follows: obtain the grasping conditions of the current picking patch operation, input the grasping conditions into the completed picking and filtering coefficient prediction model, output the picking and filtering coefficient prediction value of the grasping conditions, and determine whether the current grasping conditions are not greater than the picking quality threshold based on the picking and filtering coefficient prediction value. If so, optimize and determine the corresponding pose pre-compensation amount; otherwise, re-grab. The specific optimization method is as follows: determine the corresponding deformation compensation model based on the main grasping conditions corresponding to the current patch picking operation, extract the patch deformation features of the current grasping conditions and input them into the corresponding deformation compensation model, and the deformation compensation model outputs the corresponding pose compensation amount. If there is no master gripping condition corresponding to the current pick-and-place operation, the similarity between the current vacuum pressure value and the vacuum pressure value corresponding to the master gripping condition is calculated. The similarity of the vacuum pressure values ​​is represented by the Euclidean distance between the vacuum pressure values. The master gripping condition with the maximum similarity is determined and used as the master gripping condition corresponding to the current pick-and-place operation. The current pick-and-place operation is then stored to update the compensation relationship of the master gripping condition.

[0042] The deformation compensation correction module is used to call the drift prediction model according to the current device status, obtain the drift pre-compensation amount, fuse the pose pre-compensation amount and the drift pre-compensation amount, generate the final compensation control command, and drive the robot to complete the compensation movement according to the command.

[0043] The specific logic for fusing the pose pre-compensation amount and drift pre-compensation amount is as follows: Based on the deformation compensation model, the compensation values ​​and directions of the preset height and the robot's placement posture are output. The preset height is compensated by the drift prediction value of the robot's Z-axis, specifically expressed as the sum of the drift prediction value of the robot's Z-axis and the preset height. The placement position of the robot in the placement posture is compensated by the drift prediction values ​​of the robot's X-axis and Y-axis, thereby generating the final compensation control command and driving the robot to complete the compensation movement according to the command.

[0044] Please see Figure 4 The present invention also provides a visual self-calibration method for an automated microneedle patching device, wherein the method is used to control the aforementioned visual self-calibration system for the automated microneedle patching device, comprising: Step 1: Control the robotic arm of the equipment to measure the composite calibration target fixed in the workspace, and complete the spatial relationship calibration of the coordinate system of the vision camera, the laser profilometer and the robotic arm. The end of the robotic arm is equipped with an actuator, a vision camera and a laser profilometer. Step 2: Control the robotic arm of the equipment to perform multiple pick-up and placement operations. Based on the patch deformation data collected by the laser profilometer, establish a model of the correspondence between patch deformation characteristics and pose compensation amount under different main gripping conditions, as a deformation compensation model. Step 3: Monitor the motor temperature, mileage, and reference position feedback of the robot arm during no-load operation, and establish a drift prediction model that describes the relationship between the drift amount of the robot arm end effector and the equipment status and time. Step 4: After the robotic arm of the device performs the pick-up operation, the deformation characteristics of the patch adsorbed on the end effector are obtained, and the patch deformation characteristics are processed by the corresponding deformation compensation model according to the main gripping conditions of the pick-up operation to obtain the pose pre-compensation amount. Step 5: Based on the current device status, call the drift prediction model to obtain the drift pre-compensation amount, fuse the pose pre-compensation amount and the drift pre-compensation amount to generate the final compensation control command, and drive the robot to complete the compensation movement according to the command.

[0045] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0046] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0047] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0048] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A visual self-calibration system for an automated microneedle patch application device, characterized in that, Specifically, it includes: The data acquisition module is used to control the robotic arm of the equipment to measure the composite calibration target fixed in the workspace and to complete the spatial relationship calibration of the coordinate system of the vision camera, the laser profilometer and the robotic arm. The end effector of the robotic arm is equipped with an actuator, a vision camera and a laser profilometer. The compensation model construction module is used to control the robot arm of the equipment to perform multiple picking and placing operations. Based on the patch deformation data collected by the laser profilometer, a model is established to correspond to the patch deformation characteristics and pose compensation amount under different main gripping conditions, which serves as the deformation compensation model. The drift prediction module establishes a drift prediction model based on the motor temperature, motion mileage, and reference position feedback monitored by the robot arm during no-load operation, describing the relationship between the drift amount of the robot arm end effector and the equipment state and time. The deformation pre-compensation module is used to obtain the deformation characteristics of the patch adsorbed on the end effector after the robot arm of the device performs the patch picking operation, and to process the patch deformation characteristics using the corresponding deformation compensation model according to the main gripping conditions of the patch picking operation to obtain the pose pre-compensation amount. The deformation compensation correction module is used to call the drift prediction model according to the current device status, obtain the drift pre-compensation amount, fuse the pose pre-compensation amount and the drift pre-compensation amount, generate the final compensation control command, and drive the robot to complete the compensation movement according to the command.

2. The visual self-calibration system for an automated microneedle patching device according to claim 1, characterized in that: The specific logic for calibrating the spatial relationship between the coordinate system of the vision camera, the laser profilometer, and the robot arm is as follows: control the movement of the robot arm so that the vision camera and the laser profilometer at its end are aligned sequentially with a composite calibration target fixed on the worktable. The surface of the target has a precise array of circular centers with known world coordinates. The specific steps include: controlling a vision camera to capture target images; obtaining the camera's intrinsic parameter matrix and initial camera extrinsic parameters relative to the target coordinate system by solving a vision camera calibration algorithm; controlling a laser profilometer to scan the target surface to obtain its measurement point cloud; calculating the transformation relationship from the laser profilometer coordinate system to the target coordinate system using a least squares fitting algorithm; controlling the device's robotic arm to repeat the above process in at least three different postures; solving the fixed transformation relationship between the camera and the laser profilometer relative to the robotic arm's end flange using a hand-eye calibration algorithm; and removing the target after calibration. The intrinsic parameter matrix includes the focal length and principal point position of the visual camera, and the initial camera extrinsic parameters specifically include the spatial position and orientation of the camera relative to the target.

3. The visual self-calibration system for an automated microneedle patching device according to claim 2, characterized in that: The specific method for controlling the robotic arm of the equipment to perform multiple pick-ups and placements is as follows: set multiple sets of gripping conditions, each set of gripping conditions having different vacuum pressure values ​​or robotic arm picking postures. The specific logic for setting gripping conditions is as follows: for any vacuum pressure value, set multiple sets of robotic arm picking postures under that vacuum pressure value, combine the vacuum pressure value with the robotic arm picking postures to obtain multiple sets of gripping conditions, and summarize the gripping conditions with the same vacuum pressure value into the same type of main gripping conditions. Under various gripping conditions, the actuator at the end of the control device's robotic arm performs the action of picking up the patch sample. After each pickup, the control device's robotic arm moves the patch above a preset placement reference surface, and controls a laser profilometer to perform a full-area scan of the lower surface of the suspended patch sample, obtaining a high-density deformation point cloud of the lower surface of the patch sample, and recording the current vacuum pressure value and the robotic arm's pickup posture; the robotic arm's pickup posture specifically includes the pickup position and the robotic arm's pickup angle; after recording the data, a placement operation is performed in the current pickup state; The specific steps for performing the placement operation include: controlling the robotic arm to move the patch sample down to a preset height from the placement reference surface and pausing; cutting off the vacuum pressure value of the actuator; and using a high-speed camera to simultaneously record an analysis video from the side showing the patch sample falling freely after detaching from the actuator and contacting the placement reference surface. From the analysis video, the maximum amplitude, oscillation frequency, and damping decay time constant of the patch sample before it reaches stability are determined, and the preset height and robotic arm placement posture during placement are recorded. The robotic arm placement posture specifically includes the placement position and the robotic arm placement angle.

4. The visual self-calibration system for an automated microneedle patching device according to claim 3, characterized in that: For a high-density deformed point cloud on the lower surface of a patch sample under any gripping condition, a normal vector is set to point towards a reference plane on the back of the patch sample. The directed distance from the patch sample to the reference plane is calculated, forming a two-dimensional deformation field, where positive values ​​represent convexity and negative values ​​represent concavity. Based on the formed two-dimensional deformation field, the patch deformation characteristics of the lower surface of the patch sample under the gripping condition are calculated. The patch deformation characteristics specifically include overall flatness, overall skewness, and overall kurtosis. The overall flatness is specifically characterized by the root mean square error of the directed distance. Based on the patch deformation characteristics, a picking and filtering coefficient is calculated under the gripping condition. The gripping condition is then filtered based on the picking and filtering coefficient. The specific method for calculating the picking and filtering coefficient is as follows: The first evaluation coefficient is the reciprocal of the sum of the overall flatness and the preset compensation constant under the combination of vacuum pressure value and robot arm picking posture. The reciprocal of the sum of the overall skewness and the preset compensation constant under different combinations of vacuum pressure values ​​and robotic arm picking postures is used as the second evaluation coefficient. The reciprocal of the sum of the overall kurtosis and the preset compensation constant under different combinations of vacuum pressure values ​​and robotic arm picking postures is used as the third evaluation coefficient. Calculate the products of the first evaluation coefficient, the second evaluation coefficient, and the third evaluation coefficient with the setting permission coefficient, and use the sum of the three products as the picking and filtering coefficient of the vacuum pressure value and the robotic arm picking posture combination. Based on the picking and filtering coefficients, a corresponding picking and filtering coefficient prediction model is trained. The picking and filtering coefficient prediction model is constructed based on a neural network model. The set grasping conditions are used as inputs, namely, the vacuum pressure value and the picking posture of the robotic arm are used as the corresponding input values. The picking and filtering coefficients corresponding to the grasping conditions are used as labels to train the picking and filtering coefficient prediction model. The result is a picking and filtering coefficient prediction model with the grasping conditions as input and the corresponding picking and filtering coefficients as output.

5. The visual self-calibration system for an automated microneedle patching device according to claim 4, characterized in that: The crawling conditions are filtered based on the picking and filtering coefficient. The specific method is as follows: the crawling conditions with a picking and filtering coefficient not greater than the picking quality threshold are marked, and the crawling conditions that are not marked are excluded. For any marked grasping condition, its corresponding preset height and robot arm placement posture are used as optimization variables. The optimization objectives are to minimize the maximum amplitude, oscillation frequency, damping decay time constant, and the difference from the target placement position. Through an optimization algorithm, the optimal preset height and optimal robot arm placement posture corresponding to the grasping condition are determined. The adjustment amount required to adjust the preset height and robot arm placement posture corresponding to the grasping condition to the optimal preset height and optimal robot arm placement posture is calculated and used as the pose compensation amount corresponding to the grasping condition. The optimization algorithm specifically refers to a multi-objective genetic algorithm. The difference in target placement position specifically refers to the distance between the midpoint of the patch sample and the midpoint of the target placement position after the patch sample is placed.

6. The visual self-calibration system for an automated microneedle patching device according to claim 5, characterized in that: The specific method for training the deformation compensation model is as follows: For any main gripping condition, a corresponding deformation compensation model is constructed, and the vacuum pressure value that is consistent with the main gripping condition and is marked is extracted from all gripping conditions as the target gripping condition. The patch deformation features of the target gripping condition are used as input, and the corresponding pose compensation amount is used as the label to train the deformation compensation model, so as to obtain the deformation compensation model corresponding to the main gripping condition. The specific method for determining the pose pre-compensation amount is as follows: obtain the grasping conditions of the current picking patch operation, input the grasping conditions into the completed picking and filtering coefficient prediction model, output the picking and filtering coefficient prediction value of the grasping conditions, and determine whether the current grasping conditions are not greater than the picking quality threshold based on the picking and filtering coefficient prediction value. If so, optimize and determine the corresponding pose pre-compensation amount; otherwise, re-grab. The specific optimization method is as follows: determine the corresponding deformation compensation model based on the main grasping conditions corresponding to the current patch picking operation, extract the patch deformation features of the current grasping conditions and input them into the corresponding deformation compensation model, and the deformation compensation model outputs the corresponding pose compensation amount. If there is no master gripping condition corresponding to the current pick-up and patch operation, the similarity between the current vacuum pressure value and the vacuum pressure value corresponding to the master gripping condition is calculated. The similarity of the vacuum pressure values ​​is represented by the Euclidean distance between the vacuum pressure values. The master gripping condition with the maximum similarity is determined and used as the master gripping condition corresponding to the current pick-up and patch operation.

7. The visual self-calibration system for an automated microneedle patching device according to claim 6, characterized in that: The logic underlying the drift prediction model is as follows: The control device's robotic arm operates continuously at its classic working speed for several hours under no-load conditions. During this continuous operation, the temperature of the robotic arm's servo motors is collected, specifically the real-time temperatures of the X, Y, and Z axis servo motors. Simultaneously, the cumulative motion position of each axis is determined based on the grating ruler readings. The cumulative motion mileage of the robotic arm is calculated based on the cumulative motion position of each axis, and the absolute position drift of each axis at corresponding moments is collected. The specific method for calculating the cumulative motion mileage of the robotic arm is as follows: During the operation of the robotic arm, several sampling times are set. At any sampling time, the grating ruler readings of the X, Y and Z axes of the robotic arm are acquired. The grating ruler readings of the X, Y and Z axes of the robotic arm corresponding to the sampling time are accumulated to obtain the cumulative movement distance of the robotic arm at that sampling time. Based on the cumulative motion mileage of the robotic arm, the temperature of the servo motors of each axis, and the absolute position drift of each axis, a drift prediction model is established. Specifically, the absolute position drift data of each axis is used as the dependent variable, and the cumulative motion mileage of the robotic arm and the temperature of the servo motors of each axis are used as independent variables. The drift prediction model expression is constructed through multiple linear regression analysis. The method for establishing the drift prediction models for the X and Y axes of the robotic arm is the same as that for establishing the drift prediction model for the Z axis. Based on the drift prediction models for the X, Y, and Z axes of the robotic arm, the drift prediction values ​​for each axis of the robotic arm are determined.

8. The visual self-calibration system for an automated microneedle patching device according to claim 7, characterized in that: The logic for fusing the pose pre-compensation amount and drift pre-compensation amount is as follows: based on the deformation compensation model, the compensation value and direction of the preset height and the robot's placement posture are output. The preset height is compensated by the drift prediction value of the robot's Z-axis. Specifically, it is expressed as the sum of the drift prediction value of the robot's Z-axis and the preset height. The placement position of the robot arm in the placement posture is compensated by the drift prediction values ​​of the X and Y axes of the device, thereby generating the final compensation control command and driving the robot arm to complete the compensation movement according to the command.

9. A visual self-calibration method for an automated microneedle patching device, used to control the visual self-calibration system of the automated microneedle patching device according to any one of claims 1-8, characterized in that, include: The control device robotic arm measures a composite calibration target fixed in the workspace and completes the spatial relationship calibration of the coordinate system of the vision camera, laser profilometer and the device robotic arm. The end of the device robotic arm is equipped with an actuator, a vision camera and a laser profilometer. The control device's robotic arm performs multiple pick-up and placement operations. Based on the patch deformation data collected by the laser profilometer, a model is established to correspond to the patch deformation characteristics and pose compensation amount under different main gripping conditions, serving as a deformation compensation model. The motor temperature, mileage, and reference position feedback monitored during the no-load operation of the equipment robot are used to establish a drift prediction model that describes the relationship between the drift amount of the robot's end effector and the equipment status and time. After the robotic arm of the device performs the pick-up operation, it obtains the deformation characteristics of the patch adsorbed on the end effector, and processes the patch deformation characteristics using the corresponding deformation compensation model according to the main gripping conditions of the pick-up operation to obtain the pose pre-compensation amount. The drift prediction model is invoked based on the current device status to obtain the drift pre-compensation amount. The pose pre-compensation amount and the drift pre-compensation amount are fused to generate the final compensation control command, and the robot arm is driven to complete the compensation movement according to the command.

Images