An automated multi-pipette tip detection method and system

Abstract

This invention relates to an automated method and system for detecting the tips of multiple pipettes, belonging to the field of automated detection technology. The automated method for detecting the tips of multiple pipettes includes the following steps: using a YOLOv8 model to perform mask segmentation on the pipette tips in a clear image and calculating the angles. The mask segmentation results are used as prompt information and input along with the original image into a SAM2.1 model to generate conditional image feature memory and establish a tracker corresponding to each angle of the tip; in subsequent detection processes, only the original image is input into the SAM2.1 model, which outputs the tip mask; the system controls the tip requiring calibration to perform a displacement within the current screen, and the movement trajectory of the tip is obtained based on the mask output by the SAM2.1 segmentation model; improving detection stability: by introducing the tracking mechanism of the SAM2.1 model, the dependence on the detection results of the YOLO model is eliminated, and the tracking and detection of the pipette tip can still be achieved even when the YOLO model fails, effectively solving the problem of poor detection stability in complex backgrounds.

Description

Technical Field

[0001] This invention relates to the field of automated detection technology, and in particular to an automated method and system for detecting the tips of multiple pipettes, which is applicable to the accurate identification and positioning of the tips of multiple pipettes in scenarios such as biological experiments and chemical analysis. Background Technology

[0002] In biological experiments such as electrophysiology and adaptive PCR, pipettes are commonly used tools, and the accurate detection and positioning of their tips is crucial for experimental accuracy. Traditional pipette tip detection relies heavily on target detection models like YOLO. These models achieve good detection results in environments with uniform illumination and simple backgrounds. However, in bright-field microscopy, problems such as uneven illumination, complex backgrounds (e.g., blood vessels and tissue floaters) that resemble the needle tip, and solution crystallization often occur. This can lead to situations where the YOLO model, under certain challenging background conditions, experiences needle tip obstruction and difficulty in distinguishing nearby interfering objects, severely impacting detection stability and accuracy. Furthermore, target detection models cannot directly output the angle information of the needle tip, making it impossible to distinguish between multiple individual pipettes at different angles.

[0003] Furthermore, existing YOLO-based tracking methods, such as BoT-SORT and ByteTrack, all rely on the effectiveness of target detection. If the model cannot detect the target, tracking is impossible. While some existing technologies have attempted to combine YOLO with SAM models, most of these are merely simple model overlays without optimization for the specific scenario of pipette tip detection. They also lack effective tracking verification mechanisms, making it difficult to eliminate immovable interference. As a result, detection accuracy and stability still need improvement. Summary of the Invention

[0004] Technical issues To address the shortcomings of existing technologies, this invention provides an automated method and system for detecting the tips of multiple pipettes. The aim is to solve the problems of poor detection stability, susceptibility to interference, and inability to distinguish information from multiple pipettes in existing pipette tip detection methods under complex backgrounds, thereby improving the accuracy and reliability of detection.

[0005] Technical solution An automated method for detecting the tips of multiple pipettes includes the following steps: S1: The YOLOv8 model is used to perform mask segmentation on the pipette tip in the clear image, and the endpoint and angle information of the tip are calculated. The mask segmentation result is used as a prompt message in conjunction with the original image. Figure 1 The same input is fed into the SAM2.1 model to generate conditional image feature memory and establish a tracker corresponding to the needle tip at each angle; S2: In subsequent detection processes, only the original image is input into the SAM2.1 model, and the model outputs the pinhead mask; S3: The system controls the needle tip that needs to be calibrated to move a certain distance within the current screen. The movement trajectory of the needle tip is obtained according to the mask output by the SAM2.1 segmentation model. By calculating the composite similarity between the trajectory and the specified displacement, the system verifies whether the detected needle tip point is correct and eliminates interference objects that cannot be moved. S4: Use the information verified by the trajectory as a filtering condition to filter out unreliable mask memories in the Memory Bank, thereby enhancing the recognition and judgment capabilities of the SAM model.

[0006] The composite similarity calculation in step S3 consists of cosine similarity and coordinate component similarity, and the specific calculation method is as follows: Let the detected displacement vector be The expected displacement vector is The detected displacement distance is The expected displacement distance is ; Then the similarity of the x components is: ; y-component similarity: ; Cosine similarity: ; Final similarity: Final similarity = ; When the final similarity is greater than the set threshold, the detected pinpoint is determined to be correct.

[0007] The set threshold is 0.8.

[0008] In one embodiment, the needle tip profile is processed to calculate a rough needle body angle. Since the actual shape of a needle tip is often similar to a triangle, and the axis of the needle body is close to the longest side of the profile, the longest altitude of the circumscribed triangle can be calculated to obtain a rough estimate of the needle body angle. The needle tip profile can be viewed as a set of points. The tip of the needle is marked as , For ease of explanation, the vertical downward direction in the diagram is taken as the 0-degree reference. The angle between the vector ct from the centroid c of the needle tip contour to the tip t and the reference vector, clockwise, is taken as the needle tip angle α. The minimum circumscribed triangle ABC of the contour output by the deep learning model is calculated using OpenCV algorithms based on convex hull optimization and the Rotating Calipers method. The feet of the perpendiculars to the sides corresponding to the altitudes of their endpoints are denoted as A', B', and C'. The height length of the triangle is calculated using Heron's formula. The direction of the longest altitude endpoint pointed to by the centroid c of the contour is taken as the estimated value of ct. As shown in the figure, AA' is the longest side, and cA is the direction of the estimated vector, denoted as D.

[0009] In one embodiment, the needle tip point is calculated based on a coarse needle body angle to obtain a more accurate angle value. Among all contour points, the projection value of the needle tip point onto the needle body angle axis is the largest. An arbitrary set of contour points is selected. Let a point in the equation be denoted as , and then iterate through all other points. The endpoint positions are obtained according to the formula, and then the precise angle value is obtained from the new needle body vector ct. Since the horizontal angle of each needle placement can be known in advance, such as 45 degrees, 135 degrees, 225 degrees, 315 degrees, etc., the angle calculated from the image information can identify the identity of each needle. When there are multiple needle images on the screen, they can be easily distinguished. At the same time, if the instance segmentation model identifies other impurities as needle tips, the mask with excessive deviation from the calculated angle and the set needle body angle can also be filtered out.

[0010] An automated multi-pipette tip detection system includes: The image acquisition module is used to acquire image information from the tip of the pipette; The YOLOv8 detection module is used to detect and segment pipette tips in the initial clear image; The endpoint angle calculation module calculates the endpoint coordinates and angle of the needle tip using a mask, thus distinguishing individual pipettes. The SAM2.1 tracking module receives the detection results and original image from the YOLOv8 detection module, generates conditional image feature memory, and outputs the pinhead mask based solely on the original image in subsequent detections. The displacement control module controls the pipette tip to make a specified displacement within the screen. The similarity calculation module calculates the composite similarity between the needle tip's trajectory and a specified displacement; The memory optimization module filters the mask memories in the Memory Bank based on the trajectory verification results. The results output module outputs the final pipette tip detection results.

[0011] Beneficial effects 1. Improve detection stability: By introducing the tracking mechanism of the SAM2.1 model, the dependence on the detection results of the YOLO model is eliminated. Even when the YOLO model fails, the tracking and detection of the pipette tip can still be achieved, effectively solving the problem of poor detection stability in complex backgrounds. 2. Enhanced anti-interference capability: The composite similarity verification method can accurately distinguish between the needle tip and the interfering object, eliminate the influence of immovable interfering objects, and significantly improve the accuracy of detection; 3. Adaptive to environmental changes: The memory optimization mechanism enables the model to continuously learn and adapt to changes in complex environments during long-term detection, further improving the accuracy of detection; 4. Simple and efficient operation: The entire detection process is highly automated, requiring no manual intervention, which greatly improves the efficiency of experimental operations and reduces human error; 5. Enhanced multi-target parallel processing capability: The endpoint angle calculation module accurately distinguishes and locates multiple individual pipettes, effectively solving the problems of mutual interference and identification confusion in multi-target scenarios, and significantly improving the parallel processing efficiency and reliability of batch experiments. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of the overall process of the present invention; Figure 2 This is a schematic diagram of the YOLOv8 detection of the probe tip and the calculation of the angle in this invention; Figure 3 This is a schematic diagram of the SAM2.1 module structure and process of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] It should be noted that when a component is referred to as being "fixed to" or "set on" another component, it can be directly on the other component or there may be an intermediate component. When a component is considered to be "connected to" another component, it can be directly connected to the other component or there may be an intermediate component present. The terms "vertical," "horizontal," "upper," "lower," "left," "right," and similar expressions used in this specification are for illustrative purposes only and do not represent the only possible implementation.

[0016] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0017] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature and the second feature are in indirect contact through an intermediate medium. Furthermore, "above," "over," and "on top" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0018] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0019] The following is combined with Figures 1-3 The present invention describes an automated method and system for detecting the tips of multiple pipettes.

[0020] An automated method for detecting the tips of multiple pipettes includes the following steps: S1: The YOLOv8 model is used to perform mask segmentation on the pipette tip in the clear image, and the endpoint and angle information of the tip are calculated. The mask segmentation result is used as a prompt message in conjunction with the original image. Figure 1 The same input is fed into the SAM2.1 model to generate conditional image feature memory and establish a tracker corresponding to the needle tip at each angle; S2: In subsequent detection processes, only the original image is input into the SAM2.1 model, and the model outputs the pinhead mask; S3: The system controls the needle tip that needs to be calibrated to move a certain distance within the current screen. The movement trajectory of the needle tip is obtained according to the mask output by the SAM2.1 segmentation model. By calculating the composite similarity between the trajectory and the specified displacement, the system verifies whether the detected needle tip point is correct and eliminates interference objects that cannot be moved. S4: Use the information verified by the trajectory as a filtering condition to filter out unreliable mask memories in the Memory Bank, thereby enhancing the recognition and judgment capabilities of the SAM model.

[0021] The composite similarity calculation in step S3 consists of cosine similarity and coordinate component similarity, and the specific calculation method is as follows: Let the detected displacement vector be The expected displacement vector is The detected displacement distance is The expected displacement distance is ; Then the similarity of the x components is: ; y-component similarity: ; Cosine similarity: ; Final similarity: Final similarity = ; When the final similarity is greater than the set threshold, the detected pinpoint is determined to be correct.

[0022] The set threshold is 0.8.

[0023] We process the needle tip profile and calculate a rough needle body angle. Since the actual shape of a needle tip is often similar to a triangle, and the axis of the needle body is close to the longest side of the profile, we calculate the longest altitude of the circumscribed triangle of the profile to obtain a rough estimate of the needle body angle. The needle tip profile can be viewed as a set of points. The tip of the needle is marked as , For ease of explanation, we take the vertical downward direction in the figure as the 0-degree reference. We use the clockwise angle between the vector ct from the centroid c of the needle tip profile to the tip t and the reference vector as the needle tip angle α. We calculate the minimum circumscribed triangle ABC of the profile output by the deep learning model using OpenCV library algorithms based on convex hull optimization and Rotating Calipers. We denote the feet of the perpendiculars to the heights of their endpoints as A', B', and C'. We calculate the height length of the triangle using Heron's formula. The direction of the longest height triangle endpoint pointed to by the centroid c of the profile is used as the estimated value of ct. As shown in the figure, AA' is the longest side, and cA is the direction of the estimated vector, denoted as D.

[0024] Based on a rough needle body angle, we calculate the needle tip point to obtain a more accurate angle value. Among all contour points, the projection value of the needle tip point on the needle body angle axis is the largest. We then arbitrarily select a set of contour points. Let a point in the equation be denoted as , and then iterate through all other points. The endpoint position is obtained according to the formula, and the precise angle value is obtained from the new needle body vector ct. Since we can know the horizontal angle of each needle in advance, such as 45 degrees, 135 degrees, 225 degrees, 315 degrees, etc., the angle calculated by the image information can confirm the identity of each needle. When there are multiple needle images on the screen, they can be easily distinguished. At the same time, if the instance segmentation model identifies other impurities as needle tips, the mask with excessively large calculated angle deviation and set needle body angle can also be filtered out. An automated multi-pipette tip detection system includes: The image acquisition module is used to acquire image information from the tip of the pipette; The YOLOv8 detection module is used to detect and segment pipette tips in the initial clear image; The endpoint angle calculation module calculates the endpoint coordinates and angle of the needle tip using a mask, thus distinguishing individual pipettes. The SAM2.1 tracking module receives the detection results and original image from the YOLOv8 detection module, generates conditional image feature memory, and outputs the pinhead mask based solely on the original image in subsequent detections. The displacement control module controls the pipette tip to make a specified displacement within the screen. The similarity calculation module calculates the composite similarity between the needle tip's trajectory and a specified displacement; The memory optimization module filters the mask memories in the Memory Bank based on the trajectory verification results. The results output module outputs the final pipette tip detection results.

[0025] It should be noted that: I. Image Acquisition We collected pipette images against a brain slice background under a bright-field microscope as our dataset. The instrument used was the Scientifica SliceScope Pro 2000 integrated electrophysiology system, equipped with a 40×objective (LUMPFLFL40XW / IR, NA 0.8, Olympus) and a SciCam Pro CCD camera for capturing Infrared (IR) Differential Interference Contrast (DIC) images. The image size was 1376 x 1024 pixels, with a pixel unit of one-sixth of a micrometer and a field of view (FOV) of 229 × 171 micrometers. A total of 707 pinpoint images were collected, with manual annotation of the pinpoint outline and pinpoint. These images were divided into training and testing sets in an 8:2 ratio for training the deep learning model and validating the pinpoint detection method, respectively. Additionally, thirty sets of z-axis tomographic images of the pinpoint were included, with a tomographic range of 14 μm and a step size of 1 μm. Each image included a pinpoint focusing position image, and the image title contained information about the microscope's z-axis height at the time of capture.

[0026] II. Overview of the Method and Procedure for Needle Tip Detection Tip calibration occurs after the target (cell) point is determined. The working tip is moved to the screen directly above the current target. At this point, the tip movement distance is often several hundred micrometers, resulting in an error exceeding the radius of one cell, requiring recalibration. Tip calibration mainly includes two modules: one is static detection of the three-dimensional tip coordinates, and the other is dynamic verification of the tip coordinates.

[0027] In the static detection of the 3D needle tip, the needle tip remains stationary. The camera, positioned below the current screen, performs a layer-by-layer scan along the z-axis from bottom to top within the range [-7, 7] μm, with a 1 μm interval between each scan, for a total of 15 steps. The trained YOLO-seg model identifies the needle tip contour, detects the needle tip angle, and uses the point with the largest projection as the needle tip point in the 2D plane. Among the points calculated from each z-axis plane image, the point with the largest projection value and the largest gradient value is selected as the 3D needle tip point. However, during automated testing, the needle tip may be obscured by other impurities or interfered with by nearby objects with similar shapes, causing the YOLO-seg model to fail. In this case, the slower but more powerful SAM2.1 segmentation model can be used to detect the needle tip contour and move the needle tip to the target point. The SAM model tracks the position before and after the displacement to confirm the correctness of the needle tip point. Once the needle tip is determined to have moved to the target point, the calibration process ends.

[0028] III. Detecting the tip and angle on a 2D image stack Detecting the pinhead on a single image is the most crucial step. We chose to identify the pinhead contour and calculate the pinhead angle and pinhead point based on the contour information. The pinhead contour identification uses the instance segmentation model YOLOv8-seg. YOLOv8 is one of the state-of-the-art real-time object detection models, mainly consisting of a Backbone network, a Neck layer, and a Head layer. The Backbone network mainly consists of CBS, C2f, and SPPF modules, used to extract features from the input image; the Neck layer fuses feature maps at different scales; and the Head layer predicts feature maps for the class and box respectively. YOLO-seg additionally generates mask coefficients for multi-scale feature maps in the Head layer and uses the feature map with the highest resolution as input to the Protonet network, outputting Prototypes. The product of the mask coefficients and Prototypes is used to crop and binarize the target region to obtain the predicted mask for the target. We manually labeled the clearest tip contour and class of the pinhead image using LabelMe as the pinhead dataset, and the trained network can directly output the pinhead contour.

[0029] The second step involves processing the needle tip profile to calculate a rough estimate of the needle body angle. Since the actual shape of a needle tip is often similar to a triangle, and the axis of the needle body is close to the longest side of the profile, we calculate the longest altitude of the circumscribed triangle of the profile to obtain a rough estimate of the needle body angle. The needle tip profile can be viewed as a set of points. The tip of the needle is marked as , For ease of explanation, we take the vertical downward direction in the diagram as the 0-degree reference, and use the clockwise angle between the vector ct from the centroid c of the needle tip contour to the tip t and the reference vector as the needle tip angle α. The minimum circumscribed triangle ABC of the contour output by the deep learning model is calculated using OpenCV library algorithms based on convex hull optimization and the Rotating Calipers method. Let the feet of the perpendiculars to the sides corresponding to the altitudes of their endpoints be A', B', and C'. The height length of the triangle is calculated using Heron's formula, and the direction of the longest altitude endpoint pointed to by the contour centroid c is used as the estimate of ct. As shown in the figure, AA' is the longest side, and cA is the direction of the estimated vector, denoted as D.

[0030] The third step involves calculating the needle tip point based on the rough needle body angle to obtain a more accurate angle value. Among all contour points, the needle tip point has the largest projection value on the needle body angle axis. An arbitrary set of contour points is selected. One point in the middle is denoted as traverse all other points The endpoint positions are obtained according to the formula, and then the precise angle value is obtained from the new needle body vector ct. Since we can know the horizontal angle of each needle in advance, such as 45 degrees, 135 degrees, 225 degrees, 315 degrees, etc., the angle calculated from the image information can identify the identity of each needle. When there are multiple needle images on the screen, they can be easily distinguished. At the same time, if the instance segmentation model identifies other impurities as needle tips, the mask with excessive deviation from the calculated angle and the set needle body angle can also be filtered out.

[0031] IV. Combining needle tip motion information with SAM2.1 tracking The YOLO model, based on a convolutional neural network (CNN) architecture, can detect static needle tips and solve most problems related to uneven lighting and complex backgrounds. However, in a few challenging background conditions, the needle tip may be occluded, or there may be obstructions near the needle tip that the YOLO model cannot distinguish. To ensure the stability of needle tip detection, we added a needle tip movement verification based on SAM2.1 tracking. Other YOLO-based tracking methods, such as BoT-SORT and ByteTrack, still rely on the effectiveness of object detection; if the model cannot detect the object, it cannot track it.

[0032] Initial detection phase First, a clear image of the pipette tip is acquired and input into the YOLOv8 model. The model detects and segments the pipette tip in the image, obtaining a bounding box and segmentation mask for each tip. These detection and segmentation results are used as prompts and original... Figure 1 The same input is fed into the SAM2.1 model. The SAM2.1 model's image encoder uses a MAE pre-trained Hiera Image Encoder to extract multi-scale features from the image. The image decoder processes the multimodal input, including image features, point / box cues, memory embeddings output by the memory attention mechanism, the predicted mask of the output image, and memory features. The memory encoder fuses the image features and the output mask memory through a lightweight convolutional layer and stores them in a memory bank. At the same time, it establishes a corresponding tracker for the needle tip at each angle.

[0033] Tracking and detection phase In subsequent detection processes, no further prompts from the YOLOv8 model are input; only the real-time captured original image is fed into the SAM2.1 model. The SAM2.1 model utilizes memory information stored in the Memory Bank to generate new prompts or image codes through Memory Attention, directly outputting the pinhead mask. (Shift verification stage) The system controls the needle tip to be calibrated to move horizontally to the right within the current screen, with a displacement distance of a preset value. Based on the mask output by the SAM2.1 segmentation model, the trajectory of the needle tip during the displacement is obtained. The composite similarity between this trajectory and the specified displacement is calculated. Assuming the detected displacement vector `detectShift = (5, 0)` and the expected displacement vector `movingShift = (5, 0)`, then the detected displacement distance `distance = √(5² + 0²) = 5`, and the expected displacement distance `moving_distance = √(5² + 0²) = 5`. The similarity of the x-component, x_sim, is 5 / 5 = 1, and the similarity of the y-component, y_sim, is 0 / 0 = 1. This is a special case and can be handled accordingly in actual calculations. The cosine similarity, cos_sim, is (5×5 + 0×0) / (5×5) = 1. The final similarity is 0.3×1 + 0.3×1 + 0.4×1 = 1, which is greater than the set threshold of 0.8. Therefore, the detected needle tip is determined to be correct, and static interference objects such as experimental debris in the experimental background are excluded.

[0034] Memory optimization stage The pinhead mask memories verified as correct during trajectory testing are retained in the Memory Bank, while those mask memories deemed incorrect during displacement verification are deleted from the Memory Bank. Through continuous filtering and optimization, the SAM2.1 model can more accurately identify pipette tips in subsequent detection processes, maintaining high detection accuracy even under minor changes in lighting conditions or alterations in the position of background interference.

[0035] It should be noted that the multi-model collaboration and tracking mechanism: This invention innovatively combines the initial detection results of YOLOv8 with the SAM2.1 model, utilizing the Memory Encoder and Memory Attention modules of the SAM2.1 model to achieve tracking and detection of the pipette tip. Unlike traditional YOLO-based tracking methods, this invention establishes a Conditional image memory in the initial stage, and subsequent detection does not rely on the detection results of the YOLO model. Only the original image needs to be input to achieve the output of the tip mask, effectively solving the problem of tracking failure when the YOLO model fails.

[0036] Composite similarity verification: A composite similarity calculation method based on cosine similarity and coordinate component similarity is proposed. It not only considers the consistency of trajectory direction, but also takes into account the magnitude matching of displacement. It can more accurately verify whether the detected needle tip is correct, effectively eliminate immovable interference objects, and improve the accuracy of detection.

[0037] Memory optimization mechanism: The mask memory in the Memory Bank is filtered by the information of trajectory verification to remove unreliable memory, continuously enhance the recognition and judgment ability of the SAM model, enable the model to adapt to changes in complex environment during long-term detection, and further improve the stability of detection.

[0038] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0039] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. An automated multi-pipette tip detection method, characterized by, Includes the following steps: S1: The YOLOv8 model is used to perform mask segmentation on the pipette tip in the clear image, and the endpoint and angle information of the tip are calculated. The mask segmentation result is used as a prompt and input into the SAM2.1 model along with the original image to generate a Conditional Image Memory, and a tracker is established for each angle of the tip. S2: In subsequent detection processes, only the original image is input into the SAM2.1 model, and the model outputs the pinhead mask; S3: The system controls the needle tip that needs to be calibrated to move a certain distance within the current screen. The movement trajectory of the needle tip is obtained according to the mask output by the SAM2.1 segmentation model. By calculating the composite similarity between the trajectory and the specified displacement, the system verifies whether the detected needle tip point is correct and eliminates interference objects that cannot be moved. S4: Use the information verified by the trajectory as a filtering condition to filter out unreliable mask memories output from the memory bank, thereby enhancing the recognition and judgment capabilities of the SAM model.

2. The automated multi-pipette tip detection method of claim 1, wherein, The composite similarity calculation in step S3 consists of cosine similarity and coordinate component similarity, and the specific calculation method is as follows: Let the detected displacement vector be , the expected displacement vector be , the detected displacement distance be , and the expected displacement distance be ; Then the similarity of the x components is: ； y-component similarity: ； Cosine similarity: ； Final similarity: final similarity = ;​ When the final similarity is greater than the set threshold, the detected pinpoint is determined to be correct.

3. The automated multi-pipette tip detection method of claim 2, wherein, The set threshold is 0.

8.

4. The automated multi-pipette tip detection method according to claim 2, characterized in that, The processing of the needle tip contour needs to calculate the rough needle body angle first, because the shape of the real needle tip is often similar to a triangle, the axis direction of the needle body is close to the longest side of the contour, so the longest height of the circumscribed triangle of the contour can be calculated to obtain the rough estimation of the needle body angle. The needle tip contour can be regarded as a point set , the needle tip endpoint is denoted as , , for convenience of description, the vertical downward direction in the figure is taken as the 0 degree reference, the vector ct from the centroid point c of the needle tip contour to the tip t is taken as the reference vector, and the clockwise included angle between the two is taken as the needle tip angle a. The minimum circumscribed triangle ABC of the contour output by the deep learning model is calculated by the OpenCV library algorithm based on convex hull optimization and rotating calipers, the foot points of the longest side corresponding to the height of the end points are denoted as A', B' and C', the height length of the triangle is calculated by means of the Heron formula, and the direction of the end point of the triangle pointed by the contour centroid c in the longest height is taken as the estimated value of ct. As shown in the figure, AA' is the longest side, and cA is the direction of the estimated vector, denoted as D.

5. The automated multi-pipette tip detection method according to claim 4, characterized in that, The method calculates the needle tip point based on a rough needle body angle to obtain a more accurate angle value. Among all contour points, the projection value of the needle tip point on the needle body angle axis is the largest. An arbitrary set of contour points is then selected. One point in the middle is denoted as traverse all other points The endpoint position is obtained according to the formula, and the precise angle value is obtained from the new needle body vector ct. Since the horizontal angle of each needle placement can be known in advance, such as 45 degrees, 135 degrees, 225 degrees, 315 degrees, etc., the angle calculated by image information can confirm the identity of each needle. When there are multiple needle images on the screen, they can be easily distinguished. At the same time, if the instance segmentation model identifies other impurities as needle tips, the mask with excessively large calculated angle deviation and set needle body angle can also be filtered out.

6. An automated multi-pipette tip detection system, characterized in that, include: The image acquisition module is used to acquire image information from the tip of the pipette; The YOLOv8 detection module is used to perform masking segmentation on the pipette tip in the initial clear image; The endpoint angle calculation module calculates the endpoint coordinates and angle of the needle tip using a mask, thus distinguishing individual pipettes. The SAM2.1 tracking module receives the detection results and original image from the YOLOv8 detection module, generates conditional image feature memory, and outputs the pinhead mask based solely on the original image in subsequent detections. The displacement control module controls the pipette tip to make a specified displacement within the screen. The similarity calculation module calculates the composite similarity between the needle tip's trajectory and a specified displacement; The memory optimization module filters the mask memories in the Memory Bank based on the trajectory verification results. The results output module outputs the final pipette tip detection results.

Images