A method, device, equipment, medium and product for detecting injection speed of a pipette

Abstract

This application discloses a method, apparatus, device, medium, and product for detecting the injection speed of a pipette, relating to the field of micro-liquid processing technology. The method includes: acquiring original pipetting video; preprocessing each original image video frame of the original pipetting video and performing connected component detection to obtain the center position and area information of the label; calculating the measured distance corresponding to the preprocessed image based on the center position and area information of the label; then filtering injection segments based on the measured distances corresponding to all preprocessed images; calculating the final intra-frame injection volume of each injection image frame based on the measured distance of the injection image frames in the injection segment; converting the final intra-frame injection volume into motor control parameters; and accurately detecting the injection speed and achieving automated transfer by real-time image capture and processing of the manual pipetting process.

Description

Technical Field

[0001] This application relates to the field of micro-liquid processing technology, and in particular to a method, apparatus, equipment, medium and product for detecting the injection speed of a pipette based on machine vision. Background Technology

[0002] With the development of modern biotechnology, synthetic biology, and chemical biology, the streamlined processing of large volumes of samples has become increasingly common. These fields often require the extraction and transfer of trace amounts of liquids, i.e., micro-liquid handling technology (micropipettes). Taking molecular diagnostics as an example, processing various biological reagents and samples is a fundamental and frequent experimental operation, requiring high-precision liquid handling of biological reagents or samples with different properties. Compared to manual pipetting, automated pipetting devices offer advantages such as high throughput, good stability, and reduced manual workload. As the requirements for sample throughput and the reliability of detection results in biochemical experiments continue to increase, fully automated pipetting devices based on micro-liquid handling technology have become an inevitable trend in the development of biochemical laboratories.

[0003] In some applications of pipetting devices, due to variations in liquid properties, injection environment, and injection microstructure—such as organ-on-a-chip gel injection—high injection speeds are required, necessitating different or even varying injection rates. While many automated pipetting devices are equipped with programmable speed functions, allowing users to customize the liquid handling speed, experienced operators, despite achieving high success rates with manual injections, find it difficult to quantify their operational experience and accurately describe the magnitude and variation of their manual injection speed when using automated pipetting devices. Furthermore, the complexity and variability of injection scenarios make obtaining operational parameters through calculation or simulation, or through extensive experimentation, extremely costly in terms of both economy and time. If automated pipetting devices could directly acquire the speed parameters from manual pipetting, rapid automated injections could be achieved in complex scenarios.

[0004] Although many fully automated pipetting devices have been developed to support customizable pipetting parameters to handle complex pipetting scenarios, the acquisition of pipetting parameters mainly comes from operational experience and extensive operational testing, and there is still no simple and effective quantification method. Therefore, how to enable fully automated pipetting devices to quickly acquire the correct programmable pipetting parameters is also a problem that needs to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a method, device, equipment, medium, and product for detecting the injection speed of a pipette, which enables an automated pipetting device to learn manual speed and achieve rapid automated injection in complex scenarios.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] In a first aspect, this application provides a method for detecting pipette injection speed, including:

[0008] Acquire the original pipetting video; the original pipetting video is a video obtained by the camera capturing the pipetting injection operation; the pipette has a first label, a second label and a third label installed vertically side by side from top to bottom; the first label and the second label are fixed to the external clamp of the pipette, and the third label is fixed to the pipette body; the original pipetting video includes several original image video frames;

[0009] For each of the original image / video frames, the original image / video frame is preprocessed to obtain a preprocessed image;

[0010] Connectivity detection is performed on the preprocessed image to obtain the position coordinates of the first center, the second center, and the third center, as well as the first area, the second area, and the third area in each preprocessed image; the first area, the second area, and the third area are the areas of the first label, the second label, and the third label in the preprocessed image, respectively; the first center, the second center, and the third center are the centroids of the first area, the second area, and the third area, respectively.

[0011] The measured distance corresponding to each preprocessed image is determined based on the first pixel distance, the second pixel distance, the first area, the second area, and the third area; the first pixel distance is the pixel length between the second center and the third center on the camera plane; the second pixel distance is the pixel length between the first center and the second center on the camera plane.

[0012] The injection segment is selected based on the measured distance corresponding to all the preprocessed images;

[0013] For each injection image frame in the injection segment, the final intra-frame injection volume corresponding to the injection image frame is calculated based on the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment; the injection segment consists of several injection image frames.

[0014] The final intraframe injection volume corresponding to the injection image frame is converted into motor control parameters; the motor control parameters include the pre-frequency division coefficient and the number of motor running pulses.

[0015] Secondly, this application provides a pipette injection speed detection device, comprising:

[0016] The original pipetting video acquisition module is used to: acquire original pipetting video; the original pipetting video is a video obtained by a camera capturing the injection operation of a pipette; a first label, a second label, and a third label are vertically mounted side by side from top to bottom on the pipette; the first label and the second label are fixed on the external clamp of the pipette, and the third label is fixed on the pipette body; the original pipetting video includes several original image video frames;

[0017] The preprocessing module is used to: preprocess each of the original image / video frames to obtain a preprocessed image;

[0018] The connected component detection module is used to: perform connected component detection on the preprocessed image to obtain the position coordinates of the first center, the second center, and the third center, as well as the first area, the second area, and the third area in each preprocessed image; the first area, the second area, and the third area are respectively the areas of the first label, the second label, and the third label in the preprocessed image; the first center, the second center, and the third center are respectively the centroids of the first area, the second area, and the third area;

[0019] The distance calculation module is used to: determine the measured distance corresponding to each preprocessed image based on the first pixel distance, the second pixel distance, the first area, the second area, and the third area; the first pixel distance is the pixel length between the second center and the third center on the camera plane; the second pixel distance is the pixel length between the first center and the second center on the camera plane;

[0020] The injection segment screening module is used to: screen injection segments based on the measured distances corresponding to all the preprocessed images;

[0021] The final intra-frame injection volume calculation module is used to: for each injection image frame in the injection segment, calculate the final intra-frame injection volume corresponding to the injection image frame based on the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment; the injection segment consists of several injection image frames;

[0022] The motor control parameter conversion module is used to convert the final intra-frame injection volume corresponding to the injection image frame into motor control parameters; the motor control parameters include the pre-frequency division coefficient and the number of motor running pulses.

[0023] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described pipette injection speed detection method.

[0024] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described pipette injection speed detection method.

[0025] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the pipette injection speed detection method described above.

[0026] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0027] This application provides a method, apparatus, device, medium, and product for detecting the injection speed of a pipette. By using a machine vision-based speed acquisition method, each original image video frame of the original pipetting video is preprocessed and connected component detection is performed to obtain the tag's movement information (the tag's movement information includes the position coordinates of the first, second, and third centers in the preprocessed image, as well as the first, second, and third areas). The measured distance corresponding to the preprocessed image is calculated using the tag's movement information. Then, injection segments are selected based on the measured distances corresponding to all preprocessed images. The final intra-frame injection volume of each injection image frame is calculated based on the measured distances of the injection image frames within the injection segment. The final intra-frame injection volume is converted into motor control parameters. Through real-time image capture and processing of the manual pipetting process, the injection speed is accurately detected and automated transfer is achieved, which is of great significance for improving the adaptability of automated pipetting systems. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is an application environment diagram of a pipette injection speed detection method according to an embodiment of this application;

[0030] Figure 2 A schematic flowchart of a pipette injection speed detection method provided in an embodiment of this application;

[0031] Figure 3 A flowchart illustrating the overall process of a pipette injection speed detection method according to an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of an image processing procedure provided in an embodiment of this application;

[0033] Figure 5 A schematic diagram illustrating the distribution of labels under the influence of depth of field, provided as an embodiment of this application;

[0034] Figure 6 A schematic diagram of a depth-of-field compensation algorithm provided in an embodiment of this application;

[0035] Figure 7 A schematic diagram of a measured distance-frame graph provided in an embodiment of this application;

[0036] Figure 8(a) is a flowchart of finding the start frame of the injection segment according to an embodiment of this application; Figure 8(b) is a flowchart of finding the end frame of the injection segment according to an embodiment of this application.

[0037] Figure 9 A schematic diagram of the original and optimized injection volume-frame curves for an injection segment provided in an embodiment of this application;

[0038] Figure 10(a) is a frame-to-frame diagram of the injection volume per frame after initial optimization according to an embodiment of this application; Figure 10(b) is a frame-to-frame diagram of the injection volume per frame after secondary optimization according to an embodiment of this application;

[0039] Figure 11 A functional module diagram of a pipette injection speed detection device provided in an embodiment of this application;

[0040] Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] The pipette injection speed detection method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send the raw pipetting video to server 104. After receiving the raw pipetting video, server 104 preprocesses each raw image video frame of the raw pipetting video and performs connected component detection to obtain the center position and area information of the label. It then calculates the measured distance corresponding to the preprocessed image using the label's center position and area information. Based on the measured distances corresponding to all preprocessed images, it filters injection segments and calculates the final intra-frame injection volume for each injection image frame based on the measured distances of the injection image frames within the injection segment. The final intra-frame injection volume is then converted into motor control parameters. Server 104 can feed back the obtained motor control parameters to terminal 102. In addition, in some embodiments, the pipette injection speed detection method can also be implemented by the server 104 or the terminal 102 separately. For example, the terminal 102 can directly perform pipette injection speed detection on the original pipetting video, or the server 104 can obtain the original pipetting video from the data storage system and perform pipette injection speed detection on the original pipetting video.

[0044] The terminal 102 can be, but is not limited to, various desktop computers and laptops. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers, or it can be a cloud server.

[0045] In one exemplary embodiment, such as Figure 2 and Figure 3 As shown, a method for detecting pipette injection speed is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 207. Wherein:

[0046] Step 201: Obtain the original pipetting video; the original pipetting video is a video captured by a camera during a pipetting operation; the pipette has a first label, a second label, and a third label mounted vertically side by side from top to bottom; the first label and the second label are fixed to the external clamp of the pipette, and the third label is fixed to the pipette body; the original pipetting video includes several original image video frames.

[0047] Step 202: For each of the original image video frames, preprocess the original image video frame to obtain a preprocessed image.

[0048] Step 203: Perform connected component detection on the preprocessed image to obtain the position coordinates of the first center, the second center, and the third center, as well as the first area, the second area, and the third area in each preprocessed image; the first area, the second area, and the third area are the areas of the first label, the second label, and the third label in the preprocessed image, respectively; the first center, the second center, and the third center are the centroids of the first area, the second area, and the third area, respectively.

[0049] Step 204: Determine the measured distance corresponding to each preprocessed image based on the first pixel distance, the second pixel distance, the first area, the second area, and the third area; the first pixel distance is the pixel length between the second center and the third center on the camera plane; the second pixel distance is the pixel length between the first center and the second center on the camera plane.

[0050] Step 205: Filter the injection segments based on the measured distances corresponding to all the preprocessed images.

[0051] Step 206: For each injection image frame in the injection segment, calculate the final intra-frame injection volume corresponding to the injection image frame based on the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment; the injection segment consists of several injection image frames.

[0052] Step 207: Convert the final intraframe injection volume corresponding to the injection image frame into motor control parameters; the motor control parameters include the pre-frequency division coefficient and the number of motor running pulses.

[0053] By implementing steps 201 to 207 above, the injection speed is accurately detected and automated by real-time image capture and processing of the manual pipetting process. Furthermore, compared to commonly used optical scales and inertial measurement units, this method is relatively low in cost, noise, and error. Machine vision-based speed acquisition involves first obtaining raw pipetting video using a camera or similar device, then sequentially processing each frame of the raw video to find feature points, calibrating them, measuring the actual distance, and finally converting the measured distance into speed. The pipetting speed detection method provided in this application has no special requirements for the measured object or the object being measured, making it suitable for situations where traditional contact measurements are not feasible. The sampling frequency is mainly determined by the camera, generally between 30-1000Hz, and the accuracy is mainly determined by the distance measurement algorithm, reaching sub-millimeter level. It has the advantage of low hardware requirements and minimizing interference during the injection process.

[0054] Images and videos of the pipette operation process are captured using a smartphone. During filming, a phone holder is used to stabilize the camera and ensure image stability. Feature labels are attached to the pipette plunger for positional identification in the images. These labels are colored circular markers mounted on the external clamp of the pipette to ensure they are not obstructed during the pipetting process. This application uses color differentiation to select the color corresponding to the label in the image. Since each laboratory's equipment and instruments differ, users need to manually select the color that is absent or least present in the background as the label color.

[0055] After reading and opening the original pipetting video using OpenCV, the system continuously retrieves the next frame of the original pipetting video in a loop, performs tag detection and distance calculation, and continues this process until the entire original pipetting video is processed. Then, it performs multiple speed conversion steps and finally converts the data into motor control parameters for output.

[0056] In another exemplary embodiment of this application, the preprocessing of the original image video frame in step 202 to obtain the preprocessed image can be replaced by steps 301 to 303.

[0057] Step 301: Convert the original image / video frame from RGB color space to HSV color space to obtain HSV image / video frames;

[0058] Step 302: Binarize the HSV image video frames to obtain a binarized image;

[0059] Step 303: Perform a closing operation on the binarized image to obtain the image after closing operation; the image after closing operation is the preprocessed image.

[0060] The original image and video frames being processed are in RGB format by default. In RGB format, it's difficult to accurately select the threshold range for a specific color. Therefore, the original image and video frames are converted to HSV color space for processing. HSV color space describes color using three dimensions: hue, saturation, and brightness. This not only better matches human visual perception but also makes it easier to distinguish colors. For any coordinate point in an image, its RGB color space is (R, G, B), and its HSV color space is (H, S, V). The steps for converting an image from RGB color space to HSV color space are as follows: First, the R, G, and B values ​​need to be converted to between 0 and 1, as shown in the following formula.

[0061]

[0062] Next, calculate the values ​​of H, S, and V respectively, using the formulas shown below.

[0063]

[0064] Since OpenCV requires visualization of HSV images, it is necessary to convert each value to between 0 and 255. The converted H', S', and V' are represented by the following formulas.

[0065]

[0066] After setting the color filter, this HSV space image is simultaneously converted into a binarized image, with pixels within the labeled range turned white and others turned black. The binarized image is as follows. Figure 4 As shown in (a), basically only the color corresponding to the label is retained.

[0067] The remaining noise is removed in subsequent processing. The image is first dilated using a closing operation, followed by an erosion operation; the calculation process is shown in the following formula.

[0068]

[0069] In the formula, T is the input image, Y is the structuring element, and z is a point in the image.

[0070] Closing operations aim to eliminate noise at the edges of objects, fill voids inside objects, and smooth object boundaries, helping to connect broken parts of an object and maintain its overall shape. An image processed by closing operations looks like... Figure 4 As shown in (b) of the diagram.

[0071] After the first two steps, the area of ​​remaining noise is generally small. By iterating through the sizes of all connected components, we can directly determine which regions are labeled. OpenCV's ConnectedComponents function can detect all connected components in an image and output information such as the area, width, centroid, and top-left vertex coordinates of each component. After calling this function, we only need to iterate through all connected component information, retaining only the indices of the three regions with the largest areas. Simultaneously, we read the centroid coordinates based on these indices, arrange them sequentially according to their y-coordinates, and read other relevant information. For example... Figure 4 As shown in (b), the center positions A”, B” and C” of the first center, second center and third center of the three labels and the first area S are obtained respectively. A” Second area S B” and the third area S C” The accurate detection of the tags provides the foundation for subsequent distance and speed calculations.

[0072] During injection, the two upper labels move along with the external clamp and the pipette plunger. Therefore, the distance the plunger is pressed during pipette injection is equal to the distance the lower and middle labels change. However, since pixel distances can only be obtained from frame images, a constant standard distance is still needed as a benchmark. Since the distance between the middle and upper labels remains constant, it can be used as the benchmark for calculation. The actual distance l between the second center B and the third center C... BC The change is the same as the change in the length of the pipette plunger, that is, only l needs to be measured. BC The pipetting speed can be measured, and l can be calculated. BC The formula is shown below.

[0073]

[0074] Among them, l AB , l BC l represents the actual length between the centroids of the corresponding labels. AB Let l be the actual distance between the first center A and the second center B. A”B” , l B”C” l is the pixel length between the centroids of the label on the camera plane. A”B” The distance is the second pixel, l B”C” The distance is the first pixel.

[0075] Because of l A”B” and l B”C” It can be obtained from previous image processing, and l AB The length remains constant, and it is not necessary to obtain l. BC For the actual length, we only need to obtain its trend of change, then we can let L d =l BV / l AB We can obtain the following formula.

[0076]

[0077] However, since depth of field affects distance calculation, this application calculates the distance using the detected label centroid and area, and adds a depth-based correction coefficient for compensation. The distribution of the three labels under the influence of depth of field is as follows: Figure 5 As shown.

[0078] Since the actual size of the three labels is the same, and the actual length represented by any pixel in the preprocessed image is inversely proportional to the depth of field, that is, the square root of the actual area represented by each pixel is inversely proportional to its depth of field, the following two formulas can be derived.

[0079]

[0080] Among them, l A”r , l B”r , l C”r These are the actual lengths represented by pixels A, B, and C on line segment A and C, respectively; d A d B d C These are the distances from the centroids A, B, and C to the camera plane, respectively.

[0081] Based on formula (7), the following formula can be obtained.

[0082]

[0083] Where x is any point on line segment AC, x” is the corresponding projection point of point x on the camera plane, and d x Let l be the distance from point x to the camera plane. x”r Let x” represent the actual length of line segment A”C”.

[0084] Define a = l A”r / l B”r b = l C”r / l B”r Combining formula (8), we can obtain the principle diagram of the depth-of-field compensation algorithm, as follows: Figure 6 As shown.

[0085] From an integral perspective, let l be the line segment corresponding to all points x on line segment AB. x”r Adding them together gives the actual distance l between the first center A and the second center B. AB Therefore, the deviation between the original algorithm and the optimized algorithm in calculating the length can be derived, and thus the first error coefficient k caused by the depth of field on line segment AB can be calculated. AB The calculation formula is shown below.

[0086]

[0087] Similarly, we can obtain the second error coefficient k on line segment BC caused by depth of field. BC The calculation formula is shown below.

[0088]

[0089] By introducing the error coefficient into formula (6), the corrected measured distance L can be obtained. d The corrected measured distance is the measured distance corresponding to the preprocessed image, and its calculation formula is shown in the following formula.

[0090]

[0091] The measured distance L corresponding to the preprocessed image is calculated. dAfter this, the processing of this frame is complete, and the calculated distance is saved in a dynamic array. The program then receives the preprocessed image of the next frame and repeats the above two processing steps. When the program no longer receives the preprocessed image of the next frame, it indicates that the entire video has been processed. At this point, the program proceeds to the next step of speed conversion processing.

[0092] After the above image processing, we obtain a data structure containing the measured distance L of each preprocessed frame in the video. d An array of values. A graph is plotted with time frames on the x-axis and the measured distance of each preprocessed image frame on the y-axis. The schematic result of the measured distance-frame graph is shown below. Figure 7 As shown.

[0093] Figure 7 The process involves a typical pipetting procedure, categorized by the state of the plunger: stop, injection, stop, rebound, and stop again. During actual injection, due to the significant hysteresis caused by the air plunger and the viscosity of the injected fluid, the liquid in the pipette will remain injected for a period after the plunger stops at a certain point. If the pipette is then pulled out from the injected object, some liquid will spray directly from the tip, and the liquid in the injected object may flow back, directly causing the injection to fail. Since the stop time can be customized by the program, this application does not identify the stop time, but only collects data from the injection phase. Furthermore, the injection phase includes a stop phase where no injection occurs. Therefore, the actual distance of the plunger should remain constant. However, due to certain errors in the aforementioned distance measurement algorithm, the detected distance will fluctuate. This issue has been considered and optimized.

[0094] First, the injection segments must be selected from the process, such as... Figure 7 As shown, i s To inject the starting frame of the paragraph, i e To inject the end frame of the paragraph, i m The frame with the maximum distance is the event frame corresponding to the point where the maximum distance is measured during the entire pipetting process. n To find the minimum distance frame, which is the event frame corresponding to the point where the distance is minimized during the entire pipetting process, we need to find the i-th of the injection segment. s and i e Two dots.

[0095] In another exemplary embodiment of this application, step 205 may include steps 401 to 411.

[0096] Step 401: Determine the minimum distance frame; the minimum distance frame is the event frame corresponding to the point where the minimum distance is measured during the pipetting process.

[0097] Step 402: Using the frame with the minimum distance as the initial frame, traverse leftward from the frame with the minimum distance, and determine whether the measured distance corresponding to the current time frame satisfies the first set condition, to obtain the first judgment result; the first set condition is represented as s. i m ×(1-k a ); where s i s m The time frame is i, i m The distance measured at time; k a The selected threshold; i, i m These represent the current time frame and the frame with the maximum distance, respectively.

[0098] Step 403: If the first judgment result is yes, then replace the current time frame with the previous time frame of the current time frame and return to the step of "judging whether the measured distance corresponding to the current time frame meets the first set condition".

[0099] Step 404: If the first judgment result is negative, then determine whether the second set condition is met, and obtain the second judgment result; the second set condition is that the measured distance corresponding to the previous time frame of the current time frame is greater than the measured distance corresponding to the current time frame.

[0100] Step 405: If the second judgment result is yes, then replace the current time frame with the previous time frame of the current time frame and return to the "Judgment whether the second set condition is met" step.

[0101] Step 406: If the second judgment result is negative, then the current time frame is determined as the starting frame of the injection segment.

[0102] Step 407: Using the starting frame of the injection segment as the initial frame, traverse to the right from the starting frame of the injection segment, and determine whether the measured distance corresponding to the current time frame meets the third set condition, thus obtaining the third judgment result. The third set condition is s. i >s n ×(1+k a ), s n For time frames equal to i n The distance measured at that time.

[0103] Step 408: If the third judgment result is yes, then replace the current time frame with the next time frame of the current time frame and return to the step of "judging whether the measured distance corresponding to the current time frame meets the third set condition".

[0104] Step 409: If the third judgment result is negative, then determine whether the fourth setting condition is met, and obtain the fourth judgment result; the fourth setting condition is that the measured distance corresponding to the next time frame of the current time frame is less than the measured distance corresponding to the current time frame. The fourth setting condition can be expressed as s i+1 <s​i .

[0105] Step 410: If the fourth judgment result is yes, then replace the current time frame with the next time frame of the current time frame and return to the "Judgment whether the fourth set condition is met" step.

[0106] Step 411: If the result of the fourth judgment is negative, then the current time frame is determined as the end frame of the injection segment; the start frame of the injection segment, the end frame of the injection segment, and all time frames in between constitute the injection segment.

[0107] Since the maximum distance can occur at either the beginning or the end of the injection, while the minimum distance always occurs after the injection is completed, i.e., i s n ,i e ≤i n Then we can first find the frame i with the smallest distance. n And iterate to the left to find the starting frame i of the injection segment. s Then, traverse to the right to obtain the endpoint frame i of the injection segment. e The specific process is shown in Figure 8(a). First, starting from frame i with the smallest distance... n Start iterating to the left until s is no longer satisfied. i m ×(1-k a Until then. Here k a =0.02, mainly determined based on the error rate of subsequent single-frame ranging measurements. At this point, the current time frame i is approaching the end frame i of the injection segment. e However, it may be more effective than injecting the end frame of the paragraph. e If it's large, then as long as there's s i-1 >s i If it continues traversing to the left, then it is considered that the injection segment starting frame i has been reached. s The process of finding the end frame of the injection segment is shown in Figure 8(b). Since it is similar to the start frame of the injection segment, it will not be explained again here.

[0108] Step 206 above specifically includes steps 501 to 503.

[0109] Step 501: Based on the measured distance of the injection image frame, the measured distance of the injection start frame of the injection segment, the measured distance when the pipette plunger is fully pressed, and the maximum injection volume of the pipette, obtain the original injected volume corresponding to the injection image frame.

[0110] Step 502: Optimize the original injected volume corresponding to the injected image frame to obtain the first optimized injected volume corresponding to the injected image frame.

[0111] ​​Step 503: Perform outlier removal processing on the first-optimized injected volume corresponding to the injected image frame to obtain the second-optimized injected volume corresponding to the injected image frame; the second-optimized injected volume is the final intra-frame injection volume.

[0112] The original injected volume a corresponding to the i-th injection image frame i The calculation formula is shown below.

[0113]

[0114] Among them, s s For the detected injection segment start frame i s The corresponding measured distance; s min The distance measured when the pipette plunger is fully depressed needs to be calibrated by taking a photo beforehand; V f This represents the maximum injection volume of the pipette, expressed in μL.

[0115] Based on the original injected volume a corresponding to all injection image frames i Can draw as Figure 9 The original injected volume-frame curve is shown.

[0116] However, the injection process should only have two states: injection and stop, i.e., the original injected volume a. i The value may increase or remain unchanged over time, but Figure 9 The original injected volume-frame curve showed fluctuations and should be optimized to obtain the first optimized injected volume. The first optimization formula is shown in the following formula.

[0117]

[0118] In the formula, a i 'For the optimized injected volume of the i-th injected image frame, a' i-1 a is the original injected volume of the (i-1)th injected image frame. i+1 This represents the original injected volume of the (i+1)th injected image frame.

[0119] The processed data is as follows Figure 9 As shown in the initial optimization curve, the processed volume ensures that the injection volume can only remain constant or increase over time. However, there are also instances where the optimized injection volume is larger than the original volume at a certain point in time. Therefore, this application continues optimization. First, let's consider the first optimized injection volume *a*. i 'Converted to injection volume v per frame time' i The relationship between intra-frame injection volume and frame is shown in the following formula.

[0120] v i =a i+1'-a i '(14).

[0121] In the formula, a i+1 ' is the intra-frame injection volume of the (i+1)th injected image frame.

[0122] After the first optimization process described above, the inter-frame injection volume-frame image shown in Figure 10(a) can be obtained. It can be seen that there are three main outliers: Outlier I is that some segments that should be injected have an injection speed of 0; Outlier II is that a certain injection speed also occurs during the stopping process; Outlier III is that the injection speed is much higher than other points at certain times.

[0123] These three outliers are caused by a combination of identification errors and the preliminary optimizations mentioned above. Outliers I and III can usually be eliminated by mean filtering, but mean filtering can actually increase the range of outlier II. Therefore, outlier II should be addressed first. The handling of outlier II mainly involves identifying the entire velocity array and identifying all consecutive intervals with positive velocities. For each interval, if its length is less than 5 and the total injection volume is less than 2% of the total, it is considered an outlier interval caused by algorithm error. All velocities within this interval are set to 0, and the total velocity is retained. This sum is then added in a 1:1 ratio to the last frame of the previous interval and the first frame of the next interval, ensuring that outlier I is not affected while removing outlier II.

[0124] Then, mean filtering is applied to the processed curve. Since the sum of all injection velocities (i.e., the total injection volume) must remain unchanged after mean filtering, additional processing is required on the edge frames to obtain the secondary optimized injected volume. The secondary optimization formula is shown in the following formula.

[0125]

[0126] Among them, v i ' is the final intra-frame injection volume corresponding to the i-th injected image frame, in μL; N is the number of mean filter segments, which can be 5 in this embodiment; Z v This represents the total number of frames in the injected segment.

[0127] After the two optimization steps described above, the intra-injection volume-frame graph after secondary optimization is plotted, as shown in Figure 10(b). After processing, all three outliers are removed, and the velocity variation between adjacent frames is smaller after mean filtering. Simultaneously, it is converted back to a total injection volume-frame image, as shown... Figure 9 As shown in the secondary optimization curve, the final curve better matches the measurement results and is smoother, making it more suitable for the stable operation of the motor.

[0128] The optimized inter-frame injection speed is now obtained. Subsequent steps will involve parameter transformation to ensure consistent speed across the automated pipetting device. After parameter transformation, the injection speed within each frame will be the same, while the speed will differ between frames. The control strategy for the automated pipetting device variations involves adjusting the prescaler coefficient (PSC) to control the speed, and controlling the number of pulses at this speed within a timer interrupt function. The final output driving parameter is the PSC corresponding to any time frame i. i and the number of running pulses n i .

[0129] First, set the injection speed v for each frame. i The number of pulses p required to convert into motor motion i The calculation formula is shown below.

[0130]

[0131] Among them, V s The volume of each pulse injection for the transplanted pipette.

[0132] The number of pulses calculated by the above formula contains decimals. Since the stepper motor can only move an integer number of pulses, the number of pulses in each frame also needs to be processed. The processing strategy is to run an integer number of pulses in the current injection image frame, and add the remaining decimal part to the next injection image frame for processing. The processing formula is shown below.

[0133]

[0134] Where, n i Inject the actual number of motion pulses corresponding to the i-th image frame, n e The remaining decimal part will be added to the next injected image frame for calculation; / 1 means rounding down; %1 means dividing by 1 and taking the remainder.

[0135] At the same time, after processing, there will be n i ≤p i That is, while maintaining the same speed, the motor completes n movements. i After one pulse, before a full frame has been completed, the remaining time is added to subsequent injection image frames. The remaining time t of the current injection image frame... e The calculation formula is shown below.

[0136]

[0137] Among them, t f The original injection video is recorded at 60fps, representing the actual time per frame. f It takes 16.67ms.

[0138] Then, based on the pulse frequency f p The calculation formula can be obtained as follows.

[0139]

[0140] Where arr is the auto-reload value, f clk This is the clock frequency.

[0141] After conversion, the prescaler coefficients psc can be obtained. i The calculation formula is shown below.

[0142]

[0143] In the formula, psc i Inject the pre-frequency division coefficients of the image frame into the i-th frame.

[0144] When the injection stops, the psc is set. i =0, the number of pulses becomes the pause time, in milliseconds. To reduce the total parameter length, intervals with the same speed are merged.

[0145] This application also provides an application scenario in which the above-described pipette injection speed detection method is applied. Specifically, the pipette injection speed detection method provided in this embodiment can be applied in an injection speed transplantation scenario. The injection speed transplantation scenario includes a video acquisition stage, an injection speed detection link, and an injection speed transplantation stage. The original pipetting video enters the injection speed detection link from the video acquisition stage, obtains the corresponding motor control parameters, and then enters the downstream injection speed transplantation stage. The pipette injection speed detection method provided in this embodiment belongs to the injection speed detection link. Specifically, in the injection speed detection link process for the original pipetting video, each original image video frame of the original pipetting video can be preprocessed and connected component detection can be performed to obtain the center position and area information of the label. The measured distance corresponding to the preprocessed image is calculated using the center position and area information of the label. Then, the injection segment is selected according to the measured distance corresponding to all preprocessed images. The final intra-frame injection volume of each injection image frame is calculated according to the measured distance of the injection image frames in the injection segment, and the final intra-frame injection volume is converted into motor control parameters.

[0146] The pipette injection speed detection method of this application has the following advantages:

[0147] 1. High-precision detection: Through optimized image processing algorithms, high-precision detection of injection speed is achieved, which is suitable for a variety of complex scenarios.

[0148] 2. Low hardware cost: Utilizing smartphone cameras for video capture reduces hardware costs and equipment complexity.

[0149] 3. Automated transfer: It can learn the manual pipetting process and automatically transfer it to different experimental scenarios, improving experimental efficiency and repeatability.

[0150] 4. High adaptability: Through the design and installation of feature tags, the problems of obstruction and position change that may occur during the operation of the pipette are overcome.

[0151] Based on the same inventive concept, this application also provides a pipette injection speed detection device for implementing the pipette injection speed detection method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of the one or more pipette injection speed detection device embodiments provided below can be found in the limitations of the pipette injection speed detection method described above, and will not be repeated here.

[0152] In one exemplary embodiment, such as Figure 11 As shown, this application provides a pipette injection speed detection device including the following modules.

[0153] The original pipetting video acquisition module T1 is used to: acquire original pipetting video; the original pipetting video is a video obtained by a camera capturing the pipetting operation; the pipette has a first label, a second label and a third label installed vertically side by side from top to bottom; the first label and the second label are fixed on the external clamp of the pipette, and the third label is fixed on the pipette body; the original pipetting video includes several original image video frames.

[0154] The preprocessing module T2 is used to: preprocess each of the original image video frames to obtain a preprocessed image.

[0155] The connected component detection module T3 is used to: perform connected component detection on the preprocessed image to obtain the position coordinates of the first center, the second center, and the third center, as well as the first area, the second area, and the third area in each preprocessed image; the first area, the second area, and the third area are the areas of the first label, the second label, and the third label in the preprocessed image, respectively; the first center, the second center, and the third center are the centroids of the first area, the second area, and the third area, respectively.

[0156] The distance calculation module T4 is used to: determine the measured distance corresponding to each preprocessed image based on the first pixel distance, the second pixel distance, the first area, the second area, and the third area; the first pixel distance is the pixel length between the second center and the third center on the camera plane; the second pixel distance is the pixel length between the first center and the second center on the camera plane.

[0157] The injection segment screening module T5 is used to: screen injection segments based on the measured distances corresponding to all the preprocessed images.

[0158] The final intra-frame injection volume calculation module T6 is used to: calculate the final intra-frame injection volume corresponding to each injection image frame in the injection segment based on the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment; the injection segment consists of several injection image frames.

[0159] The motor control parameter conversion module T7 is used to convert the final intra-frame injection volume corresponding to the injection image frame into motor control parameters; the motor control parameters include the pre-frequency division coefficient and the number of motor running pulses.

[0160] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 12 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores injection rate detection data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a pipette injection rate detection method.

[0161] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0162] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0163] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0164] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0165] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0166] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0167] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0168] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting pipette injection speed, characterized in that, The method for detecting the pipette injection speed includes: Acquire the original pipetting video; the original pipetting video is a video obtained by the camera capturing the pipetting injection operation; the pipette has a first label, a second label and a third label installed vertically side by side from top to bottom; the first label and the second label are fixed to the external clamp of the pipette, and the third label is fixed to the pipette body; the original pipetting video includes several original image video frames; For each of the original image / video frames, the original image / video frame is preprocessed to obtain a preprocessed image; Connectivity detection is performed on the preprocessed image to obtain the position coordinates of the first center, the second center, and the third center, as well as the first area, the second area, and the third area in each preprocessed image; the first area, the second area, and the third area are the areas of the first label, the second label, and the third label in the preprocessed image, respectively; the first center, the second center, and the third center are the centroids of the first area, the second area, and the third area, respectively. The measured distance corresponding to each preprocessed image is determined based on the first pixel distance, the second pixel distance, the first area, the second area, and the third area; the first pixel distance is the pixel length between the second center and the third center on the camera plane; the second pixel distance is the pixel length between the first center and the second center on the camera plane. The injection segment is selected based on the measured distance corresponding to all the preprocessed images, specifically including: Determine the frame with the minimum distance; the frame with the minimum distance is the event frame corresponding to the point where the minimum distance is measured during pipetting. Using the frame with the minimum distance as the initial frame, traverse leftwards from the frame with the minimum distance, and determine whether the measured distance corresponding to the current time frame satisfies the first preset condition to obtain the first judgment result; the first preset condition is expressed as follows: ;in , Don't set time frame equal to , The distance measured at that time; The selected threshold; , These represent the current time frame, the frame with the maximum distance, and the frame with the minimum distance, respectively. If the first judgment result is yes, then replace the current time frame with the previous time frame and return to the step of "judging whether the measured distance corresponding to the current time frame meets the first set condition". If the first judgment result is negative, then it is determined whether the second set condition is met, and the second judgment result is obtained; the second set condition is that the measured distance corresponding to the previous time frame of the current time frame is greater than the measured distance corresponding to the current time frame. If the second judgment result is yes, then replace the current time frame with the previous time frame and return to the "judgment whether the second set condition is met" step; If the second judgment result is negative, then the current time frame is determined as the starting frame of the injection segment; Starting from the beginning frame of the injection segment, the system traverses to the right from the beginning frame of the injection segment, and determines whether the measured distance corresponding to the current time frame meets the third set condition, thus obtaining the third judgment result. If the third judgment result is yes, then the next time frame of the current time frame will replace the current time frame, and the process will return to the step of "judging whether the measured distance corresponding to the current time frame meets the third set condition". If the third judgment result is negative, then it is judged whether the fourth setting condition is met, and the fourth judgment result is obtained; the fourth setting condition is that the measured distance corresponding to the next time frame of the current time frame is less than the measured distance corresponding to the current time frame. If the fourth judgment result is yes, then the next time frame of the current time frame will replace the current time frame, and the process will return to the "Judgment whether the fourth set condition is met" step. If the result of the fourth judgment is negative, then the current time frame is determined as the end frame of the injection segment; the start frame of the injection segment, the end frame of the injection segment, and all time frames in between constitute the injection segment. For each injection image frame in the injection segment, the final intra-frame injection volume corresponding to the injection image frame is calculated based on the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment; the injection segment consists of several injection image frames. The final intra-frame injection volume corresponding to the injection image frame is converted into motor control parameters; the motor control parameters include the pre-frequency division coefficient and the number of motor running pulses.

2. The pipette injection speed detection method according to claim 1, characterized in that, The original image video frames are preprocessed to obtain a preprocessed image, specifically including: The original image and video frames are converted from the RGB color space to the HSV color space to obtain HSV image and video frames; The HSV image video frames are binarized to obtain a binarized image; A closing operation is performed on the binarized image to obtain the image after the closing operation; the image after the closing operation is the preprocessed image.

3. The pipette injection speed detection method according to claim 1, characterized in that, The formula for calculating the measured distance corresponding to the preprocessed image is as follows: ； in, This represents the measured distance corresponding to the preprocessed image. The distance is the second pixel. The distance is the first pixel. This is the first error coefficient caused by depth of field. This is the second error coefficient caused by depth of field. For the first area, For the second area, This is the third area.

4. The pipette injection speed detection method according to claim 1, characterized in that, Based on the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment, the final intra-frame injection volume corresponding to the injection image frame is calculated, specifically including: Based on the measured distance of the injection image frame, the measured distance of the injection start frame of the injection segment, the measured distance when the pipette plunger is fully pressed, and the maximum injection volume of the pipette, the original injected volume corresponding to the injection image frame is obtained. The original injected volume corresponding to the injection image frame is optimized to obtain the first optimized injected volume corresponding to the injection image frame. The outlier removal process is performed on the first-optimized injected volume corresponding to the injected image frame to obtain the second-optimized injected volume corresponding to the injected image frame; the second-optimized injected volume is the final intra-frame injection volume.

5. The pipette injection speed detection method according to claim 1, characterized in that, The formula for calculating the number of motor operation pulses is as follows: ； ； ； In the formula, For the first The number of motor pulses that execute a frame-injected image frame. This represents the number of pulses the motor needs to run. For the first The final intra-frame injection volume corresponding to the frame-injected image frame. The volume of each pulse injected by the transplanted pipette. The remaining decimal part; The formula for calculating the prescaler coefficient is as follows: ； In the formula, For the first Pre-frequency division coefficients of the frame-injected image frame. This is the value for automatic reloading. For clock frequency, This represents the actual time per frame of the original injection video.

6. A pipette injection speed detection device, characterized in that, The pipette injection speed detection device includes: The original pipetting video acquisition module is used to: acquire original pipetting video; the original pipetting video is a video obtained by a camera capturing the injection operation of a pipette; a first label, a second label, and a third label are vertically mounted side by side from top to bottom on the pipette; the first label and the second label are fixed on the external clamp of the pipette, and the third label is fixed on the pipette body; the original pipetting video includes several original image video frames; The preprocessing module is used to: preprocess each of the original image / video frames to obtain a preprocessed image; The connected component detection module is used to: perform connected component detection on the preprocessed image to obtain the position coordinates of the first center, the second center, and the third center, as well as the first area, the second area, and the third area in each preprocessed image; the first area, the second area, and the third area are respectively the areas of the first label, the second label, and the third label in the preprocessed image; the first center, the second center, and the third center are respectively the centroids of the first area, the second area, and the third area; The distance calculation module is used to: determine the measured distance corresponding to each preprocessed image based on the first pixel distance, the second pixel distance, the first area, the second area, and the third area; the first pixel distance is the pixel length between the second center and the third center on the camera plane; the second pixel distance is the pixel length between the first center and the second center on the camera plane; The injection segment filtering module is used to filter injection segments based on the measured distances corresponding to all the preprocessed images, specifically including: Determine the frame with the minimum distance; the frame with the minimum distance is the event frame corresponding to the point where the minimum distance is measured during pipetting. Using the frame with the minimum distance as the initial frame, traverse leftwards from the frame with the minimum distance, and determine whether the measured distance corresponding to the current time frame satisfies the first preset condition to obtain the first judgment result; the first preset condition is expressed as follows: ;in , Don't set time frame equal to , The distance measured at that time; The selected threshold; , These represent the current time frame, the frame with the maximum distance, and the frame with the minimum distance, respectively. If the first judgment result is yes, then replace the current time frame with the previous time frame and return to the step of "judging whether the measured distance corresponding to the current time frame meets the first set condition". If the first judgment result is negative, then it is determined whether the second set condition is met, and the second judgment result is obtained; the second set condition is that the measured distance corresponding to the previous time frame of the current time frame is greater than the measured distance corresponding to the current time frame. If the second judgment result is yes, then replace the current time frame with the previous time frame and return to the "judgment whether the second set condition is met" step; If the second judgment result is negative, then the current time frame is determined as the starting frame of the injection segment; Starting from the beginning frame of the injection segment, the system traverses to the right from the beginning frame of the injection segment, and determines whether the measured distance corresponding to the current time frame meets the third set condition, thus obtaining the third judgment result. If the third judgment result is yes, then the next time frame of the current time frame will replace the current time frame, and the process will return to the step of "judging whether the measured distance corresponding to the current time frame meets the third set condition". If the third judgment result is negative, then it is judged whether the fourth setting condition is met, and the fourth judgment result is obtained; the fourth setting condition is that the measured distance corresponding to the next time frame of the current time frame is less than the measured distance corresponding to the current time frame. If the fourth judgment result is yes, then the next time frame of the current time frame will replace the current time frame, and the process will return to the "Judgment whether the fourth set condition is met" step. If the result of the fourth judgment is negative, then the current time frame is determined as the end frame of the injection segment; the start frame of the injection segment, the end frame of the injection segment, and all time frames in between constitute the injection segment. The final intra-frame injection volume calculation module is used to: for each injection image frame in the injection segment, calculate the final intra-frame injection volume corresponding to the injection image frame based on the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment; the injection segment consists of several injection image frames; The motor control parameter conversion module is used to convert the final intra-frame injection volume corresponding to the injection image frame into motor control parameters; the motor control parameters include the pre-frequency division coefficient and the number of motor running pulses.

7. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the pipette injection speed detection method according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the pipette injection speed detection method according to any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the pipette injection speed detection method according to any one of claims 1-5.

Images