Pipette injection speed detection method, device, equipment, medium and product

Abstract

The invention discloses a pipette injection speed detection method, device and equipment, a medium and a product, and relates to the technical field of trace liquid treatment.The method comprises the steps that an original pipetting video is obtained, each original image video frame of the original pipetting video is preprocessed, connected domain detection is carried out, and the original image video frame of the original pipetting video is obtained; the central position and the area information of the label can be obtained, the measured distance corresponding to the preprocessed image is calculated through the central position and the area information of the label, and then the injection paragraph is screened according to the measured distance corresponding to all the preprocessed images. The final intra-frame injection volume of each injection image frame is calculated according to the measured distance of the injection image frames in the injection paragraph, the final intra-frame injection volume is converted into a motor control parameter, and the injection speed is accurately detected and automatic transplantation is realized through real-time image capture and processing in the manual pipetting process.

Description

Technical Field

[0001] The present application relates to the field of trace liquid processing technology, and in particular to a method, device, equipment, medium and product for detecting injection speed of a pipette based on machine vision. Background Art

[0002] With the development of modern biotechnology, synthetic biology and chemical biology, the process processing of large numbers of samples has become more and more common, and these fields usually require the extraction and transfer of trace amounts of liquid, that is, trace liquid handling technology (micropipetting technology). Taking the field of molecular diagnosis as an example, the processing of various biological reagents and biological samples is a basic and frequent experimental operation, which requires high-precision liquid processing of biological reagents or biological samples with different characteristics. Compared with manual pipetting operations, automatic pipetting devices have the advantages of high throughput, good stability, and reduced manual pressure. With the further improvement of the requirements for sample processing throughput and the reliability of test results in biochemical experiments, fully automatic pipetting devices based on trace liquid handling technology have become an inevitable trend in the development of biochemical laboratories.

[0003] In some occasions where pipetting devices are used, due to the different conditions such as the properties of the liquid, the injection environment, and the injection microstructure, such as the organ chip gel injection process, the injection speed requirements are high, and different or even variable injection speeds are required. Although many automatic pipetting devices are equipped with speed programmable functions, allowing users to customize the speed of the liquid handling process, for experienced operators, although a high success rate can be guaranteed during manual injection, it is difficult to quantify the operating experience and accurately describe the speed size and change process during manual injection when using automatic pipetting devices. And due to the complexity and variability of the injection scene, whether it is through calculation or simulation to obtain the operating parameters, or through long-term experiments to obtain the available parameters, there are extremely high economic and time costs. If the automatic pipetting device can directly obtain the speed parameters during manual pipetting, it can achieve rapid automated injection in complex scenarios.

[0004] Although many fully automatic pipetting devices have been developed to support customized pipetting parameters to cope with complex pipetting scenarios, the acquisition of pipetting parameters mainly comes from operating experience and a large number of operational tests, and there is no simple and effective quantification method. Therefore, how to enable the fully automatic pipetting device to quickly obtain the correct programmable pipetting parameters is also a problem to be solved. Summary of the invention

[0005] The purpose of this application is to provide a method, device, equipment, medium and product for detecting the injection speed of a pipette, which can enable an automatic pipetting device to learn the manual speed and realize rapid automated injection in complex scenarios.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a method for detecting injection speed of a pipette, comprising:

[0008] Obtaining an original pipetting video; the original pipetting video is a video obtained by shooting a pipette injection operation with a camera; the pipette is vertically and parallelly mounted with a first label, a second label, and a third label from top to bottom; the first label and the second label are fixed on an external fixture of the pipette, and the third label is fixed on the pipette body; the original pipetting video includes a plurality of original image video frames;

[0009] For each of the original image video frames, preprocessing the original image video frame to obtain a preprocessed image;

[0010] Performing connected domain detection on the preprocessed images to obtain position coordinates of a first center, a second center, and a third center in each of the preprocessed images and a first area, a second area, and a third area; the first area, the second area, and the third area are respectively the areas of the first label, the second label, and the third label in the preprocessed images; the first center, the second center, and the third center are respectively the centroids of the first area, the second area, and the third area;

[0011] Determine the measured distance corresponding to each of the preprocessed images according to the first pixel distance, the second pixel distance, the first area, the second area and the third area; the first pixel distance is the pixel length between the second center and the third center on the camera plane; the second pixel distance is the pixel length between the first center and the second center on the camera plane;

[0012] Filter the injection sections according to the measured distances corresponding to all the preprocessed images;

[0013] For each injection image frame in the injection segment, a final intra-frame injection volume corresponding to the injection image frame is calculated according to the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment; the injection segment is composed of a plurality of injection image frames;

[0014] The final intra-frame injection volume corresponding to the injection image frame is converted into motor control parameters; the motor control parameters include a pre-frequency division coefficient and the number of motor operation pulses.

[0015] In a second aspect, the present application provides a device for detecting injection speed of a pipette, comprising:

[0016] The original pipetting video acquisition module is used to: acquire the original pipetting video; the original pipetting video is a video obtained by shooting the injection operation of the pipette gun by a camera; the pipette gun is vertically and parallelly installed with a first label, a second label and a third label from top to bottom; the first label and the second label are fixed on the external clamp of the pipette gun, and the third label is fixed on the gun body of the pipette gun; the original pipetting video includes a plurality of original image video frames;

[0017] A preprocessing module is used to: for each of the original image video frames, preprocess the original image video frame to obtain a preprocessed image;

[0018] A connected domain detection module is used to: perform connected domain detection on the preprocessed images to obtain the position coordinates of the first center, the second center and the third center in each of the preprocessed images and the first area, the second area and the third area; the first area, the second area and the third area are respectively the areas of the first label, the second label and the third label in the preprocessed images; the first center, the second center and the third center are respectively the centroids of the first area, the second area and the third area;

[0019] A measured distance calculation module, used to: determine the measured distance corresponding to each of the preprocessed images according to a first pixel distance, a second pixel distance, a first area, a second area, and a third area; the first pixel distance is a pixel length between the second center and the third center on the camera plane; the second pixel distance is a pixel length between the first center and the second center on the camera plane;

[0020] An injection segment screening module is used to: screen the injection segments according to the measured distances corresponding to all the pre-processed images;

[0021] The final frame injection volume calculation module is used to: for each injection image frame in the injection segment, calculate the final frame injection volume corresponding to the injection image frame according to the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment; the injection segment is composed of a plurality of injection image frames;

[0022] The motor control parameter conversion module is used to: convert the injection volume in the final frame corresponding to the injection image frame into the motor control parameter; the motor control parameter includes a pre-frequency division coefficient and the number of motor operation pulses.

[0023] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned pipette injection speed detection method.

[0024] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned pipette injection speed detection method when executed by a processor.

[0025] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for detecting the injection speed of a pipette.

[0026] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0027] The present application provides a method, device, equipment, medium and product for detecting the injection speed of a pipette. Through a speed acquisition method based on machine vision, each original image video frame of the original pipetting video is preprocessed and connected domain detection is performed to obtain the movement information of the label (the movement information of the label includes the position coordinates of the first center, the second center and the third center in the preprocessed image and the first area, the second area and the third area). The measured distance corresponding to the preprocessed image is calculated through the movement information of the label, and then the injection segments are screened according to the measured distances corresponding to all the preprocessed images. The final intra-frame injection volume of each injection image frame is calculated according to the measured distances of the injection image frames in the injection segments, and the final intra-frame injection volume is converted into a motor control parameter. By capturing and processing real-time images of the manual pipetting process, the injection speed can be accurately detected and automated transplantation can be realized, which is of great significance to improving the adaptability of the automatic pipetting system. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 This is an application environment diagram of a method for detecting injection speed of a pipette in one embodiment of the present application;

[0030] Figure 2 A schematic flow chart of a method for detecting injection speed of a pipette provided in one embodiment of the present application;

[0031] Figure 3 This is an overall flow chart of a method for detecting the injection speed of a pipette provided in one embodiment of the present application;

[0032] Figure 4 A schematic diagram of an image processing process provided by an embodiment of the present application;

[0033] Figure 5 A schematic diagram of the distribution of tags under the influence of depth of field provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of a depth of field compensation algorithm provided in one embodiment of the present application;

[0035] Figure 7 A schematic diagram of a measured distance-frame chart provided in an embodiment of the present application;

[0036] FIG8(a) is a flowchart of searching for a start frame of an injection segment according to an embodiment of the present application; FIG8(b) is a flowchart of searching for an end frame of an injection segment according to an embodiment of the present application;

[0037] Fig. 9 A schematic diagram of the original and optimized injection volume-frame curves of the injection segment provided in one embodiment of the present application;

[0038] FIG. 10( a ) is a graph showing the injection volume per frame after the primary optimization according to an embodiment of the present application; FIG. 10( b ) is a graph showing the injection volume per frame after the secondary optimization according to an embodiment of the present application;

[0039] Fig.11 A schematic diagram of functional modules of a device for detecting injection speed of a pipette provided in one embodiment of the present application;

[0040] Fig.12 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0043] The method for detecting the injection speed of a pipette provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the original pipetting video to the server 104. After the server 104 receives the original pipetting video, it pre-processes each original image video frame of the original pipetting video and performs a connected domain detection to obtain the center position and area information of the label. The measured distance corresponding to the pre-processed image is calculated by the center position and area information of the label, and then the injection segment is screened according to the measured distance corresponding to all pre-processed images. The final intra-frame injection volume of each injection image frame is calculated according to the measured distance of the injection image frame in the injection segment, and the final intra-frame injection volume is converted into a motor control parameter. The server 104 can feedback the obtained motor control parameters to the terminal 102. In addition, in some embodiments, the pipette injection speed detection method can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly perform the pipette injection speed detection on the original pipetting video, or the server 104 can obtain the original pipetting video from the data storage system and perform the pipette injection speed detection on the original pipetting video.

[0044] The terminal 102 may be, but is not limited to, various desktop computers and laptop computers. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0045] In an exemplary embodiment, Figure 2 and Figure 3 As shown, a method for detecting the injection speed of a pipette is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 201 to 207. Among them:

[0046] Step 201: Obtain an original pipetting video; the original pipetting video is a video obtained by shooting a pipette injection operation with a camera; the pipette is vertically and side by side with a first label, a second label and a third label from top to bottom; the first label and the second label are fixed on an external clamp of the pipette, and the third label is fixed on the pipette body; the original pipetting video includes a plurality of original image video frames.

[0047] Step 202: For each of the original image video frames, preprocess the original image video frame to obtain a preprocessed image.

[0048] Step 203: Perform connected domain detection on the preprocessed images to obtain the position coordinates of the first center, the second center and the third center in each of the preprocessed images and the first area, the second area and the third area; the first area, the second area and the third area are respectively the areas of the first label, the second label and the third label in the preprocessed images; the first center, the second center and the third center are respectively the centroids of the first area, the second area and the third area.

[0049] Step 204: Determine the measured distance corresponding to each of the preprocessed images according to the first pixel distance, the second pixel distance, the first area, the second area and the third area; the first pixel distance is the pixel length between the second center and the third center on the camera plane; the second pixel distance is the pixel length between the first center and the second center on the camera plane.

[0050] Step 205: Filter the injection sections according to the measured distances corresponding to all the preprocessed images.

[0051] Step 206: for each injection image frame in the injection segment, calculate the final intra-frame injection volume corresponding to the injection image frame according to the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment; an injection segment is composed of a plurality of injection image frames.

[0052] Step 207: converting the final intra-frame injection volume corresponding to the injection image frame into motor control parameters; the motor control parameters include a pre-frequency division coefficient and the number of motor operation pulses.

[0053] Implement the above-mentioned steps 201 to 207, and accurately detect the injection speed and realize automated transplantation by capturing and processing the real-time images of the manual pipetting process. In addition, compared with the common grating rulers and inertial measurement units on the market, the cost is relatively low, the noise is low, and the error is small. Speed ​​acquisition based on machine vision is to first obtain the original pipetting video through equipment such as cameras, and then perform relevant processing on the image of each frame of the original pipetting video in turn, find the feature points, measure the actual distance after calibration, and then convert the measured distance into a method of speed. The pipette injection speed detection method provided in the present application has no special requirements for the measuring object and the measured object, and is suitable for occasions where traditional contact measurement cannot be implemented. The sampling frequency is mainly determined by the camera, generally 30-1000Hz, and the accuracy is mainly determined based on the distance measurement algorithm, which can reach sub-millimeter level. Due to the low hardware requirements, it has the advantage of minimizing the interference of the injection process.

[0054] Use a smartphone to capture images and videos of the pipette operation process. During the shooting process, use a mobile phone holder to fix the mobile phone camera to ensure the stability of the shooting. Feature labels are installed on the piston rod of the pipette for position calibration in the image. These labels are colored circular marks installed on the outer clamp of the pipette to ensure that they are not blocked during the pipetting operation. This application distinguishes based on color and filters out the color corresponding to the label in the image. Since the equipment and instruments in each laboratory are different, users need to choose the color that does not exist or is the least in the background as the label color.

[0055] After reading and opening the original pipetting video through OpenCV, the next frame of the original pipetting video is obtained in a loop, and label detection and distance calculation are performed. After the entire original pipetting video is processed, multi-step speed conversion is performed and finally converted into motor control parameter output.

[0056] In another exemplary embodiment of the present application, in the above step 202, the original image video frame is preprocessed to obtain a preprocessed image, which can be replaced by the following steps 301 to 303.

[0057] Step 301: converting the original image video frame from the RGB color space to the HSV color space to obtain an HSV image video frame;

[0058] Step 302: binarizing the HSV image video frame to obtain a binary image;

[0059] Step 303: performing a closing operation on the binary image to obtain an image processed by the closing operation; the image processed by the closing operation is the pre-processed image.

[0060] The original image video frame is processed in RGB format by default. In RGB format, it is difficult to accurately select the threshold range of a certain color, so the original image video frame is converted to HSV space for processing. HSV space describes color by three dimensions: color, depth, and brightness. It is not only more in line with human visual perception, but also easier to distinguish colors. For any coordinate point in the image, its RGB color space is (R, G, B), and the HSV color space is (H, S, V). The calculation steps for converting the image from RGB color space to HSV color space are as follows. First, the R, G, and B values ​​need to be converted to between 0 and 1. The conversion formula is shown below.

[0061]

[0062] Then calculate the values ​​of H, S, and V respectively. The calculation formulas are shown below.

[0063]

[0064] Since Opencv needs to visualize HSV images, each value needs to be converted to between 0 and 255. The converted H', S' and V' are expressed as follows.

[0065]

[0066] After setting the color filter, this frame of HSV space image is converted into a binary image at the same time, and the pixels within the label range are changed to white, and the others are changed to black. The binary image is as follows Figure 4 As shown in (a) in the figure, basically only the color corresponding to the label is retained.

[0067] The remaining noise is removed in the subsequent processing. The image is first dilated using the closed operation, and then eroded. The calculation process is shown in the following formula.

[0068]

[0069] Where T is the input image, Y is the structural element, and z is a point in the image.

[0070] Closing is used to remove noise on the edges of objects, fill holes inside objects, and smooth the boundaries of objects, which helps connect broken parts of objects and maintain their overall shape. Figure 4 As shown in (b) in .

[0071] After the first two steps, the area of ​​the remaining noise is generally small. By traversing the size of all connected domains, we can directly determine which areas are labels. Using OpenCv's connected domain detection function (ConnectedComponents), we can detect all connected domains in the image and output the area, width, center of mass, and upper left corner vertex coordinates of the connected domain. After calling this function, you only need to traverse all connected domain information and only keep the subscripts of the three areas with the largest area. At the same time, read the coordinates of the center of mass according to the subscript, and arrange them in sequence according to the size of the y coordinate, and read other relevant information separately. Figure 4 As shown in (b), the center positions A”, B” and C” of the first center, the second center and the third center of the three labels and the first area S are obtained respectively. A” , the second area S B” and the third area S C” The accurate detection of tags provides the basis for the subsequent distance and speed calculation.

[0072] During injection, the two upper labels will move with the external fixture and the push rod of the pipette. Therefore, the distance the push rod is pressed during pipette injection is equal to the distance the lower and middle labels change. However, since only pixel distances can be obtained from frame images, a constant standard distance is still required as a ruler. Since the distance between the middle and upper labels is always constant, it can be used as a ruler for calculation. Since the actual distance between the second center B and the third center C is l BC The change of l is the same as the change of the pipette push rod length, that is, only l BC The pipetting speed can be measured and the calculation l can be obtained BC The formula is as shown below.

[0073]

[0074] Among them, l AB , l BC is the actual length between the corresponding label centroids, l AB is the actual distance between the first center A and the second center B, l A”B” , l B”C” is the pixel length between the centroids of the tags on the camera plane, l A”B” is the second pixel distance, l B”C” is the first pixel distance.

[0075] Because l A”B” and l B”C” can be obtained from previous image processing, and l AB The length is fixed and there is no need to obtain l BC The actual length, we only need to get its changing trend, then we can let L d = l BV / l AB , we can get the following formula.

[0076]

[0077] However, since the depth of field affects the distance calculation, this application calculates the distance by the centroid and area of ​​the detected tags, and adds a correction factor based on the depth of field to compensate. The distribution of the three tags under the influence of the depth of field is as follows: Figure 5 shown.

[0078] Since the actual sizes of the three labels are the same, and the actual length represented by any pixel in the preprocessed image is inversely proportional to the depth of field, that is, the square root of the actual area represented by each pixel is inversely proportional to its depth of field, the following two formulas can be derived.

[0079]

[0080] Among them, l A”r , l B”r , l C”r are the actual lengths represented by pixel points A", B", and C" on line segment A"C; d A , d B , d C are the distances from the center of mass points A, B, and C to the camera plane respectively.

[0081] According to formula (7), the following formula can be obtained.

[0082]

[0083] Where x is any point on line segment AC, x' is the corresponding projection point of point x on the camera plane, and d x is the distance from point x to the camera surface, l x”r is the actual length represented by point x” on line segment A”C”.

[0084] Define a = l A”r / l B”r , b=l C”r / l B”r Combined with formula (8), we can get the principle diagram of depth of field compensation algorithm, as shown in Figure 6 shown.

[0085] From the perspective of integration, the l corresponding to all points x on line segment AB x”r Add them together to get the actual distance l between the first center A and the second center B AB Therefore, the deviation relationship between the original algorithm and the optimized algorithm in the length calculation process can be obtained, and thus the first error coefficient k caused by the depth of field on the AB line segment can be calculated. AB The calculation formula is shown below.

[0086]

[0087] Similarly, the second error coefficient k caused by the depth of field on the BC segment can be obtained: BC The calculation formula is shown below.

[0088]

[0089] Introducing the error coefficient into formula (6), we can get the corrected measured distance L d , the corrected measured distance is the measured distance corresponding to the preprocessed image, and its calculation formula is shown in the following formula.

[0090]

[0091] Calculate the measured distance L corresponding to the preprocessed image dAfter that, the processing of this frame is completed, the measured distance is saved in a dynamic array, and the control program receives the pre-processed image of the next frame, repeating the above two processing steps. When the program does not receive the next frame of pre-processed image, it means that the entire video has been processed. At this time, the next step of speed conversion processing is entered.

[0092] After the above image processing, we get a measured distance L that stores the pre-processed image of each frame in the video. d The time frame is used as the horizontal axis and the measured distance of each frame of the preprocessed image is used as the vertical axis to draw a graph. The schematic result of the measured distance-frame graph is as follows Figure 7 shown.

[0093] Figure 7 It contains a typical pipetting process, which is divided into five processes according to the state of the push rod: stop, inject, stop, rebound and stop. In the actual injection process, due to the large hysteresis caused by the air plunger and the viscosity of the injected liquid, the push rod stops at a certain position, and the liquid in the pipette gun will be delayed for a period of time. At this time, if the pipette gun is pulled out from the injected object, part of the liquid will be directly sprayed out from the gun head, and the liquid in the injected object will flow back, which will directly lead to the failure of this injection. Since the time of the stop segment can be customized through the program later, this application does not identify the stop time, but simply collects the process of the injection segment. In addition, the injection segment contains a stop segment, and there is no injection at this time, so the actual distance of the push rod should be unchanged, but because the above-mentioned distance measurement algorithm has certain errors, the detected distance will have certain fluctuations, so this problem is taken into consideration and optimized.

[0094] First, the injection segment should be screened from the process, such as Figure 7 As shown, s The starting frame of the injection paragraph, i e The end frame of the injection paragraph, i m is the maximum distance frame, that is, the event frame corresponding to the maximum distance point measured during the entire pipetting process, i n The minimum distance frame is the event frame corresponding to the minimum distance point measured during the entire pipetting process. It is necessary to find the i of the injection segment. s and i e Two points.

[0095] In another exemplary embodiment of the present application, the above step 205 may include the following steps 401 to 411.

[0096] Step 401: determine the minimum distance frame; the minimum distance frame is the event frame corresponding to the minimum distance point measured during the pipetting process.

[0097] Step 402: Taking the minimum distance frame as the initial frame, traverse from the minimum distance frame to the left to determine whether the measured distance corresponding to the current time frame meets the first set condition, and obtain a first determination result; the first set condition is represented by s i m ×(1-k a );where s i ,s m The time frame is equal to i, i m The measured distance at time k a is the selected threshold; i, i m Respectively represent the current time frame and the maximum distance frame.

[0098] Step 403: If the first judgment result is yes, the previous time frame of the current time frame replaces the current time frame, and the process returns to the step of "judging whether the measured distance corresponding to the current time frame meets the first set condition".

[0099] Step 404: If the first judgment result is no, determine whether the second set condition is met to obtain a second judgment result; the second set condition is that the measured distance corresponding to the previous time frame of the current time frame is greater than the measured distance corresponding to the current time frame.

[0100] Step 405: If the second judgment result is yes, the previous time frame of the current time frame replaces the current time frame, and returns to the step of "judging whether the second set condition is met".

[0101] Step 406: If the second judgment result is no, the current time frame is determined as the injection segment start frame.

[0102] Step 407: Taking the injection segment start frame as the initial frame, traverse rightward from the injection segment start frame to determine whether the measured distance corresponding to the current time frame meets the third setting condition, and obtain a third determination result. The third setting condition is s i >s n ×(1+k a ), s n For time frame equal to i n The measured distance at .

[0103] Step 408: If the third judgment result is yes, the next time frame of the current time frame replaces the current time frame, and the process returns to the step of "judging whether the measured distance corresponding to the current time frame meets the third set condition".

[0104] Step 409: If the third judgment result is no, determine whether the fourth setting condition is met, and obtain the fourth judgment result; the fourth setting condition is that the measured distance corresponding to the next time frame of the current time frame is less than the measured distance corresponding to the current time frame. The fourth setting condition can be expressed as s i+1 <s​i .

[0105] Step 410: If the fourth judgment result is yes, the next time frame of the current time frame replaces the current time frame, and returns to the step of "judging whether the fourth setting condition is met".

[0106] Step 411: If the fourth determination result is no, the current time frame is determined as the injection segment end frame; the injection segment start frame, the injection segment end frame and all time frames therebetween constitute the injection segment.

[0107] Since the maximum distance may appear at the beginning and the end, and the minimum distance point must appear after the injection is completed, that is, there is i s n ,i e ≤i n Then we can first find the frame i with the minimum distance n , and traverse to the left to find the injection paragraph starting frame i s , and then traverse to the right to get the injection paragraph end frame i e The specific process is shown in Figure 8(a). First, we start from the frame with the smallest distance i. n Start traversing to the left until s is not satisfied i m ×(1-k a ) until. Here k a =0.02, mainly determined by the error rate of the single-frame distance measurement measured later. At this time, the current time frame i is close to the injection segment end frame i e , but it may be faster than injecting the end frame of the paragraph i e big, then as long as there is s i-1 >s i , then continue to traverse to the left, otherwise it is considered that the traversal has reached the injection paragraph starting frame i s The process of finding the end frame of the injection segment is shown in FIG8( b ). Since it is similar to the start frame of the injection segment, it will not be further described here.

[0108] The above step 206 specifically includes the following steps 501 to 503.

[0109] Step 501: Obtain the original injected volume corresponding to the injection image frame according to the measured distance of the injection image frame, the measured distance of the injection start frame of the injection segment, the measured distance when the pipette push rod is fully pressed, and the maximum injection volume of the pipette.

[0110] Step 502: Optimizing the original injected volume corresponding to the injection image frame to obtain a first optimized injected volume corresponding to the injection image frame.

[0111] ​​Step 503: performing outlier removal processing on the primary optimized injected volume corresponding to the injection image frame to obtain the secondary optimized injected volume corresponding to the injection image frame; the secondary optimized injected volume is the final intra-frame injection volume.

[0112] The original injected volume a corresponding to the i-th injection image frame i The calculation formula is shown below.

[0113]

[0114] Among them, s s is the detected injection segment start frame i s The corresponding measured distance; s min This is the distance measured when the pipette push rod is fully pressed. It needs to be photographed and calibrated in advance. f The maximum injection volume of the pipette, in μL.

[0115] According to the original injected volume a corresponding to all injected image frames i Can be drawn as Fig. 9 The original injected volume-frame curve is shown.

[0116] However, the injection process should only have two states: injection and stop, that is, the original injected volume a i The value will increase or remain constant over time, but Fig. 9 The original injected volume-frame curve fluctuates up and down, and it should be optimized to obtain an optimized injected volume. The optimization formula is shown below.

[0117]

[0118] In the formula, a i ' is the optimized injected volume of the i-th injection image frame, a i-1 is the original injected volume of the i-1th injection image frame, a i+1 is the original injected volume of the i+1th injection image frame.

[0119] After processing, the data is Fig. 9 As shown in the initial optimization curve, after treatment, the injection volume can only remain unchanged or increase with time, but at the same time, the optimized injection volume at a certain moment will be larger than the original one, so this application continues to optimize. i 'Converted into injection volume v per frame time i , that is, the relationship between the injection volume within the frame and the frame. The conversion formula is shown below.

[0120] v i =a i+1'-a i ' (14).

[0121] In the formula, a i+1 ' is the intra-frame injection volume of the i+1th injection image frame.

[0122] After the first optimization process, the inter-frame injection volume-frame image shown in Figure 10(a) can be obtained. It can be seen that there are three main abnormal values: abnormal value I is that the injection speed is 0 in some sections that should be injected; abnormal value II is that a certain injection speed also appears during the stop process; abnormal value III is that the injection speed at certain times is much greater than that at other points.

[0123] These three outliers are caused by the recognition error and the above preliminary optimization. For outliers I and III, they can usually be eliminated by mean filtering, but mean filtering will increase the interval of outlier II, so outlier II should be processed first. The processing of outlier II is mainly to identify the entire velocity array, identify all continuous intervals with positive velocities, traverse each interval, and if its length is less than 5 and the total injection volume is less than 2% of the whole, it is determined to be an abnormal interval caused by algorithm error. All velocities in this interval are set to 0, and the sum of velocities is retained. It is added to the last frame of the previous interval and the first frame of the next interval at a ratio of 1:1, respectively, to ensure that outlier I will not be affected while removing outlier II.

[0124] Then, the processed curve is subjected to mean filtering. Since it is necessary to ensure that the sum of all injection speeds (i.e., the total injection volume) remains unchanged after mean filtering, it is necessary to perform additional processing on the edge frames to obtain the secondary optimized injected volume. The secondary optimization processing formula is shown in the following formula.

[0125]

[0126] Among them, v i ' is the final intra-frame injection volume corresponding to the i-th injection image frame, in μL; N is the number of mean filter segments, and in this embodiment, N can be 5; Z v The total number of frames in the injection segment.

[0127] After the above two optimization steps, the intra-frame injection volume-frame graph after secondary optimization is drawn, as shown in Figure 10(b). After processing, all three outliers are deleted, and the speed change between adjacent frames is smaller after mean filtering. At the same time, it is converted back to the total injection volume-frame image, as shown in Figure 10(b). Fig. 9 As shown in the secondary optimization curve in , the final curve is more consistent with the measured result, and the curve is smoother and more suitable for the stable operation of the motor.

[0128] At this point, the optimized inter-frame injection speed is obtained. The speed will be converted to ensure that it can be transplanted to the automatic pipetting device at the same speed. After the parameter transplantation, the injection speed in each frame is the same, and the speed between frames is different. The control strategy of the automatic pipetting device is to control the speed by changing the pre-scaling coefficient psc, and control the number of pulses running at this speed in the timer interrupt function. The final output drive parameter is the psc corresponding to any time frame i i And the number of running pulses n i .

[0129] First, the injection speed v of each frame i 'Converted into the number of pulses p that the motor needs to move i , the calculation formula is shown as follows.

[0130]

[0131] Among them, V s Pipette the volume injected for each pulse of transplantation.

[0132] The number of pulses calculated by the above formula has decimals. Since the stepper motor can only move an integer number of pulses, the number of pulses moving in each frame also needs to be processed. The processing strategy is that only an integer number of pulses are run in the injection image frame, and the remaining decimal part is added to the next injection image frame for processing. The processing formula is shown below.

[0133]

[0134] Among them, n i is the actual number of motion pulses corresponding to the i-th injection image frame, n e The remaining decimal part will be added to the next injection image frame for calculation; / 1 means rounding; %1 means dividing by 1 and taking the remainder.

[0135] At the same time, after processing, there will be n i ≤p i , that is, the motor moves to the same speed. i After a pulse, the time of one frame has not yet arrived, so the remaining time is also added to the subsequent injection image frame. The remaining time of the current injection image frame is t e The calculation formula is shown below.

[0136]

[0137] Among them, t f is the actual time represented by each frame of the original injection video. Here the video is shot at 60fps, that is, t f It is 16.67ms.

[0138] Then according to the pulse frequency f p The calculation formula of , we can get the following formula.

[0139]

[0140] Among them, arr is the automatic reload value, f clk is the clock frequency.

[0141] After conversion, the pre-scaling coefficient psc can be obtained i The calculation formula is shown below.

[0142]

[0143] In the formula, psc i The pre-scaling coefficient of the image frame is injected into the i-th frame.

[0144] When the injection stops, psc i = 0, the number of pulses becomes the time required to pause, in ms. In order to reduce the total length of the parameter, the intervals with the same speed are merged.

[0145] The present application also provides an application scenario, which applies the above-mentioned pipette injection speed detection method. Specifically: the pipette injection speed detection method provided in this embodiment can be applied in the injection speed transplantation scenario. The injection speed transplantation scenario includes a video acquisition link, an injection speed detection link and an injection speed transplantation link; the original pipetting video enters the injection speed detection link from the video acquisition link, obtains the corresponding motor control parameters, and enters the downstream injection speed transplantation link. The pipette injection speed detection method provided in this embodiment belongs to the injection speed detection link. Specifically, in the injection speed detection link process for the original pipetting video, each original image video frame of the original pipetting video can be preprocessed and connected domain detection can be performed to obtain the center position and area information of the label, and the measured distance corresponding to the preprocessed image is calculated by the center position and area information of the label, and then the injection segment is screened according to the measured distance corresponding to all preprocessed images, and the final intra-frame injection volume of each injection image frame is calculated according to the measured distance of the injection image frame in the injection segment, and the final intra-frame injection volume is converted into a motor control parameter.

[0146] The pipette injection speed detection method of the present application has the following advantages:

[0147] 1. High-precision detection: Through optimized image processing algorithms, high-precision detection of injection speed is achieved, which is suitable for a variety of complex scenarios.

[0148] 2. Low hardware cost: Using smartphone cameras for video capture reduces hardware costs and equipment complexity.

[0149] 3. Automated transplantation: It can learn the manual pipetting process and automatically transplant it to different experimental scenarios to improve experimental efficiency and repeatability.

[0150] 4. Strong adaptability: Through the design and installation of feature labels, the problems of occlusion and position change that may occur during the operation of the pipette can be overcome.

[0151] Based on the same inventive concept, the embodiment of the present application also provides a pipette injection speed detection device for implementing the above-mentioned pipette injection speed detection method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more pipette injection speed detection device embodiments provided below can refer to the limitations of the pipette injection speed detection method above, and will not be repeated here.

[0152] In an exemplary embodiment, Fig.11 As shown, the present application provides a pipette injection speed detection device including the following modules.

[0153] The original pipetting video acquisition module T1 is used to: acquire the original pipetting video; the original pipetting video is a video obtained by shooting the injection operation of the pipette with a camera; the pipette is vertically and parallelly installed with a first label, a second label and a third label from top to bottom; the first label and the second label are fixed on the external clamp of the pipette, and the third label is fixed on the gun body of the pipette; the original pipetting video includes a plurality of original image video frames.

[0154] The preprocessing module T2 is used to: for each of the original image video frames, preprocess the original image video frame to obtain a preprocessed image.

[0155] The connected domain detection module T3 is used to: perform connected domain detection on the preprocessed images to obtain the position coordinates of the first center, the second center and the third center in each of the preprocessed images and the first area, the second area and the third area; the first area, the second area and the third area are respectively the areas of the first label, the second label and the third label in the preprocessed image; the first center, the second center and the third center are respectively the centroids of the first area, the second area and the third area.

[0156] The measured distance calculation module T4 is used to determine the measured distance corresponding to each of the preprocessed images according to the first pixel distance, the second pixel distance, the first area, the second area and the third area; the first pixel distance is the pixel length between the second center and the third center on the camera plane; the second pixel distance is the pixel length between the first center and the second center on the camera plane.

[0157] The injection segment screening module T5 is used to screen the injection segments according to the measured distances corresponding to all the pre-processed images.

[0158] The final frame injection volume calculation module T6 is used to: for each injection image frame in the injection segment, calculate the final frame injection volume corresponding to the injection image frame according to the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment; the injection segment is composed of several injection image frames.

[0159] The motor control parameter conversion module T7 is used to convert the injection volume in the final frame corresponding to the injection image frame into the motor control parameters; the motor control parameters include the pre-frequency division coefficient and the number of motor operation pulses.

[0160] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.12 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store injection speed detection data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting injection speed of a pipette is implemented.

[0161] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0163] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0164] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0165] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0166] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0167] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0168] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for detecting the injection speed of a pipette, characterized in that: The method for detecting the injection speed of a pipette gun comprises: Obtaining an original pipetting video; the original pipetting video is a video obtained by shooting a pipette injection operation with a camera; the pipette is vertically and parallelly mounted with a first label, a second label, and a third label from top to bottom; the first label and the second label are fixed on an external fixture of the pipette, and the third label is fixed on the pipette body; the original pipetting video includes a plurality of original image video frames; For each of the original image video frames, preprocessing the original image video frame to obtain a preprocessed image; Performing connected domain detection on the preprocessed images to obtain position coordinates of a first center, a second center, and a third center in each of the preprocessed images and a first area, a second area, and a third area; the first area, the second area, and the third area are respectively the areas of the first label, the second label, and the third label in the preprocessed images; the first center, the second center, and the third center are respectively the centroids of the first area, the second area, and the third area; Determine the measured distance corresponding to each of the preprocessed images according to the first pixel distance, the second pixel distance, the first area, the second area and the third area; the first pixel distance is the pixel length between the second center and the third center on the camera plane; the second pixel distance is the pixel length between the first center and the second center on the camera plane; Filter the injection sections according to the measured distances corresponding to all the preprocessed images; For each injection image frame in the injection segment, a final intra-frame injection volume corresponding to the injection image frame is calculated according to the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment; the injection segment is composed of a plurality of injection image frames; The final intra-frame injection volume corresponding to the injection image frame is converted into motor control parameters; the motor control parameters include a pre-frequency division coefficient and the number of motor operation pulses.

2. The method for detecting the injection speed of a pipette according to claim 1, characterized in that: Preprocessing the original image video frame to obtain a preprocessed image specifically includes: Convert the original image video frame from the RGB color space to the HSV color space to obtain an HSV image video frame; Binarizing the HSV image video frame to obtain a binary image; A closing operation is performed on the binary image to obtain an image processed by the closing operation; the image processed by the closing operation is the pre-processed image.

3. The method for detecting the injection speed of a pipette according to claim 1, characterized in that: The calculation formula for the measured distance corresponding to the preprocessed image is as follows: Among them, L d Represents the measured distance corresponding to the preprocessed image, l A”B” is the second pixel distance, l B”C” is the first pixel distance, k AB is the first error coefficient caused by depth of field, k BC is the second error coefficient caused by the depth of field.

4. The method for detecting the injection speed of a pipette according to claim 1, characterized in that: The injection section is selected according to the measured distances corresponding to all the pre-processed images, specifically including: Determine the minimum distance frame; the minimum distance frame is the event frame corresponding to the minimum distance point measured during the pipetting process; Take the minimum distance frame as the initial frame, start from the minimum distance frame and traverse to the left, determine whether the measured distance corresponding to the current time frame meets the first set condition, and obtain the first judgment result; the first set condition is expressed as s i m ×(1-k a );where s i ,s m For timeframe equal to i, i m The measured distance at time k a is the selected threshold; i, i m Respectively represent the current time frame, the maximum distance frame and the minimum distance frame;​ If the first judgment result is yes, the previous time frame of the current time frame replaces the current time frame, and returns to the step of "judging whether the measured distance corresponding to the current time frame meets the first set condition"; If the first judgment result is no, then determine whether the second setting condition is met to obtain a second judgment result; the second setting condition is that the measured distance corresponding to the previous time frame of the current time frame is greater than the measured distance corresponding to the current time frame; If the second judgment result is yes, the previous time frame of the current time frame replaces the current time frame, and returns to the step of "judging whether the second set condition is met"; If the second judgment result is no, the current time frame is determined as the injection segment starting frame; Taking the injection segment start frame as the initial frame, traversing rightward from the injection segment start frame, determining whether the measured distance corresponding to the current time frame satisfies the third set condition, and obtaining a third determination result; If the third judgment result is yes, the next time frame of the current time frame replaces the current time frame, and the process returns to the step of "judging whether the measured distance corresponding to the current time frame satisfies the third setting condition"; If the third judgment result is no, then determine whether the fourth setting condition is met to obtain a fourth judgment result; the fourth setting condition is that the measured distance corresponding to the next time frame of the current time frame is smaller than the measured distance corresponding to the current time frame; If the fourth judgment result is yes, the next time frame of the current time frame replaces the current time frame, and the process returns to the step of "judging whether the fourth setting condition is met"; If the fourth determination result is no, the current time frame is determined as the injection segment end frame; the injection segment start frame, the injection segment end frame and all time frames therebetween constitute the injection segment.

5. The method for detecting the injection speed of a pipette according to claim 1, characterized in that: Calculating the final intra-frame injection volume corresponding to the injection image frame according to the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment, specifically comprising: Obtaining the original injected volume corresponding to the injection image frame according to the measured distance of the injection image frame, the measured distance of the injection start frame of the injection segment, the measured distance when the pipette push rod is fully pressed, and the maximum injection volume of the pipette; Optimizing the original injected volume corresponding to the injection image frame to obtain an optimized injected volume corresponding to the injection image frame; An outlier removal process is performed on the first optimized injected volume corresponding to the injection image frame to obtain a second optimized injected volume corresponding to the injection image frame; the second optimized injected volume is the final intra-frame injection volume.

6. The method for detecting the injection speed of a pipette according to claim 1, characterized in that: The calculation formula for the number of motor running pulses is as follows: Where n i The number of motor pulses injected into the i-th frame, p i is the number of pulses the motor needs to run, v i ' is the final intra-frame injection volume corresponding to the i-th injection image frame, V s The volume injected per pulse of the transplant pipette, n e is the remaining decimal part; The calculation formula of the pre-scaling coefficient is as follows: In the formula, psc i The pre-scaling coefficient of the image frame injected for the i-th frame, arr is the automatic reload value, f clk is the clock frequency, t f The actual time represented by each frame of the original injection video.

7. A pipette injection speed detection device, characterized in that: The pipette injection speed detection device comprises: The original pipetting video acquisition module is used to: acquire the original pipetting video; the original pipetting video is a video obtained by shooting the injection operation of the pipette gun by a camera; the pipette gun is vertically and parallelly installed with a first label, a second label and a third label from top to bottom; the first label and the second label are fixed on the external clamp of the pipette gun, and the third label is fixed on the gun body of the pipette gun; the original pipetting video includes a plurality of original image video frames; A preprocessing module is used to: for each of the original image video frames, preprocess the original image video frame to obtain a preprocessed image; A connected domain detection module is used to: perform connected domain detection on the preprocessed images to obtain the position coordinates of the first center, the second center and the third center in each of the preprocessed images and the first area, the second area and the third area; the first area, the second area and the third area are respectively the areas of the first label, the second label and the third label in the preprocessed images; the first center, the second center and the third center are respectively the centroids of the first area, the second area and the third area; A measured distance calculation module, used to: determine the measured distance corresponding to each of the preprocessed images according to a first pixel distance, a second pixel distance, a first area, a second area, and a third area; the first pixel distance is a pixel length between the second center and the third center on the camera plane; the second pixel distance is a pixel length between the first center and the second center on the camera plane; An injection segment screening module is used to: screen the injection segments according to the measured distances corresponding to all the pre-processed images; The final frame injection volume calculation module is used to: for each injection image frame in the injection segment, calculate the final frame injection volume corresponding to the injection image frame according to the measured distance of the injection image frame and the measured distance of the injection start frame of the injection segment; the injection segment is composed of a plurality of injection image frames; The motor control parameter conversion module is used to: convert the injection volume in the final frame corresponding to the injection image frame into the motor control parameter; the motor control parameter includes a pre-frequency division coefficient and the number of motor operation pulses.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for detecting the injection speed of a pipette according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting the injection speed of a pipette according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for detecting the injection speed of a pipette according to any one of claims 1 to 6 is implemented.

Images