Sample tube positioning method, sample tube grabbing method and biological storage library

Abstract

The invention relates to the technical field of biomedical treatment, particularly provides a sample tube positioning method, a sample tube grabbing method and a biological sample library, and aims to solve the problem of inaccurate sample tube grabbing precision in an existing storage library. In order to achieve the purpose, the method comprises the steps of obtaining a sample picture of a sample box; inputting the sample picture into a trained sample model to obtain a first preprocessed picture; obtaining first feature information according to the first preprocessed picture; using an image recognition algorithm for the first preprocessed picture to obtain second feature information; and determining final position information of the sample tube according to the first feature information and the second feature information. Compared with the prior art, the deep learning algorithm and the image recognition algorithm are combined, so that the position of the sample tube can be accurately positioned, the abnormal tube can be accurately grabbed, the overall positioning efficiency is improved, the recognition time is shortened, and the recognition cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of biomedical technology, and specifically provides a sample tube positioning method, a grasping method and a biological storage library. Background Art

[0002] -196 degrees deep cryogenic biological sample storage is an important basic equipment for current research in the medical and biological fields. Through the cryogenic storage of liquid nitrogen, biological tissues such as blood, stem cells and immune cells can remain active for a long time. In order to realize the information management of biological samples, each sample tube needs to be coded, and the code needs to be scanned to enter the information when the sample is stored, and the code also needs to be scanned for verification when it is out of the warehouse.

[0003] The existing grasping of sample tubes is based on fixed coordinates. The robot arm picks and places each sample tube in the storage according to the three-dimensional coordinates initially measured. Once the position is offset, the grasping accuracy is prone to inaccurate when picking and placing the tube according to the fixed coordinates. In addition, in a deep cryogenic environment, frost may form on the surface of the sample tube, and the surface of the sample tube may be reflective or blocked due to mechanical structure or other reasons, which will also affect the grasping of the robot arm.

[0004] Therefore, the present invention needs to provide a new technical solution to solve the above technical problems. Summary of the invention

[0005] The present invention aims to solve the above-mentioned technical problem, that is, to solve the problem of inaccurate grasping of sample tubes in a storage library.

[0006] In a first aspect, the present invention provides a sample tube positioning method, wherein a plurality of sample tubes are stored in a plurality of sample frames in a sample box, and the sample tube positioning method comprises: obtaining a sample image of the sample box; inputting the sample image into a trained sample model to obtain a first preprocessed image; obtaining first feature information based on the first preprocessed image; wherein the first feature information includes at least one of the position information of each sample frame and the state of the sample tube corresponding to each sample frame; obtaining second feature information based on the first preprocessed image using an image recognition algorithm; wherein the second feature information includes the position information of the sample tube corresponding to each sample frame; and determining the final position of the sample tube based on the first feature information and the second feature information.

[0007] In a specific implementation of the above-mentioned sample tube positioning method, the state of the sample tube includes at least one of a tube state, a tubeless state and an abnormal tube state; the step of "using an image recognition algorithm to obtain second feature information from the first preprocessed image" specifically includes: first using an edge segmentation algorithm to segment the first preprocessed image to obtain multiple second preprocessed images containing a single sample frame; wherein the second preprocessed image includes an image of a tube state, an image of a tubeless state and an abnormal tube state; and determining the location information of the sample tube based on the second preprocessed image.

[0008] In a specific implementation of the above-mentioned sample tube positioning method, the step of "determining the position information of the sample tube according to the second preprocessed image" specifically includes: for the image with the tube state: using a template matching algorithm to extract the position information of the sample tube in each image.

[0009] In a specific implementation of the above-mentioned sample tube positioning method, the step of "determining the position information of the sample tube according to the second preprocessed image" further includes: for the image of the abnormal tube state: using an edge detection algorithm and a contour extraction algorithm to extract the position information of the sample tube in each image.

[0010] In a specific implementation of the above-mentioned sample tube positioning method, the position information of the sample tube includes the tube diameter of the sample tube and the pixel coordinates of the center point of the sample tube; the step of "using edge detection algorithm and contour extraction algorithm to extract the position information of the sample tube in each picture" specifically includes: firstly performing edge detection on the picture of the abnormal tube state to identify the edge corresponding to the sample frame; extracting the edge of the sample frame to obtain the contour image of the sample frame; according to the contour image, obtaining the pixel coordinates of the center point of the sample frame, and using the coordinates of the center point of the sample frame as the pixel coordinates of the sample tube.

[0011] In a specific implementation of the above-mentioned sample tube positioning method, the position information of the sample frame includes the length and width of the sample frame and the pixel coordinates of the intersection of the sample frame; the step of "obtaining the final position information of the sample tube based on the first feature information and the second feature information" specifically includes: first matching the position information of the sample frame and the position information of the sample tube corresponding to the same sample tube; performing coordinate conversion on the pixel coordinates of the intersection of the sample frame and the pixel coordinates of the center point of the sample tube to obtain the pixel coordinates of the intersection of the sample frame and the pixel coordinates of the center point of the sample tube in the same coordinate system; determining the final pixel coordinates of the center point of the sample tube based on the pixel coordinates of the intersection of the sample frame and the pixel coordinates of the center point of the sample tube in the same coordinate system.

[0012] In a specific implementation of the above-mentioned sample tube positioning method, the training step of the sample model specifically includes: obtaining a preset number of sample tubes stored in different positions on each type of sample box and sample tubes in abnormal states; annotating each of the sample images to obtain an annotation file containing the sample box type, whether there are tubes and / or abnormal tubes in the sample box; performing model training on the annotation file to obtain an onnx file; and loading the onnx file to obtain the sample model.

[0013] In a second aspect, the present invention also provides a sample tube grabbing method for a biological storage repository, the biological storage repository comprising a grabbing device, a sample tube and a sample box, the sample box being provided with a plurality of sample frames, each of the sample frames storing a sample tube, the grabbing device being configured to grab the sample tube; the sample tube grabbing method comprising: obtaining a sample image of the sample box; inputting the sample image into a trained sample model to obtain first feature information; wherein the first feature information comprises at least one of the position information of each sample frame and the state of the sample tube corresponding to each sample frame; obtaining second feature information from the sample image using an image recognition algorithm; wherein the second feature information comprises the position information of the sample tube corresponding to each sample frame; determining the final position information of the sample tube based on the first feature information and the second feature information; and controlling the grabbing device to grab the sample tube according to the final position of the sample tube.

[0014] In a specific implementation of the sample tube grabbing method of the above-mentioned biological storage repository, the position information of the sample tube includes the tube diameter of the sample tube and the pixel coordinates of the center point of the sample tube; the step of "controlling the grabbing device to grab the sample tube according to the final position of the sample tube" specifically includes: converting the pixel coordinates of the center point of the sample tube into six-axis coordinates and sending them to the grabbing device; controlling the grabbing device to grab the sample tube according to the six-axis coordinates.

[0015] In a third aspect, the present invention further provides a biological storage repository, comprising the sample tube grabbing device as described above.

[0016] When adopting the above technical solution, the present invention first uses a deep learning algorithm to roughly locate the sample image, that is, first obtain the position of the sample frame, and then uses an image recognition algorithm for precise positioning, that is, obtain the position of the sample tube in each sample frame. By combining the deep learning algorithm with the image recognition algorithm, compared with the existing method, it can not only accurately locate the position of the sample tube, but also realize accurate capture of abnormal tubes, thereby improving the overall positioning efficiency and shortening the recognition time and recognition cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings, in which:

[0018] Figure 1 is a flow chart of the main steps of the sample tube positioning method of the present invention;

[0019] Figure 2 It is a flow chart of the main steps of the sample tube grabbing method of the present invention. DETAILED DESCRIPTION

[0020] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not used to limit the scope of protection of the present invention. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0021] It should be noted that, in the description of the present invention, ordinal numbers such as "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, although the various steps of the control method of the present invention are described in a specific order in this application, these orders are not restrictive, and those skilled in the art may perform the steps in a different order without departing from the basic principles of the present invention.

[0022] See first Figure 1 , which is a flow chart of the main steps of the sample tube positioning method of the present invention. Figure 1 As shown, the sample tube positioning method of the present invention includes:

[0023] S1, obtain a sample image of a sample box.

[0024] In some embodiments, a sample image of the sample box is acquired by a camera.

[0025] S2: Input the sample image into the trained sample model to obtain a first preprocessed image.

[0026] In some embodiments, the sample model is a sample model trained by a deep learning algorithm based on sufficient sample images collected in advance, and the acquired sample image is input into the sample model for comparison to obtain a first preprocessed image.

[0027] S3, obtaining first feature information according to the first preprocessed image; wherein the first feature information includes at least one of position information of each sample frame and a state of a sample tube corresponding to each sample frame;

[0028] In some embodiments, the first feature information includes one or two of the position information of each sample frame and the state of the sample tube corresponding to each sample frame. Exemplarily, the first feature information includes the position information of each sample frame and the state of the sample tube corresponding to each sample frame. The position information of the sample frame is rough position information, specifically including the size of the sample frame and the pixel coordinates of the intersection of the border of the sample frame. If the sample frame is a rectangle, the size of the sample frame is the length and width of the rectangle and the pixel coordinates (X, Y) of the intersection of the rectangular border. If the sample frame is a circle, the size of the sample frame is the length and width of the rectangular border corresponding to the circle and the pixel coordinates (X, Y) of the intersection of the rectangular border. The state of the sample tube includes one or two or three of the state with tube, the state without tube and the state of abnormal tube. Exemplarily, the state of the sample tube includes the state with tube, the state without tube and the state of abnormal tube. Specifically, while outputting the position information of each sample frame, the sample frame is numbered. For example, the position coordinates of the sample frame numbered 1 are (X1, Y1), and the state of the sample tube corresponding to the sample frame is the state with tube. In addition, abnormal tubes are tubes with frost, reflections or obstructions on the surface of the sample tubes. This is because in actual operations, the sample box is transferred between a deep low temperature environment, a low temperature environment and a normal temperature environment. Therefore, during the transportation process, frost will appear on the surface of the sample tube. Due to mechanical structure or other conditions, the surface of the sample tube will also appear reflective or obstructed.

[0029] S4, using an image recognition algorithm to obtain second feature information on the first preprocessed image; wherein the second feature information includes position information of the sample tube corresponding to each sample frame;

[0030] In some embodiments, the position information of the sample tube corresponding to each sample frame is the position information of the sample tube corresponding to the sample frame number, and the position information of the sample tube includes the diameter of the sample tube and the pixel coordinates (X', Y') of the center point of the sample tube;

[0031] S5, determining the final position information of the sample tube according to the first characteristic information and the second characteristic information.

[0032] Furthermore, in step S2, the training steps of the sample model specifically include:

[0033] Obtain a preset number of sample images of sample tubes stored in different positions on each type of sample box and sample tubes in abnormal states;

[0034] Annotate each sample image to obtain an annotation file including the sample box type, whether there are tubes and / or abnormal tubes in the sample box;

[0035] Perform model training on the labeled file to obtain the onnx file;

[0036] Load the onnx file to get the sample model.

[0037] In some embodiments, the sample pictures described above are sample pictures of sample tubes stored in different positions and in different abnormal states of each type of sample box, which are collected in advance and preset in number. There are many types of sample boxes for storing sample tubes in the prior art, mainly 10*10 (10 rows and 10 columns), 12*8 (12 rows and 8 columns), 8*6 (8 rows and 6 columns), and 6*4 (6 rows and 4 columns), so one of each of these types of sample boxes is found. Then, for each type of sample box storing sample tubes in different positions and in different abnormal states of sample tubes, pictures are taken to obtain sample pictures. Here, different positions refer to different positions of sample tubes on the sample box in each sample picture, and different abnormal states of sample tubes refer to abnormal states such as frost, fog, reflection, and obstruction. The number of sample tubes in each sample picture is not limited by this application and can be set arbitrarily. The placement position of the sample tubes can also be set arbitrarily. For example, sample tubes are placed at the four corners of the sample box for taking pictures. However, these sample pictures must include pictures of empty box state and full box state. Among them, the preset number of sample pictures recorded above means that each type of sample box needs to take a preset number of pictures, and the preset number of pictures can be 100, 200, 500, etc. The specific value of the preset number of pictures is flexibly set according to the accuracy and difficulty of recognition of the trained sample recognition model.

[0038] Further, in step S4, the step of "using an image recognition algorithm to obtain second feature information on the first preprocessed image" specifically includes:

[0039] S41, first use an edge segmentation algorithm to segment the first preprocessed image to obtain a plurality of second preprocessed images containing single sample frames; wherein the second preprocessed images include images with tubes, images without tubes, and images with abnormal tubes;

[0040] In some embodiments, the second pre-processed picture is a picture including sample frame numbers.

[0041] S42, determining the position information of the sample tube according to the second preprocessed image.

[0042] Further, in step S42, the step of “determining the position information of the sample tube according to the second preprocessed image” specifically includes:

[0043] S421, for the images with tubes: extract the position information of the sample tubes in each image using a template matching algorithm.

[0044] In some embodiments, the template matching algorithm specifically matches the image to be matched with the template image using a normalized correlation matching method, and extracts the position information of the sample tube in the matching image. The template image is a template image of a sample frame with a tube state that has been pre-collected and stored for a preset number of times.

[0045] S422, for the pictures of abnormal tube status: extract the position information of the sample tube in each picture using edge detection algorithm and contour extraction algorithm.

[0046] In some embodiments, since the sample tube in the abnormal tube state picture cannot accurately obtain the position information of the sample tube when it is in abnormal state such as frost or reflection, the position information of the sample tube is extracted using edge detection algorithm and contour extraction algorithm.

[0047] Further, in step S422, the step of “using an edge detection algorithm and a contour extraction algorithm to extract the position information of the sample tube in each image” specifically includes:

[0048] S4221, first perform edge detection on the image of the abnormal tube state to identify the edge of the corresponding sample frame;

[0049] S4222, extracting the edge of the sample frame to obtain a contour image of the sample frame;

[0050] S4223, obtaining the pixel coordinates of the center point of the sample frame according to the contour image, and using the coordinates of the center point of the sample frame as the pixel coordinates of the sample tube.

[0051] In some embodiments, if the sample frame is a rectangle, the center point of the sample frame is the intersection of the diagonal lines; if the sample frame is a circle, the center point of the sample frame is the center of the circle.

[0052] Further, in step S5, the step of “determining the final position information of the sample tube according to the first characteristic information and the second characteristic information” specifically includes:

[0053] S51, first match the position information of the sample frame corresponding to the same sample tube with the position information of the sample tube.

[0054] In some embodiments, the position information of the sample frame and the position information of the sample tube corresponding to the same sample tube are specifically the position information of the sample frame and the position information of the sample tube under the same sample frame number.

[0055] S52, performing coordinate conversion on the pixel coordinates of the intersection of the sample frame and the pixel coordinates of the center of the sample tube to obtain the pixel coordinates of the intersection of the sample frame and the pixel coordinates of the center of the sample tube in the same coordinate system.

[0056] S53, determining the final pixel coordinates of the center point of the sample tube according to the pixel coordinates of the intersection of the sample frame and the pixel coordinates of the center point of the sample tube in the same coordinate system.

[0057] To summarize, the present invention first uses a deep learning algorithm to roughly locate the sample image, that is, first obtains the position of the sample frame, and then uses an image recognition algorithm for precise positioning, that is, obtains the position of the sample tube in each sample frame. By combining the deep learning algorithm with the image recognition algorithm, compared with the existing method, it can not only accurately locate the position of the sample tube, but also accurately capture the abnormal tube, thereby improving the overall positioning efficiency and shortening the recognition time and recognition cost.

[0058] Based on the above embodiments, the present invention also provides a method for grabbing a sample tube of a biological storage library. Figure 2 , which is a flow chart of the main steps of the sample tube grabbing method of the present invention. Figure 2 As shown, the sample tube grabbing method of the present invention includes:

[0059] S11, obtaining a sample image of a sample box;

[0060] S12, inputting the sample image into the trained sample model to obtain a first preprocessed image;

[0061] S13, obtaining first feature information according to the first preprocessed image; wherein the first feature information includes at least one of position information of each sample frame and a state of the sample tube corresponding to each sample frame;

[0062] S14, using an image recognition algorithm to obtain second feature information on the first preprocessed image; wherein the second feature information includes position information of a corresponding sample tube in each sample frame;

[0063] S15, determining the final position information of the sample tube according to the first characteristic information and the second characteristic information;

[0064] S16, controlling the grabbing device to grab the sample tube according to the final position of the sample tube.

[0065] Further, in step S16, the step of “controlling the grabbing device to grab the sample tube according to the final position of the sample tube” specifically includes:

[0066] S161, converting the pixel coordinates of the center of the sample tube into six-axis coordinates and sending them to the grasping device;

[0067] S162, controlling the grasping device to grasp the sample tube according to the six-axis coordinates.

[0068] In some embodiments, the final position information of the sample tube is the relative position of the sample tube in the biological storage repository. In addition, the gripping device determines the size of the gripper opening of the gripping device according to the six-axis coordinates of the center of the sample tube and the diameter of the sample tube to ensure that the sample tube can be gripped.

[0069] Based on the above-mentioned embodiments, the present invention further provides a biological storage repository, which is characterized in that it includes the sample tube grabbing device as described above.

[0070] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A sample tube positioning method, characterized in that: The plurality of sample tubes are stored in a plurality of sample frames in a sample box, and the sample tube positioning method comprises: Obtaining a sample image of the sample box; Inputting the sample image into a trained sample model to obtain a first preprocessed image; Acquire first feature information according to the first preprocessed image; wherein the first feature information includes at least one of position information of each of the sample frames and a state of the sample tube corresponding to each of the sample frames; Using an image recognition algorithm on the first preprocessed image, obtaining second feature information; wherein the second feature information includes position information of the sample tube corresponding to each sample frame; The final position information of the sample tube is determined based on the first characteristic information and the second characteristic information.

2. The sample tube positioning method according to claim 1, characterized in that: The state of the sample tube includes at least one of a tube state, a tube-free state, and an abnormal tube state; The step of "using an image recognition algorithm to obtain second feature information from the first preprocessed image" specifically includes: First, the first pre-processed image is segmented by an edge segmentation algorithm to obtain a plurality of second pre-processed images containing single sample frames; wherein the second pre-processed images include images with tubes, images without tubes, and images with abnormal tubes; The position information of the sample tube is determined according to the second preprocessed image.

3. The sample tube positioning method according to claim 2, characterized in that: The step of "determining the position information of the sample tube according to the second preprocessed image" specifically includes: For the pictures with tubes: the position information of the sample tubes in each picture is extracted using a template matching algorithm.

4. The sample tube positioning method according to claim 3, characterized in that: The step of "determining the position information of the sample tube according to the second preprocessed image" further includes: For the pictures of the abnormal tube status: the position information of the sample tube in each picture is extracted using an edge detection algorithm and a contour extraction algorithm.

5. The sample tube positioning method according to claim 4, characterized in that: The position information of the sample tube includes the diameter of the sample tube and the pixel coordinates of the center point of the sample tube; The step of "using edge detection algorithm and contour extraction algorithm to extract the position information of the sample tube in each picture" specifically includes: Firstly, edge detection is performed on the image of the abnormal tube state to identify the edge corresponding to the sample frame; Extracting the edge of the sample frame to obtain a contour image of the sample frame; According to the contour image, the pixel coordinates of the center point of the sample frame are obtained, and the coordinates of the center point of the sample frame are used as the pixel coordinates of the sample tube.

6. The sample tube positioning method according to claim 5, characterized in that: The position information of the sample frame includes the length and width of the sample frame and the pixel coordinates of the intersection points of the sample frame; The step of "determining the final position information of the sample tube according to the first characteristic information and the second characteristic information" specifically includes: Firstly, matching the position information of the sample frame and the position information of the sample tube corresponding to the same sample tube; Performing coordinate conversion on the pixel coordinates of the intersection of the sample frame and the pixel coordinates of the center of the sample tube to obtain the pixel coordinates of the intersection of the sample frame and the pixel coordinates of the center of the sample tube in the same coordinate system; The final pixel coordinates of the center point of the sample tube are determined according to the pixel coordinates of the intersection of the sample frame and the pixel coordinates of the center point of the sample tube in the same coordinate system.

7. The sample tube positioning method according to claim 1, characterized in that: The training steps of the sample model specifically include: Obtain a preset number of sample images of sample tubes stored in different positions on each type of sample box and sample tubes in abnormal states; Annotating each of the sample images to obtain an annotation file including the sample box type, whether there are tubes and / or abnormal tubes in the sample box; Performing model training on the labeled file to obtain an onnx file; The onnx file is loaded to obtain the sample model.

8. A method for grabbing sample tubes of a biological storage library, characterized in that: The biological storage library comprises a grabbing device, a sample tube and a sample box, wherein a plurality of sample frames are arranged in the sample box, each of the sample frames stores a sample tube, and the grabbing device is arranged to grab the sample tube; The sample tube grabbing method comprises: Obtaining a sample image of the sample box; Inputting the sample image into a trained sample model to obtain a first preprocessed image; Acquire first feature information according to the first preprocessed image; wherein the first feature information includes at least one of position information of each of the sample frames and a state of the sample tube corresponding to each of the sample frames; Using an image recognition algorithm on the first preprocessed image, obtaining second feature information; wherein the second feature information includes position information of the sample tube corresponding to each sample frame; Determining final position information of the sample tube according to the first characteristic information and the second characteristic information; The grasping device is controlled to grasp the sample tube according to the final position of the sample tube.

9. The method for grabbing a sample tube of a biological storage bank according to claim 8, characterized in that: The position information of the sample tube includes the diameter of the sample tube and the pixel coordinates of the center point of the sample tube; The step of "controlling the grabbing device to grab the sample tube according to the final position of the sample tube" specifically includes: Convert the pixel coordinates of the center point of the sample tube into six-axis coordinates and send them to the grabbing device; The grasping device is controlled to grasp the sample tube according to the six-axis coordinates.

10. A biostorage storage device, characterized in that: Includes a sample tube gripping device as described in claim 9.

Images