Method for identifying and positioning mycobacterium tuberculosis colonies on inclined plane

Abstract

The invention relates to a method for identifying and positioning mycobacterium tuberculosis colonies on an inclined plane, which comprises the following steps of: converting the actual position of a bacterium picking stick into a rotary displacement matrix of a robot world coordinate system, picking mark points, and determining a plane equation of a plane where a sample tube is positioned; the culture surface in the inclined sample tube is an inclined plane, the camera continuously photographs the rotating sample tube, continuously reads a frame image, calculates the ratio of a point representing a background to an image area, obtains the signal intensity of a background area, and judges the surface of the culture medium directly facing the camera according to the signal intensity; photographing the side surface of the culture medium by a camera, fitting a culture surface into a straight line to obtain pixel coordinate values of two end points of the straight line, and determining a plane equation of the front surface of the culture medium; the camera photographs the front face of the culture medium, determines pixel coordinates of a bacterium picking point, converts the pixel coordinates into robot world coordinates according to a plane equation of the front face of the culture medium, and controls the manipulator to execute bacterium picking action through the robot world coordinates.

Description

Technical Field

[0001] The invention belongs to the technical field of colony identification and positioning, and particularly relates to a method for identifying and positioning a Mycobacterium tuberculosis colony on an inclined plane. Background Art

[0002] Mycobacterium tuberculosis, the pathogen that causes tuberculosis (TB), is an ancient disease with widespread distribution worldwide. To study the characteristics of M. tuberculosis, researchers need to culture and study the bacterium. This bacterium grows slowly, requiring long cultivation times. It is also sensitive to ultraviolet light and can be killed by exposure to direct sunlight for just a few hours. Therefore, the bacterium must be cultured in an environment protected from UV light. Traditional sterile chambers, containing multiple culture dishes, are no longer suitable for rapidly culturing large numbers of M. tuberculosis. The culture dishes are shallow, and the culture medium lacks nutrients.

[0003] At present, those skilled in the art adopt more straight and elongated sample tubes, and solid culture medium is installed in advance in the sample tube. The sample tube equipped with culture medium can be produced in batches like this, and researchers can directly purchase it, saving the time of a large amount of homemade culture medium, and the product consistency of the culture medium produced in batches is better, for experimental research provides a good basis, and the laboratory can culture mycobacterium tuberculosis in batches, thereby improving research efficiency. However, if commercially available sample tubes cultivate mycobacterium tuberculosis in a vertical posture, the culture medium top surface (i.e. the cross section of the tube body) in the tube body is the culture surface, and the culture surface area is relatively small, which limits the amount of culture. If the sample tube is tilted, the culture surface can be increased a lot, greatly improving culture efficiency. However, this area is very little for the automated bacterium colony collection research of the tilted culture surface, and manual collection is inefficient again. This area is in urgent need of a method for bacterium colony identification and positioning on the tilted culture surface. Summary of the Invention

[0004] In response to the above method, the present invention provides a method for identifying and locating Mycobacterium tuberculosis colonies on an inclined plane, comprising:

[0005] S100: The actual spatial position of the picking stick is converted into a rotational displacement matrix of the robot's world coordinate system. Fixed marking points are selected around the sample tube, and the plane equation of the plane where the sample tube is located is determined to facilitate the robot's operation of positioning the picking stick and image processing of the camera.

[0006] S200: The culture surface in the tilted sample tube is an inclined plane. The camera continuously photographs the rotating sample tube, continuously reads the camera frame image, and uses the color difference between the culture medium and the background to calculate the ratio of the background point to the image area to obtain the signal intensity of the background area. Based on the signal intensity, the culture medium surface facing the camera is determined.

[0007] S300: The camera takes a picture of the side of the culture medium, fits a straight line to the culture surface, obtains the pixel coordinate values ​​of the two end points of the straight line, and then determines the plane equation of the front of the culture medium;

[0008] S400: The camera takes a picture of the front of the culture medium to determine the pixel coordinates of the bacteria picking point. According to the plane equation of the front of the culture medium, the pixel coordinates are converted into robot world coordinates, and the robot world coordinates are used to control the manipulator to perform the bacteria picking action.

[0009] Optionally, step S100 specifically includes:

[0010] S101: Taking the center of the top of the manipulator as the origin, convert the actual spatial position of the end of the picking stick into the coordinates of the robot's world coordinate system;

[0011] S102: calibrate four marking points on the top surfaces of the two long sides of the experimental box at positions corresponding to the supports and the sample tube (i.e., on both sides of the sample tube), and determine the coordinates of the four marking points in the robot world coordinate system;

[0012] S103: Determine the center point P of the square formed by the four marking points c ; Determine the vectors of the four sides of the square; Determine the center point P c Four internal diagonal vectors to the four marked points respectively;

[0013] S104: Multiply a side vector by the corresponding internal diagonal vector to obtain a corresponding normal vector; arithmetic average the four normal vectors to obtain the normal vectors of the plane where the four marker points are located;

[0014] S105: Align the plane normal vector with the center point P c Perform fitting to obtain the plane equation of the plane where the four marked points are located;

[0015] S106: Using the pnp algorithm, determine the conversion relationship between the camera coordinate system and the robot world coordinate system based on the four marking points in step S102;

[0016] S107: Select the center point of the sample tube mouth as P5, the center point of the sample tube bottom as P6, and the straight line connecting P5 and P6 as the rotation axis of the sample tube. According to the coordinates of P5 and P6 in the robot world coordinate system, obtain the rotation axis vector.

[0017] Optionally, step S101 is specifically as follows: the rotation displacement matrix of the end of the picking stick based on the robot world coordinate system = the actual rotation displacement matrix of the end of the picking stick × the actual rotation displacement matrix of the end of the robotic arm

[0018] According to the position offset relationship between the end of the picking stick and the origin, the actual rotation displacement matrix of the end of the picking stick is obtained as follows:

[0019]

[0020] Wherein, y1 is the distance in m that the end of the picking stick deviates from the origin in the Y-axis direction; z1 is the distance in m that the end of the picking stick deviates from the origin in the Z-axis direction;

[0021] According to the position offset relationship between the end of the manipulator arm and the origin, the actual rotation displacement matrix of the end of the manipulator arm is obtained as follows:

[0022]

[0023] Where y2 is the distance in m that the end of the robot arm deviates from the origin in the Y-axis direction; z2 is the distance in m that the end of the robot arm deviates from the origin in the Z-axis direction;

[0024] The rotational displacement matrix of the end of the picking stick based on the robot's world coordinate system is a matrix with four rows and four columns. The values ​​in the first row and fourth column, the values ​​in the second row and fourth column, and the values ​​in the third row and fourth column respectively correspond to the coordinate values ​​(x0', y0', z0') of the end of the picking stick based on the robot's world coordinate system.

[0025] Optionally, in step S102, vertical lines are drawn from the center of the bottom end of the sample tube to the two long sides of the experimental box, respectively. The vertical lines are perpendicular to the long sides of the experimental box, and the intersections of the vertical lines with the two long sides are marked as two points, denoted as P2 and P3.

[0026] Draw vertical lines from the top of the backrest to the two long sides of the experimental box. The vertical lines are perpendicular to the long sides. The positions where the vertical lines intersect with the two long sides are two other marked points, recorded as P1 and P4.

[0027] Optionally, in step S102 , the camera takes a photo of the four marking points and records the image coordinates of the four marking points on the photo image, which are Pixel_1 , Pixel_2 , Pixel_3 , and Pixel_4 .

[0028] Optionally, in step S103, within the square, the position where the diagonal line connecting P1 and P3 intersects with the diagonal line connecting P2 and P4 is the center point P c ;

[0029] The vector pointing from P4 to the side of P1 is v1, the vector pointing from P1 to the side of P2 is v2, the vector pointing from P2 to the side of P3 is v3, and the vector pointing from P3 to the side of P4 is v4;

[0030] P c The internal diagonal vector pointing to P4 is vc1, P c The internal diagonal vector pointing to P1 is vc2, P cThe internal diagonal vector pointing to P2 is vc3, P c The internal diagonal vector pointing to P3 is vc4.

[0031] Optionally, step S106 is specifically: r_vec, t_vec = cv2.solvePnP (robot world coordinate 3D column, pixel coordinate 2D column, camera intrinsic parameter matrix, camera distortion coefficient),

[0032] The robot world coordinate 3D column is the three-dimensional coordinate column of the robot world coordinate system, with four marker points P1, P2, P3, and P4; the pixel coordinate 2D column is the pixel coordinate column of the four marker points on the image [[x1', y1'], [x2', y2'], [x3', y3'], [x4', y4']];

[0033] The camera intrinsic parameter matrix (camera_K) and camera distortion coefficient (camera_D) are unique to the camera. Different cameras have their own parameters and can be calibrated by themselves, such as the Zhang Zhengyou calibration method.

[0034] r_vec is the rotation change vector, which is converted into a rotation matrix using the Rodrigues formula, i.e., r_matrix = cv2.Rodrigues(r_vec)[0]. t_vec is the translation change vector.

[0035] Optionally, step S200 includes:

[0036] S201: When photographing the sample tube, provide a light-proof environment for the camera and the sample tube so that the area outside the culture medium (i.e., the background area) is black; define a viewing area to ensure that there is always a background area within the viewing area;

[0037] S202: Rotate the sample tube with the central axis of the sample tube as the axis. During the rotation, the camera continuously takes pictures of the sample tube, continuously reads the camera frame image, and quantifies the color category of each pixel in the visible area by using the color difference between the culture medium and the background;

[0038] S203: Using the KMeans algorithm, cluster the colors in the visible area into three parts. The darkest cluster in the center of the cluster is the background cluster. The ratio of the pixels in the background cluster to all the pixels in the visible area is calculated to obtain the signal strength of the background area.

[0039] S204: With the shooting time as the horizontal coordinate and the signal strength of the background area as the vertical coordinate, a relationship graph between the signal strength of the background area and time is obtained; the curve of the relationship graph has several continuous cycles, each cycle has two peaks and two troughs, the lowest trough indicates that the back is facing up, and the second lowest trough indicates that the front is facing up.

[0040] Optionally, step S300 includes:

[0041] S301: Turn the culture medium to a side-up position, take a picture, and capture the effective area in the picture;

[0042] S302: Cluster the colors in the valid area using the KMeans algorithm to select the black area (i.e., the background area); after Gaussian smoothing, obtain all points on the edge of the black area, and then perform a straight line fitting on the edge of the black area;

[0043] S303: Projecting the two endpoints of the edge of the black area onto the plane where the sample tube is located to obtain two projection points, and then combining them with the center point of the camera to obtain the culture medium plane equation;

[0044] S304: Turn the culture medium to a state where the front side faces upward, and rotate the plane equation of the culture medium by the same angle to obtain the plane equation of the front side of the culture medium.

[0045] Further optionally, in step S302, the fitLine function in opencv is used to perform straight line fitting on the edge of the black area; then, based on the image captured by the camera, the maximum number of rows and the minimum number of rows of the two endpoints of the edge of the black area are substituted into the fitted straight line equation to obtain the pixel coordinate values ​​of the two endpoints.

[0046] Further optionally, in step S303, from a top-down perspective, the line of sight is perpendicular to the plane where the sample tube is located (i.e., the plane where the four marking points obtained in step S105 are located), and the plane is extended to the position of the camera, with the center point of the camera as the camera origin Pnew0; starting from Pnew0, use a straight line to connect the endpoints at the top edge of the black area, and then extend the straight line to the plane where the four marking points are located, to obtain a projection point Pnew1; starting from Pnew0, use a straight line to connect the endpoints at the bottom edge of the black area, and then extend the straight line to the plane where the four marking points are located, to obtain a projection point Pnew2;

[0047] Step S106 obtains the translation change vector and the rotation change vector. Given the pixel coordinates of Pnew1 and Pnew2, the coordinates of Pnew1 and Pnew2 in the robot coordinate system can be obtained. Similarly, given the pixel coordinates of Pnew0, the coordinates of Pnew0 in the robot coordinate system can be obtained.

[0048] That is, Pnew0 = -(r_matrix inverse) · (t_vec), that is, Pnew0 = numpy.linalg.inv(r_matrix)@t_vec;

[0049] According to the coordinates of Pnew0, Pnew1 and Pnew2 in the robot coordinate system, the plane equation of the culture medium can be obtained (three points determine a plane). BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram of the structure of the device of the embodiment;

[0051] Figure 2 Schematic diagram of the mushroom picking stick and the robot;

[0052] Figure 3 Here are photos of the four marked locations;

[0053] Figure 4 Four marking points and the center point P c Schematic diagram of the location;

[0054] Figure 5 Schematic diagram of the relationship between the pixel coordinate system, image coordinate system, camera coordinate system and robot world coordinate system;

[0055] Figure 6 This is a photo of the state where the culture medium side faces upward;

[0056] Figure 7 This is a photo of the culture medium facing upwards;

[0057] Figure 8 is a graph showing the relationship between the signal intensity of the background area and the number of shooting frames (the unit of the vertical axis is %, and the unit of the horizontal axis is frame);

[0058] Figure 9 The captured image in step S301;

[0059] Figure 10 for Figure 9 Processed pictures;

[0060] Figure 11 Schematic diagram of the plane where the camera and sample tube are located at the four marked points;

[0061] Figure 12 A schematic diagram of inferring the position of an object from an image in existing imaging technology;

[0062] Figure 13 is a schematic diagram of inferring the position of a physical object from an image in this embodiment;

[0063] Figure 14 Schematic diagram of the picking stick in a fixed posture (the thick red line indicates the blind area);

[0064] Figure 15 This is a schematic diagram of picking bacteria when the tilt angle of the picking stick changes.

[0065] Note: Figure 6 、 Figure 7 、 Figure 9 The text on the sample tube introduces the culture medium, specifically "Neutral Roche Culture Medium (CON)" and its expiration date, and is not the name of the manufacturer or product promotional text.

[0066] In the accompanying drawings, 1-sample tube, 2-experimental box, 3-support, 4-backrest, 5-camera, 6-end of the bacteria picking stick, 7-manipulator, 8-end of the robotic arm, 9-. DETAILED DESCRIPTION

[0067] This embodiment provides a method for identifying and locating Mycobacterium tuberculosis colonies on an inclined plane, comprising:

[0068] S100: The actual spatial position of the picking stick is converted into a rotational displacement matrix of the robot's world coordinate system. Fixed marking points are selected around the sample tube, and the plane equation of the plane where the sample tube is located is determined to facilitate the robot's operation of positioning the picking stick and image processing of the camera.

[0069] S200: The culture surface in the tilted sample tube is an inclined plane. The camera continuously photographs the rotating sample tube, continuously reads the camera frame image, and uses the color difference between the culture medium and the background to calculate the ratio of the background point to the image area to obtain the signal intensity of the background area. Based on the signal intensity, the culture medium surface facing the camera is determined.

[0070] S300: The camera takes a picture of the side of the culture medium, fits a straight line to the culture surface, obtains the pixel coordinate values ​​of the two end points of the straight line, and then determines the plane equation of the front of the culture medium;

[0071] S400: The camera takes a picture of the front of the culture medium to determine the pixel coordinates of the bacteria picking point. According to the plane equation of the front of the culture medium, the pixel coordinates are converted into robot world coordinates, and the robot world coordinates are used to control the manipulator to perform the bacteria picking action.

[0072] Optional, such as Figure 1 As shown, the sample tube 1 is a conventional cylindrical tube with a closed bottom and a reclosable lid on the top, and culture medium is contained inside;

[0073] The sample tube 1 is placed on a rotatable bracket 1, which is then placed inside an experimental box 2. The bracket 1 includes a base and a support 3. The bottom of the base is located on the bottom of the experimental box, and the top of the base is hinged to the bottom of the support via a rotating shaft. When the sample tube is placed on the support, the centerline of the sample tube is parallel to the backrest 4 of the support.

[0074] A motor is installed on the side of the base, connected to a rotating shaft, which can control the tilt angle of the support backrest and the sample tube. The experimental box is a hollow rectangular parallelepiped, with its two long sides located on either side of the support, and the distance between the two long sides and the sample tube is equal.

[0075] Optionally, a second bracket is provided at one end of the experimental box, and a camera 5 is provided on the second bracket. The camera head faces the sample tube, that is, the sample tube is between the camera and the backrest, and the center line of the sample tube is parallel to the surface of the camera, so that the camera can take pictures directly from the side of the sample tube. The tilt angle of the camera head is adjusted according to the tilt angle of the sample tube.

[0076] Optionally, step S100 specifically includes:

[0077] S101: The robot grips the front end of the picking stick, and the end of the picking stick extends downward; with the center of the top of the robot as the origin, the actual spatial position of the end of the picking stick is converted into the coordinates of the robot's world coordinate system;

[0078] S102: Plane calibration: Mark four marking points on the top surfaces of the two long sides of the experimental box, corresponding to the positions of the support and the sample tube (i.e., on both sides of the sample tube). Record the actual positions of the four marking points and determine the coordinates of the four marking points in the robot world coordinate system according to the method of step S101.

[0079] S103: Determine the center point P of the square formed by the four marking points c ; Determine the vectors of the four sides of the square; Determine the center point P c Four internal diagonal vectors to the four marked points respectively;

[0080] S104: Multiply a side vector by the corresponding internal diagonal vector to obtain a corresponding normal vector; arithmetic average the four normal vectors to obtain the normal vectors of the plane where the four marker points are located;

[0081] S105: Align the plane normal vector with the center point P c Perform fitting to obtain the plane equation of the plane where the four marked points are located;

[0082] S106: Using the pnp algorithm, determine the conversion relationship between the camera coordinate system and the robot world coordinate system based on the four marking points in step S102;

[0083] S107: Select the center point of the sample tube mouth as P5, the center point of the sample tube bottom as P6, and the straight line connecting P5 and P6 as the rotation axis of the sample tube. According to the coordinates of P5 and P6 in the robot world coordinate system, obtain the rotation axis vector.

[0084] like Figure 2As shown, a conventional manipulator 7 grips the front end of the picking stick to operate the picking stick. The end of the picking stick is suspended and is used to extend into the sample tube to pick bacterial colonies on the culture surface. Therefore, determining the position of the end 6 of the picking stick is equivalent to determining the position of the specific picking point.

[0085] The manipulator is fixedly mounted outside the experimental box and close to a long side surface, and is located next to support 1. The top of the manipulator is connected to the control device. The present invention takes the manipulator as the center and selects the center of the top of the manipulator as the origin of the actual space.

[0086] Optionally, step S101 is specifically as follows: the rotation displacement matrix of the end of the picking stick based on the robot world coordinate system = the actual rotation displacement matrix of the end of the picking stick × the actual rotation displacement matrix of the end of the robotic arm

[0087] According to the position offset relationship between the end of the picking stick and the origin, the actual rotation displacement matrix of the end of the picking stick is obtained as follows:

[0088]

[0089] Wherein, y1 is the distance in m that the end of the picking stick deviates from the origin in the Y-axis direction; z1 is the distance in m that the end of the picking stick deviates from the origin in the Z-axis direction;

[0090] According to the position offset relationship between the end of the manipulator arm 8 and the origin, the actual rotation displacement matrix of the end of the manipulator arm is obtained as follows:

[0091]

[0092] Where y2 is the distance in m that the end of the robot arm deviates from the origin in the Y-axis direction; z2 is the distance in m that the end of the robot arm deviates from the origin in the Z-axis direction;

[0093] The rotational displacement matrix of the end of the picking stick based on the robot's world coordinate system is a matrix with four rows and four columns. The values ​​in the first row and fourth column, the values ​​in the second row and fourth column, and the values ​​in the third row and fourth column respectively correspond to the coordinate values ​​(x0', y0', z0') of the end of the picking stick based on the robot's world coordinate system.

[0094] Optional, such as Figure 3 As shown, in step S102, the sample tube is placed on the support, with the bottom end of the sample tube against the bottom of the support, and vertical lines are drawn from the center of the bottom end of the sample tube to the two long sides of the experimental box. The vertical lines are perpendicular to the long sides of the experimental box, and the positions where the vertical lines intersect the two long sides are marked as two points, denoted as P2 and P3;

[0095] Draw vertical lines from the top of the backrest to the two long sides of the experimental box. The vertical lines are perpendicular to the long sides. The intersections of the vertical lines with the two long sides are marked as two other points, denoted as P1 and P4.

[0096] On the side of the bracket 1 away from the camera, and viewed from a top-down angle, P1, P2, P3, and P4 are arranged in a square in a counterclockwise direction.

[0097] Optionally, in step S102, the distance P1 is offset from the origin in the Y-axis direction is y3 meters, and the distance P1 is offset from the origin in the Z-axis direction is z3 meters. The actual rotation displacement matrix of P1 is:

[0098]

[0099] Multiply the actual rotation displacement matrix of P1 by the actual rotation displacement matrix of the above-mentioned robotic arm end to obtain the rotation displacement matrix of P1 based on the robot world coordinate system. The values ​​in the fourth column of the first three rows correspond to the coordinate values ​​(x1', y1', z1') of P1's position based on the robot world coordinate system.

[0100] Similarly, the distance P2 is offset from the origin in the Y-axis direction is y4 meters, and the distance P1 is offset from the origin in the Z-axis direction is z4 meters. The actual rotation displacement matrix of P2 is:

[0101]

[0102] Multiply the actual rotation displacement matrix of P2 by the actual rotation displacement matrix of the above-mentioned robotic arm end to obtain the rotation displacement matrix of P2 based on the robot world coordinate system. The values ​​of the fourth column of the first three rows correspond to the coordinate values ​​of P2's position based on the robot world coordinate system (x2', y2', z2').

[0103] Similarly, the distance P3 is offset from the origin in the Y-axis direction is y5 meters, and the distance P3 is offset from the origin in the Z-axis direction is z5 meters. The actual rotation displacement matrix of P3 is:

[0104]

[0105] Multiply the actual rotation displacement matrix of P3 by the actual rotation displacement matrix of the above-mentioned robotic arm end to obtain the rotation displacement matrix of P3 based on the robot world coordinate system. The values ​​of the fourth column of the first three rows correspond to the coordinate values ​​of P3's position based on the robot world coordinate system (x3', y3', z3').

[0106] Similarly, the distance P4 is offset from the origin in the Y-axis direction is y6 meters, and the distance P4 is offset from the origin in the Z-axis direction is z6 meters. The actual rotation displacement matrix of P4 is:

[0107]

[0108] Multiply the actual rotation displacement matrix of P4 by the actual rotation displacement matrix of the above-mentioned robotic arm end to obtain the rotation displacement matrix of P4 based on the robot world coordinate system. The values ​​of the fourth column of the first three rows correspond to the coordinate values ​​of P4's position based on the robot world coordinate system (x4', y4', z4').

[0109] Optionally, in step S102, the camera takes a photo of the four marking points and records the pixel coordinates of the four marking points on the photo image, which are Pixel_1 = (x1', y1'), Pixel_2 = (x2', y2'), Pixel_3 = (x3', y3'), and Pixel_4 = (x4', y4').

[0110] Optional, such as Figure 4 In step S103, within the square, the intersection of the diagonal line connecting P1 and P3 and the diagonal line connecting P2 and P4 is the center point P. c ; Center point P c The coordinate value of the robot world coordinate system is (x c ',y c ', z c '), where x c '=(x1'+x2'+x3'+x4') / 4,y c '=(y1'+y2'+y3'+y4') / 4,z c '=(z1'+z2'+z3'+z4') / 4;

[0111] The vector pointing from P4 to the side of P1 is v1, the vector pointing from P1 to the side of P2 is v2, the vector pointing from P2 to the side of P3 is v3, and the vector pointing from P3 to the side of P4 is v4;

[0112] P c The internal diagonal vector pointing to P4 is vc1, P c The internal diagonal vector pointing to P1 is vc2, P c The internal diagonal vector pointing to P2 is vc3, P c The internal diagonal vector pointing to P3 is vc4. Based on the coordinates of the robot's world coordinate system of the four markers and existing mathematical geometry knowledge, the magnitude and direction of v1-v4 and vc1-vc4 can be obtained.

[0113] Optionally, in step S104, the first normal vector = v1 × vc1, the second normal vector = v2 × vc2, the third normal vector = v3 × vc3, and the fourth normal vector = v4 × vc4;

[0114] That is, the first normal vector = numpy.cross(v1,vc1), the second normal vector = numpy.cross(v2,vc2), the third normal vector = numpy.cross(v3,vc3), and the fourth normal vector = numpy.cross(v4,vc4); the numpy.cross function is used to calculate the cross product of two vectors;

[0115] The magnitude of the normal vector of the plane where the four marker points are located is the arithmetic mean of the magnitudes of the first, second, third, and fourth normal vectors.

[0116] Optionally, in step S105, the plane equation is Ax+By+Cz+D=0,

[0117] Where A is the component of the plane normal vector in the x-axis direction of the robot's world coordinate system, B is the component of the plane normal vector in the y-axis direction of the robot's world coordinate system, and C is the component of the plane normal vector in the z-axis direction of the robot's world coordinate system, all of which are dimensionless.

[0118] P c The coordinate value of the robot world coordinate system (x c ',y c ', z c ') is substituted into the above plane equation to obtain the value of D, so as to obtain the plane equation of the plane where the four marked points are located, that is, the plane equation of the plane where the sample tube is located.

[0119] Optionally, step S106 is specifically: r_vec, t_vec = cv2.solvePnP (robot world coordinate 3D column, pixel coordinate 2D column, camera intrinsic parameter matrix, camera distortion coefficient),

[0120] The robot world coordinate 3D column is a three-dimensional coordinate column of the robot world coordinate system, with four marked points P1, P2, P3, and P4. Each of the four points has coordinate values ​​x, y, and z. The robot world coordinate 3D column is [[x1', y1', z1'], [x2', y2', z2'], [x3', y3', z3'], [x4', y4', z4']];

[0121] The pixel coordinate 2D column is the pixel coordinate column of the four marker points on the image [[x1', y1'], [x2', y2'], [x3', y3'], [x4', y4']];

[0122] The camera intrinsic parameter matrix (camera_K) and camera distortion coefficient (camera_D) are unique to the camera. Different cameras have their own parameters and can be calibrated by themselves, such as the Zhang Zhengyou calibration method.

[0123] r_vec is the rotation change vector, which is converted into a rotation matrix using the Rodrigues formula, i.e., r_matrix = cv2.Rodrigues(r_vec)[0]. t_vec is the translation change vector.

[0124] Through the above method, we can obtain the external parameters of the camera, that is, the rotation and translation of the camera in the robot world coordinate system, and also connect the relationship between the pixel coordinate system, image coordinate system, camera coordinate system and robot world coordinate system, such as Figure 5 shown.

[0125] The pixel coordinate system is a two-dimensional coordinate system with its origin in the upper left corner of the image (photo), specifically referring to the horizontal and vertical pixels on the photo. The image coordinate system is a two-dimensional coordinate system with its origin on the optical axis (i.e., the center point of the imaging), i.e., the photosensitive imaging coordinates x and y in the camera. There is a linear relationship between the pixel coordinate system and the image coordinate system in translation and scaling. The camera coordinate system is a three-dimensional coordinate system in the pinhole imaging model. The image coordinate system can obtain the camera coordinate system through the imaging relationship. The robot world coordinate system can be obtained by the method of step S101.

[0126] Optionally, in step S107, according to the method of step S101, the coordinates of P5 and P6 in the robot world coordinate system are determined based on the actual rotation displacement matrix of P5 and P6 and the actual rotation displacement matrix of the end of the robot arm.

[0127] The distance P5 offsets from the origin in the Y-axis direction is y7 meters, and the distance P5 offsets from the origin in the Z-axis direction is z7 meters. The actual rotation displacement matrix of P5 is:

[0128]

[0129] Multiply the actual rotation displacement matrix of P5 by the actual rotation displacement matrix of the above-mentioned robot end to obtain the rotation displacement matrix of P5 based on the robot world coordinate system. The values ​​of the fourth column of the first three rows correspond to the coordinate values ​​of P5 based on the robot world coordinate system (x5', y5', z5');

[0130] The distance P6 offsets from the origin in the Y-axis direction is y8 meters, and the distance P6 offsets from the origin in the Z-axis direction is z8 meters. The actual rotation displacement matrix of P6 is:

[0131]

[0132] Multiply the actual rotation displacement matrix of P6 by the actual rotation displacement matrix of the above-mentioned robot end to obtain the rotation displacement matrix of P6 based on the robot world coordinate system. The values ​​in the fourth column of the first three rows correspond to the coordinate values ​​(x6', y6', z6') of the position of P6 based on the robot world coordinate system.

[0133] P5 and P6 are the two ends of the rotation axis, and the rotation axis vector can be calculated from the coordinates of P5 and P6 in the robot world coordinate system.

[0134] The tilted sample tube in step S200 means that, driven by the support, the angle between the central axis of the sample tube and the horizontal plane near the backrest is less than 90°. The culture surface is the interface between the solid culture medium in the culture tube and the air. The tilt angle of the culture surface varies depending on the tilt angle of the sample tube.

[0135] When the sample tube is placed on the support, the sample tube is placed in different states including front side up, back side up, and side up according to the different sides of the culture medium in the sample tube facing the camera; front side up means that the culture surface of the culture medium is perpendicular to the direction indicated by the camera, and the culture surface is facing the camera; back side up means that the culture surface is perpendicular to the direction indicated by the camera, and the culture surface is facing the direction indicated by the camera, that is, the culture surface is facing away from the camera; side side up means that the culture surface is parallel to the direction indicated by the camera, that is, the culture surface is facing the left or right side of the camera.

[0136] Optionally, step S200 includes:

[0137] S201: When photographing the sample tube, provide a light-proof environment for the camera and the sample tube so that the area outside the culture medium (i.e., the background area) is black; define a viewing area to ensure that there is always a background area within the viewing area;

[0138] S202: Rotate the sample tube with the central axis of the sample tube as the axis. During the rotation, the camera continuously takes pictures of the sample tube, continuously reads the camera frame image, and quantifies the color category of each pixel in the visible area by using the color difference between the culture medium and the background;

[0139] S203: Using the KMeans algorithm, the colors in the visible area are clustered into three parts. The darkest settlement in the center of the settlement is the background settlement. The ratio of the pixels in the background settlement to all the pixels in the visible area is calculated to obtain the signal strength of the background area.

[0140] S204: With the shooting time as the horizontal coordinate and the signal strength of the background area as the vertical coordinate, a relationship graph between the signal strength of the background area and time is obtained; the curve of the relationship graph has several continuous cycles, each cycle has two peaks and two troughs, the lowest trough indicates that the back is facing up, and the second lowest trough indicates that the front is facing up.

[0141] Further optionally, in step S201, when the camera can capture the complete sample tube, a visible area (i.e., the user's region of interest, ROI area) is delineated to ensure that there is always a background area (black area) within the visible area, and the light-colored area is the culture medium. Generally, the culture medium of Mycobacterium tuberculosis appears white and green in the image.

[0142] Further optionally, in step S202, the central axis of the sample tube is tilted and the sample tube can be rotated; conventional image processing technology is used to read the frame image of the camera, the color of the culture medium is light (white and green), and the color of the background is black.

[0143] In step S203, the colors in the visible area are clustered into three parts: white, green, and black. The surface of the culture medium facing the camera changes as the sample tube rotates, and the ratio of the background area (black) to the area in the visible area also changes. This ratio is converted into the size of the background area. In the present invention, the size of the background area is recorded as the signal intensity of the background area.

[0144] Further optionally, in step S204, the shooting time is proportional to the shooting frame number, so the shooting frame number can also be used as the horizontal axis, such as Figure 8 As shown, the vertical axis is in percentage (%), which represents the percentage of the black background color in the entire comparison area. The percentage values ​​will vary depending on the viewing area, but the curve relationship (the relationship between peaks and troughs) remains similar as the sample tube rotates.

[0145] The sample tube rotates continuously 360° for one cycle. The peak and valley values ​​within several cycles are calculated using the peak detection algorithm (AMPD). The peaks and valley values ​​are then clustered separately, with each clustering into at most two clusters. For peaks, only the largest peak is processed. For valleys, the lowest valley value is first reached, and then the next lowest valley value is moved forward to ensure that the rotation is in place.

[0146] The peak of the curve of the relationship diagram in step S204 indicates that the side is facing upward, and the background area is the largest at this time; the lowest trough indicates that the back is facing upward, and the background area is the smallest at this time; the second lowest trough indicates that the front is facing upward, and the background area is also relatively small at this time.

[0147] For example, in one cycle, the first peak is when the left or right side of the culture medium is facing upward, the first trough (the lowest trough) is when the back of the culture surface is facing upward, the second peak is when the right or left side of the culture medium is facing upward, and the second trough (the second lowest trough) is when the front of the culture surface is facing upward.

[0148] Optionally, step S300 includes:

[0149] S301: Turn the culture medium to a side-up position, take a picture, and capture the effective area in the picture;

[0150] S302: Cluster the colors in the valid area using the KMeans algorithm to select the black area (i.e., the background area); after Gaussian smoothing, obtain all points on the edge of the black area, and then perform a straight line fitting on the edge of the black area;

[0151] S303: Projecting the two endpoints of the edge of the black area onto the plane where the sample tube is located to obtain two projection points, and then combining them with the center point of the camera to obtain the culture medium plane equation;

[0152] S304: Turn the culture medium to a state where the front side faces upward, and rotate the plane equation of the culture medium by the same angle to obtain the plane equation of the front side of the culture medium.

[0153] Further optionally, in step S301, if Figure 9 As shown in FIG. 1 , the effective area in the intercepted image refers to the area where the side of the culture medium is an inclined line with a substantially uniform slope and approximately a straight line. The selection of the effective area can be done by visual observation.

[0154] Further optionally, the clustering in step S302 is the same as the clustering in step S203;

[0155] Before Gaussian smoothing, step S302 also includes removing noise from the image. Here, conventional operations are performed, such as using erosion and dilation to remove interference (cv2.erode, cv2.dilate, etc.); blurring and desharpening (cv2.GaussianBlur, cv2.medianBlur); and image value conversion.

[0156] Use the EdgeDrawing module in opencv to get all the points on the edge of the black area.

[0157] Further optionally, in step S302, the fitLine function in opencv is used to perform straight line fitting on the edge of the black area; Figure 10 As shown, according to the image captured by the camera, the maximum and minimum row numbers of the two endpoints at the edge of the black area are substituted into the fitted straight line equation to obtain the pixel coordinate values ​​of the two endpoints. The maximum and minimum row numbers of the above two endpoints are determined by the pixels of the image itself. Figure 10 The native resolution is 300×700, so the minimum and maximum number of lines are 0 and 699 respectively.

[0158] Further optional, such as Figure 11As shown, in step S303, from a top-down perspective, the line of sight is perpendicular to the plane where the sample tube is located (i.e., the plane where the four marking points obtained in step S105 are located, i.e., the plane with a blue frame);

[0159] Take the center point of the camera as the camera origin Pnew0, starting from Pnew0, use a straight line to connect the endpoints at the top of the edge of the black area, and then extend the straight line to the plane where the four marked points are located to obtain the projection point Pnew1; starting from Pnew0, use a straight line to connect the endpoints at the bottom of the edge of the black area, and then extend the straight line to the plane where the four marked points are located to obtain the projection point Pnew2; the plane where Pnew0, Pnew1, and Pnew2 are located is the culture medium plane, and the culture medium plane intersects with the plane where the four marked points are located, and the intersection line is the line connecting Pnew1 and Pnew2.

[0160] According to the conversion relationship obtained in S106, the plane where the four marking points are located is a known plane in the robot coordinate system, while the culture medium plane is an unknown plane. Because points Pnew1 and Pnew2 are on both planes, the coordinate values ​​of points Pnew1 and Pnew2 in the robot coordinate system can be obtained based on the known planes. Adding Pnew0, the equation of the culture medium plane can be obtained from these three points, which is:

[0161] Step S106 obtains the translation change vector and the rotation change vector. Given the pixel coordinates of Pnew1 and Pnew2, the coordinates of Pnew1 and Pnew2 in the robot coordinate system can be obtained. Similarly, given the pixel coordinates of Pnew0, the coordinates of Pnew0 in the robot coordinate system can be obtained.

[0162] That is, Pnew0 = -(r_matrix inverse) · (t_vec), that is, Pnew0 = numpy.linalg.inv(r_matrix)@t_vec;

[0163] According to the coordinates of Pnew0, Pnew1 and Pnew2 in the robot coordinate system, the plane equation of the culture medium can be obtained (three points determine a plane).

[0164] Further optionally, in step S304, according to the method of step S200, the culture medium is turned to a state where the front side is facing upward, and the rotation angle can be measured;

[0165] Method for processing the rotation of the culture medium plane equation:

[0166] S304-1: Substitute the rotation axis vector obtained in step S107 into the culture medium plane equation to obtain the intersection point Px of the rotation axis and the culture medium plane;

[0167] S304-2: Based on the rotation angle and the axis vector, use the Rodriguez rotation formula to determine the normal vector of the culture medium plane after rotation; then use the plane point formula of the normal vector of the culture medium plane after rotation to substitute the intersection point Px to obtain the plane equation of the front of the culture medium.

[0168] Optionally, in step S400, the culture medium is turned to a front-facing state. After the camera takes a picture, several picking positions are randomly selected on the image to obtain the pixel coordinates of each picking point. The pixel coordinates of the picking point are converted into coordinates of the robot's world coordinate system, and the converted coordinates are sent to the manipulator control device to guide the manipulator and the picking stick to perform the picking action.

[0169] On either side of the lens (i.e. camera) are the image and the object. In the prior art, when the two-dimensional pixel coordinates are converted into the robot world coordinates, the image (i.e. two-dimensional pixel coordinates) is determined, and the distance between the object and the camera is unknown. There can be multiple positions of the object that conform to the imaging principle. Figure 12 As shown, the red line at the camera is the image, and the red line at the sample tube and the bracket is the possible real object. It can be seen that the position of the real object is uncertain.

[0170] In the present invention, the plane equation of the front of the culture medium can be determined, so the position that conforms to the imaging principle and is on the front of the culture medium is the actual position, that is, the position of the bacteria picking point on the front of the culture medium. Figure 13 As shown, the red dot at the camera is the image, and the red dot at one end of the sample tube and the holder is the physical location of the front of the culture medium (green line) (the picking point location). Step S106 obtains the translation change vector and the rotation change vector. Knowing the pixel coordinates of the picking point, the coordinates of the picking point in the robot coordinate system can be obtained.

[0171] like Figure 14 As shown in the figure, when picking bacteria, if the tilt direction of the picking stick is fixed, there will be some blind areas on the culture medium plane. Figure 15 As shown, the present invention can also keep the position of the end of the picking stick unchanged while changing the tilt angle of the picking stick, so that the picking operation can be performed on the entire culture medium plane without blind spots. When picking, the picking ring at the end of the picking stick is allowed to slide a short distance on the culture medium plane with a certain pressure to achieve the best picking effect.

Claims

1. A method for identifying and locating Mycobacterium tuberculosis colonies on an inclined plane, characterized in that: include: S100: The actual spatial position of the picking stick is converted into a rotational displacement matrix of the robot's world coordinate system. Fixed marking points are selected around the sample tube, and the plane equation of the plane where the sample tube is located is determined to facilitate the robot's operation of positioning the picking stick and image processing of the camera. S200: The culture surface in the tilted sample tube is an inclined plane. The camera continuously photographs the rotating sample tube, continuously reads the camera frame image, and uses the color difference between the culture medium and the background to calculate the ratio of the background point to the image area to obtain the signal intensity of the background area. Based on the signal intensity, the culture medium surface facing the camera is determined. S300: The camera takes a picture of the side of the culture medium, fits a straight line to the culture surface, obtains the pixel coordinate values ​​of the two end points of the straight line, and then determines the plane equation of the front of the culture medium; S400: The camera takes a picture of the front of the culture medium to determine the pixel coordinates of the bacteria picking point. According to the plane equation of the front of the culture medium, the pixel coordinates are converted into robot world coordinates, and the robot world coordinates are used to control the manipulator to perform the bacteria picking action.

2. The method for identifying and locating Mycobacterium tuberculosis colonies on an inclined plane according to claim 1, characterized in that: Step S100 specifically includes: S101: Taking the center of the top of the manipulator as the origin, convert the actual spatial position of the end of the picking stick into the coordinates of the robot's world coordinate system; S102: Mark four marking points on both sides of the sample tube and determine the coordinates of the four marking points in the robot world coordinate system; S103: Determine the center point P of the square formed by the four marking points c ; Determine the vectors of the four sides of the square; Determine the center point P c Four internal diagonal vectors to the four marked points respectively; S104: Multiply a side vector by the corresponding internal diagonal vector to obtain a corresponding normal vector; arithmetic average the four normal vectors to obtain the normal vectors of the plane where the four marker points are located; S105: plane normal vector and center point P c Perform fitting to obtain the plane equation of the plane where the four marked points are located; S106: Determine the conversion relationship between the camera coordinate system and the robot world coordinate system based on the four marking points; S107: Select the center point of the sample tube opening as P5, the center point of the sample tube bottom as P6, and the straight line connecting P5 and P6 as the rotation axis of the sample tube to obtain the rotation axis vector.

3. The method for identifying and locating Mycobacterium tuberculosis colonies on an inclined plane according to claim 2, characterized in that: Step S101 is specifically as follows: the rotation displacement matrix of the end of the picking stick based on the robot world coordinate system = the actual rotation displacement matrix of the end of the picking stick × the actual rotation displacement matrix of the end of the robot arm According to the position offset relationship between the end of the picking stick and the origin, the actual rotation displacement matrix of the end of the picking stick is obtained as follows: Wherein, y1 is the distance in m that the end of the picking stick deviates from the origin in the Y-axis direction; z1 is the distance in m that the end of the picking stick deviates from the origin in the Z-axis direction; According to the position offset relationship between the end of the manipulator arm and the origin, the actual rotation displacement matrix of the end of the manipulator arm is obtained as follows: Where y2 is the distance in m that the end of the robot arm deviates from the origin in the Y-axis direction; z2 is the distance in m that the end of the robot arm deviates from the origin in the Z-axis direction; The rotational displacement matrix of the end of the picking stick based on the robot's world coordinate system is a matrix with four rows and four columns. The values ​​in the first row and fourth column, the values ​​in the second row and fourth column, and the values ​​in the third row and fourth column respectively correspond to the coordinate values ​​(x0', y0', z0') of the end of the picking stick based on the robot's world coordinate system.

4. The method for identifying and locating Mycobacterium tuberculosis colonies on an inclined plane according to claim 2, wherein: In step S102, vertical lines are drawn from the center of the bottom end of the sample tube to the two long sides of the experimental box. The vertical lines are perpendicular to the long sides of the experimental box. The intersections of the vertical lines with the two long sides are marked as two points, denoted as P2 and P3. Draw vertical lines from the top of the backrest to the two long sides of the experimental box. The vertical lines are perpendicular to the long sides. The intersections of the vertical lines with the two long sides are marked as two other points, denoted as P1 and P4. According to the method of step S101, the coordinates and image coordinates of the four marking points in the robot world coordinate system are obtained.

5. The method for identifying and locating Mycobacterium tuberculosis colonies on an inclined plane according to claim 4, characterized in that: In step S103, within the square, the position where the diagonal line connecting P1 and P3 intersects with the diagonal line connecting P2 and P4 is the center point P c ; The vector pointing from P4 to the side of P1 is v1, the vector pointing from P1 to the side of P2 is v2, the vector pointing from P2 to the side of P3 is v3, and the vector pointing from P3 to the side of P4 is v4; P c The internal diagonal vector pointing to P4 is vc1, P c The internal diagonal vector pointing to P1 is vc2, P c The internal diagonal vector pointing to P2 is vc3, P c The internal diagonal vector pointing to P3 is vc4.

6. The method for identifying and locating Mycobacterium tuberculosis colonies on an inclined plane according to claim 4, characterized in that: Step S106 is specifically: r_vec, t_vec = cv2.solvePnP (robot world coordinate 3D column, pixel coordinate 2D column, camera intrinsic parameter matrix, camera distortion coefficient), Among them, the robot world coordinate 3D column is the three-dimensional coordinate column of the robot world coordinate system, with four marker points P1, P2, P3, and P4; the pixel coordinate 2D column is the pixel coordinate column of the four marker points on the image; the camera intrinsic parameter matrix and camera distortion coefficient are the camera's own; r_vec is the rotation change vector, and t_vec is the translation change vector.

7. The method for identifying and locating Mycobacterium tuberculosis colonies on an inclined plane according to claim 1, wherein: Step S200 includes: S201: When photographing the sample tube, provide a light-proof environment for the camera and the sample tube, so that the area outside the culture medium is black; define a viewing area to ensure that there is always a background area in the viewing area; S202: Rotate the sample tube with the central axis of the sample tube as the axis. During the rotation, the camera continuously takes pictures of the sample tube, continuously reads the camera frame image, and quantifies the color category of each pixel in the visible area by using the color difference between the culture medium and the background; S203: Using the KMeans algorithm, clustering is performed based on the colors in the visible area. The darkest cluster in the center of the cluster is the background cluster. The ratio of the pixels in the background cluster to all the pixels in the visible area is calculated to obtain the signal strength of the background area. S204: With the shooting time as the horizontal coordinate and the signal strength of the background area as the vertical coordinate, a relationship graph between the signal strength of the background area and time is obtained; the curve of the relationship graph has several continuous cycles, each cycle has two peaks and two troughs, the lowest trough indicates that the back is facing up, and the second lowest trough indicates that the front is facing up.

8. The method for identifying and locating Mycobacterium tuberculosis colonies on an inclined plane according to claim 6, wherein: Step S300 includes: S301: Turn the culture medium to a side-up position, take a picture, and capture the effective area in the picture; S302: Cluster the colors in the valid area using the KMeans algorithm to select the black area. After Gaussian smoothing, obtain all points on the edge of the black area and then perform a straight line fitting on the edge of the black area. S303: Projecting the two endpoints of the edge of the black area onto the plane where the sample tube is located to obtain two projection points, and then combining them with the center point of the camera to obtain the culture medium plane equation; S304: Turn the culture medium to a state where the front side faces upward, and rotate the plane equation of the culture medium by the same angle to obtain the plane equation of the front side of the culture medium.

9. The method for identifying and locating Mycobacterium tuberculosis colonies on an inclined plane according to claim 8, characterized in that: In step S302, the fitLine function is used to fit a straight line to the edge of the black area. Then, based on the image captured by the camera, the maximum and minimum line numbers of the two endpoints of the edge of the black area are substituted into the fitted straight line equation to obtain the pixel coordinate values ​​of the two endpoints.

10. The method for identifying and locating Mycobacterium tuberculosis colonies on an inclined plane according to claim 9, characterized in that: In step S303, from a top-down perspective, the line of sight is perpendicular to the plane where the sample tube is located, and the plane extends to the position of the camera, with the center point of the camera as the camera origin Pnew0. Starting from Pnew0, a straight line is connected to the endpoints at the top edge of the black area, and then the line is extended to the plane where the four marked points are located to obtain the projection point Pnew1. Starting from Pnew0, use a straight line to connect the endpoints at the bottom edge of the black area, and then extend the straight line to the plane where the four marked points are located to obtain the projection point Pnew2; Step S106 obtains the translation change vector and the rotation change vector. Given the pixel coordinates of Pnew1 and Pnew2, the coordinates of Pnew1 and Pnew2 in the robot coordinate system can be obtained. Similarly, given the pixel coordinates of Pnew0, the coordinates of Pnew0 in the robot coordinate system can be obtained. According to the coordinates of Pnew0, Pnew1 and Pnew2 in the robot coordinate system, the equation of the culture medium plane can be obtained.

Images