A three-dimensional positioning method for transparent cell culture dishes based on pure binocular vision

Abstract

The present invention provides a three-dimensional positioning method for a transparent cell culture dish based on pure binocular vision, which belongs to the field of image processing technology and specifically includes: acquiring images through a binocular camera; labeling a sample data set; training a target model using a training set, and inputting the corrected image into the trained target model to extract a local image of the transparent cell culture dish; generating a heat map corresponding to a prediction frame, and extracting the two-dimensional corner coordinates corresponding to the prediction frame based on this; transforming the two-dimensional corner coordinates into the binocular camera coordinate system; using the three-dimensional corner coordinates to calculate the key point information of the transparent cell culture dish in the binocular camera coordinate system; obtaining the coordinate transformation matrix of the robotic arm base relative to the binocular camera, and transforming the key point information in the binocular camera coordinate system into the key point information in the base coordinate system based on this; and controlling the robotic arm to grasp the transparent cell culture dish. The solution of the present invention improves positioning efficiency, accuracy, and adaptability.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to a three-dimensional positioning method for a transparent cell culture dish based on pure binocular vision. Background Art

[0002] Currently, machine vision-based ranging methods use visible light cameras to capture image information of the surrounding environment, leverage camera imaging principles to build a three-dimensional relational model of the target, and then obtain the object's three-dimensional coordinate information in space through transformations between coordinate systems. This method is widely used for target identification and positioning due to its wide range of applications, reliable positioning accuracy, and low cost. However, the identification and grasping of transparent objects has always been a technical challenge in the fields of computer vision and robotic control. Due to the optical properties of transparent objects, their surfaces reflect and refract ambient light, making it difficult to extract their features using traditional methods based on color, texture, or edge detection. Furthermore, accurate three-dimensional positioning of transparent objects is crucial for automated robotic grasping tasks, and traditional vision systems struggle to meet the high-precision and high-speed grasping requirements.

[0003] It can be seen that there is an urgent need for a three-dimensional positioning method for transparent cell culture dishes based on pure binocular vision with high positioning efficiency, accuracy, speed and adaptability. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a three-dimensional positioning method for a transparent cell culture dish based on pure binocular vision, which at least partially solves the problems of poor positioning efficiency, accuracy, speed and adaptability in the prior art.

[0005] The embodiment of the present invention provides a three-dimensional positioning method for a transparent cell culture dish based on pure binocular vision, comprising:

[0006] Step 1: Use a binocular camera to collect real-time images of the transparent cell culture dish, and perform distortion correction on each frame of the image to obtain a corrected image to form a sample data set;

[0007] Step 2: Label the sample dataset using labeling software, use the transparent cell culture dish as the detection target and store it regionally in the form of a bounding box, and obtain the labeled bounding box as the training set;

[0008] Step 3: Build a lightweight backbone network in the target model, use the training set to train the target model, input the corrected image into the trained target model, extract the local image of the transparent cell culture dish and store it in the form of a prediction box;

[0009] Step 4: Use the deep learning network model to generate a heat map corresponding to the prediction box, and extract the two-dimensional corner coordinates corresponding to the prediction box based on it;

[0010] Step 5: Use the triangular similarity projection conversion algorithm to convert the two-dimensional corner point coordinates into the binocular camera coordinate system to obtain the three-dimensional corner point coordinates of the transparent cell culture dish in the binocular camera coordinate system;

[0011] Step 6, using the three-dimensional corner point coordinates to calculate the key point information of the transparent cell culture dish in the binocular camera coordinate system, wherein the key point information includes the normal vector and corner point coordinates of the bottom of the transparent cell culture dish;

[0012] Step 7: Use the calibration plate to perform hand-eye calibration on the coordinates of the manipulator in real space, obtain the coordinate transformation matrix of the manipulator base relative to the binocular camera, and convert the key point information in the binocular camera coordinate system to the key point information in the base coordinate system.

[0013] Step 8: Control the robotic arm to grab the transparent cell culture dish according to the key point information in the base coordinate system;

[0014] The step 8 specifically includes:

[0015] The center point of the bottom of the transparent cell culture dish in the base coordinate system is used as the grasping point, and its unit normal vector is used as the attitude angle. Based on this, the final execution trajectory planning from the end of the robotic arm to the grasping target is generated, and the robotic arm is controlled to grasp the transparent cell culture dish accordingly.

[0016] According to a specific implementation of the embodiment of the present invention, step 1 specifically includes:

[0017] Step 1.1: Use a binocular camera to capture real-time images of the transparent cell culture dish, and calculate the normalized coordinates after radial distortion correction based on the original pixel coordinates of each frame of the image;

[0018] In step 1.2, the normalized coordinates after radial distortion correction are converted to the pixel coordinate system using the camera's intrinsic parameter matrix to obtain the corrected image and form a sample data set.

[0019] According to a specific implementation of the embodiment of the present invention, step 2 specifically includes:

[0020] The transparent cell culture dish is used as the detection target, and each frame image in the sample dataset is annotated using a bounding box through annotation software. The annotation file is stored in txt format as a training set.

[0021] According to a specific implementation of the embodiment of the present invention, step 3 specifically includes:

[0022] Step 3.1: Input the training set into the lightweight backbone network of the target model to obtain the predicted target box and calculate the confidence score;

[0023] Step 3.2, calculate the loss function based on the predicted target box and confidence score and train the target model accordingly, where the loss function is

[0024] ;

[0025] in, is the bounding box regression loss, , represents the intersection-over-union ratio, which is used to evaluate the degree of overlap between two bounding boxes. represents the target bounding box predicted by the model, represents the acquired true target bounding box, is the cross entropy loss, , represents the true value of the label, represents the predicted value of the model, is the confidence loss, is the weight coefficient;

[0026] In step 3.3, the rectified image is input into the trained target model, and the local image of the transparent cell culture dish is extracted and stored in the form of a prediction box.

[0027] According to a specific implementation of the embodiment of the present invention, step 4 specifically includes:

[0028] Step 4.1: Based on the center point coordinates of each prediction box, use the CenterNet network model to generate a heat map of the corner points of each prediction box;

[0029] In step 4.2, based on the heat map, the two-dimensional corner coordinates corresponding to the prediction box are obtained through the non-maximum suppression method.

[0030] According to a specific implementation of the embodiment of the present invention, step 5 specifically includes:

[0031] Step 5.1, given the intrinsic parameter matrix of the binocular camera , rotation matrix , translation vector , and the two-dimensional corner coordinates identified by the binocular camera and , according to the binocular imaging principle and the principle of similar triangles, the transformation relationship between binocular cameras is obtained:

[0032] ;

[0033] in, Indicates the focal length of the right camera; Indicates the focal length of the left camera; Represents the component of the focal length of the right camera along the y-axis in the imaging plane; Represents the component of the focal length of the left camera along the y-axis in the imaging plane; represents the component of the focal length of the right camera along the x-axis in the imaging plane; represents the component of the focal length of the left camera along the x-axis in the imaging plane, are the pixel sizes of each pixel in the imaging plane of the binocular camera;

[0034] Step 5.2: According to the transformation relationship between the binocular cameras, the three-dimensional coordinate points in the left camera coordinate system are converted to the right camera coordinate system:

[0035] ;

[0036] ;

[0037] in, Represents the rotation matrix The Rank Elements of the column; is the translation vector The element in row and column 1 ,in, It is obtained according to the following formula:

[0038] ;

[0039] in, are the optical center coordinates of the binocular cameras;

[0040] In step 5.3, the expression for calculating the three-dimensional corner coordinates of the transparent cell culture dish in the binocular camera coordinate system is:

[0041] ;

[0042] ;

[0043] Where, are the three-dimensional coordinate points of the real target in the binocular camera;

[0044] Step 5.4, obtain the corresponding 3D corner coordinates according to the expression of the 3D corner coordinates of the transparent cell culture dish in the binocular camera coordinate system .

[0045] According to a specific implementation of an embodiment of the present invention, the calculation formula of the key point information in the binocular camera coordinate system is:

[0046] ;

[0047] in, is the three-dimensional corner coordinate of the bottom of the transparent cell culture dish in the binocular camera coordinate system, is the unit normal vector of the bottom of the transparent cell culture dish in the binocular camera coordinate system, The coordinates of the center point of the bottom of the transparent cell culture dish in the binocular camera coordinate system.

[0048] According to a specific implementation of the embodiment of the present invention, step 7 specifically includes:

[0049] Step 7.1: Solve the coordinate transformation matrix of the manipulator base relative to the binocular camera based on the manipulator hand-eye calibration ;

[0050] Step 7.2, according to the coordinate transformation matrix Convert the key point information in the binocular camera coordinate system to the key point information in the base coordinate system

[0051] ;

[0052] in, , Represents the three-dimensional corner coordinates of the transparent cell culture dish in the binocular camera coordinate system, Represents the component along the X-axis in the manipulator base coordinate system, Represents the component along the Y-axis in the robot base coordinate system, Represents the component along the Z-axis in the robot base coordinate system.

[0053] The three-dimensional positioning scheme of a transparent cell culture dish based on pure binocular vision in an embodiment of the present invention includes: step 1, collecting real-time images of the transparent cell culture dish through a binocular camera, and performing distortion correction on each frame of the image to obtain a corrected image to form a sample data set; step 2, labeling the sample data set using annotation software, taking the transparent cell culture dish as the detection target and storing it regionally in the form of a bounding box, and obtaining the labeled bounding box as a training set; step 3, constructing a lightweight backbone network in the target model, training the target model using the training set, and inputting the corrected image into the trained target model, extracting the local image of the transparent cell culture dish and storing it in the form of a prediction box; step 4, generating a heat map corresponding to the prediction box using a deep learning network model, and extracting the heat map based on the heat map. The two-dimensional corner coordinates corresponding to the prediction box; step 5, using the triangular similarity projection conversion algorithm to transform the two-dimensional corner coordinates into the binocular camera coordinate system, and obtain the three-dimensional corner coordinates of the transparent cell culture dish in the binocular camera coordinate system; step 6, using the three-dimensional corner coordinates to calculate the key point information of the transparent cell culture dish in the binocular camera coordinate system, wherein the key point information includes the unit normal vector and the center point coordinates of the bottom of the transparent cell culture dish; step 7, using the calibration plate to perform hand-eye calibration on the coordinates of the robotic arm in real space, obtain the coordinate transformation matrix of the robotic arm base relative to the binocular camera, and convert the key point information in the binocular camera coordinate system to the key point information in the base coordinate system based on this; step 8, according to the key point information in the base coordinate system, control the robotic arm to grab the transparent cell culture dish.

[0054] The beneficial effects of the embodiments of the present invention are as follows: through the solution of the present invention, a diversified data set is constructed and feature learning is performed in combination with deep learning technology, which improves the detection accuracy of the edges and key points of the transparent cell culture dish and reduces the impact of the transparent cell culture dish on target recognition due to light refraction and transmission. In addition, the key point detection deep learning model Centernet is combined with the prediction box-based algorithm structure to obtain the pixel points corresponding to the real coordinate points in the left and right images. By designing a simple TSP conversion algorithm, the heavy computational burden and instability brought about by the use of stereo matching for disparity calculation are avoided, and efficient positioning with low computational burden is achieved to meet real-time control requirements. Based on the three-dimensional coordinates and normal vector information of the transparent cell culture dish, the present invention realizes the conversion of two-dimensional pixel coordinates to robot base coordinates through hand-eye calibration technology, so that the robot arm can be adaptively adjusted according to the posture of the transparent cell culture dish, thereby improving positioning efficiency, accuracy and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 A schematic diagram of a process for three-dimensional positioning of a transparent cell culture dish based on pure binocular vision provided by an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of a specific implementation process of a method for three-dimensional positioning of a transparent cell culture dish based on pure binocular vision provided by an embodiment of the present invention;

[0058] Figure 3 A structural diagram of a binocular vision TSP algorithm provided by an embodiment of the present invention;

[0059] Figure 4 A diagram of a key point detection network structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0061] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0062] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present invention, those skilled in the art will appreciate that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0063] It should also be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. The illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0064] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0065] An embodiment of the present invention provides a three-dimensional positioning method for a transparent cell culture dish based on pure binocular vision, which can be applied to the positioning process of transparent objects in machine vision scenes.

[0066] See also Figure 1 , is a flow chart of a method for three-dimensional positioning of a transparent cell culture dish based on pure binocular vision provided by an embodiment of the present invention. Figure 1 and Figure 2 As shown, the method mainly includes the following steps:

[0067] Step 1: Use a binocular camera to collect real-time images of the transparent cell culture dish, and perform distortion correction on each frame of the image to obtain a corrected image to form a sample data set;

[0068] In the specific implementation, a binocular camera is used to collect real-time images of a transparent cell culture dish to construct data samples for the online detection model. Compared with objects with distinct colors, it is difficult to extract feature information of a transparent cell culture dish and the feature learning process is easily affected by external interference. Therefore, when constructing the data set, it is necessary to cover as many real-time images of the transparent cell culture dish as possible under different backgrounds and different postures. Distortion correction is performed on each frame of the image to obtain the corrected image. , as the training sample of the target detection model. In step 1, the image collected by the binocular camera system needs to be distorted to obtain The formula is as follows:

[0069] ;

[0070] In formula (1), is the original pixel coordinate; is the normalized coordinate after radial distortion correction; is the square distance from the pixel to the optical center; is the radial distortion parameter (obtained through camera calibration). In order to normalize the corrected coordinates Convert to the pixel coordinate system and use the camera's intrinsic parameter matrix (obtained through camera calibration) to obtain the corrected image, that is:

[0071] ;

[0072] In formula (2), is the corrected pixel coordinate; is the focal length of the camera; is the coordinate of the camera optical center; Both can be obtained through camera calibration.

[0073] Step 2: Label the sample dataset using labeling software, use the transparent cell culture dish as the detection target and store it regionally in the form of a bounding box, and obtain the labeled bounding box as the training set;

[0074] In specific implementation, the labeling software labelme can be used to label the collected images, and the transparent cell culture dish can be used as the detection target and stored regionally in the form of a bounding box to obtain the labeled bounding box. , so that the deep learning network model can learn the target features. In step 2, the target detection model uses the bounding box to mark the target, and the marking file is stored in the form of txt. It can be expressed as:

[0075] ;

[0076] is the coordinate of the upper left corner of the rectangular box; The coordinates of the upper right corner of the rectangle.

[0077] Step 3: Build a lightweight backbone network in the target model, use the training set to train the target model, input the corrected image into the trained target model, extract the local image of the transparent cell culture dish and store it in the form of a prediction box;

[0078] In the specific implementation, a lightweight backbone neural network in the yolov5 model is constructed, and the model is trained using the collected data samples. The model extracts the feature information of the transparent cell culture dish, and uses the corrected image as input to extract the local image information of the target and store it in the form of a prediction frame. In step 3, yolov5 extracts the features of the transparent cell culture dish through the lightweight backbone network and learns the mapping relationship:

[0079] ;

[0080] In formula (4), is the predicted target bounding box; is the category label; is the confidence score. Its loss function is defined as follows:

[0081] ;

[0082] In formula (5), is the bounding box regression loss, which can be expressed as: ; is the cross entropy loss, expressed as ; is the true label value; is the model prediction value is the confidence loss; is the corresponding weight coefficient.

[0083] Step 4: Use the deep learning network model to generate a heat map corresponding to the prediction box, and extract the two-dimensional corner coordinates corresponding to the prediction box based on it;

[0084] In the specific implementation, after extracting the local image of the target, the corner coordinates of the transparent cell culture dish are used as the data set for key point detection, and the deep learning CenterNet network model is used to generate a heat map. To extract its corner coordinates, such as Figure 4 In step 4, based on target detection, the CenterNet network model is used to extract the corner coordinates, which can be expressed as:

[0085] Hypothesized detection box The center point is: , then the Gaussian heat map of the corner points generated by CenterNet can be expressed as:

[0086] ;

[0087] In formula (6), is a factor that controls the diffusion degree of the Gaussian kernel; Heatmap generated for the network; Heat map The pixel coordinates of a point on . The corner coordinates are obtained by non-maximum suppression (NMS)

[0088] ;

[0089] In formula (7), is the number of corner points, These corner point coordinates are used to reconstruct the 3D position of the transparent cell culture dish and calculate its 3D position information.

[0090] Step 5: Use the triangular similarity projection conversion algorithm to convert the two-dimensional corner point coordinates into the binocular camera coordinate system to obtain the three-dimensional corner point coordinates of the transparent cell culture dish in the binocular camera coordinate system;

[0091] In the specific implementation, combined with binocular vision, a simple conversion algorithm from two-dimensional pixel coordinates to three-dimensional camera coordinate system is used. Its structure is as follows Figure 3 As shown, the corner coordinates extracted from the left and right target images are converted to the camera coordinate system to obtain the corner coordinates of the transparent cell culture dish in the camera coordinate system. In step 5, a concise two-dimensional pixel coordinate The TSP conversion algorithm to the camera coordinate system is as follows. This algorithm does not require the traditional binocular vision algorithm to perform extreme correction to obtain the left and right parallax, and then solve the three-dimensional coordinates in the right camera coordinate system. First, given the camera's intrinsic parameter matrix , rotation matrix , translation vector (obtained through camera calibration), and the pixels identified by the binocular camera and , according to the binocular imaging principle Figure 3 As shown, using the principle of similar triangles, we can get:

[0092] ;

[0093] According to formula (8), we can get:

[0094] ;

[0095] According to the transformation relationship between binocular cameras, the three-dimensional coordinate points in the left camera coordinate system are converted to the right camera coordinate system:

[0096] ;

[0097] According to formula (9), we can get:

[0098] ;

[0099] In formulas (8), (9), (10), and (11), represents the component of the focal length of the right camera along the y-axis in the imaging plane, Represents the component of the focal length of the left camera along the y-axis in the imaging plane; represents the component of the focal length of the right camera along the x-axis in the imaging plane; represents the component of the focal length of the left camera along the x-axis in the imaging plane; are the pixel sizes of each pixel in the imaging plane of the binocular camera; Represents the rotation matrix The Rank Elements of the column; is the translation vector The element in row and column 1 .in, It is obtained according to the following formula:

[0100] ;

[0101] In formula (12), are the optical center coordinates of the binocular camera. Therefore, the three-dimensional coordinates of the binocular camera coordinate system can be calculated as:

[0102] ;

[0103] In formula (13), are the three-dimensional coordinate points of the real target in the binocular camera, so its three-dimensional coordinates are expressed as .

[0104] Step 6, using the three-dimensional corner point coordinates to calculate the key point information of the transparent cell culture dish in the binocular camera coordinate system, wherein the key point information includes the unit normal vector and the center point coordinates of the bottom of the transparent cell culture dish;

[0105] In the specific implementation, the normal vector and corner coordinates of the bottom of the transparent cell culture dish are obtained using the obtained corner coordinates of the transparent cell culture dish in the camera coordinates, and these coordinates are sent to the driver of the six-axis robot arm. In step 6, the normal vector of the bottom of the transparent cell culture dish is solved as follows:

[0106] The three-dimensional coordinate set of the corner points of the transparent cell culture dish is known to be , its bottom normal vector can be calculated as:

[0107] ;

[0108] In formula (14), are the coordinates of the three corner points on the bottom of the transparent cell culture dish; is its normal vector; Represents a three-dimensional point set in the camera coordinate system; Represents the three-dimensional coordinates of the i-th key point in the camera coordinate system, which can be expressed as ;

[0109] In step 6, the coordinates of the corner point at the bottom center of the transparent cell culture dish are solved as follows:

[0110] ;

[0111] In formula (15), is the three-dimensional corner coordinate of the center of the bottom of the transparent cell culture dish;

[0112] Step 7: Use the calibration plate to perform hand-eye calibration on the coordinates of the manipulator in real space, obtain the coordinate transformation matrix of the manipulator base relative to the binocular camera, and convert the key point information in the binocular camera coordinate system to the key point information in the base coordinate system.

[0113] In the specific implementation, the calibration plate is used to calibrate the coordinates of the manipulator in the real space, obtain the coordinate transformation relationship of the manipulator base relative to the camera, and convert the key point information in the camera coordinate system to the key point information in the base coordinate system. In step 7, the transformation matrix from camera coordinates to manipulator coordinates is solved by manipulator hand-eye calibration. as follows:

[0114] ;

[0115] In formula (16), It can be obtained by the hand-eye calibration method, that is, . is the transformation matrix between the camera and the robotic arm motion; is the hand-eye transformation matrix; .

[0116] Step 8: Control the robotic arm to grab the transparent cell culture dish based on the key point information in the base coordinate system.

[0117] In specific implementation, the three-dimensional coordinates of the bottom corner of the transparent cell culture dish relative to the base coordinates of the robot arm are used to drive the robot arm to perform movement, thereby achieving automatic grasping of the transparent cell culture dish. The inverse kinematics of the robotic arm is calculated using the unit normal vector of the bottom of the transparent cell culture dish as follows:

[0118] ;

[0119] In formula (17), is the joint angle; is the offset of the joint; is the connecting rod length; is the connecting rod torsion angle, Indicates the The vertical coordinate in the positioning coordinates; Indicates the The horizontal coordinate in the positioning coordinates; Indicates the The vertical coordinate in the positioning coordinates; Represents the normal vector of the bottom of the transparent culture dish obtained by identification; Represents a vector The value of the first row; Represents a vector The value of the second row, Represents a vector The value of the third row; Represents a vector The value of the first row; Represents a vector The value of the second row; Represents a vector The value of the third row; Represents a vector The value of the first row; Represents a vector The value of the second row; Represents a vector The value of the third row;

[0120] The final execution trajectory planning of the joint angle of the robot arm is: , represents the initial angle, represents the initial velocity of the joint, Indicates the acceleration of the joint.

[0121] The three-dimensional positioning method of transparent cell culture dishes based on pure binocular vision provided in this embodiment improves the detection accuracy of the edges and key points of transparent cell culture dishes by constructing a diverse data set and combining deep learning technology for feature learning, and reduces the impact of light refraction and transmission on target recognition of transparent cell culture dishes. In addition, the key point detection deep learning model Centernet is combined with the algorithm structure based on the prediction box to obtain the pixel points corresponding to the real coordinate points in the left and right images. By designing a simple TSP conversion algorithm, the heavy computational burden and instability brought about by the use of stereo matching for disparity calculation are avoided, and efficient positioning under low computational burden is achieved to meet real-time control requirements. Based on the three-dimensional coordinates and normal vector information of the transparent cell culture dish, the present invention realizes the conversion of two-dimensional pixel coordinates to robot base coordinates through hand-eye calibration technology, so that the robot arm can be adaptively adjusted according to the posture of the transparent cell culture dish, thereby improving positioning efficiency, accuracy and adaptability.

[0122] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware or a combination thereof.

[0123] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A three-dimensional positioning method for transparent cell culture dishes based on pure binocular vision, characterized in that: include: Step 1: Use a binocular camera to collect real-time images of the transparent cell culture dish, and perform distortion correction on each frame of the image to obtain a corrected image to form a sample data set; Step 2: Label the sample dataset using labeling software, use the transparent cell culture dish as the detection target and store it regionally in the form of a bounding box, and obtain the labeled bounding box as the training set; Step 3: Build a lightweight backbone network in the target model, use the training set to train the target model, input the corrected image into the trained target model, extract the local image of the transparent cell culture dish and store it in the form of a prediction box; Step 4: Use the deep learning network model to generate a heat map corresponding to the prediction box, and extract the two-dimensional corner coordinates corresponding to the prediction box based on it; Step 5: Use the triangular similarity projection conversion algorithm to convert the two-dimensional corner point coordinates into the binocular camera coordinate system to obtain the three-dimensional corner point coordinates of the transparent cell culture dish in the binocular camera coordinate system; Step 6, using the three-dimensional corner point coordinates to calculate the key point information of the transparent cell culture dish in the binocular camera coordinate system, wherein the key point information includes the unit normal vector and the center point coordinates of the bottom of the transparent cell culture dish; Step 7: Use the calibration plate to perform hand-eye calibration on the coordinates of the manipulator in real space, obtain the coordinate transformation matrix of the manipulator base relative to the binocular camera, and convert the key point information in the binocular camera coordinate system to the key point information in the base coordinate system. Step 8: Control the robotic arm to grab the transparent cell culture dish according to the key point information in the base coordinate system; The step 8 specifically includes: The center point of the bottom of the transparent cell culture dish in the base coordinate system is used as the grasping point, and its unit normal vector is used as the attitude angle. Based on this, the final execution trajectory planning from the end of the robotic arm to the grasping target is generated, and the robotic arm is controlled to grasp the transparent cell culture dish accordingly.

2. The method according to claim 1, characterized in that The step 1 specifically includes: Step 1.1: Use a binocular camera to capture real-time images of the transparent cell culture dish, and calculate the normalized coordinates after radial distortion correction based on the original pixel coordinates of each frame of the image; In step 1.2, the normalized coordinates after radial distortion correction are converted to the pixel coordinate system using the camera's intrinsic parameter matrix to obtain the corrected image and form a sample data set.

3. The method according to claim 2, characterized in that The step 2 specifically includes: The transparent cell culture dish is used as the detection target, and each frame image in the sample dataset is annotated using a bounding box through annotation software. The annotation file is stored in txt format as a training set.

4. The method according to claim 3, characterized in that The step 3 specifically includes: Step 3.1: Input the training set into the lightweight backbone network of the target model to obtain the predicted target box and calculate the confidence score; Step 3.2, calculate the loss function based on the predicted target box and confidence score and train the target model accordingly, where the loss function is ； in, is the bounding box regression loss, , represents the intersection-over-union ratio, which is used to evaluate the degree of overlap between two bounding boxes. represents the target bounding box predicted by the model, represents the acquired true target bounding box, is the cross entropy loss, , represents the true value of the label, represents the predicted value of the model, is the confidence loss, is the weight coefficient; In step 3.3, the rectified image is input into the trained target model, and the local image of the transparent cell culture dish is extracted and stored in the form of a prediction box.

5. The method according to claim 4, characterized in that The step 4 specifically includes: Step 4.1: Based on the center point coordinates of each prediction box, use the CenterNet network model to generate a heat map of the corner points of each prediction box; In step 4.2, based on the heat map, the two-dimensional corner coordinates corresponding to the prediction box are obtained through the non-maximum suppression method.

6. The method according to claim 5, characterized in that The step 5 specifically includes: Step 5.1, given the intrinsic parameter matrix of the binocular camera , rotation matrix , translation vector , and the two-dimensional corner coordinates identified by the binocular camera and , according to the binocular imaging principle and the principle of similar triangles, the transformation relationship between binocular cameras is obtained: ； ； in, Indicates the focal length of the right camera; Indicates the focal length of the left camera; represents the component of the focal length of the right camera along the y-axis in the imaging plane, Represents the component of the focal length of the left camera along the y-axis in the imaging plane; represents the component of the focal length of the right camera along the x-axis in the imaging plane; represents the component of the focal length of the left camera along the x-axis in the imaging plane, are the pixel sizes of each pixel in the imaging plane of the binocular camera; Step 5.2: Based on the transformation relationship between the binocular cameras, transform the 3D coordinate points in the left camera coordinate system into the right camera coordinate system: ； ； in, Represents the rotation matrix The Rank Elements of the column; is the translation vector The element in row and column 1 ; In step 5.3, the expression for calculating the three-dimensional corner coordinates of the transparent cell culture dish in the binocular camera coordinate system is: ； Where, are the three-dimensional coordinate points of the real target in the left and right cameras respectively; Step 5.4, obtain the corresponding 3D corner coordinates according to the expression of the 3D corner coordinates of the transparent cell culture dish in the binocular camera coordinate system .

7. The method according to claim 6, characterized in that The calculation formula of the key point information in the binocular camera coordinate system is: ； in, is the three-dimensional corner coordinate of the bottom of the transparent cell culture dish in the binocular camera coordinate system, is the unit normal vector of the bottom of the transparent cell culture dish in the binocular camera coordinate system, The coordinates of the center point of the bottom of the transparent cell culture dish in the binocular camera coordinate system.

8. The method according to claim 7, characterized in that The step 7 specifically includes: Step 7.1: Solve the coordinate transformation matrix of the manipulator base relative to the binocular camera based on the manipulator hand-eye calibration ; Step 7.2, according to the coordinate transformation matrix Convert the key point information in the binocular camera coordinate system to the key point information in the base coordinate system ； in, , Represents the three-dimensional corner coordinates of the transparent cell culture dish in the binocular camera coordinate system, Represents the component along the X-axis in the manipulator base coordinate system, Represents the component along the Y-axis in the robot base coordinate system, Represents the component along the Z-axis in the robot base coordinate system.

Images