A pose recognition method of a transparent culture dish and a related device

Abstract

The application relates to the technical field of culture dish recognition, and provides a transparent culture dish pose recognition method and related equipment, which comprises the following steps: acquiring a left camera image and a right camera image of a transparent culture dish; performing segmentation and contour recognition on the left camera image to obtain a first picture construction point set of the left camera image, and performing segmentation and contour recognition on the right camera image to obtain a second picture construction point set of the right camera image; constructing an optimization problem and a constraint condition according to the first picture construction point set and the second picture construction point set; solving the optimization problem under the constraint of the constraint condition to obtain three-dimensional corner point coordinates of each corner point in the left camera image in a left camera coordinate system; and calculating the position and pose of the transparent culture dish in the left camera coordinate system according to all the three-dimensional corner point coordinates. The method can improve the accuracy of transparent culture dish pose recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of culture dish recognition, and in particular to a transparent culture dish pose recognition method and related equipment. BACKGROUND

[0002] As an important support for regenerative medicine and personalized medicine, the significance of stem cell automated preparation technology lies not only in improving preparation efficiency and consistency of quality, but also in reducing the risk and variability brought by human operation. With the wide application of stem cell therapy in clinical practice, large-scale and standardized production has become an urgent need. Through intelligent control and data-driven methods, automated technology realizes the whole-process management from cell culture, subculture to harvesting, providing a solid technical guarantee for high-throughput, precise and controllable cell preparation, and also promoting the development of the biopharmaceutical industry towards intelligent and digital transformation.

[0003] Medical transparent material culture dishes have become an important tool for stem cell culture and detection due to their excellent optical transmittance, biocompatibility and chemical stability. Such culture dishes usually use high-quality polycarbonate, glass or other transparent plastic materials, which can ensure cell growth in a non-toxic and non-polluted environment, and facilitate real-time monitoring of cell status using microscopic imaging, optical detection and other methods. In the laboratory and industrial production, transparent culture dishes help to accurately observe the cell morphology, proliferation and differentiation process, which is of great significance to improve the success rate of cell preparation and product quality, and also provides convenient conditions for subsequent cell function verification.

[0004] In the process of stem cell automated preparation, there are many technical challenges in identifying and positioning medical transparent material culture dishes using deep learning and machine vision technology. Due to the reflection, refraction and scattering effects of light on transparent materials, traditional image processing algorithms often fail to accurately capture the edges and subtle features of the culture dish, which affects the calculation and positioning of the object pose. In addition, binocular vision positioning technology is based on the principle of stereo disparity, which realizes three-dimensional reconstruction by analyzing the geometric differences between left and right images. Its core process includes two key links: first, use stereo matching algorithm to establish corresponding point pair relationship between binocular images, then calculate the depth information of the target object according to the principle of triangulation, and finally construct the three-dimensional space coordinates of the measured object. It is worth noting that existing real-time stereo matching algorithms need to traverse all pixel points in the candidate disparity range during pixel-level matching, and the search space grows quadratically with the image resolution, facing serious challenges in computing power and timeliness. More seriously, in low-texture areas, repeated structures or scenes with occlusions, false matching is easy to occur, making it difficult to reliably obtain correct matching point pairs, which is particularly evident on reflective surfaces. Therefore, there is a problem of low accuracy in the pose recognition of transparent culture dishes. SUMMARY

[0005] The application provides a transparent culture dish pose recognition method and related equipment, which can solve the problem of low accuracy of transparent culture dish pose recognition.

[0006] In a first aspect, the application provides a transparent culture dish pose recognition method, which comprises:

[0007] Obtaining a left camera image and a right camera image of the transparent culture dish;

[0008] Segmenting and recognizing the contour of the left camera image to obtain a first picture construction point set of the left camera image, and segmenting and recognizing the contour of the right camera image to obtain a second picture construction point set of the right camera image; the first picture construction point set comprises the coordinates of all contour points of the bottom surface of the transparent culture dish in the left camera image, and the second picture construction point set comprises the coordinates of all contour points of the bottom surface of the transparent culture dish in the right camera image;

[0009] Constructing an optimization problem and a constraint condition according to the first picture construction point set and the second picture construction point set; the optimization problem is that the sum of squares of distances between each contour point in the first picture construction point set and the second picture construction point set and the envelope edge to which the contour point belongs is minimum in a two-dimensional coordinate system, and the constraint condition is used to describe the contour of the transparent culture dish, and the envelope edge is an edge in a minimum envelope quadrilateral determined based on the left camera image or an edge in a minimum envelope quadrilateral determined based on the right camera image;

[0010] Solving the optimization problem under the constraint of the constraint condition to obtain three-dimensional corner point coordinates of each corner point in the left camera image in the left camera coordinate system;

[0011] Calculating the position and pose of the transparent culture dish in the left camera coordinate system according to all the three-dimensional corner point coordinates.

[0012] Optionally, the first picture construction point set of the left camera image obtained by segmenting and recognizing the contour of the left camera image comprises:

[0013] Performing semantic segmentation on the left camera image to obtain a semantic segmentation binary image;

[0014] Performing edge detection on the semantic segmentation binary image to obtain an edge binary image;

[0015] Performing contour extraction on the edge binary image to obtain the coordinates of all contour points of the bottom surface of the transparent culture dish in the left camera image;

[0016] Processing the coordinates of all the contour points by using a convex hull extraction algorithm to obtain a minimum envelope quadrilateral in the left camera image;

[0017] classifying all the contour points according to the minimum envelope quadrangle, determining the edge to which each contour point belongs from the minimum envelope quadrangle;

[0018] integrating the coordinates of all the contour points according to the edges to which the contour points belong, to obtain a first picture construction point set.

[0019] Optionally, an optimization problem and constraint conditions are constructed according to the first picture construction point set and the second picture construction point set, including:

[0020] calculating a distance expression between each contour point in the first picture construction point set and the edge to which the contour point belongs, and calculating a distance expression between each contour point in the second picture construction point set and the edge to which the contour point belongs;

[0021] constructing an index function according to all the distance expressions;

[0022] respectively for each corner point in the left camera image, constructing a left camera coordinate conversion relationship between the three-dimensional coordinates of the corner point in the left camera coordinate system and the two-dimensional coordinates of the corner point in the left camera image;

[0023] respectively for each corner point in the right camera image, constructing a right camera coordinate conversion relationship between the three-dimensional coordinates of the corner point in the left camera coordinate system and the two-dimensional coordinates of the corner point in the right camera image;

[0024] constructing an edge length expression in the left camera coordinate system;

[0025] constructing an optimization problem according to the index function, taking the left camera coordinate conversion relationship, the right camera coordinate conversion relationship and the edge length expression as constraint conditions of the optimization problem.

[0026] Optionally, the distance expression between each contour point in the first picture construction point set and the edge to which the contour point belongs is:

[0027]

[0028] wherein, represents the distance between the jth contour point belonging to the ith edge in the first picture construction point set and the ith edge, i = 1, 2, 3, 4, j = 1, 2, …, n i,L , n i,L represents the number of contour points belonging to the ith edge in the first picture construction point set, represents the horizontal coordinate of the jth contour point belonging to the ith edge in the first picture construction point set, represents the vertical coordinate of the jth contour point belonging to the ith edge in the first picture construction point set, v i,L represents the vertical coordinate of the corner point on the ith edge in the left camera image, u i,Ldenotes the horizontal coordinate of the corner point on the i-th edge in the left camera image, v i+1,L denotes the vertical coordinate of the corner point on the i-th edge in the left camera image, u i+1,L denotes the horizontal coordinate of the corner point on the i+1-th edge in the left camera image;

[0029] The distance expression between each contour point and the edge to which the contour point belongs in the second picture construction point set is as follows:

[0030]

[0031] wherein, denotes the distance between the k-th contour point belonging to the i-th edge and the i-th edge in the second picture construction point set, k = 1, 2, …, n i,R , n i,R denotes the number of contour points belonging to the i-th edge in the second picture construction point set, denotes the horizontal coordinate of the k-th contour point belonging to the i-th edge in the second picture construction point set, denotes the vertical coordinate of the k-th contour point belonging to the i-th edge in the second picture construction point set, v i,R denotes the vertical coordinate of the corner point on the i-th edge in the right camera image, u i,R denotes the horizontal coordinate of the corner point on the i-th edge in the right camera image, v i+1,R denotes the vertical coordinate of the corner point on the i+1-th edge in the right camera image, u i+1,R denotes the horizontal coordinate of the corner point on the i+1-th edge in the right camera image;

[0032] The index function is as follows:

[0033]

[0034] wherein, J(P 1,L ,P 2,L ,P 3,L ,P 4,L ) denotes the value of the index function.

[0035] Optionally, the left camera coordinate conversion relationship is as follows:

[0036]

[0037] wherein, x i,L denotes the horizontal coordinate of the corner point on the i-th edge in the left camera coordinate system, y i,L denotes the vertical coordinate of the corner point on the i-th edge in the left camera coordinate system, z i,L denotes the vertical coordinate of the corner point on the i-th edge in the left camera coordinate system, M L denotes the left camera internal parameter matrix:

[0038]

[0039] wherein f L,x represents the ratio of the focal length of the left camera to the horizontal pixel size, f L,y represents the ratio of the focal length of the left camera to the vertical pixel size, c L,x represents the horizontal coordinate of the center pixel of the left camera image, c L,y represents the vertical coordinate of the center pixel of the left camera image;

[0040] The right camera coordinate conversion relationship is:

[0041]

[0042] wherein f R,x represents the ratio of the focal length of the right camera to the horizontal pixel size, f R,y represents the ratio of the focal length of the right camera to the vertical pixel size, c R,x represents the horizontal coordinate of the center pixel of the right camera image, c R,y represents the vertical coordinate of the center pixel of the right camera image, r 1,1 represents the element of the first row and the first column in the rotation matrix, r 1,2 represents the element of the first row and the second column in the rotation matrix, r 1,3 represents the element of the first row and the third column in the rotation matrix, r 2,1 represents the element of the second row and the first column in the rotation matrix, r 2,2 represents the element of the second row and the second column in the rotation matrix, r 2,3 represents the element of the second row and the third column in the rotation matrix, r 3,1 represents the element of the third row and the first column in the rotation matrix, r 3,2 represents the element of the third row and the second column in the rotation matrix, r 3,3 represents the element of the third row and the third column in the rotation matrix, t1 represents the first element in the translation vector, t2 represents the second element in the translation vector, and t3 represents the third element in the translation vector;

[0043] The side length expression is:

[0044]

[0045] wherein L i represents the side length of the ith side of the bottom surface of the transparent culture dish, x i+1,L represents the horizontal coordinate of the corner point on the ith+1 side in the left camera coordinate system, y i+1,L represents the vertical coordinate of the corner point on the ith+1 side in the left camera coordinate system, z i+1,L represents the vertical coordinate of the corner point on the ith+1 side in the left camera coordinate system;

[0046] The optimization problem is:

[0047]

[0048] where P 1,L represents the three-dimensional corner point coordinates of the corner point on the first edge in the left camera coordinate system, P 2,L represents the three-dimensional corner point coordinates of the corner point on the second edge in the left camera coordinate system, P 3,L represents the three-dimensional corner point coordinates of the corner point on the third edge in the left camera coordinate system, and P 4,L represents the three-dimensional corner point coordinates of the corner point on the fourth edge in the left camera coordinate system.

[0049] Optionally, the position and pose of the transparent culture dish in the left camera coordinate system are calculated according to all the three-dimensional corner point coordinates, including:

[0050] The plane equation parameters of the bottom surface of the transparent culture dish in the left camera coordinate system are calculated according to all the three-dimensional corner point coordinates;

[0051] The normal vector and the center point coordinates of the bottom surface of the transparent culture dish in the left camera coordinate system are calculated according to the plane equation parameters;

[0052] The long edge vector and the short edge vector of the bottom surface of the transparent culture dish in the left camera coordinate system are calculated according to the normal vector;

[0053] The position and pose transformation matrix of the transparent culture dish in the left camera coordinate system is calculated based on the normal vector, the long edge vector, the short edge vector, and the center point coordinates; the first three columns of the position and pose transformation matrix are used to describe the pose of the transparent culture dish in the left camera coordinate system, and the last column of the position and pose transformation matrix is used to describe the position of the transparent culture dish in the left camera coordinate system.

[0054] Optionally, the plane equation parameters of the bottom surface of the transparent culture dish in the left camera coordinate system are calculated according to all the three-dimensional corner point coordinates, including:

[0055] The plane equation parameters a, b, and c are calculated by the formula:

[0056]

[0057]

[0058] where x 1,L represents the horizontal coordinate in the three-dimensional corner point coordinates of the corner point on the first edge in the left camera coordinate system, y 1,L represents the vertical coordinate in the three-dimensional corner point coordinates of the corner point on the first edge in the left camera coordinate system, and z 1,L represents the vertical coordinate in the three-dimensional corner point coordinates of the corner point on the first edge in the left camera coordinate system, x 2,L represents the horizontal coordinate in the three-dimensional corner point coordinates of the corner point on the second edge in the left camera coordinate system, y​2,L represents the vertical coordinate in the three-dimensional corner point coordinates of the corner point on the second edge under the left camera coordinate system, x 2,L represents the vertical coordinate in the three-dimensional corner point coordinates of the corner point on the second edge under the left camera coordinate system, x 3,L represents the horizontal coordinate in the three-dimensional corner point coordinates of the corner point on the third edge under the left camera coordinate system, y 3,L represents the vertical coordinate in the three-dimensional corner point coordinates of the corner point on the third edge under the left camera coordinate system, x 3,L represents the vertical coordinate in the three-dimensional corner point coordinates of the corner point on the third edge under the left camera coordinate system, x 4,L represents the horizontal coordinate in the three-dimensional corner point coordinates of the corner point on the fourth edge under the left camera coordinate system, y 4,L represents the vertical coordinate in the three-dimensional corner point coordinates of the corner point on the fourth edge under the left camera coordinate system, x 4,L represents the vertical coordinate in the three-dimensional corner point coordinates of the corner point on the fourth edge under the left camera coordinate system, x represents the pseudo-inverse of a matrix;

[0059] According to the plane equation parameters, the normal vector and the center point coordinates of the bottom surface of the transparent culture dish in the left camera coordinate system are calculated, including:

[0060] Through the formula:

[0061]

[0062] The normal vector v z is calculated by the formula:

[0063] Through the formula:

[0064]

[0065] The center point coordinates

[0066] wherein, represents the projection point of the corner point on the first edge under the left camera coordinate system, represents the projection point of the corner point on the second edge under the left camera coordinate system, represents the projection point of the corner point on the third edge under the left camera coordinate system, represents the projection point of the corner point on the fourth edge under the left camera coordinate system:

[0067]

[0068] wherein, represents the projection point of the corner point on the i-th edge under the left camera coordinate system, i = 1, 2, 3, 4, x i,L represents the horizontal coordinate in the three-dimensional corner point coordinates of the corner point on the i-th edge under the left camera coordinate system, y i,LThe z-coordinate represents the ordinate of the 3D corner point of the i-th edge in the left camera coordinate system. i,L This represents the vertical coordinate of the three-dimensional corner point of the i-th edge in the left camera coordinate system.

[0069] Optionally, calculate the long and short side vectors of the bottom surface of the transparent petri dish in the left camera coordinate system based on the normal vector, including:

[0070] Through the formula:

[0071]

[0072] Calculate the long side vector v x ;

[0073] Through the formula:

[0074] v y =v z ×v x

[0075] Calculate the short side vector v y ;

[0076] The position and orientation transformation matrix of the transparent culture dish in the left camera coordinate system is calculated based on the normal vector, long side vector, short side vector, and center point coordinates, including:

[0077] Through the formula:

[0078]

[0079] Calculate the position and attitude transformation matrix T.

[0080] Secondly, embodiments of this application provide a pose recognition device for a transparent petri dish, comprising:

[0081] The acquisition module is used to acquire left and right camera images of the transparent culture dish;

[0082] The segmentation module is used to segment and recognize contours of the left camera image to obtain a first image construction point set of the left camera image, and to segment and recognize contours of the right camera image to obtain a second image construction point set of the right camera image. The first image construction point set includes the coordinates of all contour points on the bottom surface of the transparent petri dish in the left camera image, and the second image construction point set includes the coordinates of all contour points on the bottom surface of the transparent petri dish in the right camera image.

[0083] The constructing module is configured to construct an optimization problem and constraint conditions according to the first image construction point set and the second image construction point set; the optimization problem is that the sum of squares of distances between each contour point in the first image construction point set and the second image construction point set and the edge to which the contour point belongs is minimum in a two-dimensional coordinate system; and the constraint conditions are used to describe the contour of the transparent culture dish, and the edge is an edge in a minimum envelope quadrilateral determined based on the left camera image or an edge in a minimum envelope quadrilateral determined based on the right camera image.

[0084] The solving module is configured to solve the optimization problem under the constraint of the constraint conditions to obtain three-dimensional corner point coordinates of each corner point in the left camera image in the left camera coordinate system.

[0085] The calculating module is configured to calculate the position and pose of the transparent culture dish in the left camera coordinate system according to all the three-dimensional corner point coordinates.

[0086] In a third aspect, an embodiment of the present application provides a terminal device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the pose recognition method of the transparent culture dish when executing the computer program.

[0087] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the pose recognition method of the transparent culture dish.

[0088] The above-mentioned scheme of the present application has the following advantages:

[0089] In some embodiments of the present application, the left camera image and the right camera image of the transparent culture dish are acquired, then the left camera image is segmented and contour recognition is performed to obtain a first image construction point set of the left camera image, the right camera image is segmented and contour recognition is performed to obtain a second image construction point set of the right camera image, then an optimization problem and constraint conditions are constructed according to the first image construction point set and the second image construction point set, then the optimization problem is solved under the constraint of the constraint conditions to obtain three-dimensional corner point coordinates of each corner point in the left camera image in the left camera coordinate system, and finally the position and pose of the transparent culture dish in the left camera coordinate system are calculated according to all the three-dimensional corner point coordinates. The optimization problem and the constraint conditions are constructed, the distance between the contour point and the edge to which the contour point belongs is expressed, the three-dimensional corner point coordinates of the corner point are obtained by solving the optimization problem, the distance between the contour point and the edge to which the contour point belongs is considered, the accuracy of the three-dimensional corner point coordinates of the corner point is improved, the position and pose of the transparent culture dish are calculated based on the accurate three-dimensional corner point coordinates, and the accuracy of the pose recognition of the transparent culture dish is effectively improved.

[0090] Other benefits of the application will be described in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0091] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0092] Figure 1 The flow chart of the pose recognition method of the transparent culture dish provided by an embodiment of the present application;

[0093] Figure 2 The left camera image and the right camera image provided by an embodiment of the present application;

[0094] Figure 3 The semantic segmentation binary image provided by an embodiment of the present application;

[0095] Figure 4 The edge binary image provided by an embodiment of the present application;

[0096] Figure 5 The minimum envelope quadrilateral schematic diagram provided by an embodiment of the present application;

[0097] Figure 6 The corner point schematic diagram provided by an embodiment of the present application;

[0098] Figure 7 The structural schematic diagram of the pose recognition device of the transparent culture dish provided by an embodiment of the present application;

[0099] Figure 8 The structural schematic diagram of the terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0100] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0101] It will be understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0102] It is also to be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items, and that the term “at least one of’ denotes one, or a plurality of, or any combination of the listed items.

[0103] As used in the description of the application and the appended claims, the term “if’ can be interpreted to mean “when” or “upon” or “in response to determining” or “in response to detecting” depending on the context. Similarly, the phrase “if it is determined” or “if [a described condition or event] is detected” can be interpreted to mean “upon determining” or “in response to determining” or “upon [the described condition or event] being detected” or “in response to [the described condition or event] being detected,” depending on the context.

[0104] In addition, the description in the specification of the application and the appended claims uses the terms “first,” “second,” “third,” etc. to refer to the features described, but does not imply that the features are of any relative importance.

[0105] References in the specification to “one embodiment” or “some embodiments” etc. mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrases “in one embodiment” or “in some embodiments” in various places in the specification are not necessarily all referring to the same embodiment, although they can. The terms “comprise,” “comprising,” “include,” “including,” and “has” or “having” as used in the specification, specify the presence of stated features, integers, steps, operations, elements, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof.

[0106] In view of the low accuracy of the existing pose recognition of the transparent culture dish, the embodiment of the present application provides a pose recognition method of a transparent culture dish. The pose recognition method comprises the following steps: acquiring a left camera image and a right camera image of the transparent culture dish; performing segmentation and contour recognition on the left camera image to obtain a first picture construction point set of the left camera image, and performing segmentation and contour recognition on the right camera image to obtain a second picture construction point set of the right camera image; constructing an optimization problem and a constraint condition according to the first picture construction point set and the second picture construction point set; and solving the optimization problem under the constraint of the constraint condition to obtain three-dimensional corner point coordinates of each corner point in the left camera image in a left camera coordinate system. Finally, the position and attitude of the transparent culture dish in the left camera coordinate system are calculated according to all the three-dimensional corner point coordinates. The construction of the optimization problem and the constraint condition can express the distance between the contour points of the transparent culture dish and the edges where the contour points are located. The three-dimensional corner point coordinates of the corner points are obtained by solving the optimization problem, the distance between the contour points and the edges where the contour points are located is considered, the accuracy of the three-dimensional corner point coordinates of the corner points is improved, the position and attitude of the transparent culture dish are calculated based on the accurate three-dimensional corner point coordinates, and the accuracy of the pose recognition of the transparent culture dish is effectively improved.

[0107] Next, the pose recognition method of the transparent culture dish provided by the present application is exemplarily described.

[0108] As shown in Figure 1 The pose recognition method of the transparent culture dish provided by the present application comprises the following steps:

[0109] Step 11: acquiring a left camera image and a right camera image of the transparent culture dish.

[0110] The transparent culture dish is a transparent material culture dish that needs to be recognized in pose. The left camera image is an image captured by a camera located on the left side of the transparent culture dish in a binocular camera. The right camera image is an image captured by a camera located on the right side of the transparent culture dish in the binocular camera, and the left camera image and the right camera image are acquired at the same time.

[0111] Step 12: performing segmentation and contour recognition on the left camera image to obtain a first picture construction point set of the left camera image, and performing segmentation and contour recognition on the right camera image to obtain a second picture construction point set of the right camera image.

[0112] The first picture construction point set comprises the coordinates of all contour points of the bottom surface of the transparent culture dish in the left camera image (the coordinates are two-dimensional coordinates in a two-dimensional coordinate system constructed with the left camera image as a plane), and the second picture construction point set comprises the coordinates of all contour points of the bottom surface of the transparent culture dish in the right camera image (the coordinates are two-dimensional coordinates in a two-dimensional coordinate system constructed with the right camera image as a plane).

[0113] In some embodiments of the present application, the step of segmenting and contour recognizing the left camera image to obtain the first picture construction point set of the left camera image comprises:

[0114] Firstly, the left camera image is subjected to semantic segmentation to obtain a semantic segmentation binary image.

[0115] For example, the left camera image can be subjected to semantic segmentation by using a visual target detection model YOLOV5 to obtain a semantic segmentation binary image. In the semantic segmentation binary image, the pixel value of the region determined as the quadrilateral bottom surface of the culture dish is 255, and the pixel value of the region determined as the non-quadrilateral bottom surface of the culture dish is 0.

[0116] Secondly, the semantic segmentation binary image is subjected to edge detection to obtain an edge binary image.

[0117] For example, the semantic segmentation binary image can be subjected to edge detection by using a Canny edge detection algorithm to obtain an edge binary image, wherein the pixel value of the region determined as the edge is 255, and the pixel value of the region determined as the non-edge is 0.

[0118] Thirdly, the edge binary image is subjected to contour extraction to obtain the coordinates of all contour points of the transparent culture dish bottom surface in the left camera image.

[0119] For example, the edge binary image can be subjected to contour extraction by using a Moore neighborhood tracking algorithm to obtain the coordinates of all contour points of the transparent culture dish bottom surface in the left camera image. The coordinates are coordinates in a two-dimensional coordinate system constructed with the left camera image as the plane.

[0120] Fourthly, a convex hull extraction algorithm is used to process the coordinates of all contour points to obtain the minimum envelope quadrilateral in the left camera image.

[0121] For example, the convex hull extraction algorithm is used to find the convex hull (the convex hull is the smallest convex polygon containing all points) of all contour points, and then the smallest rectangle is found from the convex hull to obtain the minimum envelope quadrilateral. The four edges of the minimum envelope quadrilateral are the four edges of the transparent culture dish bottom surface in the left camera image.

[0122] Fifthly, all contour points are classified according to the minimum envelope quadrilateral to determine the edge to which each contour point belongs from the minimum envelope quadrilateral.

[0123] For example, for each contour point, the distance between the contour point and each edge of the minimum envelope quadrilateral is calculated, and the edge with the smallest distance is taken as the edge to which the contour point belongs. The distance between the contour point and the edge can be calculated by using a mathematical formula for calculating the distance between a straight line and a point.

[0124] In the sixth step, coordinates of all contour points are integrated according to edges to which the contour points belong, to obtain a first picture construction point set.

[0125] For example, coordinates of all contour points belonging to the same edge are integrated into a set, and all sets are integrated into the first picture construction point set. The expression is as follows:

[0126]

[0127] wherein Ω i,L represents a set of coordinates of all contour points belonging to the ith edge, represents coordinates of the jth contour point belonging to the ith edge in the first picture construction point set, represents an abscissa of the jth contour point belonging to the ith edge in the first picture construction point set, represents an ordinate of the jth contour point belonging to the ith edge in the first picture construction point set.

[0128] It should be noted that the above process of segmenting and contour recognition of the right camera image to obtain the second picture construction point set of the right camera image is the same as the process of obtaining the first picture construction point set. That is, the right camera image is subjected to semantic segmentation to obtain a semantic segmentation binary image, then the semantic segmentation binary image is subjected to edge detection to obtain an edge binary image, then the edge binary image is subjected to contour extraction to obtain coordinates of all contour points of the transparent culture dish bottom surface in the right camera image (the coordinates are coordinates in a two-dimensional coordinate system constructed with the right camera image as a plane), then a convex hull extraction algorithm is used to process the coordinates of all contour points to obtain a minimum envelope quadrilateral in the right camera image, then all contour points are classified according to the minimum envelope quadrilateral to determine edges to which each contour point belongs from the minimum envelope quadrilateral, and finally coordinates of all contour points are integrated according to edges to which the contour points belong to obtain the second picture construction point set.

[0129] In step 13, an optimization problem and a constraint condition are constructed according to the first picture construction point set and the second picture construction point set.

[0130] The above optimization problem is that the sum of squares of distances between each contour point in the first picture construction point set and the second picture construction point set and an envelope edge to which the contour point belongs is minimized in a two-dimensional coordinate system, and the constraint condition is used to describe the contour of the transparent culture dish. The envelope edge is an edge in the minimum envelope quadrilateral determined based on the left camera image, or an edge in the minimum envelope quadrilateral determined based on the right camera image.

[0131] In some embodiments of the present application, the above step of constructing an optimization problem and a constraint condition according to the first picture construction point set and the second picture construction point set includes:

[0132] In the first step, the distance expression between each contour point and the edge to which the contour point belongs in the first picture construction point set is calculated, and the distance expression between each contour point and the edge to which the contour point belongs in the second picture construction point set is calculated.

[0133] The edge to which the contour point belongs is the edge of the minimum envelope quadrilateral determined in the above, for the contour point in the first picture construction point set, the edge to which the contour point belongs is the edge of the minimum envelope quadrilateral in the left camera image, and for the contour point in the second picture construction point set, the edge to which the contour point belongs is the edge of the minimum envelope quadrilateral in the right camera image.

[0134] Specifically, the distance expression between each contour point and the edge to which the contour point belongs in the first picture construction point set is:

[0135]

[0136] wherein, represents the distance between the jth contour point belonging to the ith edge in the first picture construction point set and the ith edge, i = 1, 2, 3, 4, j = 1, 2, …, n i,L , n i,L represents the number of contour points belonging to the ith edge in the first picture construction point set, represents the horizontal coordinate of the jth contour point belonging to the ith edge in the first picture construction point set, represents the vertical coordinate of the jth contour point belonging to the ith edge in the first picture construction point set, v i,L represents the vertical coordinate of the corner point on the ith edge in the left camera image, u i,L represents the horizontal coordinate of the corner point on the ith edge in the left camera image, v i+1,L represents the vertical coordinate of the corner point on the i+1th edge in the left camera image, u i+1,L represents the horizontal coordinate of the corner point on the i+1th edge in the left camera image.

[0137] The distance expression between each contour point and the edge to which the contour point belongs in the second picture construction point set is:

[0138]

[0139] wherein, represents the distance between the kth contour point belonging to the ith edge in the second picture construction point set and the ith edge, k = 1, 2, …, n i,R , n i,R represents the number of contour points belonging to the ith edge in the second picture construction point set, represents the horizontal coordinate of the kth contour point belonging to the ith edge in the second picture construction point set, denotes the vertical coordinate of the kth contour point belonging to the ith edge in the second picture construction point set, v i,R denotes the vertical coordinate of the corner point on the ith edge in the right camera image, u i,R denotes the horizontal coordinate of the corner point on the ith edge in the right camera image, v i+1,R denotes the vertical coordinate of the corner point on the ith+1 edge in the right camera image, u i+1,R denotes the horizontal coordinate of the corner point on the ith+1 edge in the right camera image.

[0140] It should be noted that the corner point on the envelope edge in the left camera image is the corner point on the bottom surface of the transparent culture dish in the left camera image, and the corner point on the envelope edge in the right camera image is the corner point on the bottom surface of the transparent culture dish in the right camera image. The corner point on the edge is determined by pre-setting, for example, when the bottom surface is a quadrilateral, the end point of one end of the first edge is taken as the corner point on the edge, and the next end point in the clockwise direction of the end point is taken as the corner point on the second edge, and so on. In the case of a quadrilateral bottom surface, i.e. having four corner points, the 4+1th corner point is considered as the 1st corner point.

[0141] Secondly, an index function is constructed according to all distance expressions.

[0142] Specifically, the index function is:

[0143]

[0144] wherein J(P 1,L ,P 2,L ,P 3,L ,P 4,L ) represents the value of the index function.

[0145] Thirdly, for each corner point in the left camera image, a left camera coordinate conversion relationship between the three-dimensional coordinates of the corner point in the left camera coordinate system and the two-dimensional coordinates of the corner point in the left camera image is constructed.

[0146] Specifically, the left camera coordinate conversion relationship is:

[0147]

[0148] wherein x i,L denotes the horizontal coordinate of the corner point on the ith edge in the left camera coordinate system, y i,L denotes the vertical coordinate of the corner point on the ith edge in the left camera coordinate system, z i,L denotes the vertical coordinate of the corner point on the ith edge in the left camera coordinate system, M L denotes the left camera internal parameter matrix:

[0149]

[0150] wherein f L,x represents the ratio of the focal length of the left camera to the horizontal pixel size, f L,y represents the ratio of the focal length of the left camera to the vertical pixel size, c L,x represents the horizontal coordinate of the center pixel of the left camera image, c L,y represents the vertical coordinate of the center pixel of the left camera image.

[0151] In the fourth step, for each corner point in the right camera image, a right camera coordinate conversion relationship between the three-dimensional coordinate of the corner point in the left camera coordinate system and the two-dimensional coordinate of the corner point in the right camera image is constructed.

[0152] Specifically, the right camera coordinate conversion relationship is:

[0153]

[0154] wherein f R,x represents the ratio of the focal length of the right camera to the horizontal pixel size, f R,y represents the ratio of the focal length of the right camera to the vertical pixel size, c R,x represents the horizontal coordinate of the center pixel of the right camera image, c R,y represents the vertical coordinate of the center pixel of the right camera image, r 1,1 represents the element in the first row and the first column of the rotation matrix, r 1,2 represents the element in the first row and the second column of the rotation matrix, r 1,3 represents the element in the first row and the third column of the rotation matrix, r 2,1 represents the element in the second row and the first column of the rotation matrix, r 2,2 represents the element in the second row and the second column of the rotation matrix, r 2,3 represents the element in the second row and the third column of the rotation matrix, r 3,1 represents the element in the third row and the first column of the rotation matrix, r 3,2 represents the element in the third row and the second column of the rotation matrix, r 3,3 represents the element in the third row and the third column of the rotation matrix, t1 represents the first element in the translation vector, t2 represents the second element in the translation vector, and t3 represents the third element in the translation vector.

[0155] In the fifth step, an edge length expression in the left camera coordinate system is constructed.

[0156] Specifically, the edge length expression is:

[0157]

[0158] wherein L idenotes the length of the i-th edge of the bottom surface of the transparent culture dish (the length is the actual length of the edge of the bottom surface of the transparent culture dish, which is a priori known information and can be obtained by actual measurement), x i+1,L denotes the horizontal coordinate of the corner point on the i+1-th edge in the left camera coordinate system, y i+1,L denotes the vertical coordinate of the corner point on the i+1-th edge in the left camera coordinate system, z i+1,L denotes the vertical coordinate of the corner point on the i+1-th edge in the left camera coordinate system.

[0159] The formula indicates that the length of the calculated envelope edge in the left camera coordinate system should be equal to the actual length of the edge of the bottom surface of the transparent culture dish.

[0160] In the sixth step, an optimization problem is constructed according to the index function, and the left camera coordinate conversion relationship, the right camera coordinate conversion relationship and the edge length expression are taken as the constraint conditions of the optimization problem.

[0161] Specifically, the optimization problem is as follows:

[0162]

[0163] wherein P 1,L denotes the three-dimensional corner point coordinates of the corner point on the first edge in the left camera coordinate system, P 2,L denotes the three-dimensional corner point coordinates of the corner point on the second edge in the left camera coordinate system, P 3,L denotes the three-dimensional corner point coordinates of the corner point on the third edge in the left camera coordinate system, P 4,L denotes the three-dimensional corner point coordinates of the corner point on the fourth edge in the left camera coordinate system.

[0164] The corner point on the edge in the left camera coordinate system is the corner point on the envelope edge in the left camera image.

[0165] In the step 14, the optimization problem is solved under the constraint of the constraint condition, and the three-dimensional corner point coordinates of each corner point on each edge in the left camera image in the left camera coordinate system are obtained.

[0166] For example, the interior point method can be used to solve the optimization problem under the constraint of the constraint condition, and the three-dimensional corner point coordinates of each corner point on each edge in the left camera image in the left camera coordinate system are obtained.

[0167] In the step 15, the position and pose of the transparent culture dish in the left camera coordinate system are calculated according to all the three-dimensional corner point coordinates.

[0168] In some embodiments of the present application, the step of calculating the position and pose of the transparent culture dish in the left camera coordinate system according to all the three-dimensional corner point coordinates comprises:

[0169] The first step is to calculate the plane equation parameters of the bottom surface of the transparent petri dish in the left camera coordinate system based on the coordinates of all three-dimensional corner points.

[0170] Specifically, through the formula:

[0171]

[0172] Calculate the parameters a, b, and c of the plane equation.

[0173] Where, x 1,L This represents the x-coordinate of the corner point in the 3D coordinate system of the first edge (i.e., the envelope edge) in the left camera coordinate system. 1,L This represents the ordinate (z) of the 3D corner point coordinates of the first edge in the left camera coordinate system. 1,L The x-coordinate represents the vertical coordinate of the 3D corner point of the first edge in the left camera coordinate system. 2,L This represents the x-coordinate of the corner point in the 3D coordinate system of the second edge (i.e., the envelope edge) in the left camera coordinate system. 2,L Z represents the ordinate of the 3D corner point coordinates of the second edge in the left camera coordinate system. 2,L The x-coordinate represents the vertical coordinate of the 3D corner point of the second edge in the left camera coordinate system. 3,L This represents the x-coordinate of the 3D corner point on the third edge (i.e., the envelope edge) in the left camera coordinate system. 3,L This represents the ordinate (z) of the 3D corner point coordinates of the third edge point in the left camera coordinate system. 3,L The x-coordinate represents the vertical coordinate of the 3D corner point of the third edge in the left camera coordinate system. 4,L This represents the x-coordinate of the 3D corner point on the 4th edge (i.e., the envelope edge) in the left camera coordinate system. 4,L This represents the ordinate (z) of the 3D corner point coordinates of the fourth edge point in the left camera coordinate system. 4,L This represents the vertical coordinate of the 3D corner point of the fourth edge in the left camera coordinate system. This represents the pseudo-inverse of a matrix.

[0174] It should be noted that after calculating the plane equation parameters, the plane equation of the bottom surface of the transparent petri dish can be obtained as: h(x,y)=ax+by+c.

[0175] The second step is to calculate the normal vector and center point coordinates of the bottom surface of the transparent petri dish in the left camera coordinate system based on the plane equation parameters.

[0176] Specifically, through the formula:

[0177]

[0178] Calculate the normal vector vz .

[0179] By formula:

[0180]

[0181] Calculate the center point coordinates

[0182] wherein, represents the projection point of the corner point on the first edge in the left camera coordinate system, represents the projection point of the corner point on the second edge in the left camera coordinate system, represents the projection point of the corner point on the third edge in the left camera coordinate system, represents the projection point of the corner point on the fourth edge in the left camera coordinate system:

[0183]

[0184] wherein, represents the projection point of the corner point on the i-th edge in the left camera coordinate system, i = 1, 2, 3, 4, x i,L represents the horizontal coordinate of the three-dimensional corner point coordinates of the corner point on the i-th edge in the left camera coordinate system, y i,L represents the vertical coordinate of the three-dimensional corner point coordinates of the corner point on the i-th edge in the left camera coordinate system, z i,L represents the vertical coordinate of the three-dimensional corner point coordinates of the corner point on the i-th edge in the left camera coordinate system.

[0185] Step 3, calculate the long edge vector and short edge vector of the bottom surface of the transparent culture dish in the left camera coordinate system according to the normal vector.

[0186] Specifically, by formula:

[0187]

[0188] Calculate the long edge vector v x .

[0189] By formula:

[0190] v y =v z ×v x

[0191] Calculate the short edge vector v y .

[0192] Step 4, calculate the position and posture transformation matrix of the transparent culture dish in the left camera coordinate system based on the normal vector, long edge vector, short edge vector and center point coordinates.

[0193] The first three columns of the position and attitude transformation matrix are used to describe the attitude of the transparent culture dish in the left camera coordinate system, and the last column of the position and attitude transformation matrix is used to describe the position of the transparent culture dish in the left camera coordinate system.

[0194] Specifically, the position and attitude transformation matrix T is calculated by the formula:

[0195]

[0196] The position and attitude transformation matrix T is calculated.

[0197] It should be noted that after obtaining the position and attitude of the transparent culture dish, the robot arm can be guided to grasp the transparent culture dish based on the position and attitude. Since the relationship between the left camera coordinate system and the right camera coordinate system relative to the base coordinate system of the robot can be obtained through hand-eye calibration, the position and attitude of the transparent culture dish relative to the left camera coordinate system or the right camera coordinate system can be solved to guide the robot arm to grasp it. In actual application, the left camera coordinate system or the right camera coordinate system is determined according to specific requirements. If the left camera coordinate system is used, the position and attitude of the culture dish in the left camera coordinate system are calculated using the process described above. If the right camera coordinate system is used, the same process as described above can be used, but when performing coordinate transformation and other steps, the two-dimensional coordinates are converted to the right camera coordinate system.

[0198] It is worth mentioning that the construction of the optimization problem and the constraint condition can describe the distance between the contour points of the transparent culture dish and the edges where the contour points are located. Solving the optimization problem obtains the three-dimensional corner point coordinates of the corner point, which considers the distance between the contour points and the edges where the contour points belong to, improves the accuracy of the three-dimensional corner point coordinates of the corner point, and based on the accurate three-dimensional corner point coordinates, the position and attitude of the transparent culture dish are calculated, which effectively improves the accuracy of the transparent culture dish pose recognition.

[0199] The method of the present application will be described in conjunction with a specific example.

[0200] The left camera image obtained by image acquisition using a binocular camera is shown in Figure 2 a, and the right camera image is shown in Figure 2 b, and semantic segmentation is performed respectively to obtain the semantic segmentation binary image of the left camera image as shown in Figure 3 a, and the semantic segmentation binary image of the right camera image as shown in Figure 3 b, edge detection is performed on the semantic segmentation binary image to obtain the left edge binary image as shown in Figure 4 a, and the right edge binary image as shown in Figure 4 b, after contour extraction and convex hull extraction algorithm processing of the edge binary image, the minimum envelope quadrilateral of the left camera image is obtained as shown in Figure 5The minimum envelope quadrilateral of the right camera image is shown as Figure 5 b, Figure 5 The middle quadrilateral is the minimum envelope quadrilateral.

[0201] In this example, the specific values of the internal parameter matrix of the two cameras are:

[0202]

[0203] The rotation matrix and the translation vector are respectively:

[0204]

[0205] t = [-62.73 33.17 x 10 -2 0.1] T

[0206] According to the steps in the foregoing and the above parameters, the optimization problem and the constraint conditions are:

[0207]

[0208] Solving the same, the three-dimensional corner point coordinates of the four corner points are:

[0209] P 1,L = [110.51, -115.37, 505.58]

[0210] P 2,L = [31.53, -162.84, 531.51]

[0211] P 3,L = [29.72, -132.49, 562.30]

[0212] P 4,L = [107.89, -85.61, 527.11]

[0213] According to the relationship between the three-dimensional corner point coordinates and the left camera image and the right camera image, the coordinates of the corner points in the left camera image are shown as Figure 6 a, the coordinates of the corner points in the right camera image are shown as Figure 6 b, Figure 6 In which the straight line represents the edge, and the circular point represents the corner point.

[0214] The calculated center point coordinates are:

[0215]

[0216] The normal vector is:

[0217] v z = [0.55 -0.54 0.63]T

[0218] The long side vector and short side vector are calculated based on the normal vector as follows:

[0219]

[0220] The calculated position and attitude transformation matrix is:

[0221]

[0222] Therefore, the method of this application can improve the accuracy of pose recognition of transparent petri dishes.

[0223] The pose recognition device for transparent petri dishes provided in this application will be described by way of example below.

[0224] like Figure 7 As shown, this application embodiment provides a pose recognition device for a transparent petri dish. The pose recognition device 700 for the transparent petri dish includes:

[0225] The acquisition module 701 is used to acquire the left and right camera images of the transparent culture dish;

[0226] The segmentation module 702 is used to segment and recognize contours of the left camera image to obtain a first image construction point set of the left camera image, and to segment and recognize contours of the right camera image to obtain a second image construction point set of the right camera image; the first image construction point set includes the coordinates of all contour points on the bottom surface of the transparent petri dish in the left camera image, and the second image construction point set includes the coordinates of all contour points on the bottom surface of the transparent petri dish in the right camera image.

[0227] The construction module 703 is used to construct an optimization problem and constraints based on the first image and the second image point set. The optimization problem is to minimize the sum of the squares of the distances between each contour point in the first image point set and the second image point set and the envelope edge to which the contour point belongs in a two-dimensional coordinate system. The constraints are used to describe the contour of the transparent petri dish. The envelope edge is the edge in the minimum envelope quadrilateral determined based on the left camera image, or the edge in the minimum envelope quadrilateral determined based on the right camera image.

[0228] The solver module 704 is used to solve the optimization problem under the constraints of the constraints, and obtain the three-dimensional corner coordinates of each corner point in the left camera image in the left camera coordinate system.

[0229] The calculation module 705 is used to calculate the position and orientation of the transparent culture dish in the left camera coordinate system based on the coordinates of all three-dimensional corner points.

[0230] It should be noted that the information interaction, execution process and the like between the above apparatuses / units are based on the same concept as the method embodiments of the present application, and the specific functions and brought technical effects can be referred to the method embodiments part, which will not be repeated here.

[0231] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the above-described functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific name of each functional unit and module is only for convenient distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0232] As shown in Figure 8 , the embodiment of the present application provides a terminal device, the terminal device D10 of the embodiment includes at least one processor D100 (only one processor is shown in the embodiment), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 executes the computer program D102 to realize the steps in any of the above method embodiments. Figure 8

[0233] ​Specifically, the processor D100 executes the computer program D102, obtains a left camera image and a right camera image of the transparent culture dish, then performs segmentation and contour recognition on the left camera image to obtain a first picture construction point set of the left camera image, and performs segmentation and contour recognition on the right camera image to obtain a second picture construction point set of the right camera image, constructs an optimization problem and a constraint condition according to the first picture construction point set and the second picture construction point set, then solves the optimization problem under the constraint of the constraint condition to obtain three-dimensional corner point coordinates of each corner point in the left camera image in a left camera coordinate system, and finally calculates the position and pose of the transparent culture dish in the left camera coordinate system according to all the three-dimensional corner point coordinates. Wherein, the construction of the optimization problem and the constraint condition can express the distance between the contour points of the transparent culture dish and the edges where the contour points are located, and the solving of the optimization problem obtains the three-dimensional corner point coordinates of the corner points, which considers the distance between the contour points and the edges where the contour points belong to, improves the accuracy of the three-dimensional corner point coordinates of the corner points, and calculates the position and pose of the transparent culture dish based on the accurate three-dimensional corner point coordinates, effectively improving the accuracy of the transparent culture dish pose recognition.

[0234] The processor D100 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor.

[0235] The storage D101 can be an internal storage unit of the terminal device D10 in some embodiments, such as a hard disk or a memory of the terminal device D10. The storage D101 can also be an external storage device of the terminal device D10 in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Further, the storage D101 can include both an internal storage unit and an external storage device of the terminal device D10. The storage D101 is used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of the computer program, and the like. The storage D101 can also be used to temporarily store data that has been output or will be output.

[0236] The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0237] The computer program product, when running on the terminal device, causes the terminal device to implement the steps in each of the above method embodiments.

[0238] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the present application implements all or part of the processes in the above embodiments, which can be completed by instructing related hardware through a computer program. The computer program can be stored in a computer readable storage medium, and the computer program, when executed by a processor, can implement the steps in each of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer readable medium at least includes any entity or device that can carry the computer program code to the pose recognition method device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. For example, a U disk, a mobile hard disk, a magnetic disk or an optical disk, and the like. In some jurisdictions, according to legislation and patent practice, the computer readable medium cannot be an electrical carrier signal and a telecommunications signal.

[0239] In the above embodiments, the description of each embodiment is focused on, and the part not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments.

[0240] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0241] The above is the preferred embodiment of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the principles described in the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A method for recognizing the pose of a transparent petri dish, characterized in that, include: Acquire left and right camera images of the transparent culture dish; The left camera image is segmented and contour recognized to obtain a first image construction point set of the left camera image, and the right camera image is segmented and contour recognized to obtain a second image construction point set of the right camera image; the first image construction point set includes the coordinates of all contour points on the bottom surface of the transparent petri dish in the left camera image, and the second image construction point set includes the coordinates of all contour points on the bottom surface of the transparent petri dish in the right camera image; An optimization problem and constraints are constructed based on the point sets constructed from the first and second images. The optimization problem is to minimize the sum of the squares of the distances between each contour point in the first and second image point sets and the envelope edge to which that contour point belongs, in a two-dimensional coordinate system. The constraints are used to describe the contour of the transparent culture dish. The envelope edge is a side in the minimum envelope quadrilateral determined based on the left camera image, or a side in the minimum envelope quadrilateral determined based on the right camera image. The optimization problem is solved under the constraints to obtain the three-dimensional corner coordinates of each corner point in the left camera image in the left camera coordinate system. The position and orientation of the transparent culture dish in the left camera coordinate system are calculated based on all three-dimensional corner coordinates.

2. The pose recognition method according to claim 1, characterized in that, The step of segmenting and contour recognition of the left camera image to obtain a first image construction point set of the left camera image includes: The left camera image is semantically segmented to obtain a semantically segmented binary image; Edge detection is performed on the semantically segmented binary image to obtain an edge binary image; Contour extraction is performed on the binary edge image to obtain the coordinates of all contour points on the bottom surface of the transparent culture dish in the left camera image; The coordinates of all contour points are processed using a convex hull extraction algorithm to obtain the minimum envelope quadrilateral in the left camera image; All contour points are classified according to the minimum envelope quadrilateral, and the edge to which each contour point belongs is determined from the minimum envelope quadrilateral; The coordinates of all contour points are integrated according to the edges to which all contour points belong, resulting in the first image construction point set.

3. The pose recognition method according to claim 1, characterized in that, The step of constructing an optimization problem and constraints based on the point set constructed from the first image and the point set constructed from the second image includes: Calculate the distance expression between each contour point in the first image construction point set and the edge to which the contour point belongs, and calculate the distance expression between each contour point in the second image construction point set and the edge to which the contour point belongs; Construct an index function based on all distance expressions; For each corner point in the left camera image, construct the left camera coordinate transformation relationship between the three-dimensional coordinates of the corner point in the left camera coordinate system and the two-dimensional coordinates of the corner point in the left camera image; For each corner point in the right camera image, construct the right camera coordinate transformation relationship between the three-dimensional coordinates of the corner point in the left camera coordinate system and the two-dimensional coordinates of the corner point in the right camera image; Construct the side length expression in the left camera coordinate system; An optimization problem is constructed based on the index function, and the coordinate transformation relationship of the left camera, the coordinate transformation relationship of the right camera, and the side length expression are used as constraints for the optimization problem.

4. The pose recognition method according to claim 3, characterized in that, The distance expression between each contour point in the first image construction point set and the edge to which the contour point belongs is: in, This represents the distance between the j-th contour point belonging to the i-th edge in the first image construction point set and the i-th edge, where i = 1, 2, 3, 4, j = 1, 2, ..., n i,L n i,L This represents the number of contour points in the first image's construction point set that belong to the i-th edge. This represents the x-coordinate of the j-th contour point belonging to the i-th edge in the first image construction point set. v represents the ordinate of the j-th contour point belonging to the i-th edge in the first image construction point set. i,L U represents the ordinate of the corner point on the i-th edge in the left camera image. i,L v represents the x-coordinate of the corner point on the i-th edge in the left camera image. i+1,L U represents the ordinate of the corner point of the (i+1)th edge in the left camera image. i+1,L This represents the x-coordinate of the corner point of the (i+1)th edge in the left camera image; The distance expression between each contour point in the second image construction point set and the edge to which the contour point belongs is: in, This represents the distance between the k-th contour point belonging to the i-th edge in the second image construction point set and the i-th edge, where k = 1, 2, ..., n. i,R n i,R This represents the number of contour points in the second image's construction point set that belong to the i-th edge. This represents the x-coordinate of the k-th contour point belonging to the i-th edge in the second image construction point set. v represents the ordinate of the k-th contour point belonging to the i-th edge in the second image construction point set. i,R U represents the ordinate of the top corner point of the i-th edge in the right camera image. i,R v represents the x-coordinate of the corner point on the i-th edge in the right camera image. i+1,R U represents the ordinate of the corner point on the (i+1)th edge in the right camera image. i+1,R This represents the x-coordinate of the corner point of the (i+1)th edge in the right camera image.

5. The pose recognition method according to claim 4, characterized in that, The index function is: Among them, J(P 1,L ,P 2,L ,P 3,L ,P 4,L ) represents the value of the index function.

6. The pose recognition method according to claim 5, characterized in that, The coordinate transformation formula for the left camera is: Where, x i,L The x-coordinate of the corner point on the i-th edge in the left camera coordinate system is y. i,L Z represents the ordinate of the corner point on the i-th edge in the left camera coordinate system. i,L M represents the vertical coordinate of the corner point on the i-th edge in the left camera coordinate system. L Represents the intrinsic parameter matrix of the left camera: Among them, f L,x f represents the ratio of the focal length of the left camera to the horizontal pixel size. L,y c represents the ratio of the focal length of the left camera to the vertical pixel size. L,x c represents the x-coordinate of the center pixel of the left camera image. L,y Represents the ordinate of the center pixel of the left camera image; The coordinate transformation formula for the right camera is: Among them, f R,x f represents the ratio of the focal length to the horizontal pixel size of the right camera. R,y c represents the ratio of the focal length of the right camera to its vertical pixel size. R,x c represents the x-coordinate of the center pixel of the right camera image. R,y r represents the ordinate of the center pixel of the right camera image. 1,1 Let r represent the element in the first row and first column of the rotation matrix. 1,2 Let r represent the element in the first row and second column of the rotation matrix. 1,3 Let r represent the element in the first row and third column of the rotation matrix. 2,1 Represents the element in the second row and first column of the rotation matrix, r 2,2 Let r represent the element in the second row and second column of the rotation matrix. 2,3 The element r in the second row and third column of the rotation matrix represents the element r. 3,1 Let r represent the element in the third row and first column of the rotation matrix. 3,2 Represents the element in the third row and second column of the rotation matrix, r 3,3 t1 represents the element in the third row and third column of the rotation matrix, t2 represents the first element in the translation vector, t3 represents the second element in the translation vector, and t3 represents the third element in the translation vector. The expression for the side length is: Among them, L i Let x represent the length of the i-th side of the bottom surface of the transparent culture dish. i+1,L The x-coordinate of the (i+1)th edge corner point in the left camera coordinate system is y. i+1,L z represents the ordinate of the (i+1)th edge corner point in the left camera coordinate system. i+1,L This represents the vertical coordinate of the (i+1)th corner point on the left camera coordinate system. The optimization problem is: Among them, P 1,L P represents the 3D corner coordinates of the first edge in the left camera coordinate system. 2,L P represents the 3D corner coordinates of the second edge point in the left camera coordinate system. 3,L P represents the 3D corner coordinates of the third edge point in the left camera coordinate system. 4,L This represents the 3D corner coordinates of the corner point on the 4th edge in the left camera coordinate system.

7. The pose recognition method according to claim 1, characterized in that, The calculation of the position and orientation of the transparent culture dish in the left camera coordinate system based on all three-dimensional corner coordinates includes: Calculate the plane equation parameters of the bottom surface of the transparent culture dish in the left camera coordinate system based on all three-dimensional corner point coordinates; Calculate the normal vector and center point coordinates of the bottom surface of the transparent culture dish in the left camera coordinate system based on the plane equation parameters; Calculate the long and short side vectors of the bottom surface of the transparent culture dish in the left camera coordinate system based on the normal vector; The position and attitude transformation matrix of the transparent petri dish in the left camera coordinate system is calculated based on the normal vector, the long side vector, the short side vector, and the center point coordinates. The first three columns of the position and attitude transformation matrix are used to describe the attitude of the transparent petri dish in the left camera coordinate system, and the last column of the position and attitude transformation matrix is ​​used to describe the position of the transparent petri dish in the left camera coordinate system.

8. The pose recognition method according to claim 7, characterized in that, The calculation of the plane equation parameters of the bottom surface of the transparent culture dish in the left camera coordinate system based on all three-dimensional corner coordinates includes: Through the formula: Calculate the parameters a, b, and c of the plane equation; Where, x 1,L This represents the x-coordinate of the 3D corner point of the first edge in the left camera coordinate system. 1,L This represents the ordinate (z) of the 3D corner point coordinates of the first edge in the left camera coordinate system. 1,L The x-coordinate represents the vertical coordinate of the 3D corner point of the first edge in the left camera coordinate system. 2,L This represents the x-coordinate of the 3D corner point of the second edge in the left camera coordinate system. 2,L This represents the ordinate (z) of the three-dimensional corner point coordinates of the second edge point in the left camera coordinate system. 2,L The x-coordinate represents the vertical coordinate of the 3D corner point of the second edge in the left camera coordinate system. 3,L This represents the x-coordinate of the 3D corner point of the third edge in the left camera coordinate system. 3,L This represents the ordinate (z) of the 3D corner point coordinates of the third edge point in the left camera coordinate system. 3,L The x-coordinate represents the vertical coordinate of the 3D corner point of the third edge in the left camera coordinate system. 4,L This represents the x-coordinate of the 3D corner point of the fourth edge in the left camera coordinate system. 4,L This represents the ordinate (z) of the 3D corner point coordinates of the fourth edge point in the left camera coordinate system. 4,L This represents the vertical coordinate of the 3D corner point of the fourth edge in the left camera coordinate system. Represents the pseudo-inverse of a matrix; The calculation of the normal vector and center point coordinates of the bottom surface of the transparent culture dish in the left camera coordinate system based on the plane equation parameters includes: Through the formula: Calculate the normal vector v z ; Through the formula: Calculate the coordinates of the center point in, This represents the projection point of the corner point of the first edge in the left camera coordinate system. This represents the projection point of the corner point of the second edge in the left camera coordinate system. This represents the projection point of the top corner point of the third edge in the left camera coordinate system. This represents the projection point of the corner point of the 4th edge in the left camera coordinate system: in, Let x represent the projection point of the corner point on the i-th edge in the left camera coordinate system, i = 1, 2, 3, 4. i,L The x-coordinate of the 3D corner point of the i-th edge in the left camera coordinate system is represented by the x-coordinate. i,L The z-coordinate represents the ordinate of the 3D corner point of the i-th edge in the left camera coordinate system. i,L This represents the vertical coordinate of the three-dimensional corner point of the i-th edge in the left camera coordinate system.

9. The pose recognition method according to claim 8, characterized in that, The calculation of the long and short side vectors of the bottom surface of the transparent culture dish in the left camera coordinate system based on the normal vector includes: Through the formula: Calculate the long side vector v x ; Through the formula: v y =v z ×v x Calculate the short side vector v y ; The calculation of the position and orientation transformation matrix of the transparent culture dish in the left camera coordinate system based on the normal vector, the long side vector, the short side vector, and the center point coordinates includes: Through the formula: Calculate the position and attitude transformation matrix T.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the pose recognition method for transparent culture dishes as described in any one of claims 1 to 9.

Images