Transparent culture dish pose estimation method and related device

Abstract

This invention provides a method and related equipment for estimating the pose of a transparent culture dish. The method acquires an RGB color image and a synchronously aligned depth image of a cell preparation stage containing a transparent culture dish. The RGB color image is input into a pre-trained lightweight target detection network for recognition, resulting in a two-dimensional bounding box of the transparent culture dish. This bounding box is used to extract the local depth map of the transparent culture dish from the depth image. The local depth map is then mapped to three-dimensional local point cloud data using distortion correction and intrinsic parameter matrices from the acquisition device. The three-dimensional local point cloud data is segmented to obtain the point cloud of the target culture dish to be grasped. This point cloud is then precisely registered with a three-dimensional standard point cloud model of the transparent culture dish to obtain the pose estimation result of the transparent culture dish, thus improving the accuracy of the pose estimation.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a method and related equipment for estimating the pose of a transparent petri dish. Background Technology

[0002] In the biomedical field, medical transparent culture dishes such as high-quality polystyrene are the most critical consumables in the entire process of cell culture, passage, and observation. However, due to the optical properties of transparent objects, their surfaces reflect and refract ambient light, making it difficult to extract their features using traditional methods based on color, texture, or edge detection. Therefore, the recognition and three-dimensional localization of transparent objects has always been a technical challenge for machine vision.

[0003] Traditional pure binocular vision positioning technology is based on the principle of stereo parallax, which places a heavy computational burden on pixel-level matching and is prone to mismatches on transparent and reflective surfaces. In addition, in actual fabrication workbenches, multiple culture dishes are often arranged closely or overlapped one on the other. Existing vision algorithms struggle to achieve accurate instance segmentation and 3D spatial positioning of overlapping transparent objects, which can easily lead to collisions or crushing of culture dishes when the collaborative robotic arm performs grasping tasks. Summary of the Invention

[0004] This invention provides a method and related equipment for estimating the pose of transparent culture dishes, with the aim of improving the accuracy of pose estimation for transparent culture dishes.

[0005] To achieve the above objectives, the present invention provides a method for estimating the pose of a transparent culture dish, comprising: Step 1: Acquire RGB color images and synchronously aligned depth images of the cell preparation stage with transparent culture dishes placed on it; Step 2: Input the RGB color image into the pre-trained lightweight object detection network for recognition to obtain the two-dimensional bounding box of the transparent petri dish; Step 3: Extract the local depth map of the transparent culture dish from the depth image based on the two-dimensional bounding box, and map the local depth map into three-dimensional local point cloud data by combining the distortion correction and intrinsic parameter matrix of the acquisition device. Step 4: Perform instance segmentation on the 3D local point cloud data to obtain the point cloud of the target petri dish to be grasped, and perform fine registration between the point cloud of the target petri dish and the 3D standard point cloud model of the transparent petri dish to obtain the pose estimation result of the transparent petri dish.

[0006] Furthermore, the loss function expression for a lightweight object detection network is: ; in, This represents the total loss value. This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. Indicates the center point of the prediction box Center point of the real frame The Euclidean distance between them This represents the diagonal distance between the predicted bounding box and the ground truth bounding box, representing the minimum closure region. This indicates a penalty for aspect ratio consistency. This represents the balance ratio coefficient. This represents the confidence loss value. Represents the classification loss value. , , All of these represent weighting coefficients.

[0007] Furthermore, based on the two-dimensional bounding box, the local depth map of the transparent culture dish is extracted from the depth image, including: Extracting the region of interest enclosed by a two-dimensional bounding box in a color image; By cropping a local depth map corresponding to the region of interest from the depth image, a local depth map of the transparent culture dish is obtained.

[0008] Furthermore, by combining distortion correction and intrinsic parameter matrices from the acquisition equipment, the local depth map is mapped into 3D local point cloud data, including: Radial and tangential distortion correction models are introduced to correct the pixel coordinates in the local depth map, thus obtaining ideal pixel coordinates; By utilizing the intrinsic parameter matrix of the acquisition device and based on the principle of pinhole imaging reverse perspective, the ideal pixel coordinates are upgraded to spatial point coordinates in the three-dimensional camera coordinate system to obtain three-dimensional local point cloud data.

[0009] Furthermore, by introducing radial and tangential distortion correction models to correct the pixel coordinates in the local depth map, the expression for the ideal pixel coordinates is obtained as follows: ; ; in, Represents the ideal pixel coordinates. Represents the pixel coordinates in the local depth map. This represents the radial distance from the pixel coordinate to the image center. This represents the radial distortion coefficient of the data acquisition device. This represents the tangential distortion coefficient of the acquisition device.

[0010] Furthermore, utilizing the intrinsic parameter matrix of the acquisition device and based on the principle of pinhole imaging reverse perspective, the expression for upgrading the ideal pixel coordinates to spatial point coordinates in the 3D camera coordinate system is as follows: ; ; ; in, This represents the coordinates of a point in the 3D camera coordinate system. Indicates the depth value. Indicates the optical center. Indicates the equivalent focal length.

[0011] Furthermore, before performing instance segmentation on the 3D local point cloud data, the following steps are also included: For any point in a 3D local point cloud, the k nearest neighbors of that point in 3D space are located using the KD-tree nearest neighbor search algorithm. Calculate the average Euclidean distance between all points in a 3D local point cloud data and the k nearest neighbors in 3D space; By removing free noise points from the 3D local point cloud data based on the mean Euclidean distance, the denoised 3D local point cloud data is obtained.

[0012] This invention also provides a transparent petri dish pose estimation device, which applies a transparent petri dish pose estimation method. The transparent petri dish pose estimation device includes: The acquisition module is used to acquire RGB color images and synchronously aligned depth images of the cell preparation stage on which transparent culture dishes are placed. The recognition module is used to input RGB color images into a pre-trained lightweight object detection network for recognition, and obtain the two-dimensional bounding box of the transparent petri dish; The mapping module is used to extract the local depth map of the transparent petri dish from the depth image based on the two-dimensional bounding box, and to map the local depth map into three-dimensional local point cloud data by combining the distortion correction and intrinsic parameter matrix of the acquisition device. The registration module is used to segment the 3D local point cloud data into instances to obtain the point cloud of the target petri dish to be grasped, and to perform fine registration between the point cloud of the target petri dish and the 3D standard point cloud model of the transparent petri dish to obtain the pose estimation result of the transparent petri dish.

[0013] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a method for estimating the pose of a transparent culture dish.

[0014] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for estimating the pose of a transparent culture dish.

[0015] The above-described solution of the present invention has the following beneficial effects: This invention acquires RGB color images and synchronously aligned depth images of a cell preparation worktable containing transparent culture dishes. The RGB color images are input into a pre-trained lightweight target detection network for recognition, yielding a two-dimensional bounding box for the transparent culture dishes. This bounding box is used to extract local depth maps of the transparent culture dishes from the depth images. Combined with distortion correction and intrinsic parameter matrices from the acquisition device, the local depth maps are mapped to three-dimensional local point cloud data. The three-dimensional local point cloud data is segmented to obtain the point cloud of the target culture dish to be grasped. This point cloud is then precisely registered with a three-dimensional standard point cloud model of the transparent culture dishes to obtain the pose estimation result of the transparent culture dishes. Compared with existing technologies, this invention effectively overcomes the challenges of depth feature extraction caused by reflection, refraction, and stacking of transparent consumables in automated stem cell preparation by integrating two-dimensional semantic extraction and local three-dimensional point cloud spatial mapping, thus improving the pose estimation accuracy of transparent culture dishes.

[0016] Other beneficial effects of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating an embodiment of the present invention; Figure 2 This is a schematic diagram of the lightweight target detection network in an embodiment of the present invention; Figure 3 This is a schematic diagram of the transparent petri dish pose estimation device according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the terminal device in an embodiment of the present invention. Detailed Implementation

[0018] To make the technical problems, solutions, and advantages of this invention clearer, a detailed description will be provided below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0020] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a locking connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0021] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0022] This invention addresses existing problems by providing a method and related equipment for estimating the pose of a transparent petri dish.

[0023] like Figure 1 As shown, an embodiment of the present invention provides a method for estimating the pose of a transparent petri dish, comprising: Step 1: Acquire RGB color images and synchronously aligned depth images of the cell preparation stage with transparent culture dishes placed on it; Step 2: Input the RGB color image into the pre-trained lightweight object detection network for recognition to obtain the two-dimensional bounding box of the transparent petri dish; Step 3: Extract the local depth map of the transparent culture dish from the depth image based on the two-dimensional bounding box, and map the local depth map into three-dimensional local point cloud data by combining the distortion correction and intrinsic parameter matrix of the acquisition device. Step 4: Perform instance segmentation on the 3D local point cloud data to obtain the point cloud of the target petri dish to be grasped, and perform fine registration between the point cloud of the target petri dish and the 3D standard point cloud model of the transparent petri dish to obtain the pose estimation result of the transparent petri dish.

[0024] In this embodiment of the invention, the cell preparation workbench is part of an automated stem cell preparation workstation, which also includes a collection device, a collaborative robotic arm, and a control center; its working process is as follows: The RGB color image and synchronously aligned depth image of the cell preparation operating table are acquired by the acquisition device and input into the control center. After the control center processes the input image as described in steps 1-4 above, it generates control commands to control the end flexible gripper on the collaborative robotic arm to complete the gripping of the transparent culture dish.

[0025] In this embodiment of the invention, since transparent cell culture dishes are prone to refraction and reflection under complex lighting conditions, it is necessary to use a hardware synchronization triggering mechanism to ensure that the output RGB color image with a resolution of 1920×1080 and the depth image are completely aligned in terms of timestamp and spatial resolution.

[0026] In embodiments of the present invention, such as Figure 2 As shown, the lightweight target detection network adopts an improved YOLO network structure. From left to right, the network consists of an input terminal, a central network layer, an output branch, and a result terminal. The connection relationships and functions of each module are as follows: The input terminal is located at the front end of the network and is used to receive RGB color images collected on site and transmit them to the backbone network of the central network layer. The front end of the central network layer is the backbone network, which uses CSPDarknet. It is responsible for receiving image data from the input end, extracting features layer by layer, and generating the initial feature map. The output end of the backbone network is connected in series with convolutional layers for feature processing. Figure 2 The diagram shows the convolution process with dimensions H×W×C and H×W×4 respectively. This central network layer also integrates a path aggregation network, which is used to splice and fuse the deep semantic features extracted by the backbone network with the shallow localization features to achieve deep semantic interaction of multi-scale features and enhance the ability to extract key features of the target. The output branch is connected in parallel to the end of the central network layer. It receives the multi-scale feature map after feature fusion and is divided into two prediction branches. The first branch is used to calculate and output the class confidence of the target to be captured, and the second branch is used to regress and calculate the width, height and center point position parameters of the target box. The result terminal is connected to the end of the output branch, receiving the regressed position parameters and category labels, mapping them onto the original image and outputting the final target detection box and confidence information (such as "flask 0.89"), thereby completing high-precision target recognition and planar localization, providing a priori basis for subsequent pose estimation.

[0027] In the training phase of the lightweight object detection network, this invention optimizes the construction of the total loss function to overcome the problem of inaccurate bounding box regression caused by blurred edges of transparent materials. The resulting loss function expression for the lightweight object detection network is as follows: ; in, This represents the total loss value. This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. Indicates the center point of the prediction box Center point of the real frame The Euclidean distance between them This represents the diagonal distance between the predicted bounding box and the ground truth bounding box, representing the minimum closure region. This represents the aspect ratio consistency penalty term, used to constrain the deformation of the prediction box to adapt to the geometry of the petri dish. , This indicates the width of the prediction box. Indicates the height of the prediction box. This represents the width of the actual bounding box. Indicates the height of the actual bounding box. This represents the balance ratio coefficient. , This represents the confidence loss value. Represents the classification loss value. , , All of these represent weighting coefficients.

[0028] The lightweight target detection network in this embodiment of the invention can accurately select the edges of the culture dish with weak features due to reflection by using gradient backpropagation of the total loss function.

[0029] Specifically, the local depth map of the transparent culture dish is extracted from the depth image based on the two-dimensional bounding box, including: Extracting the region of interest enclosed by a two-dimensional bounding box in a color image; By cropping a local depth map corresponding to the region of interest from the depth image, a local depth map of the transparent culture dish is obtained.

[0030] Specifically, by combining distortion correction and intrinsic parameter matrices from the acquisition equipment, the local depth map is mapped into 3D local point cloud data, including: Since industrial camera lenses inevitably have distortion errors, radial and tangential distortion correction models need to be introduced to correct the pixel coordinates in the local depth map before performing 3D mapping to obtain ideal pixel coordinates. By utilizing the intrinsic parameter matrix of the acquisition device and based on the principle of pinhole imaging reverse perspective, the ideal pixel coordinates are upgraded to spatial point coordinates in the three-dimensional camera coordinate system to obtain three-dimensional local point cloud data.

[0031] Specifically, radial and tangential distortion correction models are introduced to correct the pixel coordinates in the local depth map, resulting in the expression for the ideal pixel coordinates: ; ; in, Represents the ideal pixel coordinates. Represents the pixel coordinates in the local depth map. This represents the radial distance from the pixel coordinate to the image center. This represents the radial distortion coefficient of the data acquisition device. This represents the tangential distortion coefficient of the acquisition device.

[0032] Specifically, using the intrinsic parameter matrix of the acquisition device and based on the principle of pinhole imaging reverse perspective, the expression for upgrading the ideal pixel coordinates to spatial point coordinates in the 3D camera coordinate system is as follows: ; ; ; in, This represents the coordinates of a point in the 3D camera coordinate system. Indicates the depth value. Indicates the optical center. Indicates the equivalent focal length.

[0033] Since the generated 3D local point cloud data often contains abrupt depth changes caused by reflections from the petri dish material and refraction from the internal liquid, this embodiment of the invention employs a statistical outlier removal algorithm based on local Gaussian distribution to clean the 3D local point cloud data before instance segmentation. The specific steps are as follows: For any point in a 3D local point cloud, the k nearest neighbors of that point in 3D space are located using the KD-tree nearest neighbor search algorithm. Calculate the average Euclidean distance between all points in a 3D local point cloud data and the k nearest neighbors in 3D space; By removing free noise points from the 3D local point cloud data based on the mean Euclidean distance, the denoised 3D local point cloud data is obtained.

[0034] In this embodiment of the invention, the expression for calculating the average Euclidean distance between all points in the three-dimensional local point cloud data and the k nearest neighbors in the three-dimensional space is as follows: ; in, Point With the third in three-dimensional space The average Euclidean distance between the nearest neighbors.

[0035] The specific process of removing free noise points from 3D local point cloud data based on the average Euclidean distance in this embodiment of the invention is as follows: Assume the global average distance follows a Gaussian distribution. If the average Euclidean distance satisfy Then the point Points identified as free noise caused by reflection from transparent materials are discarded. Keep this point ,in, This represents the global mean. Indicates standard deviation, This represents the preset empirical adjustment coefficient.

[0036] To address the problem of interference caused by the dense stacking of culture dishes on a stem cell preparation table, this invention performs instance segmentation on three-dimensional local point cloud data to obtain the point cloud of the target culture dish to be grasped, including: Set neighborhood search radius and core point density threshold ; Based on neighborhood search radius Core point density threshold The spatial distribution density along the Z-axis divides the 3D local point cloud data into several instance clusters. The points in each cluster are all located within the neighborhood search radius, and the number of points in each cluster is greater than or equal to the core point density threshold. ; By comparing the mean Z-axis coordinates of each cluster, the cloud point cluster with the smallest absolute Z-axis value (i.e., the highest physical height and closest to the overhead camera on the actual operating table) and meeting the effective area threshold is forcibly selected, and this cloud point cluster is marked as the unique target petri dish point cloud to be captured.

[0037] Specifically, in this embodiment of the invention, an iterative nearest-point algorithm based on singular value decomposition is used to precisely register the point cloud of the target culture dish to be grasped with the three-dimensional standard point cloud model of the transparent culture dish, thereby obtaining the pose estimation result of the transparent culture dish, including: A prior three-dimensional standard point cloud model of a transparent petri dish was retrieved as a benchmark; The geometric centroids of the three-dimensional standard point cloud model of the transparent petri dish and the point cloud of the target petri dish to be grasped are calculated respectively, and the geometric centroids are translated to the origin to decenter them; Construct a 3×3 cross covariance matrix, which is used to characterize the rotational offset relationship between the two sets of decentroided point clouds; Singular value decomposition is performed on the cross covariance matrix to obtain the decomposition results; Based on the least squares optimization principle, analytical solutions for the three-dimensional rotation matrix and three-dimensional translation vector that minimize the sum of squared point cloud errors are obtained. The pose estimation results of the transparent culture dish are obtained by iterating through the determinant of the three-dimensional rotation matrix.

[0038] Specifically, the expressions for calculating the geometric centroids of the 3D standard point cloud model of the transparent petri dish and the point cloud of the target petri dish to be grasped are as follows: ; ; in, The geometric centroid of a standard three-dimensional point cloud model representing a transparent petri dish. This represents the geometric centroid of the point cloud in the petri dish from which the target object is to be captured. Indicates the number of points. The first point cloud model representing a transparent petri dish in a standard three-dimensional point cloud is shown. One point, This indicates the first point in the petri dish point cloud representing the target to be captured. One point.

[0039] Specifically, the expression for decentrifugation by translating the geometric centroid to the origin is: ; ; in, This represents a standard 3D point cloud model after centroid removal. This represents the point cloud of the petri dish containing the target to be grasped after centroid decentrifugation.

[0040] Specifically, the 3×3 cross-covariance matrix is ​​constructed as follows: ; in, Represents the cross covariance matrix. express The first in A three-dimensional coordinate point, express The first in Three-dimensional coordinate points.

[0041] Specifically, the expression for performing singular value decomposition on the cross covariance matrix is: ; in, , Both represent orthogonal matrices. This represents a singular value diagonal matrix.

[0042] Specifically, based on the least squares optimization principle, the analytical solutions for the two sets of point cloud error summations—the 3D rotation matrix and the 3D translation vector—are as follows: ; ; in, Represents a three-dimensional rotation matrix. This represents a three-dimensional translation vector.

[0043] In order to prevent erroneous reflection transformations, the determinant of the three-dimensional rotation matrix is ​​calculated in this embodiment of the invention. If the determinant of the three-dimensional rotation matrix is ​​less than 0, then by the formula The three-dimensional rotation matrix is ​​corrected, and after multiple iterations to bring the error to converge, the optimal three-dimensional rotation matrix and three-dimensional translation vector are output. The optimal three-dimensional rotation matrix and three-dimensional translation vector are the pose estimation results of the transparent culture dish.

[0044] In this embodiment of the invention, to enable the control center in the automated stem cell preparation workstation to better control the collaborative robotic arm for precise grasping, the pose estimation results of the transparent culture dish are transformed into the robotic arm's base coordinate system using a hand-eye homogeneous transformation matrix. The target grasping pose is then calculated to control the flexible end effector on the collaborative robotic arm for grasping. The transformation expression is as follows: ; in, This represents the homogeneous coordinates of the origin of the transparent petri dish in its own coordinate system. This represents the pose estimation result of the transparent petri dish. This represents the extrinsic parameter matrix obtained through hand-eye calibration, used to characterize the relative geometric transformation relationship between the camera optical center and the origin of the robot arm's base coordinate system. This represents the target grasping pose in the robot arm's base coordinate system.

[0045] The embodiments of the present invention will be illustrated by specific examples, as follows: In this embodiment of the invention, a depth camera of model ZED M and a JKZ5 collaborative robotic arm are used as the acquisition device and the collaborative robotic arm.

[0046] It should be noted that, in this embodiment of the invention, the camera coordinate system takes the optical center of the left lens of the ZED M binocular camera as the origin, and the depth information has been reprojected and fully aligned to the left camera plane after being processed by the algorithm.

[0047] For steps 1 to 2 above, the ZED M camera acquires images of the worktable containing the transparent culture dish, with a resolution of 1920×1080; After extraction by the target detection network, the center pixel coordinates of the two-dimensional bounding box of the transparent culture dish are: ; The actual factory-calibrated intrinsic parameter data of the camera is retrieved, and the physical and mechanical tolerances of the lens are also taken into account. In this embodiment of the invention, the measured intrinsic parameter matrix is... The specific value is: ; Extract the depth value near the center point from the local depth map. ; Substituting the above actual calibration data into the expression for upgrading the ideal pixel coordinates to the spatial point coordinates in the 3D camera coordinate system, we perform a 3D spatial mapping and calculate the coordinates of the point in the camera coordinate system as follows: ; ; The initial spatial positions in the camera coordinate system are obtained by solving the problem. ; Set the number of nearest neighbors Standard deviation multipliers After calculating and removing outlier noise points, the DBSCAN algorithm is used based on... The topmost target point cloud was extracted by peeling off the axis height distribution; Extract the centroid of the target point cloud With respect to the standard model centroid Perform centroid removal processing; Constructing the cross covariance matrix The optimal three-dimensional rotation matrix is ​​obtained by performing singular value decomposition (SVD) on it. Translation vector for: ; ; Retrieve the hand-eye conversion matrix of the JKZ5 robotic arm obtained from offline calibration. ; Substituting the above pose results into the transformation expression and performing a series of multiplications, the final result is calculated to guide the robotic arm to grasp. The target pose transformation matrix is: ; The control center drives the JECHA ZU5 robotic arm according to the above. Translation components in a matrix The end effector gripper, controlled by rotational attitude, completes the grasping process.

[0048] This invention acquires RGB color images and synchronously aligned depth images of a cell preparation stage containing transparent culture dishes. The RGB color images are input into a pre-trained lightweight target detection network for recognition, resulting in a two-dimensional bounding box for the transparent culture dishes. This bounding box is used to extract local depth maps of the transparent culture dishes from the depth images. The local depth maps are then mapped to three-dimensional local point cloud data using distortion correction and intrinsic parameter matrices from the acquisition device. Instance segmentation is performed on the three-dimensional local point cloud data to obtain the point cloud of the target culture dish to be grasped. The point cloud of the target culture dish to be grasped is then precisely registered with the three-dimensional standard point cloud model of the transparent culture dish to obtain the pose estimation result of the transparent culture dish. Compared with existing technologies, this invention effectively overcomes the difficulty of depth feature extraction caused by reflection, refraction, and stacking of transparent consumables in automated stem cell preparation by integrating two-dimensional semantic extraction and local three-dimensional point cloud spatial mapping, thus improving the pose estimation accuracy of the transparent culture dishes.

[0049] This invention also provides a transparent petri dish pose estimation device 100, which applies a transparent petri dish pose estimation method, such as... Figure 3 As shown, the transparent petri dish pose estimation device 100 includes: The acquisition module 101 is used to acquire RGB color images and synchronously aligned depth images of the cell preparation stage on which transparent culture dishes are placed. The recognition module 102 is used to input the RGB color image into a pre-trained lightweight object detection network for recognition, and obtain the two-dimensional bounding box of the transparent petri dish; The mapping module 103 is used to extract the local depth map of the transparent culture dish from the depth image based on the two-dimensional bounding box, and to map the local depth map into three-dimensional local point cloud data by combining the distortion correction and intrinsic parameter matrix of the acquisition device. The registration module 104 is used to perform instance segmentation on the three-dimensional local point cloud data to obtain the point cloud of the target culture dish to be grasped, and to perform fine registration between the point cloud of the target culture dish to be grasped and the three-dimensional standard point cloud model of the transparent culture dish to obtain the pose estimation result of the transparent culture dish.

[0050] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0051] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0052] This invention also provides a terminal device, such as... Figure 4 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 4 The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100. When the processor D100 executes the computer program D102, it implements the above-described transparent culture dish pose estimation method.

[0053] The terminal device D10 can be a desktop computer, laptop, handheld computer, server, server cluster, or cloud server, etc. This terminal device may include, but is not limited to, a processor D100 and a memory D101. Those skilled in the art will understand that... Figure 4 This is merely an example of terminal device D10 and does not constitute a limitation on terminal device D10. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0054] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0055] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0056] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0057] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0058] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for estimating the pose of a transparent culture dish.

[0059] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a building device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0060] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for estimating the pose of a transparent petri dish, characterized in that, include: Step 1: Acquire RGB color images and synchronously aligned depth images of the cell preparation stage with transparent culture dishes placed on it; Step 2: Input the RGB color image into a pre-trained lightweight object detection network for recognition to obtain the two-dimensional bounding box of the transparent culture dish; Step 3: Extract the local depth map of the transparent culture dish from the depth image based on the two-dimensional bounding box, and map the local depth map into three-dimensional local point cloud data by combining the distortion correction and intrinsic parameter matrix of the acquisition device. Step 4: Perform instance segmentation on the three-dimensional local point cloud data to obtain the point cloud of the target culture dish to be grasped, and perform fine registration between the point cloud of the target culture dish to be grasped and the three-dimensional standard point cloud model of the transparent culture dish to obtain the pose estimation result of the transparent culture dish.

2. The method for estimating the pose of a transparent culture dish according to claim 1, characterized in that, The loss function expression of the lightweight target detection network is: ； in, This represents the total loss value. This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. Indicates the center point of the prediction box Center point of the real frame The Euclidean distance between them This represents the diagonal distance between the predicted bounding box and the ground truth bounding box, representing the minimum closure region. This indicates a penalty for aspect ratio consistency. This represents the balance ratio coefficient. This represents the confidence loss value. Represents the classification loss value. , , All of these represent weighting coefficients.

3. The method for estimating the pose of a transparent culture dish according to claim 1, characterized in that, Extracting the local depth map of the transparent culture dish from the depth image based on the two-dimensional bounding box includes: Based on the two-dimensional bounding box, the region of interest enclosed by the two-dimensional bounding box is extracted from the color image; A local depth map corresponding to the region of interest is cropped from the depth image to obtain the local depth map of the transparent culture dish.

4. The method for estimating the pose of a transparent petri dish according to claim 3, characterized in that, By combining distortion correction and intrinsic parameter matrices from the acquisition device, the local depth map is mapped into 3D local point cloud data, including: A radial and tangential distortion correction model is introduced to correct the pixel coordinates in the local depth map to obtain ideal pixel coordinates; Using the intrinsic parameter matrix of the acquisition device and based on the principle of pinhole imaging reverse perspective, the ideal pixel coordinates are upgraded to spatial point coordinates in the three-dimensional camera coordinate system to obtain three-dimensional local point cloud data.

5. The method for estimating the pose of a transparent culture dish according to claim 4, characterized in that, A radial and tangential distortion correction model is introduced to correct the pixel coordinates in the local depth map, resulting in the expression for the ideal pixel coordinates: ； ； in, Represents the ideal pixel coordinates. Represents the pixel coordinates in the local depth map. This represents the radial distance from the pixel coordinate to the image center. This represents the radial distortion coefficient of the data acquisition device. This represents the tangential distortion coefficient of the acquisition device.

6. The method for estimating the pose of a transparent culture dish according to claim 5, characterized in that, Using the intrinsic parameter matrix of the acquisition device and based on the principle of pinhole imaging reverse perspective, the expression for upgrading the ideal pixel coordinates to spatial point coordinates in the three-dimensional camera coordinate system is as follows: ； ； ； in, This represents the coordinates of a point in the 3D camera coordinate system. Indicates the depth value. Indicates the optical center. Indicates the equivalent focal length.

7. The method for estimating the pose of a transparent culture dish according to claim 4, characterized in that, Before performing instance segmentation on the three-dimensional local point cloud data, the method further includes: For any point in the three-dimensional local point cloud data, the k nearest neighbors of the point in the three-dimensional space are located by the KD-tree nearest neighbor search algorithm; Calculate the average Euclidean distance between all points in the three-dimensional local point cloud data and the k nearest neighbors in the three-dimensional space; By removing the free noise points in the three-dimensional local point cloud data based on the average Euclidean distance, the denoised three-dimensional local point cloud data is obtained.

8. A device for estimating the pose of a transparent petri dish, characterized in that, The transparent culture dish pose estimation device, using the transparent culture dish pose estimation method as described in any one of claims 1-7, comprises: The acquisition module is used to acquire RGB color images and synchronously aligned depth images of the cell preparation stage on which transparent culture dishes are placed. The recognition module is used to input the RGB color image into a pre-trained lightweight object detection network for recognition, and obtain the two-dimensional bounding box of the transparent culture dish; The mapping module is used to extract the local depth map of the transparent culture dish from the depth image based on the two-dimensional bounding box, and to map the local depth map into three-dimensional local point cloud data by combining the distortion correction and intrinsic parameter matrix of the acquisition device. The registration module is used to perform instance segmentation on the three-dimensional local point cloud data to obtain the point cloud of the target culture dish to be grasped, and to perform fine registration between the point cloud of the target culture dish to be grasped and the three-dimensional standard point cloud model of the transparent culture dish to obtain the pose estimation result of the transparent culture dish.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the transparent culture dish pose estimation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the transparent culture dish pose estimation method as described in any one of claims 1 to 7.

Images