An image recognition-based automatic uniform plating method for tissue fragments

Abstract

This invention provides an automatic and uniform plating method for tissue fragments based on image recognition, comprising the following steps: S1, driving the end effector of a robotic arm to a designated position for precise positioning; S2, acquiring images of the tissue fragments and their circular culture dishes, and precisely locating and cropping the circular culture dish region in the image; S3, performing image enhancement and morphological preprocessing to obtain a binary image separating the tissue fragments from the circular culture dish; S4, using connected component analysis and distance transform algorithms to calculate the start point set and the end point set; S5, pairing the start point set and the end point set based on proximity; S6, transforming the three-dimensional world coordinates; S7, controlling the end effector of the robotic arm to automatically and uniformly platch the tissue fragments. By combining deep learning object detection and image processing technologies, a solution for automatic compensation of end effector positioning deviation and localization of tissue fragments and idle areas is designed.

Description

Technical Field

[0001] This invention relates to the field of cell culture and tissue engineering technology, and in particular to an automatic uniform plating method for tissue fragments based on image recognition. Background Technology

[0002] In the fields of cell culture and tissue engineering, the application of automation technology provides crucial support for research in biomedicine and tissue repair. During cell culture, the uniform distribution of tissue fragments, such as umbilical cord fragments, is essential for improving cell culture quality and the validity of experimental results. Traditional manual tissue fragment plating is not only labor-intensive but also prone to uneven plating or poor repeatability. With the continuous development of automation technology, image recognition and robotic control techniques can enable automated and uniform plating of tissue fragments such as umbilical cord fragments, thereby improving work efficiency and reducing human interference.

[0003] To ensure the uniform distribution of umbilical cord and other tissue fragments in a culture dish according to a specific pattern, thus guaranteeing high-quality cell growth, image processing technology plays a crucial role in the detection and localization of umbilical cord tissue fragments. This technology can identify umbilical cord tissue fragments in real time and accurately calculate their position and morphology, thereby providing a basis for robots to perform accurate plating operations.

[0004] However, automated tissue plating still faces many unresolved issues, such as: low efficiency in automatic tissue fragment identification; technical difficulties in identifying and setting robot motion points; small individual tissue fragments after achieving a certain distribution effect, making accurate identification by the robot's levers difficult; and positional deviations in the robot's levers leading to uneven plating of small tissue fragments. Therefore, many research gaps and technical challenges remain in the field of automated uniform tissue plating technology. Summary of the Invention

[0005] The technical problems to be solved by this invention include compensation for positioning deviations of the robot's end effector lever and automatic uniform distribution of tissue fragments. To this end, this invention provides a method for calculating the position of the end effector lever and its automatic positioning correction, as well as an image processing-based method for identifying the distribution of tissue fragments in a circular culture dish for the automatic uniform spreading of umbilical cord tissue fragments. The purpose is to accurately identify the distribution of umbilical cord tissue fragments in the culture dish, providing precise guidance for the robot to perform the automatic spreading action.

[0006] To achieve the above objectives, the present invention provides an automatic uniform plating method for tissue fragments based on image recognition, comprising the following steps:

[0007] S1. Drive the end lever of the robotic arm to the designated position to accurately locate its three-dimensional world coordinates in the world coordinate system and correct the error;

[0008] S2. Collect images of tissue fragments and their circular culture dishes. Use the YOLOv5 convolutional neural network model as the target detection model and the disk detection method to accurately locate and crop the circular culture dish area in the image to obtain the disk image.

[0009] S3. Perform image enhancement and morphological preprocessing on the disc image obtained in S2, and binarize the disc image to obtain a binarized image of the tissue fragments separated from the circular culture dish.

[0010] S4. In the binarized image obtained in S3, the clustered regions of tissue fragments are extracted using the connected component analysis method. The centroid coordinates of each larger tissue fragment cluster region are calculated as the starting point set, and the pixels in the free area that are far away from the tissue fragments are calculated as the ending point set.

[0011] S5. Pair the starting point set and ending point set obtained in S4 with the nearest points;

[0012] S6. Calculate the three-dimensional world coordinates of the starting point set and the ending point set in the world coordinate system through coordinate system transformation;

[0013] S7. Calculate the three-dimensional world coordinates of the starting point set and the ending point set relative to the end lever of the driving robotic arm, and control the end lever of the driving robotic arm to automatically and evenly spread the fragments; when the standard deviation of the cluster area is small, the process ends.

[0014] Image processing technology plays a crucial role in the detection and localization of umbilical cord tissue fragments. This technology can identify umbilical cord tissue fragments in real time and accurately calculate their position and morphology, thus providing a basis for the robot to perform accurate plating operations. Simultaneously, larger clusters of identified tissue fragments require the application of multiple robot motion points to these clusters to improve work efficiency. Furthermore, since the individual areas of tissue fragments are relatively small after reaching a certain distribution effect, it is necessary to ensure that the robot's end effector accurately reaches the identified position points in the image. Various factors can cause positioning deviations in the robot's end effector; to eliminate these deviations, they must be compensated for to the robot's world coordinates before plating operations begin.

[0015] In view of this, the present invention proposes a method for calculating the position of the end lever and automatically correcting its positioning for automatically and evenly spreading umbilical cord tissue fragments in a circular culture dish, as well as a method for identifying the distribution of tissue fragments based on image processing. This method includes locating the umbilical cord tissue fragments using image recognition technology, planning their distribution in a transparent disc, and further guiding the end lever of the robotic arm to perform a uniform spreading operation, thereby achieving a precise and uniform distribution of umbilical cord tissue fragments in the culture dish.

[0016] Preferably, step S1, which involves driving the end effector of the robotic arm to a designated position to accurately locate its three-dimensional world coordinates in the world coordinate system and correct errors, specifically includes:

[0017] S1.1 Select a black area on the working plane of the end effector lever of the robotic arm, and select a specified position point p in the image coordinate system within this area. target =[u target ,v target ] T This position is the target position for the movement of the lever at the end of the robotic arm;

[0018] S1.2 calibrates the monocular camera using Zhang Zhengyou calibration and / or hand-eye calibration methods, and obtains the 3D world coordinates P based on the robot base through matrix transformation. target_world =[X target ,Y target Z target ] T ;

[0019] The monocular camera is fixed 1m above the lever working platform, and the robotic arm base is fixed to the lever working platform.

[0020] S1.3 Based on the three-dimensional world coordinates P obtained in step S1.2 target_world The end lever of the drive robotic arm moves to the designated position point;

[0021] S1.4 Within the black area of ​​the image, the tip of the lever at the end of the robotic arm is transparent and appears white. By utilizing the color difference between the lever tip and the black area, combined with image processing techniques, the position of the lever tip is identified, and the image coordinates p of the lever tip are obtained. rod =[u rod ,v rod ] T ;

[0022] S1.5 The transformation relationship obtained through the matrix transformation in step S1.2 is used to transform the image coordinates p of the lever tip. rod =[u rod ,v rod ] T Convert to 3D world coordinates P rod_world =[X rod ,Y rod Z rod ] T .

[0023] S1.6 calculates the deviation ΔP between the 3D world coordinates of a specified point in the image and the 3D world coordinates of the lever tip. This deviation is used for positioning compensation in subsequent robotic arm movements. The deviation calculation formula is:

[0024] ΔP = Ptarget_world -Ρ rod_world (7)

[0025] S1.7 Based on the calculated deviation ΔP, the robot's three-dimensional coordinates are compensated for in subsequent operations to ensure that the lever can accurately reach the predetermined position in each operation, thereby achieving precise positioning.

[0026] Preferably, the images of the tissue fragments and their circular culture dishes collected in step S2 are acquired by taking pictures with a monocular camera;

[0027] The specific use of the YOLOv5 convolutional neural network model as the target detection model includes: using the YOLOv5 convolutional neural network model to extract the Region of Interest (ROI) of the circular culture dish in the image of tissue fragments and their circular culture dishes;

[0028] The method of using the disk detection method to accurately locate and crop the circular petri dish region in the image specifically includes: using the HoughCircles transformation circle detection algorithm in OpenCV to detect the circular petri dish region, extracting the center pixel coordinates and radius of the circular petri dish region, cropping the image, and retaining only the disk region for analysis;

[0029] The image enhancement and morphological preprocessing of the disk image obtained in S2 in step S3 includes: disconnecting the tiny connecting pixels between the tissue fragments in the disk image;

[0030] The binarization of the disk image is performed using a threshold segmentation method.

[0031] Preferably, step S4, which involves extracting clustered regions of tissue fragments using connected component analysis and employing a distance transformation algorithm to calculate the centroid coordinates of each larger tissue fragment clustered region as the starting point set and calculating the pixel points in the free area that are far from the tissue fragments as the ending point set, specifically includes:

[0032] In the binarized image, the connected component search algorithm in OpenCV is used to analyze and extract all tissue fragment cluster regions. A distance transformation is then performed on each region to generate a distance image from the pixels within each connected region to the edge. In other words, the pixel value within a white connected region in the binary image is transformed into the minimum Euclidean distance to the edge of that connected region.

[0033]

[0034] In the formula Let (u,v) be the boundary of the i-th connected region, (u,v) represent the two-dimensional coordinates of the current pixel in the image, and (u′,v′) represent the connected region R. iThe coordinates of the pixel on the boundary, D(u,v) represents the minimum Euclidean distance from the current pixel to the boundary;

[0035] In each connected region, the pixel corresponding to the maximum distance value is extracted and used as the centroid pixel coordinate of the fragment group. If the area of ​​the connected region is less than a threshold A... th If the area of ​​a connected region is greater than the threshold A, no action is taken, as it is considered to have met the maximum area requirement for cell growth. th_big Then, multiple starting points are extracted from this region, where each starting point satisfies the Euclidean distance constraint ||p|| between them. i,S -p j,S ||2>δ S δ S This represents the minimum distance threshold between starting points to prevent repeated region adjustments; the set of starting points is obtained from the above steps.

[0036] Ω S ={p i,S =[u i,S ,v i,S ,1] T |i=1,...,N S} (9)

[0037] In the formula p i,S Let N be the homogeneous coordinates (3×1 vector) of the starting point of the i-th tissue fragment in the image. S The number of starting points;

[0038] The original binary image is flipped between foreground and background. Connectivity analysis and distance transformation are performed on the current foreground region (the area not covered by tissue fragments). Pixels far from tissue fragments within the free area are extracted as candidate endpoints, and a set of endpoints is constructed.

[0039] Ω E ={p j,E =[u j,E ,v j,E ,1] T |j=1,...,N E} (10)

[0040] In the formula p j,E Let N be the homogeneous coordinates (3×1 vector) of the j-th free target point in the image. E Let p be the number of endpoints. Each endpoint satisfies the Euclidean distance constraint ||p|. i,E -p j,E ||2>δ E δ E This represents the minimum distance threshold between endpoints to prevent overlap of idle areas.

[0041] Preferably, the nearest pairing of the starting point set and the ending point set in step S5 specifically includes:

[0042] For the starting set Ω S and the final set Ω E The endpoints are assigned based on proximity, with higher priority given to pairing the endpoints with the starting points closer to the edge of the transparent disk, resulting in several sets of starting-endpoint pairs:

[0043] Ι={(p m,S ,p m,E )},m=1,...,K (11)

[0044] In the formula, K represents the number of valid start-end pairs.

[0045] Preferably, the three-dimensional world coordinates of the starting point set and the ending point set in step S6 in the world coordinate system are P. i,start_world ,Ρ j,end_world ;

[0046] Step S7, which involves calculating the three-dimensional world coordinates of the starting point set and the ending point set relative to the end effector of the robotic arm, and controlling the end effector of the robotic arm to automatically and evenly spread the fragments, specifically includes:

[0047] The pose transformation of the tool coordinate system relative to the flange coordinate system is calculated using the four-point setting method. Four known points P are selected in the workspace of the lever tip. i =[X i ,Y i Z i ,RX i ,RY i ,RZ i ] T i = 1, 2, 3, 4, RX i ,RY i ,RZ i This is the rotation angle relative to the flange coordinate system;

[0048] The position and orientation of the robot's end effector (TCP) relative to the robot's sixth axis were calculated using the four-point setup method. TCP This achieves coordinate system transformation between the robotic arm's end effector and the TCP, obtaining the three-dimensional world coordinates P of the starting and ending pixel sets with respect to the lever. i,start_world,TCP ,Ρ j,end_world,TCP To achieve precise positioning of the lever; Position TCP The definition is as follows:

[0049] Position TCP =[X TCP ,Y TCP Z TCP ,RXTCP ,RY TCP ,RZ TCP ] T (12)

[0050] By using the four-point setting method and pose transformation, the three-dimensional world coordinates P of the lever tip are obtained. i,start_world,TCP ,Ρ j,end_world,TCP The world coordinate positioning deviation ΔP obtained by the automatic correction method of the end lever positioning is compensated to the world coordinate of the end lever, and the robot end is controlled to perform a uniform dispersing operation of umbilical cord tissue fragments.

[0051] Preferably, the condition for terminating the process in step S7 when the standard deviation of the cluster region is small is:

[0052] The process termination condition is std all_areas <std th or nums action <nums th std all_areas Let std be the standard deviation of the area of ​​all connected regions. th The standard deviation threshold, ranging from 150 to 200, represents the uniformity of the area size of tissue fragments. action nums represents the number of actions of the robotic arm (one action is defined as moving from a starting point to a corresponding ending point). th The threshold for the number of robotic arm movements means that when tissue fragments are evenly distributed, the standard deviation of the area of ​​all connected regions should be relatively small. When the standard deviation consistently fails to meet the requirements, the tissue fragments should be evenly distributed when the number of robotic arm movements reaches a certain number.

[0053] If the termination condition is not met, a new round of image recognition and robot actions will begin from step S1.

[0054] Under the same technical concept, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0055] Under the same technical concept, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon; characterized in that the computer program / instructions, when executed by a processor, implement the steps of the above method.

[0056] Under the same technical concept, the present invention also provides a computer program product, including a computer program / instructions; characterized in that the computer program / instructions, when executed by a processor, implement the steps of the above-described method.

[0057] The above-described solution of the present invention has the following beneficial effects:

[0058] This patent addresses the problem that uniformly dispersing tissue fragments in a culture dish is labor-intensive and cannot be automated. It develops an image recognition-based method for automatically and uniformly spreading tissue fragments in a culture dish, using a monocular camera as the image acquisition tool. By combining deep learning object detection and image processing technologies, a solution for automatic compensation of end effector positioning deviation, tissue fragment localization, and idle area localization was designed. First, a calibration point is set in a designated image area. After transformation from pixel coordinates to a world coordinate system based on the robotic arm base, the robotic arm moves to this point, identifying the actual position of the lever tip in the image. The positioning deviation between its position in the world coordinate system and the calibration point position is calculated for subsequent robotic arm movement deviation compensation. Next, a deep learning model is used to perform object detection on the circular culture dish. Within the ROI area, a circle detection algorithm accurately identifies the region where tissue fragments need to be distributed. Subsequently, image processing methods such as image binarization, morphological methods, connected component recognition, and pixel distance transformation are used to identify the centroids of each tissue fragment and idle area. Based on the proximity principle, several sets of motion start-end pairs are formed, converted to the world coordinate system, and sent to the robotic arm to achieve motion from the start point to the end point, achieving the goal of uniform distribution of tissue fragments.

[0059] Other beneficial effects of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0060] Figure 1 This is a comparison diagram of the positioning of the robotic arm end effector lever before correction in an embodiment of the present invention;

[0061] Figure 2 This is a comparison diagram of the positioning of the robotic arm end effector lever after calibration in an embodiment of the present invention;

[0062] Figure 3 This is an example of the YOLOv5 convolutional neural network model's recognition effect on circular petri dishes in an embodiment of the present invention;

[0063] Figure 4 This is a diagram illustrating the detection effect of the disc detection method in an embodiment of the present invention;

[0064] Figure 5 This is a binarized image of the separation of tissue fragments from a circular culture dish in an embodiment of the present invention;

[0065] Figure 6-11 The image recognition point set diagrams for rounds 1, 3, 9, 12, 15, and 18 in the embodiments of the present invention are shown. Detailed Implementation

[0066] To make the technical problems, solutions, and advantages of this invention clearer, a detailed description will be provided below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0067] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0068] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a locking connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0069] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0070] This invention provides an automatic uniform plating method for tissue fragments based on image recognition, comprising the following steps:

[0071] S1. Drive the end lever of the robotic arm to the designated position to accurately locate its three-dimensional world coordinates in the world coordinate system and correct the error;

[0072] S2. Collect images of tissue fragments and their circular culture dishes. Use the YOLOv5 convolutional neural network model as the target detection model and the disk detection method to accurately locate and crop the circular culture dish area in the image to obtain the disk image.

[0073] S3. Perform image enhancement and morphological preprocessing on the disc image obtained in S2, and binarize the disc image to obtain a binarized image of the tissue fragments separated from the circular culture dish.

[0074] S4. In the binarized image obtained in S3, the clustered regions of tissue fragments are extracted using the connected component analysis method. The centroid coordinates of each larger tissue fragment cluster region are calculated as the starting point set, and the pixels in the free area that are far away from the tissue fragments are calculated as the ending point set.

[0075] S5. Pair the starting point set and ending point set obtained in S4 with the nearest points;

[0076] S6. Calculate the three-dimensional world coordinates of the starting point set and the ending point set in the world coordinate system through coordinate system transformation;

[0077] S7. Calculate the three-dimensional world coordinates of the starting point set and the ending point set relative to the end lever of the driving robotic arm, and control the end lever of the driving robotic arm to automatically and evenly spread the fragments; when the standard deviation of the cluster area is small, the process ends.

[0078] In one specific embodiment, the above process includes:

[0079] First, the following steps are used to achieve automatic compensation for the positioning deviation of the end lever:

[0080] 1. First, select a specific black area on the working plane of the lever, and then select a designated point p in the image within that area. target =[u target ,v target ] T This position is the target position that the end lever needs to reach.

[0081] 2. The monocular camera is calibrated using Zhang Zhengyou calibration, hand-eye calibration, and other processes, undergoing a series of matrix transformations to obtain the 3D world coordinates P based on the robot base. target_world =[X target ,Y target Z target ] T .

[0082] 3. Based on the three-dimensional world coordinates P calculated in step 2 target_world The end effector of the robotic arm moves to that point, such as... Figure 1 As shown.

[0083] 4. Within the black area of ​​the image, the tip of the lever at the end of the robotic arm is transparent and appears white. By utilizing the color difference between the lever tip and the black area, combined with image processing techniques, the position of the lever tip is identified, and the image coordinates p of the lever tip are obtained. rod =[u rod ,v rod ] T .

[0084] 5. Using the transformation relationship obtained in step 2, the image coordinates p of the lever tip are... rod =[u rod ,v rod ] T Convert to coordinates in a 3D world coordinate system P rod_world =[X rod ,Y rod Z rod ] T .

[0085] 6. Calculate the deviation ΔP between the 3D world coordinates corresponding to the specified position in the image and the actual 3D world coordinates of the lever tip. This deviation is used for compensation in subsequent operations. The deviation calculation formula is:

[0086] ΔP = P target_world -Ρ rod_world (13)

[0087] 7. Based on the calculated deviation ΔP, the robot's coordinates are compensated for in subsequent operations to ensure that the lever accurately reaches the predetermined position during each operation, thereby achieving precise positioning. Figure 2 As shown.

[0088] The above-mentioned automatic calibration method for lever positioning is used for precise positioning of the end of the tissue fragment mixing action, which can achieve high-precision positioning and movement of small tissue fragments. Finally, the following steps are used to perform image recognition of tissue fragments and empty areas in the culture dish:

[0089] 1. Images were acquired using a monocular camera, with each shot containing an image of umbilical cord tissue fragments and a circular culture dish.

[0090] 2. The YOLOv5 convolutional neural network model was used as the object detection model to extract the Region of Interest (ROI) of the transparent circular culture dish in the image, such as... Figure 3 As shown.

[0091] 3. In the ROI image obtained in step 2, the HoughCircles function in OpenCV is used to perform circle detection on the transparent disk region, and the center pixel coordinates (u) of the transparent disk are extracted. c ,v c Given the radius r, the image is cropped to retain only the disk region for analysis, such as... Figure 4 As shown.

[0092] 4. Perform image enhancement and morphological preprocessing on the disk image extracted in step 3, including opening and erosion operations, to break the fine connections between the tissue fragments; then, use a thresholding segmentation method to binarize the image to distinguish the tissue fragments from the disk background, such as... Figure 5As shown.

[0093] 5. In the obtained binarized image, the connected component analysis function in OpenCV is used to extract all tissue fragment cluster regions, and a distance transformation is performed on each region to generate a distance image from the pixel inside each connected region to the edge. That is, the pixel value in the white connected region in the binary image becomes the minimum pixel Euclidean distance to the edge of the connected region where the pixel is located:

[0094]

[0095] In the formula Let (u,v) be the boundary of the i-th connected region, (u,v) represent the two-dimensional coordinates of the current pixel in the image, and (u′,v′) represent the connected region R. i The coordinates of the pixel on the boundary, D(u,v) represents the minimum Euclidean distance from the current pixel to the boundary.

[0096] 6. In each connected region, extract the pixel corresponding to the maximum distance value and use it as the centroid pixel coordinate of the fragment group, where the area of ​​the connected region is less than the threshold A. th If the value is 400, no action is taken. If the area of ​​a connected region is greater than the threshold A, no action is taken. th_big If the value is 5000, then multiple starting points can be extracted from this region, where each starting point satisfies the Euclidean distance constraint ||p|| between them. i,S -p j,S ||2>δ S =15, to prevent repeated dialing of the area. The starting point set is obtained from the above operation steps:

[0097] Ω S ={p i,S =[u i,S ,v i,S ,1] T |i=1,...,N S} (15)

[0098] In the formula p i,S Let N be the homogeneous coordinates (3×1 vector) of the starting point of the i-th tissue fragment in the image. S This represents the number of starting points.

[0099] 7. Perform an inverse transformation on the background region (area not covered by tissue fragments) and foreground region in the binarized image. Using the same image processing method for extracting the starting point set, extract pixels in the free area that are far from the tissue fragments as candidate ending points. Construct the ending point set:

[0100] Ω E ={p j,E =[u j,E ,v j,E ,1]T |j=1,...,N E} (16)

[0101] In the formula p j,E Let N be the homogeneous coordinates (3×1 vector) of the j-th free target point in the image. E Let p be the number of endpoints. Each endpoint satisfies the Euclidean distance constraint ||p|. i,E -p j,E ||2>δ E =15, to prevent overlapping of idle areas.

[0102] 8. For the starting set Ω S and the final set Ω E The endpoints are assigned based on proximity, with higher priority given to pairing the starting and ending points closer to the edge of the transparent disk, resulting in several pairs of starting and ending points:

[0103] Ι={(p m,S ,p m,E )},m=1,...,K (17)

[0104] In the formula, K is the number of valid start and end point pairs, which is taken as 10.

[0105] 9. Calculate the three-dimensional world coordinates P of the starting pixel set and the ending pixel set in the world coordinate system through coordinate system transformation. i,start_world ,Ρ j,end_world .

[0106] 10. Calculate the pose transformation of the tool coordinate system relative to the flange coordinate system using the four-point setting method, and select four known points P in the workspace of the lever tip. i =[X i ,Y i Z i ,RX i ,RY i ,RZ i ] T i = 1, 2, 3, 4, RX i ,RY i ,RZ i is the rotation angle relative to the robot's coordinate system.

[0107] 11. The position and orientation of the robot's end effector (TCP) relative to the robot's sixth axis can be calculated using the four-point setup method. TCP This achieves coordinate system transformation between the robotic arm's end effector and the TCP, obtaining the three-dimensional world coordinates P of the starting and ending pixel sets with respect to the lever. i,start_world,TCP ,Ρ j,end_world,TCP This enables precise positioning of the lever. TCPThe data is as follows:

[0108] Position TCP = [-124.461, -126.348, 290.417, 0.000, 0.000, 0.000] T (18)

[0109] 12. By using the four-point setting method and pose transformation, the three-dimensional world coordinates P of the lever tip are obtained. i,start_world,TCP ,Ρ j,end_world,TCP The world coordinate positioning deviation ΔP obtained by the automatic correction method of the end lever positioning is compensated to the end lever coordinate, thereby controlling the robot end to perform the uniform spreading operation of umbilical tissue fragments.

[0110] 13. The process termination condition is std. all_areas <std th or nums action <nums th std all_areas Let std be the standard deviation of the area of ​​all connected regions. th The value is 200, nums action nums represents the number of actions of the robotic arm (one action is defined as moving from a starting point to a corresponding ending point). th A value of 500 signifies that when tissue fragments are evenly distributed, the standard deviation of the area of ​​all connected regions should be relatively small; conversely, the standard deviation should be large. If the standard deviation consistently fails to meet the requirement, the tissue fragments should also be relatively evenly distributed after a certain number of robotic arm movements. If the termination condition is not met, the next round of the same image recognition and robot actions begins from point 1. The distribution of tissue fragments after some stages of the actions is as follows: Figures 6 to 11 As shown, this continues until the process ends when the conditions for termination are met. Figures 6 to 11 As shown in Table 1, the area standard deviation is reached after 30 rounds, and the process ends.

[0111] Table 1 Figures 6 to 11 The corresponding standard deviation of tissue fragment area

[0112]

[0113] by Figure 11 For example, the three-dimensional world coordinates of all the starting and ending points of the lever are listed in Table 2.

[0114] Table 2 Figure 11 The corresponding three-dimensional world coordinates of all start and end points

[0115]

[0116]

[0117] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for automatic uniform plating of tissue fragments based on image recognition, characterized in that, Includes the following steps: S1. Drive the end lever of the robotic arm to the designated position to accurately locate its three-dimensional world coordinates in the world coordinate system and correct the error; S2. Collect images of tissue fragments and their circular culture dishes. Use the YOLOv5 convolutional neural network model as the target detection model and the disk detection method to accurately locate and crop the circular culture dish area in the image to obtain the disk image. S3. Perform image enhancement and morphological preprocessing on the disc image obtained in S2, and binarize the disc image to obtain a binarized image of the tissue fragments separated from the circular culture dish. S4. In the binarized image obtained in S3, the clustered regions of tissue fragments are extracted using the connected component analysis method. The centroid coordinates of each larger tissue fragment cluster region are calculated as the starting point set, and the pixels in the free area that are far away from the tissue fragments are calculated as the ending point set. S5. Pair the starting point set and ending point set obtained in S4 with the nearest points; S6. Calculate the three-dimensional world coordinates of the starting point set and the ending point set in the world coordinate system through coordinate system transformation; S7. Calculate the three-dimensional world coordinates of the starting point set and the ending point set relative to the end lever of the driving robotic arm, and control the end lever of the driving robotic arm to automatically and evenly spread the fragments; when the standard deviation of the cluster area is small, the process ends.

2. The method as described in claim 1, characterized in that, Step S1, which involves driving the end effector lever of the robotic arm to a designated position to accurately locate its three-dimensional world coordinates in the world coordinate system and correct errors, specifically includes: S1.1 Select a black area on the working plane of the end effector lever of the robotic arm, and select a specified position point in the image coordinate system within this area. This position is the target position for the movement of the lever at the end of the robotic arm; S1.2 calibrates the monocular camera using Zhang Zhengyou calibration and / or hand-eye calibration methods, obtaining the 3D world coordinates based on the robot base through matrix transformation. ; S1.3 Based on the three-dimensional world coordinates obtained in step S1.2 The end lever of the drive robotic arm moves to the designated position point; S1.4 Within the black area of ​​the image, the tip of the lever at the end of the robotic arm is transparent and appears white. By utilizing the color difference between the lever tip and the black area, combined with image processing techniques, the position of the lever tip is identified, and the image coordinates of the lever tip are obtained. ; S1.5 The transformation relationship obtained through the matrix transformation in step S1.2 is used to transform the image coordinates of the lever tip. Convert to 3D world coordinates ; S1.6 Calculate the deviation between the three-dimensional world coordinates of a specified point in the image and the three-dimensional world coordinates of the lever tip. This deviation is used for positioning compensation in subsequent robotic arm movements. The deviation calculation formula is as follows: S1.7 Based on the calculated deviation In subsequent operations, the robot's three-dimensional coordinates are compensated for for deviations to ensure that the lever can accurately reach the predetermined position during each operation, thus achieving precise positioning.

3. The method as described in claim 1, characterized in that, The images of the tissue fragments and their circular culture dishes collected in step S2 were acquired by taking pictures with a monocular camera; The specific use of the YOLOv5 convolutional neural network model as the target detection model includes: using the YOLOv5 convolutional neural network model to extract the Region of Interest (ROI) of the circular culture dish in the image of tissue fragments and their circular culture dishes; The method of using the disk detection method to accurately locate and crop the circular petri dish region in the image specifically includes: using the HoughCircles transformation circle detection algorithm in OpenCV to detect the circular petri dish region, extracting the center pixel coordinates and radius of the circular petri dish region, cropping the image, and retaining only the disk region for analysis; The image enhancement and morphological preprocessing of the disk image obtained in S2 in step S3 includes: disconnecting the tiny connecting pixels between the tissue fragments in the disk image; The binarization of the disk image is performed using a threshold segmentation method.

4. The method as described in claim 1, characterized in that, Step S4, which involves extracting clustered regions of tissue fragments using connected component analysis and employing a distance transform algorithm to calculate the centroid coordinates of each larger tissue fragment clustered region as the starting point set and calculating the pixels in the free area furthest from the tissue fragments as the ending point set, specifically includes: In the binarized image, the connected component search algorithm in OpenCV is used to analyze and extract all tissue fragment cluster regions. A distance transformation is then performed on each region to generate a distance image from the pixels within each connected region to the edge. In other words, the pixel value within a white connected region in the binary image is transformed into the minimum Euclidean distance to the edge of that connected region. In the formula For the first The boundaries of a connected region Represents the two-dimensional coordinates of the current pixel in the image. Represents connected regions Pixel coordinates on the boundary This represents the minimum Euclidean distance from the current pixel to the boundary. In each connected region, the pixel corresponding to the maximum distance value is extracted and used as the centroid pixel coordinate of the fragment group. If the area of ​​the connected region, i.e., the size of the tissue fragment, is less than a threshold... If the area of ​​a connected region is greater than the threshold, no action is taken, as it is considered to have met the maximum area requirement for cell growth. Then, multiple starting points are extracted from this region, where each starting point satisfies the Euclidean distance constraint between them. , This represents the minimum distance threshold between starting points to prevent repeated region adjustments; the set of starting points is obtained from the above steps. In the formula For the first The homogeneous coordinates of the starting points of each tissue fragment in the image. The number of starting points; The original binary image is flipped between foreground and background. Connectivity analysis and distance transformation are then performed on the current foreground region to extract pixels far from tissue fragments within the free area as candidate endpoint points, thus constructing an endpoint point set. In the formula For the first The homogeneous coordinates of each idle target point in the image. Let be the number of endpoints, where each endpoint satisfies the Euclidean distance constraint between them. , This represents the minimum distance threshold between endpoints to prevent overlap of idle areas.

5. The method as described in claim 1, characterized in that, The process of pairing the starting point set and the ending point set in step S5 specifically includes: For the starting set and the final set The endpoints are assigned based on proximity, with higher priority given to pairing the endpoints with the starting points closer to the edge of the transparent disk, resulting in several sets of starting-endpoint pairs: In the formula The number of valid start-end pairs.

6. The method as described in claim 1, characterized in that, The three-dimensional world coordinates of the starting point set and the ending point set in step S6 in the world coordinate system are as follows: , ; Step S7, which involves calculating the three-dimensional world coordinates of the starting point set and the ending point set relative to the end effector of the robotic arm, and controlling the end effector of the robotic arm to automatically and evenly spread the fragments, specifically includes: The pose transformation of the tool coordinate system relative to the flange coordinate system is calculated using the four-point setting method. Four known points are selected in the workspace of the lever tip. , This is the rotation angle relative to the flange coordinate system; The position and orientation of the robot's end effector TCP relative to the robot's sixth axis were calculated using the four-point setup method. This achieves coordinate system transformation between the robotic arm's end effector and the TCP, obtaining the three-dimensional world coordinates of the starting and ending pixel sets relative to the lever. , This enables precise positioning of the lever; The definition is as follows: The three-dimensional world coordinates of the lever tip are obtained through the four-point setting method and pose transformation. , The world coordinate positioning deviation obtained by the automatic correction method for end lever positioning The coordinates are compensated to the world coordinates of the end lever to control the robot end to perform a uniform dispersion operation of umbilical cord tissue fragments.

7. The method as described in claim 1, characterized in that, The condition for terminating the process in step S7 when the standard deviation of the cluster region is small is as follows: The process ends under the following conditions: or ,in Let be the standard deviation of the area of ​​all connected components, representing the degree of fluctuation in the area values ​​of each connected component. The standard deviation threshold, ranging from 150 to 200, represents the uniformity of the area size of tissue fragments. The number of actions of the robotic arm. The threshold for the number of robotic arm movements means that when tissue fragments are evenly distributed, the standard deviation of the area of ​​all connected regions should be relatively small. When the standard deviation consistently fails to meet the requirements, the tissue fragments should be evenly distributed when the number of robotic arm movements reaches a certain number. If the termination condition is not met, a new round of image recognition and robot actions will begin from step S1.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory; characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instructions stored thereon; characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instructions; characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images