Strain culture dish positioning method and system based on monocular vision

Abstract

The invention discloses a strain culture dish positioning method and system based on monocular vision. The positioning method comprises the following steps: acquiring an edge image of a culture dish by a monocular vision camera, and presetting a grabbing point in the edge image; the edge image and the template image are matched based on an improved SIFT algorithm, and two-dimensional coordinates of the grabbing points are obtained; acquiring depth information of the culture dish by a laser triangulation ranging method, and combining the depth information with the two-dimensional coordinates to form three-dimensional coordinates; according to the coordinate system conversion relation between the camera and the manipulator, the three-dimensional coordinates are converted into a manipulator coordinate system, and the manipulator is controlled to grab the culture dish at the grabbing point. The defect that a traditional monocular vision system is difficult to obtain depth information is overcome, the improved SIFT algorithm is used for matching images and obtaining two-dimensional coordinates, the depth information of the system is obtained in combination with a laser triangulation ranging method, and therefore three-dimensional coordinate information of a target is obtained, the detection efficiency and positioning precision of the monocular vision system are improved, and the detection precision of the monocular vision system is improved. The method has the characteristics of accuracy and rapidness.

Description

Technical Field

[0001] The present invention relates to the technical field of manipulator positioning, and in particular to a strain culture dish positioning method and system based on monocular vision. Background Art

[0002] Machine vision is an important branch of the rapidly developing artificial intelligence technology. In some environments that are not suitable for manual work, machine vision is often used to replace the human eye. With the widespread application of robots in various fields such as industry and agriculture, the precise positioning of manipulators, as the main research direction in the field of robotics, has also received more and more attention and research. Through visual perception, the target is accurately located and grasped, which improves the positioning speed and increases the accuracy and robustness of the manipulator's grasping. However, at present, when the manipulator locates tiny target objects, there will be errors in obtaining the three-dimensional pose of the target object, and there will also be errors in image matching and processing, which makes it difficult to achieve fast and accurate positioning. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide an accurate and rapid method and system for locating strain culture dishes based on monocular vision.

[0004] Technical solution: The method for locating a strain culture dish based on monocular vision of the present invention comprises the following steps:

[0005] An edge image of the culture dish is acquired by a monocular vision camera, and a grasping point is preset in the edge image;

[0006] Matching the edge image with a preset template image based on an improved SIFT algorithm to obtain the two-dimensional coordinates of the grasping point;

[0007] Acquire the depth information of the culture dish by laser triangulation, and combine it with the two-dimensional coordinates into three-dimensional coordinates;

[0008] According to the coordinate system conversion relationship between the camera and the manipulator, the three-dimensional coordinates are converted into the manipulator coordinate system, and the manipulator is controlled to the grasping point to grasp the culture dish.

[0009] Furthermore, the edge image is obtained by preprocessing the scene image, comprising the following steps:

[0010] The monocular vision camera is used to collect a scene image of the culture dish;

[0011] The scene image is sequentially subjected to: denoising according to a bilateral filtering algorithm, graying according to a weighted average method, and segmenting according to a maximum inter-class variance method, so as to extract the contour of the culture dish;

[0012] The contour is detected using the Canny operator to obtain an edge image.

[0013] Furthermore, obtaining the two-dimensional coordinates of the grasping point includes the following process:

[0014] The SIFT algorithm is improved, and after compressing the number of layers of the multi-scale Gaussian difference pyramid, the features of the edge image and the template image are extracted respectively, thereby determining the key points;

[0015] Generate multiple key point descriptors from the key points, superimpose the multiple key point descriptors to form a feature descriptor, calculate the Manhattan distance of the feature descriptors to determine similarity, and form a feature point set with similar key points;

[0016] A double threshold matching method is used to match the feature point set corresponding to the template image and the edge image to obtain a matching point set;

[0017] The center coordinates of the matching point set are determined as the two-dimensional coordinates of the grasping point.

[0018] Furthermore, each of the key points corresponds to three key point descriptors, and the neighborhood of the key point is divided from the center of the key point outward into a central area, a sub-central area, and an outer area that are nested in layers; at this time, the Manhattan distance formula of the corresponding feature descriptor is as follows:

[0019]

[0020] In the formula, (x i ,y i ) are the coordinates of the pixel points in the neighborhood; w1 represents the matching weight of the central area; w2 represents the matching weight of the sub-central area; w3 represents the matching weight of the outer area; i represents the dimension of the feature descriptor, i = 0, 1, ..., 23.

[0021] Furthermore, each of the key point descriptors is a square or annular area, and each square or annular area is filled with different grayscale values ​​in each gradient direction.

[0022] Furthermore, the laser triangulation ranging method is an oblique type.

[0023] Furthermore, the formula for calculating the depth information is as follows:

[0024]

[0025] Where f is the focal length of the camera; s is the distance from the intersection of the laser optical axis and the camera lens optical axis to the camera lens; x is the distance from the lens to the camera image plane; δ is the offset of the imaging point on the camera phase plane from the reference point, is the angle between the camera image plane and the optical axis of the camera lens; θ1 is the angle between the laser beam projected onto the reference surface and the normal of the culture dish; θ2 is the angle between the normal of the culture dish and the optical axis of the camera lens.

[0026] Furthermore, the process of determining the coordinate system conversion relationship between the camera and the manipulator is as follows:

[0027] The Zhang black and white chessboard calibration method is used to obtain the intrinsic parameters, extrinsic parameters and distortion parameters of the monocular vision camera;

[0028] The Eye-to-Hand model is selected, and the positions of the camera and the manipulator are quickly calibrated by hand and eye according to the intrinsic parameters, extrinsic parameters and distortion parameters to obtain the coordinate system conversion relationship between the camera and the manipulator.

[0029] Furthermore, the intrinsic parameters and extrinsic parameters are obtained after distortion correction processing.

[0030] The monocular vision-based bacterial strain culture dish positioning system of the present invention comprises:

[0031] An image acquisition module, used for acquiring an edge image of the culture dish by a monocular vision camera, and presetting a grasping point in the edge image;

[0032] A two-dimensional coordinate extraction module, used to match the edge image with a preset template image based on an improved SIFT algorithm to obtain the two-dimensional coordinates of the grasping point;

[0033] A three-dimensional coordinate combination module, used to obtain the depth information of the culture dish by laser triangulation, and combine it with the two-dimensional coordinates into three-dimensional coordinates;

[0034] The grasping module is used to transform the three-dimensional coordinates into the manipulator coordinate system according to the coordinate system transformation relationship between the camera and the manipulator, and control the manipulator to grasp the culture dish at the grasping point.

[0035] Beneficial effects: The present invention has the following significant effects: 1. Accurate: The present invention improves the deficiency of the traditional monocular vision system that it is difficult to obtain accurate coordinate information. The target culture dish image is processed based on the improved SIFT algorithm, the information of the neighborhood of the feature point is fully considered, and multiple square or ring areas are set and filled to better characterize the features. At the same time, Manhattan distance is used to judge the similarity, and the double threshold method is used for matching and other operations to optimize the feature points. In the above process, the two-dimensional coordinates of the culture dish are obtained by combining the preprocessing process of multiple algorithm fusion, and the depth information of the culture dish is obtained based on the oblique laser triangulation method, and the two-dimensional information and the depth information are integrated to obtain three-dimensional information, which can realize the accurate positioning of the culture dish; 2. Fast: The present invention adopts a fast hand-eye calibration method to establish a model conversion relationship between the culture dish and the manipulator, and adopts the oblique laser triangulation method to obtain the depth information, which integrates image matching and image positioning, improves the detection efficiency and positioning accuracy of the monocular vision system, and is faster and more convenient for positioning and matching the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the main process of the positioning method of the present invention;

[0037] Figure 2 Schematic diagram of the dual threshold matching process. DETAILED DESCRIPTION

[0038] The present invention is further explained below in conjunction with the accompanying drawings and specific implementation methods.

[0039] See also Figure 1 As shown, the method for locating a strain culture dish based on monocular vision of the present invention comprises the following steps:

[0040] An edge image of the culture dish is acquired by a monocular vision camera, and a grasping point is preset in the edge image;

[0041] Matching the edge image with a preset template image based on an improved SIFT algorithm to obtain the two-dimensional coordinates of the grasping point;

[0042] Acquire the depth information of the culture dish by laser triangulation, and combine it with the two-dimensional coordinates into three-dimensional coordinates;

[0043] According to the coordinate system conversion relationship between the camera and the manipulator, the three-dimensional coordinates are converted into the manipulator coordinate system, and the manipulator is controlled to the grasping point to grasp the culture dish.

[0044] The monocular vision-based bacterial strain culture dish positioning system of the present invention comprises:

[0045] An image acquisition module, used for acquiring an edge image of the culture dish by a monocular vision camera, and presetting a grasping point in the edge image;

[0046] A two-dimensional coordinate extraction module, used to match the edge image with a preset template image based on an improved SIFT algorithm to obtain the two-dimensional coordinates of the grasping point;

[0047] A three-dimensional coordinate combination module, used to obtain the depth information of the culture dish by laser triangulation, and combine it with the two-dimensional coordinates into three-dimensional coordinates;

[0048] The grasping module is used to transform the three-dimensional coordinates into the manipulator coordinate system according to the coordinate system transformation relationship between the camera and the manipulator, and control the manipulator to grasp the culture dish at the grasping point.

[0049] The edge image mentioned in the present invention is obtained by preprocessing the scene image, including the following steps:

[0050] First, the monocular vision camera is used to collect the scene image of the culture dish. The strain culture dish is located in an environment with sufficient and uniform light. The monocular vision camera is fixed above the strain culture dish. The visual camera takes a picture of the strain culture dish on the workbench to realize the collection of the original color image, and then uploads the collected image to the host computer for the next step of image processing.

[0051] Secondly, the scene image is sequentially subjected to: denoising according to the bilateral filtering algorithm, graying according to the weighted average method, and segmenting according to the maximum inter-class variance method to extract the outline of the culture dish. Among them, the bilateral filtering algorithm is used to denoise the image, reduce the redundant noise in the image, better retain the edge information of the culture dish in the image, avoid the problem of excessive smoothing or loss of details, and improve the accuracy of image recognition; the weighted average method is used to gray the image, and the values ​​of the R, G, and B components are weighted averaged according to the weighting coefficients based on the different sensitivities of the human eye to different colors, so as to better retain the brightness information and contrast of the image; the maximum inter-class variance method is used to segment the image, and the foreground and background of the image are separated, so as to extract the edge outline of the strain culture dish, and an initial threshold is set to divide the image into two foregrounds and backgrounds with different gray values. Through continuous iteration of the threshold, when the standard deviation of the average gray values ​​of the foreground and background of the image is the largest, it is the threshold to be determined.

[0052] Finally, the Canny operator is used to detect the edge of the strain culture dish on the outline to obtain an edge image. In addition, when the present invention presets the grasping point according to the edge image, the grasping point can be set at the center of the edge image, and when grasping, the center of the manipulator is aligned with the grasping point; the grasping point can also be set at the edge of the edge image, and when grasping, the grasping action of the manipulator is coordinated so that the finger force point of the manipulator is located at the grasping point.

[0053] The present invention obtains the two-dimensional coordinates of the grasping point including the following process:

[0054] First, the SIFT algorithm is improved, and after compressing the number of layers of the multi-scale Gaussian difference pyramid, the features of the edge image and the template image are extracted respectively, and then the key points are determined. Since the camera shooting angle is fixed and the shape of the culture dish is fixed, the scale change of the feature points of the target image is small, so that the multi-scale Gaussian difference pyramid information in the original SIFT algorithm is redundant. By comparing the relationship between the configuration of the multi-scale Gaussian difference pyramid and the number of extracted features, the algorithm time consumption, and the matching accuracy, a reasonable number of Gaussian difference pyramid layers is obtained. In this embodiment, the process is: the original 128-dimensional feature descriptor matrix X is centralized, that is, each column is zero-meaned; the centered matrix X′ is singularly decomposed to obtain a singular matrix; the eigenvalues ​​and eigenvectors are calculated, the eigenvalues ​​are sorted from large to small, and the first k eigenvectors are selected to form a matrix P; the original matrix X is projected with the matrix P to obtain a projection matrix Y, and each row in the Y matrix represents the k eigenvalues ​​of a sample in a 24-dimensional low-dimensional space.

[0055] Secondly, multiple key point descriptors are generated from the key points, and multiple key point descriptors are superimposed to form a feature descriptor. The Manhattan distance of the feature descriptors is calculated to determine the similarity, and similar key points are combined into a feature point set. Among them, each of the key points corresponds to three key point descriptors, and the neighborhood of the key point is divided into layers of circumferentially nested central area, sub-central area, and outer area from the center of the key point outward. Each of the key point descriptors is a square or annular area, and each square or annular area is filled with different grayscale values ​​in each gradient direction. In this embodiment, the square area is the central area, and the sub-central area and the outer area are annular areas.

[0056] Specifically, a 12×12 area with a total of 144 pixels is taken as the neighborhood of the feature point with the feature point as the center, and the key point descriptor is divided into three square or ring areas, namely w1, w2, and w3. Each area is divided into 8 gradient directions and filled with different grayscale values. The 8-dimensional vector representing the central area w1 and the 16-dimensional vector superposition amplitude of the sub-central ring area w2 and the outer ring area w3 constitute a 24-dimensional feature descriptor, which reduces the 128-dimensional feature point descriptor in the original algorithm to 24 dimensions. Since the closer the pixel to the feature point is, the greater the influence on the description of the feature point, the three areas are matched with different weights, and the Manhattan distance is used instead of the Euclidean distance as the similarity determination measure of the key points. The corresponding Manhattan distance formula of the feature descriptor is as follows:

[0057]

[0058] In the formula, (xi ,y i ) are the coordinates of the pixel points in the neighborhood; w1 represents the matching weight of the central area; w2 represents the matching weight of the sub-central area; w3 represents the matching weight of the outer area; i represents the dimension of the feature descriptor, i = 0, 1, ..., 23.

[0059] Afterwards, see Figure 2 , the double threshold matching method is used to match the feature point set corresponding to the template image and the edge image to obtain a matching point set. In the matching stage, the nearest neighbor distance and the next nearest neighbor distance of the first 8 dimensions of the feature descriptor are calculated. When the ratio of the nearest neighbor distance to the next nearest neighbor distance R1 is greater than the given threshold T1, the two feature descriptors do not match; if the two distance ratio R1 is less than the threshold T1, the ratio R2 of the nearest neighbor distance and the next nearest neighbor distance corresponding to the last 16 dimensions of the descriptor is calculated and compared with the threshold T2. When R2 is less than T2, it meets the matching features and the matching point pair is saved, otherwise it does not match, where T1 is less than T2. ​​The distance ratio expression is shown in the formula:

[0060]

[0061] In the formula, l f is the nearest neighbor Manhattan distance of the feature point, l s is the next nearest neighbor Manhattan distance, i=1,2.

[0062] Finally, the center coordinates of the matching point set are determined as the two-dimensional coordinates of the grab point. The center coordinates of the matching point set are shown in the formula:

[0063]

[0064] Where n is the number of elements in the matching point set of the template image and the scene image, (x i ,y i ) is the coordinate of the i-th point in the matching point set in the scene, i = 1, 2, …, n.

[0065] The laser triangulation distance measurement method adopted by the present invention is an oblique type. The red light laser obliquely projects the laser line onto the surface of the culture dish, forming a certain angle with the surface of the culture dish. The laser transmitter, the camera, and the culture dish of the strain to be identified constitute a laser triangulation distance measurement system. This embodiment selects an appropriate measurement angle between 15° and 45°, and accurately calibrates the angle. The red light laser transmitter emits a light spot on the phase plane of the strain culture dish, and calculates the distance H between the surface of the strain culture dish and the reference plane, that is, the depth information, according to the laser triangulation distance measurement principle and Gaussian imaging theorem.

[0066] Among them, Gaussian imaging theorem is shown in the formula:

[0067]

[0068] The formula for calculating the depth information is as follows:

[0069]

[0070] Where f is the focal length of the camera; s is the distance from the intersection of the laser optical axis and the camera lens optical axis to the camera lens; x is the distance from the lens to the camera image plane; δ is the offset of the imaging point on the camera phase plane from the reference point, is the angle between the camera image plane and the optical axis of the camera lens; θ1 is the angle between the laser beam projected onto the reference surface and the normal of the culture dish; θ2 is the angle between the normal of the culture dish and the optical axis of the camera lens.

[0071] The process of determining the coordinate system conversion relationship between the camera and the manipulator in the present invention is as follows:

[0072] First, the Zhang black and white chessboard calibration method is used to obtain the intrinsic parameters, extrinsic parameters and distortion parameters of the monocular vision camera. Among them, most lenses will produce a certain degree of distortion during the actual use of the camera, which will cause the captured bacterial colony image to be deformed. In order to reduce the impact of image distortion on subsequent image processing, the present invention performs distortion correction after calibrating the camera to obtain the intrinsic and extrinsic parameters to improve the accuracy, reduce the impact of image distortion on subsequent image processing, and compensate for the geometric distortion caused by hardware factors. Therefore, the intrinsic parameters and extrinsic parameters in the above process are obtained after distortion correction processing through mapping transformation.

[0073] Afterwards, the Eye-to-Hand model is selected, and the position of the camera and the manipulator is quickly calibrated by hand and eye according to the internal parameters, external parameters and distortion parameters to obtain the coordinate system conversion relationship between the camera and the manipulator. Among them, in order to simplify the cumbersome steps of the traditional hand-eye calibration method, a rapid hand-eye calibration method is adopted. First, a solid circle nine-point array drawing board with known size parameters is moved into the camera field of view, and the drawing board is photographed using a camera. The solid circle dots on the drawing board image obtained by the camera are preprocessed and identified by image processing software to obtain the coordinates of these nine points in the image. Then, the manipulator is operated to move the center point of the manipulator end effector to the top of the corresponding nine points, and the coordinates of the nine points in the manipulator coordinate system are recorded. The hand-eye affine transformation matrix is ​​obtained by matrix operation, and the hand-eye relationship matrix is ​​obtained by combining the corresponding formula. A conversion relationship model between the camera coordinate system and the manipulator base coordinate system is established, and a parameterized model of the manipulator sorting system is established, and then the position of the strain culture dish relative to the manipulator base coordinate system is calculated.

Claims

1. A method for locating a strain culture dish based on monocular vision, characterized in that: The positioning method comprises the following steps: An edge image of the culture dish is acquired by a monocular vision camera, and a grasping point is preset in the edge image; Matching the edge image with a preset template image based on an improved SIFT algorithm to obtain the two-dimensional coordinates of the grasping point; Acquire the depth information of the culture dish by laser triangulation method, and combine it with the two-dimensional coordinates into three-dimensional coordinates; According to the coordinate system conversion relationship between the camera and the manipulator, the three-dimensional coordinates are converted into the manipulator coordinate system, and the manipulator is controlled to the grasping point to grasp the culture dish.

2. The method for locating a strain culture dish based on monocular vision according to claim 1, characterized in that: The edge image is obtained by preprocessing the scene image, comprising the following steps: The monocular vision camera is used to collect a scene image of the culture dish; The scene image is sequentially subjected to: denoising according to a bilateral filtering algorithm, graying according to a weighted average method, and segmenting according to a maximum inter-class variance method, so as to extract the contour of the culture dish; The contour is detected using the Canny operator to obtain an edge image.

3. The method for locating a strain culture dish based on monocular vision according to claim 1, characterized in that: Obtaining the two-dimensional coordinates of the grasping point includes the following process: The SIFT algorithm is improved, and after compressing the number of layers of the multi-scale Gaussian difference pyramid, the features of the edge image and the template image are extracted respectively, thereby determining the key points; Generate multiple key point descriptors from the key points, superimpose the multiple key point descriptors to form a feature descriptor, calculate the Manhattan distance of the feature descriptors to determine similarity, and form a feature point set with similar key points; A double threshold matching method is used to match the feature point set corresponding to the template image and the edge image to obtain a matching point set; The center coordinates of the matching point set are determined as the two-dimensional coordinates of the grasping point.

4. The method for locating a strain culture dish based on monocular vision according to claim 3, characterized in that: Each of the key points corresponds to three key point descriptors, and the neighborhood of the key point is divided from the center of the key point outward into a central area, a sub-central area, and an outer area that are nested in layers. At this time, the Manhattan distance formula of the corresponding feature descriptor is as follows: In the formula, (x i ,y i ) are the coordinates of the pixel points in the neighborhood; w1 represents the matching weight of the central area; w2 represents the matching weight of the sub-central area; w3 represents the matching weight of the outer area; i represents the dimension of the feature descriptor, i = 0, 1, ..., 23.

5. The method for locating a strain culture dish based on monocular vision according to claim 3, characterized in that: Each of the key point descriptors is a square or annular area, and each square or annular area is filled with different gray values ​​in each gradient direction.

6. The method for locating a strain culture dish based on monocular vision according to claim 1, characterized in that: The laser triangulation distance measurement method is an oblique type.

7. The method for locating a strain culture dish based on monocular vision according to claim 6, characterized in that: The formula for calculating the depth information is as follows: Where f is the focal length of the camera; s is the distance from the intersection of the laser optical axis and the camera lens optical axis to the camera lens; x is the distance from the lens to the camera image plane; δ is the offset of the imaging point on the camera phase plane from the reference point, is the angle between the camera image plane and the optical axis of the camera lens; θ1 is the angle between the laser beam projected onto the reference surface and the normal of the culture dish; θ2 is the angle between the normal of the culture dish and the optical axis of the camera lens.

8. The method for locating a strain culture dish based on monocular vision according to claim 1, characterized in that: The process of determining the coordinate system transformation relationship between the camera and the manipulator is as follows: The Zhang black and white chessboard calibration method is used to obtain the intrinsic parameters, extrinsic parameters and distortion parameters of the monocular vision camera; The Eye-to-Hand model is selected, and the positions of the camera and the manipulator are quickly calibrated by hand and eye according to the intrinsic parameters, extrinsic parameters, and distortion parameters to obtain the coordinate system conversion relationship between the camera and the manipulator.

9. The method for locating a strain culture dish based on monocular vision according to claim 8, characterized in that: The intrinsic parameters and extrinsic parameters are obtained after distortion correction processing.

10. A strain culture dish positioning system based on monocular vision, characterized in that: include An image acquisition module, used for acquiring an edge image of the culture dish by a monocular vision camera, and presetting a grasping point in the edge image; A two-dimensional coordinate extraction module, used to match the edge image with a preset template image based on an improved SIFT algorithm to obtain the two-dimensional coordinates of the grasping point; A three-dimensional coordinate combination module, used to obtain the depth information of the culture dish by laser triangulation, and combine it with the two-dimensional coordinates into three-dimensional coordinates; The grasping module is used to transform the three-dimensional coordinates into the manipulator coordinate system according to the coordinate system transformation relationship between the camera and the manipulator, and control the manipulator to grasp the culture dish at the grasping point.

Images