Robot visual target positioning method and device

Abstract

The invention relates to a robot visual target positioning method and device, and belongs to the technical field of robot visual localization, and the method comprises the steps: carrying out the target detection of a two-dimensional image collected by a visual sensor through calling a target detection model, and obtaining a first bounding box coordinate of a target object; generating a mask image of a target area where the target object is located in the two-dimensional image; acquiring a matching point set between the mask image and a preset template image; converting the matching point set into a 3D point cloud, performing downsampling on the 3D point cloud, and determining a 3D affine rotation transformation matrix by using a least square method; extracting a feature point coordinate of a preset feature point in the mask image, and determining a first 3D coordinate of the feature point coordinate; determining anchor point constraint conditions; the 3D affine rotation transformation matrix is applied to the source point cloud based on the condition, point cloud registration between the source point cloud after downsampling and the target point cloud after downsampling is carried out, a point cloud rotation transformation matrix is obtained, and the source point cloud is converted into position information of the target object; and positioning accuracy is improved.

Description

TECHNICAL FIELD

[0001] The application relates to a robot vision target positioning method and device, and belongs to the technical field of robot vision positioning. BACKGROUND

[0002] With the development of vision sensors and computer vision algorithms, industrial robots based on machine vision for target positioning are widely used. In industrial robot applications, accurate target positioning is the key to achieving automated operation.

[0003] Traditional robot vision target positioning methods include the following:

[0004] 1. Based on the target detection method, the position and category of the target object in the image collected by the robot are determined. For example, a target detection model based on deep learning, such as YOLO, Faster R-CNN, etc., is used for target detection to achieve target positioning.

[0005] However, the accuracy of target detection is insufficient, making it difficult to adapt to complex industrial scenarios.

[0006] 2. Based on the feature points detected in the two-dimensional image, the rough pose of the target object is calculated; based on the rough pose, the corresponding 3D data points in the 3D point cloud are filtered to achieve three-dimensional positioning.

[0007] However, the synergy of image matching and point cloud registration is insufficient, making it difficult to achieve high-precision three-dimensional positioning; at the same time, the accuracy and efficiency of 3D transformation estimation are both low. SUMMARY

[0008] The application provides a robot vision target positioning method and device, which can solve the cooperative work of target detection, image matching, 3D transformation estimation and point cloud registration, and achieve high-precision target positioning. The application provides the following technical solutions:

[0009] In a first aspect, a robot vision target positioning method is provided, which comprises:

[0010] obtaining a two-dimensional image and a depth map collected by a vision sensor in a robot;

[0011] calling a target detection model to perform target detection on the two-dimensional image to obtain first bounding box coordinates of a target object in the two-dimensional image;

[0012] calling an image segmentation model based on the bounding box coordinates to generate a mask image of a target region in the two-dimensional image where the target object is located;

[0013] performing feature matching on the mask image and a preset template image to obtain a set of matching points between the mask image and the preset template image;

[0014] Based on the intrinsic matrix of the camera and the depth map, the matching point set is converted into a 3D point cloud, and after downsampling the 3D point cloud, a 3D affine rotation transformation matrix of the downsampled 3D point cloud is determined using a least squares method; the 3D point cloud includes a source point cloud and a target point cloud; the source point cloud is obtained by converting CAD modeling;

[0015] The feature point coordinates of the preset feature points in the mask image are extracted, and a first 3D coordinate of the feature point coordinates is determined;

[0016] Based on the second 3D coordinate of the preset feature points in the target point cloud and the first 3D coordinate, an anchor point constraint condition, i.e. a translation transformation matrix, for point cloud registration is determined;

[0017] With the anchor point constraint as a registration constraint, the 3D affine rotation transformation matrix is applied to the source point cloud, and after the source point cloud and the target point cloud are downsampled, point cloud registration is performed to obtain a point cloud rotation transformation matrix;

[0018] Based on the point cloud rotation transformation matrix, the source point cloud is converted into position information of the target object.

[0019] Optionally, the calling of the image segmentation model based on the bounding box coordinates to generate a mask image of a target region in which the target object is located in the two-dimensional image comprises:

[0020] Based on the multiple bounding box coordinates output by the target detection model, the area of each bounding box is determined to obtain a target bounding box with the largest area;

[0021] The bounding box information of the target bounding box is converted into a format supported by the image segmentation model to obtain converted bounding box information;

[0022] The color space of the two-dimensional image is converted into a color space supported by the image segmentation model to obtain a converted image;

[0023] The converted bounding box information and the converted image are input into the image segmentation model to generate a mask;

[0024] From multiple masks generated by the image segmentation model, a target mask with the highest score is determined;

[0025] The target mask is converted into a binary mask image.

[0026] Optionally, the feature matching of the mask image and the preset template image to obtain a matching point set between the mask image and the preset template image comprises:

[0027] The mask image is reduced to a preset size;

[0028] perform feature matching on the mask image and a preset template image using a feature matching algorithm constructed based on a general image matcher Gim and a lightweight feature matcher LightGlue, to obtain a plurality of pairs of matching points;

[0029] multiply each pair of matching points by a reduction multiple, and obtain the matching point set in the original size by upsampling.

[0030] Optionally, convert the matching point set into a 3D point cloud based on the intrinsic matrix of the camera and the depth map, and represented by the following formula:

[0031] X = (u - c_x) * D(u, v) / f_x;

[0032] Y = (v - c_y) * D(u, v) / f_y;

[0033] Z = D(u, v);

[0034] wherein D(u, v) represents a depth value of a 2D point in the matching point set with coordinates (u, v) in an image coordinate system in the depth map; (c_x, c_y) represents coordinates of an optical center of the camera in the intrinsic matrix in the image coordinate system; (f_x, f_y) represents coordinates of a focal length of the camera in the intrinsic matrix in the image coordinate system; and (X, Y, Z) represents coordinates of the 2D point (u, v) converted into the 3D point cloud.

[0035] Based on the preset feature points, align the source point cloud and the target point cloud in the XYZ direction first, and then only allow adjustment of the rotation of the point cloud in the point cloud registration.

[0036] Correspondingly, determine a 3D affine rotation transformation matrix of the 3D point cloud after downsampling using the least square method, including:

[0037] determine the 3D affine rotation transformation matrix between each pair of matching 3D points in the 3D point cloud based on minimizing an error function;

[0038] wherein the error function is represented by the following formula:

[0039] E = Σ||Q_i - (A·P_i + t) || 2 ;

[0040] wherein i represents an index of a pair of matching 3D points, P_i represents a 3D point in the source point cloud in the pair of matching 3D points, Q_i represents a 3D point in the target point cloud in the pair of matching 3D points; A represents a rotation and scaling matrix in the 3D affine rotation transformation matrix; and t represents a translation vector in the 3D affine rotation transformation matrix.

[0041] Optionally, the extracting the feature point coordinates of the preset feature points in the mask image and determining the first 3D coordinates of the feature point coordinates comprises:

[0042] detecting, based on the target detection model, the preset feature points in the mask image to obtain a center point coordinate of a second bounding box to which the preset feature points belong;

[0043] converting, based on the intrinsic matrix of the camera and the depth map, the center point coordinate into a 3D coordinate;

[0044] determining, in the 3D point cloud, the Z coordinates of each point within a preset range of the 3D coordinate;

[0045] determining, based on the Z coordinates of each point within the preset range, the Z coordinate corresponding to the center point coordinate to obtain the first 3D coordinate.

[0046] Optionally, the determining, in the 3D point cloud, the Z coordinates of each point within a preset range of the 3D coordinate comprises:

[0047] determining the size of the preset range based on the point cloud distribution of the 3D point cloud.

[0048] Optionally, the determining, based on the second 3D coordinates of the preset feature points in the target point cloud and the first 3D coordinates, the anchor point constraint condition in point cloud registration comprises:

[0049] calculating a translation vector between the second 3D coordinates and the first 3D coordinates to align the preset feature points in the two point clouds to obtain a translation transformation matrix, the translation transformation matrix being the anchor point constraint condition, and only a rotation matrix is calculated and the translation transformation matrix is no longer calculated thereafter.

[0050] Optionally, the applying the 3D affine rotation transformation matrix to the source point cloud with the anchor point constraint as the registration constraint, and performing point cloud registration on the source point cloud and the target point cloud after down-sampling the source point cloud and the target point cloud to obtain a point cloud rotation transformation matrix comprises:

[0051] down-sampling the source point cloud and the target point cloud;

[0052] performing affine transformation on the down-sampled source point cloud using the 3D affine rotation transformation matrix to obtain a transformed point cloud;

[0053] calculating a fast point feature histogram (FPFH) feature descriptor for the transformed point cloud and the down-sampled target point cloud, respectively;

[0054] The FPFH feature descriptor is used to establish a candidate corresponding relationship between the source point cloud after down-sampling and the target point cloud after down-sampling based on a random sample consensus (RANSAC) algorithm, to obtain a coarse registration result.

[0055] The position of the preset feature point is kept unchanged based on the anchor point constraint condition, to obtain a translation matrix; and an iterative closest point (ICP) algorithm is used for fine registration, to obtain the point cloud rotation transformation matrix.

[0056] Optionally, after the target detection model is used to perform target detection on the two-dimensional image to obtain first bounding box coordinates of a target object in the two-dimensional image, the method further includes:

[0057] Based on the first bounding box coordinates, a bounding box is drawn on a copy image of the two-dimensional image for visual display.

[0058] In a second aspect, a robot vision target positioning device is provided, which includes a processor and a memory; the memory stores a program, which is loaded and executed by the processor to implement the robot vision target positioning method in the first aspect.

[0059] In a fourth aspect, a computer readable storage medium is provided, which stores a program, which is loaded and executed by the processor to implement the robot vision target positioning method in the first aspect.

[0060] The beneficial effects of the present application include: providing an initial pose through image matching and 3D transformation estimation, improving the efficiency and accuracy of point cloud registration.

[0061] In addition, the 3D affine rotation transformation matrix is accurately estimated by using the least square method, improving the positioning accuracy.

[0062] In addition, the feature points detected in combination with images are used as anchor points, effectively solving the drift problem in point cloud registration.

[0063] In addition, the FPFH feature matching and the RANSAC algorithm are combined, improving the robustness of point cloud registration.

[0064] In addition, it is highly adaptable and can be applied to various industrial scenes and different types of feature points.

[0065] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, and the content of the description can be implemented as follows. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is a flowchart of a robot vision target positioning method provided by an embodiment of the present application;

[0067] Figure 2 is a flowchart of a process of generating a binary mask image according to an embodiment of the present application;

[0068] Figure 3 is a flowchart of a process of obtaining a rotation transformation matrix of a point cloud according to an embodiment of the present application;

[0069] Figure 4 is a flowchart of a process of point cloud registration using preset feature points as guides according to an embodiment of the present application;

[0070] Figure 5 is a block diagram of a robot vision target positioning device according to an embodiment of the present application;

[0071] Figure 6 is a block diagram of a robot vision target positioning device according to an embodiment of the present application. DETAILED DESCRIPTION

[0072] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings and embodiments. The following embodiments are used to illustrate the present application, but not to limit the scope of the present application.

[0073] Figure 1 is a flowchart of a robot vision target positioning method according to an embodiment of the present application. The present embodiment takes the method used in a robot equipped with a vision sensor as an example for illustration, and the vision sensor includes a camera and a depth sensor. The depth sensor includes but is not limited to a structured light camera, a Time-of-Flight (ToF) camera, or a binocular stereo vision system, etc., and the present embodiment does not limit the implementation of the depth sensor. The robot controls the vision sensor to synchronously perform image acquisition, to obtain a two-dimensional image acquired by the camera and a depth map acquired by the depth sensor. The method includes at least the following steps:

[0074] Step 101: Obtain a two-dimensional image and a depth map acquired by a vision sensor in a robot.

[0075] Exemplarily, the two-dimensional image is an RGB image.

[0076] Step 102: Call a target detection model to perform target detection on the two-dimensional image, to obtain first bounding box coordinates of a target object in the two-dimensional image.

[0077] The robot pre-obtains model files of the target detection model and an image segmentation model in the following to initialize model parameters, so that the robot can call the target detection model to perform target detection on the two-dimensional image, and call the image segmentation model to generate an accurate mask image.

[0078] Exemplarily, the target detection model is established and trained based on a YOLO model, and the image segmentation model is established and trained based on a Segment Anything Model (SAM).

[0079] After the robot loads the target detection model, the two-dimensional image is input into the target detection model for target detection to obtain a detection result. The detection result includes first bounding box coordinates of a bounding box to which each target object belongs, the bounding box being a rectangular box surrounding image content of the target object, and the first bounding box coordinates being stored in a pre-created bounding box list (boxes list).

[0080] Optionally, after the target detection model is called to perform target detection on the two-dimensional image to obtain the first bounding box coordinates of the target object in the two-dimensional image, the method further includes: based on the first bounding box coordinates, drawing the bounding box on a copy image of the two-dimensional image for visual display.

[0081] The bounding boxes obtained by the target detection model include multiple bounding boxes, and the robot draws the bounding box of each target object on the copy of the original two-dimensional image; uses a specific display mode (such as a green rectangular box) to mark the detected target; saves the image with the drawn bounding box to a designated folder for subsequent visual display.

[0082] In step 103, a mask image of a target region in which the target object is located in the two-dimensional image is generated based on the bounding box coordinates by calling the image segmentation model.

[0083] In one example, generating the mask image of the target region in which the target object is located in the two-dimensional image based on the bounding box coordinates by calling the image segmentation model includes the following steps:

[0084] In step 1031, the area of each bounding box is determined based on the multiple bounding box coordinates output by the target detection model to obtain a target bounding box with the largest area.

[0085] In step 1032, the bounding box information of the target bounding box is converted into a format supported by the image segmentation model to obtain converted bounding box information.

[0086] In step 1033, the color space of the two-dimensional image is converted into a color space supported by the image segmentation model to obtain a converted image.

[0087] In step 1034, the converted bounding box information and the converted image are input into the image segmentation model to generate a mask.

[0088] In step 1035, a target mask with the highest score is determined from multiple masks generated by the image segmentation model.

[0089] Step 1036, convert the target mask into a binary mask image (pixel value is 0 or 255).

[0090] In order to more clearly understand the generation process of the binary mask image, an example of the flow is described below with reference to Figure 2 The process includes the following steps:

[0091] Step 21, load the YOLO model and the SAM model;

[0092] Step 22, read the RGB image collected by the camera;

[0093] Step 23, call the YOLO model to detect the target in the RGB image, and obtain a detection result;

[0094] Step 24, extract the first bounding box coordinates corresponding to the target object from the detection result;

[0095] Step 25, visualize the bounding box to which the target object belongs based on the first bounding box coordinates;

[0096] Step 26, calculate the area of the bounding box to which each target object belongs;

[0097] Step 27, select the bounding box with the largest area;

[0098] Step 28, use the SAM model to perform mask prediction based on the first bounding box coordinates;

[0099] Step 29, select the target mask with the highest score;

[0100] Step 291, generate a binary mask image based on the target mask.

[0101] Step 104, perform feature matching on the mask image and the preset template image to obtain a matching point set between the mask image and the preset template image.

[0102] In one example, performing feature matching on the mask image and the preset template image to obtain a matching point set between the mask image and the preset template image includes: reducing the mask image to a preset size; using a feature matching algorithm constructed based on a generalizable image matcher (Gim) and a light glue, performing feature matching on the mask image and the preset template image to obtain a plurality of pairs of matching points; multiplying each pair of matching points by a reduction multiple to obtain the matching point set in the original size. In this embodiment, by reducing the mask image to the preset size, memory overflow can be avoided.

[0103] At step 105, based on the intrinsic matrix of the camera and the depth map, the matching point set is converted into a 3D point cloud, and after down-sampling the 3D point cloud, a 3D affine rotation transformation matrix of the down-sampled 3D point cloud is determined by using a least square method; the 3D point cloud includes a source point cloud and a target point cloud.

[0104] The source point cloud is converted from CAD modeling. By down-sampling the 3D point cloud, the number of points in the point cloud can be reduced, and subsequent calculation can be accelerated.

[0105] In one example, based on the intrinsic matrix of the camera and the depth map, the matching point set is converted into a 3D point cloud, which is expressed by the following formula:

[0106] X = (u - c_x) * D(u, v) / f_x;

[0107] Y = (v - c_y) * D(u, v) / f_y;

[0108] Z = D(u, v);

[0109] Wherein D(u, v) represents the depth value of the 2D point in the depth map in the matching point set with the coordinate (u, v) in the image coordinate system; (c_x, c_y) represents the optical center of the camera in the intrinsic matrix in the image coordinate system; (f_x, f_y) represents the focal length of the camera in the intrinsic matrix in the image coordinate system; (X, Y, Z) represents the coordinates of the 2D point (u, v) converted into the 3D point cloud;

[0110] Based on the preset feature points, the source point cloud and the target point cloud are aligned in the XYZ direction first, and then only the rotation of the point cloud is allowed to be adjusted in the point cloud registration;

[0111] Correspondingly, the 3D affine rotation transformation matrix of the down-sampled 3D point cloud is determined by using a least square method, which includes:

[0112] The 3D affine rotation transformation matrix between each pair of matching 3D points in the down-sampled 3D point cloud is determined based on minimizing an error function;

[0113] The error function is expressed by the following formula:

[0114] E = Σ||Q_i - (A·P_i + t) || 2 ;

[0115] Wherein i represents the index of a pair of matching 3D points, P_i represents the 3D point in the source point cloud in the pair of matching 3D points, Q_i represents the 3D point in the target point cloud in the pair of matching 3D points; A represents the rotation and scaling matrix in the 3D affine rotation transformation matrix; t represents the translation vector in the 3D affine rotation transformation matrix.

[0116] In step 106, the feature point coordinates of the preset feature points in the mask image are extracted, and the first 3D coordinates of the feature point coordinates are determined.

[0117] In one example, the feature point coordinates of the preset feature points in the mask image are extracted, and the first 3D coordinates of the feature point coordinates are determined, including:

[0118] In step 1061, the salient feature points of the target are identified. Specifically, the preset feature points in the mask image are detected based on the target detection model to obtain the center point coordinates of the second bounding box to which the preset feature points belong.

[0119] In step 1062, the Z coordinates of the preset feature points are determined according to the depth map. Specifically, the center point coordinates are converted into 3D coordinates based on the intrinsic matrix of the camera and the depth map; the Z coordinates of each point within the preset range of the 3D coordinates are determined in the 3D point cloud; and the Z coordinates corresponding to the center point coordinates are determined based on the Z coordinates of each point within the preset range, to obtain the first 3D coordinates.

[0120] In one example, the Z coordinates of each point within the preset range of the 3D coordinates are determined in the 3D point cloud, including: determining the size of the preset range based on the point cloud distribution of the 3D point cloud. In this way, the accuracy of calculating the Z coordinates corresponding to the center point coordinates can be improved.

[0121] In step 107, the anchor point constraint condition during point cloud registration is determined based on the second 3D coordinates of the preset feature points in the target point cloud and the first 3D coordinates.

[0122] In one example, the anchor point constraint condition during point cloud registration is determined based on the second 3D coordinates of the preset feature points in the target point cloud and the first 3D coordinates, including: calculating the translation vector between the second 3D coordinates and the first 3D coordinates to align the preset feature points in the two point clouds, to obtain a translation transformation matrix, and the translation transformation matrix is the anchor point constraint condition. Only the rotation matrix is calculated and the translation transformation matrix is no longer calculated thereafter.

[0123] In this step, by establishing the translation transformation matrix, the source point cloud and the target point cloud can be preliminarily aligned, the accuracy of the feature point coordinates is ensured, and the subsequent registration result is avoided from being affected.

[0124] In step 108, the 3D affine rotation transformation matrix is applied to the source point cloud with the anchor point constraint condition as the registration constraint, and the source point cloud and the target point cloud are registered after being down-sampled to obtain a point cloud rotation transformation matrix.

[0125] In one example, a 3D affine rotation transformation matrix is applied to a source point cloud with an anchor point constraint as a registration constraint, and point cloud registration is performed between the source point cloud and a target point cloud to obtain a point cloud rotation transformation matrix, including: downsampling the source point cloud and the target point cloud; performing affine transformation on the downsampled source point cloud using the 3D affine rotation transformation matrix to obtain a transformed point cloud; calculating Fast Point Feature Histogram (FPFH) feature descriptors for the transformed point cloud and the downsampled target point cloud respectively; establishing a candidate correspondence relationship between the downsampled source point cloud and the downsampled target point cloud based on a Random Sample Consensus (RANSAC) algorithm using the FPFH feature descriptors to obtain a coarse registration result; keeping the positions of preset feature points unchanged based on the anchor point constraint to obtain a translation matrix; and performing fine registration using an Iterative Closest Point (ICP) algorithm to obtain the point cloud rotation transformation matrix.

[0126] In order to more clearly understand the process of obtaining the point cloud rotation transformation matrix in the present application, an example of the process is described below with reference to Figure 3 The process includes the following steps:

[0127] Step 31, resizing the mask image to a preset size;

[0128] Step 32, performing feature matching on the mask image and a preset template image using Gim LightGlue to obtain multiple pairs of matching points;

[0129] Step 33, upsampling each pair of matching points back to the original size to obtain a matching point set;

[0130] Step 34, calculating 3D point coordinates corresponding to the matching point set to obtain a 3D point cloud;

[0131] Step 35, downsampling the 3D point cloud and estimating a 3D affine rotation transformation matrix of the downsampled 3D point cloud using a least squares method;

[0132] Step 36, calculating FPFH feature descriptors of the downsampled 3D point cloud;

[0133] Step 37, performing global registration on the downsampled 3D point cloud using a RANSAC algorithm;

[0134] Step 38, performing fine registration using an ICP algorithm to obtain a point cloud rotation transformation matrix.

[0135] In order to more clearly understand the point cloud registration process guided by the preset feature points in the present application, an example of the process is described below with reference to Figure 4 The process includes the following steps:

[0136] Step 41, detecting the positions of the preset feature points in the mask image;

[0137] Step 42, calculating the center point coordinates of the preset feature points;

[0138] Step 43, determining the Z coordinates of the preset feature points;

[0139] Step 44, adjusting the search radius of the 3D points around the preset feature points in the down-sampled 3D point cloud;

[0140] Step 45, calculating the translation vector of the preset feature points;

[0141] Step 46, constructing a translation transformation matrix based on the translation vector;

[0142] Step 47, performing preliminary alignment of the center of the preset feature points;

[0143] Step 48, applying anchor point constraints to perform fine registration of the down-sampled 3D point cloud.

[0144] Step 109, converting the source point cloud into the position information of the target object based on the point cloud rotation transformation matrix.

[0145] In summary, the robot vision target positioning method provided in the embodiment provides an initial pose through image matching and 3D transformation estimation, thereby improving the efficiency and accuracy of point cloud registration.

[0146] In addition, the least squares method is used to accurately estimate the 3D affine rotation transformation matrix, thereby improving the positioning accuracy.

[0147] In addition, the feature points detected by image detection are used as anchor points, effectively solving the drift problem in point cloud registration.

[0148] In addition, the combination of FPFH feature matching and RANSAC algorithm improves the robustness of point cloud registration.

[0149] In addition, it is highly adaptable and can be applied to various industrial scenes and different types of feature points.

[0150] The application scenarios of the robot vision target positioning method provided in the present application include but are not limited to positioning of industrial parts, positioning of complex-shaped parts, etc. The positioning of industrial parts and the positioning of complex-shaped parts are described below as examples.

[0151] 1. Positioning of industrial parts. The specific steps include: the robot uses the YOLO model to detect the target position and obtain the bounding box; the SAM model is used to extract the mask of the target area (i.e. the bounding box); the mask image is resized and then image matching is performed; the 3D point coordinates are calculated according to the camera intrinsic parameters and the depth map; the least squares method is used to estimate the 3D affine rotation transformation matrix; the feature point positions (such as pin holes) are detected and the center coordinates are calculated; the Z coordinates of the feature point centers are determined according to the depth map; and the point cloud registration is performed based on the feature point center constraint.

[0152] 2. Positioning of parts with complex shapes. The specific steps include: target detection and mask extraction are performed; multiple feature points (such as corner points, edges, etc.) are detected; the three-dimensional coordinates of the feature points are calculated; the multi-feature point constraint condition is established; and the point cloud registration is performed.

[0153] In summary, through the cooperative work of target detection, image matching, 3D transformation estimation and point cloud registration, high-precision target positioning is achieved. This method has the characteristics of high precision, strong robustness and good adaptability, and can be widely applied in the field of industrial robots.

[0154] Figure 5 is a block diagram of a robot vision target positioning device provided by an embodiment of the present application. The device includes at least the following modules: an image acquisition module 510, a target detection module 520, a mask prediction module 530, an image matching module 540, a first transformation module 550, a feature extraction module 560, an anchor point constraint module 570, a point cloud registration module 580, and a target positioning module 590.

[0155] The image acquisition module 510 is configured to acquire a two-dimensional image and a depth map collected by a vision sensor in a robot.

[0156] The target detection module 520 is configured to call a target detection model to perform target detection on the two-dimensional image, to obtain first bounding box coordinates of a target object in the two-dimensional image.

[0157] The mask prediction module 530 is configured to call an image segmentation model based on the bounding box coordinates to generate a mask image of a target area in which the target object is located in the two-dimensional image.

[0158] The image matching module 540 is configured to perform feature matching on the mask image and a preset template image, to obtain a matching point set between the mask image and the preset template image.

[0159] The first conversion module 550 is configured to convert the matching point set into a 3D point cloud based on an intrinsic matrix of the camera and the depth map, and determine a 3D affine rotation transformation matrix of the 3D point cloud after down-sampling the 3D point cloud; the 3D point cloud includes a source point cloud and a target point cloud; the source point cloud is obtained by CAD modeling conversion;

[0160] The feature extraction module 560 is configured to extract feature point coordinates of preset feature points in the mask image, and determine first 3D coordinates of the feature point coordinates;

[0161] The anchor point constraint module 570 is configured to determine an anchor point constraint condition in point cloud registration based on second 3D coordinates of the preset feature points in the target point cloud and the first 3D coordinates; the anchor point constraint condition is a translation transformation matrix.

[0162] The point cloud registration module 580 is configured to apply the 3D affine rotation transformation matrix to the source point cloud with the anchor point constraint condition as a registration constraint, and perform point cloud registration on the source point cloud and the target point cloud after down-sampling the source point cloud and the target point cloud, to obtain a point cloud rotation transformation matrix.

[0163] The target positioning module 590 is configured to convert the source point cloud into position information of the target object based on the point cloud rotation transformation matrix.

[0164] For related details, refer to the above method embodiments.

[0165] It should be noted that the robot vision target positioning apparatus provided in the above embodiments is only used for example in the division of the above functional modules when performing robot vision target positioning. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the robot vision target positioning apparatus is divided into different functional modules to complete all or part of the above described functions. In addition, the robot vision target positioning apparatus and the robot vision target positioning method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0166] Figure 6 is a block diagram of a controller provided in an embodiment of the present application. The apparatus at least includes a processor 601 and a memory 602.

[0167] The processor 601 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 601 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also known as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 601 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content required to be displayed by the display screen. In some embodiments, the processor 601 can further include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0168] The memory 602 can include one or more computer-readable storage media that can be non-transitory. The memory 602 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction for being executed by the processor 601 to implement the robot visual target positioning method provided by the method embodiments in the present application.

[0169] In some embodiments, the robot visual target positioning apparatus can also optionally include a peripheral device interface and at least one peripheral device. The processor 601, the memory 602, and the peripheral device interface can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface through a bus, a signal line, or a circuit board. Illustratively, the peripheral devices include, but are not limited to, radio frequency circuitry, a touch display screen, audio circuitry, and a power supply, etc.

[0170] Of course, the robot visual target positioning apparatus can also include fewer or more components, which are not limited in the present embodiment.

[0171] Optionally, the present application also provides a computer-readable storage medium, which stores a program, the program is loaded and executed by a processor to implement the robot visual target positioning method of the above-mentioned method embodiments.

[0172] Optionally, the present application also provides a computer product, comprising a computer readable storage medium, wherein a program is stored in the computer readable storage medium, and the program is loaded and executed by a processor to implement the robot vision target positioning method of the above method embodiment.

[0173] Optionally, the present application also provides an electric compressor, comprising a computer readable storage medium, wherein a program is stored in the computer readable storage medium, and the program is loaded and executed by a processor to implement the robot vision target positioning method of the above method embodiment.

[0174] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0175] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A robot visual target positioning method, characterized in that: The method comprises: Obtain two-dimensional images and depth maps collected by the visual sensor in the robot; Calling a target detection model to perform target detection on the two-dimensional image to obtain first bounding box coordinates of a target object in the two-dimensional image; Invoking an image segmentation model based on the bounding box coordinates to generate a mask image of a target area where the target object is located in the two-dimensional image; Performing feature matching on the mask image and a preset template image to obtain a matching point set between the mask image and the preset template image; Based on the camera's intrinsic parameter matrix and the depth map, the matching point set is converted into a 3D point cloud, and after downsampling the 3D point cloud, a least squares method is used to determine a 3D affine rotation transformation matrix of the downsampled 3D point cloud; the 3D point cloud includes a source point cloud and a target point cloud; the source point cloud is converted from CAD modeling; Extracting feature point coordinates of preset feature points in the mask image, and determining first 3D coordinates of the feature point coordinates; Determining an anchor point constraint condition for point cloud registration based on the second 3D coordinates of the preset feature point in the target point cloud and the first 3D coordinates, wherein the anchor point constraint condition is a translation transformation matrix; Using the anchor point constraint as a registration constraint, applying the 3D affine rotation transformation matrix to the source point cloud, performing point cloud registration after downsampling the source point cloud and the target point cloud, and obtaining a point cloud rotation transformation matrix; The source point cloud is converted into position information of the target object based on the point cloud rotation transformation matrix.

2. The method according to claim 1, characterized in that The calling of the image segmentation model based on the bounding box coordinates to generate a mask image of the target area where the target object is located in the two-dimensional image includes: Determining the area of ​​each bounding box based on the coordinates of the multiple bounding boxes output by the target detection model, and obtaining the target bounding box with the largest area; Converting the bounding box information of the target bounding box into a format supported by the image segmentation model to obtain converted bounding box information; Converting the color space of the two-dimensional image into a color space supported by the image segmentation model to obtain a converted image; Inputting the converted bounding box information and the converted image into the image segmentation model to generate a mask; Determining a target mask with a highest score from a plurality of masks generated by the image segmentation model; Convert the target mask into a binary mask image.

3. The method according to claim 1, characterized in that The performing feature matching on the mask image and the preset template image to obtain a matching point set between the mask image and the preset template image includes: reducing the mask image to a preset size; Using a feature matching algorithm based on a general image matcher Gim and a lightweight feature matcher LightGlue, feature matching is performed on the mask image and a preset template image to obtain multiple pairs of matching points; Each pair of matching points is multiplied by the reduction factor and sampled back to the original size to obtain the matching point set.

4. The method according to claim 1, wherein The matching point set is converted into a 3D point cloud based on the intrinsic parameter matrix of the camera and the depth map, which is expressed by the following formula: X=(u-c_x)*D(u,v) / f_x; Y=(v-c_y)*D(u,v) / f_y; Z=D(u,v); Wherein, D(u,v) represents the depth value of the 2D point with coordinates (u,v) in the depth map concentrated in the image coordinate system; (c_x,c_y) represents the coordinates of the optical center of the camera in the intrinsic parameter matrix in the image coordinate system; (f_x,f_y) represents the coordinates of the focal length of the camera in the intrinsic parameter matrix in the image coordinate system; (X,Y,Z) represents the coordinates of the 2D point (u,v) after being converted to a 3D point cloud; Based on the preset feature points, the source point cloud and the target point cloud are first aligned in the XYZ direction, and then only the rotation of the point cloud is allowed to be adjusted in the point cloud registration; Accordingly, the 3D affine rotation transformation matrix of the downsampled 3D point cloud is determined using the least squares method, including: Determining a 3D affine rotation transformation matrix between each pair of matching 3D points in the downsampled 3D point cloud based on a minimized error function; The error function is expressed as follows: E=Σ||Q_i-(A·P_i+t)|| 2 ; Where i represents the index of a pair of matching 3D points, P_i represents the 3D point in the source point cloud of a pair of matching 3D points, Q_i represents the 3D point in the target point cloud of a pair of matching 3D points; A represents the rotation and scaling matrix in the 3D affine rotation transformation matrix; t represents the translation vector in the 3D affine rotation transformation matrix.

5. The method according to claim 1, wherein The extracting feature point coordinates of a preset feature point in the mask image and determining first 3D coordinates of the feature point coordinates includes: Detecting preset feature points in the mask image based on the target detection model, and obtaining coordinates of a center point of a second bounding box to which the preset feature points belong; Converting the center point coordinates into 3D coordinates based on the camera's intrinsic parameter matrix and the depth map; Determine the Z coordinate of each point within the preset 3D coordinate range in the 3D point cloud; Based on the Z coordinates of each point within a preset range, the Z coordinate corresponding to the center point coordinate is determined to obtain the first 3D coordinate.

6. The method according to claim 5, characterized in that Determining the Z coordinate of each point within the preset 3D coordinate range in the 3D point cloud includes: The size of the preset range is determined based on the point cloud distribution of the 3D point cloud.

7. The method according to claim 1, characterized in that The determining of the anchor point constraint condition for point cloud registration based on the second 3D coordinates of the preset feature point in the target point cloud and the first 3D coordinates includes: Calculate the translation vector between the second 3D coordinate and the first 3D coordinate to align the preset feature point in the two point clouds, and obtain a translation transformation matrix. The translation transformation matrix is ​​the anchor point constraint condition. After that, only the rotation matrix is ​​calculated and no longer the translation transformation matrix.

8. The method according to claim 1, characterized in that The method of applying the 3D affine rotation transformation matrix to the source point cloud using the anchor point constraint as the registration constraint, performing point cloud registration after downsampling the source point cloud and the target point cloud, and obtaining a point cloud rotation transformation matrix includes: Downsampling the source point cloud and the target point cloud; Performing an affine transformation on the downsampled source point cloud using the 3D affine rotation transformation matrix to obtain a transformed point cloud; Calculating a fast point feature histogram (FPFH) feature descriptor for the transformed point cloud and the downsampled target point cloud respectively; Using the FPFH feature descriptor based on the random sampling consistency RANSAC algorithm to establish a candidate correspondence between the downsampled source point cloud and the downsampled target point cloud, and obtain a coarse registration result; Based on the anchor point constraint condition, the position of the preset feature point is kept unchanged to obtain a translation matrix; and an iterative closest point ICP algorithm is used for precise alignment to obtain the point cloud rotation transformation matrix.

9. The method according to claim 1, characterized in that After calling the target detection model to perform target detection on the two-dimensional image and obtaining the first bounding box coordinates of the target object in the two-dimensional image, the method further includes: A bounding box is drawn on the copy image of the two-dimensional image based on the first bounding box coordinates for visual display.

10. A robot visual target positioning device, characterized in that: The device includes a processor and a memory; the memory stores a program, and the program is loaded and executed by the processor to implement the robot visual target positioning method according to any one of claims 1 to 9.

Images