A method, system, device and medium for transporting a hose into a hole

Abstract

The present invention discloses a method, system, equipment and medium for transporting a hose into a hole, including performing a first color image recognition, calculating according to a first detection result, an internal parameter of a depth camera and a distortion coefficient of the depth camera, and averaging the vertex coordinates corresponding to the first detection result, controlling a mechanical arm to move toward the hole to be detected according to the coordinates of a geometric center point and a first normal vector of the hole to be detected so that the mechanical arm is located at a second position, performing a second color image recognition, comparing a projection transformation result of the second detection result with the first detection result to obtain the coordinates of the hole to be detected, calculating according to the coordinates of the hole to be detected and point cloud data of the second position to obtain a second normal vector, and controlling the mechanical arm to transport the hose into the hole to be detected according to the coordinates of the hole to be detected and the second normal vector, which can improve work efficiency and reduce safety hazards.

Description

Technical Field

[0001] The present invention relates to the technical field related to transporting a hose into a hole, and in particular to a method, system, equipment and medium for transporting a hose into a hole. Background Art

[0002] Currently, when emulsion explosives are used for blasting, the explosive hose is mainly aligned and penetrated into the explosive-filled hole through direct manual control or indirect control by remote-controlled robotic arms. This process has high safety risks and the efficiency of manual remote control is not high. Summary of the invention

[0003] The present invention aims to at least solve the technical problems existing in the prior art. To this end, the present invention provides a method, system, device and medium for transporting a hose into a hole, which can improve work efficiency and reduce safety hazards.

[0004] A first aspect of the present invention provides a method for transporting a hose into a hole, comprising the following steps:

[0005] Acquire a first hole color image at a first position, point cloud data at the first position, an internal parameter of a depth camera, and a distortion coefficient of the depth camera, wherein the first hole color image is an image taken by the depth camera at the first position, the first position is a position whose distance value to the hole to be detected is a preset first distance value, and the point cloud data at the first position is the point cloud data taken by the depth camera at the first position;

[0006] Inputting the first hole color image into the trained Yolov5 model for a first color image recognition to obtain a first detection result;

[0007] Calculate according to the first detection result, the internal parameter of the depth camera and the distortion coefficient of the depth camera to obtain the vertex coordinates corresponding to the first detection result, and average the vertex coordinates corresponding to the first detection result to obtain the coordinates of the geometric center point; calculate according to the coordinates of the geometric center point and the point cloud data of the first position to obtain the first normal vector of the hole to be detected;

[0008] Controlling the robotic arm to move toward the hole to be detected according to the coordinates of the geometric center point and the first normal vector of the hole to be detected, so that the robotic arm is located at a second position, and acquiring a second hole color image and point cloud data of the second position, wherein the second hole color image is an image taken by the depth camera at the second position, and the point cloud data of the second position is the point cloud data taken by the depth camera at the second position;

[0009] Inputting the second hole color image into the trained Yolov5 model for a second color image recognition to obtain a second detection result;

[0010] Performing projection transformation on the second detection result to obtain a projection transformation result; and comparing the projection transformation result with the first detection result to obtain the coordinates of the hole to be detected;

[0011] Calculating according to the coordinates of the hole to be detected and the point cloud data of the second position to obtain a second normal vector of the hole to be detected;

[0012] The robot arm is controlled to transport the hose into the hole to be detected according to the coordinates of the hole to be detected and the second normal vector of the hole to be detected.

[0013] According to the embodiments of the present invention, there are at least the following technical effects:

[0014] The method obtains a first hole color image at a first position, point cloud data at the first position, an internal parameter of a depth camera, and a distortion coefficient of the depth camera, wherein the first hole color image is an image taken by the depth camera at the first position, the first position is a position whose distance value to the hole to be detected is a preset first distance value, and the point cloud data at the first position is point cloud data taken by the depth camera at the first position, and the first hole color image is input into a trained Yolov5 model for a first color image recognition to obtain a first detection result, and the vertex coordinates corresponding to the first detection result are calculated according to the first detection result, the internal parameter of the depth camera, and the distortion coefficient of the depth camera, and the vertex coordinates corresponding to the first detection result are averaged to obtain the coordinates of the geometric center point; the first normal vector of the hole to be detected is obtained according to the coordinates of the geometric center point and the point cloud data of the first position, and the first normal vector of the hole to be detected is obtained according to the coordinates of the geometric center point. The first normal vector of the hole to be detected is used to control the robotic arm to move toward the hole to be detected, so that the robotic arm is located at the second position, and the second hole color image and the point cloud data of the second position are obtained, wherein the second hole color image is an image taken by the depth camera at the second position, and the point cloud data of the second position is the point cloud data taken by the depth camera at the second position. The second hole color image is input into the trained Yolov5 model for a second color image recognition to obtain a second detection result, and the second detection result is projected to obtain a projection transformation result; and the projection transformation result is compared with the first detection result to obtain the coordinates of the hole to be detected, and calculation is performed according to the coordinates of the hole to be detected and the point cloud data of the second position to obtain the second normal vector of the hole to be detected, and the robotic arm is controlled according to the coordinates of the hole to be detected and the second normal vector of the hole to be detected to transport the hose to the hole to be detected, which can improve work efficiency and reduce safety hazards.

[0015] According to some embodiments of the present invention, the Yolov5 model is trained in the following manner:

[0016] Acquire an initial hole opening image sample set, and annotate all initial hole opening images in the initial hole opening image sample set to obtain a hole opening image training sample set, wherein the annotated rectangular labels inscribe the holes to be detected in the initial hole opening images;

[0017] An initial Yolov5 model is constructed, and the initial Yolov5 model is trained according to the hole opening image training sample set to obtain the trained Yolov5 model.

[0018] According to some embodiments of the present invention, the calculating according to the first detection result, the internal parameter of the depth camera and the distortion coefficient of the depth camera to obtain the vertex coordinates corresponding to the first detection result, and averaging the vertex coordinates corresponding to the first detection result to obtain the coordinates of the geometric center point includes:

[0019] Acquire a first depth image at a first position and a depth value of the first depth image, wherein the first depth image is an image taken by the depth camera at the first position;

[0020] Performing distributed coding on the holes of the first detection result to obtain a hole detection result;

[0021] Calculate four pixel vertices of each hole according to the hole detection result;

[0022] The four pixel vertices are converted to the image coordinate system according to the intrinsic parameters of the depth camera, the distortion coefficient of the depth camera, and the depth value of the first depth image to obtain the coordinates of the camera coordinate system of the four pixel vertices, and the coordinates of the camera coordinate system of the four pixel vertices are averaged to obtain the coordinates of the geometric center point of the camera coordinate system, wherein the calculation formula for converting the four pixel vertices to the image coordinate system according to the intrinsic parameters of the depth camera, the distortion coefficient of the depth camera, and the depth value of the first depth image to obtain the coordinates of the camera coordinate system of the four pixel vertices is:

[0023] x img =(x pixel -u x ) / f x

[0024] y img =(y pixel -v y ) / f y

[0025]

[0026] x img_undis =x img *(1+k1*r 2 +k2*r 4 +k3*r 6 )+2*p1*x img *y img +p2*(r 2 +2*x img *x img )

[0027] y img_undis =y img *(1+k1*r 2 +k2*r 4 +k3*r 6 )+2*p2*x img *y img +p1*(r 2 +2*y img *y img )

[0028] z camera =Z img

[0029] x camera =z camera *x img_undis

[0030] y camera =z camera *y img_undis

[0031] Among them, f x is the internal parameter of the depth camera, f y is the internal parameter of the depth camera, u x is the internal parameter of the depth camera, v y is the internal parameter of the depth camera, k1 is the distortion coefficient of the depth camera, k2 is the distortion coefficient of the depth camera, k3 is the distortion coefficient of the depth camera, p1 is the distortion coefficient of the depth camera, p2 is the distortion coefficient of the depth camera, x pixel is the horizontal coordinate of the point in the pixel coordinate system, y pixel is the vertical coordinate of the point in the pixel coordinate system, x img is the horizontal coordinate of the point in the image coordinate system, y img is the vertical coordinate of the point in the image coordinate system, r is the distance, x img is the horizontal coordinate of the dedistorted point in the image coordinate system, y img is the ordinate of the point after dedistortion in the image coordinate system, Z img is the pixel depth value, x camera is the horizontal coordinate of the point in the camera coordinate system, ycamera is the vertical coordinate of the point in the camera coordinate system, z camera is the point depth value in the camera coordinate system;

[0032] Obtain a first rotation matrix and a first translation matrix from the robot arm end coordinate system at the first position to the camera coordinate system, obtain a second rotation matrix and a second translation matrix from the camera coordinate system at the first position to the robot arm end coordinate system, obtain a third rotation matrix and a third translation matrix from the robot arm end at the first position to the robot arm platform, and obtain a fourth rotation matrix and a fourth translation matrix from the platform coordinate system at the first position to the robot arm end coordinate system;

[0033] According to the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix, the third rotation matrix, the third translation matrix, the fourth rotation matrix, and the fourth translation matrix, the coordinates of the camera coordinate system corresponding to the geometric center point of the camera coordinate system are projected to the robot platform coordinate system through a conversion formula to obtain the coordinates of the geometric center point, wherein the conversion formula is:

[0034]

[0035] in, is the coordinate of coordinate system o2, is the coordinate of coordinate system o1, is the rotation matrix from coordinate system o1 to coordinate system O2, is the translation matrix from coordinate system O1 to coordinate system O2.

[0036] According to some embodiments of the present invention, performing a projection transformation on the second detection result to obtain a projection transformation result; and comparing the projection transformation result with the first detection result to obtain the coordinates of the hole to be detected, comprises:

[0037] Acquire a second depth image at a second position and a depth value of the second depth image, wherein the second depth image is an image taken by the depth camera at the second position;

[0038] Calculating according to the second detection result, the internal parameter of the depth camera, the distortion coefficient of the depth camera and the depth value of the second depth image to obtain the vertex coordinates corresponding to the second detection result;

[0039] According to the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix, the third rotation matrix, the third translation matrix, the fourth rotation matrix, and the fourth translation matrix, the vertex coordinates corresponding to the second detection result are projected to the camera coordinate system through a conversion formula to obtain the spatial coordinates of the camera at the first position;

[0040] Performing a projection transformation of the camera according to the spatial coordinates of the camera at the first position to obtain a projection set of the hole to be detected in the first hole color image;

[0041] Find the projection set of the hole to be detected in the first hole color image and the first hole to be detected whose intersection-and-union ratio in the first detection result is greater than 0, obtain the first hole to be detected, and acquire the coordinates of the first hole to be detected.

[0042] According to some embodiments of the present invention, the step of calculating according to the coordinates of the hole to be detected and the point cloud data of the second position to obtain the second normal vector of the hole to be detected includes:

[0043] Obtain a second rotation matrix and a second translation matrix from the camera coordinate system at the second position to the coordinate system of the end of the robotic arm, and obtain a third rotation matrix and a third translation matrix from the end of the robotic arm at the second position to the robotic arm platform;

[0044] Performing perspective transformation on the point cloud data at the second position and projecting it into a pixel coordinate system to obtain pixel coordinates of the point cloud data;

[0045] Extract the pixel coordinates in the point cloud data to meet and The first point cloud data is obtained, where α is a constant, (u ` , v ` ) is the pixel coordinate obtained by projecting the point cloud data, are the coordinates, width and height of the hole to be detected;

[0046] Filter out the first point cloud data Point cloud, and obtain the second point cloud data;

[0047] Using the Ransac algorithm to extract the plane from the point cloud of the second point cloud data to obtain the plane normal vector;

[0048] According to the second rotation matrix and the second translation matrix from the camera coordinate system of the second position to the coordinate system of the robot end and the third rotation matrix and the third translation matrix from the robot end of the second position to the robot platform, the plane normal vector is projected to the robot platform coordinate system through a conversion formula to obtain the second normal vector of the hole to be detected.

[0049] According to some embodiments of the present invention, the calculation formula for calculating the four pixel vertices of each hole according to the hole detection result is: and in, is the horizontal coordinate of the nth hole, is the ordinate of the nth hole, is the pixel width of the nth hole, is the pixel height of the nth hole.

[0050] According to some embodiments of the present invention, the method of transporting the hose into the hole further comprises:

[0051] Obtain the center, width, and height of the first detection result in the image pixel coordinate system;

[0052] If there is a hole with an intersection-and-union ratio of 0 in the projection set of the hole to be detected in the first hole color image and in the first detection result, determine whether the pixel coordinate projection in the first detection result of the hole with an intersection-and-union ratio of 0 is within the range of the width and height of the first detection result in the image pixel coordinate system: if the pixel coordinate projection in the first detection result of the hole with an intersection-and-union ratio of 0 is within the range of the width and height of the first detection result in the image pixel coordinate system, the hole with an intersection-and-union ratio of 0 is added to the first detection result; if the pixel coordinate projection in the first detection result of the hole with an intersection-and-union ratio of 0 exceeds the range of the width and height of the first detection result in the image pixel coordinate system, the hole with an intersection-and-union ratio of 0 is ignored.

[0053] A second aspect of the present invention provides a system for transporting a hose into a hole, the system for transporting a hose into a hole comprising:

[0054] a data acquisition module, configured to acquire a first hole color image at a first position, an internal parameter of a depth camera, and a distortion coefficient of the depth camera, wherein the first hole color image is an image taken by the depth camera at the first position, and the first position is a position at which the distance value to the hole to be detected is a preset first distance value;

[0055] A first color image recognition module, used for inputting the first hole color image into the trained Yolov5 model to perform a first color image recognition to obtain a first detection result;

[0056] A first detection result calculation module, configured to calculate according to the first detection result, the internal parameter of the depth camera and the distortion coefficient of the depth camera, obtain the vertex coordinates corresponding to the first detection result, and average the vertex coordinates corresponding to the first detection result to obtain the coordinates of the geometric center point;

[0057] A second position data acquisition module, used for controlling the robot arm to move toward the hole to be detected according to the coordinates of the geometric center point, so that the robot arm is located at a second position, and acquiring a second hole color image and point cloud data of the second position, wherein the second hole color image is an image taken by the depth camera at the second position, and the point cloud data of the second position is the point cloud data taken by the depth camera at the second position;

[0058] A second detection module, used for inputting the second hole color image into the trained Yolov5 model for a second color image recognition to obtain a second detection result;

[0059] A hole coordinate acquisition module to be detected, used for performing a projection transformation on the second detection result to obtain a projection transformation result; and comparing the projection transformation result with the first detection result to obtain the coordinates of the hole to be detected;

[0060] a normal vector calculation module, used for calculating according to the coordinates of the hole to be detected and the point cloud data of the second position to obtain the normal vector of the hole to be detected;

[0061] The hole operation module is used to control the robot arm to transport the hose to the hole to be detected according to the coordinates of the hole to be detected and the normal vector of the hole to be detected.

[0062] The system obtains a first hole color image at a first position, point cloud data at the first position, internal parameters of a depth camera, and a distortion coefficient of the depth camera, wherein the first hole color image is an image taken by the depth camera at the first position, the first position is a position whose distance value to the hole to be detected is a preset first distance value, and the point cloud data at the first position is point cloud data taken by the depth camera at the first position, and the first hole color image is input into a trained Yolov5 model for a first color image recognition to obtain a first detection result, and the vertex coordinates corresponding to the first detection result are calculated based on the first detection result, the internal parameters of the depth camera, and the distortion coefficient of the depth camera, and the vertex coordinates corresponding to the first detection result are averaged to obtain the coordinates of the geometric center point; the first normal vector of the hole to be detected is obtained based on the coordinates of the geometric center point and the point cloud data of the first position, and the first normal vector of the hole to be detected is obtained based on the coordinates of the geometric center point. The first normal vector of the hole to be detected is used to control the robotic arm to move toward the hole to be detected, so that the robotic arm is located at the second position, and the second hole color image and the point cloud data of the second position are obtained, wherein the second hole color image is an image taken by the depth camera at the second position, and the point cloud data of the second position is the point cloud data taken by the depth camera at the second position. The second hole color image is input into the trained Yolov5 model for a second color image recognition to obtain a second detection result, and the second detection result is projected to obtain a projection transformation result; and the projection transformation result is compared with the first detection result to obtain the coordinates of the hole to be detected, and calculation is performed according to the coordinates of the hole to be detected and the point cloud data of the second position to obtain the second normal vector of the hole to be detected, and the robotic arm is controlled according to the coordinates of the hole to be detected and the second normal vector of the hole to be detected to transport the hose to the hole to be detected, which can improve work efficiency and reduce safety hazards.

[0063] According to a third aspect of the present invention, there is provided an electronic device for transporting a hose into a hole, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the above-mentioned method for transporting the hose into the hole.

[0064] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned method of transporting a hose into a hole.

[0065] It should be noted that the beneficial effects between the second to fourth aspects of the present invention and the prior art are the same as the beneficial effects between the above-mentioned system for transporting a hose to a hole and the prior art, and will not be described in detail here.

[0066] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0068] Figure 1 is a flow chart of a method for transporting a hose into a hole according to an embodiment of the present invention;

[0069] Figure 2 It is an image division trend diagram of a method for transporting a hose into a hole according to an embodiment of the present invention;

[0070] Figure 3 FIG. 1 is a flow chart of a system for transporting a hose into a hole according to an embodiment of the present invention. DETAILED DESCRIPTION

[0071] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0072] In the description of the present invention, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0073] In the description of the present invention, it should be understood that descriptions involving orientation, such as orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0074] In the description of the present invention, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention in combination with the specific content of the technical solution.

[0075] Currently, when emulsion explosives are used for blasting, the explosive hose is mainly aligned and penetrated into the explosive-filled hole through direct manual control or indirect control by remote-controlled robotic arms. This process has high safety risks and the efficiency of manual remote control is not high.

[0076] In order to solve the above technical defects, refer to Figure 1 The present invention also provides a method for transporting a hose into a hole, comprising:

[0077] Step S101, obtaining a first hole color image at a first position, point cloud data at the first position, an internal parameter of a depth camera, and a distortion coefficient of the depth camera, wherein the first hole color image is an image taken by the depth camera at the first position, the first position is a position whose distance value to the hole to be detected is a preset first distance value, and the point cloud data at the first position is the point cloud data taken by the depth camera at the first position.

[0078] Step S102: Input the first hole color image into the trained Yolov5 model to perform a first color image recognition to obtain a first detection result.

[0079] Step S103, calculating according to the first detection result, the internal parameters of the depth camera and the distortion coefficient of the depth camera to obtain the vertex coordinates corresponding to the first detection result, and averaging the vertex coordinates corresponding to the first detection result to obtain the coordinates of the geometric center point; calculating according to the coordinates of the geometric center point and the point cloud data of the first position to obtain the first normal vector of the hole to be detected.

[0080] Step S104, controlling the robotic arm to move toward the hole to be detected according to the coordinates of the geometric center point and the first normal vector of the hole to be detected, so that the robotic arm is located at a second position, and obtaining a second hole color image and point cloud data of the second position, wherein the second hole color image is an image taken by the depth camera at the second position, and the point cloud data of the second position is the point cloud data taken by the depth camera at the second position.

[0081] Step S105: input the second hole color image into the trained Yolov5 model for a second color image recognition to obtain a second detection result.

[0082] Step S106, performing projection transformation on the second detection result to obtain a projection transformation result; and comparing the projection transformation result with the first detection result to obtain the coordinates of the hole to be detected.

[0083] Step S107 , performing calculation according to the coordinates of the hole to be detected and the point cloud data of the second position to obtain a second normal vector of the hole to be detected.

[0084] Step S108 , controlling the robot arm to transport the hose into the hole to be inspected according to the coordinates of the hole to be inspected and the second normal vector of the hole to be inspected.

[0085] The method obtains a first hole color image at a first position, point cloud data at the first position, an internal parameter of a depth camera, and a distortion coefficient of the depth camera, wherein the first hole color image is an image taken by the depth camera at the first position, the first position is a position whose distance value to the hole to be detected is a preset first distance value, and the point cloud data at the first position is point cloud data taken by the depth camera at the first position, and the first hole color image is input into a trained Yolov5 model for a first color image recognition to obtain a first detection result, and the vertex coordinates corresponding to the first detection result are calculated according to the first detection result, the internal parameter of the depth camera, and the distortion coefficient of the depth camera, and the vertex coordinates corresponding to the first detection result are averaged to obtain the coordinates of the geometric center point; the first normal vector of the hole to be detected is obtained according to the coordinates of the geometric center point and the point cloud data of the first position, and the first normal vector of the hole to be detected is obtained according to the coordinates of the geometric center point. The first normal vector of the hole to be detected is used to control the robotic arm to move toward the hole to be detected, so that the robotic arm is located at the second position, and the second hole color image and the point cloud data of the second position are obtained, wherein the second hole color image is an image taken by the depth camera at the second position, and the point cloud data of the second position is the point cloud data taken by the depth camera at the second position. The second hole color image is input into the trained Yolov5 model for a second color image recognition to obtain a second detection result, and the second detection result is projected to obtain a projection transformation result; and the projection transformation result is compared with the first detection result to obtain the coordinates of the hole to be detected, and calculation is performed according to the coordinates of the hole to be detected and the point cloud data of the second position to obtain the second normal vector of the hole to be detected, and the robotic arm is controlled according to the coordinates of the hole to be detected and the second normal vector of the hole to be detected to transport the hose to the hole to be detected, which can improve work efficiency and reduce safety hazards.

[0086] Specifically, in some embodiments, the first normal vector of the hole to be detected is obtained by calculating according to the coordinates of the geometric center point and the point cloud data of the first position, including:

[0087] Obtain a second rotation matrix and a second translation matrix from the camera coordinate system at the first position to the coordinate system of the end of the robotic arm, and obtain a third rotation matrix and a third translation matrix from the end of the robotic arm at the first position to the robotic arm platform;

[0088] Perform perspective transformation on the point cloud data at the first position and project it into a pixel coordinate system to obtain pixel coordinates of the point cloud data;

[0089] Extract pixel coordinates from point cloud data to meet and The first point cloud data is obtained, where α is a constant, (u ` , v ` ) is the pixel coordinate obtained by projecting the point cloud data, are the coordinates, width and height of the hole to be detected;

[0090] Filter out the first point cloud data and Point cloud, and obtain the second point cloud data;

[0091] The Ransac algorithm is used to extract the plane from the point cloud of the second point cloud data to obtain the plane normal vector;

[0092] According to the second rotation matrix and the second translation matrix from the camera coordinate system at the first position to the coordinate system of the robot end and the third rotation matrix and the third translation matrix from the robot end at the second position to the robot platform, the plane normal vector is projected to the robot platform coordinate system through the conversion formula to obtain the first normal vector of the hole to be detected.

[0093] In some embodiments, if the angle between the vector formed by the end point of the robot arm to the geometric center point and the normal vector is greater than 90 degrees, the normal vector direction is reversed and used as the first normal vector; if the angle between the vector formed by the end point of the robot arm to the geometric center point and the normal vector is less than 90 degrees, the normal vector is taken as the first normal vector. The coordinates of the center point of the set are translated by a set distance in the opposite direction of the first normal vector to obtain the coordinates of the first position offset center point as the second position.

[0094] In some embodiments, the Yolov5 model is trained as follows:

[0095] An initial hole opening image sample set is obtained, and all initial hole opening images in the initial hole opening image sample set are annotated to obtain a hole opening image training sample set, wherein the annotated rectangular labels are inscribed with holes to be detected in the initial hole opening images.

[0096] Construct an initial Yolov5 model, train the initial Yolov5 model according to the cave entrance image training sample set, and obtain a trained Yolov5 model.

[0097] In some embodiments, the coordinates of the vertices corresponding to the first detection result are obtained by calculating according to the first detection result, the internal parameters of the depth camera, and the distortion coefficient of the depth camera, and the coordinates of the vertices corresponding to the first detection result are averaged to obtain the coordinates of the geometric center point, including:

[0098] A first depth image at a first position and a depth value of the first depth image are obtained, wherein the first depth image is an image captured by a depth camera at the first position.

[0099] Distributed encoding is performed on the holes of the first detection result to obtain a hole detection result.

[0100] The four pixel vertices of each hole are calculated based on the hole detection results.

[0101] The four pixel vertices are converted to the image coordinate system according to the intrinsic parameters of the depth camera, the distortion coefficient of the depth camera, and the depth value of the first depth image to obtain the coordinates of the camera coordinate system of the four pixel vertices, and the coordinates of the camera coordinate system of the four pixel vertices are averaged to obtain the coordinates of the geometric center point of the camera coordinate system, wherein the four pixel vertices are converted to the image coordinate system according to the intrinsic parameters of the depth camera, the distortion coefficient of the depth camera, and the depth value of the first depth image to obtain the coordinates of the camera coordinate system of the four pixel vertices. The calculation formula is:

[0102] x img =(x pixel -u x ) / f x

[0103] y img =(y pixel -v y ) / f y

[0104]

[0105] x img_undis =x img *(1+k1*r 2 +k2*r 4 +k3*r 6 )+2*p1*x img *y img +p2*(r 2 +2*x img *x img )

[0106] y img_undis =y img *(1+k1*r 2 +k2*r 4 +k3*r 6 )+2*p2*x img *y img +p1*(r 2 +2*y img *y img )

[0107] z camera =Z img

[0108] x camera =z cameea *x img_undis

[0109] y camera =z camera *yimg_undis

[0110] Among them, f x is the internal parameter of the depth camera, f y is the internal parameter of the depth camera, u x is the internal parameter of the depth camera, v y is the internal parameter of the depth camera, k1 is the distortion coefficient of the depth camera, k2 is the distortion coefficient of the depth camera, k3 is the distortion coefficient of the depth camera, p1 is the distortion coefficient of the depth camera, p2 is the distortion coefficient of the depth camera, x pixel is the horizontal coordinate of the point in the pixel coordinate system, y pixel is the vertical coordinate of the point in the pixel coordinate system, x img is the horizontal coordinate of the point in the image coordinate system, y img is the vertical coordinate of the point in the image coordinate system, r is the distance, x img is the horizontal coordinate of the dedistorted point in the image coordinate system, y img is the ordinate of the point after dedistortion in the image coordinate system, Z img is the pixel depth value, x camera is the horizontal coordinate of the point in the camera coordinate system, y camera is the vertical coordinate of the point in the camera coordinate system, z camera It is the point depth value in the camera coordinate system.

[0111] Obtain the first rotation matrix and the first translation matrix from the robot arm end coordinate system at the first position to the camera coordinate system, obtain the second rotation matrix and the second translation matrix from the camera coordinate system at the first position to the robot arm end coordinate system, obtain the third rotation matrix and the third translation matrix from the robot arm end at the first position to the robot arm platform, and obtain the fourth rotation matrix and the fourth translation matrix from the platform coordinate system at the first position to the robot arm end coordinate system.

[0112] According to the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix, the third rotation matrix, the third translation matrix, the fourth rotation matrix, and the fourth translation matrix, the coordinates of the camera coordinate system corresponding to the geometric center point of the camera coordinate system are projected to the robot platform coordinate system through the conversion formula to obtain the coordinates of the geometric center point, where the conversion formula is:

[0113]

[0114] in, is the coordinate of coordinate system o2, is the coordinate of coordinate system o1, is the rotation matrix from coordinate system o1 to coordinate system O2, is the translation matrix from coordinate system O1 to coordinate system O2.

[0115] Specifically, in some embodiments, four pixel vertices are transformed into the image coordinate system according to the internal parameters of the depth camera, the distortion coefficients of the depth camera, and the depth values of the first depth image to obtain the coordinates of the four pixel vertices in the camera coordinate system, including:

[0116] According to the internal parameters of the depth camera, the distortion coefficients of the depth camera, and the depth values of the first depth image, the four pixel vertices are first transformed into the image coordinate system, and then the coordinates of the four pixel vertices in the camera coordinate system.

[0117] Refer to Figure 2 , in some embodiments, distributed coding is performed on the holes in the first detection result in the image. The image is divided into Row = [s / h center (rounded up) regions, and the width of each region is u0. Denote the bottommost row as the 0th row, and the row number increases from bottom to top. It is stipulated that the even rows are in the right direction and the odd rows are in the left direction in an S-shaped distribution route, as shown in Figure 3 In the digital increasing direction, the encoding regions of the center pixel values of the n detection results in the image are encoded. The encoding serial number of the detection result whose center point falls in any row m (m≥1) must be greater than the encoding serial number of the detection result whose center point falls in row k (0≤k<m). The serial number of the odd rows increases to the left, and the serial number of the even rows increases to the right. Obtain a new set of n detection results: Among them, x and y are the center pixel coordinates of the detection frame, w and h are the pixel-level width and height of the detection frame, and the h corresponding to the point with the smallest Euclidean distance between x, y and 0.5*u0, 0.5*v0 is denoted as h center , and u0 and v0 are the width and height of the first detection result in the image pixel coordinate system.

[0118] Distributed hole coding can control the overall movement route of the end of the robotic arm, which can well avoid the problem of unclear operation sequence caused by the movement of the camera with the end of the robotic arm during operation.

[0119] In some embodiments, the second detection result is subjected to a projection transformation to obtain a projection transformation result; and the projection transformation result is compared with the first detection result to obtain the coordinates of the hole to be detected, including:

[0120] Obtain the second depth image at the second position and the depth value of the second depth image, where the second depth image is an image captured by the depth camera at the second position.

[0121] Calculate according to the second detection result, the internal parameters of the depth camera, the distortion coefficients of the depth camera, and the depth value of the second depth image to obtain the vertex coordinates corresponding to the second detection result.

[0122] According to the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix, the third rotation matrix, the third translation matrix, the fourth rotation matrix, and the fourth translation matrix, the vertex coordinates corresponding to the second detection result are projected to the camera coordinate system through the transformation formula to obtain the spatial coordinates of the camera at the first position.

[0123] The projection transformation of the camera is performed according to the spatial coordinates of the camera at the first position to obtain a projection set of the hole to be detected in the first hole color image.

[0124] Find the first hole to be detected whose intersection-and-union ratio is greater than 0 between the projection set of the hole to be detected in the first hole color image and the first detection result, obtain the first hole to be detected, and acquire the coordinates of the first hole to be detected.

[0125] In some embodiments, the second normal vector of the hole to be detected is obtained by calculating according to the coordinates of the hole to be detected and the point cloud data of the second position, including:

[0126] Obtain a second rotation matrix and a second translation matrix from the camera coordinate system at the second position to the coordinate system of the end of the robotic arm, and obtain a third rotation matrix and a third translation matrix from the end of the robotic arm at the second position to the robotic arm platform;

[0127] Performing perspective transformation on the point cloud data at the second position and projecting it into a pixel coordinate system to obtain pixel coordinates of the point cloud data;

[0128] Extract pixel coordinates from point cloud data to meet and The first point cloud data is obtained by projecting the point cloud data. α is a constant, (u`, v`) is the pixel coordinate obtained by projecting the point cloud data, are the coordinates, width and height of the hole to be detected;

[0129] Filter out the first point cloud data and Point cloud, to obtain the second point cloud data;

[0130] The Ransac algorithm is used to extract the plane from the point cloud of the second point cloud data to obtain the plane normal vector;

[0131] According to the second rotation matrix and the second translation matrix from the camera coordinate system at the second position to the coordinate system of the robot end and the third rotation matrix and the third translation matrix from the robot end at the second position to the robot platform, the plane normal vector is projected to the robot platform coordinate system through the conversion formula to obtain the second normal vector of the hole to be detected.

[0132] In some embodiments, the calculation formula for calculating the four pixel vertices of each hole according to the hole detection result is: and in, is the horizontal coordinate of the nth hole, is the ordinate of the nth hole, is the pixel width of the nth hole, is the pixel height of the nth hole.

[0133] In some embodiments, the method of transporting the hose into the hole further comprises:

[0134] Get the center, width, and height of the first detection result in the image pixel coordinate system.

[0135] If there is a hole with an intersection-and-union ratio of 0 in the projection set of the hole to be detected in the first hole color image and the first detection result, it is determined whether the pixel coordinate projection in the first detection result of the hole with an intersection-and-union ratio of 0 is within the range of the width and height of the first detection result in the image pixel coordinate system: if the pixel coordinate projection in the first detection result of the hole with an intersection-and-union ratio of 0 is within the range of the width and height of the first detection result in the image pixel coordinate system, the hole with an intersection-and-union ratio of 0 is added to the first detection result; if the pixel coordinate projection in the first detection result of the hole with an intersection-and-union ratio of 0 exceeds the range of the width and height of the first detection result in the image pixel coordinate system, the hole with an intersection-and-union ratio of 0 is ignored.

[0136] In some embodiments, the first hole is operated on, and the intersection and union ratio of the real-time detection result and the serialized first detection result is calculated to determine the first frame. At the same time, the position of the second serial hole relative to the platform is updated in real time according to the intersection and union ratio. When the nth hole is operated, if there is a detection frame with an intersection and union ratio greater than 0 with the serialized n+1th hole in the field of view, the spatial position of the n+1th hole in the serial result is updated, so that the camera can approach the target hole for secondary recognition and operation during the next positioning operation.

[0137] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0138] 1. Obtain the first test result:

[0139] Acquire a first hole color image at a first position, an internal parameter of a depth camera, and a distortion coefficient of the depth camera, wherein the first hole color image is an image taken by the depth camera at a first position, and the first position is a position where the distance value to the hole to be detected is a preset first distance value.

[0140] An initial hole opening image sample set is obtained, and all initial hole opening images in the initial hole opening image sample set are annotated to obtain a hole opening image training sample set, wherein the annotated rectangular labels are inscribed with holes to be detected in the initial hole opening images.

[0141] Construct an initial Yolov5 model, train the initial Yolov5 model according to the cave entrance image training sample set, and obtain a trained Yolov5 model.

[0142] The first hole color image is input into the trained Yolov5 model for the first color image recognition to obtain the first detection result.

[0143] 2. Obtain the second test result:

[0144] A first depth image at a first position and a depth value of the first depth image are obtained, wherein the first depth image is an image captured by a depth camera at the first position.

[0145] Distributed encoding is performed on the holes of the first detection result to obtain a hole detection result.

[0146] The four pixel vertices of each hole are calculated based on the hole detection results.

[0147] The four pixel vertices are converted to the image coordinate system according to the intrinsic parameters of the depth camera, the distortion coefficient of the depth camera, and the depth value of the first depth image to obtain the coordinates of the camera coordinate system of the four pixel vertices, and the coordinates of the camera coordinate system of the four pixel vertices are averaged to obtain the coordinates of the geometric center point of the camera coordinate system, wherein the four pixel vertices are converted to the image coordinate system according to the intrinsic parameters of the depth camera, the distortion coefficient of the depth camera, and the depth value of the first depth image to obtain the coordinates of the camera coordinate system of the four pixel vertices. The calculation formula is:

[0148] x img =(x pixel -u x ) / f x

[0149] y img =(y pixel -v y ) / f y

[0150]

[0151] x img_undis =x img *(1+k1*r 2 +k2*r 4 +k3*r 6 )+2*p1*x img *y img +p2*(r 2 +2*x img *x img )

[0152] yimg_undis =y img *(1+k1*r 2 +k2*r 4 +k3*r 6 )+2*p2*x img *y img +p1*(r 2 +2*y img *y img )

[0153] z camera =Z img

[0154] x camera =z camera *x img_undis

[0155] y camera =z cameea *y img_undis

[0156] Among them, f x is the internal parameter of the depth camera, f y is the internal parameter of the depth camera, u x is the internal parameter of the depth camera, v y is the internal parameter of the depth camera, k1 is the distortion coefficient of the depth camera, k2 is the distortion coefficient of the depth camera, k3 is the distortion coefficient of the depth camera, p1 is the distortion coefficient of the depth camera, p2 is the distortion coefficient of the depth camera, x poxel is the horizontal coordinate of the point in the pixel coordinate system, y [ixel is the vertical coordinate of the point in the pixel coordinate system, x img is the horizontal coordinate of the point in the image coordinate system, y img is the vertical coordinate of the point in the image coordinate system, r is the distance, x img is the horizontal coordinate of the dedistorted point in the image coordinate system, y img is the ordinate of the point after dedistortion in the image coordinate system, Z img is the pixel depth value, x camera is the horizontal coordinate of the point in the camera coordinate system, y camera is the vertical coordinate of the point in the camera coordinate system, z camera It is the point depth value in the camera coordinate system.

[0157] Obtain the first rotation matrix and the first translation matrix from the robot arm end coordinate system at the first position to the camera coordinate system, obtain the second rotation matrix and the second translation matrix from the camera coordinate system at the first position to the robot arm end coordinate system, obtain the third rotation matrix and the third translation matrix from the robot arm end at the first position to the robot arm platform, and obtain the fourth rotation matrix and the fourth translation matrix from the platform coordinate system at the first position to the robot arm end coordinate system.

[0158] According to the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix, the third rotation matrix, the third translation matrix, the fourth rotation matrix, and the fourth translation matrix, the coordinates of the camera coordinate system corresponding to the geometric center point of the camera coordinate system are projected to the robot platform coordinate system through the conversion formula to obtain the coordinates of the geometric center point, where the conversion formula is:

[0159]

[0160] in, is the coordinate of coordinate system o2, is the coordinate of coordinate system o1, is the rotation matrix from coordinate system o1 to coordinate system O2, is the translation matrix from coordinate system O1 to coordinate system O2.

[0161] The first normal vector of the hole to be detected is calculated based on the coordinates of the geometric center point and the point cloud data of the first position. The mechanical arm is controlled to move toward the hole to be detected based on the coordinates of the geometric center point and the first normal vector of the hole to be detected, so that the mechanical arm is located at the second position, and a second hole color image and point cloud data of the second position are obtained, wherein the second hole color image is an image taken by the depth camera at the second position, and the point cloud data of the second position is the point cloud data taken by the depth camera at the second position.

[0162] The second hole color image is input into the trained Yolov5 model for a second color image recognition to obtain a second detection result.

[0163] 3. Get the coordinates of the hole to be detected:

[0164] A second depth image at a second position and a depth value of the second depth image are obtained, wherein the second depth image is an image captured by a depth camera at the second position.

[0165] A calculation is performed according to the second detection result, the internal parameter of the depth camera, the distortion coefficient of the depth camera, and the depth value of the second depth image to obtain the vertex coordinates corresponding to the second detection result.

[0166] According to the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix, the third rotation matrix, the third translation matrix, the fourth rotation matrix, and the fourth translation matrix, the vertex coordinates corresponding to the second detection result are projected to the camera coordinate system through the transformation formula to obtain the spatial coordinates of the camera at the first position.

[0167] The projection transformation of the camera is performed according to the spatial coordinates of the camera at the first position to obtain a projection set of the hole to be detected in the first hole color image.

[0168] Find the first hole to be detected whose intersection-and-union ratio is greater than 0 between the projection set of the hole to be detected in the first hole color image and the first detection result, obtain the first hole to be detected, and acquire the coordinates of the first hole to be detected.

[0169] 4. Get the second normal vector of the hole to be detected:

[0170] Obtain a second rotation matrix and a second translation matrix from the camera coordinate system at the second position to the coordinate system of the end of the robotic arm, and obtain a third rotation matrix and a third translation matrix from the end of the robotic arm at the second position to the robotic arm platform;

[0171] Performing perspective transformation on the point cloud data at the second position and projecting it into a pixel coordinate system to obtain pixel coordinates of the point cloud data;

[0172] Extract pixel coordinates from point cloud data to meet and The first point cloud data is obtained, where α is a constant, (u ` , v ` ) is the pixel coordinate obtained by projecting the point cloud data, are the coordinates, width and height of the hole to be detected;

[0173] Filter out the first point cloud data and Point cloud, to obtain the second point cloud data;

[0174] The Ransac algorithm is used to extract the plane from the point cloud of the second point cloud data to obtain the plane normal vector;

[0175] According to the second rotation matrix and the second translation matrix from the camera coordinate system at the second position to the coordinate system of the robot end and the third rotation matrix and the third translation matrix from the robot end at the second position to the robot platform, the plane normal vector is projected to the robot platform coordinate system through the conversion formula to obtain the second normal vector of the hole to be detected.

[0176] 5. Perform operations on the holes to be tested:

[0177] The robot arm is controlled according to the coordinates of the hole to be detected and the normal vector of the hole to be detected to transport the hose to the hole to be detected.

[0178] In the present invention, the distributed holes are uniformly coded and operated according to the specified route. Considering the visual detection direction during the operation, the position of the next target is updated each time the current hole is operated, which can improve the accuracy of visual detection; at the same time, during the movement, the overall recall rate of visual detection is improved through multiple detection results through intersection and union comparison. At the same time, point cloud is used for hole posture detection, which makes up for the problem of insufficient spatial information in pure visual detection.

[0179] In addition, refer to Figure 3 One embodiment of the present invention provides a system for transporting a hose into a hole, comprising a data acquisition module 1100, a first color image recognition module 1200, a first detection result calculation module 1300, a second position data acquisition module 1400, a second detection module 1500, a hole coordinate acquisition module 1600, a normal vector calculation module 1700 and a hole operation module 1800, wherein:

[0180] The data acquisition module 1100 is used to acquire a first hole color image at a first position, point cloud data at the first position, internal parameters of a depth camera, and distortion coefficients of the depth camera, wherein the first hole color image is an image taken by the depth camera at the first position, the first position is a position whose distance value to the hole to be detected is a preset first distance value, and the point cloud data at the first position is the point cloud data taken by the depth camera at the first position.

[0181] The first color image recognition module 1200 is used to input the first hole color image into the trained Yolov5 model to perform the first color image recognition to obtain the first detection result.

[0182] The first detection result calculation module 1300 is used to perform calculations based on the first detection result, the internal parameters of the depth camera and the distortion coefficient of the depth camera to obtain the vertex coordinates corresponding to the first detection result, and to average the vertex coordinates corresponding to the first detection result to obtain the coordinates of the geometric center point; and to perform calculations based on the coordinates of the geometric center point and the point cloud data of the first position to obtain the first normal vector of the hole to be detected.

[0183] The second position data acquisition module 1400 is used to control the robot arm to move toward the hole to be detected according to the coordinates of the geometric center point and the first normal vector of the hole to be detected, so that the robot arm is located at the second position, and obtains the second hole color image and the point cloud data of the second position, wherein the second hole color image is the image taken by the depth camera at the second position, and the point cloud data of the second position is the point cloud data taken by the depth camera at the second position.

[0184] The second detection module 1500 is used to input the second hole color image into the trained Yolov5 model to perform a second color image recognition to obtain a second detection result.

[0185] The hole coordinate acquisition module 1600 is used to perform a projection transformation on the second detection result to obtain a projection transformation result; and compare the projection transformation result with the first detection result to obtain the coordinates of the hole to be detected.

[0186] The normal vector calculation module 1700 is used to calculate according to the coordinates of the hole to be detected and the point cloud data of the second position to obtain a second normal vector of the hole to be detected.

[0187] The hole operation module 1800 is used to control the robot arm to transport the hose to the hole to be inspected according to the coordinates of the hole to be inspected and the second normal vector of the hole to be inspected.

[0188] The system obtains a first hole color image at a first position, point cloud data at the first position, internal parameters of a depth camera, and a distortion coefficient of the depth camera, wherein the first hole color image is an image taken by the depth camera at the first position, the first position is a position whose distance value to the hole to be detected is a preset first distance value, and the point cloud data at the first position is point cloud data taken by the depth camera at the first position, and the first hole color image is input into a trained Yolov5 model for a first color image recognition to obtain a first detection result, and the vertex coordinates corresponding to the first detection result are calculated based on the first detection result, the internal parameters of the depth camera, and the distortion coefficient of the depth camera, and the vertex coordinates corresponding to the first detection result are averaged to obtain the coordinates of the geometric center point; the first normal vector of the hole to be detected is obtained based on the coordinates of the geometric center point and the point cloud data of the first position, and the first normal vector of the hole to be detected is obtained based on the coordinates of the geometric center point. The first normal vector of the hole to be detected is used to control the robotic arm to move toward the hole to be detected, so that the robotic arm is located at the second position, and the second hole color image and the point cloud data of the second position are obtained, wherein the second hole color image is an image taken by the depth camera at the second position, and the point cloud data of the second position is the point cloud data taken by the depth camera at the second position. The second hole color image is input into the trained Yolov5 model for a second color image recognition to obtain a second detection result, and the second detection result is projected to obtain a projection transformation result; and the projection transformation result is compared with the first detection result to obtain the coordinates of the hole to be detected, and calculation is performed according to the coordinates of the hole to be detected and the point cloud data of the second position to obtain the second normal vector of the hole to be detected, and the robotic arm is controlled according to the coordinates of the hole to be detected and the second normal vector of the hole to be detected to transport the hose to the hole to be detected, which can improve work efficiency and reduce safety hazards.

[0189] It should be noted that the present system embodiment and the above-mentioned system embodiment are based on the same inventive concept, so the relevant contents of the above-mentioned method embodiment are also applicable to the present system embodiment and will not be repeated here.

[0190] The present application also provides an electronic device for transporting a hose into a hole, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements: the method for transporting a hose into a hole as described above.

[0191] The processor and the memory may be connected via a bus or other means.

[0192] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0193] The non-transitory software program and instructions required to implement the method of transporting a hose to a hole in the above embodiment are stored in the memory. When executed by the processor, the method of transporting a hose to a hole in the above embodiment is executed, for example, the above described method is executed. Figure 1 The method comprises steps S101 to S108.

[0194] The present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute: the method of transporting a hose into a hole as described above.

[0195] The computer-readable storage medium stores computer-executable instructions, which are executed by a processor or a controller, for example, by a processor in the above electronic device embodiment, so that the processor can execute the method of transporting the hose to the hole in the above embodiment, for example, executing the above described Figure 1 The method comprises steps S101 to S108.

[0196] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program units or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media generally contain computer readable instructions, data structures, program elements, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0197] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A method for transporting a hose into a hole, characterized in that: The method of transporting the hose into the hole comprises: Acquire a first hole color image at a first position, point cloud data at the first position, an internal parameter of a depth camera, and a distortion coefficient of the depth camera, wherein the first hole color image is an image taken by the depth camera at the first position, the first position is a position whose distance value to the hole to be detected is a preset first distance value, and the point cloud data at the first position is the point cloud data taken by the depth camera at the first position; Inputting the first hole color image into the trained Yolov5 model for a first color image recognition to obtain a first detection result; Calculate according to the first detection result, the internal parameter of the depth camera and the distortion coefficient of the depth camera to obtain the vertex coordinates corresponding to the first detection result, and average the vertex coordinates corresponding to the first detection result to obtain the coordinates of the geometric center point; calculate according to the coordinates of the geometric center point and the point cloud data of the first position to obtain the first normal vector of the hole to be detected; Controlling the robotic arm to move toward the hole to be detected according to the coordinates of the geometric center point and the first normal vector of the hole to be detected, so that the robotic arm is located at a second position, and acquiring a second hole color image and point cloud data of the second position, wherein the second hole color image is an image taken by the depth camera at the second position, and the point cloud data of the second position is the point cloud data taken by the depth camera at the second position; Inputting the second hole color image into the trained Yolov5 model for a second color image recognition to obtain a second detection result; Performing projection transformation on the second detection result to obtain a projection transformation result; and comparing the projection transformation result with the first detection result to obtain the coordinates of the hole to be detected; Calculating according to the coordinates of the hole to be detected and the point cloud data of the second position to obtain a second normal vector of the hole to be detected; The robot arm is controlled to transport the hose into the hole to be detected according to the coordinates of the hole to be detected and the second normal vector of the hole to be detected.

2. A method for transporting a hose into a hole according to claim 1, characterized in that: The Yolov5 model is trained as follows: Acquire an initial hole opening image sample set, and annotate all initial hole opening images in the initial hole opening image sample set to obtain a hole opening image training sample set, wherein the annotated rectangular labels inscribe the holes to be detected in the initial hole opening images; An initial Yolov5 model is constructed, and the initial Yolov5 model is trained according to the hole opening image training sample set to obtain the trained Yolov5 model.

3. A method for transporting a hose into a hole according to claim 2, characterized in that: The step of calculating according to the first detection result, the internal parameter of the depth camera and the distortion coefficient of the depth camera to obtain the vertex coordinates corresponding to the first detection result, and averaging the vertex coordinates corresponding to the first detection result to obtain the coordinates of the geometric center point includes: Acquire a first depth image at a first position and a depth value of the first depth image, wherein the first depth image is an image taken by the depth camera at the first position; Perform distributed encoding on the holes in the first detection result to obtain a hole detection result, where the distributed encoding is to divide the image into regions, the width of each region is u0, the bottommost row is denoted as the 0th row, and the row number increases from bottom to top. It is stipulated that the even rows are in the right direction and the odd rows are in the left direction in an S-shaped distribution route. Encode the directions of the encoding regions of the central pixel values of n detection results. The encoding serial number of the detection result with the center point falling in any m (m≥1)th row must be greater than the encoding serial number of the detection result with the center point falling in the k (0≤k<m)th row. The odd row serial number increases to the left, and the even row serial number increases to the right, obtaining n new groups of detection results: where x and y are the central pixel coordinates of the detection frame, w and h are the pixel-level width and height of the detection frame, and the h corresponding to the point with the smallest Euclidean distance between x, y and 0.5*u0, 0.5*v0 is denoted as h center , and u0 and v0 are the width and height of the first detection result in the image pixel coordinate system; Calculate four pixel vertices of each hole according to the hole detection result; The four pixel vertices are converted to the image coordinate system according to the intrinsic parameters of the depth camera, the distortion coefficient of the depth camera, and the depth value of the first depth image to obtain the coordinates of the camera coordinate system of the four pixel vertices, and the coordinates of the camera coordinate system of the four pixel vertices are averaged to obtain the coordinates of the geometric center point of the camera coordinate system, wherein the calculation formula for converting the four pixel vertices to the image coordinate system according to the intrinsic parameters of the depth camera, the distortion coefficient of the depth camera, and the depth value of the first depth image to obtain the coordinates of the camera coordinate system of the four pixel vertices is: x img =(x pixel -u x ) / f x y img =(y pixel -v y ) / f y x img_undis =x img *(1+k1*r 2 +k2*r 4 +k3*r 6 )+2**1*x img *y img +p2*(r 2 +2*x img *x img ) y img_undis =y img *(1+k1*r 2 +k2*r 4 +k3*r 6 )+2*p2*x img *y img +p1*(r 2 +2*y img *y img ) With camera =Z img x camera =z camera *x img_undis and camera =z camera *and img_undis Among them, f x is the internal parameter of the depth camera, f y is the internal parameter of the depth camera, u x is the internal parameter of the depth camera, v y is the internal parameter of the depth camera, k1 is the distortion coefficient of the depth camera, k2 is the distortion coefficient of the depth camera, k3 is the distortion coefficient of the depth camera, p1 is the distortion coefficient of the depth camera, p2 is the distortion coefficient of the depth camera, x Pixel is the horizontal coordinate of the point in the pixel coordinate system, y pixel is the vertical coordinate of the point in the pixel coordinate system, x img is the horizontal coordinate of the point in the image coordinate system, y img is the vertical coordinate of the point in the image coordinate system, r is the distance, x img is the horizontal coordinate of the dedistorted point in the image coordinate system, y img is the ordinate of the point after dedistortion in the image coordinate system, Z img is the pixel depth value, x camera is the horizontal coordinate of the point in the camera coordinate system, y camera is the vertical coordinate of the point in the camera coordinate system, z camera is the point depth value in the camera coordinate system; Obtain a first rotation matrix and a first translation matrix from the robot arm end coordinate system at the first position to the camera coordinate system, obtain a second rotation matrix and a second translation matrix from the camera coordinate system at the first position to the robot arm end coordinate system, obtain a third rotation matrix and a third translation matrix from the robot arm end at the first position to the robot arm platform, and obtain a fourth rotation matrix and a fourth translation matrix from the platform coordinate system at the first position to the robot arm end coordinate system; According to the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix, the third rotation matrix, the third translation matrix, the fourth rotation matrix, and the fourth translation matrix, the coordinates of the camera coordinate system corresponding to the geometric center point of the camera coordinate system are projected to the robot platform coordinate system through a conversion formula to obtain the coordinates of the geometric center point, wherein the conversion formula is: in, are the coordinates of coordinate system O2, are the coordinates of coordinate system O1, is the rotation matrix from coordinate system O1 to coordinate system O2, is the translation matrix from coordinate system O1 to coordinate system O2.

4. A method for transporting a hose into a hole according to claim 3, characterized in that: performing a projection transformation on the second detection result to obtain a projection transformation result; and comparing the projection transformation result with the first detection result to obtain the coordinates of the hole to be detected, including: Acquire a second depth image at a second position and a depth value of the second depth image, wherein the second depth image is an image taken by the depth camera at the second position; Calculating according to the second detection result, the internal parameter of the depth camera, the distortion coefficient of the depth camera and the depth value of the second depth image to obtain the vertex coordinates corresponding to the second detection result; According to the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix, the third rotation matrix, the third translation matrix, the fourth rotation matrix, and the fourth translation matrix, the vertex coordinates corresponding to the second detection result are projected to the camera coordinate system through a conversion formula to obtain the spatial coordinates of the camera at the first position; Performing a projection transformation of the camera according to the spatial coordinates of the camera at the first position to obtain a projection set of the hole to be detected in the first hole color image; Find the projection set of the hole to be detected in the first hole color image and the first hole to be detected whose intersection-and-union ratio in the first detection result is greater than 0, obtain the first hole to be detected, and acquire the coordinates of the first hole to be detected.

5. A method for transporting a hose into a hole according to claim 4, characterized in that: The step of calculating according to the coordinates of the hole to be detected and the point cloud data of the second position to obtain a second normal vector of the hole to be detected includes: Obtain a second rotation matrix and a second translation matrix from the camera coordinate system at the second position to the coordinate system of the end of the robotic arm, and obtain a third rotation matrix and a third translation matrix from the end of the robotic arm at the second position to the robotic arm platform; Performing perspective transformation on the point cloud data at the second position and projecting it into a pixel coordinate system to obtain pixel coordinates of the point cloud data; Extract the pixel coordinates in the point cloud data to meet and The first point cloud data is obtained by projecting the point cloud data. α is a constant, (u', v') is the pixel coordinate obtained by projecting the point cloud data, are the coordinates, width and height of the hole to be detected; Filter out the first point cloud data and Point cloud, and obtain the second point cloud data; Using the Ransac algorithm to extract the plane from the point cloud of the second point cloud data to obtain the plane normal vector; According to the second rotation matrix and the second translation matrix from the camera coordinate system of the second position to the coordinate system of the robot end and the third rotation matrix and the third translation matrix from the robot end of the second position to the robot platform, the plane normal vector is projected to the robot platform coordinate system through a conversion formula to obtain the second normal vector of the hole to be detected.

6. A method for transporting a hose into a hole according to claim 5, characterized in that: The calculation formula for calculating the four pixel vertices of each hole according to the hole detection result is: and in, is the horizontal coordinate of the nth hole, is the ordinate of the nth hole, is the pixel width of the nth hole, is the pixel height of the nth hole.

7. A method of transporting a hose into a hole according to claim 6, characterized in that: The method of transporting the hose into the hole also includes: Obtain the center, width, and height of the first detection result in the image pixel coordinate system; If there is a hole with an intersection-and-union ratio of 0 in the projection set of the hole to be detected in the first hole color image and in the first detection result, determine whether the pixel coordinate projection in the first detection result of the hole with an intersection-and-union ratio of 0 is within the range of the width and height of the first detection result in the image pixel coordinate system: if the pixel coordinate projection in the first detection result of the hole with an intersection-and-union ratio of 0 is within the range of the width and height of the first detection result in the image pixel coordinate system, the hole with an intersection-and-union ratio of 0 is added to the first detection result; if the pixel coordinate projection in the first detection result of the hole with an intersection-and-union ratio of 0 exceeds the range of the width and height of the first detection result in the image pixel coordinate system, the hole with an intersection-and-union ratio of 0 is ignored.

8. A system for transporting a hose into a hole, characterized in that: The method system for transporting a hose into a hole: a data acquisition module, configured to acquire a first hole color image at a first position, point cloud data at the first position, an internal parameter of a depth camera, and a distortion coefficient of the depth camera, wherein the first hole color image is an image taken by the depth camera at the first position, the first position is a position at which a distance value from the hole to be detected is a preset first distance value, and the point cloud data at the first position is point cloud data taken by the depth camera at the first position; A first color image recognition module, used for inputting the first hole color image into the trained Yolov5 model to perform a first color image recognition to obtain a first detection result; A first detection result calculation module, configured to calculate according to the first detection result, the internal parameter of the depth camera and the distortion coefficient of the depth camera to obtain the vertex coordinates corresponding to the first detection result, and average the vertex coordinates corresponding to the first detection result to obtain the geometric center point coordinates; and calculate according to the geometric center point coordinates and the point cloud data of the first position to obtain the first normal vector of the hole to be detected; A second position data acquisition module, used for controlling the robot arm to move toward the hole to be detected according to the coordinates of the geometric center point and the first normal vector of the hole to be detected, so that the robot arm is located at a second position, and acquiring a second hole color image and point cloud data of the second position, wherein the second hole color image is an image taken by the depth camera at the second position, and the point cloud data of the second position is the point cloud data taken by the depth camera at the second position; A second detection module, used for inputting the second hole color image into the trained Yolov5 model for a second color image recognition to obtain a second detection result; A hole coordinate acquisition module to be detected, used for performing a projection transformation on the second detection result to obtain a projection transformation result; and comparing the projection transformation result with the first detection result to obtain the coordinates of the hole to be detected; a normal vector calculation module, configured to calculate according to the coordinates of the hole to be detected and the point cloud data of the second position to obtain a second normal vector of the hole to be detected; The hole operation module is used to control the robot arm to transport the hose into the hole to be detected according to the coordinates of the hole to be detected and the second normal vector of the hole to be detected.

9. An apparatus for transporting a hose into a hole, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute a method for transporting a hose into a hole as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method for transporting a hose into a hole according to any one of claims 1 to 7.

Images