Pose control method, device and equipment of drainage wire disconnecting robot and medium

Abstract

The invention discloses a pose control method, device, equipment and medium for a drainage wire disconnecting robot, and the method comprises the steps: collecting an RGB image and a depth image of a cable through a camera on the robot, inputting the RGB image into a cable detection model, and obtaining a cable detection frame, further determining the pose data of the cable relative to the mechanical arm of the robot based on the cable detection frame and the depth image, and controlling the mechanical arm of the robot to operate the cable through the pose data, so that the pose of the cable is determined through the RGB image and the depth image collected by the camera; the robot can adjust the pose of the mechanical arm according to the pose of the cable, the problem that the mechanical arm and the cable incline and collide to damage the cable due to the fact that the pose of the cable cannot be determined is solved, and the mechanical arm can accurately grab the cable with the correct pose.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a posture control method, device, equipment and medium for a drainage line disconnecting robot. Background Art

[0002] In the power grid, the substation management office team often needs to come into contact with 110kV high-voltage cables and various substation equipment during daily operations. When some substation equipment needs to be inspected and repaired, the drainage lines connected to the distribution equipment need to be disconnected under power.

[0003] In order to avoid the need for manual disconnection of drainage wires in live environments, it is necessary to develop a robot that can disconnect drainage wires. The robot's mechanical arm is mounted on the cable, and the operating end of the mechanical arm needs to accurately identify the position of the cable to avoid tilting and collision during the process of grabbing the cable. Therefore, how to identify the position of the cable will affect whether the mechanical arm can successfully mount the operating end on the cable. How to accurately determine the position of the cable has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a posture control method, device, equipment and medium of a drainage line disconnecting robot, so as to control the operation end of the robot to be accurately mounted on the cable to perform drainage line disconnecting operation.

[0005] In a first aspect, the present invention provides a posture control method for a drainage wire disconnecting robot, which is applied to a robot equipped with a camera, comprising:

[0006] Collect RGB images and depth images through the camera cable;

[0007] Inputting the RGB image into a cable detection model to obtain a cable detection frame;

[0008] Determining, based on the cable detection frame and the depth image, position data of the cable relative to the robot arm;

[0009] The robot arm is controlled by the posture data to operate the cable.

[0010] Optionally, before collecting the RGB image and the depth image of the cable through the camera, the following is further included:

[0011] Set the frame rate and resolution of the camera to collect RGB images and depth images.

[0012] Optionally, the cable detection model is trained by the following steps:

[0013] Acquire a sample image, where the sample image is marked with a first detection frame of the cable;

[0014] Initialize YOLOv5 as the cable detection model;

[0015] Randomly extracting a sample image and inputting it into the cable detection model to obtain a second detection frame;

[0016] Calculating a loss rate using the first detection frame and the second detection frame;

[0017] Determine whether the training stop condition is met;

[0018] If so, determining that the cable detection model has completed training;

[0019] If not, the model parameters of the cable detection model are adjusted according to the loss rate, and the process returns to the step of randomly extracting a sample image and inputting it into the cable detection model to obtain a second detection frame.

[0020] Optionally, determining the position data of the cable relative to the robot arm based on the cable detection frame and the depth image includes:

[0021] Determining a target area corresponding to the cable detection frame in the depth image;

[0022] Convert the pixel coordinates and depth value of each pixel in the target area into the three-dimensional coordinates of the cable in the camera coordinate system;

[0023] Based on a pre-calibrated camera-to-manipulator extrinsic matrix, converting the three-dimensional coordinates of the cable in the camera coordinate system into the three-dimensional coordinates of the cable in the manipulator coordinate system;

[0024] The position data of the robot arm is calculated based on the three-dimensional coordinates of the cable in the robot arm coordinate system.

[0025] Optionally, converting the pixel coordinates and the depth value of each pixel in the target area into the three-dimensional coordinates of the cable in the camera coordinate system includes:

[0026] The pixel coordinates and depth value of each pixel are converted into the three-dimensional coordinates of the cable in the camera coordinate system by the following formula:

[0027]

[0028] Z = z;

[0029] Among them, u and v are the pixel coordinates of the pixel in the depth image, and c x 、c y is the principal point coordinate of the camera internal parameter, f x 、f y is the focal length of the camera intrinsic parameter, and z is the depth value.

[0030] Optionally, calculating the position and posture data of the robotic arm based on the three-dimensional coordinates of the cable in the robotic arm coordinate system includes:

[0031] Take the starting boundary point P of the cable on the left and right boundaries of the cable detection frame s =(x s ,y s ,z s ) and the end point P e =(x e ,y e ,z e );

[0032] The spatial direction vector v of the cable is calculated by the following formula cable :

[0033]

[0034] The deflection angle α of the cable around the x-axis is calculated by the following formula: x :

[0035]

[0036] The deflection angle α of the cable around the y-axis is calculated by the following formula: y :

[0037]

[0038] The deflection angle α of the cable around the z-axis is calculated by the following formula: z :

[0039]

[0040] Optionally, controlling a mechanical arm of the robot to operate the cable through the posture data includes:

[0041] Solving the adjustment amount of each joint of the robot arm with the position and posture data of the cable as the target;

[0042] adjusting the posture of the end of the robotic arm based on the adjustment amount;

[0043] The end of the robot arm is controlled to grasp the cable in the adjusted posture.

[0044] In a second aspect, the present invention provides a posture control device for a drainage wire disconnecting robot, which is applied to a robot equipped with a camera, comprising:

[0045] An image acquisition module, used to acquire RGB images and depth images of the cable through a camera;

[0046] A cable detection module, used for inputting the RGB image into a cable detection model to obtain a cable detection frame;

[0047] A cable posture determination module, used to determine the posture data of the cable relative to the robot arm based on the cable detection frame and the depth image;

[0048] A robotic arm control module is used to control the robotic arm of the robot to operate the cable through the posture data.

[0049] In a third aspect, the present invention provides an electronic device, the electronic device comprising:

[0050] at least one processor; and

[0051] a memory communicatively connected to the at least one processor; wherein,

[0052] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the posture control method of the drainage line disconnection robot described in any one of the first aspects of the present invention.

[0053] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the posture control method of the drainage line disconnection robot described in any one of the first aspects of the present invention when executed.

[0054] The embodiment of the present invention collects RGB images and depth images of cables through a camera on a robot, inputs the RGB images into a cable detection model to obtain a cable detection frame, further determines the position data of the cable relative to the robot's mechanical arm based on the cable detection frame and the depth image, and controls the robot's mechanical arm to operate on the cable through the position data, thereby realizing the determination of the cable's position through the RGB images and depth images collected by the camera, so that the robot can adjust the position of the mechanical arm according to the position of the cable, avoiding the problem of tilting and collision between the mechanical arm and the cable and damaging the cable due to the inability to determine the cable's position, so that the mechanical arm can accurately grasp the cable with the correct position.

[0055] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 It is a flow chart of a posture control method of a drainage wire disconnecting robot provided in Embodiment 1 of the present invention;

[0058] Figure 2 It is a flow chart of a posture control method of a drainage wire disconnecting robot provided in the second embodiment of the present invention;

[0059] Figure 3 It is a structural schematic diagram of a posture control device of a drainage wire disconnecting robot provided in a third embodiment of the present invention;

[0060] Figure 4 It is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0062] Embodiment 1

[0063] Figure 1 This is a flow chart of a method for controlling the posture of a drainage wire disconnecting robot provided in the first embodiment of the present invention. This embodiment can be applied to the situation where the posture of a cable is determined to control the robot to disconnect the drainage wire. This method can be executed by a posture control device of the drainage wire disconnecting robot. The posture control device of the drainage wire disconnecting robot can be implemented in the form of hardware and / or software, and can be configured in an electronic device, such as in the main control computer of the robot. Figure 1 As shown, the posture control method of the drainage line disconnecting robot includes:

[0064] S101. Collect RGB images and depth images of the cable through a camera.

[0065] The robot of this embodiment can be a robot used to remove drainage lines in distribution and transmission lines. Exemplarily, it can be a robot used to remove 110kV drainage lines in substations. The robot is equipped with a camera that can capture RGB images and depth images. Exemplarily, it can be equipped with a Microsoft Azure Kinect DK depth camera. The Azure Kinect DK camera is a depth sensor that supports simultaneous acquisition of RGB images and depth images, and can capture images of objects in the environment with high precision. The Azure Kinect DK camera generates depth images through optical sensors and Time of Flight (ToF) technology, provides distance information of objects through depth images, and can also obtain high-resolution RGB images for object detection and recognition.

[0066] Before capturing images, this embodiment can first set the frame rate and resolution of the camera for capturing RGB images and depth images. During the process of the camera capturing images, the RGB image and depth image corresponding to each frame can be captured simultaneously. The depth image provides distance information of the object. For each frame of data, the depth information z corresponding to the pixel coordinates (u, v) is obtained, and a spatial relationship between the depth image and the RGB image is established, such as a pixel correspondence relationship. Specifically, the resolution of the depth image and the RGB image can be the same, and each pixel in the depth image corresponds one-to-one to each pixel in the RGB image.

[0067] S102: Input the RGB image into a cable detection model to obtain a cable detection frame.

[0068] In one embodiment, the cable detection model is used to detect the cable and output a cable detection frame, wherein the cable detection model can be trained by the following steps:

[0069] S1. Obtain a sample image, where the sample image is marked with a first detection frame of a cable.

[0070] Exemplarily, a cable image dataset in various cable scenes (different lighting, angles, occlusions, etc.) may be acquired, and a first detection frame of the cable is marked in each image, where the first detection frame is a rectangular detection frame.

[0071] S2. Initialize YOLOv5 as the cable detection model.

[0072] YOLOv5 (You Only Look Once v5) is an efficient target detection algorithm that can quickly identify target objects in a single frame image and output a detection frame. YOLO divides the entire image into multiple grids through a convolutional neural network and predicts the target object and its position in each grid. In this embodiment, YOLOv5 can be initialized as a cable detection model, such as initializing the existing YOLOv5 network parameters.

[0073] S3. Randomly extract a sample image and input it into the cable detection model to obtain a second detection frame.

[0074] Specifically, one or more sample images can be randomly extracted and input into the cable detection model to output the second detection frame (x min ,y min ,x max ,y max ).

[0075] S4. Calculate the loss rate using the first detection frame and the second detection frame.

[0076] Specifically, the loss rate may be calculated using the boundary coordinates of the first detection frame and the second detection frame, and the loss rate may be calculated using a mean square error loss function.

[0077] S5. Determine whether the training stop condition is met. If so, execute S6; if not, execute S7.

[0078] The training stop condition can be that the loss rate is less than a threshold, or the number of iterative training reaches a preset number.

[0079] S6. Determine that the cable detection model has completed training.

[0080] S7. Adjust the model parameters of the cable detection model according to the loss rate, and return to S3.

[0081] Among them, the model parameters can be gradient-decreased according to the loss rate and the preset learning rate.

[0082] After the cable detection model is trained and deployed on the robot's main control computer, the RGB image captured by the camera is input into the cable detection model to obtain the cable detection frame.

[0083] S103 . Determine the position data of the cable relative to the robot arm based on the cable detection frame and the depth image.

[0084] In one embodiment, the target area corresponding to the cable detection frame can be determined from the depth image, and the image coordinates of each pixel in the target area can be converted into coordinates in the camera coordinate system and combined with the depth value to obtain the three-dimensional coordinates of each cable in the camera coordinate system. Further, through the calibration data of the camera and the robotic arm, the three-dimensional coordinates of the cable in the camera coordinate system are converted into the three-dimensional coordinates in the robotic arm coordinate system. The position and posture of the cable are calculated through the three-dimensional coordinates of the cable in the robotic arm coordinate system, for example, the direction vector of the cable, the deflection angles around the x, y, and z axes of the robotic arm, etc.

[0085] S104, controlling the robot's mechanical arm to operate the cable through the posture data.

[0086] After determining the cable's posture data, the adjustment amount of each joint of the robot arm can be solved with the cable's posture data as the target. The robot arm's posture is adjusted by the adjustment amount, and the path to the cable is planned to control the robot arm to move in the adjusted posture and grab the cable.

[0087] The embodiment of the present invention collects RGB images and depth images of cables through a camera on a robot, inputs the RGB images into a cable detection model to obtain a cable detection frame, further determines the position data of the cable relative to the robot's mechanical arm based on the cable detection frame and the depth image, and controls the robot's mechanical arm to operate on the cable through the position data, thereby realizing the determination of the cable's position through the RGB images and depth images collected by the camera, so that the robot can adjust the position of the mechanical arm according to the position of the cable, avoiding the problem of tilting and collision between the mechanical arm and the cable and damaging the cable due to the inability to determine the cable's position, so that the mechanical arm can accurately grasp the cable with the correct position.

[0088] Embodiment 2

[0089] Figure 2 This is a flow chart of a posture control method of a drainage wire disconnecting robot provided in the second embodiment of the present invention. The embodiment of the present invention is optimized on the basis of the above-mentioned first embodiment, such as Figure 2 As shown, the posture control method of the drainage line disconnecting robot includes:

[0090] S201. Collect RGB images and depth images of the cable through a camera.

[0091] S202: Input the RGB image into a cable detection model to obtain a cable detection frame.

[0092] S201 - S202 of this embodiment are similar to S101 - S102 in the first embodiment, and specific details may be referred to S101 - S102 , which will not be described in detail here.

[0093] S203: Determine a target area corresponding to the cable detection frame in the depth image.

[0094] In this embodiment, the size and resolution of the RGB image and the depth image can be the same, and the depth image corresponds to the pixels on the RGB image one by one. The size and position of the output cable detection frame are the positions in the RGB image. For example, the output line detection frame is recorded as (x min ,y min ,x max ,y max ), that is, the image coordinates of the pixels in the lower left corner and upper right corner of the rectangular detection box. Since the depth image corresponds to the pixels on the RGB image one by one, it can be obtained by (x min ,ymin ,x max ,y max )Determine the target area in the depth image.

[0095] S204: Convert the pixel coordinates and depth value of each pixel in the target area into the three-dimensional coordinates of the cable in the camera coordinate system.

[0096] In one embodiment, the pixel coordinates and depth value of each pixel can be converted into the three-dimensional coordinates of the cable in the camera coordinate system by the following formula:

[0097]

[0098] Z = z;

[0099] Among them, u and v are the pixel coordinates of the pixel in the depth image, and c x 、c y is the principal point coordinate of the camera internal parameter, f x 、f y is the focal length of the camera intrinsic parameter, and z is the depth value.

[0100] By using the above formula, the pixel coordinates of each pixel in the target area of ​​the depth map can be converted into coordinates in the camera coordinate system, that is, the three-dimensional coordinates (X, Y, Z) of the cable in the camera coordinate system can be obtained.

[0101] S205, based on a pre-calibrated camera-to-manipulator extrinsic matrix, converting the three-dimensional coordinates of the cable in the camera coordinate system into the three-dimensional coordinates of the cable in the manipulator coordinate system;

[0102] After obtaining the three-dimensional coordinates (X, Y, Z) of the cable in the camera coordinate system, the extrinsic parameter matrix T from the camera to the manipulator, which is pre-calibrated, can be used. c,e , convert the three-dimensional coordinates (X, Y, Z) of the cable in the camera coordinate system into the three-dimensional coordinates (X e ,Y e ,Z e ), the specific conversion formula is as follows:

[0103]

[0104] S206. Calculate the position and posture data of the robot arm based on the three-dimensional coordinates of the cable in the robot arm coordinate system.

[0105] Specifically, the starting boundary point P of the cable on the left and right boundaries of the cable detection frame can be taken as s =(x s ,y s ,z s ) and the end point P e =(x e ,ye ,z e ), that is, the three-dimensional coordinates of the two ends of the cable in the acquired image in the robot arm coordinate system are obtained.

[0106] The spatial direction vector v of the cable is calculated by the following formula cable :

[0107]

[0108] The deflection angle α of the cable around the x-axis is calculated by the following formula: x :

[0109]

[0110] The deflection angle α of the cable around the y-axis is calculated by the following formula: y :

[0111]

[0112] The deflection angle α of the cable around the z-axis is calculated by the following formula: z :

[0113]

[0114] The spatial direction vector v through the cable cable , the deflection angle α around the x-axis x , the deflection angle α around the y-axis y , the deflection angle α around the z axis z The posture of the cable in the robot arm coordinate system can be determined, and the distance of the cable relative to the robot arm can be determined through the depth value Z, that is, the position and posture of the cable relative to the robot arm can be determined.

[0115] S207, solving the adjustment amount of each joint of the robot arm with the cable posture data as the target.

[0116] In one embodiment, the target posture of the end effector of the robot arm can be set to a deflection angle α around each axis. x , α y , α z , calculate the angle change of each joint of the robot arm. For example, the inverse kinematics method can be used to solve the angle adjustment of each joint according to the deflection angle of the target posture so that the end effector faces the direction of the cable. Specifically, assuming that the inverse kinematics solution function is IK, the adjustment amount is:

[0117]

[0118] θ1, θ2, ..., θ nis the adjustment amount of n joints. The inverse kinematics solution function (ikine) is a function in Matlab Robotics Toolbox used to solve the joint coordinates of a robot with any degree of freedom. It is solved by an optimization algorithm and can handle the case of an under-actuated robot. For details, please refer to the existing optimization solution algorithm, which is not limited in this embodiment.

[0119] S208. Adjust the posture of the end of the robot arm based on the adjustment amount.

[0120] After solving the adjustment amount of each joint, the adjustment amount is input into the control system of the robot to rotate the end effector of the robot arm to face the cable. After the posture adjustment, the orientation of the end effector of the robot arm will be aligned with the direction of the cable, thereby avoiding relative tilt and collision with the cable during grasping, and achieving more stable cable grasping.

[0121] S209: Control the end of the robotic arm to grab the cable in the adjusted posture.

[0122] After the posture adjustment, the relative position information and relative posture of the end effector of the robot arm and the cable have been determined. The robot arm can plan the path according to actual needs, and then grasp the cable through path control. The grasping point and force can be controlled by tactile sensor feedback to ensure stability and safety. Optionally, the robot arm can continue to collect images through the camera to identify obstacles and make real-time path adjustments while moving toward the cable.

[0123] In this embodiment, the RGB image captured by the camera is first input into the cable detection model to obtain a cable detection frame, and then the target area corresponding to the cable detection frame is determined in the depth image, and the pixel coordinates of the pixel points in the target area are converted into three-dimensional coordinates in the camera coordinate system to obtain the three-dimensional coordinates of the cable in the camera coordinate system, and the three-dimensional coordinates of the cable in the camera coordinate system are further converted into the three-dimensional coordinates of the cable in the robot coordinate system through the external parameter matrix pre-calibrated by the camera and the robot arm, and finally the direction vector of the cable in the robot coordinate system and the deflection angle around the x, y, and z axes are calculated as the posture data of the cable through the three-dimensional coordinates of the cable in the robot coordinate system, and the end effector of the robot arm is controlled through the posture data, so that the posture of the cable is determined through the RGB image and the depth image captured by the camera, so that the robot can adjust the posture of the robot arm according to the posture of the cable, avoiding the problem of tilting and collision between the robot arm and the cable and damaging the cable due to the inability to determine the cable posture, so that the robot arm can accurately grasp the cable with the correct posture.

[0124] Embodiment 3

[0125] Figure 3This is a schematic diagram of the structure of a posture control device for a drainage wire disconnecting robot provided in the third embodiment of the present invention. Figure 3 As shown, the posture control device of the drainage wire disconnecting robot is applied to a robot equipped with a camera, and includes:

[0126] An image acquisition module 301 is used to acquire RGB images and depth images of the cable through a camera;

[0127] A cable detection module 302, configured to input the RGB image into a cable detection model to obtain a cable detection frame;

[0128] A cable posture determination module 303, used to determine the posture data of the cable relative to the robot arm based on the cable detection frame and the depth image;

[0129] The robot arm control module 304 is used to control the robot arm to operate the cable through the posture data.

[0130] Optionally, also include:

[0131] The camera setting module is used to set the frame rate and resolution of the camera to collect RGB images and depth images.

[0132] Optionally, the cable detection model is trained by the following modules:

[0133] A sample image acquisition module, used to acquire a sample image, wherein the sample image is marked with a first detection frame of the cable;

[0134] Model initialization module, used to initialize YOLOv5 as a cable detection model;

[0135] A training module, used for randomly extracting sample images and inputting them into the cable detection model to obtain a second detection frame;

[0136] A loss rate calculation module, used to calculate the loss rate through the first detection frame and the second detection frame;

[0137] The training condition judgment module is used to judge whether the training stop condition is met. If so, the training completion determination module is executed; if not, the parameter adjustment module is executed;

[0138] A training completion determination module, used to determine that the cable detection model has completed training;

[0139] The parameter adjustment module is used to adjust the model parameters of the cable detection model according to the loss rate and return to the training module.

[0140] Optionally, the cable posture determination module 303 includes:

[0141] A target area determination unit, configured to determine a target area corresponding to the cable detection frame in the depth image;

[0142] An image coordinate conversion unit, used for converting the pixel coordinates and depth value of each pixel in the target area into the three-dimensional coordinates of the cable in a camera coordinate system;

[0143] A camera coordinate conversion unit, configured to convert the three-dimensional coordinates of the cable in the camera coordinate system into the three-dimensional coordinates of the cable in the robot arm coordinate system based on a pre-calibrated camera-to-robot extrinsic matrix;

[0144] A posture calculation unit is used to calculate the posture data of the robot arm based on the three-dimensional coordinates of the cable in the robot arm coordinate system.

[0145] Optionally, the image coordinate conversion unit is specifically used for:

[0146] The pixel coordinates and depth value of each pixel are converted into the three-dimensional coordinates of the cable in the camera coordinate system by the following formula:

[0147]

[0148] Z = z;

[0149] Among them, u and v are the pixel coordinates of the pixel in the depth image, and c x 、c y is the principal point coordinate of the camera internal parameter, f x 、f y is the focal length of the camera intrinsic parameter, and z is the depth value.

[0150] Optionally, the pose calculation unit includes:

[0151] The line boundary point extraction subunit is used to extract the starting boundary point P of the cable on the left and right boundaries of the cable detection frame. s =(x s ,y s ,z s ) and the end point P e =(x e ,y e ,z e );

[0152] The cable direction vector calculation subunit is used to calculate the spatial direction vector v of the cable by the following formula: cable :

[0153]

[0154] The x-axis deflection angle calculation subunit is used to calculate the deflection angle α of the cable around the x-axis using the following formula: x :

[0155]

[0156] The y-axis deflection angle calculation subunit is used to calculate the deflection angle α of the cable around the y-axis using the following formula y :

[0157]

[0158] The z-axis deflection angle calculation subunit is used to calculate the deflection angle α of the cable around the z-axis using the following formula: z :

[0159]

[0160] Optionally, the robot arm control module 304 includes:

[0161] An adjustment amount solving unit, used for solving the adjustment amount of each joint of the robot arm with the posture data of the cable as the target;

[0162] A posture adjustment unit, used for adjusting the posture of the end of the robotic arm based on the adjustment amount;

[0163] The cable grabbing control unit is used to control the end of the robot arm to grab the cable in an adjusted posture.

[0164] The posture control device of the drainage wire detaching robot provided in the embodiment of the present invention can execute the posture control method of the drainage wire detaching robot provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0165] Embodiment 4

[0166] Figure 4 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0167] like Figure 4As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0168] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0169] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 executes the various methods and processes described above, such as the posture control method of the drainage line disconnection robot.

[0170] In some embodiments, the posture control method of the drainage wire disconnecting robot can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the posture control method of the drainage wire disconnecting robot described above can be performed. Alternatively, in other embodiments, the processor 41 can be configured to execute the posture control method of the drainage wire disconnecting robot by any other appropriate means (for example, by means of firmware).

[0171] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0173] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0175] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0176] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0177] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0178] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A posture control method for a drainage line disconnecting robot, characterized in that: Applicable to robots equipped with cameras, including: Collect RGB images and depth images through the camera cable; Inputting the RGB image into a cable detection model to obtain a cable detection frame; Determining, based on the cable detection frame and the depth image, position data of the cable relative to the robot arm; The robot arm is controlled by the posture data to operate the cable.

2. The method according to claim 1, characterized in that Before collecting RGB images and depth images through the camera cable, it also includes: Set the frame rate and resolution of the camera to collect RGB images and depth images.

3. The method according to claim 1, characterized in that The cable detection model is trained by the following steps: Acquire a sample image, where the sample image is marked with a first detection frame of the cable; Initialize YOLOv5 as the cable detection model; Randomly extracting a sample image and inputting it into the cable detection model to obtain a second detection frame; Calculating a loss rate using the first detection frame and the second detection frame; Determine whether the training stop condition is met; If so, determining that the cable detection model has completed training; If not, the model parameters of the cable detection model are adjusted according to the loss rate, and the process returns to the step of randomly extracting a sample image and inputting it into the cable detection model to obtain a second detection frame.

4. The method according to any one of claims 1 to 3, characterized in that: Determining the position data of the cable relative to the robot arm based on the cable detection frame and the depth image includes: Determining a target area corresponding to the cable detection frame in the depth image; Convert the pixel coordinates and depth value of each pixel in the target area into the three-dimensional coordinates of the cable in the camera coordinate system; Based on a pre-calibrated camera-to-manipulator extrinsic matrix, converting the three-dimensional coordinates of the cable in the camera coordinate system into the three-dimensional coordinates of the cable in the manipulator coordinate system; The position data of the robot arm is calculated based on the three-dimensional coordinates of the cable in the robot arm coordinate system.

5. The method according to claim 4, characterized in that Converting the pixel coordinates and the depth value of each pixel in the target area into the three-dimensional coordinates of the cable in the camera coordinate system includes: The pixel coordinates and depth value of each pixel are converted into the three-dimensional coordinates of the cable in the camera coordinate system by the following formula: Z = z; Among them, u and v are the pixel coordinates of the pixel in the depth image, and c x 、c y is the principal point coordinate of the camera internal parameter, f x 、f y is the focal length of the camera intrinsic parameter, and z is the depth value.

6. The method according to claim 4, characterized in that Calculating the position data of the robot arm based on the three-dimensional coordinates of the cable in the robot arm coordinate system, including: Take the starting boundary point P of the cable on the left and right boundaries of the cable detection frame s =(x s ,y s ,z s ) and the end point P e =(x e ,y e ,z e ); The spatial direction vector v of the cable is calculated by the following formula cable : The deflection angle α of the cable around the x-axis is calculated by the following formula: x : The deflection angle α of the cable around the y-axis is calculated by the following formula: y : The deflection angle α of the cable around the z-axis is calculated by the following formula: z :

7. The method according to any one of claims 1 to 3, characterized in that: Controlling the robot arm to operate the cable through the posture data includes: Solving the adjustment amount of each joint of the robot arm with the position and posture data of the cable as the target; adjusting the posture of the end of the robotic arm based on the adjustment amount; The end of the robot arm is controlled to grasp the cable in the adjusted posture.

8. A posture control device for a drainage line disconnecting robot, characterized in that: Applicable to robots equipped with cameras, including: An image acquisition module, used to acquire RGB images and depth images of the cable through a camera; A cable detection module, used for inputting the RGB image into a cable detection model to obtain a cable detection frame; A cable posture determination module, used to determine the posture data of the cable relative to the robot arm based on the cable detection frame and the depth image; A robotic arm control module is used to control the robotic arm of the robot to operate the cable through the posture data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the posture control method of the drainage line disconnection robot according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the posture control method of the drainage line disconnecting robot according to any one of claims 1 to 7 when executed.

Images