Environment-friendly operation and maintenance robot visual system and environment-friendly operation and maintenance robot

Abstract

The invention discloses an environment-friendly operation and maintenance robot vision system, and the system comprises an image collection module which comprises a 3D camera disposed on a mechanical arm of a robot, and is used for collecting a plane image and a depth image of a target object; the image processing module is used for preprocessing the plane image acquired by the image acquisition module and transmitting the plane image to the target detection module; the target detection module is provided with a rotating target detector obtained by training based on a YOLOv8OBB network model, and is used for detecting the image transmitted by the image processing module and outputting a target frame center point, a target frame width, a target frame height and a rotating angle; the data conversion module is used for obtaining a rotation translation matrix of the target relative to the 3D camera; and the result output module is used for converting the rotation translation matrix into a relative motion matrix of the mechanical arm and outputting the relative motion matrix to a mechanical arm control unit of the robot. Based on the scheme of the invention, an efficient and accurate water quality station robot vision system can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to an environmental protection operation and maintenance robot vision system and an environmental protection operation and maintenance robot. Background Art

[0002] Water quality is one of the important factors for human survival and health. As a key facility for water body monitoring, water quality stations are of great significance to ensuring the safety of drinking water and maintaining the sustainable use of water resources. Traditional water quality station monitoring methods mostly rely on manual sampling, which has problems such as low sampling frequency and large sampling errors, and cannot meet the needs of real-time monitoring. In recent years, the development of robotics technology has provided new solutions for water quality station monitoring. As a new type of water quality monitoring equipment, the environmental protection operation and maintenance robot of the water quality station can perform real-time monitoring, which can not only avoid the errors of manual sampling, but also improve the monitoring accuracy and efficiency. Among them, the visual system, as an important component of the robot, can collect internal image information of the water quality station, analyze and calculate water quality parameters, thereby realizing automatic monitoring.

[0003] There are many kinds of water quality monitoring parameters in water quality stations, and robots are usually required to pick up and place different kinds of test kits. However, there are errors in the navigation and positioning of robots, so the visual system is needed to correct the grasping posture to complete the pick-up and placement of the test kits. In the past, the common practice of environmental protection operation and maintenance robots was to teach and record parameters directly at the navigation points. This method depends entirely on the accuracy of navigation. Once the navigation error is large, the operation will fail and the success rate will be low. There is also a method of attaching visual markers to the test kit and using the markers to locate the position of the test kit. This method can improve the success rate of the operation to a certain extent, but the test kits in the water quality station are frequently replaced, which requires manual labeling of the markers on the test kits, and there are high precision requirements for the marker pasting posture, which invisibly increases the labor cost and cannot completely liberate manpower. Summary of the invention

[0004] Based on the above background, the present invention provides an environmental protection operation and maintenance robot vision system and an environmental protection operation and maintenance robot to improve the operating ability of the environmental protection operation and maintenance robot. The following technical solutions are specifically adopted:

[0005] The first aspect of the present invention provides an environmental protection operation and maintenance robot vision system, comprising:

[0006] An image acquisition module, which includes a 3D camera configured on the robot arm and an image storage unit, for acquiring a planar image and a depth image of a target object;

[0007] An image processing module is used to preprocess the planar image acquired by the image acquisition module and transmit the preprocessed image to the target detection module;

[0008] The target detection module is equipped with a rotating target detector trained based on the YOLOv8_OBB network model, which is used to detect the image transmitted by the image processing module and output the center point of the target box, the width and height of the target box, and the rotation angle;

[0009] A data conversion module is used to convert the rotation angle output by the target detection module into the angle between the horizontal line and the long side, and obtain the point cloud center point coordinates of the target in the camera coordinate system based on the target center point coordinates, and obtain the rotation and translation matrix of the target relative to the 3D camera by combining the point cloud center point coordinates and the rotation angle;

[0010] The result output module is used to convert the rotation and translation matrix into a relative motion matrix of the robot arm and output it to the robot arm control unit.

[0011] Furthermore, the preprocessing of the image acquired by the image acquisition module includes:

[0012] Balance the brightness of the image;

[0013] Scales the image to the specified resolution.

[0014] Furthermore, the conversion of the rotation angle output by the target detection module into the angle between the horizontal line and the long side adopts the following formula:

[0015]

[0016] Where W and H are the width and height of the target frame output by the rotating target detector, r is the rotation angle output by the rotating target detector, and r n is the angle after conversion.

[0017] Furthermore, the step of obtaining the point cloud center point coordinates of the target in the camera coordinate system based on the target center point coordinates includes:

[0018] Convert the depth image of the target object captured by the 3D camera into a point cloud;

[0019] Based on the coordinates of the target center point, a point corresponding to the target center point in the point cloud is searched as the center point of the point cloud.

[0020] Furthermore, the conversion of the rotation and translation matrix into the relative motion matrix of the robot arm adopts the following formula:

[0021]

[0022] in, The pose transformation matrix of the flange at the end of the robot arm relative to the fixture at the end of the robot arm is a known matrix. Represents the pose transformation matrix from the fixture at the end of the robot arm to the 3D camera, which is also a known matrix; Represents the rotation and translation matrix of the target relative to the 3D camera, That is the relative motion matrix of the robot arm.

[0023] Furthermore, the construction method of the rotating target detector is as follows:

[0024] The 3D camera carried by the robot traveling on the preset path collects RGB plane images, wherein the RGB plane images contain various target objects to be operated and contain the precise coordinate information of the robot when collecting the images;

[0025] Perform data augmentation and annotation on the collected image data, and divide the annotated images into training set, test set and validation set;

[0026] The YOLOv8_OBB network model is trained based on the training set, test set and validation set, and the trained network model is used as a rotation target detector.

[0027] Furthermore, the data augmentation of the collected image data includes one or more of the following methods:

[0028] Geometric transformations, including one or more of rotation, scaling, translation, or shearing;

[0029] Color transformation, including changing one or more of brightness, contrast or saturation;

[0030] Affine transformations, including distortion or stretching;

[0031] Noise addition, adding random noise to the image;

[0032] Mixing strategy generates new samples by mixing different images.

[0033] Furthermore, the image data is labeled using a rotated bounding box labeling method, and each label consists of 9 columns of data, the first column is the category label, and the second to ninth columns are the coordinates of the four corner points of the oriented bounding box (x1, y1, x2, y2, x3, y3, x4, y4), where the coordinate values ​​are normalized relative to the image size.

[0034] Furthermore, the training of the YOLOv8_OBB network model based on the training set, the test set and the validation set includes:

[0035] Performing corner point interpolation on the rotated bounding boxes in the training set, the test set, and the validation set, and calculating a minimum area rotated rectangle formed by the interpolated bounding box contour points, the rectangle including the rotation angle, the center coordinates, and the width and height information;

[0036] Determine a loss function, wherein the loss function includes at least one of a positioning loss function ProbIoU_loss, a classification loss function lcls_loss, and a frame and angle regression loss function kld_loss, wherein the positioning loss function is used to measure the similarity between the predicted bounding box and the true bounding box, and the classification loss function is used to measure the accuracy of the category prediction;

[0037] Based on the determined loss function, the YOLOv8_OBB network model is trained using the training set and the test set until the model achieves a preset effect on the validation set, thus completing the training.

[0038] The second aspect of the present invention also provides an environmental protection operation and maintenance robot, including a robot body, a robotic arm installed on the body, a 3D camera installed on the robotic arm, and at least one clamp optionally installed on the end flange of the robotic arm, the environmental protection robot is configured with the environmental protection operation and maintenance robot vision system as described in the first aspect above.

[0039] The beneficial effects of the present invention are as follows:

[0040] Compared with the traditional visual estimation based on two-dimensional images, which only uses plane information and is suitable for plane grasping, but cannot accurately locate targets with arbitrary postures in complex environments, the solution of the present invention can obtain one more dimension of information in combination with 3D camera data, thereby improving the accuracy of pose estimation. At the same time, the YOLOv8_OBB network model and multiple loss functions are used to obtain a rotating target detector to monitor the target object, and the relative motion matrix of the robot arm is finally obtained in combination with the depth image obtained by the 3D camera, which effectively improves the grasping accuracy of the robot arm.

[0041] Based on the scheme of the present invention, an efficient and accurate water quality station robot vision system can be obtained, which can not only realize the automatic placement of the reagent box, but also ensure the operation success rate when the navigation error is large, thereby improving the accuracy and efficiency of water quality station monitoring. At the same time, the present invention uses a deep learning algorithm to estimate the posture of the reagent box, which has good intelligence and adaptability, is suitable for various types of reagent boxes, and is also very convenient for the promotion and deployment of different water quality stations. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of the module composition of an embodiment of the environmental protection operation and maintenance robot vision system of the present invention. DETAILED DESCRIPTION

[0043] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0044] See attached Figure 1 , an embodiment of the present invention provides an environmental protection operation and maintenance robot vision system, comprising:

[0045] An image acquisition module, which includes a 3D camera configured on the robot arm and an image storage unit, for acquiring a planar image and a depth image of a target object;

[0046] An image processing module is used to preprocess the planar image acquired by the image acquisition module and transmit the preprocessed image to the target detection module;

[0047] The target detection module is equipped with a rotating target detector trained based on the YOLOv8_OBB network model, which is used to detect the image transmitted by the image processing module and output the center point of the target box, the width and height of the target box, and the rotation angle;

[0048] A data conversion module is used to convert the rotation angle output by the target detection module into the angle between the horizontal line and the long side, and obtain the point cloud center point coordinates of the target in the camera coordinate system based on the target center point coordinates, and obtain the rotation and translation matrix of the target relative to the 3D camera by combining the point cloud center point coordinates and the rotation angle;

[0049] The result output module is used to convert the rotation and translation matrix into a relative motion matrix of the robot arm and output it to the robot arm control unit.

[0050] In some embodiments, preprocessing the image acquired by the image acquisition module includes:

[0051] Balance the brightness of the image to reduce the impact of reflections, shadows, etc. on the detection network;

[0052] Scales the image to the specified resolution.

[0053] In some embodiments, the rotation angle output by the YOLOv8_OBB network model adopts the opencv rotation box definition standard, with an angle range of (0π / 2), which is a positive acute angle or right angle between the horizontal line and the x-axis. In the subsequent processing, it needs to be converted to the angle between the horizontal line and the long side, with an angle range of (0π). The conversion formula is as follows:

[0054]

[0055] Where W and H are the width and height of the target frame output by the rotating target detector, r is the rotation angle output by the rotating target detector, and r n is the angle after conversion.

[0056] In some embodiments, obtaining the point cloud center point coordinates of the target in the camera coordinate system based on the target center point coordinates includes:

[0057] Convert the depth image of the target object captured by the 3D camera into a point cloud;

[0058] Based on the coordinates of the target center point, a point corresponding to the target center point in the point cloud is searched as the center point of the point cloud.

[0059] In some embodiments, the rotation and translation matrix is ​​converted into the relative motion matrix of the robot arm using the following formula:

[0060]

[0061] in, The pose transformation matrix of the flange at the end of the robot arm relative to the fixture at the end of the robot arm is a known matrix. Represents the pose transformation matrix from the fixture at the end of the robot arm to the 3D camera, which is also a known matrix; Represents the rotation and translation matrix of the target relative to the 3D camera, That is the relative motion matrix of the robot arm.

[0062] In some embodiments, the rotating object detector is constructed as follows:

[0063] The 3D camera on the robot driving on the preset path collects RGB plane images, which contain various target objects that need to be operated (such as reagent kits, electrode cleaning boxes, and other targets that need to be operated), and contains the precise coordinate information of the robot when the image is collected. In order to ensure the quality of the data set, the robot will shoot under different lighting conditions, weather conditions, and time periods, so as to obtain rich and diverse image data covering a wide range of scenes.

[0064] Perform data augmentation and annotation on the collected image data, and divide the annotated images into training set, test set and validation set;

[0065] The YOLOv8_OBB network model is trained based on the training set, test set and validation set, and the trained network model is used as a rotation target detector. The YOLOv8_OBB network model is an object detection method based on the YOLOv8 framework, which is specifically optimized for the detection of rotated targets. Compared with the traditional axis-aligned bounding box (AABB), OBB can more accurately depict tilted or rotated targets.

[0066] In some embodiments, the data collected from the real scene is limited and corresponding data augmentation is required. The data augmentation of the collected image data includes one or more of the following methods:

[0067] Geometric transformations, including one or more of rotation, scaling, translation, or shearing;

[0068] Color transformation, including changing one or more of brightness, contrast or saturation;

[0069] Affine transformations, including distortion or stretching;

[0070] Noise addition, adding random noise to the image;

[0071] Mixing strategies, such as CutMix and MixUp, generate new samples by mixing different images.

[0072] In some embodiments, the augmented image data is then annotated. The image data is annotated using the Oriented Bounding Box (OBB) annotation method, that is, the coordinates of the four corner points of the target object are annotated in the image. This annotation method can accurately indicate the rotation angle and direction of the object, providing a basis for subsequent training of the rotation target detection model. Compared with the traditional YOLO series label format, YOLOv8_OBB has made some changes and uses a special label format to represent the rotated bounding box. During annotation, each label consists of 9 columns of data, the first column is the category label, and the second to ninth columns are the coordinates of the four corner points of the oriented bounding box (x1, y1, x2, y2, x3, y3, x4, y4), where the coordinate values ​​are normalized relative to the image size.

[0073] In some embodiments, training the YOLOv8_OBB network model based on the training set, the test set, and the validation set includes:

[0074] Corner point interpolation is performed on the rotated bounding boxes in the training set, test set, and validation set: more points are generated between the original four corner points through the interpolation method to form a smoother boundary to improve the accuracy of the bounding box. The cv2.minAreaRect function in OpenCV is used to calculate the minimum area rotated rectangle formed by the interpolated bounding box contour points, which contains the rotation angle, center coordinates, width and height information.

[0075] Determine a loss function, where the loss function includes at least one of a positioning loss function ProbIoU_loss, a classification loss function lcls_loss, and a border and angle regression loss function kld_loss, where the positioning loss function is used to measure the similarity between the predicted bounding box and the true bounding box, and the classification loss function is used to measure the accuracy of the category prediction.

[0076] The loss function of YOLOv8_OBB needs to take into account the characteristics of the rotated bounding box. Common loss functions include positioning loss and classification loss. Positioning loss: IoU (Intersection over Union) and its variants (such as GIoU, DIoU, etc.) are usually used to measure the similarity between the predicted bounding box and the true bounding box. Since it is a rotated bounding box, the error of the rotation angle also needs to be considered. Classification loss: The cross entropy loss function is used to measure the accuracy of category prediction. The losses that need to be calculated in YOLOv8_OBB include classification loss: lcls_loss, and border and angle regression loss: kld_loss. Usually, CIOU_loss is used for rectangular box bounding box regression, and for rotated box detection, the original CIOU_loss needs to be replaced with the rotated box calculation loss: probiou_loss or kld_loss.

[0077] ProbIoU (Probabilistic Intersection over Union) is a loss function used to evaluate the similarity between rotated bounding boxes (OBBs), which is used to optimize the detection performance of the model. ProbIoU is developed based on the concept of IoU (Intersection over Union). IoU is a common indicator for measuring the degree of overlap between two bounding boxes, which is defined as the area of ​​the intersection of the two bounding boxes divided by the area of ​​the union. For axis-aligned bounding boxes (AABBs), the calculation of IoU is relatively simple, but for rotated bounding boxes (OBBs), since their orientation and shape are more complex, a more advanced method is needed to accurately calculate the similarity between them. ProbIoU deals with the overlap problem between rotated bounding boxes by introducing probability distribution. Specifically, instead of simply calculating the IoU of two rotated bounding boxes, ProbIoU treats each rotated bounding box as a random variable and assumes that the position and orientation of each bounding box follows a certain probability distribution. The advantage of this is that ProbIoU not only considers the geometric overlap of the two rotated bounding boxes, but also considers the uncertainty of their position and orientation. It has the following advantages:

[0078] 1) Robustness: ProbIoU can better handle the problem of inaccurate overlap estimation due to position and orientation uncertainty.

[0079] 2) Flexibility: Through probability distribution, ProbIoU can better adapt to different types of rotation bounding boxes.

[0080] 3) Accuracy: ProbIoU can more accurately reflect the actual overlap between rotated bounding boxes.

[0081] In the embodiment of the present invention, during the training process of YOLOv8_OBB, ProbIoU is used as part of the loss function to guide the model to learn how to better predict the rotation bounding box. By minimizing the ProbIoU loss between the predicted box and the true box, the model can learn a more accurate rotation bounding box position and orientation.

[0082] Finally, based on the determined loss function, the YOLOv8_OBB network model is trained using the training set and the test set until the model achieves the preset effect on the validation set and the training is completed.

[0083] Another embodiment of the present invention also provides an environmental protection operation and maintenance robot, including a robot body, a robotic arm installed on the body, a 3D camera installed on the robotic arm, and at least one clamp optionally installed on the end flange of the robotic arm, the environmental protection robot is configured with the environmental protection operation and maintenance robot vision system as described in the first embodiment above.

[0084] It should be noted that some embodiments of the present invention are described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0085] The embodiments of the present invention are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the present invention.

Claims

1. An environmental protection operation and maintenance robot vision system, characterized in that: include: An image acquisition module, which includes a 3D camera configured on the robot arm and an image storage unit, for acquiring a planar image and a depth image of a target object; An image processing module is used to preprocess the planar image acquired by the image acquisition module and transmit the preprocessed image to the target detection module; The target detection module is equipped with a rotating target detector trained based on the YOLOv8_OBB network model, which is used to detect the image transmitted by the image processing module and output the center point of the target box, the width and height of the target box, and the rotation angle; A data conversion module is used to convert the rotation angle output by the target detection module into the angle between the horizontal line and the long side, and obtain the point cloud center point coordinates of the target in the camera coordinate system based on the target center point coordinates, and obtain the rotation and translation matrix of the target relative to the 3D camera by combining the point cloud center point coordinates and the rotation angle; The result output module is used to convert the rotation and translation matrix into a relative motion matrix of the robot arm and output it to the robot arm control unit.

2. The environmental protection operation and maintenance robot vision system according to claim 1, characterized in that: The preprocessing of the image collected by the image acquisition module includes: Balance the brightness of the image; Scales the image to the specified resolution.

3. The environmental protection operation and maintenance robot vision system according to claim 1, characterized in that: The conversion of the rotation angle output by the target detection module into the angle between the horizontal line and the long side adopts the following formula: Where W and H are the width and height of the target frame output by the rotating target detector, r is the rotation angle output by the rotating target detector, and r n is the angle after conversion.

4. The environmental protection operation and maintenance robot vision system according to claim 3, characterized in that: The method of obtaining the point cloud center point coordinates of the target in the camera coordinate system based on the target center point coordinates includes: Convert the depth image of the target object captured by the 3D camera into a point cloud; Based on the coordinates of the target center point, a point corresponding to the target center point in the point cloud is searched as the center point of the point cloud.

5. The environmental protection operation and maintenance robot vision system according to claim 4, characterized in that: The conversion of the rotation and translation matrix into the relative motion matrix of the robot arm adopts the following formula: in, The pose transformation matrix of the flange at the end of the robot arm relative to the fixture at the end of the robot arm is a known matrix. Represents the pose transformation matrix from the fixture at the end of the robot arm to the 3D camera, which is also a known matrix; Represents the rotation and translation matrix of the target relative to the 3D camera, That is the relative motion matrix of the robot arm.

6. The environmental protection operation and maintenance robot vision system according to any one of claims 1 to 5, characterized in that: The construction method of the rotating target detector is as follows: The 3D camera carried by the robot traveling on the preset path collects RGB plane images, wherein the RGB plane images contain various target objects to be operated and contain the precise coordinate information of the robot when collecting the images; Perform data augmentation and annotation on the collected image data, and divide the annotated images into training set, test set and validation set; The YOLOv8_OBB network model is trained based on the training set, test set and validation set, and the trained network model is used as a rotation target detector.

7. The environmental protection operation and maintenance robot vision system according to claim 6, characterized in that: The data augmentation of the collected image data includes one or more of the following methods: Geometric transformations, including one or more of rotation, scaling, translation, or shearing; Color transformation, including changing one or more of brightness, contrast or saturation; Affine transformations, including distortion or stretching; Noise addition, adding random noise to the image; Mixing strategy generates new samples by mixing different images.

8. The environmental protection operation and maintenance robot vision system according to claim 6, characterized in that: The image data is labeled using a rotated bounding box labeling method, and each label consists of 9 columns of data, the first column is the category label, and the second to ninth columns are the coordinates of the four corner points of the oriented bounding box (x1, y1, x2, y2, x3, y3, x4, y4), where the coordinate values ​​are normalized relative to the image size.

9. The environmental protection operation and maintenance robot vision system according to claim 6, characterized in that: The training of the YOLOv8_OBB network model based on the training set, the test set and the validation set includes: Performing corner point interpolation on the rotated bounding boxes in the training set, the test set, and the validation set, and calculating a minimum area rotated rectangle formed by the interpolated bounding box contour points, the rectangle including the rotation angle, the center coordinates, and the width and height information; Determine a loss function, wherein the loss function includes at least one of a positioning loss function ProbIoU_loss, a classification loss function lcls_loss, and a frame and angle regression loss function kld_loss, wherein the positioning loss function is used to measure the similarity between the predicted bounding box and the true bounding box, and the classification loss function is used to measure the accuracy of the category prediction; Based on the determined loss function, the YOLOv8_OBB network model is trained using the training set and the test set until the model achieves a preset effect on the validation set, thus completing the training.

10. An environmental protection operation and maintenance robot, comprising a robot body, a mechanical arm mounted on the body, a 3D camera mounted on the mechanical arm, and at least one clamp optionally mounted on the end flange of the mechanical arm, characterized in that: The environmental protection robot is equipped with the environmental protection operation and maintenance robot vision system as described in any one of claims 1-9.

Images