An environmental protection operation robot vision system and an environmental protection operation robot

Abstract

The application discloses an environment-friendly operation and maintenance robot vision system, which comprises an image acquisition module, an image processing module, a target detection module, a data conversion module and a result output module.The image acquisition module comprises a 3D camera arranged on a robot mechanical arm and is used for acquiring a planar image and a depth image of a target object.The image processing module is used for pre-processing the planar image acquired by the image acquisition module and conveying the planar image to the target detection module.The target detection module is configured with a rotating target detector trained based on a YOLOv8_OBB network model, is used for detecting the image conveyed by the image processing module, and outputs a target frame center point, a target frame width and height and a rotation angle.The data conversion module is used for obtaining a rotation and translation matrix of the target relative to the 3D camera.The result output module is used for converting the rotation and translation matrix into a relative motion matrix of the mechanical arm and outputting the relative motion matrix to a mechanical arm control unit of the robot.Based on the scheme of the application, an efficient and accurate water quality station robot vision system can be obtained.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, specifically to an environmental protection operation and maintenance robot vision system and an environmental protection operation and maintenance robot. Background Technology

[0002] Water quality is a crucial factor for human survival and health. Water quality monitoring stations, as key facilities for water body monitoring, are of great significance for ensuring drinking water safety and maintaining the sustainable use of water resources. Traditional water quality monitoring methods rely heavily on manual sampling, which suffers from low sampling frequency and large sampling errors, failing to meet the needs of real-time monitoring. In recent years, the development of robotics technology has provided new solutions for water quality monitoring. Water quality monitoring station environmental operation and maintenance robots, as a new type of water quality monitoring equipment, can perform real-time monitoring, not only avoiding errors from manual sampling but also improving monitoring accuracy and efficiency. Among these components, the vision system, as a vital part of the robot, can analyze and calculate water quality parameters by collecting image information from inside the water quality station, thereby achieving automated monitoring.

[0003] Water quality monitoring stations handle a wide variety of water quality parameters, often requiring robots to pick up and place different types of reagent kits. However, robot navigation and positioning can have errors, necessitating the use of a vision system to correct the grasping posture for successful kit placement. Previously, environmental maintenance robots were commonly taught and parameters were recorded directly at navigation points. This method is entirely dependent on navigation accuracy; large navigation errors lead to operation failures and a low success rate. Another method involves attaching visual markers to the reagent kits to determine their position. While this can improve the success rate to some extent, frequent reagent kit changes at water quality stations necessitate manual labeling, requiring precise label placement and increasing labor costs, thus failing to completely eliminate human intervention. Summary of the Invention

[0004] Based on the above background, this invention provides an environmental protection operation and maintenance robot vision system and an environmental protection operation and maintenance robot to improve the operational capabilities of environmental protection operation and maintenance robots. Specifically, the following technical solution is adopted:

[0005] The first aspect of this invention provides an environmental protection operation and maintenance robot vision system, comprising:

[0006] The image acquisition module includes a 3D camera configured on the robotic arm and an image storage unit for acquiring planar and depth images of the target object.

[0007] The image processing module is used to preprocess the planar images acquired by the image acquisition module and send the preprocessed images to the target detection module.

[0008] The target detection module is equipped with a rotating target detector trained based on the YOLOv8_OBB network model. It is used to detect the image input by the image processing module and output the center point of the target box, the width and height of the target box, and the rotation angle.

[0009] The data conversion module is used to convert the rotation angle output by the target detection module into the angle between the horizontal line and the long side, obtain the center point coordinates of the target in the camera coordinate system based on the center point coordinates of the target, and obtain the rotation and translation matrix of the target relative to the 3D camera by combining the center point coordinates of the point cloud and the rotation angle.

[0010] The result output module is used to convert the rotation and translation matrix into a relative motion matrix of the robotic arm and output it to the robotic arm control unit of the robot.

[0011] Furthermore, the preprocessing of the images acquired by the image acquisition module includes:

[0012] Perform brightness equalization on the image;

[0013] Scale the image to the specified resolution.

[0014] Furthermore, the conversion of the rotation angle output by the target detection module into the angle between the horizontal line and the long side is achieved using the following formula:

[0015]

[0016] Where W and H are the width and height of the target bounding box output by the rotating target detector, respectively, and r is the rotation angle output by the rotating target detector. n This is the angle after conversion.

[0017] Furthermore, the method of obtaining the point cloud center point coordinates of the target in the camera coordinate system based on the target center point coordinates includes:

[0018] Convert the depth image of the target object captured by the 3D camera into a point cloud;

[0019] Based on the coordinates of the target center point, find the corresponding point in the point cloud and use it as the center point of the point cloud.

[0020] Furthermore, the conversion of the rotation and translation matrix into the relative motion matrix of the robotic arm is achieved using the following formula:

[0021]

[0022] in, The pose transformation matrix represents the position of the end effector flange of the robotic arm relative to the fixture mounted on the end effector of the robotic arm. This matrix is ​​a known matrix. This represents the pose transformation matrix from the gripper attached to the end effector of the robotic arm to the 3D camera; this matrix is ​​also a known matrix. This represents the rotation and translation matrix of the target relative to the 3D camera. This is the relative motion matrix of the robotic arm.

[0023] Furthermore, the method for constructing the rotating target detector is as follows:

[0024] The RGB planar images are acquired by a 3D camera mounted on a robot that travels on a preset path. The RGB planar images contain various target objects that need to be processed and include the precise coordinate information of the robot when the images are acquired.

[0025] The collected image data is augmented and labeled, and the labeled images are divided into training set, test set and validation set;

[0026] The YOLOv8_OBB network model is trained based on the training set, test set, and validation set, and the trained network model is used as a rotating target detector.

[0027] Furthermore, the data augmentation of the acquired image data includes one or more of the following methods:

[0028] Geometric transformations include one or more of rotation, scaling, translation, or clipping.

[0029] Color transformation includes changing one or more of the following: brightness, contrast, or saturation;

[0030] Affine transformations include twisting or stretching;

[0031] Add noise by adding random noise to the image;

[0032] Hybrid strategies generate new samples by mixing different images.

[0033] Furthermore, the image data is labeled using a rotated bounding box labeling method. Each label consists of 9 columns of data. The first column is the category label, and the second to ninth columns are the coordinates (x1, y1, x2, y2, x3, y3, x4, y4) of the four corner points of the oriented bounding box. The coordinate values ​​are normalized relative to the image size.

[0034] Furthermore, training the YOLOv8_OBB network model based on the training set, test set, and validation set includes:

[0035] Corner point interpolation is performed on the rotated bounding boxes in the training set, test set, and validation set, and the minimum area rotation rectangle formed by the outline points of the interpolated bounding boxes is calculated. This rectangle contains the rotation angle, center coordinates, and width and height information.

[0036] Determine a loss function, which includes at least one of a localization loss function ProbIoU_loss, a classification loss function lcls_loss, and a bounding box and angle regression loss function kld_loss. The localization loss function is used to measure the similarity between the predicted bounding box and the ground truth bounding box, and the classification loss function is used to measure the accuracy of the category prediction.

[0037] Based on the determined loss function, the YOLOv8_OBB network model is trained using the training set and test set until the model achieves the preset effect on the validation set, thus completing the training.

[0038] A second aspect of the present invention also provides an environmentally friendly operation and maintenance robot, including a robot body, a robotic arm mounted on the body, a 3D camera mounted on the robotic arm, and at least one clamp optionally mounted on the end flange of the robotic arm. The environmentally friendly robot is equipped with the environmentally friendly operation and maintenance robot vision system as described in the first aspect above.

[0039] The beneficial effects of this invention are as follows:

[0040] Compared to traditional visual estimation based on 2D images, which only uses planar information and is suitable for planar grasping but cannot accurately locate targets with arbitrary postures in complex environments, the solution of this invention combines 3D camera data to acquire an additional dimension of information, thereby improving the accuracy of pose estimation. Simultaneously, a rotating target detector is obtained using a YOLOv8_OBB network model and various loss functions to monitor the target object. Combined with the depth image acquired by the 3D camera, the relative motion matrix of the robotic arm is finally obtained, effectively improving the grasping accuracy of the robotic arm.

[0041] Based on the solution of this invention, a highly efficient and accurate robotic vision system for water quality monitoring stations can be obtained. This system not only enables automatic reagent kit handling but also maintains a high success rate even with significant navigation errors, thus improving the accuracy and efficiency of water quality monitoring. Furthermore, this invention employs a deep learning algorithm to estimate the reagent kit's posture, exhibiting excellent intelligence and adaptability. It is applicable to various types of reagent kits and is highly convenient for promotion and deployment at different water quality monitoring stations. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the module composition of an embodiment of the environmental protection operation and maintenance robot vision system of the present invention. Detailed Implementation

[0043] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0044] See appendix Figure 1 This invention provides an environmental protection operation and maintenance robot vision system, comprising:

[0045] The image acquisition module includes a 3D camera configured on the robotic arm and an image storage unit for acquiring planar and depth images of the target object.

[0046] The image processing module is used to preprocess the planar images acquired by the image acquisition module and send the preprocessed images to the target detection module.

[0047] The target detection module is equipped with a rotating target detector trained based on the YOLOv8_OBB network model. It is used to detect the image input by the image processing module and output the center point of the target box, the width and height of the target box, and the rotation angle.

[0048] The data conversion module is used to convert the rotation angle output by the target detection module into the angle between the horizontal line and the long side, obtain the center point coordinates of the target in the camera coordinate system based on the center point coordinates of the target, and obtain the rotation and translation matrix of the target relative to the 3D camera by combining the center point coordinates of the point cloud and the rotation angle.

[0049] The result output module is used to convert the rotation and translation matrix into a relative motion matrix of the robotic arm and output it to the robotic arm control unit of the robot.

[0050] In some embodiments, preprocessing the images acquired by the image acquisition module includes:

[0051] Brightness equalization is performed on the image to reduce the impact of reflections, shadows, etc. on the detection network;

[0052] Scale the image to the specified resolution.

[0053] In some embodiments, the rotation angle output by the YOLOv8_OBB network model adopts the OpenCV rotation box definition standard, with an angle range of (0π / 2), which is a positive acute angle or right angle formed by the horizontal line and the x-axis. In subsequent processing, it needs to be converted to the angle between the horizontal line and the longer side, with an angle range of (0π). The conversion formula is as follows:

[0054]

[0055] Where W and H are the width and height of the target bounding box output by the rotating target detector, respectively, and r is the rotation angle output by the rotating target detector. n This is the angle after conversion.

[0056] In some embodiments, obtaining the center point coordinates of the target in the camera coordinate system based on the target center point coordinates includes:

[0057] Convert the depth image of the target object captured by the 3D camera into a point cloud;

[0058] Based on the coordinates of the target center point, find the corresponding point in the point cloud and use it as the center point of the point cloud.

[0059] In some embodiments, the rotation and translation matrix is ​​converted into the relative motion matrix of the robotic arm using the following formula:

[0060]

[0061] in, The pose transformation matrix represents the position of the end effector flange of the robotic arm relative to the fixture mounted on the end effector of the robotic arm. This matrix is ​​a known matrix. This represents the pose transformation matrix from the gripper attached to the end effector of the robotic arm to the 3D camera; this matrix is ​​also a known matrix. This represents the rotation and translation matrix of the target relative to the 3D camera. This is the relative motion matrix of the robotic arm.

[0062] In some embodiments, the rotating target detector is constructed as follows:

[0063] The robot, traveling along a pre-defined path, uses a 3D camera to capture RGB planar images. These images contain various target objects requiring processing (such as reagent kits, electrode cleaning boxes, and other objects), along with the precise coordinates of the robot at the time of image acquisition. To ensure dataset quality, the robot takes photos under different lighting conditions, weather conditions, and time periods, resulting in a rich variety of image data covering a wide range of scenarios.

[0064] The collected image data is augmented and labeled, and the labeled images are divided into training set, test set and validation set;

[0065] The YOLOv8_OBB network model is trained using the training, test, and validation sets, and then used as a rotating object detector. The YOLOv8_OBB network model is an object detection method based on the YOLOv8 framework, specifically optimized for detecting rotating objects. Compared to traditional axis-aligned bounding boxes (AABB), OBB can more accurately depict tilted or rotated objects.

[0066] In some embodiments, the data collected from real-world scenes is limited and requires corresponding data augmentation. Data augmentation of the collected image data includes one or more of the following methods:

[0067] Geometric transformations include one or more of rotation, scaling, translation, or clipping.

[0068] Color transformation includes changing one or more of the following: brightness, contrast, or saturation;

[0069] Affine transformations include twisting or stretching;

[0070] Noise addition: Add random noise to the image;

[0071] Hybrid strategies, such as CutMix and MixUp, generate new samples by mixing different images.

[0072] In some embodiments, the augmented image data is then labeled. Image data is labeled using Oriented Bounding Box (OBB) annotation, which involves labeling the coordinates of the four corner points of the target object in the image. This annotation method accurately represents the rotation angle and direction of the object, providing a foundation for subsequent training of the rotating object detection model. Compared to traditional YOLO label formats, YOLOv8_OBB makes some changes, using a special label format to represent the rotated bounding box. During annotation, each label consists of 9 columns of data: the first column is the category label, and the second to ninth columns are the coordinates (x1, y1, x2, y2, x3, y3, x4, y4) of the four corner points of the oriented bounding box, where the coordinate values ​​are normalized relative to the image size.

[0073] In some embodiments, training the YOLOv8_OBB network model based on the training set, test set, and validation set includes:

[0074] Corner interpolation is performed on the rotated bounding boxes in the training, test, and validation sets: more points are generated between the original four corner points using interpolation methods to form smoother boundaries and improve the accuracy of the bounding boxes. The `cv2.minAreaRect` function in OpenCV is then used to calculate the minimum area rotation rectangle formed by the interpolated bounding box contour points. This rectangle contains the rotation angle, center coordinates, and width and height information.

[0075] Determine a loss function, which includes at least one of a localization loss function ProbIoU_loss, a classification loss function lcls_loss, and a bounding box and angle regression loss function kld_loss. The localization loss function is used to measure the similarity between the predicted bounding box and the ground truth bounding box, and the classification loss function is used to measure the accuracy of the category prediction.

[0076] The loss function for YOLOv8_OBB needs to take into account the characteristics of rotated bounding boxes. Common loss functions include localization loss and classification loss. Localization loss: IoU (Intersection over Union) and its variants (such as GIoU, DIoU, etc.) are typically used to measure the similarity between the predicted bounding box and the ground truth bounding box. Since it's a rotated bounding box, the error of the rotation angle also needs to be considered. Classification loss: The cross-entropy loss function is used to measure the accuracy of the class prediction. The losses that need to be calculated in YOLOv8_OBB are classification loss: lcls_loss, and bounding box and angle regression loss: kld_loss. Typically, CIOU_loss is used for bounding box regression, but for rotated box detection, the original CIOU_loss needs to be replaced with the rotation box calculation loss: probiou_loss or kld_loss.

[0077] ProbIoU (Probabilistic Intersection over Union) is a loss function used to evaluate the similarity between rotated bounding boxes (OBBs) and to optimize the detection performance of a model. ProbIoU is developed based on the concept of IoU (Intersection over Union). IoU is a commonly used metric to measure the degree of overlap between two bounding boxes, defined as the area of ​​their intersection divided by the area of ​​their union. For axis-aligned bounding boxes (AABBs), calculating IoU is relatively simple, but for rotated bounding boxes (OBBs), due to their more complex orientations and shapes, a more advanced method is needed to accurately calculate their similarity. ProbIoU addresses the overlap problem between rotated bounding boxes by introducing a probability distribution. Specifically, instead of simply calculating the IoU between two rotated bounding boxes, ProbIoU treats each rotated bounding box as a random variable and assumes that the position and orientation of each bounding box follow a certain probability distribution. The advantage of this approach is that ProbIoU considers not only the geometric overlap of the two rotated bounding boxes but also the uncertainty of their position and orientation. It has the following advantages:

[0078] 1) Robustness: ProbIoU can better handle the problem of inaccurate overlap estimation caused by position and orientation uncertainties.

[0079] 2) Flexibility: By using a probability distribution, ProbIoU can better adapt to different types of rotated bounding boxes.

[0080] 3) Accuracy: ProbIoU can more accurately reflect the actual overlap between rotated bounding boxes.

[0081] In this embodiment of the invention, during the training of YOLOv8_OBB, ProbIoU is used as part of the loss function to guide the model in learning how to better predict rotated bounding boxes. By minimizing the ProbIoU loss between the predicted box and the ground truth box, the model can learn more accurate positions and orientations of rotated bounding boxes.

[0082] Finally, based on the determined loss function, the YOLOv8_OBB network model is trained using the training set and test set until the model achieves the preset effect on the validation set, thus completing the training.

[0083] Another embodiment of the present invention provides an environmental protection operation and maintenance robot, including a robot body, a robotic arm mounted on the body, a 3D camera mounted on the robotic arm, and at least one clamp optionally mounted on the end flange of the robotic arm. The environmental protection robot is equipped with the environmental protection operation and maintenance robot vision system as described in the first embodiment above.

[0084] It should be noted that the above description describes some embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0085] The embodiments of this invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this invention should be included within the protection scope of this invention.

Claims

1. An environmentally friendly maintenance robot vision system, characterized in that, The method comprises the following steps: An image acquisition module comprises a 3D camera arranged on a robot mechanical arm and an image storage unit for acquiring a planar image and a depth image of a target object; An image processing module is used for pre-processing the planar image acquired by the image acquisition module and delivering the pre-processed image to a target detection module; The target detection module is configured with a rotating target detector trained based on a YOLOv8_OBB network model, and is used for detecting the image delivered by the image processing module and outputting a target frame center point, a target frame width and height, and a rotation angle; A data conversion module is used for converting the rotation angle output by the target detection module into an angle between a horizontal line and a long side, obtaining a point cloud center point coordinate of the target in a camera coordinate system based on the target center point coordinate, and obtaining a rotation and translation matrix of the target relative to the 3D camera in combination with the point cloud center point coordinate and the rotation angle; wherein the rotation angle output by the target detection module is converted into an angle between a horizontal line and a long side by using the following formula: wherein W, H are the target frame width and height output by the rotating target detector, r is the rotation angle output by the rotating target detector, and r n is the converted included angle; and obtaining a point cloud center point coordinate of the target in a camera coordinate system based on the target center point coordinate includes: converting a depth image of the target object collected by a 3D camera into a point cloud, and finding a point corresponding to the target center point in the point cloud as a point cloud center point based on the target center point coordinate. A result output module is used for converting the rotation and translation matrix into a relative motion matrix of the mechanical arm and outputting the relative motion matrix to a mechanical arm control unit of the robot; the rotation and translation matrix is converted into the relative motion matrix of the mechanical arm by using the following formula: wherein, represents a pose transformation matrix of the mechanical arm end flange with respect to the end-of-arm tooling clamp, which matrix is a known matrix; represents a pose transformation matrix of the end-of-arm tooling clamp to the 3D camera, which matrix is also a known matrix; represents a rotation and translation matrix of the target with respect to the 3D camera, is the relative motion matrix of the mechanical arm.

2. The environmentally friendly maintenance robot vision system of claim 1, wherein, The pre-processing of the image acquired by the image acquisition module comprises: brightness equalization of the image; scaling the image size to a specified resolution.

3. The environmentally friendly maintenance robot vision system of any one of claims 1-2, wherein, The construction method of the rotating target detector is as follows: A 3D camera carried by a robot driving on a preset path acquires an RGB planar image, the RGB planar image contains various target objects that need to be worked on, and contains accurate coordinate information of the robot when the image is acquired; The acquired image data is subjected to data augmentation and labeling, and the labeled image is divided into a training set, a test set and a validation set; The YOLOv8_OBB network model is trained based on the training set, the test set and the validation set, and the trained network model is used as the rotating target detector.

4. The environmentally friendly maintenance robot vision system of claim 3, wherein, The data augmentation of the acquired image data comprises one or more of the following modes: geometric transformation, including one or more of rotation, scaling, translation or clipping; color transformation, including one or more of changing brightness, contrast or saturation; affine transformation, including distortion or stretching; noise addition, adding random noise to the image; hybrid strategy, generating new samples by mixing different images.

5. The environmentally friendly maintenance robot vision system of claim 3, wherein, The image data is labeled by using a rotating bounding box labeling method, each label is composed of 9 columns of data, the first column is a category label, and the second to ninth columns are coordinates (x1, y1, x2, y2, x3, y3, x4, y4) of four corner points of an oriented bounding box, wherein the coordinate values are normalized values relative to the image size.

6. The environmentally friendly maintenance robot vision system of claim 3, wherein, The training of the YOLOv8_OBB network model based on the training set, the test set and the validation set comprises: angle point interpolation is performed on the rotating bounding box in the training set, the test set and the validation set, and the minimum area rotating rectangle formed by the outline points of the interpolated bounding box is calculated, which contains the rotation angle, the center coordinate and the width and height information; determine a loss function, the loss function comprising at least one of a positioning loss function ProbIoU_loss for measuring the similarity between the predicted bounding box and the real bounding box, a classification loss function lcls_loss for measuring the accuracy of the class prediction, and a bounding box and angle regression loss function kld_loss; based on the determined loss function, training the YOLOv8_OBB network model using the training set and the test set until the model reaches a preset effect on the validation set, completing the training.

7. An environmentally friendly maintenance robot, comprising a robot body, a mechanical arm installed on the body, a 3D camera installed on the mechanical arm, and at least one optional clamp installed on a flange at the end of the mechanical arm, characterized in that, The environmental protection operation robot is provided with the environmental protection operation robot vision system according to any one of claims 1-6.

Images