Mirror positioning method and system based on rgbd camera combined with hed neural network

Abstract

The application provides a mirror positioning method and system based on an RGBD camera combined with an HED neural network, which comprises the following steps: collecting data of a mirror surface by using an RGBD camera, performing edge detection on original data by using an HED neural network model, generating contour features by using a boundary tracking algorithm, performing Hough line detection on the contour features, finding intersection points of all detected line segments, performing corner point screening on the intersection point set, and screening out corner points of the mirror; generating a mask graph around the mirror surface by using the corner points of the mirror, and mapping the mask graph to a depth graph; extracting wall surface point clouds behind the mirror surface by using the mask graph, and fitting a wall surface plane equation; translating the wall surface equation into a mirror surface equation by using the thickness of the mirror surface; and finally mapping the corner points of the mirror to the mirror surface equation, and converting the corner points into three-dimensional point coordinates in the mirror surface space. The application provides mirror positioning information for a mirror surface cleaning robot, and can make the robot cover more cleaning areas, thereby improving the cleaning degree.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cleaning robot algorithm, in particular to a mirror positioning method and system based on an RGBD camera combined with a HED neural network, and specifically to a mirror positioning system based on an edge detection and corner point screening algorithm of an RGBD camera combined with a HED neural network. BACKGROUND

[0002] With the improvement of modern civilization, the rapid development of economy and the improvement of people's living standards, the hotel intelligent cleaning person has gradually entered the public vision, and its development trend is also faster and faster.

[0003] At present, the cleaning of hotel rooms is mainly completed by manual method, and the cleaning process is slow and the cost is relatively high. And the function of the robot used in the hotel at present is mainly for delivery. Due to the imaging reflection problem of the mirror, the depth map will be seriously lost at the mirror surface, and in the color image, the traditional Canny edge extraction operator needs to be continuously adjusted, which is difficult to adapt in all scenes. This leads to the fact that the spatial information of the mirror cannot be returned to the robot, resulting in the robot being unable to clean the mirror. How to solve the problem of mirror positioning has always been a difficult problem for cleaning robots.

[0004] In the Chinese patent literature with publication number CN112731977A, a mirror cleaning robot distance intelligent compensation method and system are disclosed, which includes a cleaning device, a sensor module group and a robot. The robot includes a full-automatic mobile platform. The sensor module group is distributed on the cleaning device. The cleaning device is carried on the full-automatic mobile platform. The full-automatic mobile platform is moved close to the edge of the mirror, and the distance from each position of the cleaning device to the mirror is detected by the sensor module group, and the distance and rotation angle of the cleaning robot from the cleaned mirror are controlled in real time. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a mirror positioning method and system based on an RGBD camera combined with a HED neural network.

[0006] According to the mirror positioning method based on the RGBD camera combined with the HED neural network provided by the present application, the method comprises the following steps:

[0007] Step S1: data acquisition of the mirror surface by the RGBD camera, edge detection of the original data by the HED neural network model, generation of the contour feature by the boundary tracking algorithm, Hough line detection of the contour feature, intersection point set of all detected line segments, corner point screening of the intersection point set, and finally screening of the four corner points of the mirror.

[0008] Step S2: generate a mask map around the mirror using the 4 corner points of the mirror, and map it to the depth map, then extract the wall point cloud behind the mirror using the mask map, and fit the wall plane equation, finally translate the wall equation to the mirror equation using the mirror thickness, and finally map the 4 corner points of the mirror to the mirror equation to convert them to mirror space three-dimensional point coordinates.

[0009] Preferably, the HED neural network model is trained using a labeled mirror scene training set, and the trained model is used to perform edge extraction on the RGB image. In the extracted edge segments, the longest one is extracted as the mirror edge, and the outermost contour is selected.

[0010] Preferably, the intersection point screening includes the following steps:

[0011] Step S1.1: Calculate the convex set composed of all intersection points of the detected line segments;

[0012] Step S1.2: Calculate the centroid of the convex set points;

[0013] Step S1.3: According to the size of the angle between all peripheral convex set points and the centroid, the point set is sorted clockwise,

[0014] Step S1.4: Perform 3-point circulation on the sorted point set to calculate the angle between each point and the two points before and after it;

[0015] Step S1.5: According to the relationship between the mirror frame and the affine transformation of the imaging camera, define the angle greater than 60° as the corner point of the mirror, so as to exclude irrelevant points.

[0016] Preferably, the step S1.2 includes: point set 1 to point set N, coordinates are respectively:

[0017] (x1, y1), (x2, y2), (x3, y3)...(xn, yn)

[0018] Then:

[0019]

[0020] Where x result , y result are the final centroid coordinates.

[0021] Preferably, the step S2 includes the following sub-steps:

[0022] Step S2.1: Generate a mirror area using the 4 intersection points of the mirror, and generate a wall mask around the mirror using a morphological image processing method, the morphological algorithm including:

[0023] Dilation: slide the structure element s on the image f, set the gray value of the image pixel point at the anchor point position of the structure element to the maximum value of the corresponding image region pixel in the region with the structure element value of 1, the formula is as follows:

[0024] dst(x, y) = max (x1，y1)：element(x1，y1)≠0 src(x+x1, y+y1)

[0025] Wherein, dst is the generated result image, src is the original image, element is the structure element, x, y, x1, y1 are the coordinates of the pixel points in the image;

[0026] Erosion: slide the structure element s on the image f, set the gray value of the image pixel point at the anchor point position of the structure element to the minimum value of the corresponding image region pixel in the region with the structure element value of 1, the formula is as follows:

[0027] dst(x, y) = min (x1,y1):element(x1,y1)≠0 src(x+x1, y+y1)

[0028] Wherein, dst is the generated result image, src is the original image, element is the structure element, x, y, x1, y1 are the coordinates of the pixel points in the image;

[0029] Step S2.2: using the generated wall mask image to extract the wall information in the point cloud, and using the Ransac random sample consensus model algorithm to fit the wall point cloud, and generating the plane equation of the wall;

[0030] Step S2.3: using the thickness of the mirror to move the wall equation into the mirror equation, and finally mapping the four corner points detected in the image into the spatial coordinate system according to the camera's internal parameters, and transmitting the positioning information of the mirror to the robot for mirror cleaning work.

[0031] According to the mirror positioning system based on the RGBD camera combined with the HED neural network provided by the application, comprising:

[0032] Module M1: using the RGBD camera to collect data of the mirror, using the HED neural network model to detect the edges of the original data, using the boundary tracking algorithm to generate the contour features, then performing Hough line detection on the contour features, finding the intersection points of all detected line segments, performing corner point screening on the intersection point set, and finally screening out the four corner points of the mirror.

[0033] Module M2: generate the mask map around the mirror surface with the 4 corner points of the mirror, and map it to the depth map, then extract the wall point cloud behind the mirror with the mask map, and fit the wall plane equation, finally translate the wall equation to the mirror equation with the mirror thickness, and finally map the 4 corner points of the mirror to the mirror equation to convert them to 3D point coordinates in the mirror space.

[0034] Preferably, the HED neural network model is trained with a labeled mirror scene training set, and the trained model is used to perform edge extraction on the RGB image. In the extracted edge segments, the longest one is extracted as the mirror edge, and the outermost contour is selected.

[0035] Preferably, the intersection point screening includes the following modules:

[0036] Module M1.1: calculate the convex set composed of all detected intersection points of line segments;

[0037] Module M1.2: calculate the centroid of the convex set points;

[0038] Module M1.3: according to the angle size between all peripheral convex set points and the centroid, the point set is sorted clockwise,

[0039] Module M1.4: 3-point cycle is performed on the sorted point set to calculate the angle size between each point and the two previous and next points;

[0040] Module M1.5: according to the relationship between the mirror frame and the affine transformation of the imaging camera, define the angle greater than 60° as the corner point of the mirror, so as to exclude irrelevant points.

[0041] Preferably, the module M1.2 includes: point set 1 to point set N, with coordinates respectively:

[0042] (x1,y1),(x2,y2),(x3,y3)…(xn,yn)

[0043] Then:

[0044]

[0045] Where x result , y result are the final centroid coordinates.

[0046] Preferably, the module M2 includes the following modules:

[0047] Module M2.1: generate the mirror area with the 4 intersection points of the mirror, and generate the wall mask around the mirror with the morphological image processing method, the morphological algorithm including:

[0048] Dilation: slide the structure element s on the image f, set the gray value of the image pixel point at the anchor point position of the structure element to the maximum value of the image region pixel corresponding to the region with the structure element value of 1, the formula is as follows:

[0049] dst(x, y) = max (x1，y1):element(x1，y1)≠0 src(x+x1, y+y1)

[0050] Wherein, dst is the generated result image, src is the original image, element is the structure element, x, y, x1, y1 are the coordinates of the pixel points in the image;

[0051] Erosion: slide the structure element s on the image f, set the gray value of the image pixel point at the anchor point position of the structure element to the minimum value of the image region pixel corresponding to the region with the structure element value of 1, the formula is as follows:

[0052] dst(x, y) = min (x1，y1):element(x1，y1)≠0 src(x+x1, y+y1)

[0053] Wherein, dst is the generated result image, src is the original image, element is the structure element, x, y, x1, y1 are the coordinates of the pixel points in the image;

[0054] Module M2.2: use the generated wall mask image to extract the wall information in the point cloud, and use the Ransac random sample consensus model algorithm to fit the wall point cloud, and generate the plane equation of the wall;

[0055] Module M2.3: move the wall equation to the mirror equation by using the thickness of the mirror, and finally map the four corner points detected in the image to the spatial coordinate system according to the camera's internal parameters, and transmit the positioning information of the mirror to the robot for mirror cleaning work.

[0056] Compared with the prior art, the present application has the following beneficial effects:

[0057] The present application provides a complete mirror positioning scheme suitable for hotel application scenarios, which provides mirror positioning information for the mirror cleaning of the cleaning robot, so that the robot can cover more cleaning areas, thereby improving the cleaning degree. BRIEF DESCRIPTION OF DRAWINGS

[0058] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings:

[0059] Figure 1 Flow chart of mirror positioning method of the present application

[0060] Figure 2 Label sample schematic diagram (left: original image, right: labeled image) of the present application;

[0061] Figure 3 The point set ordering diagram for the present application;

[0062] Figure 4 The mirror positioning diagram for the present application.

[0063] Figure 5 The mirror positioning diagram for the present application. DETAILED DESCRIPTION

[0064] The present application will be described in detail below with specific embodiments. The following examples will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of changes and improvements can be made. These are within the scope of the present application.

[0065] Example 1

[0066] The present application discloses a mirror positioning method based on RGBD camera combined with HED neural network. When the cleaning robot is sweeping the mirror, the RGB camera is used to collect data of the mirror, then the trained HED neural network model is used to perform edge detection on the original data, the boundary tracking algorithm is used to generate contour features, then the contour features are subjected to Hough line detection, the intersection points of all detected line segments are calculated, the intersection point set is subjected to corner point screening algorithm, and finally the four corner points of the mirror in the color image are screened out.

[0067] The four corner points of the mirror are extracted from the color camera to generate a mask map around the mirror, the mask map generated by the color camera is mapped to the depth map, and the three-dimensional point cloud collected by the camera, i.e. the wall point cloud behind the mirror, is extracted by using the depth map mask map. The RANSAC algorithm is used to fit the wall plane equation of the wall point cloud, then the wall plane equation is transferred to the mirror plane equation according to the thickness of the mirror, and finally the four corner points extracted from the color image are mapped to the mirror plane equation to complete the mirror positioning function.

[0068] Specifically, the method comprises the following steps:

[0069] Step S1: using HED neural network to perform edge detection on the color image collected by the RGB camera, performing Hough line detection on the extracted edge map, extracting all straight lines and calculating the intersection points of all straight lines, then performing corner point screening on the extracted intersection points, and calculating the final four corner points of the mirror.

[0070] Step S2: generate a wall mask map around the mirror using the four corners calculated, and map it to the depth map, then extract the wall point cloud using the mask map, and fit the wall plane equation, finally translate the wall equation to the mirror equation using the mirror thickness, and finally map the four corners of the color image to the mirror equation to convert to spatial three-dimensional point coordinates and provide them to the robot for mirror cleaning. The entire scheme flow chart is shown in Figure 1

[0071] Step S1 includes the following steps:

[0072] Step S1.1: The HED network model is designed on the basis of the VGG16 network structure, which has great advantages compared to the traditional Canny operator. The effect of contour extraction is better than that of the traditional Canny operator. Moreover, the model trained by a specific data set has the function of semantic information edge detection, which can greatly exclude the interference of external edges and overcome the problems of "broken edge" and "too many interference edges" of the traditional Canny operator. For the mirror application scenario, a large number of training sets are labeled to train the HED network model, as shown in Figure 2 , and the trained model is applied to the actual application scenario to extract the edges of the RGB image.

[0073] Step S1.2: Although most of the interference edges have been excluded in the actual edge detection process, in order to improve the edge extraction accuracy, the longest midline segment of the edge is selected as the mirror edge. The outermost contour is selected to exclude the influence of the imaging object lines in the mirror on the extraction of the mirror corner points, and the accuracy of the overall mirror positioning is improved. The positioning corner point result is shown in Figure 4 , and the white circle is the corner point detection result.

[0074] Step S1.3: Hough line detection is performed on the selected lines, and the intersection points of all lines are calculated. The corner points are selected from all intersection points, and the selection steps are as follows:

[0075] 1) First, the convex set of all intersection points is calculated;

[0076] 2) Calculate the centroid of the convex set points;

[0077] Centroid calculation formula:

[0078] The point set 1 to point set N has coordinates

[0079] (x1, y1), (x2, y2), (x3, y3)...(xn, yn)

[0080] Then:

[0081] ​

[0082] Xresult, yresult are the final centroid coordinates

[0083] 3) According to the size of the angle between all peripheral convex set points and the centroid, the point set is sorted clockwise, as shown in the following figure, the point set is arranged in order from small to large P1, P2, P3, P4. Figure 3

[0084] 4) The sorted point set is subjected to 3-point circulation, and the angle size between each point and the two points before and after it is calculated.

[0085] According to the relationship between the mirror frame and the affine transformation of the imaging camera, the angle greater than 60° is defined as the corner point of the mirror, so as to exclude irrelevant points.

[0086] Step S2 includes the following steps:

[0087] Step S2.1: Using the 4 intersection points of the mirror, the mirror area is generated, and the wall mask around the mirror is generated by using the morphological image processing method. The morphological image algorithm mainly includes two kinds of dilation and erosion and some combination operations thereof

[0088] 1) Dilation: slide the structure element s on the image f, and set the gray value of the image pixel point at the anchor point position of the structure element to the maximum value of the image region pixel corresponding to the region with the structure element value of 1. The formula is as follows:

[0089] dst(x, y) = max (x1，y1):element(x1，y1)≠0 src(x+x1, y+y1)

[0090] Where dst is the generated result image, src is the original image, element is the structure element, and x, y, x1, y1 are the coordinates of the pixel points in the image.

[0091] 2) Erosion: slide the structure element s on the image f, and set the gray value of the image pixel point at the anchor point position of the structure element to the minimum value of the image region pixel corresponding to the region with the structure element value of 1. The formula is as follows:

[0092] dst(x, y) = min (x1，y1):element(x1，y1)≠0 src(x+x1, y+y1)

[0093] Where dst is the generated result image, src is the original image, element is the structure element, and x, y, x1, y1 are the coordinates of the pixel points in the image.

[0094] Step S2.2: Extract the wall information in the point cloud using the generated wall mask image, and use the Ransac random sample consensus model algorithm to fit the wall point cloud and generate the plane equation of the wall. ​

[0095] The Ransac algorithm is an iterative method for estimating the parameters of a mathematical model from a set of observations that may contain outliers, i.e. observations that have no influence on the estimated values. It can therefore also be described as an outlier detection method. In a sense, it is a non-deterministic algorithm that only produces reasonable results with a certain probability, which increases with the number of iterations.

[0096] Step S2.3: The wall surface equation is moved to the mirror surface equation by using the thickness of the mirror surface. Finally, the four corner points detected in the image are mapped to the spatial coordinate system according to the camera's internal parameters, and the positioning information of the mirror surface is transmitted to the robot for mirror cleaning work. The positioning result of the mirror surface is shown in FIG. 6, where the white circles are the positioning points. Figure 5

[0097] Embodiment 2

[0098] Embodiment 2 is a preferred example of Embodiment 1, which further illustrates the present application.

[0099] A person skilled in the art can understand the mirror positioning method based on the RGBD camera combined with the HED neural network provided by the present application as a specific embodiment of a mirror positioning system based on the RGBD camera combined with the HED neural network, i.e. the mirror positioning system based on the RGBD camera combined with the HED neural network can be realized by executing the step flow of the mirror positioning method based on the RGBD camera combined with the HED neural network.

[0100] Specifically, a mirror positioning system based on an RGBD camera combined with a HED neural network comprises:

[0101] Module M1: data acquisition of the mirror surface by using the RGBD camera, edge detection of the original data by using the HED neural network model, generation of the contour feature by using the boundary tracking algorithm, Hough line detection of the contour feature, intersection point calculation of all detected line segments, corner point screening of the intersection point set, and finally screening of the four corner points of the mirror.

[0102] Module M2: mask map generation around the mirror surface by using the four corner points of the mirror, mapping to the depth map, mask map extraction of the wall surface points behind the mirror, wall surface plane equation fitting, wall surface equation translation to the mirror surface equation by using the mirror thickness, and finally mapping of the four corner points of the mirror to the mirror surface equation and conversion to the mirror space three-dimensional point coordinates.

[0103] ​In a preferred embodiment, the HED neural network model is trained using a labeled mirror scene training set, and the trained model is used to perform edge extraction on an RGB image. Among the extracted edge segments, the longest one is extracted as the mirror edge, and the outermost contour is selected.

[0104] In a preferred embodiment, the intersection filtering includes the following modules:

[0105] Module M1.1: Calculate the convex set composed of all detected intersection points of line segments;

[0106] Module M1.2: Calculate the centroid of the convex set points;

[0107] Module M1.3: Sort the point set clockwise according to the angle between all peripheral convex set points and the centroid,

[0108] Module M1.4: Perform 3-point circulation on the sorted point set to calculate the angle between each point and the two preceding and following points;

[0109] Module M1.5: According to the relationship between the mirror frame and the affine transformation of the imaging camera, define the angle greater than 60° as the corner point of the mirror, thereby excluding irrelevant points.

[0110] In a preferred embodiment, the module M1.2 includes: point set 1 to point set N, with coordinates respectively:

[0111] (x1, y1), (x2, y2), (x3, y3)...(xn, yn)

[0112] Then:

[0113]

[0114] Where x result , y result are the final centroid coordinates.

[0115] In a preferred embodiment, the module M2 includes the following modules:

[0116] Module M2.1: Generate the mirror area using the 4 intersection points of the mirror, and generate the wall mask around the mirror using morphological image processing methods, which include:

[0117] Dilation: Slide the structure element s over the image f, and set the gray value of the image pixel at the structure element anchor position to the maximum value of the corresponding image region pixels in the region with structure element value 1. The formula is as follows:

[0118] dst(x, y) = max (x1，y1):element(x1，y1)≠0 src(x+x1, y+y1)

[0119] wherein dst is the generated result image, src is the original image, element is the structure element, x, y, x1, y1 are the coordinates of the pixel points in the image;

[0120] Erosion: slide the structure element s on the image f, set the gray value of the image pixel point at the anchor position of the structure element as the minimum value of the image pixel in the region corresponding to the region with the structure element value of 1, and the formula is as follows:

[0121] dst(x, y) = min (x1，y1):element(x1，y1)≠0 src(x+x1, y+y1)

[0122] wherein dst is the generated result image, src is the original image, element is the structure element, x, y, x1, y1 are the coordinates of the pixel points in the image;

[0123] Module M2.2: use the generated wall mask image to extract the wall information in the point cloud, and use the Ransac random sample consensus model algorithm to fit the wall point cloud and generate the plane equation of the wall;

[0124] Module M2.3: move the wall equation to the mirror equation by using the thickness of the mirror, and finally map the four corner points detected in the image to the spatial coordinate system according to the camera's internal parameters, and transfer the positioning information of the mirror to the robot for mirror cleaning work.

[0125] Those skilled in the art know that in addition to implementing the system provided by the present application and each device, module and unit thereof in a pure computer readable program code manner, the same functions can also be achieved by logically programming the method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, the system provided by the present application and each device, module and unit thereof can be considered as a hardware component, and the devices, modules and units included therein for achieving various functions can also be considered as structures within the hardware component. The devices, modules and units for achieving various functions can also be considered as both software modules for implementing methods and structures within hardware components.

[0126] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. The embodiments of the present application and the features in the embodiments can be arbitrarily combined with each other without conflict.

Claims

1. A mirror positioning method based on an RGBD camera combined with a HED neural network, characterized in that, The method comprises the following steps: Step S1: data acquisition of the mirror surface by using an RGBD camera, edge detection of the original data by using a HED neural network model, generation of contour features by using a boundary tracking algorithm, Hough line detection of the contour features, calculation of intersection points of all detected line segments, angle point screening of the intersection point set, and finally screening of four corner points of the mirror; Step S2: generation of a mask map around the mirror surface by using the four corner points of the mirror, mapping of the mask map to a depth map, extraction of wall point cloud behind the mirror surface by using the mask map, fitting of a wall plane equation, translation of the wall plane equation into a mirror surface equation by using the thickness of the mirror surface, mapping of the four corner points of the mirror to the mirror surface equation, and conversion into three-dimensional point coordinates in the mirror surface space; The step S2 comprises the following sub-steps: Step S2.1: generation of a mirror surface region by using the four intersection points of the mirror surface, and generation of a wall mask around the mirror surface by using a morphological image processing method, wherein the morphological algorithm comprises: Dilation: sliding of a structure element s on an image f, setting of a gray value of an image pixel at an anchor point position of the structure element to a maximum value of pixels in an image region corresponding to a region with a structure element value of 1, and the formula is as follows: dst(x, y) = max (x1,y1):element(x1,y1)≠0 src(x+x1, y+y1) Wherein, dst is a generated result image, src is an original image, element is a structure element, and x, y, x1, y1 are coordinates of a pixel in an image. Erosion: sliding of a structure element s on an image f, setting of a gray value of an image pixel at an anchor point position of the structure element to a minimum value of pixels in an image region corresponding to a region with a structure element value of 1, and the formula is as follows: dst(x, y) = min (x1,y1):element(x1,y1)≠0 src(x+x1, y+y1) Wherein, dst is a generated result image, src is an original image, element is a structure element, and x, y, x1, y1 are coordinates of a pixel in an image. Step S2.2: extraction of wall information in the point cloud by using the generated wall mask, fitting of the wall point cloud by using a Ransac random sample consensus model algorithm, and generation of a wall plane equation; Step S2.3: translation of the wall plane equation into a mirror surface equation by using the thickness of the mirror surface, mapping of the four corner points detected in the image to a spatial coordinate system according to an internal parameter of the camera, and transmission of positioning information of the mirror surface to a robot for mirror surface cleaning work.

2. The mirror positioning method based on RGBD camera combined with HED neural network according to claim 1, characterized in that: The HED neural network model is trained by using a mirror scene training set with labels, edge extraction of an RGB image is performed by using the trained model, the longest line segment in the extracted edge line segment is extracted as a mirror edge, and the outermost contour is screened out. 3.The mirror positioning method based on RGBD camera combined with HED neural network according to claim 1, characterized in that: The intersection point screening comprises the following steps: Step S1.1: calculation of a convex set composed of intersection points of all detected line segments; Step S1.2: calculation of a centroid of the convex set points; Step S1.3: clockwise sorting of the point set according to an included angle between all peripheral convex set points and the centroid, Step S1.4: 3-point circulation of the sorted point set, and calculation of an included angle between each point and the two points before and after it; Step S1.5: definition of a corner point of the mirror according to an affine transformation relationship between the mirror frame and an imaging camera, and exclusion of irrelevant points.

4. The mirror positioning method based on RGBD camera combined with HED neural network according to claim 3, characterized in that: The step S1.2 comprises: point set 1 to point set N, and the coordinates are respectively (x1,y1),(x2,y2),(x3,y3)…(xn,yn) Then: where x result ,y result are the final row heart coordinates. 5.A mirror positioning system based on RGBD camera combined with HED neural network, characterized in that, It comprises: Module M1: data acquisition of the mirror surface by using the RGBD camera, edge detection of the original data by using the HED neural network model, generation of the contour feature by using the boundary tracking algorithm, Hough line detection of the contour feature, calculation of the intersection points of all detected line segments, angle point screening of the intersection point set, and finally screening of the four corner points of the mirror; Module M2: generation of the mask map around the mirror surface by using the four corner points of the mirror, mapping to the depth map, extraction of the wall point cloud behind the mirror surface by using the mask map, fitting of the wall plane equation, translation of the wall surface equation to the mirror surface equation by using the mirror thickness, mapping of the four corner points of the mirror to the mirror surface equation, and conversion to the three-dimensional point coordinates in the mirror surface space; The module M2 comprises the following modules: Module M2.1: generation of the mirror surface region by using the four intersection points of the mirror, and generation of the wall mask around the mirror surface by using the morphological image processing method, the morphological algorithm comprising: Dilation: sliding the structural element s on the image f, setting the gray value of the image pixel at the anchor point position of the structural element to the maximum value of the image region pixel corresponding to the region with a structural element value of 1, and the formula is as follows: dst(x, y) = max (x1,y1):element(x1,y1)≠0 src(x+x1, y+y1) Wherein, dst is the generated result image, src is the original image, element is the structural element, x, y, x1, y1 are the coordinates of the pixel points in the image; Erosion: sliding the structural element s on the image f, setting the gray value of the image pixel at the anchor point position of the structural element to the minimum value of the image region pixel corresponding to the region with a structural element value of 1, and the formula is as follows: dst(x, y) = min (x1,y1):element(x1,y1)≠0 src(x+x1, y+y1) Wherein, dst is the generated result image, src is the original image, element is the structural element, x, y, x1, y1 are the coordinates of the pixel points in the image; Module M2.2: extraction of the wall information in the point cloud by using the generated wall mask, fitting of the wall point cloud by using the Ransac random sampling consensus model algorithm, and generation of the plane equation of the wall; Module M2.3: moving the wall surface equation to the mirror surface equation by using the thickness of the mirror, and finally mapping the four corner points detected in the image to the spatial coordinate system according to the camera's internal parameters, and transferring the positioning information of the mirror to the robot for mirror cleaning work. 6.The mirror positioning system based on RGBD camera combined with HED neural network according to claim 5, characterized in that: The HED neural network model is trained by using a labeled mirror scene training set, and the trained model is used for edge extraction of the RGB image. In the extracted edge line segments, the longest one is extracted as the mirror edge, and the outermost contour is screened out.

7. The mirror positioning system based on RGBD camera combined with HED neural network according to claim 5, characterized in that: The intersection point screening comprises the following modules: Module M1.1: calculating the convex set composed of all detected intersection points of line segments; Module M1.2: calculating the centroid of the convex set points; Module M1.3: clockwise sorting the point set according to the angle size between all peripheral convex set points and the centroid, Module M1.4: 3-point circulation of the sorted point set, and calculation of the angle size between each point and the two points before and after it; Module M1.5: defining the angle greater than 60° as the corner point of the mirror according to the relationship between the mirror frame and the affine transformation of the imaging camera, so as to exclude irrelevant points. 8.The mirror positioning system based on RGBD camera combined with HED neural network according to claim 7, characterized in that: The module M1.2 includes: point set 1 to point set N, coordinates are respectively: (x1, y1), (x2, y2), (x3, y3)…(xn, yn) then: where x result ,y result are the final row heart coordinates.

Images