Robot edgewise method and device, robot and electronic equipment

Abstract

The invention provides a robot edging method and device, a robot and electronic equipment, and belongs to the technical field of control. The method comprises the following steps: detecting a current frame image shot by a robot to obtain a working area contour in the current frame image; obtaining an initial boundary line from a plurality of contour line segments of the working area contour; performing linear fitting based on the endpoints of the initial boundary line and the contour pixel points of the initial boundary line to obtain an edgewise boundary line; and according to the edge boundary line, the relative position between the robot and the edge boundary line is adjusted. Therefore, the robot obtains the edge boundary line only by using the current frame image shot by the visual camera in real time, and accordingly adjusts the pose to realize edge walking, so that the cost is greatly reduced. Moreover, through a mode of combining image detection and linear fitting again, the edge boundary line is more accurate and is not influenced by shielding objects such as trees and buildings, and the precision and reliability of edge walking are greatly improved.

Description

Technical Field

[0001] The present application relates to the field of control technology, and in particular, to a method and device for a robot to follow the edge, a robot, and an electronic device. Background Art

[0002] During the process of performing tasks such as mowing, cleaning, and inspection, a dedicated robot needs to walk along the edge within the working area. Walking along the edge is not only one of the key factors to ensure the working efficiency of the robot and enhance the user experience, but also enables the robot to better adapt to the complex and changeable working environment. At present, the robot edge-following technology mainly includes two common methods: one is buried wire guidance; the other is map- and positioning-based guidance.

[0003] The method of buried wire guidance is to lay an underground guiding wire in advance. The robot detects the signal of the guiding wire and adjusts its own traveling direction and posture according to the change of the signal intensity. The method of map- and positioning-based guidance is to use a preset map or a map manually created by the user to obtain boundary information, and at the same time use means such as RTK (Real-Time Kinematic) or UWB (Ultra-Wideband) to obtain the position information of the robot, and accordingly guide the robot to adjust its posture to achieve the purpose of walking along the boundary.

[0004] However, the robot system adopting the buried wire guidance method has a high construction cost, the guiding wire is prone to aging and needs to be frequently replaced, and affected by weather conditions and changes in ground cover, it may cause unstable or even interrupted signal transmission. For the method of map- and positioning-based guidance, its effectiveness depends on the accuracy of the initial map establishment and positioning information. If the RTK or UWB signal is lost due to the occlusion of trees or other buildings, the robot may not be able to receive the guiding instruction normally, thus losing the path guidance. Summary of the Invention

[0005] In view of this, the purpose of the present application is to provide a method and device for a robot to follow the edge, a robot, and an electronic device, which can realize the robot's edge-following without presetting a map and positioning information.

[0006] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0007] In a first aspect, the present application provides a method for a robot to follow the edge, and the method includes:

[0008] Detect the current frame image captured by the robot to obtain a working area detection result; wherein, the detection result includes the contour of the working area in the current frame image, the contour of the working area includes multiple contour line segments, and each contour line segment includes multiple contour pixel points;

[0009] Obtain an initial boundary line from multiple said contour line segments;

[0010] Based on the endpoints of the initial boundary line and the contour pixel points of the initial boundary line, perform linear fitting to obtain an edge-aligned boundary line;

[0011] Adjust the relative position between the robot and the edge-aligned boundary line according to the edge-aligned boundary line.

[0012] Optionally, the step of performing linear fitting based on the endpoints of the initial boundary line and the contour pixel points of the initial boundary line to obtain an edge-aligned boundary line includes:

[0013] According to the orientation of the initial boundary line and the walking direction of the robot, obtain a target boundary line from each of the initial boundary lines;

[0014] For the endpoints of the target boundary line, obtain a neighborhood circle centered on the endpoint, and use the pixel points within the neighborhood circle as the neighborhood pixel points of the endpoint;

[0015] Perform linear fitting on the contour pixel points of the initial boundary line and the neighborhood pixel points to obtain an edge-aligned boundary line.

[0016] Optionally, the step of detecting the current frame image captured by the robot to obtain a work area detection result includes:

[0017] Perform semantic segmentation on the current frame image captured by the robot to obtain a segmentation mask image;

[0018] According to the category of each pixel in the segmentation mask image and the target pixel category, convert the segmentation mask image into a binary mask image; wherein, the binary mask image includes the work area;

[0019] Perform edge detection on the work area in the binary mask image to obtain a work area edge;

[0020] Based on the work area edge, perform contour detection on the image of the work area to obtain an initial contour;

[0021] Optimize the initial contour to obtain a work area contour.

[0022] Optionally, the step of optimizing the initial contour to obtain a work area contour includes:

[0023] Perform polygon fitting on the initial contour to obtain a work area contour.

[0024] Optionally, the step of obtaining an initial boundary line from multiple said contour line segments includes:

[0025] Obtain a demarcation line parallel to the width of the current frame image according to the height of the current frame image;

[0026] Use the contour line segments segmented by the demarcation line as the initial boundary lines;

[0027] For each of the initial boundary lines, obtain the orientation of the initial boundary line according to the intersection point of the initial boundary line and the demarcation line; wherein, the orientation includes a left boundary line and a right boundary line.

[0028] Optionally, before the step of detecting the current frame image captured by the robot to obtain the work area detection result, the method further includes:

[0029] Detect whether there is a charging pile in the current frame image;

[0030] If not, execute the step of detecting the current frame image captured by the robot to obtain the work area detection result;

[0031] If so, obtain the relative distance and positional relationship between the charging pile and the robot according to the charging pile area in the current frame image;

[0032] In the case where the relative distance and the positional relationship do not meet the charging pile return condition corresponding to the walking direction, execute the step of detecting the current frame image captured by the robot to obtain the work area detection result.

[0033] Optionally, after the step of obtaining the initial boundary lines from the multiple contour line segments, the method further includes:

[0034] Detect whether there is an initial boundary line whose orientation matches the walking direction of the robot;

[0035] If so, execute the step of performing linear fitting based on the endpoints of the initial boundary line and the contour pixel points of the initial boundary line to obtain the edge boundary line;

[0036] If not, control the robot to rotate to adjust its posture, after adjusting the posture, control the robot to capture, obtain a new current frame image, and return to execute the step of detecting the current frame image captured by the robot to obtain the work area detection result.

[0037] Optionally, the step of adjusting the relative position between the robot and the edge boundary line according to the edge boundary line includes:

[0038] Based on the internal and external parameter matrices of the robot vision camera, convert the pixel coordinates of the endpoints of the edge boundary line into camera coordinates;

[0039] Obtain the perpendicular distance between the robot and the edge boundary line according to the camera coordinates and trigonometric functions;

[0040] When the perpendicular distance is not equal to the edge distance, control the robot to move so that the perpendicular distance between the robot and the edge boundary line is equal to the edge distance.

[0041] In a second aspect, the present application provides a robot, including a vision camera and a controller, and the controller is communicatively connected to the vision camera;

[0042] The controller is configured to implement the robot edge-following method as described in the first aspect.

[0043] In a third aspect, the present application provides an electronic device, including a processor and a memory, where the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the robot edge-following method as described in the first aspect.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the robot edge-following method as described in the first aspect.

[0045] In a fifth aspect, the present application provides a robot edge-following device, including an image detection module, a boundary acquisition module, and an edge-following execution module;

[0046] The image detection module is configured to detect the current frame image captured by the robot to obtain a work area detection result; wherein, the detection result includes the work area contour in the current frame image, the work area contour includes multiple contour line segments, and each contour line segment includes multiple contour pixel points;

[0047] The boundary acquisition module is configured to obtain an initial boundary line from multiple contour line segments;

[0048] The boundary acquisition module is further configured to perform linear fitting based on the endpoints of the initial boundary line and the contour pixel points of the initial boundary line to obtain an edge boundary line;

[0049] The edge-following execution module is configured to adjust the relative position between the robot and the edge boundary line according to the edge boundary line.

[0050] The edge-following method, device, robot, and electronic device provided by the embodiments of the present application. The method includes: detecting a current frame image captured by the robot to obtain a work area detection result, where the detection result includes the work area contour in the current frame image, the work area contour includes multiple contour line segments, and each contour line segment includes multiple contour pixel points; obtaining an initial boundary line from the multiple contour line segments; performing linear fitting based on the endpoints of the initial boundary line and the contour pixel points of the initial boundary line to obtain an edge-following boundary line; and adjusting the relative position between the robot and the edge-following boundary line according to the edge-following boundary line.

[0051] In this way, the robot can obtain the edge-following boundary line based on the current frame image captured in real time by the vision camera and adjust its pose to achieve edge-following walking only using the vision camera, without the need for a pre-set map, pre-buried wires, or positioning devices such as RTK, UWB technology, and GPS, greatly reducing the cost. Moreover, through the combination of image detection and re-linear fitting, the edge-following boundary line is more accurate and not affected by obstacles such as trees and buildings, greatly improving the accuracy and reliability of edge-following walking.

[0052] To make the above objects, features, and advantages of the present application more obvious and understandable, the following specific preferred embodiments are given and described in detail in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0054] Figure 1 Shows the system architecture diagram of the robot edge-following system provided by the embodiments of the present application.

[0055] Figure 2 Shows the module architecture diagram of the electronic device provided by the embodiments of the present application.

[0056] Figure 3 Shows one of the flow diagrams of the robot edge-following method provided by the embodiments of the present application.

[0057] Figure 4 Shows Figure 3 The flow diagram of some sub-steps of step 12 in

[0058] Figure 5 Shows Figure 3 The flow diagram of some sub-steps of step 14 in

[0059] Figure 6 shows Figure 3 a schematic flow diagram of some sub - steps of step 16 in

[0060] Figure 7 a detection result diagram of the current frame image in an example

[0061] Figure 8 shows Figure 3 a schematic flow diagram of some sub - steps of step 18 in

[0062] Figure 9 a schematic diagram of the relative position between the robot and the boundary line in an example

[0063] Figure 10 a schematic diagram of the relative position between the robot and the boundary line in another example

[0064] Figure 11 a second schematic flow diagram of the robot edge - following method provided by the embodiment of the present application

[0065] Figure 12 a third schematic flow diagram of the robot edge - following method provided by the embodiment of the present application

[0066] Figure 13 a schematic diagram of the module architecture of the robot edge - following device provided by the embodiment of the present application

[0067] Icons: 10 - robot edge - following system; 110 - vision camera; 120 - controller; 130 - working mechanism; 140 - power mechanism; 150 - walking mechanism; 20 - electronic device; 210 - memory; 220 - processor; 230 - communication module; 30 - robot edge - following device; 310 - image detection module; 320 - boundary acquisition module; 330 - edge - following execution module. Detailed implementation manners

[0068] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Usually, the components of the embodiments of the present application described and illustrated herein can be arranged and designed in various different configurations.

[0069] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but merely represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0070] It should be noted that relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the said element.

[0071] The edge-following method for a robot provided by an embodiment of this application can be applied to Figure 1 the robot edge-following system 10 shown in the figure. The robot edge-following system 10 includes a vision camera 110, a controller 120, a working mechanism 130, a power mechanism 140, and a traveling mechanism 150 of the robot. The controller 120 is communicatively connected to the vision camera 110 and the power structure respectively by means of wire or wireless, etc.

[0072] The power mechanism 140 is configured to drive the traveling mechanism 150 and / or the working mechanism 130 according to instructions (such as movement instructions, operation instructions, etc.) of the controller 120 to realize the movement (such as movement, rotation, etc.) and operation (such as mowing, sweeping, inspection, etc.) of the robot.

[0073] The vision camera 110 is configured to collect images of the environment where the robot is located and transmit the collected video stream or images to the controller 120 in real time.

[0074] The controller 120 is configured to implement the edge-following method for a robot provided by an embodiment of this application, including: detecting a current frame image captured by the robot to obtain a work area detection result, where the detection result includes a work area contour in the current frame image, the work area contour includes a plurality of contour line segments, and each contour line segment includes a plurality of contour pixel points; obtaining an initial boundary line from the plurality of contour line segments; performing linear fitting based on the endpoints of the initial boundary line and the contour pixel points of the initial boundary line to obtain an edge-following boundary line; and adjusting the relative position between the robot and the edge-following boundary line according to the edge-following boundary line.

[0075] Among them, the vision camera 110 can be a monocular camera of the robot, and the controller 120 can be a computer device, a single-chip microcomputer, a programmable logic controller 120, a field-programmable gate array or any programmable processing device.

[0076] Please refer to Figure 2, is a block diagram of an electronic device 20, which can be Figure 1 The controller 120 in the robot edge following system 10 shown. The electronic device 20 includes a memory 210, a processor 220, and a communication module 230. The elements of the memory 210, the processor 220, and the communication module 230 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines.

[0077] Among them, the memory 210 is used to store programs or data. The memory 210 can be, but is not limited to, random access memory, read-only memory, programmable read-only memory, erasable read-only memory, electrically erasable read-only memory, etc.

[0078] The processor 220 is used to read / write the data or programs stored in the memory 210 and perform corresponding functions. For example, Figure 1 In the robot edge following system 10 shown, the processor 220 of the controller 120 executes the computer program stored in the memory 210 to implement the robot edge following method provided by the embodiments of the present application.

[0079] The communication module 230 is used to establish a communication connection between the electronic device 20 and other communication terminals through a network, and is used to send and receive data through the network. For example, Figure 1 In the robot edge following system 10 shown, the communication module 230 of the controller 120 is used to send and receive data with the vision camera 110 and the power mechanism 140 through a network.

[0080] It should be understood that Figure 2 The structure shown is only a schematic diagram of the structure of the electronic device 20, and the electronic device 20 may further include more or fewer components than those shown in Figure 2 , or have a configuration different from that shown in Figure 2 . Figure 2 Each component shown in can be implemented by hardware, software, or a combination thereof.

[0081] In order to solve the problems that a robot needs to embed wires or preset maps and positioning devices to achieve edge following walking, resulting in high costs, and being easily affected by weather and ground coverings, resulting in poor stability. Referring to Figure 3 , the embodiments of the present application provide a robot edge following method, including steps 12 to 18. And, Figure 1 The controller 120 in the robot edge following system 10 shown can Figure 2 In the structure shown, when the processor 220 reads the computer program stored in the memory 210, the execution of steps 12 to 18 is achieved.

[0082] Step 12: Detect the current frame image captured by the robot to obtain the detection result of the work area.

[0083] Among them, the detection result includes the work area contour in the current frame image. The work area contour includes multiple contour line segments, and each contour line segment includes multiple contour pixel points.

[0084] Step 14: Obtain the initial boundary line from multiple contour line segments.

[0085] Step 16: Based on the endpoints of the initial boundary line and the contour pixel points of the initial boundary line, perform linear fitting to obtain the edge-following boundary line.

[0086] Step 18: Adjust the relative position between the robot and the edge-following boundary line according to the edge-following boundary line.

[0087] Exemplarily, in combination with Figure 1 the edge-following system 10 of the robot shown, when the robot starts patrol inspection, it retreats from the charging pile to a certain distance from the front end of the charging pile, and then moves towards the work area (such as grassland, cleaning area, patrol inspection area, etc.). During the movement, the vision camera 110 collects images in real time and transmits the collected video stream to the controller 120. The controller 120 splits the video stream to obtain the current frame image, detects the current frame image captured by the robot, and obtains the work area contour in the current frame image. The work area contour includes multiple contour line segments, and each contour line segment includes multiple contour pixel points.

[0088] Furthermore, the controller 120 obtains the initial boundary line from multiple contour line segments of the work area contour, and performs linear fitting based on the endpoints of the initial boundary line and the contour pixel points of the initial boundary line to obtain the edge-following boundary line. After determining the edge-following boundary line, the controller 120 sends a control instruction to the power mechanism 140 of the walking mechanism 150, and the power mechanism 140 responds to the control instruction to drive the walking mechanism 150 to adjust the relative position between the robot and the edge-following boundary line, such as adjusting the vertical distance to be equal to the preset edge-following threshold.

[0089] During this process, if the robot enters the work area, the controller 120 will also send an instruction to the power mechanism 140 to drive the operation mechanism to perform operations, such as mowing, cleaning, or patrol inspection, etc.

[0090] In steps 12 to 18 of the above-mentioned edge-following method for the robot, the robot can use only a vision camera to obtain the edge-following boundary line based on the current frame image captured in real time by the vision camera, and accordingly adjust its pose to achieve edge-following walking without a pre-set map, without buried wires, and without relying on positioning devices such as RTK, UWB technology, and GPS, greatly reducing the cost. Moreover, by combining image detection and re-linear fitting, the edge-following boundary line is more accurate and is not affected by obstacles such as trees and buildings, greatly improving the accuracy and reliability of edge-following walking.

[0091] Among them, step 12 can adopt any feasible method to detect the current frame image. For example, the current frame image can be input into a trained convolutional neural network (such as the u-net network, DeepLab network, etc.), and the working area contour in the current frame image can be obtained by using the convolutional neural network, or the current frame image can be processed according to pre-set image processing rules to obtain the working area contour in the current frame image, and its implementation method is not limited.

[0092] In order to make the working area contour more accurate, the concept of sequentially performing semantic segmentation, edge detection, contour detection, and optimization is introduced in step 12. Refer to Figure 4 , the process of obtaining the working area contour in the current frame image in step 12 includes steps 121 to 129.

[0093] Step 121, perform semantic segmentation on the current frame image captured by the robot to obtain a segmentation mask image.

[0094] Step 123, convert the segmentation mask image into a binary mask image according to the category of each pixel and the target pixel category in the segmentation mask image.

[0095] Among them, the binary mask image includes the working area.

[0096] Step 125, perform edge detection on the working area in the binary mask image to obtain the working area edge.

[0097] Step 127, based on the working area edge, perform contour detection on the image of the working area to obtain an initial contour.

[0098] Step 129, perform polygon fitting on the initial contour to obtain the working area contour.

[0099] In this application, a semantic segmentation model is pre-deployed on the controller 120. This semantic segmentation model is a model obtained by using any one of the training methods such as supervised or unsupervised training. Based on the pixel classification rules learned during training, it infers the categories of each pixel in the input image and divides the input image into blocks of different categories. In step 121, the current frame image is fed into the semantic segmentation model for inference to obtain a segmentation mask image divided into blocks of different categories. Accordingly, based on the category of the working area (i.e., the target pixel category), the working area can be determined from the segmentation mask image.

[0100] In one example, in steps 123 to 129, the functions of binaryzation, edge detection, contour detection, and polygon fitting of OpenCV can be called in sequence to assign the pixels of the working area in the segmentation mask image a gray intensity of 255 and assign the gray intensity of the pixels of the non-working area to 0, obtaining a binary mask image. Then, edge detection is performed on the working area in the binary mask image in sequence, contour detection is performed on the image of the working area, and polygon fitting is performed on the initial contour, thereby obtaining a more accurate contour of the working area.

[0101] In another example, any other edge detection algorithm, contour detection algorithm, and polygon fitting algorithm can also be used to perform edge detection, contour detection, and polygon fitting in sequence, and the implementation method is not limited.

[0102] Through the above method, the obtained contour of the working area is closer to the true contour of the working area.

[0103] After obtaining the contour of the working area, in step 14, there are various ways to obtain the initial boundary line from multiple contour line segments of the contour of the working area. For example, the current frame image with the identification information of the contour of the working area is input into a trained boundary inference model, and the boundary inference model infers the initial boundary line, or the initial boundary line can be determined according to a preset rule, and its implementation method is not limited.

[0104] In one example, in order to reduce costs and quickly obtain an accurate initial boundary line, considering that the robot walks along the boundary line and the vision camera captures the image in front of the robot, the concept of using the contour line segment that intersects the width direction of the image captured by the vision camera as the initial boundary line is introduced in step 14. Refer to Figure 5 , the process of obtaining the initial boundary line from multiple contour line segments includes steps 141 to 145.

[0105] Step 141, obtain a dividing line parallel to the width of the current frame image according to the height of the current frame image.

[0106] Step 143, use the contour line segments segmented by the dividing line as the initial boundary line.

[0107] Step 145: For each initial boundary line, obtain the orientation of the initial boundary line based on the intersection point of the initial boundary line and the demarcation line.

[0108] Among them, the orientation includes the left boundary line and the right boundary line.

[0109] When the origin of the pixel coordinate system is at the upper left corner of the current frame image, the width of the previous frame image is consistent with the horizontal coordinate axis u of the pixel coordinate system, and the height is consistent with the vertical coordinate axis v of the pixel coordinate system, the demarcation line can be expressed as: v = k·h, where h represents the height of the current frame image and k is a coefficient.

[0110] According to the characteristics of the robot's perspective, the working area must be located in the lower half of the current frame image, which also means that the initial boundary line must be located in the lower half of the current frame image. Based on this perspective characteristic and in order to obtain the orientation of the initial boundary line according to the intersection point of the demarcation line and the initial boundary line, the k of the demarcation line is restricted to: 0.5 < k < 1.

[0111] When the contour segment is segmented by the demarcation line, that is, one end point of the contour segment is above the demarcation line and the other end point is below the demarcation line, it means that the contour segment has the same orientation as the robot's walking direction, and it is used as the initial demarcation line.

[0112] Furthermore, since the upper left corner of the pixel coordinate system is the origin of the pixel coordinate system, so when w represents the width of the current frame image and the abscissa u of the intersection point of the initial boundary line and the demarcation line j > 0.5·w, the initial boundary line is a line segment extending from the lower right corner of the current frame image, which is on the right side of the robot and is the right boundary line. On the contrary, if the abscissa u of the intersection point of the initial boundary line and the demarcation line j ≤ 0.5·w, it means that the initial boundary line is a line segment extending from the lower left corner of the current frame image, which is on the left side of the robot and is the left boundary line. Therefore, in Step 145, according to the relationship between the abscissa of the intersection point of the initial boundary line and the demarcation line and the width of the current frame, the orientation of the initial boundary line relative to the robot can be determined.

[0113] When the origin of the pixel coordinate system of the current frame image is at the lower right corner or the lower left corner, just set k to: 0 < k < 0.5 and make adaptive adjustments to the above process, which will not be elaborated here.

[0114] Through the above method, there is no need to introduce complex image processing rules and models, and the initial boundary line and its orientation can be determined quickly and accurately, greatly reducing the development cost of the robot along the edge.

[0115] After obtaining the initial boundary line, in step 16, there are various ways to obtain the edge boundary line. For example, the end points of the initial boundary line and the contour pixel points of the initial boundary line can be directly linearly fitted, or the edge boundary line can be obtained by fitting after processing according to rules first, and its implementation method is not restricted.

[0116] In an example, considering that there may be multiple initial boundary lines, in order to improve the orderliness of the robot, in step 16, the idea of introducing the initial boundary line that matches the walking direction of the robot and performing linear fitting to obtain the edge boundary line is introduced. Refer to Figure 6 , the process of obtaining the edge boundary line in step 16 includes steps 161 to 165.

[0117] Step 161: According to the orientation of the initial boundary line and the walking direction of the robot, obtain the target boundary line from each initial boundary line.

[0118] Step 163: For the end points of the target boundary line, obtain a neighborhood circle centered on the end point, and use the pixel points within the neighborhood circle as the neighborhood pixel points of the end point.

[0119] Step 165: Linearly fit the contour pixel points and neighborhood pixel points of the initial boundary line to obtain the edge boundary line.

[0120] When the walking direction of the robot is always moving along the left boundary, the left boundary line is the target boundary line. On the contrary, when the walking direction of the robot is always moving along the right boundary, the right boundary line is the target boundary line.

[0121] The radius of the neighborhood circle can be set flexibly. For example, it can be 30 pixels or 20 pixels, and its value is not restricted.

[0122] Linearly fit the neighborhood pixel points and contour pixel points within the neighborhood circles of the two end points of the target boundary line together to obtain a more accurate edge boundary line.

[0123] In other examples, for each initial boundary line, the neighborhood pixel points and contour pixel points of the two end points of the boundary line can be linearly fitted first to obtain a more accurate initial boundary line. Furthermore, the initial boundary line whose orientation matches the walking direction of the robot is used as the edge boundary line.

[0124] As Figure 7 shown, the blue line is the contour of the working area obtained after the detection in step 12. It can be seen that the blue right boundary line does not match the actual boundary. Therefore, the method of steps 161 to 165 above is used to linearly fit the contour of the working area (that is, linearly fit the contour pixel points and neighborhood pixel points of the initial boundary line) to obtain Figure 7 the red line segment in, and this red line segment coincides with the actual boundary and is more accurate.

[0125] In the above - mentioned manner, that is, first through edge detection, contour detection and polygon fitting, the contour of the working area is obtained. Then, adopting the solution of step 16 and its sub - steps, first supplement or expand the neighborhood pixel points of the endpoints for the initial boundary line, so that the pixel points covered by the initial boundary line are richer. Again, linearly fit the neighborhood pixel points and contour pixel points of the initial boundary line to obtain a more accurate edge - following boundary line. Thus, the accuracy of the edge - following boundary line is greatly improved, which helps to improve the accuracy of edge - following walking.

[0126] After obtaining a highly accurate edge - following boundary line by the above - mentioned method, there are various implementation methods for adjusting the relative position between the robot and the edge - following boundary line in step 17. For example, the vertical distance between the robot and the edge - following boundary line can be obtained, and according to the vertical distance and the set edge - following distance, the rotation angle and the moving distance are obtained. Then, first control it to adjust the angle and then move according to the moving distance. It can also be adjusted by using a preset rule.

[0127] In an example, in order to improve the position adjustment accuracy, in step 18, the idea of first converting the edge - following boundary line from the pixel coordinate system to the camera coordinate system, then calculating the vertical distance between the robot and the edge - following boundary line, and adjusting the robot pose according to this distance is introduced. Refer to Figure 8 , the process of adjusting the relative position between the robot and the edge - following boundary line according to the edge - following connection line includes steps 181 to 185.

[0128] Step 181, based on the internal and external parameter matrices of the robot vision camera, convert the pixel coordinates of the endpoints of the edge - following boundary line into camera coordinates.

[0129] Step 183, according to the camera coordinates and trigonometric functions, obtain the vertical distance between the robot and the edge - following boundary line.

[0130] Step 185, when the vertical distance is not equal to the edge - following distance, control the robot to move so that the vertical distance between the robot and the edge - following boundary line is equal to the edge - following distance.

[0131] It should be understood that no adjustment is required when the vertical distance is equal to the edge - following distance.

[0132] Since the origin of the camera coordinate system is the camera optical center, the abscissa x is horizontally to the right, and the optical axis center line is in the z - direction towards the robot's forward direction. Therefore, after obtaining the pixel coordinates (u 1 , v 1 ) and (u 2 , v 2 ) of the two endpoints of the edge - following boundary line, multiply the matrices corresponding to the two pixel coordinates by the internal and external parameter matrices of the vision camera respectively to obtain the coordinates of the two endpoints in the camera coordinate system, which can be expressed as (x1 , y 1 , z 1 ), and (x 2 , y 2 , z 2 ), or (x 1 , z 1 ), and (x 2 , z 2 ).

[0133] Furthermore, based on the camera coordinates and trigonometric functions, the perpendicular distance between the robot and the edge boundary line can be obtained.

[0134] For example, as Figure 9 shown, the robot moves along the edge to the right in the working area. Point O is the camera center (i.e., the position of the robot), point A is the projection point of endpoint P 1 on the vertical line where the camera center is located, and point B is the projection point of endpoint P 2 on the vertical line where the camera center is located. Taking the position of the robot as an endpoint and the forward direction of the robot as the extension direction, a ray is drawn to obtain the extended line of the robot's central axis ( Figure 9 OA in it is on this extended line). If this extended line does not intersect the edge boundary line, it means that the walking direction of the robot is moving away from the edge boundary line. At this time, P1P2 is the edge boundary line, and L is the distance between the camera center and the edge boundary line, that is, the perpendicular distance between the robot and the edge boundary line.

[0135] Since L = OC * sin(n), OC = AC - AO, and AC = AP1 / tanh(n), the calculation formula for the perpendicular distance between the robot and the edge boundary line is:[[]]

[0136] As Figure 10 shown, the robot moves along the edge to the right in the working area. Point O is the camera center (i.e., the position of the robot), point A is the projection point of endpoint P 1 on the vertical line where the camera center is located, and point B is the projection point of endpoint P 2 on the vertical line where the camera center is located. Taking the position of the robot as an endpoint and the forward direction of the robot as the extension direction, a ray is drawn to obtain the extended line of the robot's central axis ( Figure 10 OA in it is on this extended line). If this extended line intersects the edge boundary line, it means that the walking direction of the robot is moving closer to the edge boundary line. At this time, P1P2 is the edge boundary line, and L is the distance between the camera center and the edge boundary line, that is, the perpendicular distance between the robot and the edge boundary line.

[0137] Since \(L = OC\times\sin(n)\), \(OC = AC + AO\), and \(AC=\frac{AP1}{\tanh(n)}\), the calculation formula for the perpendicular distance between the robot and the edge boundary line is as follows:

[0138] When the robot moves along the edge to the left in the working area, the calculation method of the perpendicular distance between the robot and the edge boundary line is similar to the method when moving along the edge to the right provided above, and will not be elaborated here.

[0139] After obtaining the perpendicular distance in the above manner, the relationship between the perpendicular distance and the edge distance can be determined. Furthermore, in step 185, if the perpendicular distance is greater than or less than the edge distance, the distance difference between the two is input into the PID controller, and the PID controller uses the PID principle to adjust the position of the robot until the perpendicular distance between the robot and the edge boundary line is equal to the edge distance.

[0140] In other examples, any other control method can also be used for adjustment, which is not limited here.

[0141] In addition, considering the situation that there may be no boundary line in the current frame image that matches the edge walking direction. Therefore, in order to ensure the orderliness and stability of the robot moving along the edge, in the robot edge following method provided in the embodiments of the present application, the concept of controlling the robot to rotate and adjust its pose to find the required boundary line when there is no boundary line in the current frame image that matches the edge walking direction is introduced. Referring to Figure 11 The robot edge following method provided in the embodiments of the present application further includes steps 15 to 17.

[0142] Step 15: Detect whether there is an initial boundary line whose orientation matches the walking direction of the robot. If so, execute step 16; if not, execute step 17.

[0143] Step 17: Control the robot to rotate to adjust its posture, and after adjusting the posture, control the robot to take a picture to obtain a new current frame image. After step 17, return to execute step 12.

[0144] When the walking direction of the robot is always moving along the left boundary, if there is an initial boundary line with the orientation of the left boundary line, it means that there is an initial boundary line whose orientation matches the walking direction of the robot; otherwise, there is no such boundary line.

[0145] Similarly, when the walking direction of the robot is always moving along the right boundary, if there is an initial boundary line with the orientation of the right boundary line, it means that there is an initial boundary line whose orientation matches the walking direction of the robot; otherwise, there is no such boundary line.

[0146] Furthermore, in step 17, a unit angle can be preset. For each rotation of a unit angle, a photograph is taken to obtain a new current frame image. For example, if the unit angle is 5°, after the robot rotates 5°, a new current frame image is taken, and then step 12 is executed again. In the next detection, if there is no initial boundary line matching the walking direction, the robot continues to rotate until an initial boundary line matching the walking direction appears.

[0147] In the above manner, the situation where the robot cannot find the required boundary line and thus cannot perform edge following is avoided, ensuring the accuracy and reliability of the robot's edge following.

[0148] To make the edge following process of the robot proceed orderly, in the robot edge following method provided in the embodiment of the present application, a step of determining whether the edge following end point is reached is introduced. Referring to Figure 12 , steps 21 to 25 are further included.

[0149] Step 21: Detect whether there is a charging pile in the current frame image. If not, execute step 12; if so, execute step 23.

[0150] Input the current frame image into a pre-trained image detection model. The image detection model infers the charging pile area in the current frame image. If the charging pile area cannot be obtained, there is no charging pile; otherwise, there is a charging pile.

[0151] Step 23: Obtain the relative distance and positional relationship between the charging pile and the robot according to the charging pile area in the current frame image.

[0152] Step 25: Detect whether the relative distance and positional relationship meet the pile-return condition corresponding to the walking direction. If not, execute step 12; if so, end the edge following.

[0153] In step 23, there are various ways to obtain the relative distance between the charging pile and the robot. For example, according to the height ratio of the charging pile area in the current frame image, a conversion formula fitted based on experimental data or calibration data can be used to calculate the relative distance between the robot and the charging pile. It is also possible to convert the center point coordinates to the camera coordinate system and calculate the distance between the converted center point coordinates of the charging pile and the camera center coordinates. And the above methods are all examples, and the implementation methods are not limited.

[0154] There are also various ways to obtain the positional relationship between the charging pile and the robot. For example, draw a vertical center line on the current frame image. When more than half of the charging pile area is on the right side of the vertical center line, it is determined that the charging pile is close to the right boundary line; otherwise, it is close to the left boundary line. It is also possible to infer the positional relationship between the charging pile and the robot by a pre-trained neural network model. The implementation methods are not limited.

[0155] After obtaining the relative distance and positional relationship between the robot and the charging pile, if the robot walks along the right boundary, when the charging pile is close to the right boundary line and the relative distance is less than the distance threshold, the charging-back condition corresponding to the walking direction is satisfied; otherwise, it is not satisfied.

[0156] If the robot walks along the left boundary, when the charging pile is close to the left boundary line and the relative distance is less than the distance threshold, the charging-back condition corresponding to the walking direction is satisfied; otherwise, it is not satisfied.

[0157] The above method for detecting whether the relative distance and positional relationship satisfy the charging-back condition corresponding to the walking direction is only an example, and its implementation method is not limited.

[0158] Adopting the above method to determine in real time whether the end of walking along the edge is reached, so as to accurately control the robot to end or continue walking along the edge, improving the orderliness and comprehensiveness of the robot walking along the edge.

[0159] Based on the above steps 21 to 25, after step 17 is executed, step 21 is executed.

[0160] Based on the same concept as the above robot edge-following method, referring to Figure 13 , the embodiment of the present application further provides a robot edge-following device 30, including an image detection module 310, a boundary acquisition module 320, and an edge-following execution module 330.

[0161] The image detection module 310 is used to detect the current frame image captured by the robot to obtain a work area detection result. Among them, the detection result includes the work area contour in the current frame image, the work area contour includes multiple contour line segments, and each contour line segment includes multiple contour pixel points.

[0162] The boundary acquisition module 320 is used to obtain an initial boundary line from multiple contour line segments.

[0163] The boundary acquisition module 320 is further used to perform linear fitting based on the endpoints of the initial boundary line and the contour pixel points of the initial boundary line to obtain an edge-following boundary line.

[0164] The edge-following execution module 330 is used to adjust the relative position between the robot and the edge-following boundary line according to the edge-following boundary line.

[0165] In an example, the robot edge-following device 30 further includes a boundary search module and an end point discrimination module.

[0166] The boundary search module is used to execute steps 15 and 17 provided above.

[0167] The end point discrimination module is used to execute steps 21 to 25 provided above.

[0168] Under the collaborative action of the above-mentioned edge-following device 30 of the robot, namely the image detection module 310, the boundary acquisition module 320, and the edge-following execution module 330, the robot can use only a vision camera to obtain the edge-following boundary line based on the current frame image captured by the vision camera in real time, and adjust its pose accordingly to achieve edge-following walking without a pre-set map, without buried wires, and without relying on positioning devices such as RTK, UWB technology, and GPS, greatly reducing the cost. Moreover, through the combination of image detection and re-linear fitting, the edge-following boundary line is made more accurate and is not affected by obstacles such as trees and buildings, greatly improving the accuracy and reliability of edge-following walking.

[0169] For the specific implementation and effects of the edge-following device 30 of the robot, reference can be made to the description of the implementation of the edge-following method of the robot in the above text. For example, for the specific implementation and effects of the image detection module 310, reference can be made to the description of the relevant content in step 12 above; for the specific implementation and effects of the boundary acquisition module 320, reference can be made to the description of the relevant content in steps 14 to 16 above; for the specific implementation and effects of the edge-following execution module 330, reference can be made to the description of the relevant content in step 18 above, which will not be elaborated here.

[0170] In addition, each module of the above-mentioned edge-following device 30 of the robot can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in or independent of the processor 220 in the electronic device 20 in the form of hardware, or stored in the memory 210 of the electronic device 20 in the form of software, so that the processor 220 can call and execute the operations corresponding to each of the above modules to implement the edge-following method of the robot provided in the above text.

[0171] The embodiment of the present application also provides a robot, including a vision camera 110 and a controller 120, and the controller 120 is communicatively connected to the vision camera 110.

[0172] The controller 120 is used to implement the edge-following method of the robot provided above.

[0173] The embodiment of the present application also provides an electronic device 20, including a processor 220 and a memory 210, and the memory 210 stores a computer program that can be executed by the processor 220, and the processor 220 can execute the computer program to implement the edge-following method of the robot provided above.

[0174] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor 220, it implements the edge-following method of the robot proposed in the embodiment of the present application.

[0175] In summary, the edge-following method, device, robot, and electronic device provided in the embodiment of the present application have at least the following beneficial effects:

[0176] (1) Without pre-setting maps and positioning information, the robot's edge-following motion function is achieved only through a camera;

[0177] (2) The robot automatically detects the boundary through vision, calculates the distance between the robot and the boundary, and can combine PID control to make the robot always move along the edge;

[0178] (3) Without pre-buried wires and positioning devices, only a single monocular camera is used, which greatly reduces the cost.

[0179] In several embodiments provided by the present application, it should be understood that the disclosed device and method can also be implemented in other ways. The device embodiments described above are only illustrative. For example, the flowcharts and block diagrams in the drawings show the possible architectures, functions, and operations of the device, method, and computer program product according to multiple embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0180] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0181] When the above-mentioned functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory 210 (ROM, Read-Only Memory), random access memory 210 (RAM, Random Access Memory), magnetic disks, or optical discs.

[0182] The foregoing are only the preferred embodiments of this application and are not intended to limit this application. For those skilled in the art, this application may have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

Claims

1. A robot edge following method, characterized in that: The method comprises: Detecting the current frame image taken by the robot to obtain a work area detection result; wherein the detection result includes a work area contour in the current frame image, the work area contour includes a plurality of contour line segments, and each of the contour line segments includes a plurality of contour pixel points; Obtaining an initial boundary line from the plurality of contour line segments; Based on the endpoints of the initial boundary line and the contour pixel points of the initial boundary line, linear fitting is performed to obtain an edge boundary line; According to the edge boundary line, the relative position between the robot and the edge boundary line is adjusted.

2. The robot edge following method according to claim 1, characterized in that: The step of performing linear fitting based on the endpoints of the initial boundary line and the contour pixel points of the initial boundary line to obtain the boundary line along the edge includes: Obtaining a target boundary line from each of the initial boundary lines according to the orientation of the initial boundary lines and the walking direction of the robot; For the endpoint of the target boundary line, a neighborhood circle centered on the endpoint is obtained, and the pixel points in the neighborhood circle are used as the domain pixel points of the endpoint; Linear fitting is performed on the contour pixel points of the initial boundary line and the neighborhood pixel points to obtain an edge boundary line.

3. The robot edge following method according to claim 1 or 2, characterized in that: The step of detecting the current frame image taken by the robot to obtain the workspace detection result includes: Perform semantic segmentation on the current frame image taken by the robot to obtain a segmentation mask map; According to the category of each pixel in the segmentation mask image and the category of the target pixel, the segmentation mask image is converted into a binary mask image; wherein the binary mask image includes a working area; Performing edge detection on the working area in the binary mask image to obtain the edge of the working area; Based on the edge of the working area, contour detection is performed on the image of the working area to obtain an initial contour; Perform polygon fitting on the initial outline to obtain the working area outline.

4. The robot edge following method according to claim 1 or 2, characterized in that: The step of obtaining an initial boundary line from the plurality of contour line segments comprises: According to the height of the current frame image, a dividing line parallel to the width of the current frame image is obtained; Using the contour segment segmented by the dividing line as the initial boundary line; For each of the initial boundary lines, the orientation of the initial boundary line is obtained according to the intersection of the initial boundary line and the dividing line; wherein the orientation includes a left boundary line and a right boundary line.

5. The robot edge following method according to claim 1 or 2, characterized in that: Before the step of detecting the current frame image shot by the robot to obtain the workspace detection result, the method further includes: Detect whether there is a charging pile in the current frame image; If not, the step of detecting the current frame image taken by the robot to obtain the work area detection result is executed; If yes, then obtaining the relative distance and position relationship between the charging pile and the robot according to the charging pile area in the current frame image; When the relative distance and the position relationship do not satisfy the pile return condition corresponding to the walking direction, the step of detecting the current frame image taken by the robot to obtain the work area detection result is performed.

6. The robot edge following method according to claim 1 or 2, characterized in that: After the step of obtaining an initial boundary line from the plurality of contour line segments, the method further comprises: Detecting whether there is the initial boundary line whose orientation matches the walking direction of the robot; If yes, then performing the step of performing linear fitting based on the endpoints of the initial boundary line and the contour pixels of the initial boundary line to obtain the boundary line along the edge; If not, the robot is controlled to rotate to adjust the posture, and after adjusting the posture, the robot is controlled to shoot to obtain a new current frame image, and the step of returning to execute the detection of the current frame image shot by the robot to obtain the workspace detection result.

7. The robot edge following method according to claim 1 or 2, characterized in that: The step of adjusting the relative position between the robot and the edge boundary line according to the edge boundary line comprises: Based on the internal and external parameter matrix of the robot vision camera, the pixel coordinates of the endpoints of the edge boundary line are converted into camera coordinates; Obtaining a vertical distance between the robot and the edge boundary line according to the camera coordinates and trigonometric functions; When the vertical distance is not equal to the edge distance, the robot is controlled to move so that the vertical distance between the robot and the edge boundary line is equal to the edge distance.

8. A robot edge device, characterized in that: It includes an image detection module, a boundary acquisition module and an edge execution module; The image detection module is used to detect the current frame image taken by the robot to obtain a work area detection result; wherein the detection result includes the work area contour in the current frame image, the work area contour includes a plurality of contour line segments, and each of the contour line segments includes a plurality of contour pixel points; The boundary acquisition module is used to obtain an initial boundary line from the plurality of contour line segments; The boundary acquisition module is further used to perform linear fitting based on the endpoints of the initial boundary line and the contour pixel points of the initial boundary line to obtain the boundary line along the edge; The edge execution module is used to adjust the relative position between the robot and the edge boundary line according to the edge boundary line.

9. A robot, characterized in that: It includes a visual camera and a controller, wherein the controller is communicatively connected with the visual camera; The controller is used to implement the robot edge-following method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the robot edge-following method according to any one of claims 1 to 7.

Images