Multi-navigation-mode robot, control method and computer readable storage medium

Abstract

The invention relates to robots such as intelligent mowers, in particular to a multi-navigation-mode robot, a control method and a computer readable storage medium. The multi-navigation-mode robot comprises a boundary identification module which is used for acquiring boundary information of a working area and identifying the boundary of the working area according to the boundary information; the magnetic sensor module is used for detecting a magnetic guiding device arranged at the boundary of the working area or near the boundary of the obstacle; and the control module is used for controlling the moving mode of the robot based on the recognition result of the boundary recognition module and / or the detection result of the magnetic sensor module. The boundary recognition module and the magnetic sensor module jointly detect the boundary, so that the image detection boundary and the magnetic strip complement each other, the navigation mode reliability judgment is not needed, and the implementation is simple.

Description

Technical Field

[0001] This application relates to robots such as intelligent lawnmowers, and more specifically to a multi-navigation mode robot, a control method, and a computer-readable storage medium. Background Technology

[0002] Intelligent lawnmowers need to detect the boundaries of their working area during operation to confine their work to the area or along its boundaries. Common boundary detection methods include: 1) establishing virtual boundaries, such as controlling a lawnmower equipped with an RTK positioning module to walk around the boundary and establishing a virtual boundary based on the walking trajectory and positioning data; 2) visual boundary detection, such as installing visual sensors on the lawnmower to detect the boundaries of the working area through visual image processing.

[0003] However, single boundary detection methods or navigation modes often have certain failure scenarios. For example, RTK positioning cannot accurately calculate the relative positional relationship between the lawnmower and the boundary when satellite signals are blocked; or visual sensors cannot effectively recognize boundaries when boundary image features are not obvious. Summary of the Invention

[0004] To address the failure scenarios inherent in existing single navigation modes, this invention provides a multi-navigation mode robot, a control method, and a computer-readable storage medium.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A multi-navigation mode robot, comprising:

[0007] The boundary recognition module is used to acquire work area boundary information and identify the work area boundary based on this boundary information.

[0008] A magnetic sensor module is used to detect magnetic guiding devices located near the boundary of the working area;

[0009] The control module controls the robot's movement based on the recognition results of the boundary recognition module and / or the detection results of the magnetic sensor module.

[0010] Furthermore, the boundary recognition module includes:

[0011] The vision sensor module is used to acquire two-dimensional images of the environment and environmental depth maps or three-dimensional point cloud maps;

[0012] The image processing module is used to acquire the two-dimensional image and the environmental depth map or three-dimensional point cloud map of the vision sensor module, and to identify the boundary of the working area through image processing methods.

[0013] Furthermore, when detecting the boundary of the working area, the image processing module acquires a two-dimensional image obtained by the visual sensor module, and uses a semantic segmentation method to segment the working area in the two-dimensional image, wherein the boundary of the working area is the boundary of the working area in the two-dimensional image.

[0014] Furthermore, when the image processing module detects the boundary of the working area, the robot can also detect the boundary of obstacles based on the height information of the environmental point cloud.

[0015] Furthermore, the step of segmenting the working region in the two-dimensional image using the semantic segmentation method includes: constructing a semantic segmentation network; collecting an image dataset containing the working region, at least annotating the working region in the image dataset, and training a neural network model using the annotated image dataset; using the trained neural network model to segment the two-dimensional image acquired by the robot to obtain a working region mask; and comparing the working region mask with the original image to segment the working region in the original image.

[0016] Furthermore, the vision sensor module is a binocular camera. Two two-dimensional images are obtained through the two cameras of the binocular camera. The depth of the feature points in the two-dimensional images is calculated to obtain an environmental depth map. The three-dimensional coordinates of the feature points in the robot positioning coordinate system are calculated based on the environmental depth map to obtain the three-dimensional point cloud map.

[0017] Furthermore, when detecting the boundary of the working area, the image processing module acquires the environmental depth map or 3D point cloud map from the vision sensor module, and matches the 2D image acquired by the vision sensor module with the environmental depth map or 3D point cloud map to obtain the 3D point cloud corresponding to the boundary of the working area in the 2D image.

[0018] Furthermore, the robot detects boundaries of different detection types through the boundary recognition module or the magnetic sensor module. When the robot operates along the edge and detects a boundary of the second detection type, it selects the next boundary to operate along the edge based on the detection type of the current boundary and the detection type of the boundary of the second detection type.

[0019] Furthermore, the robot detects boundaries of different detection types through the boundary recognition module or the magnetic sensor module. When the robot operates along the edge and detects multiple detection types of boundaries, if the current boundary detection type is an image detection boundary and a magnetically guided boundary is detected, the robot searches for the direction of the magnetically guided device and switches to travel along the magnetically guided boundary. If the current boundary detection type is a magnetically guided boundary and an image detection boundary within a certain range centered on the robot is detected, the robot switches to travel along the image detection boundary.

[0020] Furthermore, when the magnetic sensor module searches for the direction of the magnetic guidance device, if the robot is set to the right edge, it will search to the left first; if the robot is set to the left edge, it will search to the right first.

[0021] Furthermore, the work area boundary includes the work area range boundary, obstacle boundary, and the boundary of sub-regions or zones within the work area.

[0022] Furthermore, the magnetic guiding device includes a magnetic strip, a magnetic marker, a magnetic nail, or an electromagnetically conductive wire.

[0023] A control method for a multi-navigation mode robot, the robot comprising a boundary recognition module, a magnetic sensor module, and a control module; the method comprising:

[0024] The robot performs tasks in the work area, including performing tasks within the work area and performing tasks along the edge;

[0025] During the operation within the work area or along the edge, the robot simultaneously detects the boundary of the work area using both a boundary recognition module and a magnetic sensor module.

[0026] Furthermore, the robot performs tasks within the work area, including performing tasks within the work area and performing tasks along the edge, including:

[0027] Step 1: In response to the user-set lawn mowing schedule time conditions, the robot performs operations within the work area at the set time;

[0028] Step 2: After certain conditions are met for the operations within the work area, switch to performing edge operations along the boundary of the work area;

[0029] Step 3: After the edge operation has been performed for a certain period of time or the operation trajectory meets certain conditions, switch back to performing the operation within the work area;

[0030] Step 4: Repeat steps 2 and 3 above until the task is completed.

[0031] Furthermore, the robot's operation within the work area includes: the robot moving forward in a straight line to perform grass mowing, and when the boundary of the work area is detected, rotating at a certain angle and moving forward in a straight line again to perform grass mowing.

[0032] Furthermore, during the process of performing the edge-following operation, the robot also includes: simultaneously detecting boundaries of different detection types through the image processing module and the magnetic sensor module in the boundary recognition module; when the robot detects only one type of boundary, it performs the edge-following operation along the detected boundary.

[0033] Furthermore, during the operation of the robot within the work area, the robot also includes: simultaneously detecting boundaries of different detection types through the image processing module and the magnetic sensor module in the boundary recognition module; when the robot detects any type of boundary, it rotates a certain angle and continues to perform the operation in a straight line.

[0034] Furthermore, when the robot detects only one type of boundary, performing the edge-following operation along the detected boundary includes: when only an image detection boundary is detected, performing PID following on the nearest boundary point using the robot's right front corner point, and following the edge using the robot's right-side edge-following method; when only a magnetically guided boundary is detected, the robot rotates to search for the magnetically guided device direction and travels along the direction that makes the peak of the magnetic induction signal at the center of the magnetic sensing array.

[0035] Furthermore, during the edge-following operation, the robot simultaneously detects boundaries of different detection types through the image processing module and the magnetic sensor module in the boundary recognition module. When a boundary of the second detection type is detected, the next boundary for the edge-following operation is selected based on the detection type of the current boundary and the detection type of the boundary of the second detection type.

[0036] Furthermore, during the edge-following operation, the robot simultaneously detects different types of boundaries through the image processing module and the magnetic sensor module in the boundary recognition module. When multiple types of boundaries are detected, if the current boundary is an image detection boundary and a magnetically guided boundary is detected, the robot searches for the direction of the magnetically guided device and switches to travel along the magnetically guided boundary. If the current boundary is a magnetically guided boundary and an image detection boundary is detected within a certain range centered on the robot, the robot switches to travel along the image detection boundary.

[0037] Furthermore, when the robot searches for the direction of the magnetic guidance device, if the robot is set to...

[0038] If the robot is set to search along the right edge, it will search to the left first; if the robot is set to search along the left edge, it will search to the right first.

[0039] Furthermore, prior to step 1, the robot is positioned at the charging station in its initial state.

[0040] Furthermore, after step 3, the process includes: after performing the work within the work area or the work along the edge to reach a certain power level, the robot searches for the boundary of the work area and returns to the charging station along the boundary of the work area.

[0041] A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it can implement the control method of the multi-navigation mode robot described above. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating the working process of a multi-navigation mode robot provided in an embodiment of the present invention.

[0043] Figure 2A The image provided in this embodiment of the invention is a two-dimensional image before segmentation using a semantic segmentation method;

[0044] Figure 2B This is a two-dimensional image segmented using a semantic segmentation method, provided in an embodiment of the present invention.

[0045] Figure 3A This is a schematic diagram illustrating the execution of a robot after detecting a boundary while it is working within an area, as provided in an embodiment of the present invention.

[0046] Figure 3B This is a schematic diagram illustrating the execution of a robot after detecting a magnetic strip while it is working in an area, according to an embodiment of the present invention.

[0047] Figure 4A This is a schematic diagram illustrating the execution status of a robot after detecting a boundary when performing edge-following operations, as provided in an embodiment of the present invention.

[0048] Figure 4B This is a schematic diagram illustrating the execution status of a robot after detecting a magnetic strip during edge-following operations, as provided in an embodiment of the present invention.

[0049] Figure 5A , 5B A schematic diagram of a non-physical boundary provided for an embodiment of the present invention;

[0050] Figure 6A , 6B This is a schematic diagram illustrating the boundary switching process of a robot when it detects multiple types of boundaries, as provided in an embodiment of the present invention.

[0051] Figure 7 This is a schematic diagram illustrating the robot's operation when the magnetic strip is located outside the working area, as provided in an embodiment of the present invention.

[0052] Figure 8A This is a schematic diagram of the robot's movement without a magnetic strip after the search direction has been set in advance, as provided in an embodiment of the present invention.

[0053] Figure 8B This is a schematic diagram of the robot's movement under magnetic strip conditions after the search direction has been prioritized, as provided in an embodiment of the present invention.

[0054] Figure 9 This is a schematic diagram of a robot determining the boundary of its working area based on the direction of its edge trajectory, provided in an embodiment of the present invention.

[0055] Figure 10 This is a schematic diagram of the priority search process of the robot when the magnetic strip is improperly positioned, as provided in an embodiment of the present invention.

[0056] Figure 11 This is a schematic diagram illustrating the misidentification of a magnetic strip as a non-working area under backlight conditions, provided by an embodiment of the present invention.

[0057] Figure 12 This is a schematic diagram illustrating the robot's execution when a magnetic strip is mistakenly identified as an image detection boundary, as provided in an embodiment of the present invention.

[0058] Figure 13A This is a grassland boundary map before the closing operation processing provided in an embodiment of the present invention;

[0059] Figure 13B This is a grassland boundary map after closing operation processing provided in an embodiment of the present invention;

[0060] Figure 14A The mask image provided in this embodiment of the invention before the closing operation processing;

[0061] Figure 14B The mask image after closing operation processing is provided in the embodiment of the present invention. Detailed Implementation

[0062] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0063] It should be noted that, according to the definition of navigation as "a method of guiding an aircraft or ship to move from one point to another along a certain route," the "navigation mode" in the instruction manual includes: the process of environmental information detection or positioning for controlling the robot's movement, as well as the process of controlling the robot's movement.

[0064] To facilitate a better understanding of the technical solutions in this invention by those skilled in the art, a magnetic strip is used as an example of a magnetic guiding device in the description. However, the aforementioned magnetic guiding device may include magnetic strips, magnetic markers, magnetic nails, electromagnetic wires, and other equivalent alternatives that can be obtained by those skilled in the art without inventive effort.

[0065] When no specific definition of "boundary" is specified, "boundary" in this invention can be understood as "work area boundary". The "work area boundary" can be the boundary of the work area range, the boundary of an obstacle, the boundary of a partition within the work area, or other boundaries used to define the robot's work area.

[0066] Furthermore, the "boundary" can be classified according to which detection module obtains the boundary, thus obtaining boundaries of different detection types. For example, the image detection boundary is the boundary obtained by the image processing module detecting the boundary of the working area, and the magnetic guidance boundary is the boundary obtained by the magnetic sensor module detecting the magnetic guidance device.

[0067] In addition, in this invention, "operating along the edge" refers to performing work along the boundary of the work area; "obstacle avoidance along the edge" refers to walking along the boundary of an obstacle to avoid it.

[0068] Example 1

[0069] When a robot is working, it needs to detect the boundaries of its work area to confine its operations to that area or along those boundaries. This boundary detection is achieved through boundary recognition modules, such as image processing or computer vision modules, when the robot is working within its area, searching for boundaries, performing edge-based tasks, or navigating obstacles. However, relying solely on boundary recognition modules has limitations; for example, when the image features of the boundary are not readily apparent, the module may fail to effectively identify the boundary.

[0070] To address the issue of boundary recognition modules failing in certain scenarios, this embodiment proposes a multi-navigation mode robot, comprising: a boundary recognition module installed on the robot to acquire work area boundary information and identify the work area boundary based on the boundary information; a magnetic sensor module to detect magnetic guidance devices (such as magnetic strips) placed near the work area boundary; and a control module to control the robot's movement mode based on the recognition results of the boundary recognition module and / or the detection results of the magnetic sensor module.

[0071] The boundary recognition module includes: a visual sensor module for acquiring a two-dimensional image of the environment and an environmental depth map or a three-dimensional point cloud map; and an image processing module for acquiring the two-dimensional image and the environmental depth map or the three-dimensional point cloud map from the visual sensor module, and recognizing the boundary of the working area through an image processing method.

[0072] The vision sensor module includes a binocular camera. Two two-dimensional images are obtained through the two cameras of the binocular camera. The depth of feature points in the two-dimensional images is calculated to obtain an environmental depth map. The three-dimensional coordinates of the feature points in the robot positioning coordinate system are calculated based on the environmental depth map to obtain the three-dimensional point cloud map.

[0073] Specifically, the image processing module detects the boundary of the working area, obtains a two-dimensional environmental image from the visual sensor module, and uses a semantic segmentation method to segment the working area and non-working area in the two-dimensional image. The boundary between the working area and non-working area is the boundary of the working area in the two-dimensional image.

[0074] When segmenting the working region and non-working region in the two-dimensional image using the semantic segmentation method described above: construct a semantic segmentation network such as FCN network, DeepLab network, Unet network, U2net network, etc.; collect an image dataset containing the aforementioned working region and non-working region; at least annotate the working region in the image dataset; train the neural network parameters using the annotated image dataset to obtain a trained neural network model; such as... Figure 2A As shown in Figure 2B, the trained neural network model is used to segment the image acquired by the robot to obtain a working region mask; as shown in Figure 2B, the working region mask is compared with the original image to segment the working region and non-working region in the original image.

[0075] The system acquires an environmental depth map or a 3D point cloud map from the vision sensor module, and matches the 2D image obtained by the vision sensor module with the environmental depth map or the 3D point cloud map to obtain the 3D point cloud corresponding to the boundary of the working area in the 2D image, that is, the 3D coordinates of the boundary points of the working area, and the coordinate system is consistent with the robot positioning coordinate system, thereby realizing the working area boundary detection based on image processing.

[0076] Furthermore, since the robot can obtain the environmental depth map or 3D point cloud map through the vision sensor module, it can also detect the obstacle boundaries in the working area based on the height information of the environmental point cloud. For example, it can obtain the obstacle point cloud with a certain height and use its two-dimensional plane coordinates as the obstacle boundary coordinates to achieve obstacle boundary detection.

[0077] During the process of the magnetic sensor module detecting the boundary, the magnetic sensor module mounted on the robot senses the magnetic strip set near the boundary of the working area, where the boundary of the working area can be the boundary of the working area range, the boundary of the obstacle, or other boundaries, and rotates to search the direction of the magnetic strip, that is, the direction in which the peak of the magnetic induction signal is located at the center of the magnetic sensing array.

[0078] like Figure 1 As shown, the working process of the multi-navigation mode robot in this embodiment is as follows:

[0079] Step 101: In the initial state, the robot is located at the charging station.

[0080] In response to the user-set lawn mowing schedule, the robot performs its work within the designated work area at the set time, randomly moving and mowing within the work area. Specifically, in step 102, the robot first retreats a specific distance from the charging station, rotates at a specific angle, and moves forward in a straight line to perform lawn mowing. Upon detecting the boundary of the work area, in step 103, the robot rotates at a certain angle and moves forward in a straight line again to perform lawn mowing.

[0081] After the robot completes the work within the work area and meets the conditions such as work time or coverage, it simultaneously detects the work area boundary through the boundary recognition module and the magnetic sensor module. Step 104: Locate the work area boundary. Step 105: Based on the type of boundary detected, perform the corresponding operation: Step 106: When an image detection boundary is detected, perform edge-following operation along the image detection boundary. Step 107: During the edge-following operation along the image detection boundary, both the image detection boundary and the magnetically guided boundary are detected simultaneously. Step 108: Switch to performing edge-following operation along the direction of the magnetically guided boundary. Step 109: When a magnetically guided boundary is detected, perform edge-following operation along the magnetically guided boundary. Step 110: During the edge-following operation along the magnetically guided boundary, both the image detection boundary and the magnetically guided boundary are detected simultaneously. Step 111: Switch to performing edge-following operation along the image detection boundary.

[0082] Step 112: After the edge operation has been performed for a set time or the operation trajectory meets a set trajectory or other specific conditions, switch back to performing the operation within the work area.

[0083] Step 113: After performing the work within the work area or the work along the edge to reach a certain power level, the robot finds the boundary of the work area and returns to the charging pile along the boundary of the work area.

[0084] During the edge operation, when the robot detects an obstacle boundary, it walks along the obstacle boundary to bypass the obstacle until it returns to the boundary of the work area, or until the obstacle bypass trajectory closes and it searches for the boundary of the work area again.

[0085] When the robot performs operations within the aforementioned work area, along the edge, or searches for the boundary of the work area, it simultaneously detects the boundary of the work area using both the boundary recognition module and the magnetic sensor module.

[0086] When performing tasks within the work area, the robot simultaneously detects boundaries using both the image processing module and the magnetic sensor module. When the robot detects a boundary of any type, it rotates by an angle and continues performing the task in a straight line. Specifically, as... Figure 3A As shown, when the robot is working within the work area and detects an image detection boundary, it rotates a certain angle and continues to perform the task in a straight line; for example... Figure 3B As shown, when the robot is working in the work area and detects the magnetic strip 31, it rotates at a certain angle and continues to perform the work in a straight line.

[0087] When performing edge-following operations, including obstacle avoidance, and the robot simultaneously detects boundaries using both the image processing module and the magnetic sensor module, if the robot detects only one type of boundary (image or magnetic stripe), it performs the edge-following operation along that detected boundary. Specifically, as... Figure 4A As shown, when only the image detection boundary 41 is detected, the machine's right front corner point is used to perform PID tracking on the nearest boundary point. Here, PID stands for Proportional-Integral-Derivative, meaning the robot follows the edge using its right-side edge-following method. Figure 4B As shown, when only the magnetic stripe 31 is detected and the image detection boundary 42 is not detected, the robot searches for the magnetic stripe 31 in the direction that makes the peak of the magnetic induction signal at the center of the magnetic sensing array.

[0088] Example 2

[0089] During edge-following operations or obstacle avoidance, the robot may detect multiple types of boundaries simultaneously. When multiple types of boundaries are detected at the same time, the robot needs to select one of the multiple types of boundaries to continue performing the edge-following task.

[0090] In existing multi-navigation mode switching technologies, switching control is mainly achieved by setting a priority selection mode and a reliability judgment mechanism. For example, when both the satellite positioning or visual navigation boundary and the magnetic guidance device boundary are detected at the same time, satellite positioning or visual navigation is prioritized. Only when satellite positioning or visual navigation is unreliable will the system switch to magnetic guidance device navigation.

[0091] However, on the one hand, switching navigation modes based on navigation reliability analysis is complex, and not all navigation systems can determine their reliability. For example, when boundary image features are not obvious, leading to errors in boundary detection, the vision system will not provide clear feedback. On the other hand, switching navigation modes based on reliability analysis still carries the risk of the robot leaving the work area. For example, ... Figure 5AAs shown, when the boundary is the boundary between a user and their neighbor's lawn, this boundary is not a physical boundary, or as... Figure 5B As shown, when the front and back are both steep slopes covered with grass, the robot's detection of the physical boundary (outermost boundary) is still reliable. Therefore, the robot will not switch to working along the magnetic strip 31, thus causing the robot to leave the working area.

[0092] To address the issues of complex implementation and robot boundary risk associated with the aforementioned reliability-based navigation switching mechanism, the multi-navigation mode robot control method in this embodiment further includes: when the robot detects a boundary of a second detection type during edge operation, selecting the next boundary for edge operation based on the current boundary detection type and the type of the detected boundary of the second detection type.

[0093] like Figure 6A As shown, when multiple detection types of boundaries are detected, if the current boundary detection type is an image detection boundary and a magnetic stripe is detected, the robot searches the direction of the magnetic stripe and switches to traveling along the direction of the magnetic stripe; for example... Figure 6B As shown, if the current boundary detection type is magnetic stripe, and an image detection boundary within a specific range centered on the robot is detected, the system switches to traveling along the image detection boundary.

[0094] The switching mechanism provided in this embodiment selects the next boundary for edge operation by comparing the current boundary type of the robot's edge operation with the detected second boundary type. Since there is no need to perform navigation mode reliability judgment, it is simple to implement.

[0095] On the other hand, the image detection boundary and the magnetic strip together limit the robot's working range, improving the robot's safety during operation. For example, if the user improperly sets the magnetic strip, causing it to be located outside the working area, such as when the magnetic strip is set... Figure 7 Even if the robot detects the magnetic strip at points A, C, and E and attempts to walk along the magnetic strip to points A, C, and E outside the working area, the robot will detect the image detection boundary within a specific range during this process, and then switch to working along the image detection boundary, thereby ensuring that the robot will not cross the boundary and always works inside the working area.

[0096] When the robot is searching for the direction of the magnetic strip, if the robot is set to travel along the right edge, it will preferentially search for the direction of the magnetic strip to the left; if the robot is set to travel along the left edge, it will preferentially search for the direction of the magnetic strip to the right.

[0097] By setting the robot's priority search direction for the magnetic strip as described above, we can ensure that the behavior logic is consistent when searching along the magnetic strip and when searching along the edge, and increase the smoothness of the switching.

[0098] Specifically, such as Figure 8AAs shown, when the robot is set to travel along the edge on the right, in the absence of a magnetic strip, the robot performs edge-tracing operations, with its right side moving close to the boundary. Figure 8B As shown, when a magnetic stripe is detected, the robot may preferentially rotate to the left or to the right to search for it. These different preferred search directions will cause the robot to travel along the edge in different directions. It can be seen that if the robot preferentially searches for the magnetic stripe to the left, the direction along the magnetic stripe is the same as the direction along the right side of the robot when there is no magnetic stripe. Similarly, if the robot preferentially rotates to the right to search for the magnetic stripe, the direction along the magnetic stripe is the same as the direction along the left side of the robot when there is no magnetic stripe.

[0099] Maintaining consistent behavior logic when the robot moves along a magnetic strip and along an edge allows the robot to use the same processing logic, reducing system complexity. For example... Figure 9 As shown, when the robot moves along the edge on its right side, the left side 94 of the robot must be the working area, and the right side 93 of the robot must be the non-working area. Furthermore, when the robot moves along the edge on its right side, after completing a full circle, if the resulting trajectory is counter-clockwise, it can be determined that the boundary the robot is following is the outer boundary 91 of the working area; if the resulting trajectory is clockwise, it can be determined that the boundary the robot is following is the boundary 92 of an isolated area within the working area. When the behavior logic along the magnetic strip is consistent with that along the edge, the above judgment can also be used during the process of following the magnetic strip.

[0100] In addition, such as Figure 10 As shown, improper placement of the magnetic stripe by the user may result in it being placed outside the working area, potentially causing the robot to leave the working area or exhibit abnormal behavior due to the system's boundary protection measures. However, since the left side of the robot is always within the working area when it is along the right edge, prioritizing leftward rotation to search the boundary allows the robot to locate the magnetic stripe inside the working area, avoiding potential boundary violations or abnormal behavior caused by prioritizing the search for the outer edge. Similarly, when the robot is always along the left edge, the right side is always within the working area, and prioritizing rightward boundary searching can achieve the same technical effect.

[0101] Example 3

[0102] When a robot is working, it needs to detect the boundaries of the work area, which can be done using boundary recognition modules such as magnetic induction modules and image processing modules. When using a magnetic induction module to identify boundaries and enable the robot to perform edge-following operations along magnetic strips, magnetic strips often need to be deployed in the work area. These magnetic strips may be misidentified as image detection boundaries by the image processing module, thus preventing the robot from properly performing edge-following operations. In other words, in implementing this embodiment, the inventors need to solve the problem of interference between different navigation mechanisms.

[0103] like Figure 12 As shown, when the robot performs edge-following operations along magnetic strip 1, the robot moves in the direction where the peak of the magnetic induction signal is located at the center of the magnetic sensing array. During this process, if the robot recognizes the magnetic guidance device as the image detection boundary, when the robot detects that there is an image detection boundary within a specific range centered on the robot body, it switches to moving along the image detection boundary and uses the front right corner of the robot to follow the nearest boundary point using PID, where PID is an abbreviation for Proportional-Integral-Derivative.

[0104] When the robot identifies the magnetic strip 1 in front of it as the image detection boundary, on the one hand, it may cause the robot to deviate from the user-preset boundary, that is, deviate from the position of the magnetic strip 1; on the other hand, when the robot performs PID following of the boundary point with the right front corner point, it may detect the magnetic strip 1 again and switch to working along the magnetic strip 1, or even repeatedly switch between the two navigation modes of working along the magnetic strip 1 and working along the image detection boundary, causing abnormal robot behavior.

[0105] To address the aforementioned issues, this embodiment proposes a multi-navigation mode robot, comprising: a boundary recognition module on the robot for detecting work area boundary information and recognizing the work area boundary based on the boundary information; a magnetic sensor module for detecting magnetic guidance devices, such as magnetic strips, located near the work area boundary; and a control module for controlling the robot's movement mode based on the recognition result of the boundary recognition module and / or the detection result of the magnetic sensor module. Specifically, when the robot performs edge-guided operations along the magnetically guided boundary, if the boundary recognition module detects an image detection boundary obtained from the real boundary, the robot switches to traveling along the image detection boundary; if the boundary recognition module detects an image detection boundary obtained from the magnetically guided device, the robot continues to travel along the magnetically guided boundary. Further, the area where the magnetically guided device is located is identified as the work area.

[0106] The boundary recognition module includes: a visual sensor module for acquiring a two-dimensional image of the environment; and an image processing module for acquiring the two-dimensional image obtained by the visual sensor module and recognizing the boundary of the working area through an image processing method.

[0107] The process of identifying the working area boundary using image processing methods includes: acquiring a two-dimensional image obtained from a visual sensor, performing semantic segmentation on the obtained two-dimensional image to segment the working area and non-working area in the two-dimensional image, and obtaining the working area boundary based on the boundary between the working area and non-working area.

[0108] The process of semantic segmentation of the obtained two-dimensional image to separate the working region and non-working region in the two-dimensional image includes: constructing a semantic segmentation network; collecting a dataset of images containing the working region, annotating the working region in the image dataset, and training a neural network model using the annotated image dataset; using the trained neural network model to perform semantic segmentation on the two-dimensional image acquired by the robot to obtain a working region mask; and comparing the working region mask with the original image to segment the working region and non-working region in the original image.

[0109] For example, after the image processing module performs semantic segmentation, the robot obtains a working region mask of the same size as the original image, which is used to indicate pixels in the original image that belong to the working region and pixels that belong to the non-working region. By comparing the working region mask with the original image, the working region, non-working region, and working region boundary in the original image can be identified.

[0110] Identifying the area where the magnetic guidance device is located as the working area includes: when annotating the working area in the image dataset, labeling the area where the magnetic guidance device is located as the working area, and using the labeled image dataset to train a neural network model.

[0111] Example 4

[0112] The difference between this embodiment and embodiment 3 is that when the area where the magnetic guidance device is located is identified as the working area, the alternative implementation method to embodiment 3 is, or based on the implementation method of embodiment 3, further includes: performing image closing operation processing on the working area mask obtained by semantic segmentation; comparing the working area mask after image closing operation processing with the original image, and segmenting the working area and non-working area in the original image.

[0113] In some embodiments, when the area where the magnetic guidance device is located is identified as the working area, the method of implementation in embodiment 3 further includes: performing image closing operation processing on the working area mask obtained by semantic segmentation.

[0114] For example, the magnetic guidance device in the training image dataset is labeled as the working area. The labeled image data is used to train a neural network. The trained neural network is then used to segment the images acquired by the robot into working and non-working areas, so as to identify the magnetic strip as the working area.

[0115] Furthermore, when the area where the magnetic guidance device is located is identified as the working area, an image closing operation is performed on the working area mask obtained from semantic segmentation. This image closing operation reduces instability caused by factors such as light reflection when identifying the magnetic strip as the working area.

[0116] In other embodiments, when the area where the magnetic guiding device is located is identified as the working area, instead of marking the area where the magnetic guiding device is located as the working area as described in Embodiment 3, this embodiment only performs image closing operation on the working area mask obtained after semantic segmentation to eliminate holes or gaps caused by magnetic strips in the mask, thereby realizing the identification of the magnetic guiding device as the working area.

[0117] In the implementation of Example 3, although the area where the magnetic guidance device is located can be identified as the working area in most cases, in scenarios such as when the direction of sunlight reflected from the magnetic strip is consistent with the direction of the visual sensor, the brightness of the magnetic strip in the image will be high, resulting in a significant change in its image features. This may cause the neural network model to be unable to stably identify the magnetic guidance device as the working area.

[0118] like Figure 11 As shown, based on the implementation method of Example 3, ideally the magnetic strip should be identified as the lawn, i.e. the working area; in backlight, the magnetic strip is reflective, causing the reflective part of the magnetic strip to be identified as the background, i.e. the non-working area.

[0119] Since the magnetic strip 1 generally occupies a small area in the image, the image closing operation processing in this embodiment can eliminate small holes and gaps caused by the magnetic strip 1 in the working area mask, thereby achieving the technical effect of recognizing the magnetic strip 1 as the working area, avoiding recognizing the magnetic strip as the image detection boundary, and thus solving the problem of mutual interference between the navigation mode based on magnetic guidance boundary and the navigation mode based on image detection boundary.

[0120] Reference Appendix Figure 13A , which is the grassland boundary before the closing operation, so magnetic stripe 1 is identified as the image detection boundary; Figure 13B The grassland boundary after the closing operation is shown, indicating that the influence of magnetic stripe 1 on boundary recognition has been largely eliminated.

[0121] Figure 14A The mask image before the closing operation; Figure 14B The image is a mask image after the closing operation, which shows that the technical effect of recognizing magnetic stripe 1 as the working area has been achieved.

[0122] Example 5

[0123] The difference between this embodiment and embodiment 3 is that, when identifying the area where the magnetic guidance device is located as the working area, in addition to the implementation method of embodiment 3, it further includes: in order to improve the stability of magnetic stripe recognition processing in the image, when training the neural network model, histogram equalization processing is performed on the brightness component of the image dataset, and the neural network is trained using the image dataset after histogram equalization processing to obtain a trained neural network model; when using the trained neural network model to perform image segmentation on the image acquired by the robot, histogram equalization processing is performed on the brightness component of the image acquired by the robot, and the image after equalization processing is input into the neural network model for image segmentation, and the working area boundary is obtained based on the image segmentation.

[0124] Specifically, the image is obtained under the RGB color model and converted to the HSI color model to obtain the image luminance I component. Based on the histogram equalization processing method, the luminance I component of the image is subjected to luminance histogram equalization to obtain equalized HSI color model data. The HSI color model data is then converted to RGB color model data to obtain the equalized image.

[0125] In the implementation of Embodiment 3 above, detecting the working area boundary through the image processing module includes: acquiring a two-dimensional image obtained by the visual sensor module; segmenting the working area and non-working area in the two-dimensional image using a semantic segmentation method; and training neural network parameters using a labeled image dataset to obtain a trained neural network model. During this process, factors such as sunlight reflection may cause the magnetic stripe brightness in the image to be high and image features to change significantly, which may lead to the neural network model being unable to stably identify the magnetic guidance device as the working area.

[0126] In this embodiment, by performing histogram equalization on the brightness components of both the neural network training image dataset and the images acquired by the robot, the brightness distribution of both the training images and the images to be segmented acquired by the robot can be made more uniform. This further makes the overall brightness characteristics of the images used to train the neural network and the real-time images to be segmented by the neural network more consistent, reducing the instability in recognition caused by significant changes in the magnetic stripe image features due to sunlight reflection.

[0127] Example 6

[0128] This embodiment provides a multi-navigation mode robot, including an image processing module for identifying the boundary of the work area; a magnetic sensor module for detecting a magnetic guidance device set near the boundary of the work area or within the work area; and a control module for controlling the robot to move in a first navigation mode or a second navigation mode, and also for controlling the robot to switch between the first navigation mode and the second navigation mode.

[0129] When the robot moves in the first navigation mode, it moves along the boundary of the working area according to the recognition results of the image processing module. The recognition results of the image processing module include the image features of the working area boundary, and further include the image detection boundary obtained based on the image features of the working area boundary.

[0130] When the robot travels in the second navigation mode, it travels along the magnetic guidance device according to the detection results of the magnetic sensor module. The detection results of the magnetic sensor module include the magnetic signal of the magnetic guidance device, and further include the magnetic guidance boundary obtained based on the magnetic signal of the magnetic guidance device.

[0131] For example, the magnetic sensor module includes at least two magnetic sensors symmetrically arranged on both sides of the robot's longitudinal axis. When the robot travels in a first navigation mode, the two magnetic sensors are located on the same side of the work area boundary. When the robot travels in a second navigation mode, in one embodiment, the two magnetic sensors are located on opposite sides of the magnetic guidance device. Furthermore, when the robot travels along a straight magnetic strip, the longitudinal axis and the straight magnetic strip are coplanar in the vertical direction, i.e., their projections on the horizontal plane overlap. In another embodiment, one of the two magnetic sensors is located approximately directly above the magnetic guidance device.

[0132] For example, the magnetic sensor module includes at least one magnetic sensor configured on the longitudinal axis of the robot. When the robot travels in a first navigation mode, the distance from the magnetic sensor to the boundary of the working area is not less than 1 / 4 of the robot's width. When the robot travels in a second navigation mode, the magnetic sensor is positioned approximately directly above the magnetic guidance device.

[0133] When the robot is traveling in the second navigation mode, it switches to the first navigation mode in response to the first recognition result of the image processing module. The first recognition result includes image features of the working area boundary detected by the image processing module within a preset range. The first recognition result may also include image features of the magnetic guidance device detected by the image processing module within the preset range.

[0134] When the robot travels in the second navigation mode, it continues to travel in the second navigation mode in response to the second recognition result of the image processing module. The second recognition result includes image features of the working area boundary that the image processing module did not detect within a preset range, and also includes image features of the magnetic guidance device that the image processing module detected only within the preset range.

[0135] After detecting image features of the magnetic guidance device within a preset range, the image processing module further includes eliminating the image features of the magnetic guidance device using a first image processing method. The first image processing method includes an image closing operation method. The first image processing method also includes a semantic segmentation method, and a process of labeling the image features of the magnetic guidance device as working region image features when constructing a semantic segmentation network.

[0136] like Figure 11 , Figure 13A or Figure 13B As shown, the image processing module also detects image features within a preset working area. It can be understood that the image features of the working area generally form a continuous, closed shape, but there may be voids or other openings within the shape or near its boundary. The image features of the magnetic guidance device include void areas within the shape enclosed by the image features of the working area or near its boundary. For example, the void area of ​​the magnetic guidance device's image features is a long, narrow strip with a width not exceeding 5cm and an aspect ratio not less than 3; or the minimum circumscribed rectangle of the void area has a width not exceeding 5cm and an aspect ratio not less than 3.

[0137] For example, when the robot is traveling in the second navigation mode, the image processing module identifies the boundary of the working area to obtain a first identification result. The first identification result includes image features of the working area boundary detected by the image processing module within a preset range. The first identification result may also include image features of the magnetic guidance device detected by the image processing module within the preset range. After detecting the image features of the magnetic guidance device, the image features of the magnetic guidance device are eliminated using a first image processing method. After eliminating the image features of the magnetic guidance device, the first identification result only includes the image features of the working area boundary. The robot then switches to traveling in the first navigation mode based on the first identification result.

[0138] For example, when the robot travels in the second navigation mode, the image processing module identifies the boundary of the working area to obtain a second identification result. The second identification result includes image features of the working area boundary that the image processing module did not detect within a preset range, and also includes image features of the magnetic guidance device that the image processing module detected only within the preset range. After detecting the image features of the magnetic guidance device, the image features of the magnetic guidance device are eliminated using a first image processing method. After eliminating the image features of the magnetic guidance device, the second identification result does not include image features of the working area boundary, nor does it include image features of the magnetic guidance device. The robot continues to travel in the second navigation mode based on the second identification result.

[0139] For example, when the robot travels in the second navigation mode, the image processing module identifies the boundary of the working area to obtain a first identification result. The first identification result includes image features of the working area boundary detected by the image processing module within a preset range, and further includes an image detection boundary obtained based on the working area boundary image features. The first identification result may also include image features of the magnetic guidance device detected by the image processing module within a preset range, and further includes an image detection boundary obtained based on the magnetic guidance device image features. After detecting the image features of the magnetic guidance device, the image features of the magnetic guidance device are eliminated using a first image processing method. After eliminating the image features of the magnetic guidance device, the first identification result only includes the working area boundary image features or the image detection boundary obtained based on the working area boundary image features. The robot then switches to travel in the first navigation mode based on the first identification result.

[0140] For example, when the robot travels in the second navigation mode, the image processing module identifies the boundary of the working area to obtain a second identification result. The second identification result includes image features of the working area boundary that the image processing module did not detect within a preset range. The second identification result also includes image features of the magnetic guidance device that the image processing module detected only within the preset range, and further includes image detection boundaries obtained due to the image features of the magnetic guidance device. After detecting the image features of the magnetic guidance device, the image features of the magnetic guidance device are eliminated using a first image processing method. After eliminating the image features of the magnetic guidance device, the second identification result does not include image features of the working area boundary or image detection boundaries obtained due to the image features of the working area boundary, nor does it include image features of the magnetic guidance device or image detection boundaries obtained due to the image features of the magnetic guidance device. The robot continues to travel in the second navigation mode based on the second identification result.

[0141] The robot in this disclosure may include one or more of the following components: processor and memory.

[0142] Optionally, the processor connects various parts of the robot using various interfaces and circuits. It executes various robot functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by calling data stored in memory. Optionally, the processor can be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor can integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and Neural-network Processing Unit (NPU). The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content displayed on the touchscreen; and the NPU is used to implement Artificial Intelligence (AI) functions.

[0143] The memory may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the data storage area may store data created based on the use of the robot, etc.

[0144] This disclosure also provides a computer-readable storage medium storing a computer program for execution by a processor to implement the control method for a multi-navigation mode robot as described in the above embodiments.

[0145] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the invention by those skilled in the art without departing from the spirit and essence of the invention. Such modifications or substitutions should all fall within the scope of the invention, or any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the invention should be covered within the protection scope of the invention. Therefore, the protection scope of the invention should be determined by the scope of the claims.

Claims

1. A multi-navigation mode robot, characterized in that, include: The boundary recognition module is used to acquire work area boundary information and identify the work area boundary based on this boundary information. A magnetic sensor module is used to detect magnetic guiding devices located near the boundary of the working area; The control module controls the robot's movement based on the recognition results of the boundary recognition module and / or the detection results of the magnetic sensor module.

2. The robot according to claim 1, characterized in that, The boundary recognition module includes: The vision sensor module is used to acquire two-dimensional images of the environment and environmental depth maps or three-dimensional point cloud maps; The image processing module is used to acquire the two-dimensional image and the environmental depth map or three-dimensional point cloud map of the vision sensor module, and to identify the boundary of the working area through image processing methods.

3. The robot according to claim 2, characterized in that, When detecting the boundary of the working area, the image processing module acquires a two-dimensional image obtained by the vision sensor module, and uses a semantic segmentation method to segment the working area in the two-dimensional image. The boundary of the working area is the working area boundary in the two-dimensional image.

4. The robot according to claim 2, characterized in that, When the image processing module detects the boundary of the working area, the robot can also detect obstacle boundaries based on the height information of the environmental point cloud.

5. The robot according to claim 3, characterized in that, The step of segmenting the working region in the two-dimensional image using a semantic segmentation method includes: constructing a semantic segmentation network; collecting an image dataset containing the working region, at least annotating the working region in the image dataset, and training a neural network model using the annotated image dataset; using the trained neural network model to segment the two-dimensional image acquired by the robot to obtain a working region mask; and comparing the working region mask with the original image to segment the working region in the original image.

6. The robot according to claim 2, characterized in that, The visual sensor module is a binocular camera. Two two-dimensional images are obtained through the two cameras of the binocular camera. The depth of the feature points in the two-dimensional images is calculated to obtain an environmental depth map. The three-dimensional coordinates of the feature points in the robot positioning coordinate system are calculated based on the environmental depth map to obtain the three-dimensional point cloud map.

7. The robot according to claim 2, characterized in that, When detecting the boundary of the working area, the image processing module acquires the environmental depth map or 3D point cloud map from the vision sensor module, and matches the 2D image acquired by the vision sensor module with the environmental depth map or 3D point cloud map to obtain the 3D point cloud corresponding to the boundary of the working area in the 2D image.

8. The robot according to claim 1, characterized in that, The robot detects boundaries of different detection types through the boundary recognition module or the magnetic sensor module. When the robot is working along the edge and detects a boundary of the second detection type, it selects the next boundary to work along the edge based on the detection type of the current boundary and the detection type of the boundary of the second detection type.

9. The robot according to claim 1, characterized in that, The robot detects boundaries of different detection types through the boundary recognition module or the magnetic sensor module. When the robot operates along the edge and detects multiple detection types of boundaries, if the current boundary detection type is an image detection boundary and a magnetically guided boundary is detected, the robot searches for the direction of the magnetically guided device and switches to travel along the magnetically guided boundary. If the current boundary detection type is a magnetically guided boundary and an image detection boundary within a certain range centered on the robot is detected, the robot switches to travel along the image detection boundary.

10. The robot according to claim 1, characterized in that, When the magnetic sensor module searches for the direction of the magnetic guidance device, if the robot is set to the right edge, it will search to the left first; if the robot is set to the left edge, it will search to the right first.

11. The robot according to claim 1, characterized in that, The work area boundary includes the work area range boundary, obstacle boundary, and the boundary of sub-regions or zones within the work area.

12. The robot according to claim 1, characterized in that, The magnetic guiding device includes a magnetic strip, a magnetic marker, a magnetic nail, or an electromagnetically conductive wire.

13. A control method for a multi-navigation mode robot, the robot comprising a boundary recognition module, a magnetic sensor module, and a control module; characterized in that, The method includes: The robot performs tasks in the work area, including performing tasks within the work area and performing tasks along the edge; During the operation within the work area or along the edge, the robot simultaneously detects the boundary of the work area using both a boundary recognition module and a magnetic sensor module.

14. The method according to claim 13, characterized in that, The robot performs tasks within the work area, including tasks within the work area and tasks along the edge, including: Step 1: In response to the user-set lawn mowing schedule time conditions, the robot performs operations within the work area at the set time; Step 2: After certain conditions are met for the operations within the work area, switch to performing edge operations along the boundary of the work area; Step 3: After the edge operation has been performed for a certain period of time or the operation trajectory meets certain conditions, switch back to performing the operation within the work area; Step 4: Repeat steps 2 and 3 above until the task is completed.

15. The method according to claim 13, characterized in that, The robot performs the following tasks within the work area: the robot moves forward in a straight line to perform grass mowing, and when it detects the boundary of the work area, it rotates at a certain angle and moves forward in a straight line again to perform grass mowing.

16. The method according to claim 13, characterized in that, During the process of performing the edge-following operation, the robot also includes: the robot simultaneously detects the boundary through the image processing module and the magnetic sensor module in the boundary recognition module to obtain boundaries of different detection types. When the robot detects only one type of boundary, it performs the edge-following operation along the detected boundary.

17. The method according to claim 13, characterized in that, During the operation of the robot within the work area, the robot also includes: simultaneously detecting boundaries of different detection types through the image processing module and the magnetic sensor module in the boundary recognition module; when the robot detects any type of boundary, it rotates a certain angle and continues to perform the operation in a straight line.

18. The method according to claim 16, characterized in that, When the robot detects only one type of boundary, the edge-following operation along the detected boundary includes: when only an image detection boundary is detected, the robot performs PID following on the nearest boundary point using the robot's right front corner point, and follows the edge using the robot's right-side edge-following method; when only a magnetically guided boundary is detected, the robot rotates to search for the magnetically guided device direction and travels along the direction that makes the peak of the magnetic induction signal at the center of the magnetic sensing array.

19. The method according to claim 13, characterized in that, During the edge operation, the robot simultaneously detects boundaries of different detection types through the image processing module and the magnetic sensor module in the boundary recognition module. When a boundary of the second detection type is detected, the next boundary for the edge operation is selected based on the detection type of the current boundary and the detection type of the boundary of the second detection type.

20. The method according to claim 13, characterized in that, During the edge-following operation, the robot simultaneously detects different types of boundaries through the image processing module and the magnetic sensor module in the boundary recognition module. When multiple types of boundaries are detected, if the current boundary is an image detection boundary and a magnetically guided boundary is detected, the robot searches for the direction of the magnetically guided device and switches to travel along the magnetically guided boundary. If the current boundary is a magnetically guided boundary and an image detection boundary is detected within a certain range centered on the robot, the robot switches to travel along the image detection boundary.

21. The method according to claim 18 or 20, characterized in that, When the robot searches for the direction of the magnetic guidance device, if the robot is set to search along the right edge, it will search to the left first; if the robot is set to search along the left edge, it will search to the right first.

22. The method according to claim 14, characterized in that, Before step 1, the robot is positioned at the charging station in its initial state.

23. The method according to claim 14, characterized in that, After step 3, the process further includes: after performing the work within the work area or the work along the edge to reach a certain power level, the robot searches for the boundary of the work area and returns to the charging station along the boundary of the work area.

24. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement the control method for the multi-navigation mode robot as described in any one of claims 13 to 23.

Images