Map construction method and apparatus, and lawn mowing robot

Abstract

The present application provides a map construction method and apparatus, and a lawn mowing robot. The method comprises: acquiring a depth image and a color image of a first operation area; identifying a target object in the depth image and the color image, and, on the basis of a relationship between a first height of the target object from the ground in the depth image and a preset threshold value and a relationship between first color information of the target object in the color image and preset color information, determining whether the target object is an obstacle; when the target object is an obstacle, acquiring position information of the obstacle; and performing map construction on the basis of the position information of the obstacle. By means of the method, the depth image and the color image of the first operation area can be used to determine whether there is an obstacle in the first operation area, and, when there is an obstacle in the first operation area, map construction is performed, thus completing map construction more efficiently, and facilitating lawn mowing robots to operate on the basis of constructed maps.

Description

Map construction method, device and lawn mowing robot

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on October 16, 2023, with application number 202311345465.6, entitled “Map construction method, device and lawn mowing robot”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of autonomous mobile robots, and more specifically, to a map construction method, device, and lawn mowing robot. Background Art

[0003] Environmental perception is one of the most important functions of autonomous exploration for lawn mower robots, and it is the most important function to ensure that lawn mower robots can operate stably and safely. Existing lawn mower robots can build maps based on environmental perception technology and automatically operate within the designated operating boundaries according to the constructed maps.

[0004] However, during the map construction process, for obstacles within the designated operating boundary, the lawn mowing robot needs to pre-demarcate the restricted areas of the obstacles within the boundary before planning the operating path. If there are many obstacles within the boundary and their shapes are complex, it will cause great trouble to the user, making the operation very troublesome and the user experience poor.

[0005] Summary of the Invention

[0006] The present application provides a map construction method, device and lawn mowing robot, which can complete map construction more efficiently so that the lawn mowing robot can operate according to the constructed map.

[0007] In a first aspect, a map construction method is provided, the method comprising: acquiring a depth image and a color image within a first working area; identifying a target object in the depth image and the color image, and determining whether the target object is an obstacle based on a relationship between a first height of the target object from the ground in the depth image and a preset threshold, and a relationship between first color information of the target object in the color image and preset color information; if the target object is the obstacle, acquiring location information of the obstacle; and constructing a map based on the location information of the obstacle to obtain a global map of the first working area.

[0008] In an embodiment of the present application, the depth image and color image in the first working area can be used to determine whether there are obstacles in the first working area, and a map can be constructed if there are obstacles in the first working area. In this way, obstacles can be identified more quickly and accurately, and map construction can be completed more efficiently, so that the lawn mowing robot can operate according to the constructed map.

[0009] In combination with the first aspect, in certain implementations of the first aspect, determining whether the target object is an obstacle based on the relationship between the first height of the target object from the ground in the depth image and a preset threshold, and the relationship between the first color information of the target object in the color image and the preset color information includes: when the first height is greater than or equal to the preset threshold and the first color information is inconsistent with the preset color information, determining that the target object is the obstacle, or when the first height is greater than or equal to the preset threshold and the first color information is consistent with the preset color information, determining that the target object is not the obstacle.

[0010] In an embodiment of the present application, whether the target object is an obstacle can be judged based on the relationship between the first height of the target object from the ground in the depth image and a preset threshold, and whether the first color information of the target object in the color image is consistent with the preset color information. In this way, obstacles can be identified more quickly and accurately, thereby completing the map construction work more efficiently.

[0011] In combination with the first aspect, in certain implementations of the first aspect, determining whether the target is an obstacle based on a relationship between a first height of the target from the ground in the depth image and a preset threshold, and a relationship between the first color information of the target in the color image and the preset color information, includes: if the first height is greater than or equal to the preset threshold, determining that the pixel value corresponding to the target in the depth image is a first value; if the first height is less than the preset threshold, determining that the pixel value corresponding to the target in the depth image is a second value; if the first color information is inconsistent with the preset color information, determining that the pixel value corresponding to the target in the color image is the first value; if the first color information is consistent with the preset color information, determining that the pixel value corresponding to the target in the color image is the second value; if both the pixel value corresponding to the target in the depth image and the pixel value corresponding to the target in the color image are the first value, determining that the target is the obstacle; or if at least one of the pixel value corresponding to the target in the depth image and the pixel value corresponding to the target in the color image is the second value, determining that the target is not the obstacle.

[0012] Optionally, the first value may be 1, and the second value may be 0.

[0013] In an embodiment of the present application, by using a method of binarizing a depth image and a color image, when the pixel value corresponding to the target object in the depth image and the pixel value corresponding to the target object in the color image are both the first value, the target object can be determined to be an obstacle; otherwise, the target object is determined not to be an obstacle. In this way, obstacles can be identified more quickly and accurately, thereby completing the map construction work more efficiently.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the acquiring of the depth image and the color image within the first working area includes: acquiring an ideal working path for the lawn mower robot to operate within a second working area, the second working area being a working area assuming that there are no obstacles within the first working area, and the ideal working path being a working path determined based on the absence of the obstacles within the second working area; controlling the lawn mower robot to operate within the first working area according to the ideal working path; and acquiring the depth image and the color image within the first working area during the operation of the lawn mower robot.

[0015] In an embodiment of the present application, the lawn mower robot can obtain depth images and color images while operating along the ideal operating path, so that the lawn mower robot can determine whether there are obstacles in the first operating area. In this way, it is possible to avoid defining restricted areas for obstacles in advance and plan the actual operating path of the lawn mower robot, thereby improving the efficiency of map construction.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the map construction based on the location information of the obstacle includes: determining a real working path based on the location information of the obstacle and the ideal working path, wherein the obstacle does not exist in the real working path; and controlling the lawn mowing robot to operate within the second working area based on the real working path.

[0017] In an embodiment of the present application, the lawn mowing robot can update the ideal working path according to the location information of the obstacle to obtain the real working path, so that the lawn mowing robot can avoid obstacles during operation without the need for additional user manipulation, which is conducive to improving the user experience.

[0018] In combination with the first aspect, in certain implementations of the first aspect, obtaining the location information of the obstacle includes: obtaining contour information of the obstacle, the contour information of the obstacle including second point cloud data; sampling the contour information of the obstacle to obtain third point cloud data, the point cloud sparsity in the third point cloud data being lower than the point cloud sparsity in the second point cloud data; and determining the location information of the obstacle based on the third point cloud data.

[0019] In an embodiment of the present application, when the contour information of the obstacle includes second point cloud data, the lawn mower robot can sample the contour information of the obstacle to obtain relatively sparse third point cloud data, and obtain the position information of the obstacle based on the third point cloud data. In this way, the amount of calculation can be reduced and the efficiency of the lawn mower robot in obtaining the position information of the obstacle can be improved.

[0020] In a second aspect, a map construction device is provided, which includes: an acquisition unit for acquiring a depth image and a color image within a first working area; a processing unit for: identifying a target object in the depth image and the color image, and determining whether the target object is an obstacle based on a relationship between a first height of the target object from the ground in the depth image and a preset threshold, and a relationship between first color information of the target object in the color image and preset color information; if the target object is the obstacle, acquiring position information of the obstacle; and constructing a map based on the position information of the obstacle to obtain a global map of the first working area.

[0021] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a program code, and when the computer program code is run on a computer, the computer executes the method in any one of the implementation modes of the first aspect.

[0022] In a fourth aspect, a lawn mowing robot is provided, comprising an acquisition device, a main body and a main control chip, wherein: the acquisition device is used to acquire depth images and color images within a first working area of ​​the lawn mowing robot; the main control chip is used to process the depth image and color image according to the method in any one of the implementation methods of the above-mentioned first aspect to obtain a global map of the first working area. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG1 is a schematic diagram of a smart device provided in an embodiment of the present application;

[0024] FIG2 is an application scenario diagram of the map construction method provided in an embodiment of the present application;

[0025] FIG3 is a schematic flow chart of a map construction method provided in an embodiment of the present application;

[0026] FIG4 is a schematic flow chart of another map construction method provided in an embodiment of the present application;

[0027] FIG5 is a schematic diagram of obstacle classification provided by an embodiment of the present application;

[0028] FIG6 is a schematic diagram of a point cloud data detection method provided in an embodiment of the present application;

[0029] FIG7 is a schematic diagram of a scenario in which the obstacle provided by an embodiment of the present application is a trampoline;

[0030] FIG8 is a scene diagram of a lawn mowing robot in operation provided by an embodiment of the present application;

[0031] FIG9 is a map construction device provided in an embodiment of the present application;

[0032] FIG10 is another map construction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The technical solution in this application will be described below with reference to the accompanying drawings.

[0034] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In this application, "at least one" refers to one or more, and "more than one" refers to two or more. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0035] In the embodiments of this application, prefixes such as "first" and "second" are used only to distinguish different description objects and have no limiting effect on the position, order, priority, quantity, or content of the described objects. The use of prefixes such as ordinal numbers in the embodiments of this application to distinguish description objects does not constitute a limitation on the described objects. For a statement of the described objects, please refer to the description in the context of the claims or embodiments, and the use of such prefixes should not constitute an unnecessary limitation.

[0036] FIG1 is a functional diagram of a smart device 100 provided in an embodiment of the present application. It should be understood that FIG1 and the related description are merely examples and do not limit the smart device in the embodiment of the present application.

[0037] The smart device 100 may include multiple subsystems, such as a sensing system 120 and a computing platform 130. Optionally, the smart device 100 may include more or fewer subsystems, and each subsystem may include one or more components. In addition, each subsystem and component of the smart device 100 may be interconnected via wired or wireless means.

[0038] The perception system 120 may include several sensors for sensing information about the environment surrounding the smart device 100. For example, the perception system 120 may include a positioning system, which may be a global positioning system (GPS), a Beidou system, or other positioning systems. The perception system 120 may include one or more of an inertial measurement unit (IMU), a laser radar, a millimeter-wave radar, an ultrasonic radar, and a camera.

[0039] Some or all functions of the smart device 100 can be controlled by the computing platform 130. The computing platform 130 may include processors 131 to 13n (n is a positive integer). A processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), or a digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationship of a hardware circuit. The logical relationship of the hardware circuit is fixed or reconfigurable. For example, the processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration file to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, the processor may also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. In addition, the computing platform 130 may also include a memory for storing instructions, and some or all of the processors 131 to 13n may call the instructions in the memory to implement corresponding functions.

[0040] The computing platform 130 can control the functionality of the smart device 100 based on input received from various subsystems, such as the sensing system 120. In some embodiments, the computing platform 130 can be used to provide control over many aspects of the smart device 100 and its subsystems.

[0041] Optionally, the above components are only an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 1 should not be understood as a limitation to the embodiments of the present application.

[0042] The smart device 100 in this application may include: road vehicles, water vehicles, air vehicles, industrial equipment, agricultural equipment, or entertainment equipment, etc. For example, the smart device can be a vehicle, which is a vehicle in a broad sense, and can be a transportation tool (such as a commercial vehicle, a passenger car, a motorcycle, a flying car, a train, etc.), an industrial vehicle (such as a forklift, a trailer, a tractor, etc.), an engineering vehicle (such as an excavator, a bulldozer, a crane, etc.), agricultural equipment (such as a lawn mower, a harvester, etc.), amusement equipment, a toy vehicle, etc. The embodiments of this application do not specifically limit the type of vehicle. For another example, the smart device can be a robot, including a lawn mowing robot, etc.

[0043] When the smart device is a lawn mower robot, environmental perception is one of the most important functions of its autonomous exploration and the most important function to ensure that the lawn mower robot can operate stably and safely. Existing lawn mower robots can build maps based on environmental perception technology and automatically operate within the designated operating boundaries according to the constructed maps.

[0044] However, during the map construction process, for obstacles within the designated operating boundary, the lawn mowing robot needs to pre-demarcate the restricted areas of the obstacles within the boundary before planning the operating path. If there are many obstacles within the boundary and their shapes are complex, it will cause great trouble to the user, making the operation very troublesome and the user experience poor.

[0045] The embodiments of the present application provide a map construction method, device, and lawn mowing robot, which can more efficiently complete map construction so that the lawn mowing robot can operate according to the constructed map.

[0046] Figure 2 illustrates an application scenario of the map construction method provided by an embodiment of the present application. In one embodiment, a lawn mower robot includes a main body, an independently operable acquisition device, and a main control chip. Before controlling the main body of the lawn mower robot to operate along a preset path within its work area, it must first obtain a map of the work area and its work boundary. Specifically, a user can hand-hold or remotely control the acquisition device on the lawn mower robot to move along the boundary of the robot's work area and scan to obtain multiple frames of partial images or multiple frames of partial images and the motion parameters of the acquisition device when each frame of partial image is acquired. The main control chip processes each frame of partial image and the motion parameters of the acquisition device when each frame of partial image is acquired to obtain an initial pose for each frame of partial image. Keyframes are selected based on parallax and other conditions, and a global optimization is performed on all keyframes to obtain a global map of the lawn mower robot's work area and the optimized poses of each keyframe that constitutes the map. The pose of each keyframe corresponds to the user's movement trajectory, and the poses of each keyframe are aggregated to form the work boundary of the lawn mower robot's work area. The lawn mower robot shown in Figure 2 may be the smart device 100 in Figure 1 , the acquisition device may be located in the perception system 120 , and the main control chip may be located in the computing platform 130 .

[0047] After obtaining the working boundary, the map construction method provided in accordance with one or more embodiments of the present application can process the working boundary of the robot mower's working area to obtain an ideal movement path, thereby controlling the robot mower to move along the ideal movement path. Simultaneously, the acquisition device can capture image data within the working area during movement, identify the locations of obstacles within the working area based on the image data, and use the obstacle location information to update the ideal movement path to obtain a true movement path, thereby facilitating subsequent control of the robot mower to perform mowing operations along the true movement path.

[0048] It should be noted that the ideal moving path in this application may refer to the working trajectory generated when there are no obstacles in the working area of ​​the lawn mower, while the actual moving path may be the working trajectory obtained by taking into account information such as the existence of internal obstacles in the working area of ​​the lawn mower. Internal obstacles may include: outdoor sofas, benches, flower beds, trees, etc. that are common on home lawns.

[0049] In one embodiment, the acquisition device may include a depth camera and a color camera, wherein the depth camera is used to acquire depth images, and the color camera is used to acquire color images. It should be noted that when a user holds or remotely controls the acquisition device to acquire image data to construct a work boundary, the depth camera and / or color camera may be used to move along the boundary of the work area and acquire multiple frames of local depth images and / or multiple frames of color images to obtain the work boundary of the mowing robot. Furthermore, in the process of acquiring the actual movement path of the mowing robot, the depth images and color images acquired by the depth camera and the color camera, respectively, need to be processed to identify obstacles within the work area.

[0050] Furthermore, when the acquisition device includes a depth camera and a color camera, the images captured by the depth camera and the color camera need to be aligned so that at least some pixels in the depth image captured by the depth camera have a one-to-one correspondence with at least some pixels in the color image captured by the color camera.

[0051] In one embodiment, the depth camera may include a binocular depth camera, or a monocular depth camera and a motion parameter recording module (generally referred to as an IMU), or a binocular depth camera and a motion parameter recording module, wherein the monocular / binocular depth camera is used to collect image data, and the motion parameter recording module is used to record the corresponding motion parameters of the camera when the camera collects each frame of image (IMU data, recording the angular velocity and linear acceleration of the acquisition device, etc.). The acquisition device can work independently and can have its own storage and processing chip. When the acquisition device is not detachable from the lawn mower robot, the acquisition device on the mobile terminal can also be used to collect data from the working area of ​​the lawn mower robot.

[0052] Furthermore, the monocular depth camera can be a structured light depth camera, a time-of-flight depth camera, etc., and the binocular depth camera can be an active binocular depth camera or a passive binocular depth camera, which is not limited in this application.

[0053] It should be noted that whether the acquisition device includes a motion parameter recording module depends on whether the acquisition device can directly obtain the absolute scale, that is, the scale relationship between the actual size of the scene captured by the acquisition device and the size of the scene in the image captured including the scene; for a binocular depth camera, it has a baseline, and the distance between the baselines is known, which can be used as reference information to directly obtain the absolute scale of the acquisition device. For a monocular depth camera, it cannot directly obtain the absolute scale, and it needs to be assisted by the motion parameter recording module to determine it.

[0054] In one embodiment, when the acquisition device used on the mobile terminal has computing power, it can process the data collected by the acquisition device to output the initial posture of each frame of local image, and transmit each frame of local image and its initial posture to the main control chip of the lawn mowing robot via wired or wireless means for processing to obtain a global map and working boundary of the working area of ​​the lawn mowing robot.

[0055] In another embodiment, when the acquisition device used on the mobile terminal does not have computing power, each frame of local image or each frame of local image and the motion parameters corresponding to each frame of local image acquired by the acquisition device on the mobile terminal can be transmitted to the main control chip on the lawn mowing robot via wired or wireless means for processing to obtain a global map and working boundary of the working area of ​​the lawn mowing robot.

[0056] It should be noted that, in addition to cameras and motion parameter recording modules, the acquisition equipment in this application can also be GPS / real-time kinematic (RTK) / ultra-wideband (UWB) / lidar and IMU. The acquisition equipment may not even include a camera or IMU, such as only including a lidar, and this application does not impose any restrictions on this.

[0057] FIG3 is a schematic flow chart of a map construction method provided in an embodiment of the present application. Method 300 may include steps S301 to S304:

[0058] S301: Acquire a depth image and a color image within a first operating area.

[0059] Optionally, the depth image and the color image may be acquired by a collection device deployed on the lawn mowing robot, and the collection device may include: a depth camera and a color camera.

[0060] In one embodiment, step S301 specifically includes: obtaining an ideal operating path for the mowing robot to operate within a second operating area, where the second operating area is an ideal working area assuming no obstacles in the first operating area. The ideal operating path is determined based on the absence of obstacles in the second operating area; and controlling the mowing robot's acquisition device to operate within the first operating area based on the ideal operating path, thereby acquiring a depth image and a color image of the first operating area. Thus, the mowing robot can acquire depth and color images while operating along the ideal operating path to facilitate determining whether there are obstacles in the first operating area. This avoids the need to pre-define no-go zones in order to plan the mowing robot's actual operating path, thereby improving map construction efficiency.

[0061] Furthermore, the mowing robot can obtain the working boundary of the second working area and determine an ideal working path based on the working boundary; wherein the ideal working path is preferably a bow-shaped working path.

[0062] S302, identifying the target object in the depth image and the color image, and determining whether the target object is an obstacle based on the relationship between the first height of the target object from the ground in the depth image and a preset threshold, and the relationship between the first color information of the target object in the color image and the preset color information.

[0063] In one embodiment, step S302 may specifically include: when the first height is greater than or equal to a preset threshold and the first color information is inconsistent with the preset color information, determining that the target object is an obstacle; or when the first height is greater than or equal to a preset threshold and the first color information is consistent with the preset color information, determining that the target object is not an obstacle (such as a trampoline or a slope).

[0064] It should be understood that the preset threshold value can be determined according to the height of the collection device relative to the ground when it is installed on the lawn mowing robot, and this application does not impose any limitation on this.

[0065] Optionally, the first height may be determined based on the coordinate value of the first point cloud data corresponding to the target object in the depth image. Further, the first height may be the z value in the coordinate (x, y, z) of the first point cloud data.

[0066] Optionally, in a lawn mowing scenario, the preset color information may be green. When the first color information of the target object is inconsistent with green, the target object is determined to be an obstacle; otherwise, the target object is determined not to be an obstacle.

[0067] In one embodiment, when it is determined that the first height of the target object is less than a preset threshold, it can be determined that the target object is not an obstacle.

[0068] In one embodiment, step S302 may specifically include: when the first height is greater than or equal to a preset threshold, determining that the pixel value corresponding to the target object in the depth image is a first value; when the first height is less than the preset threshold, determining that the pixel value corresponding to the target object in the depth image is a second value; when the first color information is inconsistent with the preset color information, determining that the pixel value corresponding to the target object in the color image is a first value; when the first color information is consistent with the preset color information, determining that the pixel value corresponding to the target object in the color image is a second value; when the pixel value corresponding to the target object in the depth image and the pixel value corresponding to the target object in the color image are both first values, determining that the target object is an obstacle, or when at least one of the pixel value corresponding to the target object in the depth image and the pixel value corresponding to the target object in the color image is the second value, determining that the target object is not an obstacle.

[0069] Optionally, the first value may be 1, and the second value may be 0. For example, when the pixel value corresponding to the target object in the depth image and the pixel value corresponding to the target object in the color image are both 1, the target object is determined to be an obstacle. For another example, when at least one of the pixel value corresponding to the target object in the depth image and the pixel value corresponding to the target object in the color image is 0, the target object is determined not to be an obstacle.

[0070] S303: When the target object is an obstacle, obtain the position information of the obstacle.

[0071] In one embodiment, step S303 specifically includes: obtaining obstacle contour information, the obstacle contour information including second point cloud data; upsampling the obstacle contour information to obtain third point cloud data, wherein the point cloud sparsity in the third point cloud data is lower than that in the second point cloud data; and determining the obstacle location information based on the third point cloud data. Upsampling the second point cloud data can reduce computational complexity and improve the efficiency of the lawn mower robot in obtaining obstacle location information.

[0072] S304: construct a map based on the location information of the obstacles to obtain a global map of the first operating area.

[0073] Specifically, multiple frames of color images and / or multiple frames of depth images obtained in the first operating area are spliced ​​to obtain a map of the first operating area, and the map is updated according to obstacle information to obtain a global map of the first operating area.

[0074] In one embodiment, method 304 further includes: assigning a time tag to each obstacle based on its residence time, with the time tag being either permanent, semi-permanent, or temporary. Temporary obstacles can be understood as obstacles that the mowing robot can remove at any time during operation. This allows the mowing robot to monitor only the presence of temporary obstacles during operation and dynamically update the global map of the first operating area in real time based on the obstacle's time tag, eliminating the need to repeatedly acquire obstacle information, thereby improving the user experience.

[0075] In one embodiment, the process following step S304 includes determining a real operating path based on the obstacle location information and the ideal operating path, wherein the real operating path contains no obstacles; and controlling the mowing robot to operate within the first operating area based on the real operating path. In this way, the mowing robot can update the ideal operating path based on the obstacle location information to obtain the real operating path, allowing the mowing robot to avoid obstacles during operation without requiring additional user control, thereby improving the user experience.

[0076] In an embodiment of the present application, the depth image and color image in the first working area can be used to determine whether there are obstacles in the first working area, and a map can be constructed if there are obstacles in the first working area. In this way, obstacles can be identified more quickly and accurately, and map construction can be completed more efficiently, so that the lawn mowing robot can operate according to the constructed map.

[0077] It should be understood that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced by each other, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0078] FIG4 is a schematic flow chart of another map construction method provided in an embodiment of the present application. Method 400 may be a specific description of steps 301 to 304 in method 300. Method 400 may include the following steps:

[0079] S401 , obtaining a working boundary of the lawn mowing robot, and generating an ideal moving path of the lawn mowing robot when working within the working boundary based on the working boundary.

[0080] In one embodiment, when a lawn mower robot needs to mow within a specified work area, it can first obtain the work boundary corresponding to the work area and generate an ideal movement path based on the work boundary. This ideal movement path can be a work trajectory generated when there are no obstacles within the lawn mower's work area. Furthermore, the ideal movement path is preferably a bow-shaped pattern, as shown in Figure 2. Compared to moving in a single direction, the bow-shaped pattern allows the lawn mower robot to move without stopping when reaching the work area boundary, achieving a continuous motion scanning effect, thereby saving time in acquiring image data. The obstacle-free work area corresponding to the work boundary can be the second work area in method 300.

[0081] S402 , controlling the mowing robot to move along an ideal moving path, and synchronously actuating the acquisition device to acquire images of a portion of the area within the working boundary at each scanning point during the movement to obtain corresponding depth images and color images.

[0082] Specifically, the ideal movement trajectory can be understood as consisting of multiple scanning points. When the mowing robot moves along the ideal movement trajectory, the acquisition device is synchronously actuated to scan the partial area within the working boundary during the movement at each scanning point to obtain corresponding depth images and color images. It should be noted that the acquisition device can obtain one or more frames of images of the corresponding area at each scanning point. When multiple frames of images are obtained, the multiple frames of images can be fused to obtain a high-precision image corresponding to the current area, thereby improving the accuracy of path planning. This application does not impose any restrictions on this. Among them, the partial area within the above-mentioned working boundary can be the first working area in method 300.

[0083] S403: Use the depth image and the color image to identify whether there is an obstacle in the area corresponding to the current scanning point.

[0084] Optionally, if an obstacle is identified in the area corresponding to the current scanning point using the depth image and the color image, the acquisition device can be controlled to acquire the outline information and the location information of the obstacle. In step S403, there are various ways to identify whether an obstacle exists in the area corresponding to the current scanning point using the depth image and the color image.

[0085] In one embodiment, step S403 specifically includes: using the depth image to identify whether there is a foreign object in the area corresponding to the current scanning point; if there is a foreign object, combining the synchronously collected color image to identify whether the current foreign object is an obstacle.

[0086] Specifically, the depth image can be converted into point cloud data to determine whether the coordinate values ​​of some points in the point cloud data are higher than a preset height threshold. If so, the target object corresponding to this part of the point cloud is regarded as a foreign object. Otherwise, the current area is divided into a grass area or a non-grass area based on the color information of the corresponding color image collected simultaneously. The grass area can be a working area where the mowing robot can perform mowing actions, and the non-grass area can be a working area where the mowing robot cannot perform mowing actions (for example, an aisle). The non-grass area can be regarded as an obstacle. For the non-grass area, the acquisition device needs to be controlled to collect the contour information of the non-grass area to perceive the location information of the obstacle, so that the mowing robot can avoid the area when performing subsequent mowing operations. The above embodiment can correspond to the method of distinguishing obstacles by color in Figure 5.

[0087] Optionally, the preset color information of the grass area may be green, and the preset color information of the non-grass area may be non-green.

[0088] It should be noted that the preset height threshold can be determined by the height of the acquisition device relative to the ground when it is mounted on the robotic mower. This preset height threshold can be the preset threshold in method 300. When the acquisition device is mounted on the robotic mower, the horizontal plane on which the acquisition device resides can be used as a reference, and the infinite distance region corresponding to its field of view can form a plane. Objects corresponding to point clouds above this plane are considered foreign objects. As shown in Figure 6, point cloud #17037 is located above the plane formed by the infinite distance region corresponding to the field of view of the acquisition device. Therefore, the object corresponding to point cloud #17037 can be considered a foreign object.

[0089] Furthermore, if the depth image is used to identify the presence of a foreign object in the area corresponding to the current scanning point, the simultaneously acquired color image is then combined to identify whether the current foreign object is a geometric obstacle. This includes: obtaining the area in the color image corresponding to the area where the foreign object is located in the depth image based on the alignment relationship between the current depth image and the color image; extracting the color information of the area in the color image, and determining whether the current foreign object is a geometric obstacle based on the color information. Specifically, since the mowing robot mows on a grassy area, the mowing robot's working area can be divided into a grassy area and an obstacle area based on the color information of the color image. If the area in the color image where the foreign object detected based on the depth image is located is a grassy area, then the current foreign object is not a geometric obstacle, and the mowing robot can perform mowing in that area. If the area in the color image where the foreign object detected based on the depth image is located is an obstacle area, then the current foreign object is a geometric obstacle, and the acquisition device can be controlled to collect the obstacle's contour information to perceive the location information of the geometric obstacle.

[0090] In one embodiment, step S403 specifically includes: performing obstacle judgment on the depth image and the color image through binarization, and then identifying whether there is an obstacle in the area corresponding to the current scanning point.

[0091] Specifically, in the process of using a depth image to judge foreign objects, the depth image can be binarized based on the comparison relationship between the target object and a preset height threshold. That is, if the height of the target object is higher than the preset height threshold, there is a foreign object in the depth image, and the pixels in the area where the foreign object is located are assigned 1, and the pixels in the area where the foreign object is not located are assigned 0, so as to obtain a depth binary image; in the process of using a color image to judge foreign objects, the color image is binarized based on the color information of the color image. If the target object is judged to be a foreign object based on the color information of the color image, the pixels in the area where the foreign object is located are assigned 1, and the pixels in the area where the foreign object is not located are assigned 0, so as to obtain a color binary image; a logical AND operation is performed on the depth binary image and the color binary image. If the operation result is 1, it indicates that there is an obstacle in the current area. Otherwise, if the operation result is 0, it indicates that there is no obstacle in the current area. The above embodiment can correspond to the method of distinguishing obstacles using geometric structures in Figure 5. In the above binarization method, 1 can be the first value in method 300, and 0 can be the second value in method 300.

[0092] In one embodiment, as shown in FIG7 , when an overhead recreational facility (e.g., a trampoline supported by multiple support poles) or a slope exists within the working area of ​​the mowing robot, the mowing robot can still perform mowing. In this case, if only a depth camera is used to detect obstacles, the depth camera can identify the overhead recreational facility or slope as an obstacle based on the detected support pole height or slope height being greater than a preset height threshold. The area where the obstacle is located can be considered a non-grass area, thereby controlling the mowing robot to avoid the area and perform mowing in another area. However, in reality, the mowing robot can perform mowing by moving around the bottom area of ​​the overhead recreational facility or along the slope. Therefore, to avoid the aforementioned misjudgment, the embodiment of the present application can, under the premise that the depth camera detects a foreign object, combine the color information corresponding to the area where the overhead recreational facility is located in the color image to determine whether the area corresponding to the overhead recreational facility or slope is a grass area. If so, the area where the overhead recreational facility or slope is located detected by the depth camera can be segmented and used as the mowing robot's executable area. Otherwise, the acquisition device can be controlled to collect the obstacle's contour information to perceive the obstacle's location information.

[0093] Optionally, once obstacles are detected, they can be considered dynamic and time-stamped based on their duration. Time tags can be categorized as permanent, semi-permanent, or temporary. Permanent obstacles, for example, flower beds and trees, remain in place during multiple robot operations. Semi-permanent obstacles, for example, benches and coffee tables, are present during the robot's current operation but absent during the next. Temporary obstacles, for example, animals and people, can be removed at any time during the robot's current operation. This implementation corresponds to the method for distinguishing obstacles based on duration of presence in Figure 5.

[0094] Among them, for permanent or semi-permanent obstacles, in the process of constructing a global map, controlling the acquisition device to collect the outline information of the obstacle to perceive the position information of the obstacle includes: when the lawn mowing robot moves according to the ideal moving trajectory to collect the global map of the working area, if an obstacle is detected, controlling the acquisition device to rotate and move along the edge contour of the obstacle to collect multiple frames of local images of the obstacle, splicing the multiple frames of local images of the obstacle to obtain the obstacle outline information, and obtaining the obstacle position information based on the obstacle outline information, thereby updating the obtained obstacle position information to the high-precision map corresponding to the current area.

[0095] Furthermore, temporary obstacles can be removed at any time during the construction of the global map. Therefore, under normal circumstances, there is no need to obtain the obstacle's location information and update the global map. However, the obstacle's location information can be used to update the map while the mowing robot is performing its mowing operation. Specifically, when the mowing robot detects that the obstacle is a temporary obstacle for the first time during the mowing operation, the mowing robot can adopt an avoidance strategy and turn around to perform the mowing operation on the next planned path. During the execution process, if the temporary obstacle is removed, the mowing robot does not need to update the map when it returns to the charging station for charging after the previous operation. Conversely, if the temporary obstacle is not removed, the mowing robot needs to obtain the obstacle's location information and update the map when it returns to the charging station for charging after the current operation.

[0096] Optionally, when the obstacle contour information is the point cloud data of the obstacle, in order to reduce the amount of calculation, the obstacle contour information can also be upsampled to obtain a sparse point cloud, and the map corresponding to the current area can be updated based on the sparse point cloud of the obstacle to update the size and position of the obstacle to the global map.

[0097] The point cloud data corresponding to the outline information of the obstacle may be the second point cloud data in method 300 , and the sparse point cloud may be the third point cloud data in method 300 .

[0098] S404: updating the ideal moving path based on the position information of the obstacle to obtain the actual moving path of the mowing robot when mowing.

[0099] Specifically, as shown in FIG8 , the mowing robot can move step by step according to the ideal moving path to collect multiple frames of local images in the working area, perceive obstacles in the current area based on the currently collected local images, and update the local map obtained based on the historically collected images, thereby obtaining a global map including complete obstacles, so as to determine the actual moving path and perform operations based on the actual moving path.

[0100] In the embodiment of the present application, by combining depth images and color images, it is possible to more accurately determine whether there are obstacles in the working area of ​​the lawn mower robot. If there are obstacles, the ideal moving path can be updated based on the position information of the obstacles to obtain the actual moving path of the lawn mower robot, so that the lawn mower robot can avoid obstacles when performing mowing operations without the need for additional user manipulation, which is conducive to improving user experience.

[0101] It should be understood that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced by each other, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0102] An embodiment of the present application also provides a device for implementing any of the above methods, wherein the device includes a unit for implementing each step performed by the lawn mowing robot in any of the above methods.

[0103] Figure 9 is a schematic diagram of a map construction device 900 provided in an embodiment of the present application. Device 900 may include an acquisition unit 910, a storage unit 920, and a processing unit 930. Acquisition unit 910 is used to acquire instructions and / or data, while storage unit 920 is used to implement corresponding storage functions and store the corresponding instructions and / or data. Acquisition unit 910 may also be referred to as a communication interface or communication unit. Processing unit 930 is used to perform data processing to enable device 900 to implement the aforementioned map construction method.

[0104] The device 900 includes: an acquisition unit 910, used to acquire a depth image and a color image within a first working area; a processing unit 930, used to: identify a target object in the depth image and the color image, and determine whether the target object is an obstacle based on the relationship between a first height of the target object from the ground in the depth image and a preset threshold, and the relationship between the first color information of the target object in the color image and the preset color information; if the target object is an obstacle, obtain the location information of the obstacle; and construct a map based on the location information of the obstacle to obtain a global map of the first working area.

[0105] Optionally, if the apparatus 900 is located in the smart device 100 , the processing unit 930 may be the processor 131 shown in FIG. 1 .

[0106] Figure 10 is a schematic diagram of another map construction device 1000 provided in an embodiment of the present application. This device 1000 can be used in the smart device 100 of Figure 1. This device 1000 includes a memory 1010, a processor 1020, and a communication interface 1030. The memory 1010, processor 1020, and communication interface 1030 are interconnected via an internal connection path. The memory 1010 is used to store instructions, and the processor 1020 is used to execute the instructions stored in the memory 1010 to control the communication interface 1030 to obtain information or enable the device 1000 to implement the aforementioned map construction method. Optionally, the memory 1010 can be coupled to the processor 1020 via an interface or integrated with the processor 1020.

[0107] It should be noted that the communication interface 1030 may be a transceiver such as, but not limited to, a transceiver. The communication interface 1030 may also include an input / output interface.

[0108] The processor 1020 stores one or more computer programs, which include instructions. When the instructions are executed by the processor 1020, the apparatus 1000 executes the map construction method in each of the above embodiments.

[0109] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 1020 or by instructions in the form of software. The method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1010, and the processor 1020 reads the information in the memory 1010 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0110] Optionally, the communication interface 1030 in FIG. 10 may implement the acquisition unit 910 in FIG. 9 , the memory 1010 in FIG. 10 may implement the storage unit 920 in FIG. 9 , and the processor 1020 in FIG. 10 may implement the processing unit 930 in FIG. 9 .

[0111] Optionally, the device 900 or the device 1000 may be located in the lawn mowing robot in FIG. 1 .

[0112] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program code. When the computer program code is executed on a computer, the computer executes any one of the methods in FIG. 3 to FIG. 8 .

[0113] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes any one of the methods in Figures 3 to 8 above.

[0114] An embodiment of the present application further provides a chip, comprising: a circuit, wherein the circuit is used to execute any one of the methods in FIG. 3 to FIG. 8 above.

[0115] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0116] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0117] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0118] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0119] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0120] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0121] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A map construction method, characterized in that: The method comprises: Acquire a depth image and a color image within a first operating area; Identify the target in the depth image and the color image, and determine whether the target is an obstacle based on a relationship between a first height of the target from the ground in the depth image and a preset threshold, and a relationship between first color information of the target in the color image and preset color information; When the target object is the obstacle, obtaining position information of the obstacle; A map is constructed according to the location information of the obstacles to obtain a global map of the first operating area.

2. The method according to claim 1, characterized in that The determining whether the target object is an obstacle according to a relationship between a first height of the target object from the ground in the depth image and a preset threshold, and a relationship between first color information of the target object in the color image and preset color information, includes: When the first height is greater than or equal to the preset threshold and the first color information is inconsistent with the preset color information, the target object is determined to be the obstacle, or When the first height is greater than or equal to the preset threshold and the first color information is consistent with the preset color information, it is determined that the target object is not the obstacle.

3. The method according to claim 1, characterized in that The determining whether the target object is an obstacle according to a relationship between a first height of the target object from the ground in the depth image and a preset threshold, and a relationship between first color information of the target object in the color image and preset color information, includes: When the first height is greater than or equal to the preset threshold, determine that the pixel value corresponding to the target object in the depth image is a first value; when the first height is less than the preset threshold, determine that the pixel value corresponding to the target object in the depth image is a second value; When the first color information is inconsistent with the preset color information, the pixel value corresponding to the target object in the color image is determined to be the first value; when the first color information is consistent with the preset color information, the pixel value corresponding to the target object in the color image is determined to be the second value; When the pixel value corresponding to the target object in the depth image and the pixel value corresponding to the target object in the color image are both the first value, the target object is determined to be the obstacle, or When at least one of a pixel value corresponding to the target object in the depth image and a pixel value corresponding to the target object in the color image is the second value, it is determined that the target object is not the obstacle.

4. The method according to any one of claims 1 to 3, characterized in that Before constructing a map according to the location information of the obstacle, the method further includes: A time tag is added to each obstacle according to the length of time the obstacle stays, and the time tag includes permanent existence, semi-permanent existence or temporary existence.

5. The method according to any one of claims 1 to 4, characterized in that: The acquiring of the depth image and the color image in the first operating area includes: Acquire an ideal working path for the mowing robot to operate in a second working area, where the second working area is a working area assuming that no obstacles exist in the first working area, and the ideal working path is a working path determined based on the absence of the obstacles in the second working area; According to the ideal operation path, controlling the lawn mowing robot to operate in the first operation area; During the operation of the lawn mowing robot, a depth image and a color image in the first operation area are acquired.

6. The method according to claim 5, characterized in that The step of constructing a map according to the location information of the obstacle to obtain a global map of the first operating area includes: Constructing a map according to the location information of the obstacle and the ideal operation path to obtain a global map of the first operation area; Determine a real working path of the first working area based on the global map, where the obstacle does not exist in the real working path; According to the real operation path, the mowing robot is controlled to perform mowing operation in the first operation area.

7. The method according to any one of claims 1 to 6, characterized in that: The obtaining the position information of the obstacle includes: Acquiring contour information of the obstacle, wherein the contour information of the obstacle includes second point cloud data; Sampling the contour information of the obstacle to obtain third point cloud data, wherein the point cloud sparsity in the third point cloud data is lower than the point cloud sparsity in the second point cloud data; The position information of the obstacle is determined according to the third point cloud data.

8. A map construction device, characterized in that: The device comprises: An acquisition unit, used for acquiring a depth image and a color image in a first operation area; A processing unit is used to identify the target in the depth image and the color image, and determine whether the target is an obstacle based on the relationship between a first height of the target from the ground in the depth image and a preset threshold, and the relationship between the first color information of the target in the color image and the preset color information; if the target is the obstacle, obtain the location information of the obstacle; and construct a map based on the location information of the obstacle to obtain a global map of the first working area.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code, and when the computer program code is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

10. A lawn mowing robot, characterized in that: It includes acquisition equipment, main body and main control chip, among which: The acquisition device is used to acquire the depth image and the color image in the first operation area of ​​the lawn mowing robot; The main control chip is used to process the depth image and the color image according to the method described in any one of claims 1 to 7 to obtain a global map of the first operating area.

Images