Mowing robot based on multi-vision and its business execution method and system

Abstract

This invention discloses a lawnmower robot based on multi-view vision and its operational method and system. The lawnmower robot includes: a multi-sensor fusion system for implementing SLAM (Simultaneous Localization and Mapping), wherein the multi-sensor fusion system includes: a multi-view vision module, an inertial measurement unit, and an odometry system; wherein the multi-view vision module is used to achieve multi-view stereo vision and obtain 3D point cloud data of the environment; and a positioning system is used to save the 3D point cloud data of the environment during the lawnmower robot's mapping process and to perform 3D point cloud matching based on the 3D point cloud data during the lawnmower robot's autonomous operation. According to the technical solution provided by this invention, the positioning accuracy of SLAM is improved, and it can be applied to complex environmental scenarios.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a lawnmower robot based on multi-view vision and its operational methods and systems. Background Technology

[0002] With the improvement of living standards, the number of commercial and residential lawns is increasing, and residential lawns are very common. Robots, with their efficient, accurate, and continuous working capabilities, are gradually replacing manual lawn mowing. The aforementioned work scenarios are complex and varied. The goal of robots in mowing is to cut the grass to the desired height. Currently, most lawn mowing robots use satellite positioning to achieve full-scene traversal of the work area. However, due to the complexity of the work environment, such as the presence of houses, trees, and uneven ground around the lawn, the robot's positioning accuracy may deviate, affecting the effectiveness and intelligence of the robot's mowing task, and potentially leading to missed mowing spots.

[0003] Currently, robots on the market use a full-scene traversal mowing method to perform lawn mowing tasks. This means that the robot is controlled to traverse the entire target work scene without discrimination, based on positioning methods such as GNSS satellite positioning or UWB positioning, to achieve lawn mowing in the work scene.

[0004] However, due to the complex working environment of lawn mowing robots, such as the presence of houses, walls, trees, and uneven ground around the lawn, there are corresponding unsuitable scenarios for using GNSS satellites, UWB, and other positioning methods, which leads to reduced positioning accuracy or even failure. Consequently, using the full-scene traversal method for mowing may result in missed mowing areas or the lawn mowing robot moving outside the mowing area. Summary of the Invention

[0005] The main objective of this invention is to disclose a multi-view vision lawnmower robot and its operational method and system, in order to at least solve the problems in related technologies where lawnmower robots operate in complex environments, and the use of positioning methods such as GNSS satellites and UWB has corresponding unsuitable scenarios, resulting in reduced positioning accuracy or even failure. Consequently, using a full-scene traversal method for lawnmower mowing may lead to missed mowing or the lawnmower robot moving outside the mowing area.

[0006] According to one aspect of the present invention, a lawnmower robot based on multi-view vision is provided.

[0007] The lawnmower robot based on multi-view vision according to the present invention includes: a multi-sensor fusion system for realizing simultaneous localization and mapping (SLAM), wherein the multi-sensor fusion system includes: a multi-view vision module, an inertial measurement unit (IMU), and an odometry, wherein the multi-view vision module is used to realize multi-view stereo vision and obtain three-dimensional point cloud data of the environment; the positioning system is used to save the three-dimensional point cloud data of the environment during the mapping process of the lawnmower robot and to perform three-dimensional point cloud matching based on the three-dimensional point cloud data of the environment during the autonomous operation of the lawnmower robot.

[0008] According to another aspect of the present invention, a method for performing tasks using a lawnmower robot based on multi-view vision is provided.

[0009] The operational execution method of the lawnmower robot based on multi-view vision according to the present invention includes: establishing a world coordinate system based on the multi-view vision module, inertial sensor, and odometer of the lawnmower robot, with a marker board or charging pile as the starting point; responding to the user's control operation on the lawnmower robot, delineating the outermost working boundary of the area to be mowed, or pre-delineating the working path of the lawnmower robot; during the execution of the control command corresponding to the control operation by the lawnmower robot, performing at least one of the following operations based on the multi-view vision module: constructing and saving a SLAM map, constructing and saving a 3D scene map, and generating and saving a semantic map; when the lawnmower robot receives the operational execution command, autonomously planning a working path within the delineated outermost working boundary and traversing the working path, or traversing the pre-delineated working path; during the autonomous execution of the operational tasks by the lawnmower robot, performing at least one of the following operations based on the multi-view vision module: updating the currently saved SLAM map, updating the currently saved 3D scene map, and updating the currently saved semantic map.

[0010] According to another aspect of the present invention, a business execution system for a lawn mowing robot based on multi-view vision is provided.

[0011] The operational execution system of the lawnmower robot based on multi-view vision according to the present invention includes: a coordinate system establishment module, used to establish a world coordinate system based on the multi-view vision module, inertial sensor, and odometer of the lawnmower robot, with a marker board or charging pile as the starting point; a delineation module, used to delineate the outermost working boundary of the area to be mowed in response to the user's control operation on the lawnmower robot, or to pre-delineate the working path of the lawnmower robot; and a map building module, used to perform at least one of the following operations based on the multi-view vision module during the execution of control commands corresponding to the control operation by the lawnmower robot: constructing SLAM. The system includes a map and save module, a 3D scene map and save module, and a semantic map and save module. A path planning module is used to autonomously plan a working path within the defined outermost working boundary and traverse the working path when the lawnmower receives a business execution instruction, or to traverse a pre-defined working path. A map update module is used to perform at least one of the following operations based on the multi-view vision module during the lawnmower's autonomous business execution: update the currently saved SLAM map, update the currently saved 3D scene map, and update the currently saved semantic map.

[0012] According to the present invention, a multi-view vision-based lawn mowing robot and its operational method and system are provided. The multi-view vision-based lawn mowing robot includes a multi-sensor fusion system, a texture compensation device, and a positioning system. The multi-view vision module in the multi-sensor fusion system is used to achieve multi-view stereo vision and obtain three-dimensional point cloud data of the environment. The positioning system saves the three-dimensional point cloud data of the environment during the mapping process of the lawn mowing robot and performs three-dimensional point cloud matching based on the three-dimensional point cloud data of the environment during the autonomous operation of the lawn mowing robot. This achieves matching between the robot's autonomous operation and the mapping process, improves the positioning accuracy of SLAM, and can be applied to complex environmental scenarios. It solves the problems that may occur when using the full-scene traversal method for lawn mowing, such as missed mowing or the lawn mowing robot moving outside the mowing area. Attached Figure Description

[0013] Figure 1 This is a structural block diagram of a lawnmower robot based on multi-view vision according to an embodiment of the present invention;

[0014] Figure 2 This is a structural block diagram of a lawnmower robot based on multi-view vision according to a preferred embodiment of the present invention;

[0015] Figure 3 This is a schematic diagram of the structure of a lawnmower robot based on multi-view vision according to a preferred embodiment of the present invention;

[0016] Figure 4This is a flowchart of a business execution method for a lawnmower robot based on multi-view vision according to a preferred embodiment of the present invention;

[0017] Figure 5 This is a structural block diagram of a multi-view vision-based lawn mowing robot's operational system according to a preferred embodiment of the present invention. Detailed Implementation

[0018] The specific implementation of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] According to an embodiment of the present invention, a lawnmower robot based on multi-view vision is provided.

[0020] Figure 1 This is a structural block diagram of a lawnmower robot based on multi-view vision according to an embodiment of the present invention. Figure 1 As shown, the multi-view vision-based lawnmower robot includes a multi-sensor fusion system 10 for achieving simultaneous localization and mapping (SLAM). The multi-sensor fusion system 10 includes a multi-view vision module 100, an inertial measurement unit (IMU) 102, and an odometry system 104. The multi-view vision module 100 is used to achieve multi-view stereo vision and obtain three-dimensional point cloud data of the environment. The positioning system 14 is used to save the three-dimensional point cloud data of the environment during the mapping process of the lawnmower robot and to perform three-dimensional point cloud matching based on the three-dimensional point cloud data of the environment during the autonomous operation of the lawnmower robot.

[0021] Figure 1 The lawnmower robot shown employs a multi-sensor fusion system architecture, including a multi-view vision module (e.g., a binocular vision module), an IMU, and an odometry system, for SLAM localization. The multi-view vision module enables multi-view stereo vision, obtaining 3D point cloud data of the environment. The localization system saves this 3D point cloud data during the lawnmower robot's mapping process and performs 3D point cloud matching based on this data during the robot's autonomous operation. This improves the localization accuracy of SLAM and makes it suitable for complex environments. It also solves problems such as missed mowing or the lawnmower robot moving outside the mowing area when using a full-scene traversal mowing method.

[0022] Preferably, such as Figure 2As shown, the lawnmower robot described above may also include at least one of the following modules: a Global Navigation Satellite System (GNSS) module 16, used to activate in specific scenarios to assist the multi-sensor fusion system in achieving robot localization; and a spectral camera 18, used to detect the ground material during the lawnmower robot's movement and, when a grassy area is detected, to determine the chlorophyll abundance information in that grassy area. That is, the lawnmower robot may also include: a GNSS module; it may also include: a spectral camera; or it may include both a GNSS module and a spectral camera. Figure 2 The example shown includes both a GNSS module and a spectrophotometer.

[0023] In the preferred implementation process, for specific scenarios with large areas and open spaces, it is also necessary to activate the GNSS module of the lawnmower robot. The GNSS module assists the above-mentioned multi-sensor fusion system to better achieve robot positioning.

[0024] In the preferred implementation, the lawnmower robot uses a spectral camera to detect the ground material during its movement. When it detects a grassy area, it determines the chlorophyll abundance information in that area. Specifically, the multi-view vision module of the multi-sensor fusion system of this application can be used to acquire image data of the grassy area, and the spectral camera can be used to acquire spectral data of the grassy area. Then, texture features and spectral features are extracted from the image data and the spectral data, respectively. The texture features and spectral features are fused, and the fused feature information is used as input. The trained target detection model is then used to calculate and output the chlorophyll abundance of the grassy area. Using the above scheme, the rapid and automatic acquisition of chlorophyll abundance data of the grassy area can be achieved.

[0025] Preferably, such as Figure 2 As shown, the above-mentioned lawn mowing robot may further include: a texture compensation device 12, used to disperse the emitted infrared point light source into light spots or stripes to form a texture pattern; a brightness sensing device 20, used to sense the brightness of the current ambient light; and a brightness compensation device 22, connected to the brightness sensing device 20, used to perform brightness compensation when the brightness of the current environment is lower than a predetermined ambient brightness threshold, and dynamically adjust the compensation brightness value according to the sensing result of the brightness sensing device 20.

[0026] In the optimized implementation process, the lawnmower robot's multi-view vision module, combined with a texture compensation device, achieves multi-view stereo vision and obtains 3D point cloud data of the environment. Lawnmower robots typically operate in complex outdoor environments with significant ambient light variations. When the brightness sensing device (e.g., a light sensor module) detects changes in ambient light brightness, it is necessary to dynamically adjust the brightness compensation value (e.g., supplementary lighting) in real time to improve the lawnmower robot's adaptability to various outdoor scenarios (e.g., low-light scenarios).

[0027] Preferably, such as Figure 2 As shown, the above-mentioned lawnmower robot may further include: a mapping system 24 connected to the positioning system 18, wherein the mapping system 24 is used to perform at least one of the following functions:

[0028] The mapping system 24 is used to filter the above-mentioned environmental 3D point cloud data to remove sparse obstacles, and to build a 3D scene map based on the above-mentioned filtered environmental 3D point cloud data.

[0029] The mapping system 24 is used to perform semantic recognition of environmental objects based on the image data obtained by the above-mentioned multi-view vision module, and generate a semantic map based on the recognition results.

[0030] In the preferred implementation process, based on a multi-view vision module and a texture compensation device, multi-view stereo vision can be achieved to obtain 3D point cloud data of the environment. Based on this 3D point cloud data, a 3D map of the environmental scene can be constructed. The mapping system filters the 3D point cloud data, removing sparse obstacles (e.g., filtering out grass) while retaining other obstacles.

[0031] In the specific implementation process, based on the environmental 3D point cloud data to the point cloud projection depth image, the projection points of each point cloud data located on the aforementioned point cloud projection depth image can be clustered to obtain clusters corresponding to each obstacle. The specific steps are as follows: Based on the planar dimensions of the aforementioned point cloud projection depth image, the aforementioned point cloud projection depth image is divided into multiple grids, each grid containing point cloud data falling within the spatial range of that grid. For the aforementioned multiple grids, clustering is performed according to their planar relative positions to obtain clusters corresponding to each obstacle, and cluster labels corresponding to each cluster are generated.

[0032] Then, the features of each cluster on the point cloud projection depth image are extracted; the specific steps are as follows: determine the distribution range image of each cluster on the point cloud projection depth image; according to the distribution range image of each cluster on the point cloud projection depth image, divide each of the above distribution range images into rows to obtain row range images; based on the number of projections that can be accommodated and the actual number of projections of the point cloud data in the row range image, obtain the above row sparsity features; based on the number of row range images and the variance between the number of projections of the point cloud data in the row range image and the actual number of projections, obtain the above column density non-uniformity features; based on the distribution range image of each cluster on the point cloud projection depth image, and the number of projections that can be accommodated and the actual number of projections of the point cloud data in the distribution range image, obtain the average density features.

[0033] Finally, the extracted features corresponding to each cluster are input into the pre-trained classification model to obtain the probability of grass corresponding to each cluster, and the obstacles corresponding to the clusters with grass probabilities higher than a preset threshold (e.g., 50%) are removed from the obstacle sequence.

[0034] In the preferred implementation process, the mapping system can also use deep learning algorithms to perform semantic recognition of environmental objects based on image data obtained from a multi-view vision module (e.g., a binocular vision module), and construct a semantic map using semantic objects. The constructed semantic map can be used for lawnmower robot interaction and obstacle avoidance.

[0035] The following combination Figure 3 The preferred embodiments described above are further described below.

[0036] Figure 3 This is a structural schematic diagram of a lawnmower robot based on multi-view vision according to a preferred embodiment of the present invention. Figure 3 As shown, the lawnmower robot includes a binocular vision module comprising two cameras 30, which are spaced apart and positioned in front of the robot. These cameras can be mounted horizontally or tilted downwards at a certain pitch angle (e.g., greater than 0 degrees and less than 30 degrees). The robot also includes a texture compensation device 32, positioned alongside the two cameras 30 in front of the robot. The combination of the cameras 30 and the texture compensation device 32 enables binocular stereo vision. A spectral camera 34 is also positioned alongside the two cameras 30 in front of the robot. Based on the spectral camera 34, the robot detects the ground material. When a grassy area is detected, the chlorophyll abundance information in that area can be determined. The lawnmower robot can then construct a full-scene chlorophyll abundance map based on the chlorophyll abundance information and the robot's SLAM pose data. The lawnmower also includes: a brightness sensor 36 for sensing the brightness of the current ambient light, and a brightness compensation device 38 for compensating for brightness when the brightness of the current environment is lower than a predetermined ambient brightness threshold. The brightness sensor 36 and the brightness compensation device 38 are arranged side by side with the binocular camera 30 in front of the lawnmower. A GNSS module 40 is arranged at the rear of the robot for activation in specific scenarios to assist the multi-sensor fusion system in achieving robot positioning.

[0037] In addition, the IMU is installed inside the robot, fixedly mounted in a direction parallel or perpendicular to the vehicle body. Figure 3 Not shown in the image; the odometer is located inside the wheel hub motor. Figure 3 Not shown in the diagram; both the positioning and mapping systems are located inside the robot. Figure 3 Not shown in the image.

[0038] According to an embodiment of the present invention, a business execution method for a lawn mowing robot based on multi-view vision is also provided.

[0039] Figure 4 This is a flowchart of a business execution method for a lawnmower robot based on multi-view vision according to a preferred embodiment of the present invention. Figure 4 As shown, the operational method of this multi-view vision-based lawnmower robot includes:

[0040] Step S401: Based on the multi-view vision module, inertial sensor and odometer of the above-mentioned lawnmower robot, establish a world coordinate system with the marker board or charging pile as the starting point position;

[0041] Step S402: In response to the user's control operation on the above-mentioned lawn mowing robot, define the outermost working boundary of the area to be mowed, or predefine the working path of the above-mentioned lawn mowing robot;

[0042] Step S403: During the process of the lawnmower robot executing the control instructions corresponding to the above control operations, at least one of the following operations is performed based on the above multi-view vision module: constructing and saving a SLAM map, constructing and saving a 3D scene map, and generating and saving a semantic map;

[0043] Step S404: When the lawnmower receives the business execution instruction, it autonomously plans the work path within the defined outermost work boundary and traverses the work path, or it traverses the predefined work path.

[0044] Step S405: During the autonomous execution of tasks by the lawnmower robot, at least one of the following operations is performed based on the multi-view vision module: updating the currently saved SLAM map, updating the currently saved 3D scene map, and updating the currently saved semantic map.

[0045] Step S406: After the lawnmower robot autonomously performs its tasks, it returns to the aforementioned marker board or the aforementioned charging station and saves the updated SLAM map and / or the updated 3D scene map and / or the updated semantic map.

[0046] Figure 4The business execution method of the lawn mowing robot based on multi-view vision shown can realize multiple working modes in the stage of responding to user control operations on the lawn mowing robot (where user control operations on the lawn mowing robot include multiple methods, such as the user pushing the machine to perform corresponding operations, the user controlling the lawn mowing robot to perform corresponding operations on the client, etc.). The first mode is to define the outermost working boundary of the area to be mowed. The second mode is to predefine the working path of the lawn mowing robot. In the process of the lawn mowing robot executing the control commands corresponding to the above control operations, a map can be built. For example, a SLAM map can be built and saved, a 3D scene map can be built and saved, and a semantic map can be generated and saved. When the lawnmower receives a task execution command, in the first working mode, it can autonomously plan and traverse a working path within the defined outermost working boundary. In the second working mode, it tracks and traverses a pre-defined working path. Furthermore, during the autonomous execution of tasks, the lawnmower can update the maps created by the robot during the user-controlled operation phase. For example, it can update the currently saved SLAM map, the currently saved 3D scene map, and the currently saved semantic map. Using this method, the positioning accuracy and intelligence level of the lawnmower are effectively improved, and its working efficiency is significantly increased.

[0047] Preferably, during the process of the lawnmower robot executing the control command corresponding to the control operation, at least one of the following operations is performed based on the multi-view vision module:

[0048] The process involves recording point or region feature information in the environment to construct and save a SLAM map; combining the multi-view vision module with the texture compensation device to construct and save a 3D scene map; and performing semantic recognition of environmental objects based on the image data obtained from the multi-view vision module, generating and saving a semantic map based on the recognition results. The process of recording point or region feature information in the environment to construct and save a SLAM map may further include: extracting and recording at least one point feature, at least one line feature, at least one surface feature, or semantic feature in the current environment in real time based on the robot's multi-view vision module, constructing and saving a SLAM map.

[0049] During the autonomous operation of the aforementioned lawnmower robot, at least one of the following operations is performed based on the aforementioned multi-view vision module: real-time tracking and correction of the aforementioned point feature information or region feature information to update the SLAM map and save it; updating the aforementioned 3D scene mapping based on the aforementioned 3D point cloud data of the environment and saving it; performing environmental object semantic recognition based on the image data obtained by the aforementioned multi-view vision module and updating the aforementioned semantic map based on the recognition results and saving it. The real-time tracking and correction of the aforementioned point feature information or region feature information to update the SLAM map and saving it may further include the following processing: binding at least one point feature, at least one line feature, at least one surface feature, or semantic feature to a keyframe; when SLAM co-viewing or loop closure occurs, performing pose optimization on the aforementioned at least one point feature, at least one line feature, at least one surface feature, or semantic feature bound to the aforementioned keyframe; performing map correction and optimization on the SLAM map corresponding to the current scene; and saving the corrected and optimized SLAM map.

[0050] SLAM employs a feature-based localization approach. Because outdoor lighting conditions vary significantly throughout the day, SLAM uses a feature extraction method that is insensitive to changes in light. During the mapping process, SLAM extracts and records scene environmental features in real time. These environmental features can be independent single feature points, combinations of multiple feature points, line features, area features, or semantic features. During SLAM operation, map correction and optimization are performed in real time based on co-view and loop closure. When ambient light changes, the brightness compensation device needs to be dynamically adjusted in real time to improve adaptability to low-light scenes.

[0051] In the preferred implementation process, the lawnmower's business execution method mainly includes two stages: the mapping stage (the stage in which the lawnmower executes control commands corresponding to user control operations) and the autonomous business execution stage (the lawnmower traverses the autonomously planned work path within the defined outermost work boundary to execute business, or the lawnmower tracks the pre-defined work path to be executed).

[0052] Specifically, the mapping phase may include the following processing:

[0053] 1. The lawnmower robot establishes a world coordinate system starting from a marker board or charging station; the marker on the marker board can be Apriltag or other types of markers, such as ARTag, ARToolkit, aruco, etc.

[0054] 2. Users can manually push the lawnmower to define the outermost working boundary of the current lawnmower task (first working mode) or define the predetermined working path of the lawnmower (second working mode).

[0055] 3. As the user pushes the lawnmower forward, the lawnmower's multi-view vision module (e.g., binocular vision module) actively performs SLAM localization and records environmental features (including point features and area features).

[0056] 4. During the process of the user pushing the lawnmower, the lawnmower's mapping system can record point feature information or area feature information in the environment to build a SLAM map; during the process of the user pushing the lawnmower, the lawnmower's mapping system can build a 3D scene map based on stereo vision point cloud; during the process of the user pushing the lawnmower, the lawnmower's mapping system can perform semantic recognition of environmental objects based on object recognition, and build a semantic map from the semantic objects. The map built above can be used to realize the lawnmower's interaction and obstacle avoidance functions.

[0057] 5. After the lawnmower robot defines the outermost working boundary (first working mode) or the predetermined working path (second working mode), it returns to the starting position of the marker board or charging station to complete the mapping phase and stores the currently constructed SLAM map and point cloud and / or 3D map and point cloud and / or semantic map.

[0058] Specifically, the autonomous execution phase of a lawnmower robot's tasks may include the following processes:

[0059] 1. Responding to the user's choice of working mode, when the first working mode is selected, the lawnmower autonomously plans its motion trajectory within the outermost working boundary (e.g., using a spiral path planning method or an arc path planning method) to traverse the area; when the second working mode is selected, the lawnmower autonomously tracks the defined trajectory and traverses the trajectory.

[0060] 2. During the autonomous execution of tasks by the lawnmower robot, the SLAM module tracks and corrects the stored environmental features in real time according to changes in the scene environment, and updates the recorded environmental features (including single-point features and area features). The lawnmower robot can also update the 3D scene map based on stereo vision point cloud. The lawnmower robot can also perform environmental object semantic recognition based on object recognition and update the semantic map. The updated map can be used to realize lawnmower robot interaction and obstacle avoidance.

[0061] 3. After completing the task in either the first or second working mode, the lawnmower robot returns to the marker or charging station starting point to complete the task and stores the updated SLAM map and point cloud, 3D map and point cloud, and semantic map.

[0062] Preferably, during the process of the lawnmower executing the control command corresponding to the control operation, the following processing may also be included: detecting the ground material based on the spectral camera of the lawnmower, and determining the chlorophyll abundance in the grass area when a grass area is detected; and constructing a chlorophyll abundance map corresponding to the current scene using the chlorophyll abundance information in combination with the current pose data of the lawnmower.

[0063] Preferably, during the autonomous execution of the above-mentioned lawnmower robot's tasks, the following processing may also be included: based on multiple saved chlorophyll abundance maps of different scenarios, performing a relocation operation on the current scenario, and selecting a chlorophyll abundance map matching the current scenario from the multiple chlorophyll abundance maps of different scenarios according to the relocation result; determining the task execution area to be entered by the lawnmower robot based on the pre-saved chlorophyll abundance map corresponding to the current scenario; and determining the task execution strategy corresponding to the chlorophyll abundance information according to the chlorophyll abundance information in the chlorophyll abundance map.

[0064] In the optimized implementation process, the lawnmower robot uses a spectral camera to detect the ground material. When it detects a grassy area and its chlorophyll abundance, it can combine SLAM pose data to construct a chlorophyll abundance map of the work scene area. Based on the full-scene chlorophyll abundance map, it can: prevent the robot from entering non-grass areas; and plan the mowing based on chlorophyll abundance information and corresponding grass density information, so as to achieve different mowing intensity and speed for different grass densities. In other words, the mowing operation is matched with the chlorophyll abundance map. For example, since chlorophyll abundance information is related to grass density information, multiple levels can be preset according to grass density. Each level corresponds to different mowing intensity and robot speed. When the lawnmower robot autonomously executes its operation, it can determine the level based on the grass density information in the current grassy area, and then determine the corresponding operation strategy, such as the robot's speed or mowing intensity.

[0065] Preferably, during the autonomous execution of the above-mentioned lawnmower robot's tasks, at least one of the following processes is also included:

[0066] The above-mentioned multi-view vision module and / or spectral camera are used to detect the ground material, and the corresponding processing method of the above-mentioned lawn mowing robot is determined based on the detection results of the ground material; for example, if the ground material is detected and the ground is not grass, the mowing is stopped.

[0067] The above-mentioned multi-view vision module and / or the above-mentioned spectral camera are used to detect the ground material. Based on the detection results of the ground material, combined with the current pose data of the above-mentioned lawn mowing robot (the pose data is obtained through SLAM positioning), the outermost working boundary of the above-mentioned area to be mowed is jointly located. Using this method, the lawn mowing robot can be prevented from entering non-grass areas due to SLAM positioning offset, thereby avoiding certain risks.

[0068] By using the aforementioned multi-view vision module and / or the aforementioned spectral camera to detect the ground material, when a non-grassland area is detected in the region enveloped by the outermost working boundary, the aforementioned mowing robot is controlled to shut down the mowing function module and move across the non-grassland area. The robot can better reduce risks by moving across the area after shutting down the mowing function module.

[0069] The aforementioned multi-view vision module and / or spectral camera are used to detect the ground material, and the detection results are recorded in a world map corresponding to the aforementioned world coordinate system. The world map can be used to better realize robot motion planning and control.

[0070] Preferably, during the autonomous execution of the above-mentioned lawn mowing robot, the following processing may also be included: performing environmental object semantic recognition based on the image data obtained by the above-mentioned multi-view vision module, determining dynamic objects and static objects according to the identified object categories; updating the current semantic map in real time according to the environmental object semantic recognition results, recording the addition time of newly added dynamic objects in the current semantic map, and updating the semantic map in real time according to the movement of the dynamic object if the position of the dynamic object is detected to change after a predetermined time.

[0071] In the preferred implementation process, the lawnmower robot performs object recognition and semantic mapping based on a multi-view vision module (e.g., a binocular vision module). For example, it identifies flower and grass areas, tables and chairs, pedestrians, pets, etc. in the lawn. The identified static or dynamic objects are processed according to different strategies. For example, static objects are mapped and maintained, while dynamic objects are detected and maintained for a predetermined time. If the position of the dynamic object changes after the predetermined time, the semantic map is updated in real time according to the movement of the dynamic object.

[0072] Preferably, the above method may further include the following processing: when the multi-view vision module is in a scene with insufficient light or overexposure, perform linearization or nonlinear brightness adjustment operation on the image data acquired by the multi-view vision module; and perform at least one of the following operations based on the adjusted image data: SLAM operation, object recognition operation, and 3D point cloud matching operation.

[0073] During the preferred implementation process, the outdoor lighting environment varies greatly throughout the day, which may cause the visual sensors of the multi-view vision module to be dim or overexposed. Therefore, it is necessary to adjust the image brightness of the acquired images. The image brightness adjustment methods include non-linear or linear brightness stretching. Preferably, a non-linear brightness stretching method is used to achieve adaptive brightness and contrast enhancement of the entire image. Based on the brightness-adjusted image, SLAM localization, object recognition, and 3D point cloud matching are performed to improve accuracy and effect.

[0074] The linearization brightness adjustment operation for the image data acquired by the multi-view vision module can be performed in the following way: convert the RGB color space of the image pixels to the HSL color space or the HSV color space; use the light brightness adjustment factor to increase or decrease the brightness part of the HSL color space or the lightness part of the HSV color space; and convert the adjusted HSL color space or HSV color space back to the RGB color space.

[0075] The nonlinear brightness adjustment operation performed on the image data acquired by the multi-view vision module can be carried out in the following way: using the light brightness adjustment factor, the red, green and blue values ​​of the image pixels acquired by the multi-view vision module are adjusted to obtain a new image brightness value. The light brightness adjustment factor is obtained by looking up the light brightness adjustment table using light intensity information and the original brightness of the acquired image.

[0076] Preferably, after delineating the outermost working boundary of the area to be mowed in response to the user's control operation on the above-mentioned lawn mowing robot, the process may further include at least one of the following:

[0077] The system outputs and presents the aforementioned 3D scene mapping or semantic map to the user terminal; it receives information from the user terminal regarding non-business execution areas set by the user within the outermost working boundary based on the aforementioned 3D scene mapping or semantic map; and it determines the area where the lawnmower robot will perform its business based on the outermost working boundary and the aforementioned non-business execution area information. In other words, the user can further set non-business execution areas (i.e., prohibited lawnmower areas) within the lawnmower area enclosed by the outermost working boundary using a mobile terminal device (such as a mobile phone).

[0078] In response to the user's control of the lawnmower robot to delineate the boundary of the non-business execution area within the outermost working boundary (for example, after the user pushes the lawnmower robot or uses a remote control to control the lawnmower robot to delineate the outermost working boundary, the user pushes the lawnmower robot or uses a remote control to control the lawnmower robot to continue delineating the non-mowing area inside the outermost working boundary), the area where the lawnmower robot is to perform its business is determined based on the outermost working boundary and the boundary of the non-business execution area.

[0079] Preferably, the above-mentioned business execution method may further include at least one of the following processes:

[0080] During the autonomous execution of the above-mentioned lawn mowing robot, when the lawn mowing robot needs to interrupt the current execution of ...

[0081] During the autonomous execution of the above-mentioned lawn mowing robot, the execution time information and the area covered by the operation are recorded and statistically analyzed; that is, the lawn mowing robot supports the statistics of working area and working time.

[0082] During the autonomous execution of the above-mentioned lawnmower robot, based on the above-mentioned 3D scene mapping corresponding to the current scene, and / or based on the above-mentioned semantic map corresponding to the current scene, the corresponding obstacle avoidance strategy is selected in combination with the identified obstacle categories; that is, the lawnmower robot supports obstacle avoidance based on scene maps constructed from 3D point clouds and semantic objects.

[0083] After the lawnmower autonomously performs its tasks, it retraces the work path or the autonomously planned path to perform the overall lawn mowing and recycling operation; that is, the lawnmower supports unified lawn mowing and recycling.

[0084] According to an embodiment of the present invention, a business execution system for a lawn mowing robot based on multi-view vision is also provided.

[0085] Figure 5 This is a structural block diagram of a multi-view vision-based lawnmower robot's operational system according to a preferred embodiment of the present invention. Figure 5As shown, the operational execution system of the multi-view vision-based lawnmower robot includes: a coordinate system establishment module 50, used to establish a world coordinate system based on the multi-view vision module, inertial sensor, and odometer of the lawnmower robot, with a marker board or charging station as the starting point; a delineation module 52, used to delineate the outermost working boundary of the area to be mowed in response to the user's control operation on the lawnmower robot, or to pre-delineate the working path of the lawnmower robot; and a map construction module 54, used to perform at least one of the following operations based on the multi-view vision module during the execution of control commands corresponding to the control operation by the lawnmower robot: constructing SLAM. The system includes a map and save module 56, a 3D scene map and save module 57, and a semantic map and save module 58. The path planning module 56 is used to autonomously plan a working path within the defined outermost working boundary and traverse the working path when the lawnmower receives a business execution instruction, or to traverse a pre-defined working path. The map update module 58 is used to perform at least one of the following operations based on the multi-view vision module during the autonomous execution of business by the lawnmower: updating the currently saved SLAM map, updating the currently saved 3D scene map, and updating the currently saved semantic map.

[0086] Figure 5 The operational principle of the multi-view vision-based lawnmower robot's business execution system, as shown, can be found in [reference needed]. Figure 4 The description will not be repeated here. The above-mentioned business execution system effectively improved the positioning accuracy and intelligence level of the lawnmower robot, and also significantly increased its working efficiency.

[0087] In summary, using the embodiments provided by this invention, the multi-view vision-based lawnmower robot includes a multi-sensor fusion system, the texture compensation device, and a positioning system. The multi-view vision module in the multi-sensor fusion system, combined with the texture compensation device, achieves multi-view stereo vision, obtaining 3D point cloud data of the environment. The positioning system saves the 3D point cloud data during the lawnmower robot's mapping process and performs 3D point cloud matching based on the environmental 3D point cloud data during the robot's autonomous operation, achieving matching between the robot's autonomous operation and mapping process. This improves the positioning accuracy of SLAM and is applicable to complex environments. Furthermore, the lawnmower robot performs semantic recognition of environmental objects based on the multi-view vision module and creates a semantic map from these semantic objects. This semantic map can be used for lawnmower robot interaction and obstacle avoidance. In addition, the lawnmower robot also includes a spectral camera that detects ground material during its movement. When a grassy area is detected, the chlorophyll abundance information in the grassy area is determined. Combined with the lawnmower robot's current pose data, the chlorophyll abundance information is used to construct a chlorophyll abundance map corresponding to the current scene. A full-scene chlorophyll abundance map can prevent the robot from entering non-grassland areas. Based on the chlorophyll abundance information in the map, and corresponding information such as the density of the grass, mowing planning is performed to achieve different mowing intensity and speed for different grass densities. Using the solution provided in this invention, the positioning accuracy and intelligence level of the lawnmower robot can be improved, further enhancing its working efficiency.

[0088] The above-disclosed embodiments are merely a few specific examples of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A lawnmowing robot based on multi-view vision, characterized in that, include: A multi-sensor fusion system for simultaneous localization and mapping is provided, wherein the multi-sensor fusion system includes: a multi-view vision module, an inertial measurement unit, and an odometer; wherein the multi-view vision module is used to combine with a texture compensation device to achieve multi-view stereo vision and obtain three-dimensional point cloud data of the environment. The texture compensation device is used to disperse the emitted infrared point light source into light spots or stripes to form a texture pattern. A spectral camera is used to detect the ground material during the movement of the lawnmower robot. When a grassy area is detected, the chlorophyll abundance information in the grassy area is determined. Combined with the current pose data of the lawnmower robot, a chlorophyll abundance map corresponding to the current scene is constructed using the chlorophyll abundance information. The chlorophyll abundance map based on the whole scene can prevent the lawnmower robot from entering non-grassy areas. The multi-view vision module acquires image data of the grassy area, and the spectral camera acquires spectral data of the grassy area. Texture features are extracted from the image data, and spectral features are extracted from the spectral data. The texture features and spectral features are fused. The fused feature information is used as input, and the trained target detection model is used to calculate and output the chlorophyll abundance of the grassy area. A positioning system is used to save the environmental 3D point cloud data during the mapping process of the lawnmower robot, and to perform 3D point cloud matching based on the environmental 3D point cloud data during the autonomous operation of the lawnmower robot.

2. The lawnmower robot according to claim 1, characterized in that, Also includes: The Global Navigation Satellite System (GNSS) module is used to activate in specific scenarios to assist the multi-sensor fusion system in achieving robot positioning.

3. The lawnmower robot according to claim 1, characterized in that, Also includes: A brightness sensor is used to sense the brightness of the current ambient light. A brightness compensation device is used to perform brightness compensation when the brightness of the current environment is lower than a predetermined ambient brightness threshold, and to dynamically adjust the compensation brightness value according to the sensing result of the brightness sensing device.

4. The lawnmower robot according to claim 1, characterized in that, Also includes: A mapping system, wherein the mapping system is configured to perform at least one of the following functions: The mapping system is used to filter the environmental 3D point cloud data to remove sparse obstacles, and to build a 3D scene map based on the filtered environmental 3D point cloud data. The mapping system is used to perform semantic recognition of environmental objects based on the image data obtained by the multi-view vision module, using a deep learning algorithm, and to generate a semantic map based on the recognition results.

5. A method for executing tasks using a multi-view vision-based lawnmowing robot as described in any one of claims 1 to 4, characterized in that, include: Based on the multi-view vision module, inertial measurement unit, and odometer of the lawnmower robot, a world coordinate system is established with the marker board or charging pile as the starting point. In response to the user's control operation on the lawn mowing robot, the outermost working boundary of the area to be mowed is defined, or the working path of the lawn mowing robot is predefined; During the process of the lawnmower robot executing control commands corresponding to the control operation, at least one of the following operations is performed based on the multi-view vision module: constructing and saving a SLAM map, constructing and saving a 3D scene map, and generating and saving a semantic map. When the lawnmower receives a work execution instruction, it autonomously plans a work path within the defined outermost work boundary and traverses the work path, or it traverses the pre-defined work path. During the autonomous execution of tasks by the lawnmower robot, at least one of the following operations is performed based on the multi-view vision module: updating the currently saved SLAM map, updating the currently saved 3D scene map, and updating the currently saved semantic map. After the lawnmower robot autonomously performs its tasks, it returns to the marker board or the charging station and saves the updated SLAM map and / or the updated 3D scene map and / or the updated semantic map.

6. The business execution method according to claim 5, characterized in that, During the process of the lawnmower robot executing control commands corresponding to the control operation, at least one of the following operations is performed based on the multi-view vision module: recording point feature information or region feature information in the environment to construct and save a SLAM map; constructing and saving a 3D scene map by combining the 3D point cloud data of the environment obtained by the multi-view vision module with the texture compensation device; and performing semantic recognition of environmental objects based on the image data obtained by the multi-view vision module and generating and saving a semantic map based on the recognition results. During the autonomous execution of the lawnmower robot's tasks, at least one of the following operations is performed based on the multi-view vision module: real-time tracking and correction of the point feature information or region feature information to update the SLAM map and save it; updating the 3D scene map based on the environmental 3D point cloud data and saving it; performing environmental object semantic recognition based on the image data obtained by the multi-view vision module and updating the semantic map based on the recognition results and saving it.

7. The business execution method according to claim 6, characterized in that, Recording point feature information or region feature information in the environment to construct and save a SLAM map includes: extracting and recording at least one point feature, at least one line feature, at least one surface feature, or semantic feature in the current environment in real time based on the robot's multi-view vision module, constructing a SLAM map and saving it; Real-time tracking and correction of the point feature information or region feature information to update the SLAM map and save it includes: binding at least one point feature, at least one line feature, at least one surface feature, or semantic feature to a keyframe; when SLAM co-view or loop closure occurs, performing pose optimization on the at least one point feature, at least one line feature, at least one surface feature, or semantic feature bound to the keyframe respectively; performing map correction and optimization on the SLAM map corresponding to the current scene; and saving the corrected and optimized SLAM map.

8. The business execution method according to claim 5, characterized in that, The process of the lawnmower autonomously performing its tasks also includes: Based on the saved chlorophyll abundance maps of multiple different scenes, a relocation operation is performed on the current scene, and the chlorophyll abundance map that matches the current scene is selected from the multiple chlorophyll abundance maps of different scenes according to the relocation result. Based on a pre-saved chlorophyll abundance map corresponding to the current scene, the business execution area to be entered by the lawn mowing robot is determined. Based on the chlorophyll abundance information in the chlorophyll abundance map, determine the business execution strategy corresponding to the chlorophyll abundance information.

9. The business execution method according to claim 5, characterized in that, The autonomous execution of tasks by the lawnmower robot also includes at least one of the following: The multi-view vision module and / or spectral camera are used to detect the ground material, and the current processing method of the lawn mowing robot is determined based on the detection results of the ground material. The ground material is detected using the multi-view vision module and / or the spectral camera. Based on the detection results of the ground material and combined with the current pose data of the lawnmower robot, the outermost working boundary of the area to be mowed is jointly located. The multi-view vision module and / or the spectral camera are used to detect the ground material. When a non-grass area is detected in the region enveloped by the outermost working boundary, the lawnmower robot is controlled to turn off the lawnmower function module and move across the non-grass area. The ground material is detected using the multi-view vision module and / or the spectral camera, and the detection results are recorded in a world map corresponding to the world coordinate system.

10. The business execution method according to claim 5, characterized in that, The process of the lawnmower autonomously performing its tasks also includes: Based on the image data obtained by the multi-view vision module, environmental object semantic recognition is performed, and dynamic and static objects are determined according to the recognized object categories. The semantic map is updated in real time based on the semantic recognition results of environmental objects. The addition time of newly added dynamic objects in the current semantic map is recorded. If the position of the dynamic object changes after a predetermined time, the semantic map is updated in real time based on the movement of the dynamic object.

11. The business execution method according to claim 5, characterized in that, Also includes: When the multi-view vision module is in a scene with insufficient light or overexposure, a linear or non-linear brightness adjustment operation is performed on the image data acquired by the multi-view vision module. Perform at least one of the following operations based on the adjusted image data: SLAM operation, object recognition operation, and 3D point cloud matching operation.

12. The business execution method according to claim 5, characterized in that, In response to user control operations on the lawnmower robot, after defining the outermost working boundary of the area to be mowed, the system also includes at least one of the following: The three-dimensional scene mapping or semantic map is output and presented to the user terminal. The user terminal receives non-business execution area information set by the user in the outermost working boundary based on the three-dimensional scene mapping or semantic map. The area of ​​the lawn mowing robot to perform business is determined according to the outermost working boundary and the non-business execution area information. In response to user control, the lawnmower robot delineates the boundary of the non-business execution area within the outermost working boundary, and determines the area where the lawnmower robot will perform its business based on the outermost working boundary and the boundary of the non-business execution area.

13. The business execution method according to claim 5, characterized in that, It also includes at least one of the following: During the autonomous execution of the lawn mowing robot's tasks, when the lawn mowing robot needs to interrupt the current task, it records the current working position and the area of ​​completed tasks. After returning to the current working position, it continues to execute the unfinished part of the task corresponding to the task execution instruction. During the autonomous execution of the lawnmower robot's tasks, the execution time and coverage area are recorded and statistically analyzed. During the autonomous execution of tasks by the lawnmower robot, a 3D scene map is built based on the current scene, and / or a semantic map is built based on the current scene, combined with the identified obstacle categories to select the corresponding obstacle avoidance strategy. After the lawnmower robot autonomously performs its tasks, it retraces the path to be worked or the autonomously planned path to perform the overall lawn mowing operation.

14. A business execution system for a lawnmower robot based on multi-view vision as described in any one of claims 1 to 4, characterized in that, include: The coordinate system establishment module is used to establish a world coordinate system based on the multi-view vision module, inertial measurement unit, and odometer of the lawnmower robot, with the marker board or charging pile as the starting point position. The delineation module is used to respond to the user's control operation on the lawn mowing robot, delineate the outermost working boundary of the area to be mowed, or pre-delineate the working path of the lawn mowing robot; The map building module is used to perform at least one of the following operations based on the multi-view vision module during the process of the lawn mowing robot executing control commands corresponding to the control operation: building and saving a SLAM map, building and saving a 3D scene map, and generating and saving a semantic map. The path planning module is used to autonomously plan a working path within the defined outermost working boundary and traverse the working path when the lawn mowing robot receives a business execution instruction, or to traverse the predefined working path. The map update module is used to perform at least one of the following operations based on the multi-view vision module during the autonomous execution of the lawnmower robot's business: update the currently saved SLAM map, update the currently saved 3D scene map, and update the currently saved semantic map.

Images