Smart lawnmower and smart lawnmowing system

Abstract

The application discloses an intelligent mowing system, which comprises an intelligent mower and a mobile terminal. The mobile terminal comprises a camera, which is used for collecting image data of the environment around the intelligent mower; an inertial measurement unit, which is used for detecting the pose data of the intelligent mower; an interface, which is connected with the intelligent mower and is used for data transmission; a memory, which is used for storing at least an application program for controlling the working or walking of the intelligent mower; and a processor, which is used for calling the application program, fusing the image data collected by the camera and the pose data collected by the inertial measurement unit, performing instant positioning and map construction of the intelligent mower, and generating navigation and mowing instructions according to a preset program. The intelligent mower comprises a main body, a fixing device arranged on the main body and used for fixing the mobile terminal to the intelligent mower, an interface connected with the mobile terminal and used for data transmission, and a controller electrically connected with the interface and used for controlling the intelligent mower according to the instructions of the mobile terminal when the interface of the intelligent mower is connected with the interface of the mobile terminal.

Description

TECHNICAL FIELD

[0001] The present application relates to a mower and a mowing system, in particular, an intelligent mower and an intelligent mowing system. BACKGROUND

[0002] With the rise and popularity of smart home, the technology of intelligent mower is progressing, and the degree of acceptance by the family is gradually increasing. Because it does not need manpower to push and follow, it greatly reduces the user's laboriousness and saves the user's time. The navigation positioning of the existing intelligent mower generally uses GPS with ordinary positioning accuracy to identify the region, uses boundary line signal and inertial measurement unit (IMU) to realize the calculation of accurate position, but this scheme usually has low positioning accuracy, cannot realize real-time positioning and navigation, and is difficult to obtain efficient path planning and complete area coverage. The high-precision positioning scheme, such as the RTK scheme based on satellite signal, or the UWB scheme based on radio signal, and so on, the hardware cost and system reliability of these schemes have always been the bottleneck of limiting their application. In addition, for the intelligent mower working autonomously, even if the high-precision positioning is obtained at any cost, it is far from enough, because of the lack of deep understanding of the surrounding environment, the mower cannot easily cope with the complex situations of road surface, obstacles, light, etc. SUMMARY

[0003] In order to solve the problems of the prior art, the main purpose of the present application is to provide an intelligent mower with low cost, higher positioning accuracy and deeper understanding of the surrounding environment.

[0004] To achieve the above purpose, the present application adopts the following technical scheme:

[0005] An intelligent mowing system includes an intelligent mower and a mobile terminal. The mobile terminal includes a camera for collecting image data of the environment around the intelligent mower, an inertial measurement unit for detecting the pose data of the intelligent mower, an interface for connecting with the intelligent mower for data transmission, a memory for storing at least an application program for controlling the working or walking of the intelligent mower, and a processor for calling the application program, fusing the image data obtained by the camera and the pose data obtained by the inertial measurement unit, performing instant positioning and map construction of the intelligent mower, generating navigation and mowing instructions according to a preset program, and sending the navigation and mowing instructions to the intelligent mower through the interface. The intelligent mower includes a main body, a fixing device arranged on the main body for fixing and mounting the mobile terminal to the intelligent mower, an interface for connecting with the mobile terminal for data transmission, and a controller electrically connected with the interface, for controlling the intelligent mower according to the navigation and mowing instructions of the mobile terminal when the interface of the intelligent mower is connected with the interface of the mobile terminal.

[0006] Optionally, the fixing device of the intelligent mower comprises an elastic clamping mechanism capable of elastically clamping a mobile terminal with a size of 4 inches to 12 inches.

[0007] Optionally, the application program can distinguish the grassland from the non-grassland according to the feature points of the two-dimensional plane in the image data and the texture features of the grassland, and automatically generate the boundary of the mowing area by the real-time positioning and the map construction with the discrete anchor points of the boundary between the grassland and the non-grassland.

[0008] Optionally, the intelligent mower further comprises a blade, and the application program can distinguish the grassland from the non-grassland according to the feature points of the two-dimensional plane in the image data and the texture features of the grassland, and stop rotating the blade when the current working plane is not the grassland.

[0009] Optionally, the application program can determine the type of the current working plane according to the feature points of the two-dimensional plane in the image data and the texture features of the common types of ground preset by the application program, and control the intelligent mower to drive to the ground with greater hardness among the multiple types of ground when the current working plane comprises multiple types of ground.

[0010] Optionally, the application program further comprises an object recognition program, and the application program can select a corresponding obstacle avoidance strategy according to the obstacle category recognized by the object recognition program.

[0011] Optionally, the mobile terminal further comprises a global satellite positioning system sensor, and the application program uses the positioning result of the global satellite positioning system sensor to filter and correct the result of the real-time positioning and the map construction.

[0012] Optionally, the intelligent mower further comprises a lighting lamp, and the application program calculates the light intensity of the current environment according to the image data and sends an instruction to turn on the lighting lamp when the light intensity is lower than a first light intensity threshold.

[0013] Optionally, the intelligent mowing system further comprises an interactive display interface, and the user can view the real-time image collected by the camera through the interactive display interface and superimpose a virtual fence on the real-time image, and the application program adds the anchor points of the virtual fence to the anchor point set of the mowing area boundary.

[0014] Optionally, the intelligent mowing system further comprises an interactive display interface, and the user can view the real-time image collected by the camera through the interactive display interface and superimpose a virtual obstacle on the real-time image, and the application program records the anchor points of the virtual obstacle and plans a path to bypass the virtual obstacle.

[0015] The application discloses an intelligent mowing system, which comprises an intelligent mower and a mobile terminal. The intelligent mower comprises a camera for collecting image data of the environment around the intelligent mower, an inertial measurement unit for detecting the pose data of the intelligent mower, and an interface for connecting with the mobile terminal for data transmission. The mobile terminal comprises an interface for connecting with the intelligent mower for data transmission, a memory for storing at least an application program for controlling the working or walking of the intelligent mower, and a processor for calling the application program, obtaining the image data and the pose data from the intelligent mower, fusing the image data and the pose data, performing instant positioning and map construction of the intelligent mower, generating navigation and mowing instructions according to a preset program, and sending the navigation and mowing instructions to the intelligent mower. When the intelligent mower is connected with the mobile terminal, the navigation and mowing instructions of the mobile terminal control the intelligent mower.

[0016] Optionally, the interface of the intelligent mower comprises a wireless communication device, the interface of the mobile terminal comprises a wireless communication device, and wireless data transmission can be realized between the intelligent mower and the mobile terminal.

[0017] Optionally, the interface of the intelligent mower comprises an application program interface, and the application program interface defines the data communication protocol and format between the intelligent mower and the mobile terminal.

[0018] Optionally, the application program comprises mowing preference parameters which can be edited by a user.

[0019] Optionally, the intelligent mower further comprises a global satellite positioning system sensor, and the application program uses the positioning result of the global satellite positioning system sensor to filter and correct the result of the instant positioning and map construction.

[0020] Optionally, the intelligent mower further comprises a lighting lamp, and the application program calculates the illumination intensity of the current environment according to the image data and sends an instruction to turn on the lighting lamp when the illumination intensity is lower than a first illumination intensity threshold.

[0021] Optionally, the application program can distinguish the grassland from the non-grassland according to the feature points of the two-dimensional plane in the image data and the texture features of the grassland, and automatically generate the boundary of the mowing area by taking the boundary of the grassland and the non-grassland as discrete anchor points and through the instant positioning and map construction.

[0022] Optionally, the mobile terminal further comprises an interactive display interface, a user can view the real-time image collected by the camera through the interactive display interface, and superimpose a virtual fence on the real-time image, and the application program adds the anchor points of the virtual fence to the anchor point set of the boundary of the mowing area.

[0023] Optionally, the mobile terminal further comprises an interactive display interface, through which a user can view the real-time image captured by the camera and superimpose virtual obstacles on the real-time image, and the application program records the anchor points of the virtual obstacles and plans a path to bypass the virtual obstacles.

[0024] The application has the advantages of fusing visual and inertial information through a mobile terminal, saving the manufacturing cost of the intelligent mower, and achieving higher precision positioning and deep understanding of the environment. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a side view of an intelligent mower according to an embodiment of the application;

[0026] Figure 2 is a side view of an intelligent mower according to an embodiment of the application;

[0027] Figure 3A is Figure 2 is a perspective view of a telescopic support of a camera of the intelligent mower shown in

[0028] Figure 3B is Figure 3A is a sectional view of the telescopic support of the camera of the intelligent mower shown in

[0029] Figure 3C is Figure 3A is a sectional view of the telescopic support of the camera of the intelligent mower shown in

[0030] Figure 4A is a side view of an intelligent mower according to an embodiment of the application in a non-working state;

[0031] Figure 4B is Figure 4A is a side view of an intelligent mower according to an embodiment of the application in a working state;

[0032] Figure 5A is a side view of an intelligent mower according to an embodiment of the application in a non-working state;

[0033] Figure 5B is Figure 5A is a side view of an intelligent mower according to an embodiment of the application in a working state;

[0034] Figure 6 is Figure 1 is a schematic view of an inertial measurement unit of the intelligent mower shown in

[0035] Figure 7 is a schematic view of a dual inertial measurement unit of the intelligent mower according to an embodiment of the application;

[0036] Figure 8This is a system schematic diagram of an intelligent lawnmower according to an embodiment of this application;

[0037] Figure 9 This is a flowchart of a Simultaneous Localization and Mapping (SLAM) algorithm according to an embodiment of this application;

[0038] Figure 10 This is a flowchart of a sensor fusion algorithm according to an embodiment of this application;

[0039] Figure 11A This is a display interface in a boundary recognition mode according to an embodiment of this application;

[0040] Figure 11B This is a display interface under another boundary recognition mode according to an embodiment of this application;

[0041] Figure 12 This is a schematic diagram of the road surface recognition and selection function according to an embodiment of this application;

[0042] Figure 13A This is a schematic diagram of an obstacle recognition function according to an embodiment of this application;

[0043] Figure 13B This is another schematic diagram of the obstacle recognition function according to an embodiment of this application;

[0044] Figure 14 This is a flowchart of an obstacle avoidance algorithm according to an embodiment of this application;

[0045] Figure 15 This is a display interface for setting virtual obstacles according to an embodiment of this application;

[0046] Figure 16 This is a schematic diagram of a smart lawnmower and a camera set in the scene, according to another embodiment of this application;

[0047] Figure 17A yes Figure 16 The diagram shows a data transmission architecture between a smart lawnmower and a camera set up in the scene.

[0048] Figure 17B yes Figure 16 The diagram shows another data transmission architecture between the smart lawnmower and the camera set up in the scene;

[0049] Figure 17C yes Figure 16 The diagram shows the data transmission architecture of the smart lawnmower, the cameras installed in the scene, and the cloud server.

[0050] Figure 18 This is a side view of an intelligent lawn mowing system according to another embodiment of this application;

[0051] Figure 19A yesFigure 18 side view of the fixing device of the smart mower shown;

[0052] Figure 19B is Figure 19A side view of the chuck of the fixing device of the smart mower shown when retracted;

[0053] Figure 19C is Figure 19A side view of the chuck of the fixing device of the smart mower shown when extended;

[0054] Figure 20 is a side view of a smart mower in a smart mower system according to another embodiment of the present application;

[0055] Figure 21A is a schematic view of an inertial measurement unit of a mobile terminal in a smart mower system according to another embodiment of the present application;

[0056] Figure 21B is a schematic view of a camera of a mobile terminal in a smart mower system according to another embodiment of the present application;

[0057] Figure 21C is a schematic view of an interface of a mobile terminal in a smart mower system according to another embodiment of the present application;

[0058] Figure 22A is a first data transmission architecture diagram of a smart mower system according to another embodiment of the present application;

[0059] Figure 22B is a second data transmission architecture diagram of a smart mower system according to another embodiment of the present application;

[0060] Figure 22C is a third data transmission architecture diagram of a smart mower system according to another embodiment of the present application;

[0061] Figure 22D is a fourth data transmission architecture diagram of a smart mower system according to another embodiment of the present application;

[0062] Figure 22E is a fifth data transmission architecture diagram of a smart mower system according to another embodiment of the present application. DETAILED DESCRIPTION

[0063] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] As Figure 1As shown, this application proposes an intelligent lawnmower 110, including: a cutting blade 112 for cutting grass; a main body 113 for mounting the cutting blade 112; wheels 114 that can rotate and support the main body 113; a light 119 for illumination; a camera assembly 120 for acquiring image information of the lawnmower's surrounding environment; an inertial measurement unit (IMU) 122 for acquiring the lawnmower's pose information; and a processor ( Figure 1 (Not shown in the image), electrically connected to the camera assembly 120 and the inertial measurement unit 122, used for calculating and processing information acquired by the camera assembly 120 and the inertial measurement unit 122; memory ( Figure 1 (Not shown in the image) is used to store the control program 145 for controlling the operation of the intelligent lawnmower 110. The processor can call the control program 145 to fuse the image information of the lawnmower's surrounding environment collected by the camera component 120 and the lawnmower's pose information data collected by the inertial measurement unit 122 to realize Simultaneous Localization and Mapping (SLAM) of the lawnmower, and generate corresponding navigation and mowing commands according to preset logic and real-time data to control the behavior of the intelligent lawnmower 110.

[0065] Optionally, the camera assembly 120 can be mounted on the front of the smart lawnmower 110, see [reference]. Figure 1 The camera assembly 120 installed at the front of the lawnmower 110 can effectively capture image information of the environment in front of the intelligent lawnmower 110. Compared to image information from the sides or rear of the lawnmower, the image information from the front of the lawnmower is more valuable for navigation and obstacle avoidance. Optionally, the camera assembly 120 can also be mounted on the upper front of the lawnmower via a bracket 123, such as... Figure 2 As shown. By raising the bracket 123, the vertical distance between the camera assembly 120 and the ground increases, thereby increasing the field of view of the camera assembly 120 and making the line of sight less likely to be obstructed by near-ground obstacles such as weeds.

[0066] Optionally, the bracket 123 is a telescopic device. For example... Figures 3A-3CThe shown support 123 is composed of a pin 392 telescopic sleeve. The tube part of the pin 392 telescopic sleeve includes two hollow tubes inside and outside, and the wires of the camera assembly 120 pass through the cavity between the two tubes. The outer tube 394 has a plurality of holes 395 arranged along the length direction of the outer tube 394. The inner tube 391 has a hole, and the cavity of the inner tube 391 has a head of a smooth pin 392 perpendicular to the hole, and the pin 392 is connected to a spring 393, one end of the spring 393 is fixed to the inner wall of the inner tube 391, and the other end is connected to the bottom of the pin 392, and always gives the pin 392 an outward force, so that the head of the pin 392 extends outward through the hole of the inner tube 391 when not pushed by other external forces. When the outer tube 394 is sleeved on the inner tube 391, one of the plurality of holes 395 arranged along the length direction of the outer tube 394 is aligned with the hole of the inner tube 391, and the head of the pin 392 will pass through the hole of the inner tube 391 and the hole 395 of the outer tube 394 aligned with the hole of the inner tube 391 and extend outward in turn, in the form of a latch, to fix the outer tube 394 relative to the inner tube 391. The length adjustment of the support 123 is achieved by changing the position of the outer tube 394 of the pin 392 telescopic sleeve relative to the inner tube 391: first, press the head of the pin 392 into the inner tube 391 against the force of the spring 393, when the head of the pin 392 is roughly in the same plane as the hole 395 of the outer tube 394, quickly slide the outer tube 394 to the desired position, re-align another hole 395 of the outer tube 394 with the hole of the inner tube 391, and then allow the pin 392 to naturally release to extend the head of the pin 392 out of the hole of the inner tube 391 and the other hole 395 of the outer tube 394 aligned with the hole of the inner tube 391. At this time, the pin 392 fixes the outer tube 394 relative to the inner tube 391 at the new position. The telescopic support 123 makes the position adjustment of the camera assembly 120 more convenient, at the same time enhances the protection of the camera assembly 120, and prolongs the service life thereof. The support 123 can also be telescopic through other structures, or the telescopic structure is not purely mechanical structure, but a combination of mechanical and electrical, and is electrically connected with the processor of the intelligent mower 110. The processor can adjust the length of the support 123 to adjust the height of the camera assembly 120 according to the image information collected by the camera assembly 120. The present application is not limited to a specific embodiment, as long as the support 123 of the camera assembly 120 can be telescopic, it falls within the scope of the present application.

[0067] Further, in cooperation with the telescopic support 123, the main body 113 of the intelligent mower 110 can be provided with an inwardly recessed accommodating cavity 115, see Figures 4A-4BThe top opening of the receiving cavity 115 is located on the upper surface of the lawnmower body 113. The bracket 123 is fixed inside the receiving cavity 115 by fastening mechanisms such as screws and nuts. A cover plate 118 is located on the top of the receiving cavity 115, and the cover plate 118 can be opened and closed. For example, the cover plate 118 is hinged to one side of the top opening of the receiving cavity 115, including a first position when open. Figure 4B ) and the second position when closed ( Figure 4A Alternatively, the cover 118 consists of a sliding cover and a sliding cover guide that can slide back and forth, including a first position covering the top opening of the receiving cavity 115 and a second position exposing the opening of the receiving cavity 115. The advantage of the receiving cavity 115 and the cover 118 in conjunction with the retractable bracket 123 is that when the smart lawnmower 110 is not in use, the bracket 123 can be shortened and the cover 118 closed, allowing the camera assembly 120 to be hidden and stored inside the lawnmower body 113. This is not only neater and more aesthetically pleasing, but also waterproof, dustproof, and lightproof, reducing the frequency of camera cleaning and delaying aging. Before the smart lawnmower 110 is in operation, the cover 118 is opened and the bracket 123 is extended, allowing the camera assembly 120 to extend out of the receiving cavity 115 of the smart lawnmower 110 to capture images around the smart lawnmower 110. The specific shapes of the receiving cavity 115 and the cover plate 118 are not limited in this application; in addition, the specific position of the receiving cavity 115 can be determined according to the position of the motor, PCB board and other devices of the smart lawnmower 110, so as to facilitate the acquisition of image information around the smart lawnmower 110 and minimize the impact on the arrangement of the components inside the main body 113 of the smart lawnmower 110. There are no restrictions in this application, and Figure 4 is just an exemplary illustration.

[0068] In addition, the bracket 123 can also be configured as a foldable structure, see [reference needed]. Figures 5A-5B On the upper surface of the main body 113 of the smart lawnmower 110, a groove 117 is provided to accommodate the bracket 123 and the camera assembly 120. The bracket 123 is hinged to a point on the top surface of the main body 113 of the smart lawnmower 110, allowing the bracket 123 to rotate around the hinge point, overcoming a certain amount of friction when moved by hand. When not in use, the bracket 123 is rotated around the hinge point until it lies flat and stored in the groove 117 on the top surface of the main body 113 of the smart lawnmower 110. Figure 5A This improves aesthetics and tidiness, reduces the space required for storing the smart lawnmower 110, and enhances camera protection, extending its service life. During operation, the bracket 123 is erected, as... Figure 5BThe standing angle of the support can be adjusted as needed. Furthermore, a rotatable connecting mechanism such as a damping shaft structure or a ball structure can be used between the support 123 and the camera assembly 120, so that the user can freely adjust the angle of the camera assembly 120 as needed before starting the intelligent mower 110; or the rotatable connecting mechanism is not a pure mechanical structure, but a mechanical and electrical combination, and is electrically connected to the processor of the intelligent mower 110. The processor can automatically adjust the angle of the camera assembly 120 according to the image information collected by the camera assembly 120. It should be noted that the extension, folding, and rotation design of the support 123 of the camera assembly 120 above are examples and are not limited to the specific embodiments in the examples, and should not limit the protection scope of the present application according to the examples.

[0069] The camera assembly 120 can include a single or double (multiple) camera. In the ranging principle, monocular cameras and binocular (multiple) cameras are completely different. Binocular (multiple) cameras are similar to human eyes and mainly determine the distance through the parallax calculation of two (multiple) images collected by two (multiple) cameras at the same time. Therefore, binocular (multiple) cameras can perform depth estimation without relying on other sensing devices when stationary, but the depth range and accuracy are limited by the baseline (the distance between the optical centers of the two cameras) and the resolution of the binoculars, and the operation of parallax is quite resource-consuming, which has the disadvantages of complex configuration, large amount of calculation, and high energy consumption. The image frame collected by the monocular camera is a two-dimensional projection of the three-dimensional space, which loses the depth information of the environment. Only when the camera moves, the distance can be calculated through the parallax formed by the movement of the object on the image. This disadvantage can be alleviated to some extent by fusing the pose data collected by the inertial measurement unit. For example, the algorithm of the monocular visual fusion inertial measurement system (VINS-Mono) is widely used in devices that rely on positioning, such as robots and drones, due to its low cost, small size, and low power consumption. VINS-Mono can calculate the movement and rotation of the camera itself according to the shift of feature points between the front and rear frames captured by the camera, and is not limited by signal interference like GPS sensors. Therefore, the specific number of cameras included in the camera assembly 120 is not strictly limited in the present application.

[0070] In addition to the common single, double (multi-) camera, the camera assembly 120 can also include a depth camera, also known as an RGB-D camera. The biggest feature of the RGB-D camera is that it can measure the distance between the object and the RGB-D camera by actively emitting light to the object and receiving the returned light like a laser sensor through infrared structured light or Time-of-Flight (ToF) principle. Compared with the double (multi-) camera which calculates through software, the RGB-D camera obtains depth through physical measurement means, saving a lot of calculation. The commonly used RGB-D cameras at present include Microsoft's Kinect, Intel's RealSense, etc. But limited by the accuracy and measurement range of the sensor, the depth camera still has many problems such as narrow measurement range, large noise, small field of view, easy to be disturbed by sunlight, unable to measure transmissive materials, etc. Therefore, it is usually more applied to indoor scenes than outdoor scenes. If you want to apply the RGB-D camera on the intelligent mower 110, it is inseparable from the fusion with other sensors, and it is suitable to be used when the sunlight is not strong.

[0071] The inertial measurement unit 122 at least includes an accelerometer and a gyroscope. The accelerometer is a sensor used to measure linear acceleration. When the rigid body is in a stationary state relative to the earth, the linear acceleration is 0, but due to the influence of gravity, when the linear acceleration of the rigid body is measured using the accelerometer, there will be a reading of about 9.81 m / s 2 on the axis pointing vertically downward to the center of the earth; similarly, under the action of gravity, when the reading of the accelerometer on the rigid body is 0, the rigid body is in a free-fall state, and in fact there is a vertical downward 9.81 m / s 2The actual acceleration. Micro-Electro-Mechanical System (MEMS) sensors are widely used in smart home appliances. The internal structure of a MEMS accelerometer is a spring-mass microstructure. When there is an acceleration along the deformation axis of the micro spring-mass, the micro spring will deform. The deformation of the micro spring is measured using microelectronic methods, and the acceleration along the axis is measured. Due to such a structure, the MEMS accelerometer cannot measure the actual acceleration of a rigid body, but only gives the acceleration measurement along its measurement axis. In actual use, three sets of MEMS measurement systems are usually used to form a three-axis measurement system, which measures the acceleration components of the actual acceleration along the three orthogonal measurement axes, and calculates the actual acceleration from the acceleration components along the three orthogonal measurement axes. A gyroscope is a sensor used to measure the angular velocity of a rigid body. Similar to the MEMS accelerometer, the MEMS gyroscope can only measure the angular velocity component around a single measurement axis, so when used, it is also integrated and packaged as a three-axis gyroscope with three orthogonal measurement axes, which measures the rotation components of the angular velocity of the rigid body along the three measurement axes, and finally synthesizes the actual angular velocity of the rigid body. In the usual x-y-z coordinate system, the angle of rotation around the reference coordinate system x-axis is defined as the roll angle, the angle of rotation around the y-axis is defined as the pitch angle, and the angle of rotation around the z-axis is defined as the yaw angle.

[0072] Generally, an inertial measurement unit 122 includes three single-axis accelerometers and three single-axis gyroscopes, which measure the angular velocity and acceleration of an object in three-dimensional space and calculate the attitude of the object. Further, the inertial measurement unit 122 can also include a magnetometer. The magnetometer, also known as the geomagnetic, magnetic sensor, can be used to test the magnetic field strength and direction, and to locate the device. The principle of the magnetometer is similar to that of the compass, which can measure the angle between the current device and the four directions of east, south, west, and north. The six-axis or nine-axis sensor is an integrated sensor module, which reduces the circuit board and overall space. The data accuracy of the integrated sensor not only depends on the accuracy of the device itself, but also involves correction after welding assembly and supporting algorithms for different applications. Suitable algorithms can fuse data from multiple sensors to compensate for the shortcomings of individual sensors in calculating accurate positions and directions. Generally, the IMU sensor is preferably arranged at the center of gravity of the object; therefore, preferably, the inertial measurement unit 122 can be arranged at the center of gravity G of the smart mower 110, as shown in Figure 6 Due to the low cost of the inertial measurement unit 122, in an embodiment, two inertial measurement units 122 can also be arranged to improve the accuracy and stability of the IMU data, as shown in Figure 7As shown. On the one hand, the relative angular velocity and relative acceleration between the target object and the moving reference frame can be obtained based on the difference in output from the two inertial measurement units 122; on the other hand, the redundant design of the dual inertial measurement units 122 ensures the stability of positioning by monitoring the status of the two inertial measurement units 122 in real time and immediately switching to the other inertial measurement unit 122 when one inertial measurement unit 122 malfunctions.

[0073] The system diagram of the Smart Lawn Mower 110 is as follows: Figure 8 As shown, the system includes a power module 701, a sensor module 702, a control module 703, a drive module 704, and an actuator 705. The power module 701 supplies power to the drive module 704, control module 703, and sensor module 702. To meet the autonomous movement requirements of the intelligent lawnmower 110, the power module 701 preferably includes a battery pack providing direct current. The sensor module 702 includes at least a camera assembly 120 and an inertial measurement unit 122. The intelligent lawnmower 110 may also be equipped with other sensors such as GPS sensors, collision sensors, and drop sensors; information collected by these other sensors can also be comprehensively referenced during processing. The control module 703 includes: an input module 141 for receiving various raw data collected or detected by the sensor module 702; a processor 142 for logic operations, which can be a CPU or a microcontroller with high data processing speed; a memory 144 for storing various data and control programs 145; and an output module 143 for converting control commands into motor drive commands and sending them to the drive controller 161 of the motor drive switch. The drive module 704 includes a motor drive switch circuit 162, a drive controller 161, and a motor 163. Figure 8 The motor drive switching circuit 162 shown uses the most common MOSFET switch. The drive controller 161 controls the switching of the MOSFET switch by applying a voltage to its gate. The orderly switching of the MOSFET switch causes the motor windings to conduct in an orderly manner, thereby driving the motor 163 to rotate. Figure 8Only one common motor driving circuit is shown, and the disclosure is not limited to the specific implementation of the motor driving circuit. The rotation of the motor 163 directly or indirectly drives the actuator 705 through a transmission mechanism. The actuator 705 of the intelligent mower 110 mainly includes the blade 112 and the wheel 114, and optionally, the blade 112 and the wheel 114 are respectively driven by independent motors 163. Optionally, the left and right rear wheels 114 can also be respectively driven by independent motors 163, so as to realize more flexible turning and posture adjustment. The control program 145 stored in the memory 144 mainly consists of two modules, namely the positioning and mapping module 146 and the function application module 147, wherein the positioning and mapping module 146 is the basis of the function application module 147. The positioning and mapping module 146 solves the basic problems of where the intelligent mower 110 is, what the map is, and how the surrounding environment is, and tracks the position of the intelligent mower 110 and constructs the understanding of the real world when the intelligent mower 110 moves, that is, simultaneous localization and mapping (SLAM); based on the solution of the basic problems, the function application module 147 can realize specific functions such as demarcation of the mowing area boundary, intelligent obstacle avoidance, road surface recognition and selection, navigation combination, intelligent lighting, etc. Of course, this classification is mainly for understanding and elaboration, and in specific implementation, the positioning and mapping module 146 and the function application module 147 are not completely separated two parts, and the process of realizing the function application module 147 itself also deepens the understanding of the real world, and the result will also be fed back to the positioning and mapping module 146, so as to continuously improve the map.

[0074] For the intelligent mower 110, the implementation of simultaneous localization and mapping (SLAM) requires the fusion of image data from the camera assembly 120 and pose data from the inertial measurement unit 122 (also known as sensor fusion). The reason is that visual sensors such as cameras work well in most textured scenes, but if they encounter scenes with few features such as glass or white walls, they basically cannot work. Although the inertial measurement unit can measure angular velocity and acceleration, it must be time-integrated to obtain the position or attitude of the object, and moreover, the inertial components based on micro-electro-mechanical systems (MEMS) inevitably have system bias. The two factors together result in a very large cumulative error / drift over a long period of time, but for rapid motion in a short period of time, the relative displacement data has high accuracy. In rapid motion, the camera will have motion blur, or there will be too little overlap between two frames to perform feature matching. With the inertial measurement unit, a better pose estimate can be obtained even in the time period when the camera data is invalid. If the camera is fixed in place, the pose estimate obtained from visual information will also be fixed. Therefore, in slow motion, visual data can effectively estimate and correct the drift in the readings of the inertial measurement unit, so that the pose estimate after slow motion is still valid. As can be seen, the visual data and the IMU data are highly complementary, and the fusion of the data of the camera assembly 120 and the inertial measurement unit 122 can improve the accuracy and stability of positioning and mapping.

[0075] Because the types of data measured by the camera assembly 120 and the inertial measurement unit 122 (visual measurement of the coordinates of the object projected on the pixel plane, while the inertial measurement unit measures the three-dimensional acceleration and rotational angular velocity of the object) and the measurement rates (visual is subject to frame rate and image processing speed, and the sampling rate of the camera can only reach dozens of frames per second, while the inertial measurement unit can easily reach a sampling rate of hundreds or even thousands of frames per second) are quite different, additional errors will be introduced when the data of the two is fused, whether the motion measured by the inertial measurement unit is converted into object coordinates (the bias is accumulated when integrated) or the visual quantity is converted into motion (the calculated acceleration is greatly oscillated due to the positioning bias when differentiated). Therefore, detection and optimization need to be introduced in the data fusion process. Generally, compared to differentiating the visual quantity, the motion measured by the inertial measurement unit is usually integrated into the object coordinates before being fused with the visual quantity. For example Figure 9 As shown in the flowchart, the key modules in the entire flowchart can be divided into the following parts: image and IMU data preprocessing, initialization, local optimization, mapping, key frame extraction, loop detection, and global optimization. The main functions of each module are as follows:

[0076] Image and IMU data preprocessing: For the image frames collected by the camera assembly 120, feature points are extracted, and optical flow tracking is performed using KLT pyramids to prepare for subsequent visual-only initialization to solve the pose of the intelligent mower 110. For the IMU data collected by the inertial measurement unit 122, pre-integration is performed to obtain the current time pose, velocity, and rotation angle, and the pre-integrated increments between adjacent frames, as well as the pre-integrated covariance matrix and Jacobian matrix, are calculated for use in subsequent optimization.

[0077] Initialization: In the initialization, first, visual-only initialization is performed to solve the relative pose of the intelligent mower 110; then, the IMU pre-integration is aligned to solve the initialization parameters.

[0078] Local optimization: For the sliding window, visual-inertial local optimization is performed, i.e., visual constraints and IMU constraints are placed in a large objective function for nonlinear optimization. The local optimization only optimizes the variables in the current frame and the previous n frames (e.g., n is 4) in the window, and the local optimization outputs a more accurate pose of the intelligent mower 110.

[0079] Mapping: Through the obtained pose, the depths of the corresponding feature points are calculated using the triangulation method, and the current environment map is reconstructed simultaneously. In the SLAM model, the map refers to the set of all landmark points. Once the positions of the landmark points are determined, it can be said that the mapping is completed.

[0080] Key frame extraction: Key frames are selected image frames that can be recorded without redundancy. The selection criteria for key frames are that the displacement between the current frame and the previous frame exceeds a certain threshold or the number of matched feature points is less than a certain threshold.

[0081] Loop detection: Loop detection, also known as closed-loop detection, saves the previously detected image key frames. When the intelligent mower 110 returns to the same place it has been through before, it can determine whether it has been there through the matching relationship of the feature points.

[0082] Global optimization: Global optimization is performed when loop detection occurs, using visual constraints and IMU constraints, and adding loop detection constraints for nonlinear optimization. Global optimization is performed based on local optimization, and outputs a more accurate pose of the intelligent mower 110, and updates the map.

[0083] In the above algorithm, the local optimization is the optimization of the image frames in the sliding window, and the global optimization is the optimization of all key frames. Only using local optimization has low accuracy and poor global consistency, but high speed and high IMU utilization; only using global optimization has high accuracy and good global consistency, but slow speed and low IMU utilization; the combination of the two can complement each other, making the positioning result more accurate. The output pose is a 6 degree of freedom (6DoF) pose, which refers to the three-dimensional movement (translation) of the intelligent mower 110 in the x-y-z direction plus the pitch / roll / roll (rotation). In the fusion process, by aligning the IMU estimated pose sequence and the visual estimated pose sequence, the real scale of the trajectory of the intelligent mower 110 can be estimated, and the IMU can well predict the pose of the image frame and the position of the feature point in the next frame image, improve the matching speed of the feature tracking algorithm and the robustness of the algorithm to fast rotation, and finally the gravity vector provided by the accelerometer in the IMU can convert the estimated position into the world coordinate system required for actual navigation.

[0084] Compared with the 2D / 3D position output by the global positioning system (GPS) with poor accuracy (in meters), the 6 degree of freedom pose output by SLAM has high accuracy (in centimeters), and does not depend on the strength of satellite signals and is not disturbed by other electromagnetic signals. However, the process of SLAM has the problem of high energy consumption compared with the low operation and low power consumption of GPS positioning, and since the intelligent mower 110 works outdoors, the camera sensor needs to be cleaned frequently, and if it is not cleaned in time, it may cause the collected image frames to be blurred and cannot provide effective visual data. Moreover, in order to accurately solve the SLAM problem, the intelligent mower 110 needs to repeatedly observe the same area to realize closed-loop motion, so the system uncertainty is constantly accumulated until the closed-loop motion occurs. Especially when the lawn is wide and the surrounding is empty, there is a lack of feature reference, the intelligent mower 110 performs large closed-loop motion, and the system uncertainty may cause the failure of closed-loop checking, resulting in the failure of SLAM global optimization and large positioning deviation. In the environment where the lawn is wide and the surrounding is empty, the satellite signal interference is small, and the GPS positioning result is usually stable and accurate, and GPS is currently widely used and has low price, so the intelligent mower 110 can also be equipped with a GPS sensor and use GPS+SLAM combined navigation.

[0085] The combined positioning mode composed of the camera assembly 120, the inertial measurement unit 122, and the GPS can refer to Figure 10, first determine the reliability of each sensor data, when all sensors are failed, stop and send maintenance reminder; when two sensors are failed, use the remaining one sensor to navigate for a short period of time, such as 3s, and continuously detect whether the data validity of the failed sensor is restored during this period, and add the restored sensor data to the subsequent positioning and navigation calculation, if no other sensor is restored within this short period of time, stop and send maintenance reminder; when only one sensor is failed, use the remaining two sensors for positioning and navigation, if the GPS sensor is failed, use AR fusion visual-inertial SLAM for positioning and navigation, if the camera is failed, use IMU data to verify the self-consistency of GPS results, and filter and correct the absolute positioning data that cannot be self-consistent, if the IMU is failed, perform visual-inertial SLAM (VSLAM) and send the VSLAM result and the GPS positioning result at this time to the Kalman filter at the same time after processing each frame of image, and continuously detect whether the data validity of the failed sensor is restored, and add the restored sensor data to the subsequent positioning and navigation calculation, if the mowing work is completed, and the sensor is still not restored after returning to the charging station, an abnormal reminder is sent; when three sensors are working normally, use the GPS positioning result to filter and correct the pose and environment map generated by AR fusion visual-inertial SLAM.

[0086] In practical applications, the process of simultaneous localization and mapping (SLAM) can be realized by open-source AR software packages, and different application program interfaces (APIs) can be called to realize rich functions, for example, ARCore is a software platform for building augmented reality applications launched by Google, which realizes simultaneous localization and mapping (SLAM) based on the fusion of image data and IMU data, and its three major functions integrate virtual content with the real world seen through the camera: 1. Motion tracking: allows machines to understand and track their position and pose relative to the real world; 2. Environment understanding: allows machines to detect various surfaces (such as ground, table, wall, etc. horizontal or vertical surfaces) through feature point clustering, and know their boundaries, size and location; 3. Light estimation: allows machines to estimate the current lighting conditions of the environment. In addition to Google's ARCore, Apple's ARKit and Huawei's AR Engine are also software packages that can provide similar functions.

[0087] In an embodiment, the function application module 147 of the control program 145 of the intelligent mower 110 can distinguish the grassland from the non-grassland according to the feature points of the two-dimensional plane in the image frame, and stop the rotation of the blade 112 if the working surface where the mower is currently located is not the grassland, in comparison with the texture features of the grassland; and generate the boundary of the mowing area along the boundary between the grassland and the non-grassland automatically in combination with the motion tracking function of the software package such as ARCore. Further, the intelligent mower 110 can also cooperate with the interactive display interface to display the constructed map and the boundary of the mowing area through the interactive display interface, and let the user confirm and modify. In the confirmation process, in order to facilitate the user to identify the boundary line more intuitively and carefully, two identification modes can be set. One identification mode is to display the boundary line of the mowing area on the two-dimensional map on the interactive display interface, see Figure 11A , in the two-dimensional map, the lawn 222 is located between the house 223 and the road 224, and the boundary line 221 of the mowing area is represented by a thick dashed line. The user can manually adjust the boundary line 221 in the two-dimensional map on the interactive display interface, for example, drag a section of the boundary line 221 up, down, left or right, or delete, add (draw with a finger) a section of the boundary line 221. If the user wishes, the user can also choose to directly enter this identification mode and draw all the boundary lines 221 on the two-dimensional map on the interactive display interface with a finger. The other identification mode is to display the virtual fence 211 icon superimposed on the real-time image collected by the camera assembly 120 on the interactive display interface, see Figure 11BIn this recognition mode, the boundary line automatically generated by the intelligent mower 110 is displayed in the form of a virtual fence 211 icon. The user can manually adjust the position of the virtual fence 211 icon superimposed on the real image on the interactive display interface, for example, pull or push the virtual fence 211 closer or farther away, or delete and add a segment of the virtual fence 211. Moreover, with the motion tracking function of software packages such as ARCore, the user can check the appropriateness of the virtual fence 211 from various angles during the movement and angle switching of the camera assembly 120. Compared with the boundary line 221 on the two-dimensional map, the virtual fence 211 icon superimposed on the real image is more intuitive and accurate, and it is convenient for the user to determine the accurate position of the virtual fence 211 (i.e., the boundary line) according to the specific ground conditions (e.g., terrain, vegetation type). During the confirmation process, the user can combine the two modes, first overall check whether the boundary line on the two-dimensional map meets the expectations, and adjust it if it does not. Then, for the boundary that needs special attention, check the virtual fence 211 icon superimposed on the real image, and fine-tune it if necessary. When the boundary of the mowing area is confirmed by the user, the intelligent mower 110 will store the confirmed boundary line (including the virtual fence 211) in the form of discrete anchor point coordinates. The position of the boundary line (discrete anchor points) will not change with the movement of the intelligent mower 110, and the intelligent mower 110 will work within the boundary of the mowing area during path planning. It is worth noting that the interactive display interface can be a component on the intelligent mower 110, a separate display device, or the interactive display interface of a mobile terminal such as a mobile phone or tablet that can interact with the intelligent mower 110.

[0088] In an embodiment, the function application module of the control program 145 of the intelligent mower 110 can identify different plane materials. In addition to identifying lawns and non-lawns, the intelligent mower 110 can also analyze the feature points of the two-dimensional plane in the image frames collected by the camera assembly 120, and according to the different plane textures (i.e., feature point distribution rules), compare with the texture characteristics of common types of planes preset in the control program 145, to identify different types of ground (including water surface). If the intelligent mower 110 simultaneously walks on different material grounds, due to the different support force and friction of the wheels 114 of the intelligent mower 110 on different hardness and different material grounds, it is easy to cause the intelligent mower 110 to bounce, tilt, and direction to be skewed, etc. Therefore, when the intelligent mower 110 walks on a non-lawn, for example, during the journey from one lawn to another, and identifies that there are multiple grounds with different feature point textures (i.e., different hardness) in the front area 212, it selects to walk on one of the grounds with greater hardness. See Figure 12When the intelligent lawnmower 110 detects multiple road surfaces in the area 212 directly in front of it—cement and dirt—with the cement surface on the left and the dirt surface on the right, the road selection program in control program 145 plans a path, controlling the intelligent lawnmower 110 to adjust its direction and travel to the left until it detects that the area 128 directly in front of it is entirely cement. Then, it adjusts its direction back to the original planned direction. This road selection is beneficial for the intelligent lawnmower 110's movement control, machine maintenance, and safety. In the road selection program, the environmental understanding function of software packages such as ARCore can be used to classify surfaces of different materials, and the texture features of common planes can be compared to assist the intelligent lawnmower 110 in determining the plane type. After determining the plane type, the program selects the harder surface according to the ground type-hardness comparison table stored in memory and controls the intelligent lawnmower 110's direction of travel accordingly. In addition, by comparing the texture features with common planes and judging the positional relationship between planes, the smart lawnmower 110 can identify terrain such as water surfaces, steps, and cliffs that may pose a risk of falling and damage to the smart lawnmower 110, making the function of automatically generating the boundary of the mowing area more complete.

[0089] In one embodiment, the functional application module of the control program 145 of the intelligent lawnmower 110 may further include an AI object recognition program, which calculates the category information of obstacles from the image data acquired by the camera component 120, thereby enabling the intelligent lawnmower 110 to actively and intelligently avoid obstacles. Different obstacle avoidance strategies and appropriate avoidance distances are adopted for different categories of obstacles to balance mowing coverage and obstacle avoidance success rate. Figures 13A-13B For a selected object, the object recognition program outputs a category and its corresponding confidence probability (C:P), where the confidence probability P ranges from 0 to 1. The control program 145 may also include a confidence threshold P1, for example, P1 = 0.7. It will accept judgments greater than the confidence threshold and proceed to select an obstacle avoidance strategy, such as... Figure 13A (bird: 0.99); however, judgments based on values ​​less than or equal to the confidence threshold are not accepted, such as... Figure 13B In the cases of (bird: 0.55) and (bird: 0.45), if the distance D between the obstacle and the smart lawnmower 110 is greater than the recognition threshold distance D3, then the robot continues to drive normally and uses the next frame or the next n frames of images for object recognition. The control program 145 makes a higher confidence probability object recognition judgment as the smart lawnmower 110 approaches the obstacle. If the distance D between the obstacle and the smart lawnmower 110 is less than or equal to the recognition threshold distance D3, then a long-distance avoidance strategy is adopted, for example, driving around the obstacle at a distance of 0.5m.

[0090] like Figure 14As shown, different obstacle avoidance strategies are taken according to the category of the obstacle. If the detected obstacle is a leaf, branch, pinecone, or even animal excrement, which are materials that can be cut by the blade 112 and can naturally decompose, the intelligent mower 110 can ignore the obstacle and continue to travel along the original path. Among them, although animal excrement is likely to dirty the blade 112 and chassis of the intelligent mower 110, similar to soil, these dirt will be more or less cleaned up in frequent cutting, so there is no need to avoid. If the detected obstacle is an animal, such as a person, bird, squirrel, dog, etc., a first threshold distance D1 and a second threshold distance D2 can be preset. When the distance D between the intelligent mower 110 and the detected animal obstacle is greater than the first threshold distance D1, the intelligent mower 110 travels normally along the original path. When the distance D between the intelligent mower 110 and the detected animal obstacle is less than or equal to the first threshold distance D1 and greater than the second threshold distance D2, the intelligent mower 110 travels at a slower speed and emits a warning sound to prompt the animal, such as a person, bird, squirrel, dog, etc., to find the intelligent mower 110 and actively avoid it. When the distance D between the intelligent mower 110 and the detected animal obstacle is less than or equal to the second threshold distance D2, in order to avoid accidentally causing harm to people and animals, the intelligent mower 110 adopts a long-distance avoidance strategy. If the detected obstacle is a plastic toy, shovel, rope, or other movable (temporary) small-volume object, in order to avoid accidentally causing damage to these small-volume objects, the intelligent mower 110 can maintain a distance to avoid, or in other words, adopt a long-distance avoidance strategy, and send a cleaning prompt to the user to clean up the small-volume objects on the lawn. In addition, for animal obstacles and movable (temporary) obstacles, the intelligent mower 110 can store the coordinates of the obstacle and the coordinates of the avoidance area while taking avoidance actions, and before the mowing is completed, if the image data collected by the camera assembly 120 shows that the obstacle at the obstacle coordinate has been removed, the intelligent mower 110 plans a return path and mows the previously avoided area. If the detected obstacle is a tree, garden furniture (e.g., a bench, a swing), or other immovable (permanent) large-volume object, the intelligent mower 110 can adopt a close-distance avoidance strategy, i.e., slowing down and trying to get close to the obstacle as much as possible, to maximize the mowing coverage, for example, to travel around the obstacle at a distance of 0.1 m, or when the intelligent mower 110 is equipped with a collision sensor, a slight collision at a slow speed will not cause much damage to these large-volume objects, so the closest distance avoidance can be achieved through the collision sensor. At the same time, the intelligent mower 110 can store the actual avoidance path and optimize it when the processor 142 is idle, so that the next time the same obstacle is avoided, the mowing coverage is maintained while the efficiency of the avoidance path is improved.

[0091] In addition to identifying real obstacles from the images acquired by the camera assembly 120, the user can also manually superimpose virtual obstacles 215 on the real-time images captured by the camera assembly 120 displayed on the interactive display interface and adjust the orientation, size, and dimensions of the virtual obstacles 215, as shown. Figure 15 With the motion tracking function of software packages such as ARCore, the user can check the appropriateness of the virtual obstacles 215 from various angles during the movement and angle transformation of the camera assembly 120. The position and size information of the virtual obstacles 215 will be recorded in the form of anchor points, and the virtual obstacles 215 will not change with the movement of the smart mower 110. In this way, when the smart mower 110 moves in the real working area, it can compare its current position with the position information of the virtual obstacles 215 in real time and avoid obstacles to avoid "collision" with the virtual obstacles 215. The function of the virtual obstacles 215 facilitates the user to customize special mowing ranges according to specific circumstances. For example, there is a flower bed without a fence on the lawn, which looks like an ordinary lawn in some seasons. In order to avoid the smart mower from stepping into the flower bed when mowing, the user can add a virtual obstacle 215 with the same bottom area as the actual flower bed area on the real-time image of the flower bed captured by the camera assembly 120 displayed on the interactive display interface. For another example, there is a dog house on the lawn. A large dog house will be automatically determined as a large-volume item that cannot be moved by the control program 145 described above and a close-range obstacle avoidance strategy will be adopted to improve the mowing coverage. However, considering that the dog may be in the dog house, in order to avoid the operation of the smart mower 110 from disturbing and scaring the dog, the user can superimpose virtual obstacles 215 or virtual fences 211 around the image of the dog house captured by the camera assembly 120 displayed on the interactive display interface to enclose a larger non-working area. Further, since ARCore can track trackable objects such as planes and feature points over time, the virtual obstacles can also be anchored to specific trackable objects to ensure that the relationship between the virtual obstacles and the trackable objects remains stable. For example, anchoring the virtual obstacle 215 to the dog house, then moving the dog house later, the virtual obstacle 215 will track the movement of the dog house without the need for the user to re-set the virtual obstacle.

[0092] In an embodiment, the function application module of the control program 145 of the smart mower 110 can detect the light state of the surrounding environment. With the light estimation function of software packages such as ARCore, the smart mower 110 can know the light intensity L of the surrounding environment and adjust the lighting lamp 119 of the smart mower 110 accordingly. The control program 145 can preset a first light intensity threshold L1, when the light intensity L of the surrounding environment is less than the first light intensity threshold L1, the smart mower 110 turns on the lighting lamp 119 to compensate for light. In addition, different working modes can also be set, according to the light intensity and direction, reasonably arrange the time of mowing and select different working modes. For example, when it is detected that the light of the surrounding environment is very weak, for example, when the light intensity L of the surrounding environment is less than the second light intensity threshold L2 (L2 < L1), if the user does not command to mow immediately, return to the charging station, enter the charging mode or standby mode, because the lawn is most vulnerable to fungal and pest damage when there is no light; if the user commands to mow immediately, turn on the lighting lamp 119 and mow in silent mode to reduce the disturbance of the mower noise to the quiet night. When it is detected that the light of the surrounding environment is very strong, for example, when the light intensity L of the surrounding environment is greater than the third light intensity threshold L3 (L3 > L1), if the user does not command to mow at this time, return to the charging station, enter the charging mode or standby mode, because strong sunlight can easily scorch the cut grass; if the user commands to mow immediately, mow in fast mode to reduce the time of the mower exposed to the sun to reduce aging caused by UV radiation. When it is detected that the light of the surrounding environment is suitable, for example, when the light intensity L of the surrounding environment is greater than or equal to the first light intensity threshold L1 and less than or equal to the third light intensity threshold L3, the mower can be mowed in the normal mode.

[0093] In addition to the light state of the environment, the image data collected by the camera assembly 120, combined with AI object recognition operation, can also be used as a basis for judging the mowing time and mode selection. For example, when it is detected that there is dew on the vegetation, if the user does not command to mow immediately, return to the charging station, enter the charging mode or standby mode, because the dew will reduce the cutting efficiency, and even cause stall, in addition, the wet lawn and the like are easy to leave tire marks, affecting the appearance. When it is detected that there is frost or snow on the vegetation, if the user does not command to mow immediately, return to the charging station, enter the charging mode or standby mode, because the cold weather is not conducive to the recovery of the cut grass cut.

[0094] It is worth mentioning that AR software packages such as ARCore do not have good object recognition capabilities by themselves, for example, the environment understanding function of ARCore itself detects, distinguishes and demarcates 2D surfaces by clustering feature points on the plane, rather than judging the surface of the object by object recognition. Even if the control program 145 of the intelligent mower 110 introduces some texture features of common types of planes to assist in plane type judgment, it is still far from real object recognition. Therefore, in actual use, the implementation of functions such as obstacle recognition and environment recognition still needs to rely on other AI software packages with object recognition functions, for example, TensorFlow of Google company, among which TensorFlow Lite is a set of tools that help developers run TensorFlow models on mobile devices, embedded devices and IoT devices. It supports device-side machine learning inference (without sending data back and forth between devices and servers), has low latency, and the binary file is small. Of course, the intelligent mower 110 can also include a wireless network connection device 150 to hand over the object recognition work to the cloud server 200. Since the cloud server 200 has powerful cloud storage and cloud computing functions, it can use the TensorFlow framework to continuously improve the training set and model, thereby giving more accurate judgments.

[0095] In fact, when the intelligent mower 110 includes the wireless network connection device 150, the control program 145 can send the fusion operation of the visual data and the IMU data, and even the operation tasks of the entire positioning and mapping module 146 and the functional application module 147 to the cloud server 200 for processing. The cloud server 200 fuses, positions, maps, judges, and generates navigation and mowing instructions according to the preset program based on the uploaded data. At this time, the control program 145 of the intelligent mower 110 only needs to be responsible for obtaining data from the camera 120 and the inertial measurement unit 122, pre-processing and uploading the obtained data, and downloading instructions and outputting from the cloud server 200, without performing complex AR and / or AI operations, reducing the requirements for the processor 142 of the intelligent mower 110 and saving chip costs. Similarly, when the intelligent mower 110 includes the wireless network connection device 150, the control program 145 can also send the fusion operation of the visual data and the IMU data, and even the operation tasks of the entire positioning and mapping module 146 and the functional application module 147 to other devices that can perform wireless data transmission with the intelligent mower 110, such as the application program of a mobile terminal. At this time, the control program 145 of the intelligent mower 110 can be understood as providing an application program interface (API) to realize the communication function between the intelligent mower 110 and the mobile terminal and define the data communication protocol and format between the intelligent mower 110 and the application program of the mobile terminal. Through this application program interface, the application program of the mobile terminal can obtain image and pose data from the intelligent mower 110, and generate navigation and mowing instruction data after a series of high-complexity AR and / or AI operations according to the preset program, and then return the instruction data to the intelligent mower 110 through the application program interface, thereby realizing the control of the intelligent mower 110 by the mobile terminal. The application program of the mobile terminal can also provide parameters that can be selected and modified by the user, such as mowing time preference, mowing height preference, etc., to facilitate the user to obtain customized intelligent control of the intelligent mower 110 according to their own needs. Therefore, reserving the application program interface on the intelligent mower 110 not only reduces the requirements for the processor 142 of the intelligent mower 110 and saves chip costs, but also facilitates the user to control the intelligent mower 110 through other devices.

[0096] In another embodiment, the camera for collecting image information can also be installed in the environment scene. For example, referring to Figure 16 The intelligent mower 210 itself does not have a camera, and instead one or more cameras 190 are installed on the roof and / or the top of the charging pile 180. Since there is no need to install a bracket or reserve a storage cavity, the housing structure of the intelligent mower 210 is more flexible, for example, Figure 16The smart mower 210 shown uses the appearance design of the power head, which is modern and beautiful. One or more cameras 190 arranged in the scene have a wireless connection device 191 for wireless connection with the smart mower 210 or connection to a wireless network, such as a user's home wifi network, to upload the collected image data to the cloud server 200. One or more cameras 190 can use a rotatable camera commonly available on the market to obtain a wider viewing angle and more accurate positioning. The main components of the smart mower 210 are similar to the smart mower 110, and the same components of the two will not be repeated here. The main difference between the two is that the smart mower 210 does not have a camera arranged directly on the main body or mounted on the main body through a support or other connecting mechanism to move synchronously with the smart mower 210. Moreover, the smart mower 210 has a wireless connection device 250 that can receive image data sent by one or more cameras 190 or can access the Internet and interact with the cloud server 200 to exchange data. It is worth noting that for the smart mower 110 in the previous embodiment, since the sensors (camera assembly 120, inertial measurement unit 122, etc.) are integrated into the mower main body 113, the sensors and the control module are connected by wire, so the wireless connection device 150 is not necessary, but for the purpose of improving computing power, upgrading convenience, using big data, and reducing chip cost, the smart mower 110 can also have a wireless connection device 150 such as a wireless network card, a mobile network receiver, etc. For the smart mower 210 in this embodiment, since the camera 190 is separated from the main body of the smart mower 210, the data transmission between them depends on wireless connection, so one or more cameras 190 and the smart mower 210 both rely on wireless connection devices (the camera 190 includes a wireless connection device 191, and the smart mower 210 includes a wireless connection device 250) to realize wireless transmission, for example, one or more cameras 190 respectively send the collected image data to the smart mower 210 for operation processing.

[0097] The high-level architecture of the control module of the intelligent mower 210 can refer to the intelligent mower 110 of the previous embodiment, but since the image information collected by the one or more cameras 190 arranged in the scene has a different perspective from the image information collected by the camera assembly 120 located on the intelligent mower 110, the control program 245 of the intelligent mower 210 is also different from the control program 145 of the intelligent mower 110: the control program 245 of the intelligent mower 210 mainly estimates the position of the intelligent mower 210 in the camera visible area using a visual target tracking algorithm, and generates navigation and mowing instructions accordingly. The one or more cameras 190 can send raw image data or processed data to the intelligent mower 210. When there is only one camera 190, the control program 245 of the intelligent mower 210 uses a single-view target tracking algorithm to estimate its own position; when there are multiple cameras 190, the control program 245 of the intelligent mower 210 uses a multi-view target tracking algorithm to estimate its own position. The multi-view target tracking algorithm includes a centralized multi-view target tracking algorithm and a distributed multi-view target tracking algorithm: under the centralized technology, the data transmission mode between the multiple cameras 190 and the intelligent mower is as follows Figure 17A ; under the distributed technology, the data transmission mode between the multiple cameras 190 and the intelligent mower is as follows Figure 17B . Figure 17A The intelligent mower 210 in the above-mentioned distributed multi-view target tracking algorithm actually plays the role of the Fusion Center in the centralized multi-view target tracking algorithm, and each camera 190 sends the collected image data to the intelligent mower 210 for operation and processing. Figure 17B In the above-mentioned distributed multi-view target tracking algorithm, each camera 190 completes the collection and processing of video data locally, and interacts and fuses information with other cameras 190 through the network. For example, each camera 190 fuses the position estimate calculated from the image collected by itself and the position estimate obtained from the adjacent camera 190 to obtain a new position estimate, and sends the new position estimate to the next adjacent camera 190 until the desired accuracy is reached, and then the camera 190 that reaches the desired accuracy sends the position estimate to the intelligent mower 210. The control program 245 of the intelligent mower 210 generates navigation and mowing instructions according to the obtained position estimate and the information of other sensors of the intelligent mower 210 (if any). Compared with the centralized technology, the distributed technology has the advantages of low bandwidth requirement, small system power consumption, high real-time performance, and strong reliability. The distributed multi-view target tracking algorithm reduces the requirement for the processor chip of the intelligent mower 210, but the data processing capability of the camera 190 is improved, which is suitable for the case where the lawn is large and the scene is complex, and a large number of cameras 190 are used; while the centralized multi-view target tracking algorithm is suitable for the case where the lawn is small and the scene is simple, and a small number of cameras 190 are used.

[0098] Alternatively, one or more cameras 190 and the smart lawnmower 210 may each be equipped with a wireless connectivity device 191 capable of accessing the internet, such as a wireless network card or a mobile network receiver, and integrate and compute data from multiple devices through a cloud server 200. One or more cameras 190, the smart lawnmower 210, and the cloud server 200 can be connected via... Figure 17C The architecture facilitates data interaction. One or more cameras 190 each upload the raw image data or pre-processed data they have collected to the cloud server 200. Based on the data from the one or more cameras 190, the cloud server 200 selects a single-view target tracking algorithm or a multi-view target tracking algorithm. After calculating the real-time position estimate of the smart lawnmower 210, it sends the corresponding positioning estimate and map information to the smart lawnmower 210. The control program 245 of the smart lawnmower 210 then integrates data from other sensors (if any) to generate navigation and mowing commands. Alternatively, the smart lawnmower 210 can also upload data collected by its other sensors to the cloud server 200 via a wireless network. After calculating the real-time position estimate of the smart lawnmower 210, the cloud server 200, based on the preset program stored in the cloud server 200 and the other sensor data uploaded by the smart lawnmower 210, directly issues navigation and mowing commands corresponding to the current situation and sends them to the smart lawnmower 210.

[0099] This application also proposes a lower-cost solution, namely an intelligent lawnmower system 100, comprising an intelligent lawnmower 310 and a mobile terminal 130. The mobile terminal 130 can be a mobile phone, tablet, or wristband, or any device equipped with a camera, inertial measurement unit (IMU), and computing unit. Since the mobile terminal 130 provides the camera and IMU, the intelligent lawnmower 310 itself does not need to include a camera or IMU, reducing production costs. Data transmission between the intelligent lawnmower 310 and the mobile terminal 130 can be achieved through wired or wireless communication. Figure 18 As shown, the intelligent lawn mowing system 100 can adopt an intelligent lawnmower 310, including: a cutting blade 312 for cutting grass; a main body 313 for mounting the cutting blade 312; wheels 314 for rotating and supporting the main body 313; a fixing device 316 disposed on the main body 313 for fixing the mobile terminal 130 to the intelligent lawnmower 310; an interface 311 disposed on the main body 313 for cooperating with the interface 131 of the mobile terminal 130 to form a wired connection for data transmission; and a controller (not shown) electrically connected to the interface 311, which controls the behavior of the intelligent lawnmower 310 according to the instruction data received by the interface 311 when the interface 311 is connected to the mobile terminal 130.

[0100] In an embodiment, the structure of the fixing device 316 is described with reference to Figures 19A-19C , Figure 19A The fixing device 316 includes a first baffle 381, a second baffle 382, a support plate 383, a support rod 384 and a base 385. The first baffle 381 and the second baffle 382 are parallel, respectively located at both ends of the support plate 383 and protrude outward from the same side of the support plate 383 to form opposite hooks, thereby facilitating the fixation of the mobile terminal 130 such as a mobile phone or a tablet between the first baffle 381 and the second baffle 382. Specifically, the surfaces of the support plate 383, the first baffle 381 and the second baffle 382 that contact the mobile terminal 130 such as a mobile phone or a tablet are also provided with a silica gel lining, which increases the friction between the support plate 383, the first baffle 381 and the second baffle 382 and the mobile terminal 130 such as a mobile phone or a tablet, preventing the mobile terminal 130 such as a mobile phone or a tablet from being shaken off due to bumps caused by uneven ground during the travel of the intelligent mower 310. At the same time, the silica gel lining also has a certain elasticity, which can buffer the collision between the mobile terminal 130 such as a mobile phone or a tablet and the support plate 383, the first baffle 381 and the second baffle 382 during the bumps, reducing the wear of the mobile terminal 130 such as a mobile phone or a tablet and the support plate 383, the first baffle 381 and the second baffle 382, and improving the service life. The lining material of the support plate 383 and the first baffle 381 and the second baffle 382 is not limited in this document, and various silica gel, rubber and other materials can be used as long as they have the functions of anti-skid and buffering.

[0101] As Figures 19B-19CAs shown, the distance between the first baffle 381 and the second baffle 382 is L1, for example, 10 cm, when the mobile terminal 130 is not installed, which is suitable for the size of the mobile terminal 130 such as mobile phones and tablets on the market (at present, most mobile terminals such as mobile phones and tablets are between 4 inches and 12 inches), and the distance between the first baffle 381 and the second baffle 382 can be changed, in other words, the second baffle 382 can be translated relative to the first baffle 381, or the first baffle 381 can be translated relative to the second baffle 382, so that the distance between the two baffles changes, thereby firmly clamping mobile terminals 130 of different sizes such as mobile phones and tablets. For example, by providing a tension spring 386 and an extension rod 387 on the back of the support plate 383, the first baffle 381 can be translated in the direction away from or close to the second baffle 382. For the sake of description, the movement of the first baffle 381 in the direction away from the second baffle 382 is called outward stretching, and the movement of the first baffle 381 in the direction close to the second baffle 382 is called inward contraction. Specifically, the second baffle 382 is fixedly connected to the support plate 383, and the first baffle 381 is fixedly connected to the top end of the extension rod 387 away from the second baffle 382 on the back of the support plate 383. One end of the tension spring 386 is connected to the second baffle 382, and the other end is connected to the end of the extension rod 387 close to the second baffle 382, so the tension of the tension spring 386 always pulls the extension rod 387 towards the second baffle 382, even if the extension rod 387 is inwardly contracted. The whole composed of the support plate 383, the telescopic mechanism and the first and second baffles 382 can also be called a chuck.

[0102] When the mobile terminal 130 is not installed, the tension spring 386 pulls the extension rod 387 toward the second baffle 382 until the first baffle 381 abuts against the end of the support plate 383. At this time, the first baffle 381 is fixed in the first position abutting against the end of the support plate 383 under the tension of the tension spring 386 and the reaction force of the contact surface at the end of the support plate 383. When installing a mobile terminal 130 such as a mobile phone or tablet, the user first grasps the first baffle 381 and pulls the extension rod 387 outward. Then, the mobile terminal 130 is placed flat on the support plate 383, between the first baffle 381 and the second baffle 382. The user then releases the first baffle 381, causing it and the extension rod 387 to retract inward under the tension of the tension spring 386 until the first baffle 381 touches the edge of the mobile terminal 130. At this point, the first baffle 381 is fixed in a second position against the edge of the mobile terminal 130 under the tension of the tension spring 386 and the reaction force of the contact surface at the edge of the mobile terminal 130. It is understood that when clamping mobile terminals 130 of different sizes, there will be multiple second positions with slightly different specific locations. Here, we collectively refer to these positions fixed against the edge of the mobile terminal 130 as the second positions of the first baffle 381. The maximum distance between the first baffle 381 and the second baffle 382 is L2, and the difference between L2 and L1 is ΔL. ΔL represents the extension and retraction of the clamp of the fixing device 316. For example, L2 can be 19cm, then ΔL is 9cm. This fixing device 316 of the mobile terminal 130 can fix mobile terminals 130 such as mobile phones and tablets with a width or length between 10cm and 19cm. In actual use, if the mobile terminal 130 is small, such as a mobile phone, the mobile phone can be clamped vertically between the first baffle 381 and the second baffle 382, ​​that is, the first baffle 381 and the second baffle 382 clamp the longer side of the mobile phone; if the mobile terminal 130 is large, such as a tablet computer, the tablet computer can be clamped horizontally between the first baffle 381 and the second baffle 382, ​​that is, the first baffle 381 and the second baffle 382 clamp the shorter side of the tablet computer. There are many clamps on the market, and although their structures differ, many of them can firmly clamp mobile terminals 130 of different sizes. Due to their wide use and low price, this application does not limit the specific structure of the clamp, as long as it can firmly clamp mobile terminals 130 of different sizes.

[0103] The base 385 of the fixing device 316 can be directly fixed to the surface of the main body 313 of the intelligent lawnmower 310 by fastening mechanisms such as screws and nuts, for example. Figure 18 As shown, this design requires minimal structural modifications to existing smart lawnmowers and is low-cost; however, it falls short in terms of aesthetics and tidiness. Alternatively, as... Figure 20The main body 313 of the intelligent mower 310 is provided with an inwardly recessed accommodating cavity 315, the top opening of the accommodating cavity 315 is located on the upper surface of the main body 313 of the intelligent mower 310, the base 385 of the fixing device 316 is fixed in the accommodating cavity 315 by a fastening mechanism such as a screw nut, and the accommodating cavity 315 is provided with a cover plate 318 which can be opened and closed. For example, the cover plate 318 is hinged on one side of the top opening of the accommodating cavity 315 and includes a first position when opened and a second position when closed. Alternatively, the cover plate 318 is composed of a sliding cover and a sliding cover guide rail which can slide back and forth, including a first position covering the top opening of the accommodating cavity 315 and a second position exposing the opening of the accommodating cavity 315. The advantages of the accommodating cavity 315 and the cover plate 318 are that the fixing device 316 can be hidden and stored in the main body 313 of the intelligent mower 310 when the intelligent mower 310 is not in use, which is neat and beautiful on the one hand, and can prevent water, dust and light from entering on the other hand, reducing the cleaning needs of the fixing device 316 and delaying aging. As shown in Figure 20 the interface 311 can also be arranged on the inner wall of the accommodating cavity 315, thereby reducing the invasion of dust, water and other substances. The specific form of the accommodating cavity 315 and the cover plate 318 is not limited in the present application; in addition, the specific position of the accommodating cavity 315 can be determined according to the position of the motor, PCB board and other devices of the intelligent mower 310 to facilitate the collection of image information around the intelligent mower 310 and minimize the influence on the arrangement of the internal elements of the main body 313 of the intelligent mower 310, and it is appropriate to minimize the influence, Figure 20 only for exemplary display.

[0104] During non-working hours, the fixing device 316 of the mobile terminal 130 is concealed within the main body 313 of the smart lawnmower 310. Therefore, before the smart lawnmower 310 with the mobile terminal 130 in operation, the clamp of the fixing device 316 needs to extend beyond the main body 313 of the smart lawnmower 310 so that the camera 132 of the mobile terminal 130 can collect image information around the smart lawnmower 310. To achieve this, the support rod 384 of the fixing device 316 can be designed as a telescopic structure, for example, referring to the inner and outer double tube structure of the bracket 123 in the first embodiment. Before the smart lawnmower 310 with the mobile terminal 130 in operation, the inner tube of the support rod 384 is pulled outward, increasing the length of the entire support rod 384, thereby allowing the clamp to extend beyond the main body 313 of the smart lawnmower 310. When the smart lawnmower 310 is not in operation or is not equipped with the mobile terminal 130, the inner tube of the support rod 384 is pushed back inward, shortening the overall length of the support rod 384 so that it is completely housed in the receiving cavity 315 of the smart lawnmower 310. This application does not limit the specific telescopic structure of the support rod 384 of the fixing device 316, as long as it can achieve the effect of extension and retraction. Other structures that achieve similar effects, such as flexible or foldable support rods 384, also fall within the protection scope of this application.

[0105] from Figure 19A It can also be seen that a rotatable connection is formed between the support rod 384 and the clamp via a damping pivot structure or a ball bearing structure 388. The advantage of this is that when the smart lawnmower 310 is equipped with the mobile terminal 130, the user can freely adjust the angle of the clamp according to the actual working conditions and the specific position of the camera 132 of the mobile terminal 130. This adjustment is also the angle at which the mobile terminal 130 is fixed, i.e., the angle at which the camera 132 of the mobile terminal 130 collects image information of the environment surrounding the smart lawnmower 310. This application does not limit the specific structure of the rotatable connection; it only needs to achieve the rotatable effect. In some examples, the support rod 384 consists of multiple short rods connected sequentially, which can be folded to save space and the angle of the clamp can be adjusted using the hinge points between the short rods. With the aid of the fixing device 316, when the mobile terminal 130 is fixed on the main body 313 of the smart lawnmower 310, and the position of the mobile terminal 130 is stationary relative to the smart lawnmower 310, it can be considered that the image information of the surrounding environment collected by the camera 132 of the mobile terminal 130 is the image information of the surrounding environment of the smart lawnmower 310, and the pose information collected by the inertial measurement unit 133 of the mobile terminal 130 is the pose information of the smart lawnmower 310.

[0106] See Figures 21A-21CThe mobile terminal 130 comprises a camera 132 for collecting image data of the environment around the intelligent mower 310, an inertial measurement unit 133 for detecting position and attitude data of the intelligent mower 310, an interface 131 for data transmission and charging, a memory (not shown) for storing an application program 135 for controlling the operation of the intelligent mower 310, and a processor (not shown) electrically connected to the camera 132 and the inertial measurement unit 133, for calling the application program 135 and calculating and processing the information collected by the camera 132 and the inertial measurement unit 133. The processor can call the application program 135 to fuse the data obtained by the camera 132 and the inertial measurement unit 133 to realize simultaneous localization and mapping (SLAM) of the intelligent mower 310, and generate corresponding navigation and mowing instructions according to the preset logic and real-time data to control the behavior of the intelligent mower 310. Common mobile terminals 130 on the market, such as mobile phones and tablets, have monocular cameras 132 or binocular (multi-) cameras 132. In terms of ranging principle, monocular cameras 132 and binocular (multi-) cameras 132 are completely different. Binocular (multi-) cameras 132 are similar to human eyes, and mainly determine the distance through parallax calculation of two images, which can estimate the depth when stationary, so that the accuracy of the data is better, but the operation of parallax consumes a lot of resources, and has the disadvantages of large amount of calculation and high energy consumption. Although the image frames collected by the monocular camera 132 lose the depth information of the environment, this disadvantage can be alleviated to some extent by fusing the pose data collected by the inertial measurement unit 133, for example, according to the shift of feature points between the front and rear frames photographed by the monocular camera 132, and then fusing the pose data collected by the inertial measurement unit 133 to calculate the movement and rotation of the camera itself. Therefore, the number of cameras 132 possessed by the mobile terminal 130 is not strictly limited in the present application.

[0107] The inertial measurement unit 133 at least includes an accelerometer and a gyroscope, and further can include a magnetometer. Taking an Android phone as an example, the IMU data of the Android phone includes 9 items of data of the accelerometer (3 axes), the gyroscope (3 axes), and the magnetometer (3 axes). Generally, the IMU is placed at the center of gravity of the object, but the inertial measurement unit 133 of the mobile terminal 130 fixed on the fixing device 316 is generally at a linear distance of tens of centimeters (for example, 30 centimeters) from the center of gravity G of the intelligent mower 310. In order to alleviate this problem, a sensor position offset compensation parameter can be set when the application program 135 processes the IMU data. The sensor position offset compensation parameter can include 3-axis data (X, Y, Z). Among them, X represents the front-rear distance between the inertial measurement unit 133 of the mobile terminal 130 and the center of gravity G of the intelligent mower 310, and a positive value indicates that the center of gravity G of the intelligent mower 310 is in front of the inertial measurement unit 133 of the mobile terminal 130, and a negative value indicates that the center of gravity G of the intelligent mower 310 is behind the inertial measurement unit 133 of the mobile terminal 130. Y represents the left-right distance between the inertial measurement unit 133 of the mobile terminal 130 and the center of gravity G of the intelligent mower 310, and a positive value indicates that the center of gravity G of the intelligent mower 310 is on the right side of the inertial measurement unit 133 of the mobile terminal 130, and a negative value indicates that the center of gravity G of the intelligent mower 310 is on the left side of the inertial measurement unit 133 of the mobile terminal 130. Z represents the up-down distance between the inertial measurement unit 133 of the mobile terminal 130 and the center of gravity G of the intelligent mower 310, and a positive value indicates that the center of gravity G of the intelligent mower 310 is below the inertial measurement unit 133 of the mobile terminal 130, and a negative value indicates that the center of gravity G of the intelligent mower 310 is above the inertial measurement unit 133 of the mobile terminal 130.

[0108] In addition to the camera 132 and the inertial measurement unit 133, the mobile terminal 130 can further include other sensors such as a GPS sensor, and preset corresponding sensor fusion (Sensor Fusion) logic code in the application program 135. The process of visual-inertial fusion SLAM performed by the application program 135, and the process involving more sensor fusion, including the application involving specific functions such as the generation of the mowing area boundary, the selection of the road surface, the intelligent obstacle avoidance, the setting of the virtual fence and the virtual obstacle, the intelligent lighting, and the selection of the mowing opportunity, will not be described here again due to the similarity with the control program 145 of the intelligent mower 110.

[0109] There are various ways to realize the communication between the intelligent mower 310 and the mobile terminal 130, which will be described in detail in the following embodiments. Figures 22A-22EIn the present application, the specific communication mode between the intelligent mower 310 and the mobile terminal 130 is not limited, for example, a male type C interface can be arranged on the second baffle 382 of the fixing device 316, when the mobile terminal 130 is fixed on the fixing device 316, the female type C interface of the mobile terminal is inserted on the male type C interface of the fixing device 316, that is, the data transmission between the mobile terminal 130 and the intelligent mower 310 can be realized. However, this connection mode limits the type of interface, if the interface type of the user's mobile terminal 130 is different from the preset interface type of the intelligent mower 310, then an adapter needs to be used. The problem of interface incompatibility can be solved by connecting the two interfaces with a separate data line, for example Figure 22A The intelligent mower 310 has a USB data transmission interface 311, if the mobile terminal 130 has a type C data transmission interface 131, through a USB-type C data line, one end of which is connected to the USB data transmission interface 311 of the intelligent mower 310, and the other end is connected to the type C data transmission interface 131 of the mobile terminal 130, the data transmission between the mobile terminal 130 and the intelligent mower 310 can be realized. If the data transmission interface 131 of the user's mobile terminal 130 is an Android data interface, then a USB-Android data line is needed, one end of which is connected to the USB data transmission interface 311 of the intelligent mower 310, and the other end is connected to the Android data transmission interface 131 of the mobile terminal 130, so that the data transmission between the mobile terminal 130 and the intelligent mower 310 can be realized. The advantage of using a separate data line for transmission is that it can adapt to the extension or rotation of the fixing device 316. In addition, the charging head of the mobile terminal 130 such as mobile phone and tablet is generally using USB transmission interface, that is, the end of the charging line of the mobile terminal 130 such as mobile phone and tablet connected to the charging head is basically USB transmission interface, which not only improves the universality of the USB data transmission interface 311 of the intelligent mower 310, but also can be provided by the user because of this data line, that is, the charging line of the mobile terminal 130 such as mobile phone and tablet, further reducing the cost of the intelligent mower 310.

[0110] When wired connection is adopted, the application program 135 of the mobile terminal 130 calls the image data collected by the camera 132 and the pose data collected by the inertial measurement unit 133, and fuses the two types of data for simultaneous localization and mapping (SLAM). This process can call open-source AR resource packages. For example, the application program 135 developed for an Apple mobile terminal 130 can call the ARKit development tool set, and the application program 135 developed for an Android mobile terminal 130 can call the ARCore development tool set. According to the results output by the simultaneous localization and mapping (SLAM), the application program 135 of the mobile terminal 130 generates specific navigation and mowing instructions according to a preset program, and returns them to the intelligent mower 310, as indicated by the solid arrow in FIG. 13. Figure 22A The preset program can include a plurality of application functions, such as automatic generation of mowing boundaries, virtual fence setting, road surface recognition, intelligent obstacle avoidance, virtual obstacle setting, etc. The preset program can also call resource packages with object recognition functions, such as TensorFlow Lite, to implement object recognition functions. Alternatively, considering that the intelligent mower 310 itself can also include other sensors such as collision sensors and fall sensors, the intelligent mower 310 can send the data collected by these sensors to the mobile terminal 130, as indicated by the dashed arrow in FIG. 14. After being coordinated by the application program 135 of the mobile terminal 130, the specific navigation and mowing instructions are generated according to the preset program, and are transmitted to the intelligent mower 310 through wired transmission, as indicated by the solid arrow in FIG. 15. Figure 22A Figure 22A

[0111] Further, on the basis of the communication between the intelligent mower 310 and the mobile terminal 130 described above, the intelligent mower 310 can also send the data collected by the sensors to the mobile terminal 130, as indicated by the dashed arrow in FIG. 16. After being coordinated by the application program 135 of the mobile terminal 130, the specific navigation and mowing instructions are generated according to the preset program, and are transmitted to the intelligent mower 310 through wired transmission, as indicated by the solid arrow in FIG. 17. Figure 22B ​​As shown, the mobile terminal 130 also includes a wireless network connection device 134, which can implement data transmission with the cloud server 200, so that the application program 135 of the mobile terminal 130 does not need to complete all operations locally on the mobile terminal 130, but part or all of the operations are completed on the cloud server 200, for example, in the process of simultaneous localization and mapping (SLAM), all image data collected by the camera 132 and angular velocity and acceleration data collected by the inertial measurement unit 133 are uploaded to the cloud server 200 for fusion; or, data preprocessing is first performed locally on the mobile terminal 130, for example, feature point extraction of image frames, and then the preprocessed data is sent to the cloud server 200 for fusion, so as to reduce the dependence on wireless communication rate. In addition to simultaneous localization and mapping (SLAM), the cloud server 200 can also run other program logics, and by virtue of the capabilities of cloud computing and cloud storage, the cloud server 200 can play an advantage in functions such as obstacle recognition, boundary recognition, road surface recognition, and path planning. The mobile terminal 130 can also upload the user's settings and preferences to the cloud server 200, for example, mowing height preference, lawn printing anchor point, etc.; the cloud server 200 can also autonomously obtain relevant information such as weather season from the Internet, so as to generate navigation and mowing instructions to control the behavior of the intelligent mower 310. The application program 135 of the mobile terminal 130 obtains the instructions from the cloud server 200, and then transmits the instructions to the intelligent mower 310 through wired transmission.

[0112] Alternatively, wireless data transmission can also be adopted between the intelligent mower 310 and the mobile terminal 130. As shown in FIG. 3B, the intelligent mower 310 is provided with a wireless network connection device 350, and the mobile terminal 130 is provided with a wireless network connection device 134. Figure 22C Because the distance between the intelligent mower 310 and the mobile terminal 130 is always very close when the intelligent mower 310 works with the mobile terminal 130, short-distance wireless communication, such as Bluetooth, ZigBee, NFC, etc., can be realized between the intelligent mower 310 and the mobile terminal 130, and this scheme requires that both the intelligent mower 310 and the mobile terminal 130 have matching short-distance wireless communication devices, for example, both the intelligent mower 310 and the mobile terminal 130 have Bluetooth. Figures 22A-22B Compared with the wired communication shown in FIG. 3A, the scheme of short-distance wireless communication actually only changes the wired interface between the intelligent mower 310 and the mobile terminal 130 into a wireless interface, and there is no difference in other aspects (transmission content, system architecture, etc.).

[0113] Alternatively, the mobile terminal 130 is provided with a wireless network connection device 134 such as a wireless network card or a wlan module, and the intelligent mower 310 is provided with a wireless network connection device 350 such as a wireless network card or a wlan module, as shown in FIG. 3C. Figure 22DWhen the user's lawn is fully covered by wireless network, both the mobile terminal 130 and the smart mower 310 can connect to the cloud server 200 through the wireless network. The application program 135 of the mobile terminal 130 can upload all the image data collected by the camera 132 and the angular velocity and acceleration data collected by the inertial measurement unit 133 to the cloud server 200 for AR fusion; or, the mobile terminal 130 can perform data preprocessing such as feature point extraction locally, and then send the preprocessed data to the cloud server 200 for AR fusion, so as to reduce the dependence on communication rate. At the same time, the smart mower 310 can also upload information (if any, Figure 22D The cloud server 200 returns the calculation result to the mobile terminal 130, and then the mobile terminal 130 returns the result to the smart mower 310. Compared with the case where the cloud server 200 returns the calculation result to the mobile terminal 130, the cloud server 200 directly returns the result to the smart mower 310 has the advantage of reducing delay. Figure 22B The cloud server 200 returns the calculation result to the mobile terminal 130, and then the mobile terminal 130 returns the result to the smart mower 310. Compared with the case where the cloud server 200 returns the calculation result to the mobile terminal 130, the cloud server 200 directly returns the result to the smart mower 310 has the advantage of reducing delay.

[0114] When the user's lawn is not fully covered by wireless network due to large area or other reasons, the above-mentioned scheme also has a remedial implementation method, see Figure 22E Since the mobile terminal 130 such as mobile phone generally has mobile network receiving 137 and wifi hotspot 138 functions, the mobile terminal 130 can convert the mobile network signal received by the mobile terminal 130 into wifi signal and send it out, and the smart mower 310 has wireless network connection equipment 350 such as wireless network card or wlan module, which can realize wireless communication with the cloud server 200 through the wifi network emitted by the wifi hotspot 138 of the mobile terminal 130. When the smart mower 310 and the mobile terminal 130 are not in the same wifi network, for example, the smart mower 310 accesses the network through the hotspot network of the mobile terminal 130, while the mobile terminal 130 accesses the network through the mobile network, the cloud server 200 may not be able to automatically identify the pairing of the smart mower 310 and the mobile terminal 130. At this time, the application program 135 and the smart mower 310 can increase the ID of the smart mower 310 as an identification code when uploading data, and the smart mower 310 can use the ID of the smart mower 310 as a voucher when obtaining instructions.

[0115] Compared with the first embodiment, the above-mentioned intelligent mowing system 100 integrating the intelligent mowing machine 310 with the mobile terminal 130 reduces the hardware requirements for the intelligent mowing machine 310, not only saving the cost of the camera 132 and the inertial measurement unit 133, but also reducing the requirements for the processing chip of the intelligent mowing machine 310 by transferring the AR operation with higher requirements for computing resources to the application program on the mobile terminal 130, thereby saving the chip cost. In addition, people use mobile terminals 130 more frequently in daily life; the application program 135 on the mobile terminal 130 is more convenient to upgrade, maintain and expand with the help of various application market platforms. For example, the application program 135 V1.0.0 version can be pure local operation, and the application program 135 V1.2.0 version can mainly rely on local operation, but the pictures that need to be identified will be uploaded to the cloud server 200, and the type of obstacles will be more accurately judged with the help of big data. Of course, from another point of view, the mobile terminal 130 and the intelligent mowing machine 310 are fixed when the intelligent mowing machine 310 is working, which will also bring some inconvenience to the user, because many people are now used to not leaving their mobile phones, and only when the mobile phone is charging will the mobile phone be left for a while. In order to alleviate the anxiety of the user caused by the separation of the mobile phone as much as possible, and in order to prevent the remaining power of the mobile terminal 130 from being too low to complete a complete mowing task, the intelligent mowing machine 310 can be configured to charge the battery of the mobile terminal 130 with the battery pack of the intelligent mowing machine 310 when the mobile terminal 130 is connected. At the same time, in order to avoid the problem that the intelligent mowing machine 310 still insists on charging the mobile terminal 130 when the power of the intelligent mowing machine 310 itself is insufficient, causing problems such as a sudden decrease in working time and over-discharge of the battery pack, a charging threshold can be set, for example, 70%. That is, if the remaining power of the battery pack of the intelligent mowing machine 310 is greater than or equal to 70%, the connected mobile terminal 130 is charged; if the remaining power of the battery pack of the intelligent mowing machine 310 is less than 70%, the connected mobile terminal 130 is not charged. It should be noted that here, 70% is only an example and does not limit the protection scope of the case, as long as a scheme is set to determine whether the intelligent mowing machine 310 charges the connected mobile terminal 130 according to the threshold of the remaining power of the intelligent mowing machine 310, which falls within the protection scope of the present application.

[0116] The basic principles, main features and advantages of the present application are shown and described above. Those skilled in the art should understand that the above-mentioned embodiments do not limit the present application in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present application.

Claims

1. An intelligent mowing system, comprising an intelligent mower and a mobile terminal: wherein the mobile terminal comprising: a camera for collecting image data of the environment around the intelligent mower; an inertial measurement unit for detecting pose data of the intelligent mower; an interface for connecting with the intelligent mower for data transmission; a memory for storing at least an application program for controlling the operation or movement of the intelligent mower; a processor for calling the application program, fusing the image data collected by the camera and the pose data collected by the inertial measurement unit, performing real-time positioning and map construction of the intelligent mower, generating navigation and mowing instructions according to a preset program, and sending the navigation and mowing instructions to the intelligent mower through the interface; the intelligent mower comprising: a main body; a fixing device arranged on the main body for fixedly mounting the mobile terminal to the intelligent mower; an interface for connecting with the mobile terminal for data transmission; a controller electrically connected with the interface, for controlling the intelligent mower according to the navigation and mowing instructions of the mobile terminal when the interface of the intelligent mower is connected with the interface of the mobile terminal; wherein the mobile terminal in the intelligent mowing system is a device different from the intelligent mower, and the mobile terminal comprises a mobile phone, a tablet computer, or a bracelet.

2. The intelligent mowing system of claim 1, wherein: The fixing device of the intelligent mower comprises an elastic clamping mechanism capable of elastically clamping a mobile terminal with a size of 4 inches to 12 inches.

3. The intelligent mowing system of claim 1, wherein: The application program can distinguish grassland from non-grassland according to the feature points of the two-dimensional plane in the image data and the texture characteristics of the grassland, and automatically generate the boundary of the mowing area by taking the boundary between the grassland and the non-grassland as discrete anchor points through the real-time positioning and map construction.

4. The intelligent mowing system of claim 1, wherein: The intelligent mower further comprises a blade, and the application program can distinguish grassland from non-grassland according to the feature points of the two-dimensional plane in the image data and the texture characteristics of the grassland, and stop rotating the blade when the current working plane is not grassland.

5. The intelligent mowing system of claim 1, wherein: The application program can determine the type of the current working plane according to the feature points of the two-dimensional plane in the image data and the texture characteristics of the common types of ground preset by the application program, and control the intelligent mower to move towards the ground with greater hardness among the multiple types of ground when the current working plane comprises multiple types of ground.

6. The intelligent mowing system of claim 1, wherein: The application program further comprises an object recognition program, and the application program can select a corresponding obstacle avoidance strategy according to the obstacle category recognized by the object recognition program.

7. The intelligent mowing system of claim 1, wherein: The mobile terminal further comprises a global satellite positioning system sensor, and the application program uses the positioning result of the global satellite positioning system sensor to filter and correct the result of the real-time positioning and map construction.

8. The intelligent mowing system of claim 1, wherein: The intelligent mower further comprises a lighting lamp, and the application program calculates the light intensity of the current environment according to the image data and sends an instruction to turn on the lighting lamp when the light intensity is lower than a first light intensity threshold.

9. The intelligent mowing system of claim 1, further comprising: An interactive display interface, through which a user can view the real-time image collected by the camera and superimpose a virtual fence on the real-time image, and the application program adds the anchor points of the virtual fence to the anchor point set of the mowing area boundary.

10. The intelligent mowing system of claim 1, further comprising: An interactive display interface is provided, through which a user can view real-time images captured by the camera and superimpose virtual obstacles on the real-time images. The application program records the anchor points of the virtual obstacles and plans a path to bypass the virtual obstacles.

11. An intelligent mowing system, comprising an intelligent mower and a mobile terminal: wherein The intelligent mower comprises: a camera configured to capture image data of an environment surrounding the intelligent mower; an inertial measurement unit configured to detect pose data of the intelligent mower; an interface configured to connect with the mobile terminal and perform data transmission; The mobile terminal comprises: an interface configured to connect with the intelligent mower and perform data transmission; a memory configured to store at least an application program for controlling the intelligent mower to work or walk; a processor configured to call the application program, acquire the image data and the pose data from the intelligent mower, fuse the image data and the pose data, perform instant positioning and map construction of the intelligent mower, generate navigation and mowing instructions according to a preset program, and send the navigation and mowing instructions to the intelligent mower; wherein the mobile terminal in the intelligent mowing system is a device different from the intelligent mower, and the mobile terminal comprises a mobile phone, a tablet computer, or a bracelet; the control program of the intelligent mower acquires data from the camera and the inertial measurement unit and sends the data to the mobile terminal, and the application program of the mobile terminal performs fusion operation; when the intelligent mower is connected with the mobile terminal, the navigation and mowing instructions of the mobile terminal control the intelligent mower.

12. The intelligent mowing system of claim 11, wherein: The interface of the intelligent mower comprises a wireless communication device, the interface of the mobile terminal comprises a wireless communication device, and wireless data transmission can be achieved between the intelligent mower and the mobile terminal.

13. The intelligent mowing system of claim 11, wherein: The interface of the intelligent mower comprises an application program interface, which defines a data communication protocol and format between the intelligent mower and the mobile terminal.

14. The intelligent mowing system of claim 11, wherein: The application program comprises mowing preference parameters that can be edited by a user.

15. The intelligent mowing system of claim 11, wherein: The intelligent mower further comprises a global satellite positioning system sensor, and the application program uses a positioning result of the global satellite positioning system sensor to filter and correct a result of the instant positioning and map construction.

16. The intelligent mowing system of claim 11, wherein: The intelligent mower further comprises a lighting lamp, and the application program calculates an illumination intensity of a current environment according to the image data and sends an instruction to turn on the lighting lamp when the illumination intensity is lower than a first illumination intensity threshold.

17. The intelligent mowing system of claim 11, wherein: The application program can distinguish grassland from non-grassland according to a two-dimensional plane feature point in the image data and a texture feature of the grassland, and take a boundary between the grassland and the non-grassland as a discrete anchor point to automatically generate a mowing area boundary through the instant positioning and map construction.

18. The intelligent mowing system of claim 11, wherein: The mobile terminal further comprises an interactive display interface, through which a user can view real-time images captured by the camera and superimpose a virtual fence on the real-time images, and the application program adds anchor points of the virtual fence to a set of anchor points of the mowing area boundary.

19. The intelligent mowing system of claim 11, wherein: The mobile terminal further comprises an interactive display interface, through which a user can view a real-time image captured by the camera and superimpose virtual obstacles on the real-time image, and the application program records anchor points of the virtual obstacles and plans a path to bypass the virtual obstacles.

Images