Lawn mower control method and device, lawn mower and computer program product

Abstract

The invention belongs to the technical field of hay mowers, and provides a hay mower control method and device, a hay mower and a computer program product. The mower analyzes the environment image to obtain semantic segmentation maps of different environment element types, so that the boundary, obstacles, aisles and mowing areas of the lawn can be effectively identified; on the basis of environment boundary recognition, intelligent path planning is started, and a semantic environment map can be formed through superposition of a feature point map and a semantic segmentation map of a computer vision module; the information of the semantic environment map can be combined with the real-time position information of the positioning module of the mower, and then the optimal mowing path can be determined. The mower disclosed by the embodiment of the invention is capable of ensuring the accurate positioning of the mower under the condition of no fixed boundary line by integrating various technologies, and is suitable for various complex environments; therefore, a traditional boundary line can be replaced, dependence on a physical boundary line is eliminated, and autonomous positioning and working of the mower in a boundary-line-free environment are achieved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of lawn mowers, and particularly relates to a method and device for controlling a lawn mower, the lawn mower and a computer program product. BACKGROUND

[0002] Early lawn mowers are controlled by manual operation, but most lawn mowers are developing towards automation and intelligence, and many intelligent lawn mowers have appeared on the market.

[0003] However, the intelligent lawn mowers of the prior art usually need to rely on pre-buried boundary lines (such as underground electric lines) to demarcate the working area of the intelligent lawn mower, but such a conventional method has defects such as complicated installation, poor flexibility, low positioning accuracy, and limits intelligent path planning. With the progress of automation technology, there is an urgent need for a system that does not require boundary lines, is easy to install and can be positioned with high accuracy to improve the intelligent level of the lawn mower. SUMMARY

[0004] The embodiments of the application provide a method and device for controlling a lawn mower, the lawn mower and a computer program product, aiming to solve the technical problem of the conventional intelligent lawn mower mentioned above, which is caused by relying on boundary lines and has the problems of installation and inaccurate positioning.

[0005] In a first aspect, the embodiments of the application provide a method for controlling a lawn mower, applied to the lawn mower, and the method comprises:

[0006] collecting an environment image of an initial working area range;

[0007] identifying environment elements of the environment image to obtain semantic segmentation maps of different environment element types; wherein each semantic segmentation map represents an environment element;

[0008] calling a computer vision module to process the environment image, extract and solve feature points in the environment image to generate a three-dimensional feature point cloud with scale information, and perform spatial superposition and data correlation between the three-dimensional feature point cloud and the semantic segmentation map to generate a semantic environment map; wherein the semantic environment map contains semantic labels and environment element position coordinates of each environment element, and the semantic labels are used to represent mowable areas and non-mowable areas;

[0009] acquiring real-time position information of the lawn mower during movement of the lawn mower;

[0010] setting a mowing work path of a virtual working area of the lawn mower according to the real-time position information of the lawn mower, the environment element position coordinates of each environment element in the environment distribution map and the semantic labels of each environment element in the environment distribution map.

[0011] Beneficial Effects: By analyzing environmental images, the lawnmower obtains semantic segmentation maps of different environmental elements, effectively identifying lawn boundaries, obstacles, walkways, and mowing areas. Based on environmental boundary recognition, the device begins intelligent path planning. This is achieved by overlaying feature point maps from the computer vision module with the semantic segmentation maps to form a semantic environment map. The information from this semantic environment map is then combined with the real-time location information from the lawnmower's positioning module to determine the optimal mowing path. This embodiment of the lawnmower, by integrating multiple technologies, ensures precise positioning even without fixed boundary lines, adapting to various complex environments. Furthermore, it can replace traditional boundary lines, eliminating dependence on physical boundaries and enabling autonomous positioning and operation in boundaryless environments.

[0012] In a possible implementation of the first aspect, acquiring the real-time location information of the lawnmower during its movement includes:

[0013] The positioning module is invoked, enabling it to receive real-time dynamic differential signals from the base station. The positioning module then calculates the real-time dynamic differential signals using carrier phase analysis and outputs the real-time location information of the lawnmower.

[0014] Preferably, the method further includes:

[0015] When the positioning module receives a real-time dynamic differential signal from the base station, it scores the real-time dynamic differential signal according to a preset evaluation method to obtain a signal scoring result.

[0016] The signal scoring result is compared with the quality scoring threshold;

[0017] If the signal score is not less than the quality score threshold, the real-time dynamic differential signal is calculated, and the real-time position information of the lawnmower is output.

[0018] Preferably, the lawnmower is further equipped with an inertial measurement unit and a wheel speed meter; the inertial measurement unit is used to collect the inertial data of the lawnmower, and the wheel speed meter is used to obtain the wheel speed information of the lawnmower;

[0019] After the step of comparing the signal scoring result with the quality scoring threshold, the method further includes:

[0020] If the signal score is less than the quality score threshold, the real-time pose estimation information of the lawnmower is determined based on the feature point map in the semantic environment map, the inertial data of the lawnmower, and the wheel speed information.

[0021] Accordingly, setting the mowing path of the lawnmower's virtual work area based on the real-time location information of the lawnmower, the location coordinates of each environmental element in the environmental distribution map, and the semantic tags of each environmental element in the environmental distribution map includes:

[0022] Based on the lawnmower's temporal pose estimation information, the location coordinates of each environmental element in the environmental distribution map, and the semantic labels of each environmental element in the environmental distribution map, the lawnmower's virtual work area mowing path is set.

[0023] Preferably, determining the real-time pose estimation information of the lawnmower based on the feature point map in the semantic environment map, the inertial data of the lawnmower, and the wheel speed information includes:

[0024] During the movement of the lawnmower, the relative displacement of the lawnmower with respect to each environmental element in the feature point map is calculated, and the undetermined relative displacement of the lawnmower is calculated by combining the position coordinates of each environmental element.

[0025] The short-term travel distance of the lawnmower is calculated based on the inertia data and wheel speed information of the lawnmower;

[0026] The temporal pose estimation information of the lawnmower is determined based on the undetermined relative displacement and the short-term travel distance.

[0027] Beneficial effects: Even with limited signal from the positioning module, the lawnmower can maintain accurate positioning by combining feature point maps from the environmental image with inertial data and wheel speed information.

[0028] In a possible implementation of the first aspect, the signal score result of the real-time dynamic differential signal can be calculated by: using the average signal-to-noise ratio of the real-time dynamic differential signal, the total number of satellites that the positioning module can connect to, and the accuracy set factor of the satellites.

[0029] Preferably, after the step of determining the real-time pose estimation information of the lawnmower, the method includes:

[0030] The displacement estimate of the wheel speed gauge is calculated based on the wheel speed information;

[0031] Calculate the ratio of the real-time pose estimation information of the lawnmower to the displacement estimation of the wheel speed meter. If the ratio meets the preset slippage condition, determine that the lawnmower has slipped.

[0032] If the lawnmower slips, the wheel speed information from the wheel speed meter is stopped, and a new real-time pose estimation of the lawnmower is calculated based on the undetermined relative displacement and the inertial data.

[0033] Beneficial effects: Even in complex terrain where it is easy to slip, the lawnmower can still ensure its positioning accuracy by stopping the use of the wheel speed information from the wheel speed meter and calculating the new real-time pose estimation information of the lawnmower based on the undetermined relative displacement and the inertial data.

[0034] Secondly, this application also proposes a lawnmower control device, the device comprising:

[0035] The image acquisition module is used to acquire environmental images of the initial work area.

[0036] The semantic segmentation module is used to identify environmental elements in an environmental image and obtain semantic segmentation maps of different environmental element types; wherein each semantic segmentation map represents an environmental element.

[0037] A computer vision module is used to generate a feature point map for the environment image, and to overlay the feature point map with the semantic segmentation map to form a semantic environment map; wherein, the semantic environment map includes semantic labels and location coordinates of each environmental element, and the semantic labels are used to represent mowing areas and non-mowing areas.

[0038] The positioning module is used to acquire the real-time location information of the lawnmower during its movement.

[0039] The control module is used to set the mowing path of the lawnmower's virtual work area based on the real-time location information of the lawnmower, the location coordinates of each environmental element in the environmental distribution map, and the semantic tags of each environmental element in the environmental distribution map.

[0040] Beneficial Effects: By analyzing environmental images through the device's semantic segmentation module, the lawnmower obtains semantic segmentation maps of different environmental element types, effectively identifying lawn boundaries, obstacles, walkways, and mowing areas. Based on environmental boundary recognition, the device begins intelligent path planning, forming a semantic environment map by overlaying feature point maps from the computer vision module and semantic segmentation maps. Combining the information from the semantic environment map with the real-time location information from the positioning module, the optimal mowing path can be determined. This embodiment of the lawnmower, by integrating multiple technologies, ensures precise positioning even without fixed boundary lines, adapting to various complex environments. Furthermore, it can replace traditional boundary lines, eliminating dependence on physical boundary lines and enabling autonomous positioning and operation in boundaryless environments.

[0041] Thirdly, this application also proposes a lawnmower, which includes a camera interface, a memory, a processor, and a lawnmower control program stored in the memory and executable on the processor; when the processor executes the lawnmower control program, it implements the steps of the lawnmower control method described in the first aspect above.

[0042] Fourthly, this application also proposes a computer program product that stores a lawnmower control program, which, when executed by a processor, implements the steps of the lawnmower control method described in the first aspect above.

[0043] It is understood that the beneficial effects of the third and fourth aspects mentioned above can be found in the relevant descriptions in the first or second aspects mentioned above, and will not be repeated here. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the structure of the lawnmower provided in the embodiments of this application;

[0046] Figure 2 This is a schematic flowchart of an embodiment of a lawnmower control method provided in this application;

[0047] Figure 3 This is a schematic flowchart of another embodiment of the lawnmower control method provided in this application;

[0048] Figure 4 This is a schematic flowchart of another embodiment of the lawnmower control method provided in this application;

[0049] Figure 5 This is a schematic diagram of the structure of a lawnmower control device provided in an embodiment of this application. Detailed Implementation

[0050] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0051] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0052] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0053] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0054] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0055] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0056] Understandably, the applicant of this invention has found that conventional smart lawnmowers in the prior art typically rely on pre-buried boundary lines (such as underground power lines) to delineate the work area. However, this approach has the following drawbacks:

[0057] Installation is complicated: it requires manual installation of wires, which is time-consuming and labor-intensive.

[0058] Poor flexibility: Once the boundary lines are fixed, they are difficult to adjust and cannot adapt to the needs of multiple scenarios;

[0059] Low positioning accuracy: It cannot provide real-time accurate location information of the lawnmower, which limits intelligent path planning.

[0060] To address the aforementioned technical problems of conventional intelligent lawnmowers, which rely on boundary lines leading to inconsistencies in installation and inaccurate positioning, this application provides a lawnmower control method, device, lawnmower, and computer program product. This enables a lawnmower control solution that eliminates the need for boundary lines, is easy to install, and achieves high-precision positioning, thereby improving the intelligence level of lawnmowers.

[0061] refer to Figure 1 , Figure 1 This is a schematic diagram of the structure of a lawnmower according to an embodiment of this application. The lawnmower includes: at least one processor 1001, a communication bus 1002, a memory 1005, and a lawnmower control program 1006 stored in the memory 1005 and executable on the at least one processor 1001. When the processor 1001 executes the lawnmower control program 1006, it implements the steps in the lawnmower control method embodiment of this application.

[0062] The lawnmower also includes a camera interface 1003 for connecting a camera. The lawnmower can transmit images of its surrounding environment as it moves to the processor 1001 via the camera interface 1003. In some embodiments, the camera interface 1003 can connect to the lawnmower's internal image acquisition module (e.g., an image sensor); in other embodiments, the camera interface can also connect to an external camera device. Those skilled in the art will understand that… Figure 1 This is merely an example of a lawnmower and does not constitute a limitation on lawnmowers. It may include more or fewer parts than shown, or combine certain parts, or use different parts.

[0063] The lawnmower may also be equipped with an inertial measurement unit and a wheel speed meter; the inertial measurement unit is used to collect the inertial data of the lawnmower, and the wheel speed meter is used to obtain the wheel speed information of the lawnmower.

[0064] The processor 1001 may be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0065] In some embodiments, the memory 1005 may be an internal storage unit of the lawnmower, such as a hard drive or RAM. In other embodiments, the memory 1005 may be an external storage device of the lawnmower, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 1005 may include both internal and external storage units. The memory 1005 is used to store the operating system and computer-executable programs for the lawnmower control methods.

[0066] Accordingly, when the processor 1001 of this application executes the lawnmower control program 1006, it implements the steps of the lawnmower control method of this application, such as... Figure 2 As shown, steps A1 to A5 are included:

[0067] Step A1: Call the image acquisition module to acquire environmental images of the initial working area, which is the working area created for the first time after the lawnmower is started;

[0068] For example, the initial work area can be a map area initially created by the user, such as an area where the user wants the lawnmower to perform mowing work. After the user starts the lawnmower in this embodiment, the lawnmower can travel a circle according to the map boundary initially created by the user. That is, during this circle, the lawnmower can call the image acquisition module to collect environmental images of the initial work area. For example, the camera installed on the lawnmower continuously collects image data of the surrounding environment, and these environmental images are transmitted to the lawnmower's processor for processing.

[0069] Step A2: Call the semantic segmentation module to identify environmental elements in the environmental image and obtain semantic segmentation maps of different environmental element types; wherein, each semantic segmentation map represents an environmental element;

[0070] For example, the semantic segmentation module in this embodiment can be a deep learning-based AI module. The lawnmower's processor 1001 can call the semantic segmentation module to use a trained convolutional neural network (CNN) model to perform recognition processing on the environmental image. The CNN model in this embodiment has been trained on a large amount of environmental image data and can identify different environmental elements. The environmental elements in this embodiment cover objects (such as obstacles) and areas (such as working areas and non-working areas), such as lawns, obstacles, road edges, etc., and assign each pixel in the environmental image to a specific category (such as grass, rocks, trees, etc.).

[0071] After processing by the convolutional neural network model, each pixel in the environmental image is labeled with a different category, which reflects different types of environmental elements. For example, lawn areas, obstacle areas, etc. The system generates an "environment segmentation map" based on this information, obtaining semantic segmentation maps of different environmental element types, and each semantic segmentation map represents an environmental element.

[0072] The lawnmower in this embodiment can identify environmental features such as lawn boundaries and obstacles through an AI semantic segmentation module, directly replacing physical boundary lines and providing virtual boundary constraints.

[0073] Step A3: Call the computer vision module to process the environmental image, extract and solve the feature points in the environmental image to generate a three-dimensional feature point cloud with scale information, and spatially overlay and data associate the three-dimensional feature point cloud with the semantic segmentation map to generate a semantic environment map; wherein, the semantic environment map contains semantic labels and location coordinates of each environmental element, and the semantic labels are used to represent mowing areas and non-mowing areas.

[0074] The computer vision module in this embodiment can be a VSLAM (Visual Simultaneous Localization and Mapping) module. By calling the computer vision module to process the environmental image, it can not only generate feature points in the environmental image, but also calculate the precise three-dimensional coordinates of the feature points. The calculated precise three-dimensional coordinates of the feature points (i.e., a three-dimensional feature point cloud with scale information) are spatially aligned and fused with semantic information (i.e., a semantic segmentation map) to construct a dense semantic environment map containing precise coordinate information and semantic labels.

[0075] The lawnmower in this embodiment can overlay the semantic segmentation map from AI semantic segmentation and the feature point map from the VSLAM computer vision module to generate a dense semantic environment map representing the environmental boundary, which helps in the subsequent step A5 to generate a virtual work area without relying on physical boundary lines.

[0076] Step A4: Invoke the positioning module to obtain the real-time location information of the lawnmower during its movement;

[0077] Understandably, the positioning module in this embodiment can be an RTK (Real-time kinematic) positioning module. After the lawnmower starts, it first uses the RTK positioning module to perform high-precision positioning and obtain the lawnmower's real-time location information. This embodiment can provide high-precision positioning data through the RTK positioning module, ensuring that the lawnmower always knows its own location while performing tasks.

[0078] For example, during its movement, the lawnmower invokes the RTK positioning module, which receives a real-time dynamic differential signal from the base station. The module then calculates the real-time dynamic differential signal using carrier phase and outputs the lawnmower's real-time position information, which can be centimeter-level accurate position coordinates.

[0079] Step A5: Based on the real-time location information of the lawnmower, the location coordinates of each environmental element in the environmental distribution map, and the semantic tags of each environmental element in the environmental distribution map, set the mowing path of the lawnmower's virtual work area.

[0080] Understandably, the lawnmower in this embodiment analyzes environmental images using an AI semantic segmentation module to obtain semantic segmentation maps of different environmental element types, effectively identifying lawn boundaries, obstacles, walkways, and mowing areas. Based on environmental boundary recognition, intelligent path planning begins. A semantic environment map is formed by overlaying the feature point map from the VSLAM computer vision module with the semantic segmentation map. The information from this semantic environment map is then combined with the real-time location information from the RTK positioning module to determine the optimal mowing path. By integrating multiple technologies, the lawnmower in this embodiment can ensure precise positioning even without fixed boundary lines, adapting to various complex environments. Furthermore, it can replace traditional boundary lines, eliminating dependence on physical boundary lines and enabling autonomous positioning and operation in boundaryless environments.

[0081] For example, in a specific application scenario, the input data for dynamically generating the mowing path scheme for the virtual work area in this embodiment is:

[0082] The positioning module provides real-time location information: it provides the lawnmower's current high-precision global coordinates (such as latitude and longitude or position in a local coordinate system);

[0083] Environmental distribution map: contains the following information: semantic labels of environmental elements (such as "lawn", "flower bed", "stones", "obstacles", etc.); location coordinates of environmental elements (e.g., generated by overlaying VSLAM feature point map and semantic segmentation map);

[0084] Virtual work area: This can be a user-preset or automatically identified area that can be mowed (such as a lawn area);

[0085] Planning objectives can include: covering all mowing areas (semantically labeled "lawn"); avoiding non-mowing areas (such as "flower beds" or "obstacles"); and optimizing routes to reduce repetition or omissions. This example plans detours in advance by combining semantic labels and coordinate data; and the RTK real-time location and VSLAM visual feature point map complement each other to ensure continuity when the signal is lost.

[0086] In some embodiments, the lawn mowing path planned in step A5 can be dynamically updated. For example, the semantic segmentation module processes the environmental images captured by the camera in real time and quickly feeds them back to the lawnmower's processor 1001. The processed semantic segmentation map is combined with the data from the positioning module and the data from the VSLAM computer vision module, and adjustments are made according to real-time changes in obstacles and environment (such as avoiding newly appearing obstacles and adjusting the lawnmower's movement direction). The position and path of the lawnmower are updated in real time to ensure the smooth progress of the task.

[0087] In addition, in other embodiments, when the lawnmower completes its task and covers the virtual work area of ​​step A5, the processor can use the AI ​​module to confirm whether the task is completed (e.g., all mowing areas have been completed and none have been missed). If the task is completed, the processor issues a stop command, and the lawnmower automatically stops, ready to start the next job.

[0088] Further, refer to Figure 3 In another embodiment, step A4 further includes:

[0089] Step A401: When the positioning module receives the real-time dynamic differential signal from the base station, the real-time dynamic differential signal is scored according to a preset evaluation method to obtain a signal scoring result;

[0090] In this embodiment, the signal scoring result can be calculated using the average signal-to-noise ratio of the real-time dynamic differential signal, the total number of satellites that the RTK positioning module can connect to, and the accuracy set factor of the satellites.

[0091] Exemplarily, in this embodiment, the real-time kinematic differential signal can be scored by the following formula:

[0092] Q rtk = α·SNR + β·number of satellites + γ·GDOP

[0093] where SNR represents the signal-to-noise ratio, that is, the average signal-to-noise ratio of the real-time kinematic differential signal received by the RTK positioning module;

[0094] The number of satellites represents the total number of satellites that the RTK positioning module can connect to;

[0095] GDOP represents the geometric dilution of precision, that is, the geometric dilution of precision calculated from the geometric positions of satellites;

[0096] α, β, γ are the weights of each index in the model. α is the first weight coefficient, β is the second weight coefficient, and γ is the third weight coefficient;

[0097] Exemplarily, the setting basis of the weight coefficients α, β, γ can be obtained through the following methods: The setting of these parameters is based on the following two principles: the analysis of the actual influence weight of the index on the positioning quality; the results of a large number of field tests and statistical fittings;

[0098] For example, by conducting field sampling tests in different application environments (such as open areas, semi-occluded areas, and under-forest occluded areas), statistically analyzing the influence degree of each index on the positioning error. Finally, the following recommended parameters are fitted: α = 0.5: SNR has a significant influence on the RTK positioning accuracy and directly reflects the reception quality, so the weight is the highest; β = 0.3: Although the number of satellites is important, after exceeding a certain number (such as > 8), the improvement of the accuracy slows down; γ = 0.2: GDOP represents the geometric distribution quality of satellites and affects the stability, and a certain weight is given.

[0099] The above weight coefficients can also be adjusted according to specific hardware and application scenarios.

[0100] Step A401a: Compare the signal scoring result Q rtk with the quality scoring threshold T1;

[0101] When Q rtk ≥ T1, execute the following step A401b, that is, preferentially use RTK positioning;

[0102] When Q rtk < T1, execute step A401c to automatically switch to the tightly coupled mode of visual positioning and mapping (VSLAM) and inertial navigation and odometer fusion (INS);

[0103] Furthermore, it should be noted that, in order to achieve a balance between sensitivity and robustness in mode switching, this embodiment can use an empirical statistical method to set the quality scoring threshold T1. For example, in 200 hours of field operation records, the RTK positioning error and Q can be compared. rtk The scores were fitted and analyzed; "high precision" was defined as positioning error ≤ ±5cm; the Qrtk score corresponding to more than 90% of high precision cases was selected as the quality score threshold T1.

[0104] In other embodiments, the weighting coefficients α, β, γ and the quality score threshold T1 can also be dynamically adjusted by machine learning models (such as linear regression, support vector machines, etc.) to adapt to specific environments and hardware platforms.

[0105] Step A401b: If the signal score result is not less than the quality score threshold, the real-time dynamic differential signal can be calculated by carrier phase and the real-time position information of the lawnmower can be output.

[0106] Step A401c: If the signal score result is less than the quality score threshold, determine the real-time pose estimation information of the lawnmower based on the feature point map in the semantic environment map, the inertial data of the lawnmower, and the wheel speed information.

[0107] For example, when the signal score result Qrtk < quality score threshold T1, it automatically switches to a tightly coupled mode of visual localization and mapping (VSLAM) and inertial navigation and odometry fusion (INS).

[0108] Understandably, Visual Localization and Mapping (VSLAM) corresponds to the feature point map in the aforementioned environmental image, while Inertial Navigation and Odometer Fusion (INS) corresponds to the aforementioned inertial data and wheel speed information.

[0109] The lawnmower needs to know its position (X, Y, Z) and direction (angle) in real time, and the above tightly coupled mode achieves this through the following two steps:

[0110] (1) Visual localization (VSLAM part)

[0111] During the movement of the lawnmower, the relative displacement of the lawnmower with respect to each environmental element in the feature point map is calculated, and the undetermined relative displacement of the lawnmower is calculated by combining the position coordinates of each environmental element.

[0112] Understandably, the lawnmower uses its image acquisition module (camera) to capture images of the surrounding environment while it is moving. The lawnmower's AI semantic segmentation module can then identify semantic segmentation maps in the environmental images, including different types of environmental elements (such as lawns, trees, and rocks).

[0113] The computer vision module extracts "feature points" of environmental elements (such as the texture of tree bark and the edge of stone in the image) from these semantic segmentation maps and memorizes the location coordinates of these environmental element "feature points".

[0114] It should be noted that when the lawnmower moves, the camera observes the same feature points moving from the left to the right of the screen. Through geometric calculations (triangulation), the distance and degree of rotation of the lawnmower can be estimated. Then, the relative displacement of the lawnmower is output (e.g., "moved 0.5 meters to the right"). It should be noted that this "relative displacement" may have errors and needs to be corrected using subsequent inertial assistance technology.

[0115] (2) Inertial Assist (INS Part)

[0116] The short-term travel distance of the lawnmower is calculated based on the inertia data and wheel speed information of the lawnmower;

[0117] For example, the aforementioned inertial data may include the acceleration and angular velocity of the lawnmower measured by the inertial measurement unit (e.g., "sudden acceleration" or "left turn").

[0118] The wheel speed information mentioned above can be the number of revolutions the wheels make during the lawnmower's movement recorded by the wheel speed meter, the estimated short-term travel distance, for example, by integrating the acceleration to obtain the speed, and then integrating again to obtain the displacement (similar to "estimate how far it has traveled by counting steps"), and finally outputting the short-term travel distance (such as "it has traveled 1.2 meters in 10 seconds"). It should be noted that this "short-term travel distance" may have errors and needs to be corrected in conjunction with the aforementioned visual positioning VSLAM technology.

[0119] (3) Combination of visual localization (VSLAM part) and inertial assistance (INS part):

[0120] The real-time pose estimation information of the lawnmower is determined based on the undetermined relative displacement and the short-term movement distance (instead of the real-time position information of the aforementioned A401b);

[0121] For example, this embodiment directly integrates the visual feature point observation data (undetermined relative displacement) of VSLAM with the short-time movement distance of the inertial aid device (inertial measurement unit measurement and wheel speedometer), and can output a more robust pose estimate through filtering or optimization algorithms (such as ESKF, nonlinear optimization);

[0122] For example, taking the error-state Kalman Filter (ESKF) filtering algorithm as an example, the data fusion process (using ESKF as an example) inputs the undetermined relative displacement obtained from visual positioning VSLAM technology, the inertial data of the lawnmower sensed by the inertial measurement unit, and the wheel speed information of the lawnmower recorded by the wheel speed meter into the error-state Kalman filter (ESKF) to optimize the pose estimation in real time. The inertial data is used to constrain the feature point matching of various environmental elements in the feature point map of visual positioning VSLAM technology, reducing mismatches (such as interference from dynamic objects), and ultimately outputting more robust real-time pose estimation information for the lawnmower.

[0123] For example, in the specific working applications of lawnmowers:

[0124] The lawnmower's camera captured a tree, and the VSLAM computer vision module output, "The tree is on the right side of the screen, indicating that the lawnmower has moved 0.3 meters to the left." Simultaneously, the inertial measurement unit (IMU) detected: "The accelerometer shows that it did indeed accelerate to the left; the wheels rotated X times, indicating a movement of 0.28 meters."

[0125] At this point, the lawnmower can execute a tightly coupled algorithm of visual localization and mapping (VSLAM) and inertial navigation and odometry fusion (INS):

[0126] Comparing the "0.3 meters" output by the computer vision module with the "0.28 meters" output by the IMU, it was found that the IMU might have underestimated the distance (e.g., due to wheel slippage). The system automatically corrected the distance to "actual movement of 0.3 meters" and adjusted the IMU's error model. The final output showed the pose estimation information: "Currently located at map coordinates (X=5.2, Y=3.1), facing north."

[0127] Accordingly, step A5 further includes:

[0128] A5: Based on the lawnmower's temporal pose estimation information, the location coordinates of each environmental element in the environmental distribution map, and the semantic labels of each environmental element in the environmental distribution map, set the lawnmower's virtual work area mowing path.

[0129] The beneficial effect of this embodiment is that, even when the signal from the RTK positioning module is limited, the lawnmower can ensure the positioning accuracy by combining feature point maps in the environmental image (VSLAM visual positioning) with inertial data and wheel speed information (INS).

[0130] Furthermore, in another embodiment, reference is made to... Figure 4 The method further includes:

[0131] The displacement estimate of the wheel speed gauge is calculated based on the wheel speed information;

[0132] Step A401d: During the movement of the lawnmower, calculate the ratio of the real-time pose estimation information of the lawnmower to the displacement estimation of the wheel speed meter. If the ratio meets the preset slippage condition, determine that the lawnmower has slipped.

[0133] Understandably, when a lawnmower is moving on the ground, wheel speed sensors and inertial measurement units (such as IMU sensors) can monitor the rotation and angular velocity of the lawnmower wheels. However, due to factors such as uneven or slippery ground, the wheels may slip, causing a difference between the displacement estimate of the wheel speed sensor and the actual displacement.

[0134] For example, in order to detect slippage, this embodiment uses the ratio of the real-time pose estimation information of the lawnmower to the displacement estimation of the wheel speed meter as the criterion for determining slippage:

[0135] K = ΔX odom / ΔX VIO

[0136] Where, ΔX odom This represents the displacement estimate from the wheel speed gauge; ΔX VIO This represents real-time pose estimation information.

[0137] When |1-K|>0.3, the system considers slippage to have occurred. In this case, visual displacement (provided by VIO) will be used instead of wheel speed gauge data for displacement correction to ensure the accuracy of the cutter's position. The use of absolute values ​​is to account for slippage, which can be either stationary slippage or slippage on a slope.

[0138] For example, when |1-K|>0.3, the system considers slippage to have occurred. In the event of slippage by the lawnmower, the system stops using the wheel speed information from the wheel speed meter to ensure the position accuracy of the lawnmower; and recalculates the real-time pose estimation information, including: recalculating the new real-time pose estimation information of the lawnmower based on the undetermined relative displacement obtained from visual positioning VSLAM and the inertial data of the lawnmower.

[0139] It should be noted that the "preset slip condition |1-K|>0.3" uses an absolute value to determine slippage because slippage can be divided into slippage in place and slippage on a slope. 0.3 is a slippage pre-set value, which is a value taken based on the test results. In other embodiments (such as different road surfaces), the slippage pre-set value can also be other values. As long as |1-K|>"slippage pre-set value" is met, it is determined that the lawnmower has slipped.

[0140] The advantages of this embodiment are as follows: it can detect whether the lawnmower is slipping in real time. After slipping is detected, the lawnmower can combine the visual information (from the VSLAM computer vision module) and the inertial data (from the inertial measurement unit) to re-estimate the real-time pose of the lawnmower. It can provide a relatively stable pose estimation in a short time, especially in environments where the RTK signal is weak or completely lost, it can make up for the lack of positioning accuracy. Furthermore, this embodiment only relies on the lawnmower's camera and inertial measurement unit to complete the slipping detection, without the need to set up additional sensors for the lawnmower, thus reducing hardware costs.

[0141] In one embodiment, such as Figure 5 As shown, the present invention also provides a lawnmower control device, which can be configured inside the lawnmower and includes an image acquisition module 10, a semantic segmentation module 20, a computer vision module 30, a positioning module 40, and a control module 50; as in the foregoing embodiments, in a specific implementation, the computer vision module 30 can be a VSLAM module, and the positioning module 40 can be an RTK positioning module.

[0142] For example, the functions of each module include:

[0143] Image acquisition module 10: Used to acquire environmental images of the initial work area;

[0144] For example, the image acquisition module 10 can correspond to the camera of a lawnmower.

[0145] Semantic segmentation module 20 is used to identify environmental elements in an environmental image and obtain semantic segmentation maps of different environmental element types; wherein each semantic segmentation map represents an environmental element;

[0146] For example, the semantic segmentation model 20 used in this embodiment is trained using a convolutional neural network (CNN). In order to ensure that the model can accurately identify boundaries and obstacles in different environments, the training dataset includes a variety of scenarios (such as different weather, different lighting conditions, different terrains, etc.). Through continuous training and optimization, the model can continuously improve its accuracy, especially in complex environments (such as scenarios with multiple obstacles or irregular edges of lawns).

[0147] Furthermore, in some embodiments, the semantic segmentation module 20 can process environmental images acquired by the image acquisition module 10 in real time and quickly feed them back to the lawnmower's processor. The processed semantic segmentation map is combined with data from the RTK positioning module 40 and the VSLAM computer vision module 30 to update the lawnmower's position and path in real time. Through the calculations of the semantic segmentation module 20, this device can make rapid decisions when the environment changes, such as avoiding newly appearing obstacles and adjusting the lawnmower's direction of movement.

[0148] The computer vision module 30 is used to generate a feature point map for the environment image, and to overlay the feature point map with the semantic segmentation map to form a semantic environment map (3D map); wherein, the semantic environment map (3D map) includes semantic labels of each environmental element and location coordinates of the environmental element, and the semantic labels are used to represent mowing areas and non-mowing areas.

[0149] The positioning module 40 is used to acquire the real-time location information of the lawnmower during its movement.

[0150] The control module 50 is used to set the mowing path of the virtual working area of ​​the lawnmower based on the real-time location information of the lawnmower, the location coordinates of each environmental element in the environmental distribution map, and the semantic tags of each environmental element in the environmental distribution map.

[0151] It should be noted that the lawnmower control device in this embodiment can be understood as a chip in the lawnmower; the information interaction and execution process between the above modules are based on the same concept as the method embodiment of this application, and their specific functions and technical effects can be found in the method embodiment section, which will not be repeated here.

[0152] Furthermore, this application also provides a computer program product that stores a lawnmower control program. When the lawnmower control program is executed by a processor, it implements the steps of the lawnmower control method embodiment described above.

[0153] Furthermore, it should be noted that if the integrated unit described in this application is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0154] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0155] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0156] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0157] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A lawnmower control method, applied to a lawnmower, characterized in that, The method includes: The image acquisition module is invoked to acquire environmental images of the initial work area. The semantic segmentation module is invoked to identify environmental elements in the environmental image, resulting in semantic segmentation maps of different environmental element types; wherein each semantic segmentation map represents an environmental element. The computer vision module is invoked to process the environmental image, extract and solve the feature points in the environmental image to generate a three-dimensional feature point cloud with scale information, and the three-dimensional feature point cloud is spatially superimposed and data associated with the semantic segmentation map to generate a semantic environment map; wherein, the semantic environment map includes semantic labels and location coordinates of each environmental element, and the semantic labels are used to represent mowing areas and non-mowing areas. The positioning module is invoked to obtain the real-time location information of the lawnmower during its movement. Based on the real-time location information of the lawnmower, the location coordinates of each environmental element in the environmental distribution map, and the semantic tags of each environmental element in the environmental distribution map, the lawnmower's virtual work area mowing path is set.

2. The method as described in claim 1, characterized in that, The invocation of the positioning module to obtain the real-time location information of the lawnmower during its movement includes: The positioning module is invoked, enabling it to receive real-time dynamic differential signals from the base station. The positioning module then calculates the real-time dynamic differential signals using carrier phase analysis and outputs the real-time location information of the lawnmower.

3. The method as described in claim 1, characterized in that, The method further includes: When the positioning module receives a real-time dynamic differential signal from the base station, it scores the real-time dynamic differential signal according to a preset evaluation method to obtain a signal scoring result. The signal scoring result is compared with the quality scoring threshold; If the signal score is not less than the quality score threshold, the real-time dynamic differential signal is calculated, and the real-time position information of the lawnmower is output.

4. The method as described in claim 3, characterized in that, The lawnmower is also equipped with an inertial measurement unit and a wheel speed meter; the inertial measurement unit is used to collect the inertial data of the lawnmower, and the wheel speed meter is used to obtain the wheel speed information of the lawnmower. After the step of comparing the signal scoring result with the quality scoring threshold, the method further includes: If the signal score is less than the quality score threshold, the real-time pose estimation information of the lawnmower is determined based on the feature point map in the semantic environment map, the inertial data of the lawnmower, and the wheel speed information. Accordingly, setting the mowing path of the lawnmower's virtual work area based on the real-time location information of the lawnmower, the location coordinates of each environmental element in the environmental distribution map, and the semantic tags of each environmental element in the environmental distribution map includes: Based on the lawnmower's temporal pose estimation information, the location coordinates of each environmental element in the environmental distribution map, and the semantic labels of each environmental element in the environmental distribution map, the lawnmower's virtual work area mowing path is set.

5. The method as described in claim 4, characterized in that, The step of determining the real-time pose estimation information of the lawnmower based on the feature point map in the semantic environment map, the inertial data of the lawnmower, and the wheel speed information includes: During the movement of the lawnmower, the relative displacement of the lawnmower with respect to each environmental element in the feature point map is calculated, and the undetermined relative displacement of the lawnmower is calculated by combining the position coordinates of each environmental element. The short-term travel distance of the lawnmower is calculated based on the inertia data and wheel speed information of the lawnmower; The temporal pose estimation information of the lawnmower is determined based on the undetermined relative displacement and the short-term travel distance.

6. The method according to any one of claims 3 to 5, characterized in that, The signal score of the real-time dynamic differential signal is calculated in the following manner: The signal scoring result is calculated using the average signal-to-noise ratio of the real-time dynamic differential signal, the total number of satellites that the positioning module can connect to, and the accuracy set factor of the satellites.

7. The method as described in claim 4 or 5, characterized in that, After the step of determining the real-time pose estimation information of the lawnmower, the method includes: The displacement estimate of the wheel speed gauge is calculated based on the wheel speed information; Calculate the ratio of the real-time pose estimation information of the lawnmower to the displacement estimation of the wheel speed meter. If the ratio meets the preset slippage condition, determine that the lawnmower has slipped. If the lawnmower slips, the wheel speed information from the wheel speed meter is stopped, and a new real-time pose estimation of the lawnmower is calculated based on the undetermined relative displacement and the inertial data.

8. A lawnmower control device, characterized in that, The device includes: The image acquisition module is used to acquire environmental images of the initial work area. The semantic segmentation module is used to identify environmental elements in an environmental image and obtain semantic segmentation maps of different environmental element types; wherein each semantic segmentation map represents an environmental element. A computer vision module is used to generate a feature point map for the environment image, and to overlay the feature point map with the semantic segmentation map to form a semantic environment map; wherein, the semantic environment map includes semantic labels and location coordinates of each environmental element, and the semantic labels are used to represent mowing areas and non-mowing areas. The positioning module is used to acquire the real-time location information of the lawnmower during its movement. The control module is used to set the mowing path of the lawnmower's virtual work area based on the real-time location information of the lawnmower, the location coordinates of each environmental element in the environmental distribution map, and the semantic tags of each environmental element in the environmental distribution map.

9. A lawnmower, characterized in that, The lawnmower includes a camera interface, a memory, a processor, and a lawnmower control program stored in the memory and executable on the processor; when the processor executes the lawnmower control program, it implements the steps of the lawnmower control method described in the first aspect above.

10. A computer program product storing a lawnmower control program, characterized in that, When the lawnmower control program is executed by the processor, it implements the steps of the lawnmower control method as described in any one of claims 1-7.

Images