Mowing robot positioning method and apparatus, and electronic device and readable storage medium

Abstract

Provided in the embodiments of the present application are a mowing robot positioning method and apparatus, and an electronic device and a readable storage medium. The method comprises: during an operation process of a mowing robot, collecting corresponding data by means of a wheel speed meter, an inertial positioning apparatus and an image sensing device, which are arranged on the mowing robot; analyzing collected wheel speed meter data to obtain a first residual term, analyzing inertial positioning data to obtain a second residual term, and analyzing image data to determine a third residual term; and using the first residual term, the second residual term and the third residual term to construct a cost function, and obtaining the current location information of the mowing robot by means of minimizing the cost function. In the present application, tight coupling is performed, by means of the method, on pieces of data which are collected by a plurality of sensors, thereby facilitating an improvement in the accuracy of a positioning result of a mowing robot.

Description

Lawn mowing robot positioning method, device, electronic device and readable storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 21, 2023, with application number 202310749631.2 and application name “Mowing robot positioning method, device, electronic device and readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the technical field of lawn mowing robots, and in particular to a lawn mowing robot positioning method, device, electronic device, and readable storage medium. Background Art

[0003] A lawnmower robot is a comprehensive robotic system that integrates multiple functions, including environmental perception, dynamic path planning, and behavioral control, freeing users from the tedious task of lawn maintenance. Precise positioning is crucial for lawnmowers, as it is crucial for various scenarios, including path planning for lawn maintenance within a certain range and automatic return to the charging station.

[0004] Currently, lawn mower robots obtain satellite positioning signals from mobile base stations. These signals are then verified using carrier phase real-time kinematic (RTK) base station calibration information to determine positioning accuracy. However, factors such as building obstructions and atmospheric interference can affect satellite positioning signals, causing the robot's positioning to drift and resulting in low positioning accuracy.

[0005] Summary of the Invention

[0006] In view of the above problems, the embodiments of the present application provide a lawn mowing robot positioning method, device, electronic device and readable storage medium, which improve the positioning accuracy of the lawn mowing robot.

[0007] A first aspect of an embodiment of the present application provides a positioning method for a lawn mowing robot, the method comprising:

[0008] Acquiring wheel speed meter data, inertial positioning data, and image data collected by an image sensing device provided on the lawn mowing robot;

[0009] Obtaining a first residual term according to the wheel speed meter data;

[0010] Obtaining a second residual term according to the inertial positioning data;

[0011] Calculating a visual reprojection error based on the image data, and obtaining a third residual term based on the visual reprojection error;

[0012] A cost function is constructed according to the first residual term, the second residual term, and the third residual term, and the current position information of the lawn mowing robot is obtained by minimizing the cost function.

[0013] In an optional manner, constructing a cost function according to the first residual term, the second residual term, and the third residual term, and obtaining the current position information of the lawn mowing robot by minimizing the cost function includes:

[0014] determining a first weight, a second weight, and a third weight; wherein the first weight is associated with the tachometer data; the second weight is associated with the inertial positioning data; and the third weight is associated with the image data;

[0015] Obtain a first data item according to the first residual item and the first weight, obtain a second data item according to the second residual item and the second weight, and obtain a third data item according to the third residual item and the third weight;

[0016] The cost function is obtained by adding the first data item, the second data item, and the third data item.

[0017] In an optional manner, obtaining the first residual term according to the wheel speed meter data includes:

[0018] performing integration processing on the wheel speed meter data to obtain first position information;

[0019] The first residual term is obtained according to the first position information and initial position estimation information.

[0020] In an optional manner, obtaining the second residual term according to the inertial positioning data includes:

[0021] Performing pre-integration processing on the inertial positioning data to obtain second position information;

[0022] The second residual item is obtained according to the second position information and the initial position estimation information.

[0023] In an optional manner, calculating a visual reprojection error according to the image data, and obtaining a third residual term according to the visual reprojection error, includes:

[0024] Performing image recognition on the image data to determine an image containing a reference landmark and determining an actual pixel position of the reference landmark in the corresponding image; the reference landmark is a reference object within the operating area of ​​the mowing robot;

[0025] For the reference landmark, determining an estimated pixel position of the reference landmark in the corresponding image, and obtaining a visual reprojection error based on the estimated pixel position and the actual pixel position;

[0026] The visual reprojection error is determined to be the third residual term.

[0027] In an optional manner, before constructing the cost function according to the first residual term, the second residual term, and the third residual term, the method further includes:

[0028] Get the marginalized residual term;

[0029] The constructing a cost function according to the first residual term, the second residual term, and the third residual term includes:

[0030] A cost function is constructed according to the first residual term, the second residual term, the third residual term, and the marginalized residual term.

[0031] In an optional manner, determining the first weight, the second weight, and the third weight includes:

[0032] determining the first weight according to the noise covariance corresponding to the wheel speed meter on the lawn mowing robot;

[0033] Determining the covariance of the pre-integrated noise term of the inertial positioning data according to the second weight;

[0034] The third weight is determined according to the noise covariance corresponding to the image sensor device on the lawn mowing robot.

[0035] A second aspect of the embodiments of the present application provides a positioning device for a lawn mowing robot, comprising:

[0036] A data acquisition module, configured to acquire wheel speed meter data, inertial positioning data, and image data collected by an image sensor device mounted on the mowing robot;

[0037] a residual calculation module, configured to obtain a first residual term based on the wheel speed meter data; obtain a second residual term based on the inertial positioning data; and calculate a visual reprojection error based on the image data, and obtain a third residual term based on the visual reprojection error;

[0038] The position determination module is configured to construct a cost function according to the first residual term, the second residual term, and the third residual term, and obtain the current position information of the mowing robot by minimizing the cost function.

[0039] A third aspect of the embodiments of the present application provides a computer storage medium storing at least one executable instruction, which enables a processor to implement the lawn mowing robot positioning method as described in the first aspect of the embodiments of the present application and any one of the first aspects.

[0040] According to a fourth aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor;

[0041] The memory is configured to store computer program instructions; the processor is configured to run the computer program instructions, so that the electronic device implements the lawn mowing robot positioning method as described in the first aspect of the embodiment of the present application and any one of the first aspects.

[0042] In a fifth aspect of the embodiments of the present application, a computer program product is provided, comprising: computer program instructions; an electronic device runs the computer program instructions, so that the electronic device implements the lawn mowing robot positioning method as described in the first aspect of the embodiments of the present application and any one of the first aspects.

[0043] The present application provides a robot lawn mower positioning method, device, electronic device, and readable storage medium. During the robot lawn mower's operation, the method collects corresponding data using a wheel speedometer, an inertial positioning device, and an image sensor device mounted on the robot lawn mower. The method then analyzes the collected wheel speedometer data to obtain a first residual term, analyzes the inertial positioning data to obtain a second residual term, and analyzes the image data to determine a third residual term. A cost function is constructed using the first, second, and third residual terms, and the current position of the robot lawn mower is determined by minimizing the cost function. The present application tightly couples the data collected by multiple sensors in this manner, which helps improve the accuracy of the robot lawn mower's positioning results.

[0044] The above description is only an overview of the technical solutions of the embodiments of the present application. In order to more clearly understand the technical means of the embodiments of the present application, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] FIG1 is a schematic structural diagram of a lawn mowing robot provided in one embodiment of the present application;

[0047] FIG2 is a schematic diagram of a flow chart of a positioning method for a lawn mowing robot provided in an embodiment of the present application;

[0048] FIG3 is a schematic flow chart of a positioning method for a lawn mowing robot provided in another embodiment of the present application;

[0049] FIG4 is a schematic structural diagram of a positioning device for a lawn mowing robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0051] The term "and / or" herein simply describes an association relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent three situations: A exists, A and B exist simultaneously, and B exists. Furthermore, the character " / " herein generally indicates that the related objects are in an "or" relationship. Furthermore, the terms "first," "second," and so on, in the specification and claims of this application or the accompanying drawings, are used to distinguish between different objects, not to describe a specific order, and may explicitly or implicitly include one or more of these features.

[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0053] Please refer to FIG1 , which is a schematic diagram of the structure of a lawn mower robot provided in one embodiment of the present application. As shown in FIG1 , the lawn mower robot 100 includes a memory 102, a memory controller 104, one or more (only one is shown) processors 106, a peripheral interface 108, an image sensor device 110, a wheel speed meter 112, and an inertial positioning device 114. These components communicate with each other via one or more communication buses / signal lines 116.

[0054] The memory 102 can be used to store software programs and modules, such as the computer program instructions / modules corresponding to the lawn mowing robot positioning method and device provided in the embodiments of the present application. The processor 106 executes various functional applications and data processing by running the computer program instructions and modules stored in the memory 104, thereby realizing the lawn mowing robot positioning method provided in the present application.

[0055] The memory 102 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory) or a volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory, for example, flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, etc. The RAM may include static RAM or dynamic RAM. In some embodiments, the memory 102 may be an internal storage unit of the lawn mower robot 100, for example, a hard disk or memory of the lawn mower robot 100. In other embodiments, the memory 102 may also be an external storage device of the lawn mower robot 100, such as a plug-in hard drive, a smart media card (SMC), a secure digital (SD) card, or a flash card equipped on the lawn mower robot 100. In some instances, the memory 102 may further include a memory located remotely from the lawn mower robot 100, which may be connected to the lawn mower robot 100 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. Access to the memory 102 by the processor 106 and other possible components may be controlled by the storage controller 104.

[0056] The peripheral interface 108 couples various input / output devices to the processor 106 and the memory 102. The processor 106 runs various software and instructions in the memory 102 to perform various functions of the lawn mowing robot 100 and perform data processing.

[0057] In some embodiments, the peripheral interface 108, the processor 106, and the memory controller 104 may be implemented in a single chip. In other embodiments, they may be implemented in separate chips.

[0058] In the embodiment of the present application, the processor 106 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 106 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, such as a single-chip microcomputer.

[0059] The image sensing device 110 is used to collect image data during the operation of the lawn mower robot 100. The image sensing device 110 can be, but is not limited to, a camera, a lidar device, etc. The image data collected by the image sensing device 110 can be stored in the memory 102, and the processor 106 reads the image data from the memory 102 to perform positioning of the lawn mower robot.

[0060] The wheel speed meter 112 is used to collect data such as the wheel speed (i.e., linear velocity), timestamp, angular velocity, and heading angle of the robot mower's wheels during operation as wheel speed meter data. The wheel speed meter data collected by the wheel speed meter 112 can be stored in the memory 102, and the processor 106 reads the wheel speed meter data from the memory 102 to locate the robot mower. One or more wheel speed meters 112 can be provided on the robot mower 100. For example, a wheel speed meter can be provided on each of the four wheels (the left front, right front, left rear, and right rear) of the robot mower 100. Alternatively, wheel speed meters can be provided on some wheels, such as on the left front and right front wheels, or on the left rear and right rear wheels. The wheel speed meter 112 can be a Hall-type wheel speed sensor or a magnetoelectric wheel speed sensor, which is not limited in this application. A wheel speed meter can also be referred to as a wheel encoder, odometer, or other names.

[0061] The inertial positioning device 114 can be an inertial measurement unit (IMU) or an inertial navigation system (INS). The IMU can collect inertial positioning data such as the acceleration, angular velocity, and timestamp of the mowing robot and store them in the memory 102. The INS can collect acceleration and posture data of the mowing robot and directly output the position information of the mowing robot through calculation.

[0062] In the present application, the processor 106 is specifically configured to tightly couple the data collected by the image sensing device 110 , the wheel speed meter 112 , and the inertial positioning device 114 , thereby determining the current position of the robotic lawn mower 100 .

[0063] Based on the embodiment shown in FIG1 , the lawn mower robot 100 may further include: a transceiver module 118 , an audio module 120 , a touchscreen module 122 , a button module 124 , a movement module 126 , a mowing module 128 , and the like. The transceiver module 118 may be a radio frequency module for receiving and transmitting electromagnetic waves, converting electromagnetic waves into electrical signals, and thereby communicating with a communication network or other devices. For example, the transceiver module 118 may transmit data collected by the image sensor 110 , the wheel speed meter 112 , and the inertial positioning device 114 within a time period, as well as the current position of the lawn mower robot determined by the processor 106 , to a server or a terminal device (such as a smartphone) associated with the lawn mower robot. The audio module 120 is used to collect and play audio data, among other things. The touchscreen module 122 provides both an input and output interface between the lawn mower robot 100 and the user, facilitating the user's input of control commands to the lawn mower robot 100. The button module 124 also provides an interface for the user to input input to the lawn mower robot 100. The user can press different buttons to make the lawn mower robot 100 perform different functions. The movement module 126 is used to realize the movement operation of the lawn mower robot 100. The movement module 126 includes at least wheels and a wheel drive structure. The wheels include a left wheel and a right wheel. The mowing module 128 is used to perform mowing operations. The mowing module 128 includes at least a cutter disc and a cutter disc drive structure. It will be understood that the structure shown in Figure 1 is only illustrative, and the lawn mower robot 100 may also include more or fewer components than shown in Figure 1, or have a different configuration than shown in Figure 1. The components shown in Figure 1 can be implemented using hardware, software, or a combination thereof.

[0064] FIG2 is a flow chart of a method for positioning a lawn mower robot according to an embodiment of the present application. This embodiment uses a lawn mower robot positioning device (hereinafter referred to as the positioning device) as an example for illustration. Referring to FIG2 , the method of this embodiment includes:

[0065] S201: Acquire wheel speed meter data, inertial positioning data, and image data collected by an image sensor device provided on the lawn mowing robot.

[0066] While the robot mower is operating within its designated operating area, the wheel speedometer, inertial positioning device, and image sensor device continuously collect data and store it in the robot's memory. The frequency of data collection by the wheel speedometer, inertial positioning device, and image sensor device may vary.

[0067] Wheel speed meter data can include the timestamp of the collected data, the wheel speed, heading angle, and angular velocity of the robot's wheels. Inertial positioning data can include the robot's acceleration, attitude angle, and the timestamp of the collected data collected by the IMU; or the robot's position information and the timestamp of the collected data collected by the INS.

[0068] In some embodiments, the positioning device uses a sliding time window optimization method to locate the mowing robot. Therefore, the positioning device obtains the current position information of the mowing robot by analyzing and calculating the wheel speed meter data, inertial positioning data, and image data within the sliding time window. The positioning device can periodically locate the mowing robot, and a positioning cycle can be used as a sliding time window. Alternatively, the time range of the sliding time window can be determined based on the number of frames of image data. For example, the time range corresponding to 10 consecutive frames of image data is used as a sliding time window. This application does not limit the method for determining the sliding time window.

[0069] In addition, since the data collection frequencies of the wheel speed meter, the inertial positioning device, and the image sensor may be different, the moments of each frame image in the sliding time window can be used as a reference to determine each moment in the sliding time window.

[0070] S202: Obtain a first residual term according to the wheel speed meter data.

[0071] The positioning device can read the wheel speed meter data within the sliding time window from the memory, integrate the wheel speed meter data to obtain the first position information of each moment in the sliding time window; then, for each moment in the sliding time window, the first residual term of each moment is obtained according to the error between the first position information of the moment and the initial position estimation information of the corresponding moment.

[0072] The initial position estimation information is the initial system state quantity, and may include the estimated position of the mowing robot at each moment within the sliding time window. The initial position estimation information may be, but is not limited to, calculated using inertial positioning data.

[0073] The first position information may include the coordinate values ​​of the mowing robot in a set coordinate system. Similarly, the initial position estimate information may include the estimated coordinate values ​​of the mowing robot in the set coordinate system. The starting position of the mowing robot during this lawn maintenance operation may be determined as the origin of the set coordinate system, with the mowing robot's movement direction as the horizontal axis and the direction perpendicular to the movement direction of the mowing robot as the vertical axis.

[0074] The error between the first position information and the initial position estimation information can be realized by, but is not limited to, the Marr distance.

[0075] S203: Obtain a second residual term according to the inertial positioning data.

[0076] Among them, the positioning device can read the inertial positioning data within the sliding time window from the memory. When the inertial positioning data is IMU data, the positioning device can obtain the second position information at each moment by pre-integrating the inertial positioning data between any two adjacent moments in the sliding time window. The pre-integration processing can solve the relative posture between two adjacent frames of data without the need to re-integrate all IMU data, which can reduce the amount of calculation and improve positioning efficiency. When the inertial positioning data is INS data, the positioning device can directly obtain the second position information of the lawn mower robot at each moment in the sliding time window from the memory.

[0077] Then, for each moment in the sliding time window, the error between the second position information at that moment and the initial position estimate information is calculated to obtain the second residual term at that moment, thereby obtaining the second residual term at each moment. The error between the second position information and the initial position estimate information can be, but is not limited to, realized by using the Marr distance.

[0078] S204: Calculate a visual reprojection error according to the image data, and obtain a third residual term according to the visual reprojection error.

[0079] The visual reprojection error refers to the difference between the projection of a three-dimensional space point in the real world on the image plane (i.e., the pixel point on the image) and the reprojection. Among them, reprojection refers to the second projection. The first projection refers to the projection of the three-dimensional space point onto the image when the image sensing device collects image data, that is, the three-dimensional space point in the world coordinate system is mapped to the image coordinate system corresponding to the image sensing device (also called the camera coordinate system), and then converted to the pixel coordinate system to obtain the pixel coordinates. According to the pixel coordinates of some feature points in the image, the coordinates of the three-dimensional space point (not real) are obtained by reverse deduction using geometric information, and the coordinate conversion relationship from the three-dimensional space point to the pixel point can be obtained. Reprojection refers to the second projection, which refers to the re-projection based on the calculated virtual three-dimensional space point and coordinate conversion relationship. The difference between the pixel points obtained in the two projection processes is the visual reprojection error.

[0080] Multiple dedicated reference landmarks for positioning can be pre-set within the robot's operating area. These landmarks can be placed as close to the edge of the operating area as possible, or in locations where lawn maintenance is not required, to minimize the impact of the landmarks on the robot. Alternatively, the landmarks can be objects already present in the robot's operating area. This application does not limit the use of landmarks. The image data captured by the image sensor may include pixels corresponding to one or more reference landmarks. In this application, the reference landmarks are used as feature points in three-dimensional space to calculate visual reprojection errors.

[0081] Determining the visual reprojection error can be done by following these steps:

[0082] First, the positioning device may read image data within a sliding time window from a memory, the image data including multiple frames, and perform image recognition on the image data to determine whether the image contains a reference landmark and the actual pixel location of the reference landmark in the image. The positioning device may perform image recognition on the image data using an image recognition model, which may be, but is not limited to, a deep neural network model, a convolutional network model, and the like. For example, the image recognition model may process the input image to extract Harris corners, track adjacent frames using pyramid optical flow, and remove outliers using RANSAC, thereby identifying the reference landmarks in the image and their actual pixel locations in the image.

[0083] Next, the visual reprojection error is calculated based on the pixel points of the reference landmark in the corresponding image. Images with the same reference landmark can be divided into the same image set according to the chronological order of image acquisition and the reference landmark. Then, for a reference landmark, the reference landmark is converted from its first observed camera coordinate system to the current camera coordinate system to obtain an estimated pixel position. The error between the estimated pixel position and the actual pixel position is then calculated to obtain the visual reprojection error. The visual reprojection error is the third residual term. Calculating the error between the estimated pixel position and the actual pixel position can be achieved, but is not limited to, using the Marr distance.

[0084] It should be noted that one frame of image may contain multiple reference landmarks, and therefore one frame of image may be divided into image sets corresponding to different reference landmarks.

[0085] It should be noted that the execution order of step S202 and step S204 is not particular, and in this embodiment, parallel execution is used as an example.

[0086] S205 : Constructing a cost function according to the first residual term, the second residual term, and the third residual term, and obtaining the current position information of the mowing robot by minimizing the cost function.

[0087] A cost function maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. In applications, it is often solved by minimizing the cost function. In this application, the cost function is constructed by fusing the first residual term between the wheel speed meter data and the inertial positioning data with the second residual term of the image data. The current position information of the lawn mower robot is solved by minimizing the cost function.

[0088] The cost function can be constructed in the following ways:

[0089] First, a first weight, a second weight, and a third weight are determined, wherein the first weight is related to the wheel speed meter data, the second weight is related to the inertial positioning data, and the third weight is related to the image data. In some embodiments, the first weight can be determined based on the noise covariance corresponding to the wheel speed meter on the lawn mower robot. The second weight can be determined based on the covariance of the pre-integrated noise term of the inertial positioning data collected by the inertial positioning device. The third weight can be determined based on the noise covariance corresponding to the image sensing device on the lawn mower robot. The first weight, the second weight, and the third weight can be the inverse of the corresponding covariance. The first weight and the third weight can be obtained based on the calibration data when the wheel speed meter and image sensing device on the lawn mower robot are calibrated in advance and stored in the memory of the lawn mower robot.

[0090] The larger the covariance, the less reliable the corresponding observation data. Since the weight is the inverse of the covariance, the larger the weight, the more reliable the corresponding observation data. For example, a larger first weight indicates a more reliable wheel speed meter observation. A larger second weight indicates a more reliable inertial positioning device observation. A larger third weight indicates a more reliable image observation.

[0091] Next, the first weight is fused with the first residual item to obtain the first data item; the second weight is fused with the second residual item to obtain the second data item; the third weight is fused with the third residual item to obtain the third data item; the first data item, the second data item and the third data item are added together to obtain the cost function.

[0092] The fusion of weights and residuals can be achieved through multiplication. For example, the first weight is multiplied by the first residual at each moment in the sliding time window, and the sum is calculated to obtain the first data item. The second weight is multiplied by the second residual at each moment in the sliding time window, and the sum is calculated to obtain the second data item. The third weight is multiplied by the reprojection errors corresponding to the images of the observed reference landmarks, and the sum is calculated to obtain the third data item.

[0093] For example, the cost function can be expressed as formula (1):

[0094] In formula (1), c(x) represents the cost function; represents the first residual term corresponding to the k-th frame image in the sliding time window; W p represents the first weight; K represents the total number of images in the sliding time window; T represents transposition; W represents the second residual term corresponding to the k-th frame image in the sliding time window; s represents the second weight; represents the visual reprojection error of the reference landmark l in the j-th frame image, i.e., the third residual term; β l represents the image set where the reference landmark l appears; W r Represents the third weight.

[0095] The system state vector x can be obtained by minimizing the cost function. The system state vector x may include the states of all image sensing devices within a sliding time window (including position, orientation, velocity, accelerometer bias, and gyroscope bias), external parameters from the image sensing device to the inertial positioning device, and the inverse depth of some feature points. The position estimation information of the lawn mower robot can be obtained by integrating the elements in the system state vector x, and the position estimation information is determined as the current position information of the lawn mower robot. The process of minimizing the cost function is similar to the existing method of solving the cost function and can be implemented using any existing method, which is not limited by this application.

[0096] The method provided in this embodiment collects data from a wheel speedometer, inertial positioning device, and image sensor installed on the mowing robot during operation. The method then analyzes the collected wheel speedometer data to obtain a first residual term, analyzes the inertial positioning data to obtain a second residual term, and analyzes the image data to determine a third residual term. A cost function is constructed using the first, second, and third residual terms, and the current position of the mowing robot is determined by minimizing the cost function. This method tightly couples the data collected by multiple sensors, thereby improving the accuracy of the mowing robot's positioning results.

[0097] FIG3 is a flow chart of a positioning method for a lawn mowing robot according to another embodiment of the present application. Referring to FIG3 , the method of this embodiment includes:

[0098] S301: Acquire wheel speed meter data, inertial positioning data, and image data collected by an image sensor device provided on the lawn mowing robot.

[0099] S302: Obtain a first residual term according to the wheel speed meter data.

[0100] S303: Obtain a second residual term according to the inertial positioning data.

[0101] S304: Calculate a visual reprojection error according to the image data, and obtain a third residual term according to the visual reprojection error.

[0102] Steps S301 to S304 of this embodiment are similar to steps S201 to S204 of the embodiment shown in FIG2 , and reference may be made to the detailed description of the embodiment shown in FIG2 . For the sake of brevity, they will not be repeated here.

[0103] S305: Determine the marginalized residual term.

[0104] Marginalization involves removing older or substandard frames from the sliding time window, targeting the oldest or second-most recent frame. The goal of marginalization is to not calculate the visual reprojection error corresponding to the reference landmarks in these removed frames, while preserving their constraints on other frames within the sliding time window.

[0105] The positioning device can determine whether the next latest frame in the sliding time window is a key frame. If the next latest frame is a key frame, the oldest frame in the sliding time window, its observed reference landmarks and associated inertial positioning data are marginalized out, and converted into a priori terms and added to the overall cost function. If the next latest frame is not a key frame, the next latest frame and its visual observation edges are discarded, but the inertial positioning data of the next latest frame is retained. Through the above process, the prior term is constructed, and the system state quantity in the sliding time window is optimized. That is, when the lawn mowing robot is positioned, the prior term is added to the cost function as a marginalized residual term and optimized together, thereby obtaining the latest state estimation result without losing historical information. Among them, the construction of the prior term can be achieved using the Schur complement formula.

[0106] Among them, marginalization can also be achieved by using other strategies, which are not limited in this application.

[0107] It should be noted that the execution order of steps S302, S303, S304 and S305 is not particular.

[0108] S306 : Constructing a cost function according to the first residual term, the second residual term, the third residual term, and the marginalized residual term, and obtaining the current position information of the mowing robot by minimizing the cost function.

[0109] Among them, the implementation method of determining the first data item based on the first residual item, determining the second data item based on the second residual item, and determining the third data item based on the third residual item can be determined by referring to the detailed description of step S205 in the embodiment shown in Figure 2, and will not be repeated here.

[0110] When constructing the cost function, for the marginalized residual term, the marginalized residual term and the transposed matrix of the marginalized residual term can be multiplied to obtain a fourth data term, and then the fourth data term is added to the first data term, the second data term, and the third data term to obtain the cost function. The cost function provided in this embodiment is shown in formula (2):

[0111] e m represents the marginalized residual term.

[0112] Similar to the embodiment shown in FIG2 , the system state vector x is calculated by minimizing formula (2), and the position estimation information of the lawn mower robot is obtained by performing integration processing based on the elements in the system state vector x, and the position estimation information is determined as the current position information of the lawn mower robot.

[0113] The method of this embodiment, when positioning the lawn mower robot based on data collected by multiple sensors, also considers the influence of the oldest image or the second-newest frame on positioning, and quantifies this influence as a marginalized residual term and adds it to the cost function. This ensures that the marginalized images and related data constrain other frames during the overall optimization, but the marginalized data are no longer calculated, thereby improving the positioning speed and ensuring the accuracy of the positioning results.

[0114] FIG4 is a schematic diagram of the structure of a positioning device for a lawn mowing robot provided in an embodiment of the present application. Referring to FIG4 , the device 400 provided in this embodiment includes:

[0115] The data acquisition module 401 is used to acquire wheel speed meter data, inertial positioning data and image data collected by an image sensor device provided on the lawn mowing robot.

[0116] The residual calculation module 402 is configured to obtain a first residual term based on the wheel speed meter data; obtain a second residual term based on the inertial positioning data; and calculate a visual reprojection error based on the image data and obtain a third residual term based on the visual reprojection error.

[0117] The position determination module 403 is configured to construct a cost function according to the first residual term, the second residual term, and the third residual term, and obtain the current position information of the mowing robot by minimizing the cost function.

[0118] In an optional manner, the position determination module 403 is specifically used to determine a first weight, a second weight, and a third weight; the first weight is related to the wheel speed meter data; the second weight is related to the inertial positioning data, and the third weight is related to the image data; a first data item is obtained according to the first residual item and the first weight, a second data item is obtained according to the second residual item and the second weight, and a third data item is obtained according to the third residual item and the third weight; and the first data item, the second data item, and the third data item are added together to obtain the cost function.

[0119] In an optional manner, the residual calculation module 402 is specifically configured to perform integration processing on the wheel speed meter data to obtain first position information; and obtain the first residual term according to the first position information and initial position estimation information.

[0120] In an optional manner, the residual calculation module 402 is specifically configured to perform pre-integration processing on the inertial positioning data to obtain second position information; and obtain the second residual term according to the second position information and the initial position estimation information.

[0121] In an optional manner, the residual calculation module 402 is specifically used to perform image recognition on the image data to determine an image containing a reference landmark and to determine the actual pixel position of the reference landmark in the image to which it belongs; the reference landmark is a reference object within the working area of ​​the lawn mowing robot; for the reference landmark, the estimated pixel position of the reference landmark in the image to which it belongs is determined, and a visual reprojection error is obtained based on the estimated pixel position and the actual pixel position; and the visual reprojection error is determined to be the third residual term.

[0122] In an optional manner, the residual calculation module 402 is further configured to obtain a marginalized residual term.

[0123] The position determination module 403 is specifically configured to construct a cost function according to the first residual term, the second residual term, the third residual term, and the marginalized residual term.

[0124] In an optional manner, the position determination module 403 is specifically used to determine the first weight based on the noise covariance corresponding to the wheel speed meter on the lawn mowing robot; determine the second weight based on the covariance of the pre-integrated noise term of the inertial positioning data; and determine the third weight based on the noise covariance corresponding to the image sensing device on the lawn mowing robot.

[0125] The device provided in this embodiment can be used to implement the technical solution of any of the previous method embodiments. Its implementation principles and technical effects are similar and will not be repeated here for the sake of brevity.

[0126] Another embodiment of the present application also provides an electronic device, comprising: a memory and a processor, wherein the memory is configured to store computer program instructions, and the processor is configured to execute the computer program instructions, so that the processor can perform the functional actions specified in each step or a combination of steps in flowcharts 2 and 3, thereby realizing the positioning of the lawn mowing robot.

[0127] Another embodiment of the present application further provides a computer-readable medium (also referred to as a readable storage medium), which may be a computer-readable signal medium or a computer-readable medium. A processor in a computer reads computer program instructions stored in the computer-readable medium, enabling the processor to execute the functional actions specified in each step or combination of steps in flowcharts 2 and 3, and to generate a device that implements the functional actions specified in each block or combination of blocks in the block diagram.

[0128] Computer-readable media include but are not limited to electronic, magnetic, optical, electromagnetic, infrared memory or semiconductor systems, devices or apparatuses, or any appropriate combination of the foregoing. The memory is used to store program codes or instructions, the program codes include computer operating instructions, and the processor is used to execute the program codes or instructions of the lawn mowing robot positioning method stored in the memory.

[0129] For the definitions of memory and processor, please refer to the description of the aforementioned computer device embodiment and will not be repeated here.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0131] Each functional unit or module in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0133] In the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" described in the present application does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application can be implemented with the aid of hardware comprising several different elements and with the aid of a suitably programmed computer. In a unit claim that lists several means, several units of these means may be embodied by the same hardware item. The use of first, second, and third etc. does not indicate any order and these words may be interpreted as names. The steps in the above embodiments should not be understood as limiting the order of execution unless otherwise specified.

[0134] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A lawn mowing robot positioning method, characterized in that: The method comprises: Acquire wheel speed meter data, inertial positioning data of the lawn mowing robot, and image data collected by an image sensing device disposed on the lawn mowing robot; Obtaining a first residual term according to the wheel speed meter data; Obtaining a second residual term according to the inertial positioning data; Calculating a visual reprojection error according to the image data, and obtaining a third residual term according to the visual reprojection error; A cost function is constructed according to the first residual term, the second residual term and the third residual term, and the current position information of the lawn mowing robot is obtained by minimizing the cost function.

2. The method according to claim 1, characterized in that: The step of constructing a cost function according to the first residual term, the second residual term and the third residual term, and obtaining the current position information of the lawn mowing robot by minimizing the cost function includes: Determine a first weight, a second weight, and a third weight; the first weight is related to the tachometer data; the second weight is related to the inertial positioning data, and the third weight is related to the image data; Obtain a first data item according to the first residual item and the first weight, obtain a second data item according to the second residual item and the second weight, and obtain a third data item according to the third residual item and the third weight; The cost function is obtained by adding the first data item, the second data item and the third data item.

3. The method according to claim 1, characterized in that: The step of obtaining a first residual term according to the wheel speed meter data comprises: Integrating the wheel speed meter data to obtain first position information; The first residual term is obtained according to the first position information and initial position estimation information.

4. The method according to claim 1, characterized in that: The obtaining a second residual term according to the inertial positioning data comprises: Performing pre-integration processing on the inertial positioning data to obtain second position information; The second residual term is obtained according to the second position information and the initial position estimation information.

5. The method according to claim 1, characterized in that The calculating the visual reprojection error according to the image data, and obtaining the third residual term according to the visual reprojection error, comprises: Performing image recognition on the image data to determine an image containing a reference landmark and determining an actual pixel position of the reference landmark in the corresponding image; the reference landmark is a reference object within the operating area of ​​the lawn mowing robot; For the reference landmark, determining an estimated pixel position of the reference landmark in the image to which it belongs, and obtaining a visual reprojection error according to the estimated pixel position and the actual pixel position; The visual reprojection error is determined as the third residual term.

6. The method according to claim 1, characterized in that Before constructing the cost function according to the first residual term, the second residual term and the third residual term, the method further includes: Get the marginalized residual term; The constructing a cost function according to the first residual term, the second residual term and the third residual term includes: constructing a cost function according to the first residual term, the second residual term, the third residual term and the marginalized residual term.

7. The method according to claim 2, characterized in that The determining of the first weight, the second weight and the third weight comprises: Determine the first weight according to the noise covariance corresponding to the wheel speed meter on the lawn mowing robot; determining the second weight according to the covariance of the pre-integrated noise term of the inertial positioning data; The third weight is determined according to the noise covariance corresponding to the image sensor device on the lawn mowing robot.

8. A lawn mowing robot positioning device, characterized in that: The device comprises: A data acquisition module, used to acquire wheel speed meter data, inertial positioning data of the lawn mowing robot, and image data collected by an image sensor device arranged on the lawn mowing robot; A residual calculation module, configured to obtain a first residual term according to the wheel speed meter data; obtain a second residual term according to the inertial positioning data; and calculate a visual reprojection error according to the image data, and obtain a third residual term according to the visual reprojection error; The position determination module is used to construct a cost function according to the first residual term, the second residual term and the third residual term, and obtain the current position information of the lawn mowing robot by minimizing the cost function.

9. An electronic device, characterized in that: The robot comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the robot positioning method according to any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the robot lawn mowing positioning method according to any one of claims 1 to 7 are implemented.

Images