Control method of a self-moving device and self-moving device

Abstract

The application relates to a control method of a self-moving device and the self-moving device. The self-moving device comprises a non-scanning single-frame pulse imaging sensor configured to acquire three-dimensional point cloud data of a surrounding environment of the self-moving device, and a vision sensor configured to acquire image data of the surrounding environment of the self-moving device. The control method of the self-moving device comprises: acquiring the three-dimensional point cloud data and the image data; and determining a pose of the self-moving device according to at least the three-dimensional point cloud data and the image data. Another control method of the self-moving device determines the pose of the self-moving device according to three-dimensional point cloud data and image data acquired by a sensor module. The sensor module comprises a mounting shell, a laser radar sensor and a vision sensor, the laser radar sensor and the vision sensor are both mounted on the mounting shell, and have a preset external parameter calibration relationship; and the sensor module is configured to be capable of being mounted on the self-moving device as a whole.

Description

Technical Field

[0001] This application relates to the field of self-moving device technology, and in particular to a control method for a self-moving device and a self-moving device. Background Technology

[0002] Self-moving devices, such as automatic lawnmowers, can move autonomously and complete tasks without direct human control or operation, significantly reducing reliance on manpower and thus gaining increasing popularity among users. One of the core prerequisites for autonomous movement of self-moving devices is their ability to obtain real-time and accurate location information in the working environment, i.e., positioning capability.

[0003] Among related technologies, single-LiDAR positioning technology relies on geometric structures. In geometrically degraded scenarios such as open planes, long straight corridors, and symmetrical structures, insufficient point cloud constraints can easily lead to pose drift. Meanwhile, single-vision positioning technology is extremely sensitive to lighting and texture, and is prone to losing feature tracking in low-light, strong-light, and solid-color wall scenarios. This results in positioning failure or decreased accuracy, making reliable positioning difficult to achieve. Summary of the Invention

[0004] This application provides a control method for a self-moving device and a self-moving device, which can effectively improve positioning accuracy and enhance positioning robustness.

[0005] In a first aspect, this application provides a control method for a self-moving device, the self-moving device comprising: A non-scanning single-frame pulse imaging sensor is configured to acquire three-dimensional point cloud data of the surrounding environment of the self-moving device; and A visual sensor is configured to acquire image data of the surrounding environment of the self-moving device; The control method for the self-moving device includes: Acquire the 3D point cloud data and the image data; The pose of the self-moving device is determined based at least on the three-dimensional point cloud data and the image data.

[0006] In some implementations, determining the pose of the self-moving device based at least on the 3D point cloud data and the image data includes: The pose of the self-moving device is solved by joint optimization using a tightly coupled state estimation model, based at least on the three-dimensional point cloud data and the image data.

[0007] In some embodiments, the self-moving device further includes a state detection unit configured to detect motion state data of the self-moving device; Determining the pose of the self-moving device based at least on the 3D point cloud data and the image data includes: The pose of the self-moving device is determined based on the 3D point cloud data, the image data, and the motion state data.

[0008] In some embodiments, the state detection unit includes an inertial measurement unit; the motion state data includes acceleration and angular velocity; And / or, the state detection unit includes a wheeled odometer; the motion state data includes displacement and yaw angle.

[0009] In some implementations, determining the pose of the self-moving device based on the 3D point cloud data, the image data, and the motion state data includes the following steps: Data preprocessing: Spatiotemporal alignment and motion distortion correction are performed on the 3D point cloud data, the image data, and the motion state data to obtain preprocessed multi-source data; State prediction: Based on the motion state data, predict the current state of the self-moving device and output a priori state estimate; and Tightly Coupled Observation Update: Based on the multi-source data and the global map, determine the geometric constraint residual of the 3D point cloud data and the photometric constraint residual of the image data. Based on the geometric constraint residual and the photometric constraint residual, use an iterative error state filtering model to jointly update the prior state estimate to obtain the posterior state estimate, so as to determine the pose of the self-moving device.

[0010] In some embodiments, the spatiotemporal alignment and motion distortion correction of the 3D point cloud data, the image data, and the motion state data to obtain preprocessed multi-source data includes: Based on the motion state data, an inter-frame motion model is constructed, and based on the inter-frame motion model, motion distortion correction is performed on the three-dimensional point cloud data; And / or, time-align the 3D point cloud data, the image data, and the motion state data according to the timestamp; spatially align the 3D point cloud data, the image data, and the motion state data according to the spatial synchronization parameters, and project them onto the same coordinate system; thus obtaining spatiotemporally synchronized multi-source data.

[0011] In some implementations, the step of predicting the current state of the self-moving device based on the motion state data and outputting a priori state estimate includes: Based on the motion state data, an inter-frame motion model is constructed; The posterior state estimate obtained from the previous frame is used as the initial prediction value. The current state of the self-moving device is predicted based on the inter-frame motion model, and the prior state estimate is output. The prior state estimation includes the prior pose and its covariance matrix.

[0012] In some implementations, the global map includes a global point cloud map and a global visual map; The step of constructing the geometric constraint residual of the 3D point cloud data and the photometric constraint residual of the image data based on the multi-source data and the global map includes: The preprocessed 3D point cloud data is projected onto the global point cloud map, neighboring points are retrieved to fit the local plane, the distance residual from the point to the local plane is calculated, and the Jacobian matrix of the distance residual is constructed to obtain the geometric constraint residual. Based on the preprocessed image data and the global visual map, the photometric error residual between the current image frame and the reference image frame is calculated, and the Jacobian matrix of the photometric error residual is constructed to obtain the photometric constraint residual.

[0013] In some embodiments, the iterative error state filtering model is an iterative error state Kalman filter model; Based on the geometric constraint residuals and the photometric constraint residuals, an iterative error state filtering model is used to jointly update the prior state estimate to obtain the posterior state estimate, including: Substitute the geometric constraint residual and the photometric constraint residual into the iterative error state Kalman filter model to jointly update the prior state estimate and obtain the posterior state estimate. The posterior state estimation includes the posterior pose of the self-moving device and its covariance matrix.

[0014] In some implementations, the global map includes a global point cloud map and a global visual map; The control method for the self-moving device further includes: The global point cloud map is updated based on the three-dimensional point cloud data; The global visual map is updated based on the image data and the 3D point cloud data.

[0015] In some embodiments, the self-moving device includes a sensor module, which includes a mounting housing, the non-scanning single-frame pulse imaging sensor, and the vision sensor. Both the non-scanning single-frame pulse imaging sensor and the vision sensor are mounted on the mounting housing, and there is a preset external parameter calibration relationship between the non-scanning single-frame pulse imaging sensor and the vision sensor; the sensor module is configured to be mounted as a whole on the self-moving device. In some implementations, the vision sensor is a monocular camera.

[0016] In some embodiments, the self-moving device is an automatic lawnmower; the automatic lawnmower includes: body; A walking assembly, mounted on the machine body, includes walking wheels and a self-drive motor for driving the walking wheels to move; A mowing assembly, mounted on the machine body, includes a cutting blade and a mowing motor for driving the cutting blade to rotate; and The control module is configured to connect to and control the self-driven motor and the lawnmower motor.

[0017] Secondly, this application also provides a control method for a self-moving device, the self-moving device including a sensor module, the sensor module comprising: Mounting housing; A lidar sensor is configured to acquire three-dimensional point cloud data of the surrounding environment of the self-moving device; and A visual sensor is configured to acquire image data of the surrounding environment of the self-moving device; Both the lidar sensor and the vision sensor are mounted on the mounting housing, and there is a preset external parameter calibration relationship between the lidar sensor and the vision sensor; the sensor module is configured to be mounted as a whole on the self-moving device; The control method for the self-moving device includes: Acquire the 3D point cloud data and the image data; The 3D point cloud data and the image data are time-aligned according to the timestamp; the 3D point cloud data and the image data are spatially aligned according to the preset extrinsic parameter calibration relationship and projected onto the same coordinate system; thus obtaining spatiotemporally synchronized multi-source data. Based on the multi-source data, the pose of the self-moving device is determined.

[0018] Thirdly, this application provides a self-moving device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the above-described control method for the self-moving device.

[0019] Compared with the prior art, this application has the following beneficial effects: it can effectively improve positioning accuracy and enhance positioning robustness. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the working scenario of a self-moving device according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a self-moving device according to an embodiment of this application; Figure 3 This is a flowchart of a control method for a self-moving device according to an embodiment of this application; Figure 4 This is a flowchart illustrating a control method for a self-moving device according to another embodiment of this application; Figure 5 This is a flowchart of a control method for a self-moving device according to another embodiment of this application; Figure 6 This is a schematic electrical block diagram of a self-moving device according to an embodiment of this application; Figure 7 This is a schematic electrical block diagram of a self-moving device according to yet another embodiment of this application; Figure 8 This is a schematic circuit block diagram of an electronic device according to an embodiment of this application; Figure 9 This is a schematic electrical block diagram of an automatic lawnmower according to an embodiment of this application. Detailed Implementation

[0021] The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the application. For example, terms such as "upper," "lower," "front," and "rear" that indicate orientation or positional relationship are based solely on the orientation or positional relationship shown in the accompanying drawings and are used only for the convenience of describing the application and simplifying the description, and are not intended to indicate or imply that the device / component referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the application.

[0022] Reference Figure 1 The self-moving device 10, such as an automatic lawnmower, operates within its designated work area. When not in use or needing charging, it returns to the charging station 20 to dock or recharge. The self-moving device 10 is equipped with sensors that perceive its surrounding environment to achieve autonomous positioning and map building, thus providing a basis for navigation. Of course, the self-moving device can also be an indoor smart home device, such as an automatic robot vacuum cleaner or a robotic vacuum cleaner.

[0023] Among related technologies, single-LiDAR positioning technology relies on geometric structures. In geometrically degraded scenarios such as open planes, long straight corridors, and symmetrical structures, insufficient point cloud constraints can easily lead to pose drift. Meanwhile, single-vision positioning technology is extremely sensitive to lighting and texture, and is prone to losing feature tracking in low-light, strong-light, and solid-color wall scenarios. This results in positioning failure or decreased accuracy, making reliable positioning difficult to achieve.

[0024] To address the aforementioned problems, this application provides a control method for a self-moving device. Taking an automatic lawnmower as an example, refer to... Figure 2 The self-moving device 10 includes: a non-scanning single-frame pulse imaging sensor 101 (also known as a detection module that achieves multi-dimensional imaging without mechanical scanning), configured to acquire three-dimensional point cloud data of the surrounding environment of the self-moving device; and a vision sensor 102, configured to acquire image data of the surrounding environment of the self-moving device. (Refer to...) Figure 3 The control methods for self-moving devices include: S11. Acquire 3D point cloud data and image data; S12. Determine the pose of the self-moving device based at least on the 3D point cloud data and image data.

[0025] Non-scanning single-frame pulse imaging sensors refer to sensors that can achieve multi-dimensional imaging with a single laser pulse emission without mechanical scanning. They can acquire three-dimensional information within the entire field of view using two-dimensional or multi-dimensional arrays of laser emitters and receivers, achieving instantaneous imaging. In specific embodiments, this can be a solid-state area array lidar (also known as Flash LiDAR). Compared to traditional mechanical lidar that uses rotating mirrors to scan the surrounding environment, solid-state area array lidar can achieve faster and more accurate scanning by using solid-state emitter and receiver arrays.

[0026] It is understandable that 3D point cloud data acquired by a non-scanning single-frame pulse imaging sensor has high accuracy, which can improve 3D perception performance, and the data real-time performance is higher compared to mechanical scanning sensors. Furthermore, by combining 3D point cloud data and image data, sensor complementarity can be achieved, taking into account both geometric accuracy and texture constraints. When geometric degradation occurs, visual texture constraints are used as a fallback, and when visual failure occurs in scenarios such as low light and weak texture, geometric constraints of LiDAR are used to maintain stability. This effectively solves the scene adaptation shortcomings of single sensors, and achieves a dual improvement in positioning robustness and accuracy, enabling more accurate and real-time determination of the pose of mobile devices.

[0027] Reference Figure 9 The automatic lawnmower includes: a body 104; a walking assembly mounted on the body 104, including walking wheels 1101 and a self-drive motor 1102 for driving the walking wheels 1101 to move; a mowing assembly mounted on the body 104, including cutting blades 1201 and a mowing motor 1202 for driving the cutting blades 1201 to rotate; and a control module 130 configured to connect to and control the self-drive motor 1102 and the mowing motor 1202. It should be noted that the components of the walking assembly and the mowing assembly can be installed inside the body or on the body; they can be enclosed or partially exposed; they can be fixed or relatively movable; no specific limitations are made here.

[0028] Furthermore, in some embodiments, step S12 includes: solving for the pose of the self-moving device by performing joint optimization through a tightly coupled state estimation model based at least on 3D point cloud data and image data.

[0029] It is understandable that, compared to loose coupling, where each sensor independently calculates its pose and then fuses the data, tight coupling can make fuller use of the complementary characteristics between the sensors, and perform deep fusion directly at the raw data level, achieving bidirectional, high-frequency interaction and correction, thereby achieving a better balance in positioning accuracy, system robustness and adaptability to complex environments.

[0030] Among related technologies, laser inertial odometry (LIO) relies on the geometric point cloud of a lidar and pre-integration by an inertial measurement unit (IMU) to achieve positioning. It has the advantages of being illumination-independent and geometrically robust. However, in geometrically degenerate scenarios such as open planes, long straight corridors, and symmetrical structures, insufficient point cloud constraints can easily lead to pose drift. Furthermore, pure geometric observation lacks texture information, and mismatches are prone to occur in repetitive structural scenes. At the same time, registration of dense point clouds incurs significant computational costs. Visual inertial odometry (VIO) uses visual features fused with an IMU for positioning. It has significant advantages in terms of lightweight design and excellent accuracy in scenes with rich textures. However, it is extremely sensitive to illumination and texture: feature tracking is easily lost in scenes with low light, strong light, and solid-color walls. Monocular VIO suffers from scale ambiguity and is prone to divergence in high-speed motion and dynamic interference scenarios, thus limiting its robustness. Both LIO and VIO suffer from low positioning accuracy and poor robustness.

[0031] To address the aforementioned issues, in some embodiments, the self-moving device further includes a state detection unit configured to detect motion state data of the self-moving device. Step S12 includes: determining the pose of the self-moving device based on the 3D point cloud data, image data, and motion state data.

[0032] Furthermore, the state detection unit includes an inertial measurement unit (IMU) 105; motion state data includes acceleration and angular velocity. Exemplarily, the inertial measurement unit integrates a three-axis micro-electro-mechanical systems (MEMS) accelerometer and a three-axis MEMS gyroscope. The accelerometer measures the acceleration of the self-moving device body in three directions (forward-backward, left-right, up-down), including the gravitational acceleration component; the gyroscope measures the angular velocity of the self-moving device body about three axes. For example, acceleration and angular velocity in the three directions x, y, z. The sampling frequency of the IMU is much higher than that of the lidar; in this embodiment, it is set to 200Hz. The raw data output by the IMU includes sensor bias error and measurement noise, which need to be compensated for in subsequent processing. The IMU data resides in the IMU's own sensor coordinate system.

[0033] It is understandable that combining 3D point cloud data, image data, and motion state data allows for joint optimization of pose through triple observation constraints. The inertial measurement unit provides high-frequency predictions, while the lidar can eliminate dynamic interference and provide static priors for vision. The three work together to suppress cumulative drift, balancing geometric accuracy and texture constraints, significantly improving the robustness, accuracy, and generalization ability of positioning in complex scenes, extreme lighting, and dynamic environments.

[0034] Furthermore, referring to Figure 4 Step S12 includes the following steps: S121. Data preprocessing: Spatiotemporal alignment and motion distortion correction are performed on 3D point cloud data, image data and motion state data to obtain preprocessed multi-source data. S122, State Prediction: Based on motion state data, predict the current state of the self-moving device and output a prior state estimate; and S123. Tightly Coupled Observation Update: Based on multi-source data and the global map, determine the geometric constraint residuals of the 3D point cloud data and the photometric constraint residuals of the image data. Based on the geometric constraint residuals and photometric constraint residuals, use the iterative error state filtering model to jointly update the prior state estimate to obtain the posterior state estimate, so as to determine the pose of the self-moving device.

[0035] By using data preprocessing, state prediction, and tightly coupled observation updates, positioning accuracy and robustness can be effectively improved.

[0036] Furthermore, in step S121, data preprocessing also includes: filtering the 3D point cloud data and image data for data validity according to preset filtering conditions, removing invalid data; and downsampling the 3D point cloud data and image data. It can be understood that removing invalid data avoids interference from outliers, thereby improving system robustness; downsampling retains key features, reduces data volume, saves computational overhead, and improves system real-time performance.

[0037] During operation, the self-moving device scans the field of view (e.g., 170° × 80°) directly in front of it, using its built-in driver to parse and receive the point cloud data, thus obtaining the aforementioned three-dimensional point cloud data. The three-dimensional point cloud data includes the spatial coordinates (x, y, z) of each point, its reflection intensity, and its row and column indices (row, col). The spatial coordinates (x, y, z) characterize the geometric position of each point in three-dimensional space; the reflection intensity characterizes the echo signal intensity of the target surface corresponding to each point; the row and column indices (row, col) are calculated based on the vertical pitch angle of each laser point, and the column index (col) is calculated based on the horizontal azimuth angle of each laser point. These row and column indices (row, col) construct a mapping relationship from two-dimensional indices to three-dimensional coordinates.

[0038] Because LiDAR acquires a complete frame of point cloud data not instantaneously, but requires a scanning cycle, typically 50-200 milliseconds, during which the mobile device itself is in motion, points in a single frame of point cloud come from different poses at different times. This causes point cloud distortion, severely affecting the accuracy of point cloud matching, localization, and mapping. Therefore, motion distortion correction is necessary for 3D point cloud data. Since different sensors have different acquisition times, frequencies, and coordinate systems, directly treating data observed at different times and in different spaces as equivalent will cause a mismatch between observations and reality, leading to fusion errors or divergence. Therefore, spatiotemporal alignment of multi-sensor data is required.

[0039] In step S121, spatiotemporal alignment and motion distortion correction are performed on the 3D point cloud data, image data, and motion state data to obtain preprocessed multi-source data. This includes: constructing an inter-frame motion model based on the motion state data, and performing motion distortion correction on the 3D point cloud data based on the inter-frame motion model; temporally aligning the 3D point cloud data, image data, and motion state data based on timestamps; spatially aligning the 3D point cloud data, image data, and motion state data based on spatial synchronization parameters, and projecting them onto the same coordinate system; thus obtaining spatiotemporally synchronized multi-source data.

[0040] In one specific embodiment, step S121 specifically includes the following steps: IMU Data Preprocessing: High-frequency IMU data is acquired, zero-bias initialization and gravity vector estimation are completed, and an IMU pre-integration model, i.e., an inter-frame motion model, is constructed to provide high-frequency constraints for motion prediction. For example, the self-moving device remains absolutely still after startup, and high-frequency IMU data is acquired within a preset duration (e.g., 5 seconds). The initial zero bias of the gyroscope and accelerometer is calculated using the IMU data during the still period: the gyroscope zero bias is the average value of the gyroscope's three-axis angular velocity measurements during the still period; the average value of the accelerometer's three-axis acceleration measurements during the still period is calculated, and the accelerometer zero bias is this average value minus the gravitational acceleration (9.8 m / s²). Gravity Vector Estimation: The average value of the accelerometer's three-axis measurements during the still period is directly used as the initial estimate of the gravity vector in the current device coordinate system. Based on the IMU data, an IMU pre-integration model between adjacent keyframes is constructed. The specific method for constructing the IMU pre-integration model is existing technology and will not be elaborated here.

[0041] 3D point cloud data preprocessing: The raw 3D point cloud data is denoised and downsampled, and motion distortion correction is performed using IMU pre-integration results to eliminate point cloud distortion caused by carrier motion during laser scanning. For example, the denoising and downsampling of the raw 3D point cloud data includes: First, filtering adjacent points according to a filtering distance threshold to remove outliers, ensuring the sparsity and uniformity of the point cloud and obtaining a stable number of points. Second, voxel division is performed on the filtered point cloud, and the points within each voxel are clustered. Then, the clustering results are denoised: classes with fewer than a preset threshold of points are removed (considered noise); if multiple classes exist within a voxel, it indicates that the region has rich features but is not stable enough for localization, and these are removed; finally, large classes with sufficient points and continuous features are retained. Finally, the classes are sorted in descending order of point count, with higher priority for classes with more points. Within each class, points are cyclically sampled from the center outwards until a fixed number of points are obtained. After the above processing, a representative point cloud with environmental characteristics can be obtained, providing strong data support for subsequent localization. Motion distortion correction is performed by combining IMU pre-integration results, including: using IMU pre-integration results to compensate points at different times in a laser scan frame to the same time (such as the end time of the frame); for each point in a frame, based on the timestamp of the point, the pose change of the point from the measurement time to the end time of the frame is calculated using IMU pre-integration results, and the point is compensated to a unified time to obtain motion distortion corrected 3D point cloud data.

[0042] Image data preprocessing: Construct an image pyramid to achieve coarse-to-fine alignment, remove overexposed / underexposed invalid areas, and obtain preprocessed image data.

[0043] Multi-sensor data spatiotemporal alignment: Time-align 3D point cloud data, image data, and motion state data according to timestamps; Spatially align 3D point cloud data, image data, and motion state data according to extrinsic parameters between non-scanning single-frame pulse imaging sensors, vision sensors, and IMUs, as well as camera intrinsic parameters, and project them to the same coordinate system (e.g., IMU coordinate system); obtain spatiotemporally synchronized multi-source data.

[0044] It should be noted that the extrinsic parameters between the non-scanning single-frame pulse imaging sensor 101, the vision sensor 102, and the IMU refer to the spatial transformation relationship between the coordinate systems of the three sensors. Specifically, this includes the rotation matrix and translation vector between the coordinate systems of the three sensors. Based on this extrinsic parameter calibration relationship, the data from the non-scanning single-frame pulse imaging sensor 101, the vision sensor 102, and the IMU can be transformed to the same coordinate system, thereby achieving spatial alignment of multi-sensor data. The camera intrinsic parameters refer to the inherent parameters of the camera itself, used to characterize the mapping relationship between three-dimensional spatial points and two-dimensional image pixels. They can convert the two-dimensional image pixel coordinates acquired by the camera into spatial coordinates in the three-dimensional camera coordinate system, or project three-dimensional spatial points onto the two-dimensional image plane. Specifically, the camera intrinsic parameters include focal length, principal point coordinates, and distortion coefficients. The camera intrinsic parameters are fixed values ​​pre-calibrated and stored at the factory, remaining constant during the use of the vision sensor, and do not require recalibration before each power-on or operation.

[0045] In step S122, the current state of the self-moving device is predicted based on the motion state data, and a prior state estimate is output. This includes: constructing an inter-frame motion model based on the motion state data; using the posterior state estimate obtained from the previous frame as the initial prediction value, predicting the current state of the self-moving device based on the inter-frame motion model, and outputting a prior state estimate; wherein, the prior state estimate includes the prior pose and its covariance matrix.

[0046] In one specific embodiment, step S122 specifically includes the following steps: using the posterior state estimate obtained from the previous frame as the initial prediction value, predicting the current state of the self-moving device using the IMU pre-integration result, and outputting the prior state estimate, which includes: prior pose (rotation + translation), velocity, IMU zero bias (gyroscope zero bias, acceleration zero bias) and its covariance matrix; using state propagation on the manifold, the prior state estimate is used as the prior distribution of the iterative error state filtering model (such as the iterative error state Kalman filter model, also known as IESKF), providing initial values ​​for subsequent tightly coupled observation updates. The specific implementation method of predicting the current state of the self-moving device using the IMU pre-integration result and outputting the prior state estimate is existing technology and will not be described in detail here.

[0047] The global map includes a global point cloud map and a global visual map. In step S123, based on the multi-source data and the global map, the geometric constraint residuals of the 3D point cloud data and the photometric constraint residuals of the image data are constructed. This includes: projecting the preprocessed 3D point cloud data onto the global point cloud map using a laser-inertial odometry (LIO) subsystem, retrieving neighboring points to fit a local plane, calculating the distance residuals from the points to the local plane, and constructing the Jacobian matrix of the distance residuals to obtain the geometric constraint residuals; and using a visual-inertial odometry (VIO) subsystem, calculating the photometric error residuals between the current image frame and the reference image frame based on the preprocessed image data and the global visual map, and constructing the Jacobian matrix of the photometric error residuals to obtain the photometric constraint residuals.

[0048] In some embodiments, the iterative error state filtering model is an iterative error state Kalman filter model (also known as IESKF). In step S123, the iterative error state filtering model is used to jointly update the prior state estimate based on the geometric constraint residual and the photometric constraint residual to obtain the posterior state estimate. This includes: substituting the geometric constraint residual and the photometric constraint residual into the iterative error state Kalman filter model to jointly update the prior state estimate and obtain the posterior state estimate; wherein, the posterior state estimate includes the posterior pose of the self-moving device and its covariance matrix. This achieves complementary geometric and texture constraints, improving the accuracy and robustness of state estimation. The iterative error state Kalman filter model (also known as IESKF) is existing technology and will not be described further here.

[0049] In one specific embodiment, step S123 specifically includes the following steps: A direct observation model using frame-to-map (FT-MAP) is employed, alternately or jointly updating the LIO and VIO subsystems. An IESKF maximum a posteriori probability optimization objective function is constructed, and the optimal state is iteratively solved. Specifically: The LIO subsystem update includes: projecting the preprocessed 3D point cloud data onto the global point cloud map, retrieving neighboring points to fit the local plane; calculating the distance residuals from the points to the local plane, constructing the Jacobian matrix of the distance residuals, and obtaining the geometric constraint residuals; The VIO subsystem update includes: extracting map points within the current field of view of the visual sensor from the global visual map, removing map points with inconsistent depth or occlusion through depth consistency verification and occlusion detection; using the sparse direct method, taking the image patch to which the map point is attached as a reference, calculating the photometric error residual between the current image frame and the reference image frame, and constructing the Jacobian matrix of the photometric error residual to obtain the photometric constraint residual.

[0050] When data from both sensors arrive, the geometric constraint residuals of the LIO subsystem and the photometric constraint residuals of the VIO subsystem are jointly substituted into the iterative error state Kalman filter model to jointly update the prior state estimate until the update converges, thus obtaining the posterior state estimate. The posterior state estimate includes the posterior pose of the self-moving device and its covariance matrix. The posterior state estimate is the pose of the self-moving device obtained through calculation.

[0051] In some embodiments, the state detection unit further includes a wheel odometer (also known as an ODOM) 106; the motion state data also includes displacement and yaw angle. For example, displacement and yaw angle in the x and y directions of the self-moving device in the machine coordinate system. Exemplarily, the wheel odometer is an incremental photoelectric encoder mounted on the left and right drive wheels of the self-moving device. The encoder calculates the angle through which the wheel rotates by detecting the pulse signals generated when the wheel rotates. Combined with the known wheel diameter and transmission ratio, the displacement of the self-moving device can be calculated. The yaw angle of the self-moving device can also be calculated by the speed difference between the left and right wheels. The sampling frequency of the wheel odometer is typically around 50Hz.

[0052] It is understandable that when environmental perception is completely degraded (e.g., lack of texture or geometric features) causing both LiDAR and vision to fail, wheel odometry can still provide reliable motion information, maintain short-term positioning, overcome degradation, significantly improve the robustness of the system in extreme scenarios, and further enhance positioning reliability and accuracy.

[0053] In some embodiments, the control method for the self-moving device further includes: calculating wheel speed residuals based on motion state data collected by a wheel odometer, constructing a Jacobian matrix of the wheel speed residuals to obtain wheel speed constraint residuals; and jointly substituting the geometric constraint residuals, photometric constraint residuals, and wheel speed constraint residuals into an iterative error state Kalman filter model to jointly update the prior state estimate and obtain the posterior state estimate. It can be understood that in tightly coupled observation updates, adding a wheel speed constraint residual term can suppress IMU drift and improve positioning stability and accuracy under degraded environments.

[0054] In other embodiments, the wheeled odometer is also used for state prediction, fusing with IMU data to obtain a priori state estimates. Specifically, it can be fused with IMU data through weighted averaging or loosely coupled correction to jointly calculate the priori state estimate. Furthermore, the wheeled odometer can serve as a backup for the IMU, independently performing motion state prediction when IMU data fails, thereby improving the robustness of the system and the reliability of state prediction.

[0055] The control method for self-moving devices also includes: updating the global point cloud map based on 3D point cloud data; and updating the global visual map based on image data and 3D point cloud data.

[0056] In one specific embodiment, in response to the convergence of the tightly coupled observation update in step S123, the bimodal map is incrementally updated to maintain the matching accuracy of subsequent observations. The bimodal map includes a global point cloud map and a global visual map. For the global point cloud map: the point cloud is stored using an ikd-Tree data structure, which supports dynamic downsampling and balanced tree indexing, enabling fast nearest neighbor retrieval; valid point clouds in the current frame are incrementally inserted into the map. For the global visual map: based on the 3D points in the global point cloud map, each point is associated with a corresponding image patch and observation pose, and a hash voxel index is constructed to accelerate retrieval; simultaneously, high-quality visual observation points are added, and invalid or duplicate points are removed to maintain map compactness.

[0057] In some embodiments, refer to Figure 2 The self-moving device 10 includes a sensor module 100, which includes a mounting housing 103, a non-scanning single-frame pulse imaging sensor 101, and a vision sensor 102. The non-scanning single-frame pulse imaging sensor 101 and the vision sensor 102 are both mounted on the mounting housing 103, and there is a preset external parameter calibration relationship between the non-scanning single-frame pulse imaging sensor 101 and the vision sensor 102. The sensor module 100 is configured to be mounted on the self-moving device 10 as a whole. In this embodiment, the non-scanning single-frame pulse imaging sensor 101 and the vision sensor 102 are integrated into the same sensor module, instead of being installed independently in different locations of the self-moving device. This results in better time synchronization, and the integrated design enables a rigid connection between the two sensors. The non-scanning single-frame pulse imaging sensor 101 and the vision sensor 102 have pre-calibrated extrinsic parameters that remain constant during use, eliminating the need for online estimation or frequent recalibration. This reduces the uncertainty of parameters in state estimation, lowers the complexity of the fusion algorithm, and improves positioning accuracy and system reliability.

[0058] It should be noted that both the non-scanning single-frame pulse imaging sensor 101 and the vision sensor 102 are mounted on the mounting housing 103. They can be fixedly mounted inside the same mounting housing 103, or they can be fixedly mounted on the same mounting housing 103. As long as the relative position between the two can be fixed, ensuring that the sensor module 100 can be mounted as a whole on the self-moving device 10, it is acceptable. The extrinsic parameter calibration relationship between the non-scanning single-frame pulse imaging sensor 101 and the vision sensor 102 refers to the spatial transformation relationship between the coordinate systems of the two sensors. Specifically, the extrinsic parameter calibration relationship includes the rotation matrix and translation vector between the coordinate systems of the two sensors. Based on this extrinsic parameter calibration relationship, the 3D point cloud data of the non-scanning single-frame pulse imaging sensor 101 can be transformed into the coordinate system of the vision sensor 102, or the image data of the vision sensor 102 can be transformed into the coordinate system of the non-scanning single-frame pulse imaging sensor 101, thereby achieving spatial alignment of the data from the two sensors. The preset extrinsic parameter calibration relationship between the non-scanning single-frame pulse imaging sensor 101 and the vision sensor 102 should be understood as follows: the relative poses of the non-scanning single-frame pulse imaging sensor 101 and the vision sensor 102 are fixed, and there is a pre-calibrated spatial transformation relationship between them. Therefore, in actual operation, the pre-calibrated extrinsic parameters can be directly called for data fusion, without the need to recalibrate the extrinsic parameters before each power-on or operation. A non-scanning single-frame pulse imaging sensor and a vision sensor are positioned facing the travel direction of the self-moving device to acquire environmental data in front of the self-moving device. Specifically, they can be mounted in front of or on top of the self-moving device. (See reference...) Figure 2 The non-scanning single-frame pulse imaging sensor 101 includes a laser emitter array and a laser receiver array that works in conjunction with the laser emitter array to acquire three-dimensional point cloud data. The self-moving device includes a body 104, with the laser emitter array and laser receiver array fixedly disposed relative to the body 104. It can be understood that, compared to scanning sensors, the non-scanning single-frame pulse imaging sensor, fixedly disposed on the body 104, has higher mechanical strength and a longer lifespan.

[0059] The non-scanning single-frame pulse imaging sensor 101 has a horizontal field of view of 90°~170° and a vertical field of view of 30°~90°. Exemplarily, the non-scanning single-frame pulse imaging sensor 101 is a solid-state area array lidar, with a horizontal field of view of 90°, 120°, 135°, 150°, 170°, etc., and a vertical field of view of 30°, 50°, 80°, 90°, etc. As a specific example, the solid-state area array lidar has a horizontal field of view of 120° and a vertical field of view of 80°. As another specific example, the solid-state area array lidar has a horizontal field of view of 170° and a vertical field of view of 80°. As yet another specific example, the solid-state area array lidar has a horizontal field of view of 120° and a vertical field of view of 50°. It is understood that the aforementioned field of view is determined by the characteristics of the sensor. The non-scanning single-frame pulse imaging sensor is configured with preferred horizontal and vertical field of view to balance detection range and high detection accuracy. The ranging range of the non-scanning single-frame pulse imaging sensor 101 is 0.2m to 100m. For example, the ranging range can be 0.2m to 50m, 0.2m to 60m, 0.2m to 70m, 0.2m to 80m, 0.2m to 90m, 0.2m to 100m, etc. In this embodiment, the field of view of the non-scanning single-frame pulse imaging sensor 101 is 170° (horizontal) × 80° (vertical), the frame rate is 10Hz, and the ranging range is 0.2m to 70m.

[0060] The visual sensor 102 is a monocular camera. While monocular cameras lack depth perception capabilities, when fused with a LiDAR system, the ranging capability of the LiDAR can compensate for this deficiency, enabling reliable positioning. Compared to binocular cameras, this configuration significantly reduces hardware costs while meeting performance requirements.

[0061] To address the aforementioned problems, this application also provides a control method for a self-moving device. (Refer to...) Figure 2 The self-moving device includes a sensor module, which comprises: a mounting housing; a LiDAR sensor configured to acquire 3D point cloud data of the self-moving device's surrounding environment; and a vision sensor configured to acquire image data of the self-moving device's surrounding environment. Both the LiDAR sensor and the vision sensor are mounted on the mounting housing, and a preset extrinsic parameter calibration relationship exists between them. The sensor module is configured to be mounted as a single unit on the self-moving device. (Refer to...) Figure 5 The control methods for self-moving devices include: S21. Acquire 3D point cloud data and image data; S22. Time-align the 3D point cloud data and image data according to the timestamp; spatially align the 3D point cloud data and image data according to the preset extrinsic parameter calibration relationship, and project them onto the same coordinate system; obtain spatiotemporally synchronized multi-source data; S23. Determine the pose of the self-moving device based on multi-source data.

[0062] It is understood that the non-scanning single-frame pulse imaging sensor 101 and the vision sensor 102 are integrated into the same sensor module, rather than being installed independently in different locations of the self-moving device. This results in better time synchronization, and the integrated design enables a rigid connection between the two. The non-scanning single-frame pulse imaging sensor 101 and the vision sensor 102 have pre-calibrated extrinsic parameters that remain constant during use, eliminating the need for online estimation or frequent recalibration. This reduces the uncertainty of parameters in state estimation, lowers the complexity of the fusion algorithm, and improves positioning accuracy and system reliability.

[0063] In some embodiments, the lidar sensor is a non-scanning single-frame pulse imaging sensor, specifically a solid-state area array lidar sensor. The vision sensor is a monocular camera. In other embodiments, the lidar sensor can also be an optical phased array lidar, and the vision sensor can also be a binocular camera, a multi-view camera, an RGB-D depth camera, etc.

[0064] The control methods for the self-moving device described in the foregoing embodiments can be freely combined and applied to this embodiment as long as there is no conflict or contradiction. For example, step S23 in this embodiment can be specifically implemented based on step S12 in the foregoing embodiments, and will not be described again here.

[0065] Reference Figure 6 and Figure 7To address the aforementioned problems, this application also provides a self-moving device 10, comprising: a moving module 110 for driving the self-moving device 10 to move; a working module 120 for performing working tasks; a control module 130 for controlling the operation of the moving module 110 and the working module 120; a battery module 140 for providing power to the moving module 110, the working module 120, and the control module 130; and a sensor module 150. The sensor module 150 includes a non-scanning single-frame pulse imaging sensor 101, a vision sensor 102, an inertial measurement unit 105, and a wheeled odometer 106. The control module 130 includes a memory and a processor. The memory stores a computer program executable by the processor, and the processor executes the computer program to implement the aforementioned control method for the self-moving device. The sensor module 100 is communicatively connected to the control module 130. Taking an automatic lawnmower as an example, the moving module 110 is the walking component of the automatic lawnmower, including walking wheels and a self-driving motor for driving the walking wheels to move. The working module 120 is a mowing assembly, including a cutting blade and a mowing motor for driving the cutting blade to rotate. The control module 130 connects to and controls the self-drive motor and the mowing motor.

[0066] Reference Figure 8 To address the aforementioned issues, this application also provides a self-moving device, including an electronic device 300. The electronic device 300 includes a memory 302 and a processor 301. The memory 302 stores a computer program executable by the processor 301. When the processor 301 executes the computer program, it implements the aforementioned control method for the self-moving device.

[0067] To address the aforementioned problems, this application also provides a computer-readable storage medium storing a processor-executable computer program configured to cause the processor to implement the aforementioned self-moving device control method when executing the computer program.

[0068] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0069] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

Claims

1. A control method for a self-moving device, characterized in that, The self-moving device includes: A non-scanning single-frame pulse imaging sensor is configured to acquire three-dimensional point cloud data of the surrounding environment of the self-moving device; and A visual sensor is configured to acquire image data of the surrounding environment of the self-moving device; The control method for the self-moving device includes: Acquire the 3D point cloud data and the image data; The pose of the self-moving device is determined based at least on the three-dimensional point cloud data and the image data.

2. The control method for a self-moving device according to claim 1, characterized in that, Determining the pose of the self-moving device based at least on the 3D point cloud data and the image data includes: The pose of the self-moving device is solved by joint optimization using a tightly coupled state estimation model, based at least on the three-dimensional point cloud data and the image data.

3. The control method for a self-moving device according to claim 1, characterized in that, The self-moving device further includes a state detection unit, which is configured to detect motion state data of the self-moving device; Determining the pose of the self-moving device based at least on the 3D point cloud data and the image data includes: The pose of the self-moving device is determined based on the 3D point cloud data, the image data, and the motion state data.

4. The control method for a self-moving device according to claim 3, characterized in that, The state detection unit includes an inertial measurement unit; the motion state data includes acceleration and angular velocity; And / or, the state detection unit includes a wheeled odometer; the motion state data includes displacement and yaw angle.

5. The control method for a self-moving device according to claim 3, characterized in that, Determining the pose of the self-moving device based on the 3D point cloud data, the image data, and the motion state data includes the following steps: Data preprocessing: Spatiotemporal alignment and motion distortion correction are performed on the 3D point cloud data, the image data, and the motion state data to obtain preprocessed multi-source data; State prediction: Based on the motion state data, predict the current state of the self-moving device and output a priori state estimate; and Tightly Coupled Observation Update: Based on the multi-source data and the global map, determine the geometric constraint residual of the 3D point cloud data and the photometric constraint residual of the image data. Based on the geometric constraint residual and the photometric constraint residual, use an iterative error state filtering model to jointly update the prior state estimate to obtain the posterior state estimate, so as to determine the pose of the self-moving device.

6. The control method for a self-moving device according to claim 5, characterized in that, The process of performing spatiotemporal alignment and motion distortion correction on the 3D point cloud data, the image data, and the motion state data to obtain preprocessed multi-source data includes: Based on the motion state data, an inter-frame motion model is constructed, and based on the inter-frame motion model, motion distortion correction is performed on the three-dimensional point cloud data; And / or, time-align the 3D point cloud data, the image data, and the motion state data according to the timestamp; spatially align the 3D point cloud data, the image data, and the motion state data according to the spatial synchronization parameters, and project them onto the same coordinate system; thus obtaining spatiotemporally synchronized multi-source data.

7. The control method for a self-moving device according to claim 5, characterized in that, The step of predicting the current state of the self-moving device based on the motion state data and outputting a priori state estimate includes: Based on the motion state data, an inter-frame motion model is constructed; The posterior state estimate obtained from the previous frame is used as the initial prediction value. The current state of the self-moving device is predicted based on the inter-frame motion model, and the prior state estimate is output. The prior state estimation includes the prior pose and its covariance matrix.

8. The control method for a self-moving device according to claim 5, characterized in that, The global map includes a global point cloud map and a global visual map; The step of constructing the geometric constraint residual of the 3D point cloud data and the photometric constraint residual of the image data based on the multi-source data and the global map includes: The preprocessed 3D point cloud data is projected onto the global point cloud map, neighboring points are retrieved to fit the local plane, the distance residual from the point to the local plane is calculated, and the Jacobian matrix of the distance residual is constructed to obtain the geometric constraint residual. Based on the preprocessed image data and the global visual map, the photometric error residual between the current image frame and the reference image frame is calculated, and the Jacobian matrix of the photometric error residual is constructed to obtain the photometric constraint residual.

9. The control method for a self-moving device according to claim 5, characterized in that, The iterative error state filtering model is an iterative error state Kalman filter model; Based on the geometric constraint residuals and the photometric constraint residuals, an iterative error state filtering model is used to jointly update the prior state estimate to obtain the posterior state estimate, including: Substitute the geometric constraint residual and the photometric constraint residual into the iterative error state Kalman filter model to jointly update the prior state estimate and obtain the posterior state estimate. The posterior state estimation includes the posterior pose of the self-moving device and its covariance matrix.

10. The control method for a self-moving device according to claim 5, characterized in that, The global map includes a global point cloud map and a global visual map; The control method for the self-moving device further includes: The global point cloud map is updated based on the three-dimensional point cloud data; The global visual map is updated based on the image data and the 3D point cloud data.

11. The control method for a self-moving device according to claim 1, characterized in that, The self-moving device includes a sensor module, which includes a mounting housing, the non-scanning single-frame pulse imaging sensor, and the vision sensor. Both the non-scanning single-frame pulse imaging sensor and the vision sensor are mounted on the mounting housing, and there is a preset external parameter calibration relationship between the non-scanning single-frame pulse imaging sensor and the vision sensor; The sensor module is configured to be installed as a whole on the self-moving device.

12. The control method for a self-moving device according to claim 1, characterized in that, The visual sensor is a monocular camera.

13. The control method for a self-moving device according to claim 1, characterized in that, The self-moving device is an automatic lawnmower; the automatic lawnmower includes: body; A walking assembly, mounted on the machine body, includes walking wheels and a self-drive motor for driving the walking wheels to move; A mowing assembly, mounted on the machine body, includes a cutting blade and a mowing motor for driving the cutting blade to rotate; and The control module is configured to connect to and control the self-driven motor and the lawnmower motor.

14. A control method for a self-moving device, characterized in that, The self-moving device includes a sensor module, which comprises: Mounting housing; A lidar sensor is configured to acquire three-dimensional point cloud data of the surrounding environment of the self-moving device; and A visual sensor is configured to acquire image data of the surrounding environment of the self-moving device; Both the lidar sensor and the vision sensor are mounted on the mounting housing, and there is a preset external parameter calibration relationship between the lidar sensor and the vision sensor; the sensor module is configured to be mounted as a whole on the self-moving device; The control method for the self-moving device includes: Acquire the 3D point cloud data and the image data; The 3D point cloud data and the image data are time-aligned according to the timestamp; the 3D point cloud data and the image data are spatially aligned according to the preset extrinsic parameter calibration relationship and projected onto the same coordinate system; thus obtaining spatiotemporally synchronized multi-source data. Based on the multi-source data, the pose of the self-moving device is determined.

15. A self-moving device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the control method for the self-moving device according to any one of claims 1 to 14.

Images