Multi-sensor fusion intelligent mowing robot and system

Abstract

The application provides a multi-sensor fusion intelligent mowing robot and system, which has the characteristics of including a driving part, a data acquisition part, an image preprocessing part for preprocessing a visible light image to obtain a preprocessed image, an image processing part for outputting the preprocessed image to a GrassBEV network model to obtain lawn segmentation information, a robot relative position coordinate, and a category and a position of a predicted object, a global position estimation part for obtaining a robot global position estimation according to acquisition information and the robot relative position coordinate, and a path planning and instruction generation part for inputting the robot global position estimation, the lawn segmentation information, and the position of the object with the category of an obstacle to a GrassRoutePlan network model to obtain a predicted trajectory of a mowing task and a corresponding control command. In summary, the method can efficiently, stably, and robustly complete the mowing task at a low cost.

Description

Technical Field

[0001] This invention belongs to the field of intelligent garden robots, specifically relating to an intelligent lawn mowing robot and system with multi-sensor fusion. Background Technology

[0002] Intelligent lawnmower robots are a crucial component of smart garden systems, primarily used to automate lawn mowing, replacing manual labor. In recent years, visual recognition algorithms using cameras have been developed to determine lawn boundaries and obstacles. However, algorithms based solely on computer graphics have fixed feature requirements for lawn and obstacle images. Each user's lawn terrain, color, condition, and even environmental factors like weather and lighting vary greatly, meaning the images captured by the camera are likely different from the original test dataset. Lawnmower robots also need to handle complex scenes and dynamic obstacles, leading to low recognition accuracy and a high likelihood of misjudgments. Robots relying solely on GPS or RTK positioning cannot overcome interference from surrounding buildings, trees, and other obstructions, causing them to sometimes go out of bounds or even fall into rivers.

[0003] In addition to traditional methods, lawnmower robots rely on automated control to operate. Traditional control methods involve collecting data from sensors, inputting it into a conventional feedback system, and performing simple logical checks. This approach cannot handle most of the complex situations the robot encounters. Robot control requires handling various complex situations, such as avoiding potholes, navigating uneven grass in a straight line, and dealing with slipping, getting stuck, narrow passages, isolated areas, and so on. Therefore, lawnmower robots urgently need higher recognition accuracy and more intelligent control methods to enhance their generalization ability and robustness, enabling them to adapt to complex lawn environments.

[0004] Therefore, there are still several pain points and problems that need to be solved in the field of lawn mowing robots. Summary of the Invention

[0005] This invention is made to solve the above-mentioned problems, and aims to provide an intelligent lawn mowing robot and system with multi-sensor fusion.

[0006] This invention provides a multi-sensor fusion intelligent lawnmower robot, characterized by: a drive unit for moving the intelligent lawnmower robot; a data acquisition unit for acquiring visible light images, GPS location information, inertial measurement angular velocity information, inertial measurement acceleration information, and motor encoder speed information of the scene in which the intelligent lawnmower robot is located; an image preprocessing unit for preprocessing the visible light images to obtain preprocessed images; and an image processing unit, including a GrassBEV network model, for outputting the preprocessed images to the GrassBEV network model to obtain lawn segmentation information, robot relative position coordinates, and predicted object categories. The system comprises: a global position estimation unit, which estimates the robot's global position based on GPS location information, inertial measurement angular velocity information, inertial measurement acceleration information, motor encoder speed information, and the robot's relative position coordinates; a path planning and command generation unit, including a GrassRoutePlan network model, which inputs the robot's global position estimate, lawn segmentation information, and the positions of objects classified as obstacles into the GrassRoutePlan network model to obtain the predicted trajectory of the lawn mowing task and the corresponding control commands; and a control unit, which controls the drive unit according to the control commands, enabling the intelligent lawn mowing robot to mow the lawn according to the predicted trajectory.

[0007] The intelligent lawnmower robot with multi-sensor fusion provided by this invention may also have the following features: the preprocessing includes image denoising, backlight processing, image segmentation, image enhancement, and standard normalization. Image denoising removes noise from the image through a pre-trained deep learning model. Backlight processing reduces the impact of backlight on the image by using high dynamic range technology and backlight compensation technology. Image segmentation extracts the regions of interest in the image through threshold segmentation, region segmentation, and gradient segmentation. Image enhancement strengthens the details and features of the image through local contrast enhancement and adaptive enhancement. Standard normalization standardizes each pixel of the image.

[0008] The multi-sensor fusion intelligent lawnmower robot provided by this invention may also have the following features: The GrassBEV network model includes: a feature extraction module for extracting features from a preprocessed image to obtain multi-scale features; a feature enhancement module, including a feature pyramid network, for processing the multi-scale features to obtain multiple image feature maps; a first position encoding module for adding a corresponding position encoding to each image feature map to obtain a corresponding first encoded feature map; a bird's-eye view transformation module, storing geometric transformation data, for aggregating feature points from each first encoded feature map to a bird's-eye view based on the geometric transformation data to obtain a corresponding bird's-eye view feature map; a second position encoding module for adding a corresponding position encoding to each bird's-eye view feature map to obtain a corresponding second encoded feature map; and a multi-head self-attention module for performing multi-head self-attention processing on each second encoded feature map to obtain a corresponding second encoded feature map. The system comprises the following modules: a first normalization module, which adds each bird's-eye view feature to its corresponding first encoded feature map and normalizes it to obtain a first normalized feature; a multi-head cross-attention module, which performs multi-head cross-attention processing on each first normalized feature in conjunction with multi-scale features to obtain a corresponding fused feature; a second normalization module, which adds each fused feature to the query of its corresponding normalized feature and normalizes it to obtain a corresponding second normalized feature; a feature optimization module, including a feedforward neural network, which inputs each second normalized feature into the feedforward neural network and adds the output of the feedforward neural network to the second normalized feature to obtain a corresponding fused bird's-eye view feature map; and a detection and segmentation module, which generates lawn segmentation information, robot relative position coordinates, and predicted object category and location based on all fused bird's-eye view feature maps.

[0009] The intelligent lawnmower robot with multi-sensor fusion provided by this invention may also have the following features: the drive unit includes moving wheels, and the data acquisition unit includes: a GPS sensor for acquiring GPS location information; an inertial measurement unit for acquiring inertial measurement angular velocity information and inertial measurement acceleration information; and a motor encoding speed unit including a motor encoder for obtaining the robot's linear velocity and angular velocity as motor encoding speed information. The inertial measurement angular velocity information includes angular velocity values ​​on the X, Y, and Z axes, and the inertial measurement acceleration information includes acceleration values ​​on the X, Y, and Z axes.

[0010] The intelligent lawnmower robot with multi-sensor fusion provided by this invention may also have the following features: the process of generating the robot's global position estimate by the global position estimation unit includes the following steps: Step S1, constructing an initial state estimate and an initial covariance matrix; Step S2, calculating the predicted state based on the initial state estimate and calculating the predicted covariance based on the covariance matrix; Step S3, calculating the Kalman gain based on the predicted covariance; Step S4, calculating the system state based on GPS position information, inertial measurement angular velocity information, inertial measurement acceleration information, motor encoder speed information, and the robot's relative position coordinates; Step S5, obtaining an updated predicted state based on the Kalman gain, the predicted state, and the system state; Step S6, obtaining an updated predicted covariance based on the Kalman gain and the predicted covariance; Step S7, repeating steps S5 to S6 until an updated predicted state is obtained as the robot's global position estimate.

[0011] The multi-sensor fusion intelligent lawnmower robot provided by this invention may also have the following features: the GrassRoutePlan network model includes: a multi-scale extraction module for extracting multi-scale features from the robot's global position estimation, lawn segmentation information, and the positions of objects classified as obstacles, to obtain multiple image feature maps; an encoding module for adding position encoding to each image feature map to obtain corresponding encoded feature maps; a first attention module for performing multi-head self-attention processing on each encoded feature map to obtain corresponding first feature maps; a second attention module for performing multi-head cross-attention processing on each first feature map, combining the robot's global position estimation, lawn segmentation information, and the positions of objects classified as obstacles, to obtain corresponding second feature maps; a normalization module for performing layer normalization on each second feature map to obtain corresponding normalized features; and a prediction module for obtaining predicted trajectories and control commands based on all normalized features.

[0012] The present invention also provides a multi-sensor fusion intelligent lawn mowing robot system, characterized by including: an intelligent lawn mowing robot and a charging station, wherein the intelligent lawn mowing robot is any of the above-mentioned multi-sensor fusion intelligent lawn mowing robots, and the intelligent lawn mowing robot is charged through the charging station.

[0013] The multi-sensor fusion intelligent lawnmower system provided by this invention may also have the following features: the intelligent lawnmower further includes: a power detection unit, used to detect the remaining power of the intelligent lawnmower, and generate a feedback signal when the remaining power is less than the power threshold; the control unit, based on the feedback signal and combined with the position of an object classified as a charging pile, controls the path planning and command generation unit to generate a predicted trajectory and control command leading to the charging pile, and controls the intelligent lawnmower to move towards the charging pile for charging through the control command.

[0014] The multi-sensor fusion intelligent lawn mowing robot system provided by this invention may also have the following features: the charging pile is equipped with vector graphic markers, which are located on the same side of the charging device of the charging pile. The intelligent lawn mowing robot also includes a charging calibration unit, which generates a charging docking path trajectory and corresponding docking control commands based on the images of the vector graphic markers collected by the data acquisition unit. The control unit controls the drive unit to move through the docking control commands, so that the intelligent lawn mowing robot docks with the charging device.

[0015] The role and effect of invention

[0016] According to the multi-sensor fusion intelligent lawnmower robot and system of the present invention, firstly, the image processing unit extracts bird's-eye view features from the preprocessed image and reconstructs environmental and bird's-eye view features to generate accurate lawn segmentation information, robot relative position coordinates, and predicted object categories and positions. Secondly, the global position estimation unit calibrates the position of the intelligent lawnmower robot based on sensor data collected by the data acquisition unit and robot relative position coordinates generated by the image processing unit, greatly reducing reliance on traditional high-precision positioning sensors and reducing the cost of the intelligent lawnmower robot by one-third. Thirdly, the path planning and command generation unit predicts the path for mowing while avoiding obstacles and generates control commands based on the robot's global position estimate generated by the global position estimation unit, the lawn segmentation information generated by the image processing unit, and obstacle positions, thereby simplifying the complexity of manually calculating path trajectories and improving the robustness of path planning operations. Therefore, the multi-sensor fusion intelligent lawnmower robot and system of the present invention can complete lawnmowing tasks efficiently, stably, cost-effectively, and robustly. Attached Figure Description

[0017] Figure 1 This is a block diagram of the intelligent lawnmower robot system in an embodiment of the present invention;

[0018] Figure 2 This is a block diagram of an intelligent lawnmower robot in an embodiment of the present invention;

[0019] Figure 3This is a block diagram of the GrassBEV network model in an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of the process by which the global position estimation unit generates the robot's global position estimate in an embodiment of the present invention;

[0021] Figure 5 This is a block diagram of the GrassRoutePlan network model in an embodiment of the present invention;

[0022] Figure 6 This is a schematic diagram of vector graphic markers in an embodiment of the present invention. Detailed Implementation

[0023] To make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, provide a detailed description of the intelligent lawn mowing robot and system based on multi-sensor fusion of the present invention.

[0024] This embodiment provides an intelligent lawn mowing robot system, which avoids the positioning and control errors caused by buildings and surrounding environment in traditional positioning systems in intelligent lawn mowing environment. It is an efficient, stable, low-cost and robust system.

[0025] Figure 1 This is a block diagram of an intelligent lawnmower robot system in an embodiment of the present invention.

[0026] like Figure 1 As shown, the intelligent lawnmower system 100 includes an intelligent lawnmower 10 and a charging station 20. The intelligent lawnmower 10 is charged via the charging station 20.

[0027] Figure 2 This is a block diagram of an intelligent lawnmower robot in an embodiment of the present invention.

[0028] like Figure 2 As shown, the intelligent lawnmower robot 10 includes: a drive unit 11, a data acquisition unit 12, an image preprocessing unit 13, an image processing unit 14, a global position estimation unit 15, a path planning and instruction generation unit 16, a power detection unit 17, a charging calibration unit 18, and a control unit 19 that controls the above units.

[0029] The drive unit 11 includes wheels for moving the intelligent lawnmower robot 10. In this embodiment, the wheels can be replaced with any moving device that can move the intelligent lawnmower robot 10.

[0030] The data acquisition unit 12 is used to acquire visible light images, GPS location information, inertial measurement angular velocity information, inertial measurement acceleration information and motor coding speed information of the scene where the intelligent lawn mowing robot 10 is located. It includes a GPS sensor 121, an inertial measurement unit 122 and a motor coding speed unit 123.

[0031] GPS sensor 121 is used to collect GPS location information. In this embodiment, GPS sensor 121 parses the information according to the NMEA protocol to obtain latitude and longitude coordinates, altitude, speed, etc., and its message format is: "$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47".

[0032] The inertial measurement unit 122 is used to collect inertial measurement angular velocity information and inertial measurement acceleration information.

[0033] The inertial measurement angular velocity information includes angular velocity values ​​on the X, Y, and Z axes, and the inertial measurement acceleration information includes acceleration values ​​on the X, Y, and Z axes.

[0034] The motor encoding speed unit 123 includes a motor encoder, which is used to obtain the robot's linear velocity and angular velocity as motor encoding speed information. The specific calculation formula is as follows:

[0035]

[0036]

[0037] In the formula R L and R R V represents the radius of the left and right moving wheels, respectively. L and V R These are the differential velocities of the left and right moving wheels, respectively; V is the robot's linear velocity; and R is the robot's angular velocity.

[0038] The image preprocessing unit 13 is used to preprocess the visible light image to obtain a preprocessed image.

[0039] The preprocessing includes image denoising, backlight processing, image segmentation, image enhancement, and standard normalization.

[0040] Image denoising removes noise from images using a pre-trained deep learning model. In this embodiment, the deep learning model is DnCNN, which replaces traditional linear and nonlinear filtering. By leveraging the powerful representation learning capabilities of deep convolutional neural networks, noise signals are suppressed or removed from the input image, thereby reconstructing a clearer and cleaner image.

[0041] In this embodiment, DnCNN comprises three parts: the first part is a convolutional layer plus an activation function, which contains 32 filters of size 3×3×c to generate 32 feature maps, followed by the activation function. Here, c represents the number of image channels; if the lawnmower robot's camera input is a color image, then c = 3. The second part consists of multiple combined layers containing convolutional layers, batch normalization layers, and activation functions, including 64 filters of size 3×3×64. A batch normalization layer is added between the convolutional layers and the activation function. The third part is a convolutional layer containing c filters of size 3×3×64 to output a residual image. The relationship between the noise-removed image (clean image P), the noisy input image N, and the output residual image R is P = NR.

[0042] Backlight processing employs high dynamic range (HDR) technology and backlight compensation technology to reduce the impact of backlight on the image. In this embodiment, HDR technology is used to synthesize multiple images with different exposures to obtain a wider brightness range and better image detail. Backlight compensation technology is used to reduce the impact of an overly bright background and improve the visibility of the foreground area by adjusting the camera's exposure parameters or using image processing algorithms.

[0043] Image segmentation involves extracting the regions of interest from an image through thresholding, region segmentation, and gradient segmentation.

[0044] In this embodiment, gradient-based segmentation utilizes the gradient information of the image for segmentation, primarily processing distant boundary information in the scene. Specific methods include edge detection and edge connection. Threshold-based segmentation divides the image into two regions with different brightness levels by setting a brightness threshold, facilitating further image processing. Region-based segmentation divides the image into continuous regions with similar features. To help the lawnmower robot more easily learn to distinguish between different regions or categories, specific methods include region growing and watershed algorithms.

[0045] Image enhancement enhances the detail and features of an image through local contrast enhancement and adaptive enhancement.

[0046] This embodiment uses local contrast enhancement to improve the local contrast of the image and highlight the details of the lawn boundary. Specifically, it uses a Gaussian filter to smooth the image and then subtracts the original image from the smoothed image. The expression is as follows:

[0047] output(i,j)=input(i,j)-Gaussian_filtered(i,j),

[0048] In the formula, output(i,j) represents the pixel values ​​in the enhanced image, input(i,j) represents the pixel values ​​in the original image, and Gaussian_filtered(i,j) represents the pixel values ​​in the image after the original image has been smoothed using a Gaussian filter.

[0049] In this embodiment, at different times outdoors, different regions of an image often encounter varying lighting conditions or noise levels. Therefore, adaptive enhancement is needed to adjust the enhancement parameters based on the characteristics of local regions to adapt to the needs of different areas. Adaptive enhancement typically involves dynamically adjusting the enhancement level based on the local features of the image to ensure appropriate enhancement for different regions. Its expression is:

[0050] output(i,j)=input(i,j)×adjustment_factor(i,j),

[0051] In the formula, output(i,j) represents the pixel value in the enhanced image, input(i,j) represents the pixel value in the original image, and adjustment_factor is the adjustment factor calculated based on the local features of the pixel.

[0052] Standard normalization is the process of standardizing each pixel of an image.

[0053] In this embodiment, the standard normalization is Z-score normalization, which is used to convert the data into a standard normal distribution with a mean of 0 and a standard deviation of 1. The collected values ​​of pixels in the image are 0 to 255. Each pixel in the image is normalized to a value between 0 and 1, ensuring that each channel of each image is normalized.

[0054] The image processing unit 14 includes a GrassBEV network model, which outputs the preprocessed image to the GrassBEV network model to obtain lawn segmentation information, robot relative position coordinates, and predicted object category and position.

[0055] Figure 3 This is a block diagram of the GrassBEV network model in an embodiment of the present invention.

[0056] like Figure 3 As shown, the GrassBEV network model 300 includes a feature extraction module 301, a feature enhancement module 302, a first position encoding module 303, a bird's-eye view transformation module 304, a second position encoding module 305, a multi-head self-attention module 306, a first normalization module 307, a multi-head cross-attention module 308, a second normalization module 309, a feature optimization module 310, and a detection and segmentation module 311.

[0057] The feature extraction module 301 is used to extract features from the preprocessed image to obtain multi-scale features. In this embodiment, the feature extraction module 301 includes ResNet50 for feature extraction.

[0058] The feature enhancement module 302 includes a feature pyramid network for processing multi-scale features to obtain multiple image feature maps.

[0059] The first position encoding module 303 is used to add a corresponding position encoding to each image feature map to obtain the corresponding first encoded feature map.

[0060] The calculation expression for the position code in this embodiment is:

[0061]

[0062]

[0063] In the formula, x and y are the coordinates in the image feature map, and d is the feature dimension.

[0064] The bird's-eye view conversion module 304 stores geometric transformation data, which is used to aggregate feature points in each first encoded feature map into the bird's-eye view based on the geometric transformation data, so as to obtain the corresponding bird's-eye view feature map.

[0065] In this embodiment, the geometric transformation data includes the camera's intrinsic parameter matrix and extrinsic parameter matrix. The intrinsic parameter matrix is ​​used to describe the camera's focal length, principal point, and other information, while the extrinsic parameter matrix is ​​used to describe the camera's position and orientation in the world coordinate system.

[0066] The expressions for the intrinsic parameter matrix and the extrinsic parameter matrix are:

[0067]

[0068]

[0069] In the formula, K is the intrinsic parameter matrix, [R|t] is the extrinsic parameter matrix, and f x f y c is the focal length. x c y Principal point coordinates, r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 and r 33 Let t be an element in the rotation matrix R. x t y and t zis an element in the translation vector t.

[0070] The bird's-eye view conversion module 304 performs homogeneous coordinate transformation based on geometric transformation data, then performs perspective transformation, and finally aggregates the feature points mapped to the ground plane into the bird's-eye view, and uses interpolation methods to process multi-scale features.

[0071] The expression for homogeneous coordinate transformation is:

[0072] P camera =K[R|t]P world ,

[0073] In the formula P world P is a point in the world coordinate system. camera This is a point in the camera coordinate system.

[0074] The expression for perspective shift is:

[0075] P BEV =HP camera ,

[0076] In the formula, H is the projection matrix, and P BEV The point is in the bird's-eye view coordinate system.

[0077] The second position encoding module 305 is used to add corresponding position codes to each bird's-eye view feature map to obtain the corresponding second encoded feature map. In this embodiment, the position code generation method of the second position encoding module 305 is the same as the position code generation method in the first position encoding module 303.

[0078] The multi-head self-attention module 306 is used to perform multi-head self-attention processing on each second encoded feature map to obtain the corresponding bird's-eye view features.

[0079] In this embodiment, the multi-head self-attention module 306 performs global feature integration within the bird's-eye view BEV space, modeling the relationships between different BEV features. In the BEV representation space, a series of queries are generated, representing regions of interest at different locations and scales. Specifically, for each bird's-eye view BEV feature map, the multi-head self-attention mechanism models and captures the relationships between features at different locations within the BEV space.

[0080] The expression for the attention mechanism is:

[0081]

[0082] In the formula, Q is the query matrix, K is the key matrix, V is the value matrix, and d kdenoted as the dimension of the key matrix. Q, K, and V all originate from the same modal bird's-eye view, the BEV feature view, i.e., the second encoded feature map, where Q represents the location of the query target.

[0083] The multi-head attention mechanism applies the above attention mechanism separately to each head, where the formula for each head is:

[0084]

[0085] Then, the outputs of each head are concatenated and transformed through a linear layer. The expression for multi-head attention is then:

[0086] MultiHead(Q,K,V)=Concat(head1,...,head h W O .

[0087] The first normalization module 307 is used to add each bird's-eye view feature to the corresponding first encoded feature map and normalize it to obtain the corresponding first normalized feature.

[0088] The multi-head cross-attention module 308 is used to perform multi-head cross-attention processing on each first normalized feature and multi-scale features to obtain the corresponding fused features.

[0089] In this embodiment, the expression for multi-head cross-attention is the same as that for the multi-head attention mechanism. The difference is that multi-scale features are used as k and V in multi-head cross-attention, and the first normalized feature query is used as Q. Therefore, the fused features integrate the complementary information of BEV and RGB feature maps, which can better capture the details and contextual information of the scene.

[0090] The second normalization module 309 is used to add the fused feature to the query of the corresponding normalized feature for each fused feature, and then normalize it to obtain the corresponding second normalized feature. In this embodiment, the addition operation can effectively transmit information and alleviate the gradient vanishing problem, while normalization improves the stability of the model.

[0091] The feature optimization module 310 includes a feedforward neural network, which is used to input each second normalized feature into the feedforward neural network and add the output of the feedforward neural network to the second normalized feature to obtain the corresponding fused bird's-eye view feature map.

[0092] In this embodiment, the feedforward neural network consists of two fully connected layers and an activation function ReLU, which is used to enhance feature representation, capture nonlinear relationships, and adjust feature dimensions to capture more complex relationships.

[0093] The detection and segmentation module 311 is used to generate lawn segmentation information, robot relative position coordinates, and predicted object category and location based on all fused bird's-eye view feature maps.

[0094] In this embodiment, the detection and segmentation module 311 includes a detection head and a segmentation head. The detection head performs target detection, generates the robot's relative position coordinates and predicts the object's category and location, and the segmentation head performs semantic segmentation to generate lawn segmentation information, i.e., the lawn segmentation map in the bird's-eye view.

[0095] The global position estimation unit 15 is used to obtain the global position estimate of the robot based on GPS position information, inertial measurement angular velocity information, inertial measurement acceleration information, motor encoder speed information and the robot's relative position coordinates.

[0096] Figure 4 This is a schematic diagram of the process by which the global position estimation unit generates the robot's global position estimate in an embodiment of the present invention.

[0097] like Figure 4 As shown, the process by which the global position estimation unit 15 generates a global position estimate for the robot includes the following steps:

[0098] Step S1: Construct the initial state estimate and the initial covariance matrix.

[0099] Step S2: Calculate the predicted state based on the initial state estimate, and calculate the predicted covariance based on the covariance matrix. In this embodiment, the expressions for the predicted state and the predicted covariance are:

[0100]

[0101]

[0102] In the formula To predict the state, P k To predict covariance, F k-1 Q is the state transition matrix, also known as the Jacobian matrix. k-1 For process noise covariance, u k For the control input, f is the system dynamics equation.

[0103] The expression for system state x is:

[0104]

[0105] In the formula p x and p y Let x and y be the robot's global position coordinates, v be the linear velocity, θ be the attitude information (i.e., the orientation angle relative to the y-axis), and ω be the angular velocity.

[0106] Step S3: Calculate the Kalman gain based on the predicted covariance. Kalman gain K k The expression is:

[0107] Step S4: The system state is calculated based on GPS location information, inertial measurement angular velocity information, inertial measurement acceleration information, motor encoder speed information, and the robot's relative position coordinates. In this embodiment, the system state is observed using the sensors of GPS sensor 121, inertial measurement unit 122, and motor encoder speed unit 123, and a measurement model corresponding to each sensor is determined. This measurement model describes how the corresponding sensor observations are correlated with the system state.

[0108] Step S5: Based on the Kalman gain, the predicted state, and the system state, obtain the updated predicted state. The expression for the updated predicted state is:

[0109] Step S6: Based on the Kalman gain and the prediction covariance, obtain the updated prediction covariance. The expression for the updated prediction covariance is: P k =(IK k H k )P k .

[0110] Step S7: Repeat steps S5 to S6 until an updated predicted state is obtained as the robot's global position estimate.

[0111] The path planning and command generation unit 16 includes a GrassRoutePlan network model, which is used to input the robot's global position estimate, lawn segmentation information and the position of objects classified as obstacles into the GrassRoutePlan network model to obtain the predicted trajectory of the lawn mowing task and the corresponding control commands.

[0112] Figure 5 This is a block diagram of the GrassRoutePlan network model in an embodiment of the present invention.

[0113] like Figure 5 As shown, the GrassRoutePlan network model 400 includes a multi-scale extraction module 401, an encoding module 402, a first attention module 403, a second attention module 404, a normalization module 405, and a prediction module 406.

[0114] The multi-scale extraction module 401 is used to extract multi-scale features from the robot's global position estimation, lawn segmentation information, and the positions of objects classified as obstacles, resulting in multiple image feature maps. In this embodiment, the multi-scale extraction module 401 includes a traditional convolutional neural network (CNN) model for multi-scale feature extraction.

[0115] The encoding module 402 is used to add positional encoding to each image feature map to obtain the corresponding encoded feature map. In this embodiment, the positional encoding generation method of the encoding module 402 is the same as the positional encoding generation method in the first positional encoding module 303.

[0116] The first attention module 403 is used to perform multi-head self-attention processing on each encoded feature map to obtain the corresponding first feature map. In this embodiment, the multi-head self-attention processing of the first attention module 403 is calculated in the same way as the multi-head self-attention processing of the multi-head self-attention module 306.

[0117] The second attention module 404 is used to perform multi-head cross-attention processing on each first feature map, combining the robot's global position estimation, lawn segmentation information, and the position of objects classified as obstacles, to obtain the corresponding second feature map.

[0118] In this embodiment, the multi-head cross-attention processing of the second attention module 404 is the same as the multi-head cross-attention processing of the multi-head cross-attention module 308, but Q in the multi-head cross-attention processing of the second attention module 404 is a feature representation corresponding to the robot's future trajectory and control commands.

[0119] The normalization module 405 is used to perform layer normalization on each second feature map to obtain the corresponding normalized features.

[0120] The prediction module 406 is used to obtain the predicted trajectory and control commands based on all normalized features. In this embodiment, the prediction module 406 includes a trajectory prediction header and a control command header. The trajectory prediction header is used to generate the predicted trajectory, and the control command header is used to generate the control commands.

[0121] The power detection unit 17 is used to detect the remaining power of the intelligent lawnmower robot 10 and generate a feedback signal when the remaining power is less than the power threshold.

[0122] The charging calibration unit 18 is used to generate a charging docking path trajectory and corresponding docking control commands based on the image of the vector graphic markers acquired by the data acquisition unit 12.

[0123] The control unit 19 stores control programs that control the operation of each unit.

[0124] When the intelligent lawn mowing robot is performing a lawn mowing task, the control unit 19 controls the drive unit 11 according to the control command, so that the intelligent lawn mowing robot 10 mows the lawn according to the predicted trajectory.

[0125] When the power detection unit 17 generates a feedback signal, the control unit 19, based on the feedback signal and the position of the object classified as a charging pile, controls the path planning and command generation unit 16 to generate a predicted trajectory and control command leading to the charging pile 20, and controls the intelligent lawnmower robot 10 to move to the designated position of the charging pile 20 through the control command to achieve charging.

[0126] The charging pile 20 is equipped with vector graphic markings, and the vector graphic markings are located on the same side of the charging device of the charging pile 10.

[0127] Figure 6 This is a schematic diagram of vector graphic markers in an embodiment of the present invention.

[0128] After the intelligent lawnmower robot 10 needs to be charged and moves to the vicinity of the charging pile 20, it will rotate to search for and identify vector graphic marks on the charging pile 20. When the charging calibration unit 18 generates the charging docking path trajectory and the corresponding docking control command, the control unit 19 controls the drive unit 11 to move, that is, adjust the angle and moving distance, so that the intelligent lawnmower robot 10 docks with the charging device.

[0129] The role and effect of the embodiments

[0130] According to the multi-sensor fusion intelligent lawnmower robot and system involved in this embodiment, firstly, the image processing unit extracts bird's-eye view features from the preprocessed image and reconstructs environmental and bird's-eye view features to generate accurate lawn segmentation information, robot relative position coordinates, and predicted object categories and positions. Secondly, the global position estimation unit calibrates the position of the intelligent lawnmower robot based on sensor data collected by the data acquisition unit and robot relative position coordinates generated by the image processing unit, greatly reducing the reliance on traditional high-precision positioning sensors and reducing the cost of the intelligent lawnmower robot by one-third. Thirdly, the path planning and command generation unit predicts the path for mowing while avoiding obstacles and generates control commands based on the robot's global position estimate generated by the global position estimation unit, the lawn segmentation information generated by the image processing unit, and the obstacle positions, thereby simplifying the complexity of manually calculating the path trajectory and improving the robustness of the path planning operation. In summary, this method can complete the lawnmower task efficiently, stably, cost-effectively, and robustly.

[0131] Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A multi-sensor fusion intelligent mowing robot, characterized in that, include: The drive unit is used to move the intelligent lawnmower robot; The data acquisition unit is used to acquire visible light images, GPS location information, inertial measurement angular velocity information, inertial measurement acceleration information, and motor coding speed information of the scene where the intelligent lawnmower robot is located. An image preprocessing unit is used to preprocess the visible light image to obtain a preprocessed image; The image processing unit includes a GrassBEV network model, which outputs the preprocessed image to the GrassBEV network model to obtain lawn segmentation information, robot relative position coordinates, and predicted object category and location. The GrassBEV network model includes: The feature extraction module is used to extract features from the preprocessed image to obtain multi-scale features; The feature enhancement module includes a feature pyramid network for processing the multi-scale features to obtain multiple image feature maps; The first position encoding module is used to add a corresponding position encoding to each of the image feature maps to obtain the corresponding first encoded feature map; The bird's-eye view conversion module stores geometric transformation data and is used to aggregate feature points in each of the first encoded feature maps into the bird's-eye view based on the geometric transformation data to obtain the corresponding bird's-eye view feature map. The second position encoding module is used to add corresponding position codes to each of the bird's-eye view feature maps to obtain the corresponding second encoded feature map; The multi-head self-attention module is used to perform multi-head self-attention processing on each of the second encoded feature maps to obtain the corresponding bird's-eye view features; The first normalization module is used to add the bird's-eye view feature to the corresponding first encoded feature map and normalize it to obtain the corresponding first normalized feature. The multi-head cross-attention module is used to perform multi-head cross-attention processing on each of the first normalized features and the multi-scale features to obtain the corresponding fused features. The second normalization module is used to add the fusion feature to the query of the corresponding normalized feature for each fusion feature, and then normalize it to obtain the corresponding second normalized feature. The feature optimization module includes a feedforward neural network, which is used to input each of the second normalized features into the feedforward neural network and add the output of the feedforward neural network to the second normalized features to obtain the corresponding fused bird's-eye view feature map. The detection and segmentation module is used to generate the lawn segmentation information, the robot's relative position coordinates, and the predicted object's category and location based on all the fused bird's-eye view feature maps; The global position estimation unit is used to obtain a global position estimate of the robot based on the GPS position information, the inertial measurement angular velocity information, the inertial measurement acceleration information, the motor encoder speed information, and the robot's relative position coordinates; The path planning and command generation unit includes a GrassRoutePlan network model, which is used to input the robot's global position estimate, the lawn segmentation information, and the position of the object classified as an obstacle into the GrassRoutePlan network model to obtain the predicted trajectory of the lawn mowing task and the corresponding control command. The control unit controls the drive unit according to the control command, so that the intelligent lawnmower robot mows the lawn according to the predicted trajectory.

2. The intelligent lawnmower robot with multi-sensor fusion according to claim 1, characterized in that: wherein, The preprocessing includes image denoising, backlight processing, image segmentation, image enhancement, and standard normalization. The image denoising process removes noise from the image using a pre-trained deep learning model. The backlight processing involves using high dynamic range technology and backlight compensation technology to reduce the impact of backlight on the image. The image segmentation process involves extracting the regions of interest from the image through threshold segmentation, region segmentation, and gradient segmentation. The image enhancement method involves strengthening the image's detail and features through local contrast enhancement and adaptive enhancement. The standard normalization is to standardize each pixel of the image.

3. The multi-sensor fusioned intelligent mowing robot of claim 1, Its features are: The drive unit includes movable wheels. The data acquisition unit includes: A GPS sensor is used to collect the GPS location information; An inertial measurement unit is used to collect the inertial measurement angular velocity information and the inertial measurement acceleration information; The motor encoding speed unit includes a motor encoder, used to obtain the robot's linear velocity and angular velocity as the motor encoding speed information. The inertial measurement angular velocity information includes angular velocity values ​​on the X, Y, and Z axes. The inertial measurement acceleration information includes acceleration values ​​on the X, Y, and Z axes.

4. The multi-sensor fusioned intelligent mowing robot of claim 1, Its features are: in, The process by which the global position estimation unit generates the robot's global position estimate includes the following steps: Step S1: Construct the initial state estimate and the initial covariance matrix; Step S2: Calculate the predicted state based on the initial state estimate, and calculate the predicted covariance based on the covariance matrix; Step S3: Calculate the Kalman gain based on the predicted covariance; Step S4: Calculate the system state based on the GPS location information, the inertial measurement angular velocity information, the inertial measurement acceleration information, the motor encoder speed information, and the robot's relative position coordinates; Step S5: Based on the Kalman gain, the predicted state, and the system state, obtain the updated predicted state; Step S6: Based on the Kalman gain and the prediction covariance, obtain the updated prediction covariance; Step S7: Repeat steps S5 to S6 until the updated predicted state is obtained as the global position estimate of the robot.

5. The multi-sensor fusioned intelligent mowing robot of claim 1, Its features are: in, The GrassRoutePlan network model includes: A multi-scale extraction module is used to extract multi-scale features from the robot's global position estimation, the lawn segmentation information, and the position of the object classified as an obstacle, to obtain multiple image feature maps; The encoding module is used to add positional encoding to each of the image feature maps to obtain the corresponding encoded feature maps; The first attention module is used to perform multi-head self-attention processing on each of the encoded feature maps to obtain the corresponding first feature map; The second attention module is used to perform multi-head cross-attention processing on each of the first feature maps, combining the robot's global position estimation, the lawn segmentation information, and the position of the object classified as an obstacle, to obtain the corresponding second feature map; The normalization module is used to perform layer normalization on each of the second feature maps to obtain the corresponding normalized features; The prediction module is used to obtain the predicted trajectory and the control command based on all the normalized features.

6. An intelligent mowing robot system characterized in that, include: Intelligent lawnmower robots and charging stations The intelligent lawnmower robot described herein is the multi-sensor fusion intelligent lawnmower robot as described in any one of claims 1 to 5. The intelligent lawnmower robot is charged via the charging station.

7. The intelligent lawnmower robot system according to claim 6, characterized in that: wherein The intelligent lawnmower also includes: The power detection unit is used to detect the remaining power of the intelligent lawnmower robot and generate a feedback signal when the remaining power is less than a power threshold. Based on the feedback signal and the location of the object classified as a charging pile, the control unit controls the path planning and command generation unit to generate a predicted trajectory and control command leading to the charging pile, and controls the intelligent lawnmower robot to move towards the charging pile for charging through the control command.

8. The intelligent lawnmower robot system according to claim 7, characterized in that: wherein The charging pile is equipped with vector graphic markings. The vector graphic markers are set on the same side of the charging device of the charging pile. The intelligent lawnmower also includes: The charging calibration unit is used to generate a charging docking path trajectory and corresponding docking control commands based on the image of the vector graphic markers acquired by the data acquisition unit. The control unit controls the drive unit to move via the docking control command, so that the intelligent lawnmower robot docks with the charging device.

Images