Multi-sensor fusion intelligent mowing robot and system

Through a multi-sensor fusion system, combining image processing and sensor data, lawn segmentation information and prediction trajectory are generated, which solves the problem of recognition accuracy and control robustness of intelligent mowing robots in complex scenarios, and achieves efficient, stable and low-cost Grass mowing mission.

CN119014201AActive Publication Date: 2024-11-26FUDAN UNIVERSITY
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411112793.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-26
Estimated Expiration
2044-08-14

AI Technical Summary

Technical Problem

Existing intelligent mowing robots have low recognition accuracy in complex lawn scenes, making it difficult to deal with dynamic obstacles and environmental interference, resulting in misjudgment and positioning errors. Traditional control methods cannot cope with complex situations, resulting in problems such as robots being out of bounds or being stuck.

Method used

The multi-sensor fusion system is adopted, including the driver unit, data acquisition unit, image preprocessing, image processing unit, global position estimation unit and path planning unit. The GrassBEV network model and GrassRoutePlan network model are used to combine GPS, inertial measurement and motor coded data. Perform image preprocessing and feature extraction to generate lawn segmentation information and prediction trajectories to achieve higher recognition accuracy and control robustness.

Benefits of technology

The recognition accuracy and control capabilities of the intelligent mowing robot are improved, the dependence on traditional high-precision positioning sensors is reduced, the cost is reduced, and efficient, stable and low-cost mowing tasks are achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119014201A_ABST
    Figure CN119014201A_ABST
Patent Text Reader

Abstract

The invention provides a multi-sensor fusion intelligent mowing robot and system, and the robot is characterized in that the robot comprises a driving part; a data acquisition unit; the image preprocessing part is used for preprocessing the visible light image to obtain a preprocessed image; the image processing part is used for outputting the preprocessed image to the GrassBEV network model to obtain lawn segmentation information, relative position coordinates of the robot and the category and position of a predicted object; the global position estimation part is used for obtaining robot global position estimation according to the acquired information and the relative position coordinates of the robot; and the path planning and instruction generation part is used for inputting the global position estimation of the robot, the lawn segmentation information and the position of the object of which the type is the obstacle into the GrassRoutePlan network model together to obtain a predicted trajectory of a mowing task and a corresponding control command. In a word, the method can efficiently and stably complete the mowing task with low cost and high robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of intelligent garden robots, and in particular relates to an intelligent lawn mowing robot and system with multi-sensor fusion. Background Art

[0002] The intelligent lawn mowing robot system is a vital component of the intelligent garden system, and is mainly used to replace manual automatic mowing. In recent years, visual recognition algorithms have been used through cameras to determine lawn boundaries and obstacles. However, the algorithms in pure computer graphics have fixed feature requirements for lawn and obstacle images, and the terrain, color, state, and even the weather and light of each user's lawn are ever-changing. The images collected by the camera are likely to be different from the original test data set. The lawn mowing robot also needs to face complex scenes and dynamic obstacles, which will show low recognition accuracy and is extremely prone to misjudgment. Robots that are simply positioned with GPS or RTK cannot solve the interference of various interference objects such as surrounding buildings and trees on positioning, which often causes the machine to go out of bounds or even fall into the river.

[0003] In addition, traditional lawn mowing robots use automated control to control the robot's operations. The traditional control method is to collect data from sensors, input it into a conventional feedback system, and perform simple logical judgments, which cannot traverse most of the complex situations encountered by the robot. The robot's control needs to handle a variety of complex situations, such as avoiding pits, walking in parallel straight lines even though the grass is rough, as well as a series of situations such as slipping, getting stuck, narrow passages, isolated islands, etc. Therefore, lawn mowing robots urgently need higher recognition accuracy and more intelligent control methods to enhance the robot's generalization ability and robustness to deal with things, and to adapt to complex lawn scenes.

[0004] Therefore, in lawn scenarios, there are still many pain points in smart lawn mowing robots that need to be solved urgently. Summary of the invention

[0005] The present invention is made to solve the above-mentioned problems, and aims to provide an intelligent lawn mowing robot and system with multi-sensor fusion.

[0006] The present invention provides a multi-sensor fusion intelligent lawn mowing robot, which has the following characteristics: a driving unit, used to make the intelligent lawn mowing robot move; a data acquisition unit, used to collect visible light images, GPS position information, inertial measurement angular velocity information, inertial measurement acceleration information and motor encoding speed information of the scene where the intelligent lawn mowing robot is located; an image preprocessing unit, used to preprocess the visible light image to obtain a preprocessed image; an image processing unit, including a GrassBEV network model, used to output the preprocessed image to the GrassBEV network model, obtain lawn segmentation information, robot relative position coordinates and predicted object category and position; a global position estimation unit, used to obtain the robot's global position estimation based on GPS position information, inertial measurement angular velocity information, inertial measurement acceleration information, motor encoding speed information and robot relative position coordinates; a path planning and instruction generation unit, including a GrassRoutePlan network model, used to input the robot's global position estimation, lawn segmentation information and the position of objects classified as obstacles into the GrassRoutePlan network model to obtain a predicted trajectory of the mowing task and a corresponding control command; a control unit, which controls the drive unit according to the control command, so that the intelligent mowing robot mows according to the predicted trajectory.

[0007] The multi-sensor fusion intelligent lawn mowing robot provided by the present invention may also have the following features: wherein, preprocessing includes image denoising, backlight processing, image segmentation, image enhancement and standard normalization, image denoising removes image noise through a pre-trained deep learning model, backlight processing uses high dynamic range technology and backlight compensation technology to reduce the impact of backlight on the image, image segmentation extracts areas of interest in the image through threshold segmentation, region segmentation and gradient segmentation, image enhancement strengthens the details and features of the image through local contrast enhancement and adaptive enhancement, and standard normalization standardizes each pixel of the image.

[0008] The multi-sensor fusion intelligent lawn mowing robot provided by the present invention may also have the following features: wherein the GrassBEV network model includes: a feature extraction module, which is used to extract features from the preprocessed image to obtain multi-scale features; a feature enhancement module, which includes a feature pyramid network, which is used to process the multi-scale features to obtain multiple image feature maps; a first position encoding module, which is used to add a corresponding position code to each image feature map to obtain a corresponding first encoded feature map; a bird's-eye view conversion module, which stores geometric transformation data and is used to aggregate feature points in each first encoded feature map to a bird's-eye view according to the geometric transformation data to obtain a corresponding bird's-eye view feature map; a second position encoding module, which is used to add a corresponding position code to each bird's-eye view feature map to obtain a corresponding second encoded feature map; a multi-head self-attention module, which is used to perform multi-head self-attention processing on each second encoded feature map to obtain a corresponding a first normalization module for adding the bird's-eye view features to the corresponding first encoding feature map for each bird's-eye view feature and performing normalization to obtain the corresponding first normalized feature; a multi-head cross-attention module for performing multi-head cross-attention processing on each first normalized feature in combination with the multi-scale feature to obtain the corresponding fused feature; a second normalization module for adding the fused feature to the query of the corresponding normalized feature for each fused feature and performing normalization to obtain the corresponding second normalized feature; a feature optimization module including a feedforward neural network for inputting the second normalized feature into the feedforward neural network for each second normalized feature, and adding the output of the feedforward neural network to the second normalized feature to obtain the corresponding fused bird's-eye view feature map; a detection and segmentation module for generating lawn segmentation information, robot relative position coordinates and predicted object category and position according to all fused bird's-eye view feature maps.

[0009] The multi-sensor fusion intelligent lawn mowing robot provided by the present invention may also have the following features: wherein the driving unit includes a moving wheel, and the data acquisition unit includes: a GPS sensor for collecting GPS position information; an inertial measurement unit for collecting inertial measurement angular velocity information and inertial measurement acceleration information; a motor encoding speed unit including a motor encoder for obtaining the robot linear velocity and the robot angular velocity as the motor encoding speed information, the inertial measurement angular velocity information includes angular velocity values ​​on the three axes of X, Y, and Z, and the inertial measurement acceleration information includes acceleration values ​​on the three axes of X, Y, and Z.

[0010] In the multi-sensor fusion intelligent lawn mowing robot provided by the present invention, it can also have the following characteristics: wherein, the process of the global position estimation unit generating the global position estimation of the robot includes the following steps: step S1, constructing an initialization state estimate and an initial covariance matrix; step S2, calculating the predicted state according to the initialization state estimate, and calculating the predicted covariance according to the covariance matrix; step S3, calculating the Kalman gain according to the predicted covariance; step S4, calculating the system state according to the GPS position information, the inertial measurement angular velocity information, the inertial measurement acceleration information, the motor encoding speed information and the robot relative position coordinates; step S5, obtaining an updated predicted state according to the Kalman gain, the predicted state and the system state; step S6, obtaining an updated predicted covariance according to the Kalman gain and the predicted covariance; step S7, repeating steps S5 to S6 until the updated predicted state is obtained as the global position estimate of the robot.

[0011] The multi-sensor fusion intelligent lawn mowing robot provided by the present invention may also have the following features: wherein the GrassRoutePlan network model includes: a multi-scale extraction module, which is used to perform multi-scale feature extraction on the robot's global position estimation, lawn segmentation information, and the position of objects classified as obstacles to obtain multiple image feature maps; an encoding module, which is used to perform position encoding and addition on each image feature map to obtain a corresponding encoding feature map; a first attention module, which is used to perform multi-head self-attention processing on each encoding feature map to obtain a corresponding first feature map; a second attention module, which is used to perform multi-head cross-attention processing on each first feature map in combination with the robot's global position estimation, lawn segmentation information, and the position of objects classified as obstacles to obtain a corresponding second feature map; a normalization module, which is used to perform layer normalization on each second feature map to obtain a corresponding normalized feature; a prediction module, which is used to obtain a predicted trajectory and control command based on all normalized features.

[0012] The present invention also provides a multi-sensor fusion intelligent lawn mowing robot system having the following characteristics, including: an intelligent lawn mowing robot and a charging pile, wherein the intelligent lawn mowing robot is any one of the multi-sensor fusion intelligent lawn mowing robots mentioned above, and the intelligent lawn mowing robot is charged through the charging pile.

[0013] The multi-sensor fusion intelligent lawn mowing robot system provided by the present invention may also have the following features: wherein the intelligent lawn mowing robot also includes: a power detection unit, for detecting the remaining power of the intelligent lawn mowing robot, and generating a feedback signal when the remaining power is less than a power threshold; the control unit controls the path planning and instruction generation unit to generate a predicted trajectory and control commands leading to the charging pile based on the feedback signal and the position of an object classified as a charging pile, and controls the intelligent lawn mowing robot to move to the charging pile for charging through the control command.

[0014] The multi-sensor fusion intelligent lawn mowing robot system provided by the present invention may also have the following features: wherein, the charging pile is provided with a vector graphic mark, and the vector graphic mark is provided on the same side of the charging device of the charging pile, and the intelligent lawn mowing robot also includes: a charging calibration unit, which is used to generate a charging docking path trajectory and a corresponding docking control command according to the image of the vector graphic mark collected by the data acquisition unit, and the control unit controls the movement of the driving unit through the docking control command to dock the intelligent lawn mowing robot with the charging device.

[0015] Functions and Effects of the Invention

[0016] According to the multi-sensor fusion intelligent lawn mowing robot and system involved in the present invention, first, the image processing unit extracts the bird's-eye view features of the pre-processed image, and reconstructs the environmental features and the bird's-eye view features, thereby generating accurate lawn segmentation information, robot relative position coordinates, and predicted object categories and positions; second, the global position estimation unit calibrates the position of the intelligent lawn mowing robot according to the sensor data collected by the data acquisition unit and the robot relative position coordinates generated by the image processing unit, greatly reducing the dependence on traditional high-precision positioning sensors, and can reduce the cost of the intelligent lawn mowing robot by 1 / 3. Third, the path planning and instruction generation unit predicts the path of avoiding obstacles and mowing at the same time and generates control commands according to the robot global position estimation generated by the global position estimation unit, the lawn segmentation information and obstacle positions generated by the image processing unit, thereby simplifying the complexity of artificially calculating the path trajectory and improving the robustness of the path planning operation. Therefore, the multi-sensor fusion intelligent lawn mowing robot and system of the present invention can complete the mowing task efficiently, stably, at low cost and with strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a block diagram of an intelligent lawn mowing robot system in an embodiment of the present invention;

[0018] Figure 2 is a block diagram of an intelligent lawn mowing robot in an embodiment of the present invention;

[0019] Figure 3is a block diagram of a GrassBEV network model in an embodiment of the present invention;

[0020] Figure 4 is a schematic diagram of a process of generating a global position estimate of a robot by a global position estimation unit in an embodiment of the present invention;

[0021] Figure 5 is a block diagram of a GrassRoutePlan network model in an embodiment of the present invention;

[0022] Figure 6 Schematic diagram of a vector graphic mark in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate the multi-sensor fusion intelligent lawn mowing robot and system of the present invention.

[0024] The present embodiment provides an intelligent lawn mowing robot system, which is used to avoid positioning errors and control errors caused by traditional positioning systems affected by buildings and surroundings in an intelligent lawn mowing environment. It is an efficient, stable, low-cost and robust system.

[0025] Figure 1 is a block diagram of an intelligent lawn mowing robot system in an embodiment of the present invention.

[0026] like Figure 1 As shown, the intelligent lawn mowing robot system 100 includes: an intelligent lawn mowing robot 10 and a charging pile 20. The intelligent lawn mowing robot 10 is charged through the charging pile 20.

[0027] Figure 2 is a block diagram of an intelligent lawn mowing robot in an embodiment of the present invention.

[0028] like Figure 2 As shown, the intelligent lawn mowing robot 10 includes: a driving unit 11, a data acquisition unit 12, an image preprocessing unit 13, an image processing unit 14, a global position estimation unit 15, a path planning and instruction generation unit 16, a power detection unit 17, a charging calibration unit 18 and a control unit 19 for controlling the above-mentioned units.

[0029] The driving unit 11 includes moving wheels for moving the intelligent lawn mowing robot 10. In this embodiment, the moving wheels can be replaced by any moving device that can move the intelligent lawn mowing robot 10.

[0030] The data acquisition unit 12 is used to collect visible light images, GPS position information, inertial measurement angular velocity information, inertial measurement acceleration information and motor encoding speed information of the scene where the intelligent lawn mowing robot 10 is located, and includes a GPS sensor 121, an inertial measurement unit 122 and a motor encoding speed unit 123.

[0031] The GPS sensor 121 is used to collect GPS location information. In this embodiment, the GPS sensor 121 parses the information according to the NMEA protocol to obtain longitude and latitude coordinates, altitude, speed, etc. The message format is: "$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47".

[0032] The inertial measurement unit 122 is used to collect inertial measurement angular velocity information and inertial measurement acceleration information.

[0033] The inertial measurement angular velocity information includes the angular velocity values ​​on the X, Y, and Z axes, and the inertial measurement acceleration information includes the acceleration values ​​on the X, Y, and Z axes.

[0034] The motor encoding speed unit 123 includes a motor encoder, which is used to obtain the robot linear velocity and the robot angular velocity as the motor encoding speed information. The specific calculation formula is as follows:

[0035]

[0036]

[0037] Where R L and R R are the radii of the left and right moving wheels, V L and V R are the differential speeds of the left and right moving wheels, V is the linear speed of the robot, and R is the angular speed of the robot.

[0038] The image preprocessing unit 13 is used to preprocess the visible light image to obtain a preprocessed image.

[0039] Among them, preprocessing includes image denoising, backlighting processing, image segmentation, image enhancement and standard normalization.

[0040] Image denoising removes image noise through a pre-trained deep learning model. In this embodiment, the deep learning model is DnCNN, which replaces traditional linear filtering and nonlinear filtering, and uses the powerful representation learning ability of deep convolutional neural networks to suppress or remove noise signals from the input image, thereby reconstructing a clearer and cleaner image.

[0041] In this embodiment, DnCNN includes three parts: the first part is a convolution layer + activation function, which contains 32 filters of size 3×3×c for generating 32 feature maps, followed by an activation function, where c represents the number of image channels. The camera input of the mowing robot is a color image, so c=3; the second part is a combination of multiple convolution layers, batch normalization layers, and activation functions, which contain 64 filters of size 3×3×64, and a batch normalization layer is added between the convolution layer and the activation function; the third part is a convolution layer, which contains c filters of size 3×3×64 to output the residual image. The relationship between the denoised image, i.e., the clean image P, the noisy input image N, and the output residual image R is P=NR.

[0042] Backlight processing uses high dynamic range technology and backlight compensation technology to reduce the impact of backlight on the image. In this embodiment, high dynamic range technology is used to synthesize multiple images with different exposures to obtain a wider brightness range and better image details. Backlight compensation technology is used to reduce the impact of over-brightness of the background and improve the visibility of the foreground area by adjusting the camera's exposure parameters or using image processing algorithms.

[0043] Image segmentation is to extract the area of ​​interest in the image through threshold segmentation, region segmentation and gradient segmentation.

[0044] In this embodiment, the gradient-based segmentation method uses the gradient information of the image to perform segmentation, mainly processing the distant boundary information in the scene. Specific methods include edge detection, edge connection, etc. Threshold-based segmentation is to divide the image into two areas with higher and lower brightness by setting a brightness threshold, which is convenient for further processing of the image later. The region-based segmentation method divides the image into continuous areas with similar features. In order to make it easier for the mowing robot to learn to distinguish different areas or different categories, specific methods include region growing, watershed algorithm, etc.

[0045] Image enhancement is to strengthen the details and features of the image through local contrast enhancement and adaptive enhancement.

[0046] In this embodiment, local contrast enhancement is used to enhance the local contrast of the image and highlight the details of the lawn boundary. The specific method is to use a Gaussian filter to smooth the image and subtract the smoothed image from the original image. The expression is:

[0047] output(i,j)=input(i,j)-Gaussian_filtered(i,j),

[0048] Where output(i,j) is the pixel value in the enhanced image, input(i,j) is the pixel value in the original image, and Gaussian_filtered(i,j) is the pixel value in the image after smoothing the original image using a Gaussian filter.

[0049] In this embodiment, different areas of the image often have different lighting conditions or noise levels at different time points outdoors, so adaptive enhancement is needed to adjust the enhancement parameters according to the characteristics of the local area to meet the needs of different areas. Adaptive enhancement usually involves dynamically adjusting the degree of enhancement according to the local characteristics of the image to ensure appropriate enhancement of different areas, and its expression is:

[0050] output(i,j)=input(i,j)×adjustment_factor(i,j),

[0051] Where output(i,j) is the pixel value in the enhanced image, input(i,j) is the pixel value in the original image, and adjustment_factor is the adjustment factor calculated based on the local features of the pixel.

[0052] Standard normalization is to standardize each pixel of the image.

[0053] In this embodiment, the standard normalization is Z-score normalization, which is used to convert the data into a standard normal distribution with a mean of 0 and a standard deviation of 1. The acquisition value of the pixel point in the image is 0-255, and each pixel point in the image is standardized to a value between 0 and 1, ensuring that each channel of each image is standardized.

[0054] The image processing unit 14 includes a GrassBEV network model, which is used to output the preprocessed image to the GrassBEV network model to obtain lawn segmentation information, robot relative position coordinates and predicted object category and position.

[0055] Figure 3 is a block diagram of a GrassBEV network model in an embodiment of the present invention.

[0056] like Figure 3 As shown, the GrassBEV network model 300 includes a feature extraction module 301, a feature enhancement module 302, a first position encoding module 303, a bird's-eye view conversion module 304, a second position encoding module 305, a multi-head self-attention module 306, a first normalization module 307, a multi-head cross-attention module 308, a second normalization module 309, a feature optimization module 310 and a detection and segmentation module 311.

[0057] The feature extraction module 301 is used to extract features from the preprocessed image to obtain multi-scale features. In this embodiment, the feature extraction module 301 includes ResNet50 for feature extraction.

[0058] The feature enhancement module 302 includes a feature pyramid network, which is used to process multi-scale features to obtain multiple image feature maps.

[0059] The first position encoding module 303 is used to add a corresponding position code to each image feature map to obtain a corresponding first encoded feature map.

[0060] The calculation expression of the position code in this embodiment is:

[0061]

[0062]

[0063] Where x and y are the coordinates in the image feature map, and d is the feature dimension.

[0064] The bird's-eye view conversion module 304 stores geometric transformation data, and is used to aggregate feature points in each first coded feature map into a bird's-eye view according to the geometric transformation data to obtain a corresponding bird's-eye view feature map.

[0065] In this embodiment, the geometric transformation data includes the camera's intrinsic parameter matrix and extrinsic parameter matrix. The intrinsic parameter matrix is ​​used to describe the camera's focal length, principal point and other information, and the extrinsic parameter matrix is ​​used to describe the camera's position and posture in the world coordinate system.

[0066] The expressions of the internal parameter matrix and the external parameter matrix are:

[0067]

[0068]

[0069] Where K is the internal parameter matrix, [R|t] is the external parameter matrix, and f x 、f y is the focal length, c x 、c y is the coordinate of the principal point, r 11 、r 12 、r 13 、r 21 、r 22 、r 23 、r 31 、r 32 and r 33 is the element in the rotation matrix R, t x ,t y and t zis the element in the translation vector t.

[0070] The bird's-eye view conversion module 304 performs homogeneous coordinate conversion according to the geometric transformation data, and then performs perspective conversion, and finally aggregates the feature points mapped to the ground plane into the bird's-eye view, and uses an interpolation method to process multi-scale features.

[0071] The expression for homogeneous coordinate transformation is:

[0072] P camera =K[R|t]P world ,

[0073] Where P world is a point in the world coordinate system, P camera is a point in the camera coordinate system.

[0074] The expression of perspective conversion is:

[0075] P BEV =HP camera ,

[0076] Where H is the projection matrix, P BEV is a point in the bird's-eye view coordinate system.

[0077] The second position coding module 305 is used to add corresponding position coding to each bird's-eye view feature map to obtain a corresponding second coding feature map. In this embodiment, the position coding generation method of the second position coding module 305 is the same as the position coding generation method of the first position coding module 303.

[0078] The multi-head self-attention module 306 is used to perform multi-head self-attention processing on each second encoded feature map to obtain the corresponding bird's-eye view features.

[0079] In this embodiment, the multi-head self-attention module 306 performs global feature integration within the bird's-eye view BEV space to model the relationship between different BEV features. In the BEV representation space, a series of queries are generated, which represent regions of interest at different locations and scales. Specifically, for each bird's-eye view BEV feature map, the multi-head self-attention mechanism is used to model the relationship between features at different locations in the BEV space.

[0080] The expression of the attention mechanism is:

[0081]

[0082] Where Q is the query matrix, K is the key matrix, V is the value matrix, and d kis the dimension of the key matrix. Q, K, and V all come from the same modality bird's-eye view BEV feature view, i.e., the second encoded feature map, where Q represents the location of the query target.

[0083] The multi-head attention mechanism uses the above attention mechanism for each head, where the formula for each head is:

[0084]

[0085] Then the output of each head is concatenated and transformed through a linear layer, the expression of multi-head attention is:

[0086] MultiHead(Q,K,V)=Concat(head1,...,head h )W O .

[0087] The first normalization module 307 is used to add each bird's-eye view feature to the corresponding first coded feature map and perform normalization to obtain a corresponding first normalized feature.

[0088] The multi-head cross attention module 308 is used to perform multi-head cross attention processing on each first normalized feature in combination with the multi-scale feature to obtain the corresponding fusion feature.

[0089] The expression of the multi-head cross attention in this embodiment is the same as the expression form of the multi-head attention mechanism. The difference is that multi-scale features are used as k and V in the multi-head cross attention, and the first normalized feature query is used as Q. Therefore, the fused feature fuses the complementary information of the BEV and RGB feature maps, which can better capture the details and contextual information of the scene.

[0090] The second normalization module 309 is used to add the query of the fused feature and the corresponding normalized feature to each fused feature, and perform normalization to obtain the corresponding second normalized feature. In this embodiment, the addition operation can effectively transmit information and alleviate the gradient disappearance problem, and normalization improves the stability of the model.

[0091] The feature optimization module 310 includes a feedforward neural network, which is used to input each second normalized feature into the feedforward neural network, and add the output of the feedforward neural network to the second normalized feature to obtain a corresponding fused bird's-eye view feature map.

[0092] In this embodiment, the feedforward neural network consists of two fully connected layers and an activation function ReLU, which is used to enhance feature representation, capture nonlinear relationships, and adjust feature dimensions to capture more complex relationships.

[0093] The detection and segmentation module 311 is used to generate lawn segmentation information, robot relative position coordinates and predict the category and position of the object according to all fused bird's-eye view feature maps.

[0094] In this embodiment, the detection and segmentation module 311 includes a detection head and a segmentation head. The detection head performs target detection to generate the robot's relative position coordinates and predict the category and position of the object. The segmentation head performs semantic segmentation tasks to generate lawn segmentation information, that is, a lawn segmentation map in a bird's-eye view.

[0095] The global position estimation unit 15 is used to obtain the global position estimation of the robot according to the GPS position information, the inertial measurement angular velocity information, the inertial measurement acceleration information, the motor encoder speed information and the relative position coordinates of the robot.

[0096] Figure 4 It is a schematic diagram of a flow chart of a global position estimation unit generating a global position estimation of a robot in an embodiment of the present invention.

[0097] like Figure 4 As shown, the process of generating the robot global position estimation unit 15 includes the following steps:

[0098] Step S1, constructing the initial state estimate and the initial covariance matrix.

[0099] Step S2, the predicted state is calculated based on the initialization state estimation, and the predicted covariance is calculated based on the covariance matrix. The expressions of the predicted state and the predicted covariance in this embodiment are:

[0100]

[0101]

[0102] In the formula is the predicted state, P k is the prediction covariance, F k-1 is the state transfer matrix, i.e., the Jacobian matrix, Q k-1 is the process noise covariance, u k is the control input and f is the system dynamics equation.

[0103] The expression of the system state x is:

[0104]

[0105] Where p x and p y are the x- and y-coordinates in the robot's global position coordinate system, v is the linear velocity, θ is the posture information, i.e., the orientation angle relative to the y-axis, and ω is the angular velocity.

[0106] Step S3, calculate the Kalman gain based on the predicted covariance. Kalman gain K k The expression is:

[0107] Step S4, the system state is calculated based on the GPS position information, the inertial measurement angular velocity information, the inertial measurement acceleration information, the motor encoding speed information and the robot relative position coordinates. That is, in this embodiment, the system state is observed by the sensors of the GPS sensor 121, the inertial measurement unit 122 and the motor encoding speed unit 123, and the measurement model corresponding to each sensor is determined, and the measurement model is used to describe how the corresponding sensor observation is associated with the system state.

[0108] Step S5, obtaining an updated predicted state according to the Kalman gain, the predicted state and the system state. The expression for updating the predicted state is:

[0109] Step S6, obtaining an updated prediction covariance according to the Kalman gain and the prediction covariance. The expression for updating the prediction covariance is: k =(IK k H k ) k .

[0110] Step S7, repeating steps S5 to S6 until an updated predicted state is obtained as the robot's global position estimate.

[0111] The path planning and instruction generation unit 16 includes a GrassRoutePlan network model, which is used to input the robot's global position estimation, lawn segmentation information and the positions of objects classified as obstacles into the GrassRoutePlan network model to obtain the predicted trajectory of the mowing task and the corresponding control command.

[0112] Figure 5 It is a block diagram of the GrassRoutePlan network model in an embodiment of the present invention.

[0113] like Figure 5 As shown, the GrassRoutePlan network model 400 includes a multi-scale extraction module 401, an encoding module 402, a first attention module 403, a second attention module 404, a normalization module 405 and a prediction module 406.

[0114] The multi-scale extraction module 401 is used to perform multi-scale feature extraction on the robot global position estimation, lawn segmentation information and the position of objects classified as obstacles to obtain multiple image feature maps. In this embodiment, the multi-scale extraction module 401 includes a traditional convolutional neural network model CNN for multi-scale feature extraction.

[0115] The encoding module 402 is used to add position coding to each image feature map to obtain a corresponding coding feature map. The position coding generation method of the encoding module 402 in this embodiment is the same as the position coding generation method of the first position coding module 303.

[0116] The first attention module 403 is used to perform multi-head self-attention processing on each encoded feature map to obtain a corresponding first feature map. The multi-head self-attention processing of the first attention module 403 in this embodiment is calculated in the same manner as the multi-head self-attention processing of the multi-head self-attention module 306.

[0117] The second attention module 404 is used to perform multi-head cross-attention processing on each first feature map, combining the robot's global position estimation, lawn segmentation information and the position of objects classified as obstacles, to obtain a corresponding second feature map.

[0118] In this embodiment, the steps of the multi-head cross-attention processing of the second attention module 404 are the same as those of the multi-head cross-attention processing of the multi-head cross-attention module 308, but Q in the multi-head cross-attention processing of the second attention module 404 is a feature representation corresponding to the robot's future trajectory and control command.

[0119] The normalization module 405 is used to perform layer normalization on each second feature map to obtain corresponding normalized features.

[0120] The prediction module 406 is used to obtain the predicted trajectory and control command according to all normalized features. In this embodiment, the prediction module 406 includes a trajectory prediction header and a control command header. The trajectory prediction header is used to generate the predicted trajectory, and the control command header is used to generate the control command.

[0121] The power detection unit 17 is used to detect the remaining power of the intelligent lawn mowing robot 10 and generate a feedback signal when the remaining power is less than a power threshold.

[0122] The charging calibration unit 18 is used to generate a charging docking path trajectory and a corresponding docking control command according to the image of the vector graphic mark collected by the data collection unit 12.

[0123] The control unit 19 stores a control program for controlling the operation of each unit.

[0124] When the intelligent lawn mowing robot performs a lawn mowing task, the control unit 19 controls the driving unit 11 according to the control command, so that the intelligent lawn mowing robot 10 mows according to the predicted trajectory.

[0125] When the power detection unit 17 generates a feedback signal, the control unit 19 controls the path planning and instruction generation unit 16 to generate a predicted trajectory and control command leading to the charging pile 20 based on the feedback signal and the position of the object classified as a charging pile, and controls the intelligent lawn mowing robot 10 to move to the designated position of the charging pile 20 through the control command to achieve charging.

[0126] The charging pile 20 is provided with a vector graphic mark, and the vector graphic mark is arranged on the same side of the charging device of the charging pile 10 .

[0127] Figure 6 Schematic diagram of a vector graphic mark in an embodiment of the present invention.

[0128] When the smart lawn mower robot 10 needs to be charged and moves around the charging pile 20, it will rotate and search and identify the vector graphic mark on the charging pile 20. When the charging calibration unit 18 generates a charging docking path trajectory and a corresponding docking control command, the control unit 19 controls the movement of the driving unit 11 through the docking control command, that is, adjusts the angle and moving distance, so that the smart lawn mower robot 10 docks with the charging device.

[0129] Functions and Effects of the Embodiments

[0130] According to the multi-sensor fusion intelligent lawn mowing robot and system involved in this embodiment, first, the image processing unit extracts the bird's-eye view features of the pre-processed image, and reconstructs the environmental features and the bird's-eye view features, thereby generating accurate lawn segmentation information, robot relative position coordinates, and predicted object categories and positions; second, the global position estimation unit calibrates the position of the intelligent lawn mowing robot based on the sensor data collected by the data acquisition unit and the robot relative position coordinates generated by the image processing unit, greatly reducing the dependence on traditional high-precision positioning sensors, and can reduce the cost of the intelligent lawn mowing robot by 1 / 3. Third, the path planning and instruction generation unit predicts the path of mowing while avoiding obstacles and generates control commands based on the robot global position estimation generated by the global position estimation unit, the lawn segmentation information and obstacle positions generated by the image processing unit, thereby simplifying the complexity of manually calculating the path trajectory and improving the robustness of the path planning operation. In short, this method can complete the mowing task efficiently, stably, at low cost and with strong robustness.

[0131] Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-sensor fusion intelligent lawn mowing robot, characterized in that: include: A driving unit, used to move the intelligent lawn mowing robot; A data acquisition unit, used to collect visible light images, GPS location information, inertial measurement angular velocity information, inertial measurement acceleration information and motor encoding speed information of the scene where the intelligent lawn mowing robot is located; An image preprocessing unit, used for preprocessing the visible light image to obtain a preprocessed image; An image processing unit, including a GrassBEV network model, is used to output the preprocessed image to the GrassBEV network model to obtain lawn segmentation information, robot relative position coordinates, and predicted object category and position; A global position estimation unit, used to obtain a global position estimation of the robot according to the GPS position information, the inertial measurement angular velocity information, the inertial measurement acceleration information, the motor encoding speed information and the robot relative position coordinates; A path planning and command generation unit, including a GrassRoutePlan network model, is used to input the robot global position estimation, the lawn segmentation information and the position of the object classified as an obstacle into the GrassRoutePlan network model to obtain a predicted trajectory of a mowing task and a corresponding control command; The control unit controls the driving unit according to the control command so that the intelligent lawn mowing robot mows the lawn according to the predicted trajectory.

2. The multi-sensor fusion intelligent lawn mowing robot according to claim 1, characterized in that: in, The preprocessing includes image denoising, backlighting processing, image segmentation, image enhancement and standard normalization. The image denoising removes the noise of the image through a pre-trained deep learning model. The backlight processing is to use high dynamic range technology and backlight compensation technology to reduce the impact of backlight on the image. The image segmentation is to extract the area of ​​interest in the image through threshold segmentation, region segmentation and gradient segmentation. The image enhancement is to strengthen the details and features of the image through local contrast enhancement and adaptive enhancement. The standard normalization is to standardize each pixel of the image.

3. The multi-sensor fusion intelligent lawn mowing robot according to claim 1, Features: in, The GrassBEV network model includes: A feature extraction module, used to extract features from the preprocessed image to obtain multi-scale features; A feature enhancement module, including a feature pyramid network, for processing the multi-scale features to obtain a plurality of image feature maps; A first position coding module, used for adding a corresponding position code to each of the image feature maps to obtain a corresponding first coding feature map; a bird's-eye view conversion module storing geometric transformation data, for aggregating feature points in each of the first coded feature maps into a bird's-eye view according to the geometric transformation data, to obtain a corresponding bird's-eye view feature map; A second position coding module is used to add a corresponding position code to each of the bird's-eye view feature maps to obtain a corresponding second coded feature map; A multi-head self-attention module, used for performing multi-head self-attention processing on each of the second encoded feature maps to obtain corresponding bird's-eye view features; A first normalization module, configured to add each of the bird's-eye view features to a corresponding first coded feature map and perform normalization to obtain a corresponding first normalized feature; A multi-head cross attention module, used for performing multi-head cross attention processing on each of the first normalized features in combination with the multi-scale features to obtain corresponding fusion features; A second normalization module is used to add the fused feature to the query of the corresponding normalized feature for each fused feature, and perform normalization to obtain a corresponding second normalized feature; a feature optimization module, comprising a feedforward neural network, for each of the second normalized features, inputting the second normalized features into the feedforward neural network, and adding the output of the feedforward neural network to the second normalized features to obtain a corresponding fused bird's-eye view feature map; The detection and segmentation module is used to generate the lawn segmentation information, the robot relative position coordinates and the category and position of the predicted object according to all the fused bird's-eye view feature maps.

4. The multi-sensor fusion intelligent lawn mowing robot according to claim 1, Features: Wherein, the driving part comprises a moving wheel, The data collection unit comprises: A GPS sensor, used to collect the GPS location information; An inertial measurement unit, used for collecting the inertial measurement angular velocity information and the inertial measurement acceleration information; The motor encoding speed unit includes a motor encoder, which is used to obtain the robot linear speed and the robot angular speed as the motor encoding speed information. The inertial measurement angular velocity information includes angular velocity values ​​on the X, Y, and Z axes. The inertial measurement acceleration information includes acceleration values ​​on three axes: X, Y, and Z.

5. The multi-sensor fusion intelligent lawn mowing robot according to claim 1, Features: in, The process of generating the robot global position estimation unit comprises the following steps: Step S1, constructing an initial state estimate and an initial covariance matrix; Step S2, estimating and calculating a predicted state according to the initialization state, and calculating a predicted covariance according to the covariance matrix; Step S3, calculating the Kalman gain according to the predicted covariance; Step S4, calculating the system state according to the GPS position information, the inertial measurement angular velocity information, the inertial measurement acceleration information, the motor encoding speed information and the relative position coordinates of the robot; Step S5, obtaining an updated predicted state according to the Kalman gain, the predicted state and the system state; Step S6, obtaining an updated prediction covariance according to the Kalman gain and the prediction covariance; Step S7, repeating steps S5 to S6 until the updated predicted state is obtained as the global position estimate of the robot.

6. The multi-sensor fusion intelligent lawn mowing robot according to claim 1, Features: in, The GrassRoutePlan network model includes: A multi-scale extraction module, used for performing multi-scale feature extraction on the robot global position estimation, the lawn segmentation information and the position of the object classified as an obstacle, to obtain a plurality of image feature maps; An encoding module, used for adding position encoding to each of the image feature maps to obtain a corresponding encoding feature map; A first attention module is used to perform multi-head self-attention processing on each of the encoded feature maps to obtain a corresponding first feature map; A second attention module is used to perform multi-head cross attention processing on each of the first feature maps in combination with the robot global position estimation, the lawn segmentation information and the position of the object classified as an obstacle, so as to obtain a corresponding second feature map; A normalization module, used for performing layer normalization on each of the second feature maps to obtain corresponding normalized features; A prediction module is used to obtain the predicted trajectory and the control command according to all the normalized features.

7. An intelligent lawn mowing robot system, characterized in that: include: Smart lawn mowing robot and charging pile, Wherein, the intelligent lawn mowing robot is the multi-sensor fusion intelligent lawn mowing robot according to any one of claims 1 to 6, The intelligent lawn mowing robot is charged through the charging pile.

8. The intelligent lawn mowing robot system according to claim 7, characterized in that: in, The intelligent lawn mowing robot also includes: The power detection unit is used to detect the remaining power of the intelligent lawn mowing robot and generate a feedback signal when the remaining power is less than a power threshold. The control unit controls the path planning and instruction generation unit to generate a predicted trajectory and control command leading to the charging pile based on the feedback signal and the position of the object classified as a charging pile, and controls the intelligent lawn mowing robot to move to the charging pile for charging through the control command.

9. The intelligent lawn mowing robot system according to claim 8, characterized in that: in, The charging pile is provided with a vector graphic mark, The vector graphic mark is arranged on the same side of the charging device of the charging pile, The intelligent lawn mowing robot also includes: a charging calibration unit, configured to generate a charging docking path trajectory and a corresponding docking control command according to the image of the vector graphic mark collected by the data collection unit, The control unit controls the movement of the driving unit through the docking control command, so that the intelligent lawn mowing robot docks with the charging device.

Citation Information

Patent Citations

  • Wireless automatic charging calibration device and method for unmanned sweeper

    CN111021295A

  • Unmanned mower trajectory tracking control method and computer readable storage medium

    CN114568108A

  • Lawn 3D printing system and method based on intelligent mowing robot

    CN114868514A

  • Tunnel crack automatic repairing method, device and system and readable storage medium

    CN115182747A

  • Transform-based high-precision map real-time prediction method and system

    CN116071721A