Mower cutting missing detection method and system based on UWB and camera fusion

Through the integration of UWB and camera technology, automatic detection and re-cutting of lawn leaked areas is achieved, the problem of missing mowers is solved, the efficiency and accuracy of mowing are improved, and the efficient management of lawns is achieved.

CN119090837APending Publication Date: 2024-12-06SUZHOU SHENGLEI POWER MASCH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411182108.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Unmanned lawn mowers are prone to missed areas during the mowing process, resulting in low mowing efficiency and accuracy, and lack of effective detection and re-cutting mechanisms.

Method used

Using the fusion technology based on UWB and camera, the lawn images are collected through the drone equipped with a depth camera, image filtering, grass height calculation and area segmentation, mark the missing cutting area, and convert the coordinate information into the UWB coordinate system and send it to the cutting The mower carries out the mowing operation.

Benefits of technology

Automatic detection and precise re-cutting of lawn leaked areas is realized, which improves mowing efficiency and accuracy, can cover larger detection areas and reduces manual intervention.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090837A_ABST
    Figure CN119090837A_ABST
Patent Text Reader

Abstract

The invention discloses a mower cutting missing detection method and system based on UWB and camera fusion, and the method comprises the steps: 1, enabling an unmanned plane to carry a depth camera, and collecting a depth image of a target lawn region, the method comprises the following steps: step 1, carrying out initial image filtering processing, grass height calculation, image area segmentation and average height calculation on the basis of UWB coordinates of an unmanned aerial vehicle, and marking a cutting-missing area to obtain an overall cutting-missing area of a target lawn area, and step 2, for the overall cutting-missing area, obtaining coordinates of the cutting-missing area in a UWB coordinate system through coordinate conversion, and the information is sent to the mower to execute the recutting operation of the miss-cutting area. According to the invention, the problem that the mowing mode of the unmanned mower has miss-mowing can be solved, the mowing accuracy and efficiency are further improved, the unmanned aerial vehicle is used for accurately detecting the mowing area and marking the miss-mowing area, and the position of the marked area is transmitted to the mower in real time for supplementary mowing operation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of lawnmower detection, and in particular relates to a lawnmower missed mowing detection method and system based on UWB and camera fusion. Background Art

[0002] Traditional mowing methods require manual operation, which consumes a lot of time and manpower. With the development of technology, unmanned lawn mowers that can work autonomously and do not require human supervision have significantly improved mowing efficiency. This saves a lot of time and cost for managers of large lawns, gardens or public green spaces. In modern agriculture and gardening, precise lawn management is the key to improving efficiency and saving resources. In view of the problem of missed mowing in the mowing method of unmanned lawn mowers, in order to further improve the accuracy and efficiency of mowing, how to use drones to accurately detect the mowing area and mark the missed mowing area, and transmit the marked area location to the lawn mower in real time for re-mowing operations, is an urgent problem to be solved. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention provides a lawn mower missed mowing detection method based on UWB and camera fusion, which is based on computer vision, sensor data fusion, target detection and other technologies, and can automatically detect the missed mowing area of ​​the grass and convert it into coordinate information to be sent to the lawn mower to realize the missed mowing operation.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions: a lawn mower missed mowing detection method based on UWB and camera fusion, comprising the following steps:

[0005] S1. The drone is equipped with a depth camera to collect a depth image of the target lawn area. Then, based on the UWB coordinates of the drone, the initial image filtering process, grass height calculation, image area segmentation and average height calculation, and missed cutting area marking are performed to obtain the overall missed cutting area of ​​the target lawn area.

[0006] S2. For the entire missed-cut area, coordinates of the missed-cut area in the UWB coordinate system are obtained through coordinate conversion, and the coordinates are sent to the lawn mower to perform a re-cutting operation on the missed-cut area.

[0007] Furthermore, in the aforementioned step S1, the image filtering process is specifically: convolving the depth image D(x, y) with a Gaussian kernel function to smooth the image, and the Gaussian kernel function is as follows:

[0008]

[0009] Where σ is the standard deviation, which controls the smoothness of the filter, and the depth image D(x,y) is the depth value of the pixel (x,y), that is, the distance from the camera to the ground;

[0010] The filtered depth image D f (x,y) is:

[0011]

[0012] Where k is the radius of the filter.

[0013] Furthermore, in the aforementioned step S1, the grass height H(x, y) is calculated as follows:

[0014] H(x,y)=h d -D f (x,y)

[0015] Among them, h d is the altitude at which the drone is flying.

[0016] Furthermore, the aforementioned step S1 of segmenting the image area and calculating the average height includes the following sub-steps:

[0017] S1.1, segmentation area: the depth image is divided into M×N small blocks, each small block R ij The size of is w×h;

[0018] S1.2. For each small block, calculate its average height as follows:

[0019]

[0020] Furthermore, in the aforementioned step S1, marking the missed area is specifically: for each small block R ij If the average height cm, then record the four corner points of the small block (x ci ,y ci ,D f (x ci ,y ci )), where D(x ci ,y ci ) is the pixel (x ci ,y ci ), h is the height preset according to the grass depth requirement, i=1,2,3,4.

[0021] Furthermore, the aforementioned step S2 includes the following sub-steps:

[0022] S2.1. Determine the locations of the four UWB base stations: The locations of the base stations are expressed as (x i ,y i ,z i ), where i = 1, 2, 3, 4, one of the UWB 1 The base station is the coordinate origin (0,0,0);

[0023] S2.2. Convert the positions of other UWB base stations to UWB 1 The coordinate system with the origin as UWB 1 The position is (x 1 ,y 1 ,z 1 ), other UWB base stations relative to UWB 1 The relative position is:

[0024] (x i ′,y i ′,z i ′)=(x i -x 1 ,y i -y 1 ,z i -z 1 )

[0025] S2.3. Determine the position of the depth camera through the onboard UWB tag: Specifically, the UWB tag installed on the depth camera is used to obtain the camera's position information T through the UWB positioning system. c And direction information, that is, the rotation matrix R c ;

[0026] S2.4, converting the points in the depth camera coordinate system to the UWB base station coordinate system includes sub-steps S2.4.1 to S2.4.2: S2.4.1, converting the image pixel coordinates to the camera coordinates: The depth value corresponding to the pixel (u, v) in the image is D(u, v), according to the camera's intrinsic parameter matrix: focal length f x ,f y and the principal point (c x ,c y ) Convert pixel coordinates to camera coordinates (x c ,y c ,z c ):

[0027]

[0028] z c =D(u,v)

[0029] (x ci ,y ci ,D f (x ci ,y ci )) Substitute the above formula to get the position of the pixel in the image in the camera coordinate system: P camera =

[0030] (x c ,y c ,z c);S2.4.2、Convert the camera coordinates to the UWB coordinate system: According to the camera's position information: Position P camera and the rotation matrix R c , convert the point in the camera coordinate system to the point in the UWB coordinate system:

[0031] P UWB =P camera ·R c +T c

[0032] Among them, T c is the position of the camera in the UWB base station coordinate system;

[0033] S2.5. Obtain the UWB coordinates of the four corner points of the missed-cut area, and send the coordinates to the lawn mower to perform the supplementary cutting operation after the coordinates are determined.

[0034] The present invention also provides a lawn mower missed mowing detection system based on UWB and camera fusion, comprising a drone equipped with a UWB tag and a depth camera, a UWB base station located in a target lawn area, an industrial computer, and a lawn mower control module;

[0035] The drone is used to collect depth images of the target lawn area;

[0036] The UWB base station is used to establish a UWB coordinate system according to the UWB tag carried by the drone;

[0037] The industrial computer is used to perform initial image filtering processing, grass height calculation, image area segmentation and average height calculation, and mark missed cutting areas to obtain the overall missed cutting area of ​​the target lawn area for the depth image collected by the drone.

[0038] The lawn mower control module is used to obtain the coordinates of the missed-cut area in the UWB coordinate system through coordinate conversion for the entire missed-cut area, and send them to the lawn mower to perform a supplementary mowing operation on the missed-cut area.

[0039] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0040] The invention can cover a larger detection area by using aerial drones and ground mowers to work together. The drone can provide a wide-angle view and global information from the air. Through precise positioning and target detection, the invention can quickly convert the detected missed mowing area into specific coordinate information and instruct the mower to perform precise make-up mowing operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of a lawn mower missed mowing detection system that integrates UWB and camera.

[0042] Figure 2 It is a schematic diagram of a prototype of a lawn mower missed mowing detection system that integrates UWB and camera.

[0043] Figure 3 It is a schematic diagram of coordinate system transformation of the lawn mower missed mowing detection system that integrates UWB and camera.

[0044] Figure 4 This is a schematic diagram of the installation positions of the airborne UWB and camera of the lawn mower missed mowing detection system that integrates UWB and camera.

[0045] Figure 5 It is a flow chart of the algorithm for detecting missed-cut areas of a lawn mower missed-cut detection system that integrates UWB and camera.

[0046] Figure 6 It is a coordinate system conversion flow chart of the lawn mower missed mowing detection system that integrates UWB and camera. DETAILED DESCRIPTION

[0047] In order to better understand the technical content of the present invention, specific embodiments are given and described as follows in conjunction with the accompanying drawings.

[0048] Various aspects of the invention are described herein with reference to the accompanying drawings, in which many illustrative embodiments are shown. The embodiments of the invention are not limited to those described in the accompanying drawings. It should be understood that the invention is implemented by any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed in the invention are not limited to any implementation. In addition, some aspects disclosed in the invention may be used alone or in any appropriate combination with other aspects disclosed in the invention.

[0049] like Figure 1 System diagram and Figure 2 The prototype diagram of the present invention provides a lawn mower missed mowing detection system based on UWB and camera fusion, including a drone equipped with a UWB tag and a depth camera, a UWB base station located in a target lawn area, an industrial computer, and a lawn mower control module;

[0050] like Figure 4 As shown, the drone is equipped with a UWB tag and a depth camera installation position, and the depth camera is used to collect depth images of the target lawn area;

[0051] UWB base station, used to establish a UWB coordinate system based on the UWB tag carried by the drone;

[0052] The industrial computer is used to perform initial image filtering processing, grass height calculation, image area segmentation and average height calculation on the depth image collected by the drone for the target lawn area, and mark the missed cutting area to obtain the overall missed cutting area of ​​the target lawn area;

[0053] refer to Figure 6 The lawn mower control module is used to obtain the coordinates of the missed-cut area in the UWB coordinate system through coordinate conversion for the entire missed-cut area, and send them to the lawn mower to perform the missed-cut area supplementary mowing operation.

[0054] The present embodiment also provides a lawn mower missed cutting detection method based on UWB and camera fusion, comprising the following steps: S1, a drone equipped with a depth camera collects a depth image of a target lawn area, and then obtains the overall missed cutting area of ​​the target lawn area through initial image filtering processing, grass height calculation, image area segmentation and average height calculation, and marking the missed cutting area based on the UWB coordinates of the drone.

[0055] S2. For the entire missed-cut area, coordinates of the missed-cut area in the UWB coordinate system are obtained through coordinate conversion, and the coordinates are sent to the lawn mower to perform a re-cutting operation on the missed-cut area.

[0056] When it is implemented, the steps are as follows:

[0057] Step 1: Arrange 4 UWB base stations on the ground, and establish the UWB coordinate system with one UWB base station as the origin.

[0058] Step 2: Use a drone equipped with UWB and depth camera. The drone takes off above the lawn and obtains the grass environment data through the downward camera, and obtains the three-dimensional position information of the drone through the airborne UWB. During the flight, the drone flies at an altitude of h d The depth image obtained by the depth camera provided by other airborne sensors is D(x,y), where D(x,y) is the depth value of the pixel (x,y), that is, the distance from the camera to the ground.

[0059] Step 3: Transmit the three-dimensional position information obtained by the drone and the data obtained by the camera to the industrial computer;

[0060] Step 4: The industrial computer calculates the height information of the grass in the lawn through the camera data, and determines the rectangular area with four corner vertices marking the area with higher average height as the missed cutting area;

[0061] Step 5: The three-dimensional position information of the UAV obtained through UWB in step 1 is integrated with the four corner vertex information of the missed area in step 3, and the four corner coordinates of the missed area in the UWB coordinate system are obtained after the coordinate system is converted;

[0062] Step 6: Send the UWB coordinate information of the missed-cut area to the lawn mower to perform a make-up cut.

[0063] refer to Figure 5, Flowchart of the algorithm for detecting missed areas. The depth image may contain noise, such as noise caused by sensor errors or environmental factors. In order to improve the accuracy of the data, the initial image is Gaussian filtered, and the image is convolved with the Gaussian kernel function to smooth the image and reduce noise. The specific image filtering processing of the industrial computer is: the depth image D(x,y) is convolved with the Gaussian kernel function to smooth the image. The Gaussian kernel function is as follows:

[0064]

[0065] Where σ is the standard deviation, which controls the smoothness of the filter, and the depth image D(x,y) is the depth value of the pixel (x,y), that is, the distance from the camera to the ground;

[0066] The filtered depth image D f (x,y) is:

[0067]

[0068] Where k is the radius of the filter.

[0069] After filtering the initial depth image, the filtered depth map D is obtained. f (x,y). The drone is at height h d During flight, the depth camera acquires the depth data D f (x,y) is the depth value relative to the camera. The height of the grass H(x,y) is calculated by the difference in the measured data:

[0070] H(x,y)=h d -D f (x,y)

[0071] The image is then segmented into small regions and the average height of each region is calculated.

[0072] Segmenting the image region and calculating the average height includes the following sub-steps:

[0073] S1.1, segmentation area: the depth image is divided into M×N small blocks, each small block R ij The size of is w×h;

[0074] S1.2. For each small block, calculate its average height as follows:

[0075]

[0076] Marking the missed area is as follows: for each small block R ij If the average height cm, then record the four corner points of the small block (x ci ,y ci,D f (x ci ,y ci )), where D(x ci ,y ci ) is the pixel (x ci ,y ci ), h is the height preset according to the grass depth requirement, i=1,2,3,4.

[0077] refer to Figure 6 Coordinate system conversion flow chart: Step S2 includes the following sub-steps:

[0078] S2.1. Determine the locations of the four UWB base stations: The locations of the base stations are expressed as (x i ,y i ,z i ), where i = 1, 2, 3, 4, one of the UWB 1 The base station is the coordinate origin (0,0,0), see Figure 3 shown.

[0079] S2.2. Convert the positions of other UWB base stations to UWB 1 The coordinate system with the origin as UWB 1 The position is (x 1 ,y 1 ,z 1 ), other UWB base stations relative to UWB 1 The relative position is:

[0080] (x i ′,y i ′,z i ′)=(x i -x 1 ,y i -y 1 ,z i -z 1 )

[0081] S2.3. Determine the position of the depth camera through the onboard UWB tag: Specifically, the UWB tag installed on the depth camera is used to obtain the camera's position information T through the UWB positioning system. c And direction information, that is, the rotation matrix R c ;

[0082] S2.4, converting the points in the depth camera coordinate system to the UWB base station coordinate system includes sub-steps S2.4.1 to S2.4.2: S2.4.1, converting the image pixel coordinates to the camera coordinates: The depth value corresponding to the pixel (u, v) in the image is D(u, v), according to the camera's intrinsic parameter matrix: focal length f x ,f yand the principal point (c x ,c y ) Convert pixel coordinates to camera coordinates (x c ,y c ,z c ):

[0083]

[0084] z c =D(u,v)

[0085] (x ci ,y ci ,D f (x ci ,y ci )) Substitute the above formula to get the position of the pixel in the image in the camera coordinate system: P camera =

[0086] (x c ,y c ,z c );S2.4.2、Convert the camera coordinates to the UWB coordinate system: According to the camera's position information: Position P camera and the rotation matrix R c , convert the point in the camera coordinate system to the point in the UWB coordinate system:

[0087] P UWB =P camera ·R c +T c

[0088] Among them, T c is the position of the camera in the UWB base station coordinate system;

[0089] S2.5. Obtain the UWB coordinates of the four corner points of the missed-cut area, and send the coordinates to the lawn mower to perform the supplementary cutting operation after the coordinates are determined.

[0090] Although the present invention has been described above with preferred embodiments, it is not intended to limit the present invention. A person skilled in the art of the present invention may make various modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A lawn mower missed cutting detection method based on UWB and camera fusion, characterized in that: The following steps are involved: S1. The drone is equipped with a depth camera to collect a depth image of the target lawn area. Then, based on the UWB coordinates of the drone, the initial image filtering process, grass height calculation, image area segmentation and average height calculation, and missed cutting area marking are performed to obtain the overall missed cutting area of ​​the target lawn area. S2. For the entire missed-cut area, coordinates of the missed-cut area in the UWB coordinate system are obtained through coordinate conversion, and the coordinates are sent to the lawn mower to perform a re-cutting operation on the missed-cut area.

2. According to claim 1, a lawn mower missed cutting detection method based on UWB and camera fusion, characterized in that: In step S1, the image filtering process is specifically: convolving the depth image D(x, y) with a Gaussian kernel function to smooth the image. The Gaussian kernel function is as follows: Where σ is the standard deviation, which controls the smoothness of the filter, and the depth image D(x,y) is the depth value of the pixel (x,y), that is, the distance from the camera to the ground; The filtered depth image D f (x,y) is: Where k is the radius of the filter.

3. The lawn mower missed cutting detection method based on UWB and camera fusion according to claim 2, characterized in that: In step S1, the grass height H(x,y) is calculated as follows: H(x,y)=h d -D f (x,y) Among them, h d is the altitude at which the drone is flying.

4. The lawn mower missed cutting detection method based on UWB and camera fusion according to claim 1, characterized in that: Segmenting the image area and calculating the average height in step S1 includes the following sub-steps: S1.1, segmentation area: the depth image is divided into M×N small blocks, each small block R ij The size of is w×h; S1.

2. For each small block, calculate its average height as follows:

5. The lawn mower missed cutting detection method based on UWB and camera fusion according to claim 1, characterized in that: In step S1, the specific steps of marking the missed area are: for each small block R ij If the average height cm, then record the four corner points of the small block (x ci ,y ci ,D f (x ci ,y ci )), where D(x ci ,y ci ) is the pixel (x ci ,y ci ), h is the height preset according to the grass depth requirement, i=1,2,3,4.

6. The lawn mower missed cutting detection method based on UWB and camera fusion according to claim 1, characterized in that: Step S2 includes the following sub-steps: S2.

1. Determine the locations of the four UWB base stations: The locations of the base stations are expressed as (x i ,y i ,z i ), where i = 1, 2, 3, 4, and one of the UWB1 base stations is preset as the coordinate origin (0, 0, 0); S2.

2. Convert the positions of other UWB base stations to a coordinate system with UWB1 as the origin; the position of UWB1 is (x1, y1, z1), and the relative positions of other UWB base stations relative to UWB1 are: (x i ′,y i ′,z i ′)=(x i −x1,y i -y1,z i −z1) S2.

3. Determine the position of the depth camera through the onboard UWB tag: Specifically, the UWB tag installed on the depth camera is used to obtain the camera's position information T through the UWB positioning system. c And direction information, that is, the rotation matrix R c ; S2.4, converting the points in the depth camera coordinate system to the UWB base station coordinate system includes sub-steps S2.4.1 to S2.4.2: S2.4.

1. Convert image pixel coordinates to camera coordinates: The depth value corresponding to the pixel (u, v) in the image is D(u, v). According to the camera's intrinsic parameter matrix: focal length f x ,f y and the principal point (c x ,c y ) Convert pixel coordinates to camera coordinates (x c ,y c ,z c ): z c =D(u,v) (x ci ,y ci ,D f (x ci ,y ci )) Substitute the above formula to get the position of the pixel in the image in the camera coordinate system: P camera =(x c ,y c ,z c );S2.4.2、Convert the camera coordinates to the UWB coordinate system: According to the camera's position information: Position P camera and the rotation matrix R c , convert the point in the camera coordinate system to the point in the UWB coordinate system: P UWB =P camera ·R c +T c Among them, T c is the position of the camera in the UWB base station coordinate system; S2.

5. Obtain the UWB coordinates of the four corner points of the missed-cut area, and send the coordinates to the lawn mower to perform the supplementary cutting operation after the coordinates are determined.

7. A lawn mower missed mowing detection system based on UWB and camera fusion, characterized in that: It includes a drone equipped with a UWB tag and a depth camera, a UWB base station located in the target lawn area, an industrial computer, and a lawn mower control module; The drone is used to collect depth images of the target lawn area; The UWB base station is used to establish a UWB coordinate system according to the UWB tag carried by the drone; The industrial computer is used to perform initial image filtering processing, grass height calculation, image area segmentation and average height calculation, and mark missed cutting areas to obtain the overall missed cutting area of ​​the target lawn area for the depth image collected by the drone. The lawn mower control module is used to obtain the coordinates of the missed-cut area in the UWB coordinate system through coordinate conversion for the entire missed-cut area, and send them to the lawn mower to perform a supplementary mowing operation on the missed-cut area.