Three-dimensional perception of lawn trampling behavior detection methods, devices, media and robots

Abstract

This invention discloses a robot-based 3D perception method for detecting lawn trampling behavior, comprising: constructing a point cloud map of a preset scene, the point cloud map including a lawn area and a first 3D bounding box annotating the lawn area; acquiring 3D laser point cloud data frames; performing human detection on the 3D laser point cloud data frames using a preset human detection network to obtain pedestrians in the 3D laser point cloud data frames and a second 3D bounding box annotating the pedestrians; mapping the pedestrians and the second 3D bounding boxes in the 3D laser point cloud data frames onto the point cloud map of the preset scene to obtain a point cloud frame map; calculating the intersection-union ratio (IUGR) between the first and second 3D bounding boxes in the point cloud frame map; and generating lawn trampling behavior information based on the IUGR. This invention solves the problems of high false detection rate, low detection accuracy, and low robustness in existing technologies for detecting lawn trampling behavior.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a robot-based three-dimensional perception method, device, medium, and robot for detecting lawn trampling behavior. Background Technology

[0002] To protect the park environment and plant growth, park management typically designates certain three-dimensional areas, such as lawns, as no-walking zones, prohibiting pedestrians from walking or trampling on them. Currently, one method involves using fixed visible light cameras to monitor these zones. However, this method is costly, has limited coverage, and struggles to capture the entire scene. Another approach uses non-fixed visible light cameras, patrolling and capturing images of these zones, which addresses the issues of high cost and limited coverage. However, this method relies on two-dimensional images for three-dimensional spatial reasoning, making it prone to false detections, such as misidentifying trees or fallen leaves as lawns. Furthermore, it is limited by lighting conditions, only suitable for daytime with ample sunlight, resulting in low accuracy and poor robustness in detecting lawn trampling. Summary of the Invention

[0003] This invention provides a robot-based three-dimensional perception method, device, medium, and robot for detecting lawn trampling behavior, in order to solve the problems of high false detection rate, low detection accuracy, and low robustness in existing technologies for detecting lawn trampling behavior.

[0004] A robot-based three-dimensional perception method for detecting lawn trampling behavior, the method comprising:

[0005] Construct a point cloud map of a preset scene, the point cloud map including a lawn area and a first three-dimensional bounding box that marks the lawn area;

[0006] Acquire 3D laser point cloud data frames, which are collected by the robot through 3D laser radar when it patrols the preset scene along a designated route.

[0007] A preset human detection network is used to perform human detection on the 3D laser point cloud data frame to obtain pedestrians in the 3D laser point cloud data frame and a second three-dimensional bounding box that labels the pedestrians;

[0008] The pedestrians and the second three-dimensional bounding box in the 3D laser point cloud data frame are mapped onto the point cloud map of the preset scene to obtain the point cloud frame map.

[0009] Calculate the intersection-union ratio (IUGR) between the first and second 3D bounding boxes in the point cloud frame map, and generate lawn trampling behavior information based on the IUGR.

[0010] Optionally, the first three-dimensional bounding box for marking the lawn area is a cuboid bounding box with a height equal to the preset average human height.

[0011] Optionally, before performing human detection on the 3D laser point cloud data frame using a preset human detection network, the method further includes:

[0012] Construct a pre-defined point cloud object detection model PV-RCNN;

[0013] Obtain the original sample data, and perform downsampling processing on the original sample data to obtain training sample data that is consistent with the line count of the 3D LiDAR.

[0014] The training sample data is expanded.

[0015] The point cloud target detection model PV-RCNN was trained using the expanded training sample data to obtain the human detection network.

[0016] Optionally, constructing the preset point cloud object detection model PV-RCNN includes:

[0017] The point cloud detection range of the point cloud target detection model PV-RCNN is modified, wherein the detection range of the X-axis is [-10, 10] meters, the detection range of the Y-axis is [-10, 10] meters, and the detection range of the Z-axis is [-2, 2] meters;

[0018] The Sparse3DConv 3D sparse convolutional network structure based on voxel-based in the point cloud object detection model PV-RCNN is optimized into a 2DConv network structure based on point pillars, and the VGG-like network is modified into the Mobilenetv3 mobile network.

[0019] Optionally, mapping the pedestrians and the second 3D bounding box in the 3D laser point cloud data frame to the point cloud map of the preset scene to obtain the point cloud frame map includes:

[0020] Point cloud matching is performed on the 3D laser point cloud data frame to obtain the frame transformation matrix corresponding to the 3D laser point cloud data frame.

[0021] Obtain the 3D coordinate information of the vertices corresponding to the second 3D bounding box;

[0022] The vertex 3D coordinate information is mapped onto the point cloud map of the preset scene based on the frame transformation matrix to obtain the point cloud frame map corresponding to the 3D laser point cloud data frame.

[0023] Optionally, calculating the intersection-union ratio (IUGR) between the first and second 3D bounding boxes in the point cloud frame map, and generating lawn trampling behavior information based on the IUGR, includes:

[0024] Obtain the first 3D bounding box set and the second 3D bounding box set from the point cloud frame map;

[0025] Traverse the first set of 3D bounding boxes and the second set of 3D bounding boxes to construct bounding box pairs, each bounding box pair including a first 3D bounding box and a second 3D bounding box;

[0026] Traverse each bounding box pair, calculate the intersection-union ratio (CUP) of the bounding box pair, and obtain the CUP set.

[0027] Traverse the intersection-union ratio (CUNR) values ​​in the CUNR set and compare the CUNR values ​​with a preset CUNR threshold.

[0028] Obtain the cross-union ratio (CUN) value greater than the CUN threshold and its corresponding first and second 3D bounding boxes to generate lawn trampling behavior information corresponding to the 3D laser point cloud frame.

[0029] Optionally, after generating lawn trampling behavior information, the method further includes:

[0030] Acquire 3D laser point cloud data frames within a preset window, where one preset window contains several consecutive 3D laser point cloud data frames;

[0031] Acquire lawn trampling behavior information corresponding to each 3D laser point cloud data frame;

[0032] The number of frames in each lawn area where lawn trampling occurred is counted based on the lawn trampling behavior information.

[0033] If the number of frames showing grass trampling in a lawn area is greater than or equal to a preset frame threshold, then an early warning message will be generated for that lawn area and the early warning message will be reported.

[0034] A robot-based three-dimensional perception lawn trampling behavior detection device, the device comprising:

[0035] A construction module is used to construct a point cloud map of a preset scene, wherein the point cloud map includes a lawn area and a first three-dimensional bounding box that marks the lawn area;

[0036] The acquisition module is used to acquire 3D laser point cloud data frames, which are collected by the robot through 3D laser radar when it patrols the preset scene along a specified route.

[0037] The detection module is used to perform human detection on the 3D laser point cloud data frame using a preset human detection network, so as to obtain the pedestrian in the 3D laser point cloud data frame and the second three-dimensional bounding box that marks the pedestrian.

[0038] The mapping module is used to map the pedestrians and the second three-dimensional bounding box in the 3D laser point cloud data frame to the point cloud map of the preset scene to obtain the point cloud frame map.

[0039] The generation module is used to calculate the intersection-union ratio between the first 3D bounding box and the second 3D bounding box in the point cloud frame map, and generate lawn trampling behavior information based on the intersection-union ratio.

[0040] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the robot-based three-dimensional perception lawn trampling behavior detection method described above.

[0041] A robot includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the robot-based three-dimensional perception lawn trampling behavior detection method described above.

[0042] This invention constructs a point cloud map of a preset scene, including a lawn area and a first 3D bounding box marking the lawn area; acquires 3D laser point cloud data frames, which are collected by a robot patrolling the preset scene along a designated route using 3D LiDAR; uses a preset human detection network to perform human detection on the 3D laser point cloud data frames, obtaining pedestrians in the 3D laser point cloud data frames and a second 3D bounding box marking the pedestrians, thereby reducing camera costs and increasing detection coverage; maps the pedestrians and the second 3D bounding boxes in the 3D laser point cloud data frames onto the point cloud map of the preset scene, obtaining a point cloud frame map; calculates the intersection-union ratio (IUGR) between the first and second 3D bounding boxes in the point cloud frame map, and generates lawn trampling behavior information based on the IUGR; thereby improving the detection performance of lawn trampling behavior, greatly increasing the detection rate and recall rate, and solving the problems of high false detection rate, low detection accuracy, and low robustness in existing technologies for detecting lawn trampling behavior. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the implementation of a robot-based three-dimensional perception lawn trampling behavior detection method according to an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram of a robot inspection provided in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of a second enclosure box provided in an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of a robot-based three-dimensional perception lawn trampling behavior detection device provided in an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the internal structure of a robot according to one embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] The robot-based 3D perception lawn trampling behavior detection method provided in this invention dynamically analyzes pedestrians and their trampling behavior in each lawn area using a 3D point cloud map and robot inspection 3D perception. It does not depend on the location of the acquisition device or the environmental conditions of the scene, reducing camera costs and increasing detection coverage. Moreover, it greatly improves the detection rate and accuracy of lawn trampling behavior, solving the problems of high false detection rate, low detection accuracy and low robustness in existing technologies when detecting lawn trampling behavior.

[0051] The following is a detailed description of the robot-based 3D perception lawn trampling behavior detection method provided in this embodiment of the invention. This embodiment of the invention is based on robot inspection, with a 3D LiDAR mounted on the robot's head as the main data acquisition device. The robot continuously patrols back and forth on a road in a preset scene to detect abnormal phenomena such as pedestrians trampling on the lawn, i.e., lawn trampling behavior.

[0052] Figure 1 This invention provides a robot-based three-dimensional perception method for detecting lawn trampling behavior. Figure 1 As shown, the robot-based three-dimensional perception lawn trampling behavior detection method includes:

[0053] In step S101, a point cloud map of a preset scene is constructed, the point cloud map including a lawn area and a first three-dimensional bounding box that marks the lawn area.

[0054] The preset scenarios include, but are not limited to, parks, industrial parks, and other similar areas. This embodiment constructs a 3D point cloud map of the preset scenarios. A robot is controlled to move along a designated path within the preset scenarios. During this movement, point cloud data is collected using 3D LiDAR and uploaded to a server. The server then constructs a point cloud map based on the point cloud data using a preset matching and alignment algorithm. The designated path should cover the lawn area of ​​the preset scenarios and form a closed loop.

[0055] After obtaining the point cloud map, corresponding 3D bounding boxes are marked on the point cloud map according to the lawn area in the preset scene. For ease of description, this is referred to as the first 3D bounding box. The first 3D bounding box marked on the point cloud map corresponds one-to-one with the lawn area, and pedestrians are not allowed to step on or linger there.

[0056] Optionally, the first three-dimensional bounding box is a cuboid bounding box with a height equal to a preset average human height. As a preferred example of the invention, the first three-dimensional bounding box is 1.7 meters. For ease of understanding, Figure 2 This is a schematic diagram of a robot inspection provided in an embodiment of the present invention.

[0057] In step S102, a 3D laser point cloud data frame is acquired. The 3D laser point cloud data frame is collected by the robot through a 3D laser radar while it is patrolling the preset scene along a designated route.

[0058] After constructing the point cloud map of the preset scene, a patrol route is determined, traversing the lawn area included in the point cloud map to facilitate subsequent detection and analysis of lawn trampling behavior. The robot is controlled to patrol the preset scene along the patrol route. During the patrol, the 3D LiDAR on the robot collects 3D LiDAR point cloud data in real time, obtaining the corresponding 3D LiDAR point cloud data frame for each moment.

[0059] Lawn trampling behavior analysis is performed on the 3D laser point cloud data frames acquired by the 3D LiDAR. In one embodiment, each consecutive 3D laser point cloud data frame can be analyzed; in another embodiment, a frame skipping method can be used to select 3D laser point cloud data frames for analysis.

[0060] In step S103, a preset human detection network is used to perform human detection on the 3D laser point cloud data frame to obtain pedestrians in the 3D laser point cloud data frame and a second three-dimensional bounding box that labels the pedestrians.

[0061] After acquiring a 3D laser point cloud data frame at a certain moment, this embodiment uses a neural network to perform human body detection on the 3D laser point cloud data frame.

[0062] Optionally, this embodiment employs a high-precision point cloud object detection model (Point-VoxelFeature Set Abstraction for 3D Object Detection, abbreviated as PV-RCNN) from deep learning for human detection. Before performing human detection on the 3D laser point cloud data frame using a preset human detection network, the method further includes:

[0063] Construct a pre-defined point cloud object detection model PV-RCNN;

[0064] Obtain the original sample data, and perform downsampling processing on the original sample data to obtain training sample data that is consistent with the line count of the 3D LiDAR.

[0065] The training sample data is expanded.

[0066] The point cloud target detection model PV-RCNN was trained using the expanded training sample data to obtain the human detection network.

[0067] In constructing the point cloud object detection model PV-RCNN, this embodiment optimizes the original PV-RCNN model to enable real-time operation on GPU hardware platforms. Optimizations include, but are not limited to, optimization of the detection range, PointPillar feature optimization, and network structure optimization.

[0068] Optionally, constructing the preset point cloud object detection model PV-RCNN includes:

[0069] The point cloud detection range of the point cloud target detection model PV-RCNN is modified, wherein the detection range of the X-axis is [-10, 10] meters, the detection range of the Y-axis is [-10, 10] meters, and the detection range of the Z-axis is [-2, 2] meters;

[0070] The Sparse3DConv 3D sparse convolutional network structure based on voxel-based architecture in the point cloud object detection model PV-RCNN is optimized into a PointPillar 2D Conv network structure, and the VGG-like network is modified into the Mobilenetv3 mobile network. Here, VGG is an abbreviation for Visual Geometry Group.

[0071] Here, this embodiment limits the point cloud detection range to: the horizontal axis X range is between -10 meters and 10 meters, the vertical axis Y range is between -10 meters and 10 meters, and the vertical axis Z range is between -2 meters and 2 meters. The Voxel-Based Sparse3DConv network structure in the point cloud object detection model PV-RCNN is optimized to a PointPillar 2DConv network structure to improve the inference speed of the neural network. Simultaneously, regarding the network structure, this embodiment modifies the original VGG-like network to a Mobilenetv3 network.

[0072] The original sample data includes open-source data and self-collected labeled data. The self-collected labeled data is 16-line 3D laser point cloud data acquired by 3D LiDAR. The open-source data is usually 64-line or 128-line data. Therefore, it is necessary to perform downsampling processing on the open-source data to obtain training sample data with the same line count as the 3D LiDAR, i.e., 16-line open-source data.

[0073] The expansion process includes, but is not limited to, translation, rotation, scaling, and random copy-pasting. This embodiment further expands the training sample data by translating, rotating, scaling, or randomly copying and pasting the 16-line open-source data and self-collected labeled data.

[0074] The expanded training sample data is input into the point cloud object detection model PV-RCNN for network training, generating a human detection network. This human detection network is used to detect humans in the input 3D laser point cloud data frames, obtaining each pedestrian and its corresponding 3D bounding box in the 3D laser point cloud data frame. To distinguish it from the first 3D bounding box labeled the lawn area, it is denoted here as the second 3D bounding box. Optionally, Figure 3 This is a schematic diagram of the second enclosure box provided in an embodiment of the present invention.

[0075] In step S104, the pedestrians and the second three-dimensional bounding box in the 3D laser point cloud data frame are mapped onto the point cloud map of the preset scene to obtain the point cloud frame map.

[0076] After obtaining the pedestrians and the second 3D bounding box in the 3D laser point cloud data frame, this embodiment transforms the coordinates of the second 3D bounding box used to annotate the pedestrians to the point cloud map of the preset scene based on the coordinate transformation matrix of the current frame, generating a point cloud frame map. Optionally, as a preferred example of the present invention, the step S104 of mapping the pedestrians and the second 3D bounding box in the 3D laser point cloud data frame to the point cloud map of the preset scene to obtain the point cloud frame map includes:

[0077] In step S1041, point cloud matching is performed based on the 3D laser point cloud data frame to obtain the frame transformation matrix corresponding to the 3D laser point cloud data frame.

[0078] Here, because the robot is constantly moving and collecting 3D laser point cloud data frames, the transformation matrix corresponding to the point cloud map of the preset scene is different for each 3D laser point cloud data frame. This embodiment calculates the transformation matrix of the current 3D laser point cloud data frame through point cloud matching. Each 3D laser point cloud data frame corresponds to one frame transformation matrix. Different 3D laser point cloud data frames correspond to different frame transformation matrices.

[0079] In step S1042, the vertex 3D coordinate information corresponding to the second 3D bounding box is obtained.

[0080] The second three-dimensional bounding box for annotating the human body is a three-dimensional cuboid, and therefore each second three-dimensional bounding box includes 8 vertices and their corresponding three-dimensional coordinate information.

[0081] In step S1043, the vertex 3D coordinate information is mapped to the point cloud map of the preset scene according to the frame conversion matrix to obtain the point cloud frame map corresponding to the 3D laser point cloud data frame.

[0082] This embodiment converts the 3D coordinate information of the vertices corresponding to the second 3D bounding box to the point cloud map, thereby mapping the second 3D bounding box to the point cloud map of the preset scene. During the conversion process, the 3D coordinate information of the vertices is multiplied by the frame conversion matrix using matrix multiplication, and the resulting 3D coordinate information is then labeled on the point cloud map. By traversing the 3D coordinate information of each vertex corresponding to the second 3D bounding box, the mapping of the second 3D bounding box is achieved, resulting in a point cloud frame map.

[0083] It should be understood that the point cloud frame map corresponds one-to-one with the 3D laser point cloud data frame, which integrates a first three-dimensional bounding box that marks the lawn area and a second three-dimensional bounding box that marks pedestrians from the 3D laser point cloud data frame.

[0084] In step S105, the intersection-union ratio (IUU) between the first 3D bounding box and the second 3D bounding box in the point cloud frame map is calculated, and lawn trampling behavior information is generated based on the IUU.

[0085] After converting the second 3D bounding box in the 3D laser point cloud data frame to the point cloud map of the preset scene using a frame transformation matrix, this embodiment calculates the 3D intersection-over-union (IOU) ratio based on the first and second 3D bounding boxes to determine whether pedestrians are trampling on the grass area. Optionally, as a preferred example of the present invention, step S105, which calculates the IOU ratio between the first and second 3D bounding boxes in the point cloud frame map and generates grass trampling behavior information based on the IOU ratio, includes:

[0086] In step S1051, the first 3D bounding box set and the second 3D bounding box set in the point cloud frame map are obtained.

[0087] In this embodiment, the first set of 3D bounding boxes is denoted as box_grass_map, and includes at least one first 3D bounding box; the second set of 3D bounding boxes is denoted as boss_person_map, and includes at least one second 3D bounding box. Both the first and second 3D bounding boxes are cuboid in shape.

[0088] In step S1052, the first set of three-dimensional bounding boxes and the second set of three-dimensional bounding boxes are traversed to construct bounding box pairs, each of which includes a first three-dimensional bounding box and a second three-dimensional bounding box.

[0089] In this embodiment, a bounding box pair is formed by randomly selecting one 3D bounding box from the first 3D bounding box set `box_grass_map` and the second 3D bounding box set `bos_person_map`. Specifically, a first 3D bounding box is selected from the first 3D bounding box set `box_grass_map`, and a second 3D bounding box is randomly selected from the second 3D bounding box set `bos_person_map`. The combination of the first 3D bounding box and the second 3D bounding box in each bounding box pair is unique.

[0090] In step S1053, each bounding box pair is traversed, and the intersection-union ratio (CUI) value corresponding to the bounding box pair is calculated to obtain the CUI set.

[0091] In this embodiment, the Intersection over Union (IOU) value corresponding to each bounding box pair is calculated based on the first 3D bounding box and the second 3D bounding box contained in each bounding box pair. By traversing each set of bounding box pairs, the IOU value corresponding to each bounding box pair is obtained, thereby obtaining the IOU set corresponding to the point cloud frame map.

[0092] Optionally, the calculation formula for the intersection-union ratio (IOU) is: IOU = (A∩B) / (A∪B) = (A∩B) / (A+B)-(A∩B), where A represents the first three-dimensional bounding box in the bounding box pair, and B represents the second three-dimensional bounding box in the bounding box pair.

[0093] In step S1054, the cross-union ratio values ​​in the cross-union ratio set are traversed, and the cross-union ratio values ​​are compared with a preset cross-union ratio threshold.

[0094] The intersection-union ratio (CUNR) threshold is preset and serves as a criterion for determining whether grass trampling behavior exists in the first and second 3D bounding boxes of the bounding box pair. For each CUNR value in the CUNR set, the CUNR value is compared with the preset CUNR threshold.

[0095] In step S1055, an intersection-union ratio (IU) value greater than the IU threshold and its corresponding first and second 3D bounding boxes are obtained to generate lawn trampling behavior information corresponding to the 3D laser point cloud frame.

[0096] If the intersection-union ratio (IU) is greater than the IU threshold, it indicates that the overlap between the first 3D bounding box and the second 3D bounding box in the bounding box pair is very high. Therefore, it is considered that pedestrians have trampled on the grass in the lawn area marked by the first 3D bounding box, and the lawn area marked by the first 3D bounding box is obtained as lawn trampling behavior information in the point cloud frame map. As a preferred example of the present invention, the IU threshold is 0.9.

[0097] Iterate through each intersection-union ratio (IU) value in the IU set and compare it with the IU threshold to obtain all grass areas in the point cloud frame map where pedestrians have trampled, and generate grass trampling behavior information corresponding to the 3D laser point cloud frame at the current moment.

[0098] In summary, this invention employs a robotic inspection method to detect pedestrian and lawn trampling behavior within a preset scene. This eliminates reliance on the location of the data acquisition device and the environmental conditions of the scene, reducing camera costs and increasing detection coverage. By combining point cloud maps with 3D laser point cloud data frames, and fusing the first 3D bounding box marking the lawn area with the second 3D bounding box marking pedestrians, a comprehensive judgment is made on whether pedestrian trampling behavior has occurred. This significantly improves the detection performance of lawn trampling behavior, increasing detection rate and recall rate. It is independent of the scene environment and applicable to various scenarios and lighting conditions, effectively solving the problems of high false detection rate, low detection accuracy, and poor robustness in existing technologies for detecting lawn trampling behavior.

[0099] Optionally, to reduce the impact of environmental influences and network performance on the lawn trampling behavior judgment results, a multi-frame judgment strategy is adopted in another embodiment of the present invention. After generating lawn trampling behavior information in step S105, the method further includes:

[0100] Acquire 3D laser point cloud data frames within a preset window, where one preset window contains several consecutive 3D laser point cloud data frames;

[0101] Acquire lawn trampling behavior information corresponding to each 3D laser point cloud data frame;

[0102] The number of frames in each lawn area where lawn trampling occurred is counted based on the lawn trampling behavior information.

[0103] If the number of frames showing grass trampling in a lawn area is greater than or equal to a preset frame threshold, then an early warning message will be generated for that lawn area and the early warning message will be reported.

[0104] Optionally, as a preferred example of the present invention, the preset window includes 30 consecutive 3D laser point cloud data frames, and the preset frame count threshold is preferably 20 frames. This embodiment comprehensively judges several consecutive 3D laser point cloud data frames within the preset window. If the number of frames showing grass trampling behavior in a lawn area within the preset window reaches 20 or more, it can be confirmed with a high probability that grass trampling behavior has occurred in the lawn area. An early warning message is then generated for the lawn area and reported.

[0105] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0106] In one embodiment, the present invention also provides a robot-based three-dimensional perception lawn trampling detection device, which corresponds one-to-one with the robot-based lawn trampling detection method described in the above embodiments. For example... Figure 4 As shown, the robot-based 3D perception lawn trampling detection device includes a construction module 41, an acquisition module 42, a detection module 43, a mapping module 44, and a generation module 45. Detailed descriptions of each functional module are as follows:

[0107] Module 41 is used to construct a point cloud map of a preset scene, wherein the point cloud map includes a lawn area and a first three-dimensional bounding box that marks the lawn area.

[0108] The acquisition module 42 is used to acquire 3D laser point cloud data frames, which are collected by the robot through 3D laser radar when it patrols the preset scene along a specified route.

[0109] The detection module 43 is used to perform human detection on the 3D laser point cloud data frame using a preset human detection network, and to obtain the pedestrian in the 3D laser point cloud data frame and the second three-dimensional bounding box that marks the pedestrian.

[0110] The mapping module 44 is used to map the pedestrians and the second three-dimensional bounding box in the 3D laser point cloud data frame to the point cloud map of the preset scene to obtain the point cloud frame map.

[0111] The generation module 45 is used to calculate the intersection-union ratio between the first 3D bounding box and the second 3D bounding box in the point cloud frame map, and generate lawn trampling behavior information based on the intersection-union ratio.

[0112] Optionally, the first three-dimensional bounding box for marking the lawn area is a cuboid bounding box with a height equal to the preset average human height.

[0113] Optionally, before performing human detection on the 3D laser point cloud data frame using a preset human detection network, the device further includes a training module, which is used for:

[0114] The building unit is used to build a pre-defined point cloud object detection model, PV-RCNN.

[0115] A downsampling unit is used to acquire original sample data and perform downsampling processing on the original sample data to obtain training sample data that is consistent with the line count of the 3D LiDAR.

[0116] An expansion unit is used to expand the training sample data.

[0117] The training unit is used to train the point cloud target detection model PV-RCNN using the expanded training sample data to obtain the human detection network.

[0118] Optionally, the building unit is used for:

[0119] The point cloud detection range of the point cloud target detection model PV-RCNN is modified, wherein the detection range of the X-axis is [-10, 10] meters, the detection range of the Y-axis is [-10, 10] meters, and the detection range of the Z-axis is [-2, 2] meters;

[0120] The Sparse3DConv 3D sparse convolutional network structure based on voxel-based in the point cloud object detection model PV-RCNN is optimized into a 2DConv network structure based on point pillars, and the VGG-like network is modified into the Mobilenetv3 mobile network.

[0121] Optionally, the mapping module 44 includes:

[0122] A matching unit is used to perform point cloud matching based on the 3D laser point cloud data frame to obtain the frame transformation matrix corresponding to the 3D laser point cloud data frame.

[0123] The vertex acquisition unit is used to acquire the 3D coordinate information of the vertices corresponding to the second 3D bounding box;

[0124] The mapping unit is used to map the vertex 3D coordinate information onto the point cloud map of the preset scene according to the frame transformation matrix, so as to obtain the point cloud frame map corresponding to the 3D laser point cloud data frame.

[0125] Optionally, the generation module 45 includes:

[0126] The set acquisition unit is used to acquire the first 3D bounding box set and the second 3D bounding box set in the point cloud frame map;

[0127] The bounding box pair acquisition building unit is used to traverse the first set of 3D bounding boxes and the second set of 3D bounding boxes to construct bounding box pairs. Each bounding box pair includes a first 3D bounding box and a second 3D bounding box.

[0128] The intersection-union ratio (CUNR) calculation unit is used to traverse each bounding box pair, calculate the CUNR value corresponding to the bounding box pair, and obtain the CUNR set.

[0129] The comparison unit is used to traverse the intersection-union ratio values ​​in the intersection-union ratio set and compare the intersection-union ratio values ​​with a preset intersection-union ratio threshold.

[0130] The generation unit is used to obtain the cross-union ratio value that is greater than the cross-union ratio threshold and its corresponding first three-dimensional bounding box and second three-dimensional bounding box, and generate the lawn trampling behavior information corresponding to the 3D laser point cloud frame.

[0131] Optionally, after generating information on lawn trampling behavior, the device further includes an early warning module, which is used to:

[0132] Acquire 3D laser point cloud data frames within a preset window, where one preset window contains several consecutive 3D laser point cloud data frames;

[0133] Acquire lawn trampling behavior information corresponding to each 3D laser point cloud data frame;

[0134] The number of frames in each lawn area where lawn trampling occurred is counted based on the lawn trampling behavior information.

[0135] If the number of frames showing grass trampling in a lawn area is greater than or equal to a preset frame threshold, then an early warning message will be generated for that lawn area and the early warning message will be reported.

[0136] Specific limitations regarding the robot-based 3D perception lawn trampling detection device can be found in the limitations of the robot-based lawn trampling detection method described above, and will not be repeated here. Each module in the aforementioned robot-based lawn trampling detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in memory as software, so that the processor can call and execute the corresponding operations of each module.

[0137] In one embodiment, a robot is provided whose internal structure diagram can be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The robot's processor provides computational and control capabilities. The robot's memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The robot's network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a robot-based three-dimensional perception method for detecting lawn trampling.

[0138] In one embodiment, a robot is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0139] Construct a point cloud map of a preset scene, the point cloud map including a lawn area and a first three-dimensional bounding box that marks the lawn area;

[0140] Acquire 3D laser point cloud data frames, which are collected by the robot through 3D laser radar when it patrols the preset scene along a designated route.

[0141] A preset human detection network is used to perform human detection on the 3D laser point cloud data frame to obtain pedestrians in the 3D laser point cloud data frame and a second three-dimensional bounding box that labels the pedestrians;

[0142] The pedestrians and the second three-dimensional bounding box in the 3D laser point cloud data frame are mapped onto the point cloud map of the preset scene to obtain the point cloud frame map.

[0143] Calculate the intersection-union ratio (IUGR) between the first and second 3D bounding boxes in the point cloud frame map, and generate lawn trampling behavior information based on the IUGR.

[0144] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0145] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0146] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting a three-dimensional perception of lawn trampling behavior based on a robot, characterized by, The method comprises: constructing a point cloud map of a preset scene, the point cloud map comprising a lawn area and a first three-dimensional bounding box labeling the lawn area; acquiring a 3D laser point cloud data frame, the 3D laser point cloud data frame being acquired by a 3D laser radar when a robot patrols along a specified route in the preset scene; performing human body detection on the 3D laser point cloud data frame using a preset human body detection network to obtain pedestrians in the 3D laser point cloud data frame and a second three-dimensional bounding box labeling the pedestrians; mapping the pedestrians and the second three-dimensional bounding box in the 3D laser point cloud data frame to the point cloud map of the preset scene to obtain a point cloud frame map; calculating an intersection-over-union value between the first three-dimensional bounding box and the second three-dimensional bounding box in the point cloud frame map, and generating lawn stepping behavior information according to the intersection-over-union value; the first three-dimensional bounding box labeling the lawn area is a cuboid bounding box, and the height is a preset average human height; the mapping of the pedestrians and the second three-dimensional bounding box in the 3D laser point cloud data frame to the point cloud map of the preset scene to obtain a point cloud frame map comprises: performing point cloud matching according to the 3D laser point cloud data frame to obtain a frame conversion matrix corresponding to the 3D laser point cloud data frame; acquiring vertex three-dimensional coordinate information corresponding to the second three-dimensional bounding box; mapping the vertex three-dimensional coordinate information to the point cloud map of the preset scene according to the frame conversion matrix to obtain a point cloud frame map corresponding to the 3D laser point cloud data frame; before performing human body detection on the 3D laser point cloud data frame using a preset human body detection network, the method further comprises: constructing a preset point cloud target detection model PV-RCNN; acquiring original sample data, performing down-sampling processing on the original sample data to obtain training sample data consistent with the number of lines of the 3D laser radar; performing capacity expansion processing on the training sample data; training the point cloud target detection model PV-RCNN using the training sample data after capacity expansion processing to obtain a human body detection network; the construction of the preset point cloud target detection model PV-RCNN comprises: modifying the point cloud detection range of the point cloud target detection model PV-RCNN, wherein the detection range of the X axis is [-10, 10] meters, the detection range of the Y axis is [-10, 10] meters, and the detection range of the Z axis is [-2, 2] meters; optimizing the Voxel-Base-based 3D sparse convolution network structure Sparse3DConv in the point cloud target detection model PV-RCNN to a PointPillar-based 2DConv network structure, and modifying the network in a VGG form to a Mobilenetv3 mobile network; the calculation of the intersection-over-union value between the first three-dimensional bounding box and the second three-dimensional bounding box in the point cloud frame map, and the generation of lawn stepping behavior information according to the intersection-over-union value comprises: acquiring a first three-dimensional bounding box set and a second three-dimensional bounding box set in the point cloud frame map; traversing the first and second bounding box sets to construct bounding box pairs, each bounding box pair including a first bounding box and a second bounding box; traversing each bounding box pair to calculate an intersection over union value corresponding to the bounding box pair to obtain an intersection over union value set; traversing the intersection over union value set to compare the intersection over union value with a preset intersection over union threshold value; obtaining an intersection over union value greater than the intersection over union threshold value and the first and second bounding boxes corresponding to the intersection over union value to generate lawn stepping behavior information corresponding to the 3D laser point cloud frame.

2. The robot-based three-dimensional perception lawn-treading behavior detection method according to claim 1, wherein, After generating the lawn stepping behavior information, the method further includes: obtaining 3D laser point cloud data frames in a preset window, wherein a preset window includes a plurality of continuous 3D laser point cloud data frames; obtaining lawn stepping behavior information corresponding to each 3D laser point cloud data frame; counting the number of frames in which a lawn region appears lawn stepping behavior according to the lawn stepping behavior information; if the number of frames in which a lawn region appears lawn stepping behavior is greater than or equal to a preset frame threshold, generating a warning information for the lawn region and reporting the warning information.

3. A robot-based three-dimensional perception turf-treading detection device applied to the turf-treading behavior detection method of claims 1-2, characterized in that, The device includes: a construction module configured to construct a point cloud map of a preset scene, the point cloud map including lawn regions and first bounding boxes labeling the lawn regions; an acquisition module configured to acquire 3D laser point cloud data frames collected by a 3D laser radar when a robot patrols a designated route in the preset scene; a detection module configured to perform human body detection on the 3D laser point cloud data frames using a preset human body detection network to obtain pedestrians in the 3D laser point cloud data frames and second bounding boxes labeling the pedestrians; a mapping module configured to map the pedestrians and the second bounding boxes in the 3D laser point cloud data frames to the point cloud map of the preset scene to obtain a point cloud frame map; a generation module configured to calculate intersection over union values between the first and second bounding boxes in the point cloud frame map and generate lawn stepping behavior information according to the intersection over union values.

4. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 3. The computer program, when executed by a processor, implements the robot-based three-dimensional perception lawn stepping detection method of any one of claims 1 to 2.

5. A robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor, when executing the computer program, implements the robot-based three-dimensional perception lawn stepping detection method of any one of claims 1 to 2.

Images