Methods, equipment, moving devices and storage media for detecting obstacle motion status

Abstract

This invention provides a method, device, mobile device, and storage medium for detecting the motion state of obstacles. The method includes: projecting a 3D point cloud of a scene observed by a lidar onto a ground plane and rasterizing it into a 2D grid map; constructing a grid map of the scene using multiple consecutive frames of the 2D grid map; shrinking empty areas based on distance errors to create gaps between occupied grids adjacent to the shrunken empty areas and the shrunken empty areas; projecting the 3D point cloud of the scene in the current frame observed by the lidar onto the grid map, and determining the motion state corresponding to the 3D point cloud based on its location in the grid map; and detecting the motion state of obstacles based on the motion state of the 3D point cloud. This invention eliminates the need for point cloud clustering to obtain the motion state attributes of the point cloud, avoiding target mismatches caused by occlusion and clustering errors, and does not require specific target speed.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and more particularly to a method, device, mobile device, and storage medium for detecting the motion state of obstacles. Background Technology

[0002] With the rapid development of the service industry, various types of service robots (such as cleaning robots and warehouse robots) have emerged. These service robots aim to increase work efficiency and reduce the workload of human workers. Robots not only need to accurately detect obstacles in the environment but also need to determine the motion state of these targets in order to plan their routes more efficiently and perform tasks better. Therefore, motion state estimation is a crucial step in environmental perception. Due to the movement of the robot, the distance between point clouds of different targets is often very close, and there may be mutual occlusion and sudden appearances. Therefore, cluster-based target motion estimation is often inaccurate. How to accurately determine the motion state of point clouds has always been a problem that the industry is constantly exploring and urgently needs to solve.

[0003] Commonly used sensors for acquiring point clouds in service robots include LiDAR and RGB-D cameras. The method for environmental perception using point clouds involves first detecting and clustering targets within the point cloud, and then estimating the motion state of the targets. There are generally two methods for target detection in point clouds: one is to directly obtain targets from the point cloud through point cloud segmentation and clustering; the other is to use image detection algorithms in the RGB images from the RGB camera to obtain targets, and then map the targets to the point cloud based on pixel position relationships. However, regardless of the method, inaccurate point cloud clustering, occlusion, or different observation positions can lead to significant differences in the point cloud shape of the same target across multiple frames.

[0004] Based on target detection, a common method for target tracking and velocity estimation is as follows: For the point cloud target in the previous frame, its historical motion state is combined with methods such as Kalman filtering to predict its position in the current frame. Then, methods such as the Hungarian matching algorithm or the KM algorithm are used to match the predicted target position with the target position in the current frame. For the matched targets, the observed and predicted positions in the current frame are combined with weighted values ​​using methods such as Kalman filtering to finally obtain the output position of the current frame, thus completing target tracking. By fitting the trajectory of the target's center point across the output positions of multiple frames, information such as the target's motion direction and velocity can be obtained. Summary of the Invention

[0005] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:

[0006] Existing techniques require higher accuracy and consistency for target detection. Since target tracking is achieved through matching, a match can only be made if the same target is accurately detected in both frames. However, in reality, service robots operate in densely populated and chaotic environments such as indoor spaces and commercial areas. In these scenarios, the close proximity of targets makes point cloud clustering prone to errors. When multiple objects are close together, it's easy for the same target to be segmented into two or more targets in one frame, only to merge into a larger target in another. In such cases, the targets in consecutive frames are not the same thing, making a logical match impossible. Even if a forced match is made, motion estimation will naturally be incorrect. Furthermore, due to occlusion or different observation positions, the point cloud shape of the same target can vary significantly across multiple frames. In such situations, motion estimation will be highly inaccurate.

[0007] Current methods for determining whether a target is a moving object typically rely on the amount of time the target has moved. This method is prone to errors when dealing with objects that move at relatively slow speeds.

[0008] In order to at least address the problem in existing technologies that, in environments with dense and chaotic targets, multiple targets are difficult to cluster, making it impossible to accurately determine the motion state of the targets, and that if the target's motion speed is slow, the motion state is also easily misjudged.

[0009] In a first aspect, embodiments of the present invention provide an obstacle motion state detection method, applied to a mobile device equipped with a lidar, comprising:

[0010] The three-dimensional point cloud of the scene observed by the lidar is projected onto the ground plane and rasterized into a two-dimensional raster map.

[0011] A grid map of the scene is constructed using a series of two-dimensional grid images. The grid map includes: occupied grids containing obstacles and empty areas outside the occupied grids.

[0012] The free area is shrunk based on a predetermined distance error so that there is a gap between the occupied grid adjacent to the free area before shrinkage and the free area after shrinkage.

[0013] The three-dimensional point cloud of the scene observed by the lidar in the current frame is projected onto the grid map, and the motion state corresponding to the three-dimensional point cloud is determined according to the position area of ​​the three-dimensional point cloud in the grid map.

[0014] The motion state of the obstacle is detected based on the motion state corresponding to the three-dimensional point cloud.

[0015] In a second aspect, embodiments of the present invention provide an obstacle motion state detection and execution device, comprising:

[0016] The rasterization module is used to project the 3D point cloud of the scene observed by the lidar onto the ground plane and rasterize it into a 2D raster map.

[0017] A grid map construction module is used to construct a grid map of the scene using continuous multi-frame two-dimensional grid maps, wherein the grid map includes: occupied grids with obstacles and empty areas outside the occupied grids;

[0018] The region shrinking module is used to shrink the free area based on a predetermined distance error, so that there is a gap between the occupied grid adjacent to the free area before shrinking and the free area after shrinking.

[0019] The dynamic and static determination module is used to project the three-dimensional point cloud of the scene observed by the lidar in the current frame onto the grid map, and determine the motion state corresponding to the three-dimensional point cloud based on the position area of ​​the three-dimensional point cloud in the grid map.

[0020] The motion state detection module is used to detect the motion state of the obstacle based on the motion state corresponding to the three-dimensional point cloud.

[0021] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the obstacle motion state detection method according to any embodiment of the present invention.

[0022] Fourthly, embodiments of the present invention provide a mobile device, including a main body and an electronic device according to any embodiment of the present invention mounted on the main body.

[0023] Fifthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the obstacle motion state detection method of any embodiment of the present invention.

[0024] Sixthly, embodiments of the present invention also provide a computer program product, which, when run on a computer, causes the computer to execute the obstacle motion state detection method described in any one of the embodiments of the present invention.

[0025] The beneficial effects of this invention are as follows: It obtains the motion state attributes of the point cloud without requiring point cloud clustering, thus avoiding incorrect target matching due to occlusion and clustering errors, and also avoiding incorrect dynamic / static judgments caused by unstable target detection. Furthermore, this method does not determine the target's dynamic / static status by explicitly comparing changes in the target before and after a fixed time period, therefore it does not require a specific target speed. This avoids incorrect judgments caused by failing to track high-speed moving objects, as well as incorrect judgments caused by low-speed moving objects exhibiting minimal changes over a period of time. Since this method fully considers the sensor's occlusion model, it will not produce incorrect judgments regarding changes in the target point cloud shape caused by occlusion. Because there are no target point cloud detection, clustering, and tracking steps, the computational complexity is very low. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of an obstacle motion state detection method provided in an embodiment of the present invention;

[0028] Figure 2 This is a two-dimensional grid schematic diagram of an obstacle motion state detection method provided in an embodiment of the present invention;

[0029] Figure 3 This is a grid map of an obstacle motion state detection method provided in an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of an obstacle motion state detection and execution device according to an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of an embodiment of an electronic device for detecting the motion state of an obstacle, provided by an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Those skilled in the art will recognize that embodiments of this application can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0034] For ease of understanding, the technical terms used in this application are explained below:

[0035] The term "mobile device" as used in this application includes, but is not limited to, vehicles with L0-L5 autonomous driving technology levels as defined by the Society of Automotive Engineers International (SAE International) or the Chinese national standard "Classification of Driving Automation for Automobiles".

[0036] In some embodiments, the mobile device may be a vehicle or robot having various functions, such as:

[0037] (1) Passenger transport function, such as family cars, buses, etc.;

[0038] (2) Cargo carrying function, such as ordinary trucks, box trucks, trailer trucks, enclosed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, special structure trucks, etc.;

[0039] (3) Tool functions, such as logistics delivery vehicles, automated guided vehicles (AGVs), patrol vehicles, cranes, hoists, excavators, bulldozers, loaders, road rollers, loaders, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor scrubbers, water sprinkler trucks, sweeping robots, food delivery robots, shopping guide robots, lawnmowers, golf carts, etc.

[0040] (4) Entertainment functions, such as recreational vehicles, amusement park automatic driving devices, balance bikes, etc.;

[0041] (5) Special rescue functions, such as fire trucks, ambulances, power repair vehicles, and engineering emergency rescue vehicles.

[0042] like Figure 1 The diagram shown is a flowchart of an obstacle motion state detection method according to an embodiment of the present invention, including the following steps:

[0043] S11: Project the 3D point cloud of the scene observed by the lidar onto the ground plane and rasterize it into a 2D raster map;

[0044] S12: Construct a grid map of the scene using continuous multi-frame two-dimensional grid images, wherein the grid map includes: occupied grids with obstacles and empty areas outside the occupied grids;

[0045] S13: Shrink the free area based on a predetermined distance error so that there is a gap between the occupied grid adjacent to the free area before shrinking and the free area after shrinking;

[0046] S14: Project the 3D point cloud of the scene observed by the lidar in the current frame onto the grid map, and determine the motion state corresponding to the 3D point cloud based on the location area of ​​the 3D point cloud in the grid map;

[0047] S15: Detect the motion state of the obstacle based on the motion state corresponding to the three-dimensional point cloud.

[0048] In this embodiment, the method is applied to service robots in specific indoor application scenarios, such as the dense and chaotic scenes of shopping malls and restaurants. In a shopping mall environment, numerous customers wander around in small groups, their movement routes are not fixed, and different tourists walk towards and away from each other, intersecting and then separating, resulting in considerable chaos. In shopping mall environments, the paths are usually narrow. Taking a hot pot restaurant as an example, customers will frequently visit the self-service condiment station to get seasonings, fruits, and snacks. In such a confined space, customers are not only densely packed but also chaotically grouped. In the above scenarios, if tourists / customers intersect, multiple tourists may be misidentified as one, or a single tourist may be mistakenly identified as multiple tourists due to the complexity of their background. Therefore, since it is difficult to cluster target tourists in this scenario, this method avoids the requirements for target detection and clustering accuracy in the aforementioned methods. It uses point cloud motion state judgment without clustering to determine the motion state of obstacles (for the autonomous driving of service robots, other robots and pedestrians in the scene are obstacles) observed by the service robot.

[0049] For step S11, service robots are typically equipped with cameras and LiDAR, which allows them to obtain a 3D point cloud of the robot within the scene. Since this method requires computation within a 2D point cloud grid, the point cloud is first converted into a 2D grid form through spatial partitioning.

[0050] Specifically, the process of projecting the three-dimensional point cloud of the scene observed by the lidar onto the ground plane and rasterizing it into a two-dimensional raster image includes:

[0051] The three-dimensional point cloud of the scene observed in real time by lidar is projected onto the ground plane to obtain a ground plane with two-dimensional point cloud.

[0052] The ground plane with the two-dimensional point cloud is rasterized to obtain a two-dimensional raster map.

[0053] In this embodiment, during the service robot's movement, a camera is used to obtain two-dimensional images of the road in real time. For three-dimensional point clouds, the three-dimensional point clouds need to be projected onto the ground plane based on the known camera extrinsic parameters to obtain a ground plane map with two-dimensional point clouds.

[0054] The ground plane with two-dimensional point cloud is then rasterized to obtain a two-dimensional raster map composed of multiple grids.

[0055] For step S12, since the process in step 1 is acquired in real time, multiple consecutive frames of two-dimensional grid images can be obtained. Each grid in the two-dimensional grid image corresponds to a specific grid state, including occupied grids with obstacles and empty areas outside the occupied grids. Specifically, the empty areas include: empty grids and unknown grids;

[0056] The methods for determining the grid include:

[0057] The observation area is determined by the lidar projected onto the ground plane and each two-dimensional point cloud;

[0058] In the observation area, the grid cell containing the two-dimensional point cloud is determined to be an occupied grid cell with obstacles.

[0059] In the observation area, unoccupied grid cells are defined as free grid cells without obstacles.

[0060] Grids outside the observation area are identified as unknown grids.

[0061] In this embodiment, the grid is divided into three states: idle, occupied, and unknown. Specifically, the method involves projecting the sensor's coordinates onto a two-dimensional grid image, connecting the sensor to each two-dimensional point cloud in the grid image, and using the Bresenham algorithm (or other algorithms, which will not be elaborated here) to determine the observation area by identifying the grids traversed by the connecting lines. Grids traversed by the connecting lines within the observation area are considered idle grids, indicating the absence of obstacles on these grids. Grids containing the two-dimensional point clouds are considered occupied grids, indicating the presence of obstacles within these grids. Grids not traversed by the connecting lines outside the observation area are considered unknown grids, indicating that these grids are obscured by obstacles and cannot be observed; therefore, it is impossible to determine whether these grids contain obstacles. The specific two-dimensional grid image is shown below. Figure 2 As stated above.

[0062] As one implementation, when there is no two-dimensional point cloud in a certain direction observed by the lidar, the grid covered by the sensor at a preset distance in that certain direction is determined as an empty grid without obstacles.

[0063] In this embodiment, ideally, the sensor's observation range in any direction within real space is limited. Therefore, it should be possible to connect observed point clouds within a 360-degree radius centered on the sensor, and these connections should completely cover all grids surrounding the sensor. However, in reality, due to limitations in the sensor's field of view or obstacles exceeding its measurement range—for example, in a restaurant setting where the service robot is relatively low and there are various partitioned areas—the LiDAR on the service robot cannot observe the entire range. In other words, the connection from the center of the LiDAR sensor to the point cloud cannot completely cover a 360-degree range.

[0064] When there is no point cloud in a certain direction, the grid points in that direction are considered unknown grids because they have not been observed. This leads to a large number of erroneous unknown grids in the grid map. Therefore, this algorithm adds a processing step to assume that the grids within a certain distance in the direction of the sensor are idle, even if there is no point cloud at a certain angle.

[0065] During the service robot's operation, for each frame of observation, after constructing a grid map as described above, the grid maps of each frame are accumulated in the world coordinate system according to the vehicle's positioning information to form the final grid map. Specifically, each frame's grid map can be drawn onto a transparent layer, where idle grids are completely transparent, occupied grids are black and opaque, and unknown grids are gray and semi-transparent. This multi-frame accumulation is like overlaying multiple layers and then observing the overall transparency. If it is pure black, it is an occupied grid; if it is completely transparent, it is an idle grid; other states are unknown grids (the above are only illustrative implementation steps, and the corresponding colors are not limited and can be adjusted according to different scenarios).

[0066] Regarding step S13, since the configurable mobile devices of this method vary, the self-positioning accuracy of different mobile devices also differs. Furthermore, some mobile devices are equipped with lidar, and since the models of lidar differ, the observation errors of the lidar also differ. These errors are usually recorded in the instruction manuals of the mobile devices and lidar.

[0067] Specifically, the predetermined distance error includes: the distance error of the mobile device's own positioning and / or the distance error observed by the lidar. For example, the mobile device's instruction manual indicates that the positioning distance error is ±5 meters.

[0068] As one implementation, shrinking the idle area based on a predetermined distance error includes:

[0069] The number of grid shrinkages in the free area adjacent to the occupied grid is determined based on the predetermined distance error;

[0070] The empty area adjacent to the occupied grid is shrunk by the amount of grid shrinkage.

[0071] In this embodiment, for example, each meter in the real-world grid corresponds to one unit distance in the grid. If each grid cell is 2 units, then along the adjacent line connecting occupied and free grid cells, shrink inwards by 5 units (in the direction of free grid cells). The shrinkage range is 2.5 grid cells inwards. Then, count the number of grid cells shrunk inwards by 2.5 grid cells along the adjacent line (the number of shrunken grid cells can be determined by rounding or rounding up; this is not limited here) to determine the number of shrunken grid cells. This shrunken grid cell portion is the interval area.

[0072] For step S14, for the motion / static detection of the current frame, the 3D point cloud of the scene in the current frame is projected onto the raster map after the shrinking process in step S13. The motion properties of the point cloud in the current frame are then determined.

[0073] As one implementation method, determining the motion state corresponding to the 3D point cloud based on its location region in the grid map includes:

[0074] If a first point cloud appears in the vacant area after the contraction, then the motion state of the first point cloud is determined to be dynamic; and / or

[0075] If a second point cloud appears in the occupied grid or the interval area, the motion state of the second point cloud is determined to be static.

[0076] The step of detecting the motion state of the obstacle based on the motion state corresponding to the three-dimensional point cloud includes:

[0077] The motion state of the obstacle is detected based on the motion states of the first point cloud and the second point cloud.

[0078] In this implementation, the location region in the grid map includes the shrunken free area, occupied grids, and interval regions. The shrunken free area is the region without obstacles in historical observations in the world coordinate system. The 3D point cloud of the latest sensor frame is then transformed to the world coordinate system and projected onto the grid map. If the point cloud appears within the shrunken free area, it indicates that the point is dynamic. If a point cloud appears within an occupied grid or interval region, its motion state is determined to be static, meaning the point is stationary or moving away from the robot (for example, in a restaurant environment where a service robot delivers food, the customer sits in place, the robot delivers the dinner, reverses, and then turns to perform the next task, resulting in the customer's point cloud moving away from the robot).

[0079] For static obstacle point clouds in a scene, they belong to occupied grids. Occupied grids and free grids are adjacent, but due to the shrinking of free grids, occupied grids are placed at a distance outside of free grids. This ensures that point clouds near occupied grids cannot enter free grids, and thus ensures that point clouds of static obstacles cannot enter free grids. (Simply put, if the service robot can observe a wall 10 meters away in the past, it means there is nothing obstructing its line of sight between the service robot and the wall, meaning there should be nothing within that distance. However, if the moving robot observes something 2 meters away in the current frame, it means that this is something newly appearing in the current frame. Since things don't appear out of thin air, they must have moved there in some way, that is, they are "movable".)

[0080] Specifically, such as Figure 3 As shown in the figure, the circle represents the service robot, the gray area where the service robot is located is the idle area determined in the historical steps, the lines in the black area are the static points observed in the current frame (such as partitions in a restaurant environment), and the two clusters of points in the gray area are the dynamic points observed in the current frame.

[0081] For step S15, the motion state of obstacles in different grids is determined by observing the dynamic and static states of each grid in the grid diagram.

[0082] This implementation demonstrates that the motion state attributes of the point cloud can be obtained without point cloud clustering, thus avoiding incorrect target matching due to occlusion and clustering errors, as well as incorrect dynamic / static judgments caused by unstable target detection. Furthermore, this method does not determine the target's dynamic / static status by explicitly comparing changes before and after a fixed time period, therefore it does not require a specific target speed. This avoids incorrect judgments due to the inability to track high-speed moving objects, as well as incorrect judgments due to minimal changes in low-speed moving objects over a period of time. Since this method fully considers the sensor's occlusion model, it will not produce incorrect judgments regarding changes in the target point cloud shape caused by occlusion. The absence of target point cloud detection, clustering, and tracking steps results in very low computational complexity.

[0083] like Figure 4 The diagram shown is a structural schematic of an obstacle motion state detection execution device according to an embodiment of the present invention. The system can execute the obstacle motion state detection method described in any of the above embodiments and is configured in the terminal.

[0084] This embodiment provides an obstacle motion state detection execution device 10, which includes: a rasterization module 11, a raster map construction module 12, a region shrinkage module, a dynamic and static determination module 13, a dynamic and static determination module 14, and a motion state detection module 15.

[0085] The system includes the following modules: a rasterization module 11, which projects the 3D point cloud of the scene observed by the lidar onto the ground plane and rasterizes it into a 2D raster map; a raster map construction module 12, which constructs a raster map of the scene using multiple consecutive frames of 2D raster maps, wherein the raster map includes: occupied grids containing obstacles and empty areas outside the occupied grids; a region shrinking module 13, which shrinks the empty areas based on a predetermined distance error, so that there are gaps between the occupied grids adjacent to the empty areas before shrinking and the empty areas after shrinking; a motion / static determination module 14, which projects the 3D point cloud of the scene observed by the lidar in the current frame onto the raster map and determines the motion state corresponding to the 3D point cloud based on the location of the 3D point cloud in the raster map; and a motion state detection module 15, which detects the motion state of the obstacles based on the motion state corresponding to the 3D point cloud.

[0086] Furthermore, the dynamic and static determination module is used for:

[0087] If a first point cloud appears in the vacant area after the contraction, then the motion state of the first point cloud is determined to be dynamic; and / or

[0088] If a second point cloud appears in the occupied grid or the interval area, the motion state of the second point cloud is determined to be static.

[0089] Furthermore, the motion state detection module is used for:

[0090] The motion state of the obstacle is detected based on the motion states of the first point cloud and the second point cloud.

[0091] Furthermore, the rasterization module is used for:

[0092] The three-dimensional point cloud of the scene observed in real time by lidar is projected onto the ground plane to obtain a ground plane with two-dimensional point cloud.

[0093] The ground plane with the two-dimensional point cloud is rasterized to obtain a two-dimensional raster map.

[0094] Furthermore, the raster map construction module is used for:

[0095] The observation area is determined by the lidar projected onto the ground plane and each two-dimensional point cloud;

[0096] In the observation area, the grid cell containing the two-dimensional point cloud is determined to be an occupied grid cell with obstacles.

[0097] In the observation area, unoccupied grid cells are defined as free grid cells without obstacles.

[0098] Grids outside the observation area are identified as unknown grids.

[0099] Furthermore, the raster map construction module is used for:

[0100] When there is no two-dimensional point cloud in a certain direction observed by the lidar, the grid covered by the sensor at a preset distance in that direction is determined as an empty grid without obstacles.

[0101] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the obstacle motion state detection method in any of the above method embodiments.

[0102] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0103] The three-dimensional point cloud of the scene observed by the lidar is projected onto the ground plane and rasterized into a two-dimensional raster map.

[0104] A grid map of the scene is constructed using a series of two-dimensional grid images. The grid map includes: occupied grids containing obstacles and empty areas outside the occupied grids.

[0105] The free area is shrunk based on a predetermined distance error so that there is a gap between the occupied grid adjacent to the free area before shrinkage and the free area after shrinkage.

[0106] The three-dimensional point cloud of the scene observed by the lidar in the current frame is projected onto the grid map, and the motion state corresponding to the three-dimensional point cloud is determined according to the position area of ​​the three-dimensional point cloud in the grid map.

[0107] The motion state of the obstacle is detected based on the motion state corresponding to the three-dimensional point cloud.

[0108] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, the obstacle motion state detection method in any of the above method embodiments is executed.

[0109] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform an application obstacle motion state detection method.

[0110] In some embodiments, the present invention also provides a mobile device, including a body and an electronic device according to any of the foregoing embodiments mounted on the body. The mobile device may be an unmanned vehicle, such as an unmanned sweeper, unmanned floor scrubber, unmanned logistics vehicle, unmanned passenger car, unmanned sanitation vehicle, unmanned minibus / bus, truck, mining truck, etc., or it may be a robot, etc.

[0111] In some embodiments, the present invention also provides a computer program product that, when run on a computer, causes the computer to execute any one of the obstacle motion state detection methods described in the embodiments of the present invention.

[0112] Figure 5 This is a schematic diagram of the hardware structure of an electronic device for an obstacle motion state detection method according to another embodiment of this application, as shown below. Figure 5 As shown, the device includes:

[0113] One or more processors 510 and memory 520, Figure 5 Taking a processor 510 as an example, the device for the obstacle motion state detection method may also include an input device 530 and an output device 540.

[0114] The processor 510, memory 520, input device 530, and output device 540 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0115] The memory 520, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the obstacle motion state detection method in the embodiments of this application. The processor 510 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 520, thereby implementing the obstacle motion state detection method in the above-described method embodiments.

[0116] The memory 520 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 520 may optionally include memory remotely located relative to the processor 510, and these remote memories may be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0117] Input device 530 can receive input numerical or character information. Output device 540 may include display devices such as a display screen.

[0118] The one or more modules are stored in the memory 520, and when executed by the one or more processors 510, they execute the obstacle motion state detection method in any of the above method embodiments.

[0119] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0120] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0121] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the obstacle motion state detection method of any embodiment of this invention.

[0122] The electronic device in this application embodiment can be an autonomous driving domain controller applied to a mobile device. The autonomous driving domain controller is communicatively connected to various sensors (such as lidar, cameras, millimeter-wave radar, ultrasonic radar, inertial measurement units, wheel speedometers, etc.) mounted on the mobile device. It acquires environmental perception data and vehicle speed information through these sensors, extracts obstacle information and generates the position information of the mobile device based on the acquired environmental perception data and vehicle speed information, and performs path planning based on the position information and obstacle information.

[0123] The electronic device in this application embodiment may also exist in various forms, including but not limited to:

[0124] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0125] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0126] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0127] (4) Other mobile devices with data processing functions.

[0128] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting the motion state of an obstacle, comprising: The three-dimensional point cloud of the scene observed by the lidar is projected onto the ground plane and rasterized into a two-dimensional raster map. A grid map of the scene is constructed using a series of two-dimensional grid images. The grid map includes: occupied grids containing obstacles and empty areas outside the occupied grids. Based on the distance error of the mobile device's own positioning and / or the distance error observed by the lidar, the number of grid shrinkages in the idle area is dynamically calculated, and the idle area adjacent to the occupied grid is shrunk by the number of grid shrinkages, so that there is a gap between the occupied grid adjacent to the idle area before shrinkage and the idle area after shrinkage. The three-dimensional point cloud of the scene observed by the lidar in the current frame is projected onto the grid map, and the motion state corresponding to the three-dimensional point cloud is determined according to the position area of ​​the three-dimensional point cloud in the grid map. The motion state of the obstacle is detected based on the motion state corresponding to the three-dimensional point cloud. The step of determining the motion state of the 3D point cloud based on its location region in the grid map includes: If a first point cloud appears in the vacant area after the contraction, then the motion state of the first point cloud is determined to be dynamic; and / or If a second point cloud appears in the occupied grid or the interval area, the motion state of the second point cloud is determined to be static. The step of detecting the motion state of the obstacle based on the motion state corresponding to the three-dimensional point cloud includes: The motion state of the obstacle is detected based on the motion states of the first point cloud and the second point cloud.

2. The method according to claim 1, characterized in that, The process of projecting the 3D point cloud of the scene observed by the lidar onto the ground plane and rasterizing it into a 2D raster image includes: The three-dimensional point cloud of the scene observed in real time by lidar is projected onto the ground plane to obtain a ground plane with two-dimensional point cloud. The ground plane with the two-dimensional point cloud is rasterized to obtain a two-dimensional raster map.

3. The method according to claim 2, characterized in that, The free area includes: free grid cells and unknown grid cells; The methods for determining the grid include: The observation area is determined by the lidar projected onto the ground plane and each two-dimensional point cloud; In the observation area, the grid cell containing the two-dimensional point cloud is determined to be an occupied grid cell with obstacles. In the observation area, unoccupied grid cells are defined as free grid cells without obstacles. Grids outside the observation area are identified as unknown grids.

4. The method according to claim 3, characterized in that, When there is no two-dimensional point cloud in a certain direction observed by the lidar, the grid covered by the sensor at a preset distance in that direction is determined as an empty grid without obstacles.

5. An obstacle motion state detection and execution device, characterized in that, include: The rasterization module is used to project the 3D point cloud of the scene observed by the lidar onto the ground plane and rasterize it into a 2D raster map. A grid map construction module is used to construct a grid map of the scene using continuous multi-frame two-dimensional grid maps, wherein the grid map includes: occupied grids with obstacles and empty areas outside the occupied grids; The area shrinking module is used to dynamically calculate the number of grid shrinkages in the idle area based on the distance error of the mobile device's own positioning and the distance error of the lidar observation, and shrink the idle area adjacent to the occupied grid by the number of grid shrinkages, so that there is a gap between the occupied grid adjacent to the idle area before shrinkage and the idle area after shrinkage. The dynamic and static determination module is used to project the three-dimensional point cloud of the scene observed by the lidar in the current frame onto the grid map, and determine the motion state corresponding to the three-dimensional point cloud based on the position area of ​​the three-dimensional point cloud in the grid map. The motion state detection module is used to detect the motion state of the obstacle based on the motion state corresponding to the three-dimensional point cloud. The step of determining the motion state of the 3D point cloud based on its location region in the grid map includes: If a first point cloud appears in the vacant area after the contraction, then the motion state of the first point cloud is determined to be dynamic; and / or If a second point cloud appears in the occupied grid or the interval area, the motion state of the second point cloud is determined to be static. The step of detecting the motion state of the obstacle based on the motion state corresponding to the three-dimensional point cloud includes: The motion state of the obstacle is detected based on the motion states of the first point cloud and the second point cloud.

6. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-4.

7. A mobile device, characterized in that, include: The electronic device as described in claim 6.

8. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1-4.

Images