Blind spot memory methods, devices, mobile devices, and storage media

Abstract

This invention provides a blind zone memory method, device, mobile device, and storage medium. The method includes: converting point cloud data collected continuously from multiple sensors across multiple frames into a world coordinate system to generate a multi-frame two-dimensional grid map corresponding to static obstacles; recording the corresponding observation sensor for each grid in the i-th frame of the multi-frame two-dimensional grid map of the static obstacles; adjusting the obstacle presence probability of each grid in the (i-1)-th frame of the two-dimensional grid map to determine the obstacle presence probability in the i-th frame of the two-dimensional grid map; and using the obstacle presence probability in the i-th frame of the two-dimensional grid map to memorize the static obstacles within the blind zones of multiple sensors. This invention solves the problem of applying probabilistic grid algorithms in multi-sensor scenarios, avoids the trailing phenomenon produced by traditional probabilistic grid algorithms when dealing with dynamic objects, and optimizes the calculation of obstacle presence probabilities in the grid.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and more particularly to a blind spot memory method, device, mobile device, and storage medium. Background Technology

[0002] With the rapid development of autonomous driving, various service robot products are emerging in large numbers, such as cleaning robots and warehouse robots. The purpose of these robots is to improve work efficiency and reduce the workload of human workers. Because the sensors installed on the robots have visual blind spots, the robots must not only accurately detect obstacles in their environment, but also maintain the state of targets entering the blind spots, that is, remember them, in order to plan their routes more efficiently and thus perform their tasks better.

[0003] Because the positions of dynamic obstacles change over time, robots cannot maintain this information once they enter a blind spot. Only the positions of static obstacles can be retained; therefore, blind spot memory for static obstacles is a crucial part of environmental perception. Furthermore, if an obstacle is removed from the environment and the robot returns to the location where it was previously observed, the previous memory should be cleared. Generally, robots are not equipped with only one sensor but have multiple sensors with overlapping fields of view. How to correctly remember and clear the results from multiple sensors is a very important issue.

[0004] For blind spot memory of obstacles, a probabilistic raster algorithm is typically used. This algorithm accumulates the state of rasteres in the environment over multiple frames based on a Bayesian probability model. First, for a given frame, when the sensor observes an obstacle point cloud in a spatial raster, that raster is marked as occupied. Simultaneously, based on the beam model, this node is connected to the rasteres marked as "occupied" and designated as an "idle raster"—a raster without any obstacles. If a raster is not observed, its state is unknown. During multi-frame observation, if a raster is marked as occupied in a frame, the probability of an obstacle in that raster increases; conversely, if a raster is marked as idle, the probability of an obstacle in that raster decreases. By accumulating the probabilities of obstacles in a raster based on occupied and idle states, a sufficiently high probability indicates the presence of an obstacle, while a sufficiently low probability indicates the absence of an obstacle.

[0005] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:

[0006] Current technologies generally do not consider the scenario of multiple sensors, and there is no clear method for handling overlapping areas of the field of view of multiple sensors. Furthermore, current research has not yet fused LiDAR with sensors of different modes such as RGBD. Additionally, existing methods exhibit a strong "tailing" phenomenon when using the same probability grid accumulation method for moving objects as for stationary objects, leaving incorrect grids in the areas traversed by the moving object. Summary of the Invention

[0007] To at least address the problems described above in the prior art, in a first aspect, embodiments of the present invention provide a blind spot memory method, applied to a mobile device equipped with multiple sensors, comprising:

[0008] Point cloud data collected by multiple sensors in multiple consecutive frames are uniformly converted to the world coordinate system to obtain a continuous multi-frame two-dimensional raster image in the world coordinate system.

[0009] Identify moving targets in the multi-frame two-dimensional grid map, delete all grids occupied by the moving targets in the multi-frame two-dimensional grid map, and generate a multi-frame two-dimensional grid map corresponding to static obstacles.

[0010] In the i-th frame of the multi-frame two-dimensional grid map of the static obstacle, the corresponding observation sensor is recorded for each grid within the observation area of ​​the multiple sensors.

[0011] Based on the state of the grids within the observation areas of the multiple sensors in the i-th frame of the two-dimensional grid image, the probability of the existence of obstacles in the grids in the (i-1)-th frame of the two-dimensional grid image is adjusted to determine the probability of the existence of obstacles in the i-th frame of the two-dimensional grid image.

[0012] Using the probability of obstacle presence in the i-th frame of the two-dimensional grid image, the static obstacles in the blind zones of the multiple sensors are memorized.

[0013] Secondly, embodiments of the present invention provide an execution device for blind zone memory, comprising:

[0014] The raster image determination module is used to uniformly convert point cloud data collected by multiple sensors in multiple consecutive frames to the world coordinate system to obtain a continuous multi-frame two-dimensional raster image in the world coordinate system.

[0015] The deletion module is used to identify moving targets in the multi-frame two-dimensional grid map, delete all grids occupied by the moving targets in the multi-frame two-dimensional grid map, and generate a multi-frame two-dimensional grid map corresponding to static obstacles.

[0016] The observation sensor recording module is used to record the corresponding observation sensors for each grid within the observation area of ​​the multiple sensors in the i-th frame of the multi-frame two-dimensional grid image of the static obstacle.

[0017] The existence probability determination module is used to adjust the obstacle existence probability of the grid in the (i-1)th frame of the two-dimensional grid image based on the state of the grid in the observation area of ​​the multiple sensors in the i-th frame of the two-dimensional grid image, and determine the obstacle existence probability in the i-th frame of the two-dimensional grid image.

[0018] The blind zone memory module is used to memorize static obstacles in the blind zones of the multiple sensors by utilizing the probability of obstacle presence in the i-th frame two-dimensional grid image.

[0019] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the blind spot memory method of any embodiment of the present invention.

[0020] Fourthly, embodiments of the present invention provide a mobile device, including a main body and an electronic device according to any embodiment of the present invention mounted on the main body.

[0021] Fifthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the blind zone memory method of any embodiment of the present invention.

[0022] Sixthly, embodiments of the present invention also provide a computer program product that, when the computer program product is run on a computer, causes the computer to execute any one of the blind spot memory methods described in the embodiments of the present invention.

[0023] The beneficial effects of this invention are as follows: the sensor source is recorded for each grid in the two-dimensional grid diagram, which solves the problem of applying the probabilistic grid algorithm in the case of multiple sensors, eliminates dynamic grids in the two-dimensional grid diagram, and thus avoids the trailing phenomenon generated by the traditional probabilistic grid algorithm when facing dynamic objects; at the same time, considering the actual driving situation of the mobile device, the calculation of the probability of the existence of obstacles in the grid is optimized, and the determined probability of the existence of obstacles is more accurate. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a blind spot memory method provided in an embodiment of the present invention;

[0026] Figure 2 This is a grid diagram illustrating a blind spot memory method according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the deletion grid of a blind spot memory method provided in an embodiment of the present invention;

[0028] Figure 4 This is a multi-frame raster diagram of a blind zone memory method provided in an embodiment of the present invention;

[0029] Figure 5 This is a multi-frame raster diagram of a blind zone memory method provided in another embodiment of the present invention;

[0030] Figure 6 This is a schematic diagram of the overall process of a blind spot memory method provided in an embodiment of the present invention;

[0031] Figure 7 This is a schematic diagram of the structure of an execution device for blind zone memory according to an embodiment of the present invention;

[0032] Figure 8 This is a schematic diagram of an embodiment of an electronic device with blind spot memory provided by an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Those skilled in the art will recognize that embodiments of this application can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0035] For ease of understanding, the technical terms used in this application are explained below:

[0036] The term "mobile device" as used in this application includes, but is not limited to, vehicles with the six levels of autonomous driving technology (L0-L5) as defined by the Society of Automotive Engineers International (SAE International) or the Chinese national standard "Classification of Driving Automation for Automobiles".

[0037] In some embodiments, the mobile device may be a vehicle or robot having various functions, such as:

[0038] (1) Passenger carrying function, such as family cars, buses, etc.;

[0039] (2) Cargo carrying function, such as ordinary trucks, box trucks, trailer trucks, enclosed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, special structure trucks, etc.;

[0040] (3) Tool functions, such as logistics delivery vehicles, automated guided vehicles (AGVs), patrol vehicles, cranes, hoists, excavators, bulldozers, loaders, road rollers, loaders, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor scrubbers, water sprinkler trucks, sweeping robots, food delivery robots, shopping guide robots, lawnmowers, golf carts, etc.

[0041] (4) Entertainment functions, such as recreational vehicles, amusement park automatic driving devices, balance bikes, etc.;

[0042] (5) Special rescue functions, such as fire trucks, ambulances, power repair vehicles, and engineering emergency rescue vehicles.

[0043] like Figure 1 The diagram shown is a flowchart of a blind spot memory method according to an embodiment of the present invention, which includes the following steps:

[0044] S11: Convert the point cloud data collected by multiple sensors in multiple consecutive frames to the world coordinate system to obtain a continuous multi-frame two-dimensional raster image in the world coordinate system.

[0045] S12: Identify the moving target in the multi-frame two-dimensional grid map, delete all the grids occupied by the moving target in the multi-frame two-dimensional grid map, and generate a multi-frame two-dimensional grid map corresponding to the static obstacle.

[0046] S13: In the i-th frame of the multi-frame two-dimensional grid map of the static obstacle, record the corresponding observation sensor for each grid in the observation area of ​​the multiple sensors.

[0047] S14: Based on the state of the grids within the observation area of ​​the multiple sensors in the i-th frame two-dimensional grid image, adjust the probability of the existence of obstacles in the (i-1)-th frame two-dimensional grid image to determine the probability of the existence of obstacles in the i-th frame two-dimensional grid image;

[0048] S15: Using the probability of obstacle presence in the i-th frame two-dimensional grid image, memorize the static obstacles in the blind zones of the multiple sensors.

[0049] In this embodiment, the mobile device described in this method has the ability to accurately locate itself. This positioning can be based on a global map or local dead reckoning. Furthermore, this method addresses the blind spot memory of multiple sensors; therefore, it requires knowledge of the field of view and calibration extrinsic parameters of each sensor, and calculates the polygonal area projected onto the ground from each sensor's visible region based on these parameters.

[0050] For step S11, for example, consider a smart cleaning robot in a shopping mall. This mobile device is equipped with multiple sensors and has its own positioning function. As the smart cleaning robot moves continuously within the mall, its multiple sensors continuously collect data frame by frame. For example, if the sensor is a lidar, point cloud data can be collected. The smart cleaning robot then converts this point cloud data collected continuously from multiple sensors into a unified world coordinate system.

[0051] As one implementation method, the step of uniformly converting point cloud data collected by multiple sensors in multiple consecutive frames to the world coordinate system to obtain a continuous multi-frame two-dimensional raster image in the world coordinate system includes:

[0052] Based on the calibration extrinsic parameters of each sensor and the positioning information of the mobile device, the point cloud data collected by multiple sensors in multiple consecutive frames are uniformly converted to the world coordinate system.

[0053] The point cloud data, uniformly transformed to the world coordinate system, is segmented into foreground and background to obtain the foreground point cloud.

[0054] The foreground point cloud is projected onto the ground plane, and sampling is performed based on a preset grid size to obtain a two-dimensional grid map in the world coordinate system.

[0055] In this embodiment, since the sensors are located at different positions on the intelligent cleaning robot, it is necessary to use the calibration extrinsic parameters of each sensor and the positioning information of the intelligent cleaning robot to uniformly transform the point clouds collected by these sensors into the world coordinate system. With these parameters, the point clouds collected by multiple sensors can be uniformly transformed into the world coordinate system for unified processing.

[0056] Specifically, for example, the raw point cloud output by RGBD is a 3D point p1(x,y,z) in each RGBD coordinate system. The 2D LiDAR is horizontally mounted at a known height h above the ground, so its output point is also a 3D point p2(x,y,h) in the LiDAR coordinate system. The extrinsic parameters of each sensor consist of six numbers: a three-degree-of-freedom translation vector T(tx,ty,tz) and a three-degree-of-freedom rotation vector (roll,pitch,yaw). The rotation can be represented by a 3x3 rotation matrix R. Thus, each point p_sensor in the sensor coordinate system can be transformed to the vehicle coordinate system as follows: p_car = R * p_sensor + T. Similarly, vehicle positioning information can be represented as a translation vector T' and a 3x3 rotation matrix R', and can be transformed to the world coordinate system as follows: p_world = R' * p_car + T'.

[0057] To further identify obstacles in the point cloud data, the point cloud, which has been uniformly transformed to the world coordinate system, is segmented into foreground and background. The background data does not contain obstacles that affect the intelligent cleaning robot, such as walls and the ground, which are not relevant to obstacle identification. These point clouds are then excluded to obtain the foreground point cloud.

[0058] The processed foreground cloud is projected onto the ground plane, and a grid of fixed size is used for sampling. This allows the intelligent cleaning robot to obtain a two-dimensional grid map in the world coordinate system of its current field of view using the foreground cloud. Figure 2 The image shown is an example of a two-dimensional grid diagram (there are multiple sensors, but only the fields of view of three sensors are listed in the diagram. From left to right, the largest one on the left is the first sensor, the one with the flat frame in the middle is the second sensor, and the one on the right is the field of view of the third sensor).

[0059] For step S12, because the existing technology uses the same probability grid accumulation method for moving objects as for stationary objects, a strong trailing phenomenon occurs. However, blind spot memory is used to remember the position of obstacles in order to plan the route more efficiently and thus perform the operation better. But the position of dynamic obstacles is constantly changing and cannot remain in a certain area. Even if the position of dynamic obstacles is remembered, it will not be of much help in subsequent route planning, because the dynamic obstacles have already moved away, and planning the route based on the remembered position of the dynamic obstacles will be inaccurate.

[0060] Therefore, it is necessary to delete moving targets in the multi-frame 2D grid map. It should be noted that as long as a grid has been occupied by a moving target, for example, if a grid in the first frame of the 2D grid map determines that it is observing a moving target, even if the same grid in the second frame of the 2D grid map determines that it is not observing a moving target, the grid will be deleted in both frames. After deletion, a multi-frame 2D grid map corresponding to static obstacles is generated.

[0061] As one implementation method, identifying moving targets in the multi-frame two-dimensional raster image includes:

[0062] Target detection and tracking are performed on objects in the point cloud data of multiple consecutive frames to identify moving targets in the multi-frame two-dimensional raster image.

[0063] In this embodiment, target detection and tracking refers to detecting targets using a specific method, such as point cloud clustering. Targets detected using these point cloud clustering methods are labeled with IDs, and then these IDs are used to correlate preceding and following frames, thus achieving target tracking.

[0064] Static / dynamic target identification refers to identifying moving targets based on the motion of the same target ID across multiple frames. This step may require combining the robot's own positioning information to determine whether the robot is actually moving in the world coordinate system (for example, if the robot itself is stationary but surrounding obstacles are moving, the obstacles are dynamic. However, if the robot is moving but a nearby obstacle appears stationary, this is because the obstacle is in the same cloud as the robot; therefore, in this case, the obstacle that appears stationary to the robot may actually be "dynamic"). After identifying the moving target, the point cloud grids containing the moving target are discarded, retaining only the point cloud grids of stationary objects, such as... Figure 3 As shown, the top left image represents the 2D raster image of the first frame, the top right image represents the 2D raster image of the second frame, the bottom left image represents the 2D raster image of the third frame, and the bottom right image represents the 2D raster image of the third frame with dynamic obstacles removed (X represents the deleted raster).

[0065] For step S13, in the i-th frame of the multi-frame two-dimensional grid diagram of the static obstacle, the corresponding observation sensor is recorded for each grid within the observation area of ​​the sensor. For example, if grid A in the first frame of the two-dimensional grid diagram is observed by sensor M and its state is determined to be occupied, then the observation sensor recorded for grid A is sensor M. If grid A in the second frame of the two-dimensional grid diagram is observed by sensor N and its state is determined to be occupied, then the observation sensors recorded for grid A are sensor M and sensor N.

[0066] For step S14, the probability of obstacle presence in the grid in the (i-1)th frame of the two-dimensional grid is adjusted based on the state of the grid within the observation area of ​​the multiple sensors in the i-th frame of the two-dimensional grid.

[0067] In one implementation, the state of the grid includes: occupied grid and idle grid;

[0068] Increasing the probability of obstacle presence in the (i-1)th frame of the two-dimensional grid image includes:

[0069] For the occupied grid cells within the observation areas of the multiple sensors in the i-th frame of the two-dimensional grid image, the probability is increased based on the probability of the occupied grid cells being obstacles in the (i-1)-th frame of the two-dimensional grid image.

[0070] For example, during the cleaning process of an intelligent cleaning robot, if a certain grid is occupied in the first frame and is still occupied in the second frame, this further strengthens the possibility that the grid is occupied. In general, for the same grid, if it is occupied for multiple consecutive frames, the probability of an obstacle in the grid gradually increases.

[0071] In one implementation, the state of the grid includes: occupied grid and idle grid;

[0072] The probability reduction of the obstacle presence probability in the (i-1)th frame of the two-dimensional grid image includes:

[0073] For the empty grids in the observation areas of the multiple sensors in the i-th frame two-dimensional grid image, determine the observation sensors corresponding to the empty grids in the 1st to i-1th frame two-dimensional grid images. When the sensor of the empty grid in the i-th frame is determined to belong to the observation sensors corresponding to the 1st to i-1th frame two-dimensional grid images, the probability of the existence of obstacles in the i-1th frame two-dimensional grid image is reduced.

[0074] When the sensor that determines the empty grid in the i-th frame does not belong to the corresponding observation sensor in the two-dimensional grid diagram of the first to the (i-1)th frames, the obstacle existence probability in the two-dimensional grid diagram of the (i-1)th frame is not updated.

[0075] However, considering the actual use of intelligent cleaning robots, a completely smooth driving experience cannot be guaranteed. For example, the grout lines on the floor can cause slight vibrations when the robot travels on such surfaces, resulting in fluctuations in the sensor's field of view. For instance, a grid cell might be occupied in frames 1, 2, and 3, but become clear as empty in frame 5 due to vibrations, only to return to being occupied in frames 6 and 7. The probability of an obstacle being present when the grid is empty in frame 4 is unstable. If the probability is continuously lowered, the final probability of obstacle presence will be inaccurate. Therefore, considering real-world scenarios, adjustments need to be made specifically for different situations.

[0076] To more clearly illustrate the probability calculation process for the different scenarios of increased and decreased probability, the two-dimensional grid image is simplified, retaining only static obstacles (the image of which is a signboard placed on the ground in a shopping mall). Figure 4 As shown, the left, center, and right sides represent the two-dimensional raster images of the first, second, and third frames, respectively. In the two-dimensional raster images, black represents occupied grid cells, and white represents idle grid cells. The first sensor is positioned horizontally on the left, and the second sensor is positioned vertically on the right.

[0077] For example, take grid (6,4) of the first frame of the two-dimensional grid image as an example (where 6 refers to the sixth grid from left to right and 4 refers to the fourth grid from top to bottom). At this time, grid (6,4) is observed by the second sensor (that is, the second sensor is the observation sensor), and it occupies the grid. At this time, the probability of the obstacle in grid (6,4) is +P.

[0078] In the second frame of the two-dimensional grid image, grid (6,4) is still observed by the second sensor. At this time, it is an idle grid. The second sensor is an observation sensor. In this case, the probability of the obstacle in grid (6,4) is -P. At this time, the probability of the obstacle in this grid in the first frame of the two-dimensional grid image is P. The probability of the obstacle in this grid in the second frame of the two-dimensional grid image is PP=0.

[0079] In the third frame of the two-dimensional grid image, grid (6,4) is still observed by the second sensor. At this time, it is an occupied grid. The second sensor is an observation sensor. In this case, the probability of the obstacle in grid (6,4) is +P. At this time, the probability of the obstacle in this grid in the second frame of the two-dimensional grid image is 0. The probability of the obstacle in this grid in the third frame of the two-dimensional grid image is 0 + P = P.

[0080] For example, consider grid (5,8) in the first frame of the 2D grid image. At this time, grid (5,8) is observed simultaneously by the first sensor and the second sensor (that is, the first sensor and the second sensor are the observation sensors), and it is also an occupied grid. It should be noted that although grid (5,8) is observed simultaneously by the first sensor and the second sensor, the probability of an obstacle existing in grid (5,8) is only +P.

[0081] Considering some special circumstances, such as Figure 5 As shown, the left, middle and right sides are the two-dimensional grid images of the first frame, the second frame and the third frame, respectively. In the second frame, the field of view of each sensor in the two-dimensional grid image changes due to the shaking of the intelligent cleaning robot.

[0082] For example, take grid (5,8) of the first frame of the two-dimensional grid image. At this time, grid (5,8) is observed by the first sensor (that is, the first sensor is the observation sensor), and it occupies the grid. At this time, the probability of the obstacle in grid (5,8) is +P.

[0083] However, grid (5,8) in the second frame of the two-dimensional grid image is observed by the second sensor. At this time, it is an idle grid, and the second sensor is not an observation sensor. In this case, the probability of the obstacle in grid (5,8) is not adjusted (not -P). At this time, the probability of the obstacle in this grid in the first frame of the two-dimensional grid image is P, and the probability of the obstacle in this grid in the second frame of the two-dimensional grid image is still P without adjustment.

[0084] In the third frame of the two-dimensional grid image, grid (5,8) is observed by the first sensor. At this time, it is an occupied grid. The first sensor is an observation sensor. In this case, the probability of the obstacle in grid (5,8) is +P. At this time, the probability of the obstacle in grid (5,8) in the second frame of the two-dimensional grid image is P. The probability of the obstacle in grid (5,8) in the third frame of the two-dimensional grid image is P + P = 2P.

[0085] Additionally, the probability of obstacle presence is not adjusted for areas not visible to the sensor (i.e., the grid outside the frame in the diagram).

[0086] By performing multi-frame continuous processing in the above manner, the probability of obstacles existing in the grid cells of the two-dimensional grid image is updated. After processing multiple consecutive frames, some grid cells will have a relatively high occupancy probability. The occupancy grid cells of the current frame can be output by judging the grid cell probability according to the set threshold.

[0087] For step S15, the probability of obstacle existence in each frame of the two-dimensional grid image determined in step S14 is used to determine the probability of obstacle existence in the frame of the two-dimensional grid image to be memorized, so as to memorize the static obstacles in the blind zone of multiple sensors.

[0088] Specifically, the step of memorizing static obstacles within the blind zones of the multiple sensors using the obstacle presence probability in the i-th frame of the two-dimensional grid image includes:

[0089] The observation areas of the multiple sensors change in the world coordinate system as the mobile device moves. When it is determined that the static obstacle leaves the observation area of ​​the multiple sensors, blind spot memory is performed based on the probability of the obstacle's existence in the previous frame before the static obstacle leaves the observation area of ​​the multiple sensors.

[0090] In this embodiment, the polygonal range of the visible area of ​​each sensor projected onto the ground is calculated based on the intrinsic and extrinsic parameters of each sensor. As the vehicle moves, the visible area of ​​the sensors also moves in the world coordinate system. When a grid moves outside the visible area, updating that grid stops, and it is maintained at the state of the frame before leaving the visible area. In summary, the overall steps of this method are as follows: Figure 6 As stated above.

[0091] As can be seen from this implementation, the sensor source is recorded for each grid in the two-dimensional grid diagram, which solves the application of the probabilistic grid algorithm in the case of multiple sensors. It eliminates dynamic grids in the two-dimensional grid diagram, thereby avoiding the trailing phenomenon caused by the traditional probabilistic grid algorithm when facing dynamic objects. At the same time, considering the actual driving situation of the mobile device, the calculation of the probability of the existence of obstacles in the grid is optimized, and the determined probability of the existence of obstacles is more accurate.

[0092] like Figure 7 The diagram shown is a schematic diagram of a blind zone memory execution device provided in an embodiment of the present invention. The execution device can execute the blind zone memory method described in any of the above embodiments and is configured in a terminal.

[0093] The blind spot memory execution device 10 provided in this embodiment includes: a grid map determination module 11, a deletion module 12, an observation sensor recording module 13, an existence probability determination module 14, and a blind spot memory module 15.

[0094] The system comprises the following modules: a grid map determination module 11, which converts point cloud data collected by multiple sensors in multiple consecutive frames to a world coordinate system to obtain a continuous multi-frame two-dimensional grid map in the world coordinate system; a deletion module 12, which identifies moving targets in the multi-frame two-dimensional grid map and deletes all grids occupied by the moving targets in the multi-frame two-dimensional grid map to generate a multi-frame two-dimensional grid map corresponding to static obstacles; an observation sensor recording module 13, which records the corresponding observation sensors for all grids within the observation area of ​​the multiple sensors in the i-th frame of the multi-frame two-dimensional grid map of static obstacles; an existence probability determination module 14, which adjusts the obstacle existence probability of the grids in the (i-1)-th frame of the two-dimensional grid map based on the state of the grids within the observation area of ​​the multiple sensors in the i-th frame of the two-dimensional grid map to determine the obstacle existence probability in the i-th frame of the two-dimensional grid map; and a blind zone memory module 15, which uses the obstacle existence probability in the i-th frame of the two-dimensional grid map to memorize the static obstacles in the blind zones of the multiple sensors.

[0095] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the blind memory method in any of the above method embodiments;

[0096] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0097] Point cloud data collected by multiple sensors in multiple consecutive frames are uniformly converted to the world coordinate system to obtain a continuous multi-frame two-dimensional raster image in the world coordinate system.

[0098] Identify moving targets in the multi-frame two-dimensional grid map, delete all grids occupied by the moving targets in the multi-frame two-dimensional grid map, and generate a multi-frame two-dimensional grid map corresponding to static obstacles.

[0099] In the i-th frame of the multi-frame two-dimensional grid map of the static obstacle, the corresponding observation sensor is recorded for each grid within the observation area of ​​the multiple sensors.

[0100] Based on the state of the grids within the observation areas of the multiple sensors in the i-th frame of the two-dimensional grid image, the probability of the existence of obstacles in the grids in the (i-1)-th frame of the two-dimensional grid image is adjusted to determine the probability of the existence of obstacles in the i-th frame of the two-dimensional grid image.

[0101] Using the probability of obstacle presence in the i-th frame of the two-dimensional grid image, the static obstacles in the blind zones of the multiple sensors are memorized.

[0102] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions / modules corresponding to the methods in the embodiments of this invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, the blind-zone memory method in any of the above method embodiments is executed.

[0103] This invention also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform an application blind spot memory method.

[0104] The electronic device provided in this embodiment of the invention may be, for example, an autonomous driving domain controller applied to an autonomous vehicle. It is communicatively connected to various sensors mounted on the autonomous vehicle, acquires environmental perception data and vehicle speed information through these sensors, extracts obstacle information based on the acquired environmental perception data, generates the location information of the autonomous vehicle based on the vehicle speed information, and performs path planning based on the location information and obstacle information.

[0105] In some embodiments, the present invention also provides a mobile device, including a body and an electronic device according to any of the foregoing embodiments mounted on the body. The mobile device may be an unmanned vehicle, such as an unmanned sweeper, unmanned floor scrubber, unmanned logistics vehicle, unmanned passenger car, unmanned sanitation vehicle, unmanned minibus / bus, truck, mining truck, etc., or it may be a robot, etc.

[0106] In some embodiments, the present invention also provides a computer program product that, when run on a computer, causes the computer to execute any one of the blind spot memory methods described in the embodiments of the present invention.

[0107] Figure 8 This is a schematic diagram of the hardware structure of an electronic device for a blind spot memory method according to another embodiment of this application, as shown below. Figure 8 As shown, the device includes:

[0108] One or more processors 810 and memory 820, Figure 8 Taking a processor 810 as an example, the device for the blind spot memory method may also include an input device 830 and an output device 840.

[0109] The processor 810, memory 820, input device 830, and output device 840 can be connected via a bus or other means. Figure 8Taking the example of a connection between China and Israel via a bus.

[0110] The memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the blind memory method in the embodiments of this application. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 820, thereby implementing the blind memory method in the above-described method embodiments.

[0111] The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 820 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 820 may optionally include memory remotely located relative to the processor 810, and these remote memories can be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0112] Input device 830 can receive input numerical or character information. Output device 840 may include display devices such as a display screen.

[0113] The one or more modules are stored in the memory 820, and when executed by the one or more processors 810, the blind zone memory method in any of the above method embodiments is executed.

[0114] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0115] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0116] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the blind spot memory method of any embodiment of this invention.

[0117] The electronic devices described in this application exist in various forms, including but not limited to:

[0118] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0119] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0120] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0121] (4) Other electronic devices with data processing functions.

[0122] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A blind spot memory method, applied to a mobile device equipped with multiple sensors, comprising: Point cloud data collected continuously from multiple sensors in multiple frames are uniformly converted to the world coordinate system to obtain a continuous multi-frame two-dimensional raster image in the world coordinate system. Identify moving targets in the multi-frame two-dimensional grid map, delete all grids occupied by the moving targets in the multi-frame two-dimensional grid map, and generate a multi-frame two-dimensional grid map corresponding to static obstacles. In the i-th frame of the multi-frame two-dimensional grid map of the static obstacle, the corresponding observation sensor is recorded for each grid within the observation area of ​​the multiple sensors. Based on the state of the grids within the observation areas of the multiple sensors in the i-th frame of the two-dimensional grid image, the grid states include: occupied grids and idle grids. The obstacle existence probability of the grid in the (i-1)-th frame of the two-dimensional grid image is adjusted to determine the obstacle existence probability in the i-th frame of the two-dimensional grid image, including: reducing the obstacle existence probability of the grid in the (i-1)-th frame of the two-dimensional grid image. For idle grids within the observation areas of the multiple sensors in the i-th frame of the two-dimensional grid image, the observation sensors corresponding to the idle grids in the first to (i-1)-th frames of the two-dimensional grid image are determined. When the sensor of the idle grid is determined to belong to the corresponding observation sensor in the first to (i-1)-th frames of the two-dimensional grid image, the obstacle existence probability in the (i-1)-th frame of the two-dimensional grid image is reduced. When the sensor of the idle grid is determined not to belong to the corresponding observation sensor in the first to (i-1)-th frames of the two-dimensional grid image, the obstacle existence probability in the (i-1)-th frame of the two-dimensional grid image is not updated. Using the probability of obstacle presence in the i-th frame of the two-dimensional grid image, the static obstacles in the blind zones of the multiple sensors are memorized.

2. The method according to claim 1, characterized in that, The adjustment of the obstacle presence probability in the (i-1)th frame of the two-dimensional grid map includes: The probability of the existence of obstacles in the (i-1)th frame of the two-dimensional grid map is increased.

3. The method according to claim 2, characterized in that, The state of the grid includes: occupied grid and idle grid; The step of increasing the probability of obstacle presence in the (i-1)th frame of the two-dimensional grid image includes: For the occupied grid cells within the observation areas of the multiple sensors in the i-th frame of the two-dimensional grid image, the probability is increased based on the probability of the occupied grid cells being obstacles in the (i-1)-th frame of the two-dimensional grid image.

4. The method according to claim 1, characterized in that, The method of memorizing static obstacles within the blind zones of the multiple sensors using the obstacle presence probability in the i-th frame of the two-dimensional grid image includes: The observation areas of the multiple sensors change in the world coordinate system as the mobile device moves. When it is determined that the static obstacle leaves the observation area of ​​the multiple sensors, blind spot memory is performed based on the probability of the obstacle's existence in the previous frame before the static obstacle leaves the observation area of ​​the multiple sensors.

5. The method according to claim 1, characterized in that, The process of uniformly converting point cloud data collected continuously from multiple sensors across multiple frames to a world coordinate system to obtain a continuous multi-frame two-dimensional raster image in the world coordinate system includes: Based on the calibration extrinsic parameters of each sensor and the positioning information of the mobile device, the point cloud data collected by multiple sensors in multiple consecutive frames are uniformly converted to the world coordinate system. The point cloud data, uniformly transformed to the world coordinate system, is segmented into foreground and background to obtain the foreground point cloud. The foreground point cloud is projected onto the ground plane, and sampling is performed based on a preset grid size to obtain a two-dimensional grid map in the world coordinate system.

6. The method according to claim 1, characterized in that, The identification of moving targets in the multi-frame two-dimensional raster image includes: Target detection and tracking are performed on objects in the point cloud data of multiple consecutive frames to identify moving targets in the multi-frame two-dimensional raster image.

7. A device for executing blind-spot memory, comprising: The raster image determination module is used to uniformly convert point cloud data collected by multiple sensors in multiple consecutive frames to the world coordinate system to obtain a continuous multi-frame two-dimensional raster image in the world coordinate system. The deletion module is used to identify moving targets in the multi-frame two-dimensional grid map, delete all grids occupied by the moving targets in the multi-frame two-dimensional grid map, and generate a multi-frame two-dimensional grid map corresponding to static obstacles. The observation sensor recording module is used to record the corresponding observation sensors for each grid within the observation area of ​​the multiple sensors in the i-th frame of the multi-frame two-dimensional grid image of the static obstacle. An obstacle existence probability determination module is used to adjust the obstacle existence probability of the grid in the (i-1)th frame of the two-dimensional grid image based on the state of the grids in the observation areas of the multiple sensors in the i-th frame, where the grid states include occupied grids and idle grids. This adjustment includes: reducing the obstacle existence probability of the grid in the (i-1)th frame of the two-dimensional grid image; for idle grids in the observation areas of the multiple sensors in the i-th frame of the two-dimensional grid image, determining the observation sensors corresponding to the idle grids in the first to (i-1)th frames of the two-dimensional grid image; when the sensor of the idle grid is determined to belong to the corresponding observation sensor in the first to (i-1)th frames of the two-dimensional grid image, reducing the obstacle existence probability in the (i-1)th frame of the two-dimensional grid image; and when the sensor of the idle grid is determined not to belong to the corresponding observation sensor in the first to (i-1)th frames of the two-dimensional grid image, not updating the obstacle existence probability in the (i-1)th frame of the two-dimensional grid image. The blind zone memory module is used to memorize static obstacles in the blind zones of the multiple sensors by utilizing the probability of obstacle presence in the i-th frame two-dimensional grid image.

8. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-6.

9. A mobile device comprising a body and an electronic device according to claim 8 mounted on the body.

10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-6.

Images