Obstacle detection system, mobile device, obstacle detection method, and recording medium

Abstract

An obstacle detection system includes a three-dimensional distance sensor that measures a distance to a road surface on which a mobile device travels and measures a distance in a height direction, and an obstacle detection unit that detects an obstacle on the road surface when the mobile device is stopped. The obstacle detection unit includes a height history data generation unit that generates height history data supplemented with height information held until the mobile device is stopped. The height information indicates a height from the road surface and is based on an amount of travel of the mobile device over a predetermined period of time until the mobile device is stopped.

Description

Technical Field

[0001] This invention relates to an obstacle detection system, a mobile device, an obstacle detection method, and a recording medium. Background Technology

[0002] For autonomous mobile devices operating outdoors, safety technologies are needed to detect, stop, or avoid obstacles. Various sensors, such as red, green, and blue (RGB) depth cameras (so-called RGB D cameras), stereo cameras, radar, and ultrasonic sensors, are used in these safety technologies. Currently, sensors employing light detection and ranging or laser imaging detection and ranging (LiDAR) sensors are commonly used in safety technologies. LiDAR is a technology used to measure, for example, the distance to an object and the shape of the object, based on information about the light reflected from the object by illuminating it with a laser. Specifically, two-dimensional (2D) LiDAR sensors with the ability to detect obstacles in a two-dimensional plane are typically used.

[0003] Patent document (PTL) 1 discloses a technique for performing one-dimensional laser scans K times using a ranging device mounted on a mobile device and continuously detecting obstacles with concave shapes in all scans from the (Ka)th scan to the Kth scan.

[0004] Reference List

[0005] Patent documents

[0006] PTL 1

[0007] Japanese Unexamined Patent Application Publication No. 2000-075032 Summary of the Invention

[0008] Technical issues

[0009] According to existing technology, when the mobile device stops, obstacles are not detected in areas or blind spots outside the sensor's observation range due to obstruction, for example, by the housing of the mobile device.

[0010] Solution to the problem

[0011] This disclosure provides an obstacle detection system comprising: a three-dimensional distance sensor for measuring the distance to a road surface on which a mobile device travels and for measuring the distance in the vertical direction; and an obstacle detection unit for detecting obstacles on the road surface when the mobile device stops. The obstacle detection unit includes a height history data generation unit for generating height history data supplemented with height information stored up to the point where the mobile device stops. The height information represents the height above the road surface and is based on the amount of travel by the mobile device during a predetermined time period until the mobile device stops.

[0012] This disclosure provides a mobile device comprising: a driving device that travels on a road surface; a three-dimensional distance sensor for measuring distances to the road surface and distances in the vertical direction; and an obstacle detection unit for detecting obstacles on the road surface when the driving device stops. The obstacle detection unit includes a height history data generation unit for generating height history data supplemented with height information stored up to the point where the driver stops. The height information represents the height above the road surface and is based on the amount of travel by the driving device over a predetermined time period until the driving device stops.

[0013] This disclosure provides a recording medium carrying computer-readable code that, when executed by a computer system, causes the computer system to perform a method. The method includes detecting obstacles on a road surface when a mobile device stops. Detecting obstacles includes generating height history data supplemented with height information stored up to the point where the mobile device stops. The height information represents the height above the road surface and is based on the amount of travel made by the mobile device over a predetermined time period until the mobile device stops.

[0014] This disclosure provides an obstacle detection method, including detecting obstacles on a road surface when a mobile device stops. Obstacle detection includes generating height history data, supplemented by height information stored up to the point where the mobile device stops. The height information represents the height above the road surface and is based on the amount of time the mobile device has traveled during a predetermined time period until the mobile device stops.

[0015] Beneficial effects of the present invention

[0016] According to one aspect of this disclosure, when the mobile device is stopped, obstacles in areas outside the blind spot or the observation range of a sensor can be detected due to obstruction, for example, by the housing of the mobile device. Attached Figure Description

[0017] A more complete understanding of the embodiments of this disclosure and its many accompanying advantages and features can be readily obtained and understood through the following detailed description with reference to the accompanying drawings.

[0018] [ Figure 1 ]

[0019] Figure 1 This is an external perspective view of the mobile device used as an obstacle detection system.

[0020] [ Figure 2 ]

[0021] Figure 2 yes Figure 1 Side view of the mobile device shown.

[0022] [ Figure 3 ]

[0023] Figure 3 Viewed from above Figure 1 The diagram shows a plan view of the mobile device.

[0024] [ Figure 4 ]

[0025] Figure 4 This is an explanation Figure 1 The block diagram of the hardware configuration of the mobile device described herein.

[0026] [ Figure 5 ]

[0027] Figure 5 It is shown that... Figure 1 The diagram shows the functional configuration related to autonomous driving of the mobile device.

[0028] [ Figure 6 ]

[0029] Figure 6 It is shown that... Figure 5 The diagram shows the block diagram of the speed control-related functional configuration of the steering control unit.

[0030] [ Figure 7 ]

[0031] Figure 7 This is a diagram showing the outline of the curb detection.

[0032] [ Figure 8 ]

[0033] Figure 8 It is a diagram showing the outline of the ditch inspection;

[0034] [ Figure 9A ]

[0035] Figure 9A This is a graph showing the results of obstacle detection on the road surface.

[0036] [ Figure 9B ]

[0037] Figure 9B This is a graph showing the results of obstacle detection on the road surface.

[0038] [ Figure 10 ]

[0039] Figure 10 It is by Figure 1 The flowchart shown illustrates the process of obstacle detection on the road surface performed by the mobile device.

[0040] [ Figure 11 ]

[0041] Figure 11 This is a flowchart of the process of storing point cloud data.

[0042] [ Figure 12 ]

[0043] Figure 12 This is a flowchart of the process of generating an elevation map.

[0044] [ Figure 13 ]

[0045] Figure 13 It is a diagram showing the outline of noise removal;

[0046] [ Figure 14A ]

[0047] Figure 14A This is a graph showing the results of noise removal;

[0048] [ Figure 14B ]

[0049] Figure 14B This is a graph showing the results of noise removal;

[0050] [ Figure 15A ]

[0051] Figure 15A It is a map showing the elevation.

[0052] [ Figure 15B ]

[0053] Figure 15B It is a map showing the elevation.

[0054] [ Figure 16 ]

[0055] Figure 16 This is a diagram illustrating an overview of blind spot handling in obstacle detection.

[0056] [ Figure 17 ]

[0057] Figure 17 It is always a flowchart of the process of algorithm execution.

[0058] [ Figure 18 ]

[0059] Figure 18 This is a flowchart of the process of generating a historical elevation map.

[0060] [ Figure 19 ]

[0061] Figure 19 This is a flowchart of the process of removing a moved object region.

[0062] [ Figure 20 ]

[0063] Figure 20 This is a flowchart of the client-server process for using the height history map clearing service.

[0064] [ Figure 21 ]

[0065] Figure 21 This is a sequence diagram illustrating the process of invoking the height history map clearing service.

[0066] [ Figure 22 ]

[0067] Figure 22 This is a sequence diagram of another process that invokes the height history map clearing service.

[0068] [ Figure 23 ]

[0069] Figure 23 This is a flowchart of the process of generating a map of height obstacles.

[0070] [ Figure 24 ]

[0071] Figure 24 This is a flowchart of the process of generating a differential obstacle map.

[0072] [ Figure 25 ]

[0073] Figure 25 This is a flowchart of the process of generating a map of ditch obstacles.

[0074] [ Figure 26 ]

[0075] Figure 26 This is a flowchart of the process of generating an obstacle map.

[0076] [ Figure 27 ]

[0077] Figure 27 This is a diagram showing an overview of the additional processing applied to the obstacle map.

[0078] [ Figure 28A ]

[0079] Figure 28A This is a diagram showing the results of obstacle detection without additional processing of the obstacle map.

[0080] [ Figure 28B ]

[0081] Figure 28B This is a diagram showing the results of obstacle detection, including additional processing of the obstacle map.

[0082] [ Figure 29 ]

[0083] Figure 29 This is a diagram showing a car as an obstacle detection system; and

[0084] [ Figure 30 ]

[0085] Figure 30 This is a diagram showing the structure of an obstacle detection system.

[0086] The accompanying drawings are intended to depict embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Unless explicitly stated otherwise, the drawings should not be considered to be drawn to scale. Furthermore, in several views, the same or similar reference numerals denote the same or similar parts. Detailed Implementation

[0087] In describing the embodiments shown in the accompanying drawings, specific terminology has been used for clarity. However, this disclosure is not intended to be limited to the specific terminology chosen, and it should be understood that each particular element includes all technical equivalents that have similar functionality, operate in a similar manner, and achieve similar results.

[0088] Referring now to the accompanying drawings, embodiments of this disclosure are described below. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0089] The obstacle detection system, mobile device, obstacle detection method, and procedure are described in detail below with reference to the accompanying drawings.

[0090] Configuration of mobile devices

[0091] Figure 1This is an external perspective view of the mobile device 1, which is an obstacle detection system. The mobile device 1 is a mobile device, such as a vehicle or robot, that travels on rough ground and is used for equipment inspection in facilities such as factories.

[0092] In this specification, the Y direction corresponds to the lateral width direction of the mobile device 1. The X direction corresponds to the travel direction of the mobile device 1, and the Z direction corresponds to the height direction of the mobile device 1. In the following description, the coordinates represented by X, Y, and Z can be referred to as robot coordinates. In other words, robot coordinates are coordinates centered on the mobile device 1.

[0093] The mobile device 1, which serves as an obstacle detection system, includes tracked mobile bodies 11a and 11b and a main body 10.

[0094] Tracked mobile bodies 11a and 11b are units that serve as drives for the mobile device 1. Each of the tracked mobile bodies 11a and 11b uses a metal or rubber belt.

[0095] Compared to tire-based mobility devices (such as automobiles), tracked mobility devices have a wider contact area with the ground and can travel more stably even in environments with poor footing, for example. While tire-based mobility devices require space to turn, tracked mobility devices can perform so-called spin turns and therefore can turn smoothly even in confined spaces.

[0096] In this disclosure, turning in place refers to turning at a location by rotating the left and right tracks in opposite directions at a constant speed with the center of the vehicle body as the axis.

[0097] Two tracked mobile bodies 11a and 11b are mounted across the main body 10, enabling the mobile device 1 to travel. The number of tracked mobile bodies is not limited to two, and can be three or more. For example, the mobile device 1 may include three tracked mobile bodies arranged in parallel, enabling the mobile device 1 to travel. Alternatively, the mobile device 1 may include four tracked mobile bodies arranged like tires on the front, rear, right, and left sides of a car.

[0098] Tracked mobile bodies 11a and 11b each have a triangular shape. In the following description, tracked mobile bodies 11a and 11b can be collectively referred to as "tracked mobile body 11". The advantage of the triangular tracked mobile body 11 is that, for example, when the longitudinal length of the mobile body is constrained, the contact area between the tracked mobile body and the ground can be maximized. As mentioned above, this configuration can increase driving stability. In the case of so-called tank tracks, where the upper side (drive wheel side) is longer than the lower side (lower wheel side), the contact area with the ground is generally smaller when the longitudinal length is constrained, thus making driving unstable. As mentioned above, the tracked mobile body 11 is effective in improving driving performance, especially when applied to a relatively small mobile device 1.

[0099] The main body 10 supports the movement of tracked mobile bodies 11a and 11b and includes a controller for controlling the drive of the mobile bodies 1. The main body 10 also includes a battery to supply power for driving the tracked mobile bodies 11a and 11b.

[0100] The main body 10 of the mobile device 1 includes a power button 12, a start button 13, an emergency stop button 14, a status indicator light 15, and bumpers 16 and 17. A sensor mount 18 is attached to the upper portion of the main body 10 of the mobile device 1 via rods 19. A distance sensor 112 for horizontal detection and a distance sensor 113 for tilt detection are mounted on the sensor mount 18. The sensor mount 18 is supported by four rods 19 on the main body 10.

[0101] The power button 12 is an operating device pressed by a person near the mobile device 1 to turn the power of the mobile device 1 on or off. The start button 13 is an operating device pressed by a person near the mobile device 1 to start the two tracked mobile bodies 11a and 11b. The emergency stop button 14 is an operating device pressed by a person near the mobile device 1 to stop the movement of the mobile device 1.

[0102] The status indicator light 15 is a notification device for indicating the status of the mobile device 1. When the status of the mobile device 1 changes due to, for example, a decrease in the remaining battery power, the status indicator light 15 illuminates to notify nearby people of the change in the status of the mobile device 1. The status indicator light 15 also illuminates in response to the detection of an anomaly, such as an obstacle hindering the movement of the mobile device 1.

[0103] although Figure 1 The mobile device 1 includes a status indicator light 15, but the number of status indicator lights 15 is not limited to this, and there may be two or more. As an alternative to or supplement to the status indicator light 15, a notification device may include, for example, a speaker, to output an alarm sound indicating the status of the mobile device 1.

[0104] Bumpers 16 and 17 are respectively provided on the front and rear sides of the main body 10, and protect, for example, the main body 10 of the moving device 1 and the tracked moving bodies 11a and 11b.

[0105] In the mobile device 1, a distance sensor 112 for horizontal detection is mounted on a sensor mounting bracket 18 at a height of 0.5 meters in front of the main body 10, with the direction of travel of the main body 10 as the front. The distance sensor 112 for horizontal detection is a three-dimensional (3D) distance sensor that uses 3D light detection and ranging (LiDAR) to measure distances in three-dimensional space.

[0106] The 3D lidar sensor used as the distance sensor 112 employs a non-repetitive scanning method, has a 360-degree field of view, and can measure a conical range of 70.4 degrees observed from the center of the sensor's detection surface, as well as distances up to 90 meters. The non-repetitive scanning method involves scanning in a floral pattern while minimizing phase shift in both the horizontal and vertical directions, and is characterized by the point cloud coverage within the measurement range increasing over time.

[0107] Distance sensor 112 illuminates an object (such as an obstacle) on the road surface with a laser, measures the distance to the object and the direction of the object based on the time it takes for the laser to reflect from the object, and obtains the measurement results as data.

[0108] In the mobile device 1, a distance sensor 113 for tilt direction detection is mounted on a sensor mount 18 with the front facing the direction of travel of the main body 10. This mounting position is determined to allow detection of the road surface up to the front of the bumper 16 in front of the mobile device 1. The distance sensor 113 for horizontal detection is a 3D distance sensor similar to the distance sensor 112, employing 3D LiDAR to measure distances in three-dimensional space. The road surface is the surface surrounding the mobile device 1.

[0109] The driving surface is not limited to outdoor areas, but can also be indoor areas. In addition, the driving surface is not limited to flat surfaces, but includes, for example, surfaces with grooves or steps and uneven, rough surfaces.

[0110] Figure 2 This is a side view of the mobile device 1. Figure 3 This is a plan view of the moving device 1 as seen from above.

[0111] The distance sensor 113 for tilt direction detection is mounted such that the illumination angle of the laser beam is a predetermined depression angle relative to the horizontal driving surface. The distance sensor 113 for tilt direction detection is positioned at a height of 0.8 meters, a depression angle of 52 degrees, and an elevation angle of 7 degrees, with the driving direction of the main body 10 as the front. The distance sensor 113 performs tilt direction detection in a flipped state. This mounting position is determined to enable detection of the driving surface up to a position in the height direction, in front of the bumper 16 at the front of the moving device 1.

[0112] The distance sensor 113 for tilt direction detection has blind spots in the foot and height directions due to its mounting position and its relationship with the measurement range of the 3D LiDAR. Furthermore, since the distance sensor 113 for tilt direction detection is mounted at the aforementioned position in the mobile device 1, a portion of the distance sensor 113's field of view is blocked by the four rods 19 supporting the sensor mounting base 18, which also provides the blind spot. Figure 3 The blind spot range is shown, and the blind spot distance from the front bumper 16 is 0.325 meters.

[0113] Distance sensor 113 illuminates an object (such as an obstacle) on the road surface with a laser, measures the distance to the object and the direction of the object based on the time it takes for the laser to reflect from the object, and obtains the measurement results as data.

[0114] like Figure 2 As shown, in the vertical direction, the measurement range has a depression angle of 52 degrees and an elevation angle of 7 degrees. As mentioned above, in the horizontal direction, the measurement range is 360 degrees.

[0115] The installation position of the distance sensor 113 has appropriate values, which depend on the width and length of the tracks of the tracked moving bodies 11a and 11b and the size, width, depth and height of the object to be detected.

[0116] Hardware configuration of mobile devices

[0117] refer to Figure 4 This describes the hardware configuration of mobile device 1. Figure 4 This is a block diagram showing the hardware configuration of mobile device 1.

[0118] like Figure 4As shown, the mobile device 1 includes a central processing unit (CPU) 101, a memory 102, an auxiliary memory 103, a camera 111, distance sensors 112 and 113, a navigation satellite system 114, an inertial measurement unit (IMU) 115, a battery 121, motor drivers 122a and 122b (left and right), drive motors 132a and 132b (left and right), brake drivers 123a and 123b (left and right), brake motors 133a and 133b (left and right), a power switch 141, a start switch 142, and an emergency stop switch 143.

[0119] CPU 101 controls the entire operation of the mobile device 1. CPU 101 operates according to the driving program stored in memory 102 to achieve, for example... Figure 5 The obstacle detection unit 155 shown is a functional unit. The CPU 101 operates according to a driving program stored in the memory 102 to control the movement of the mobile device 1 using an obstacle map generated by the map generation unit. The memory 102 is a temporary storage area for programs (such as driving programs) executed by the CPU 101. The auxiliary memory 103 stores data such as the driving program executed by the CPU 101.

[0120] The driving program to be executed on the mobile device 1 can be recorded as a computer-readable recording medium, such as an optical disc read-only memory (CD-ROM), flexible disc (FD), optical disc recordable (CD-R), or digital multifunction disc (DVD), as a file that can be installed or executed by a computer.

[0121] The operating program to be executed on mobile device 1 can be stored on a computer connected to a network such as the Internet, and can be downloaded from the computer via the network. Alternatively, the operating program to be executed on mobile device 1 can be provided or distributed via a network such as the Internet. Furthermore, the operating program to be executed on mobile device 1 can be pre-loaded in, for example, the ROM of mobile device 1.

[0122] Motor drives 122a and 122b are drives for travel motors 132a and 132b, respectively, included in the two tracked moving bodies 11a and 11b. Brake drives 123a and 123b are drives for brake motors 133a and 133b, respectively, included in the two tracked moving bodies 11a and 11b. In the following description, motor drives 122a and 122b may be collectively referred to as motor drive 122, travel motors 132a and 132b may be collectively referred to as travel motor 132, brake drives 123a and 123b may be collectively referred to as brake drive 123, and brake motors 133a and 133b may be collectively referred to as brake motor 133.

[0123] The motor driver 122 and the brake driver 123 receive commands from the CPU 101 to control the drive motor 132 and the brake motor 133, respectively.

[0124] The power switch 141 is a switch that turns the power supply to the moving device 1 on or off. The power switch 141 is the same as the power button 12 mentioned above (see...). Figure 1 The start switch 142 is used to start the tracked moving parts 11a and 11b. The start switch 142 is operated in conjunction with the aforementioned start button 13 (see...). Figure 1 A) Pressing and combining operation. Emergency stop switch 143 is used to stop tracked moving parts 11a and 11b in an emergency. Emergency stop switch 143 is combined with pressing the aforementioned emergency stop button 14 (see...). Figure 1 A) To operate.

[0125] Camera 111 includes a spherical camera, a stereo camera, and an infrared camera. As described above, distance sensor 112 is a 3D LiDAR sensor for horizontal detection. As described above, distance sensor 113 is a 3D LiDAR sensor for tilt detection.

[0126] Navigation satellite system 114 receives radio waves from satellites and measures the position of mobile device 1 on Earth based on the received results. Navigation satellite system 114 uses Real-Time Kinematic (RTK) - Global Navigation Satellite System (GNSS).

[0127] In location determination (positioning) using RTK-GNSS positioning, a high accuracy with an error of approximately a few centimeters is achieved when a high-accuracy positioning solution (Fix solution) in RTK-GNSS positioning is obtained.

[0128] Navigation satellite system 114 includes two antennas for GNSS reception. When there is a certain distance or a greater difference between the position information obtained through the two antennas, the reliability (accuracy) of the position information determined by navigation satellite system 114 is low.

[0129] IMU 115 includes a triaxial accelerometer sensor and a rotational angular velocity sensor. The mobile device 1 uses the measurement data from IMU 115 to detect the tilt of the main body 10 of the mobile device 1, and corrects the height difference relative to the road surface measured by distance sensor 113 based on the tilt of the mobile device 1.

[0130] IMU 115 is preferably located near distance sensor 113. When distance sensor 113 is mounted on body 10 via an arm, IMU 115 is preferably mounted on the arm. IMU 115 also serves as a self-localizing IMU for determining the posture of mobile device 1.

[0131] This configuration eliminates the inconvenience of the height position detected by the distance sensor 113 changing as the body 10 moves back and forth on rough ground.

[0132] Used as a functional unit (such as Figure 5 The CPU 101 of the obstacle detection unit 155 shown detects obstacles based on decision parameters customized for the tracked mobile bodies 11a and 11b of the mobile device 1. These parameters are configured to prevent ditches of a certain width and obstacles of a certain height that the tracked mobile bodies 11a and 11b of the mobile device 1 can pass through from being detected as obstacles.

[0133] The following describes the operations related to the autonomous driving of mobile device 1.

[0134] Figure 5 This is a block diagram illustrating the functional configuration related to the autonomous driving of mobile device 1. For example... Figure 5 As shown, autonomous driving of mobile device 1 involves functions related to self-localization of mobile device 1 using navigation satellite system 114 and functions related to driving control of mobile device 1.

[0135] The self-localization of mobile device 1 is described.

[0136] like Figure 5 As shown, the mobile device 1 includes an odometer unit 151 and a self-positioning unit 152, which are functions related to the self-positioning of the mobile device 1.

[0137] For the self-localization of mobile device 1, high-precision position information from navigation satellite system 114 is used as described above. However, the position update cycle is approximately 200 milliseconds to 1 second and varies considerably. Therefore, odometer unit 151 performs odometer calculations to determine the self-position of mobile device 1 based on the rotation angle ω of the drive motors 132a and 132b that drive the two tracked mobile bodies 11a and 11b, respectively. The rotation angle ω of the drive motors 132a and 132b is obtained from Hall sensor pulses used to drive the drive motors 132a and 132b or from external sensors such as encoders or tachometers.

[0138] The self-localization unit 152 outputs a self-localization result (X, Y, θ). The self-localization unit 152 corrects the mileage information calculated by the odometry unit 151 to achieve stable self-localization. More specifically, the self-localization unit 152 uses position information obtained by the navigation satellite system 114 (including RTK-GNSS) and attitude changes measured by the IMU 115 to correct the mileage information.

[0139] Incidentally, due to factors such as reflective objects, the position information of RTK GNSS may suddenly shift by tens of centimeters to several meters. To prevent such sudden shifts in self-localization from causing unstable operation, when the difference between the position information obtained through the two antennas is within a certain amount, the self-localization unit 152 corrects the mileage information calculated by the odometer unit 151. Conversely, when the difference between the position information obtained through the two antennas is equal to or greater than a certain amount, or when a high-precision positioning solution (Fix solution) in RTK GNSS positioning is not obtained, the self-localization unit 152 performs self-localization using only the calculation results of the odometer unit 151.

[0140] The speed control of the mobile device 1 is described below.

[0141] like Figure 5 As shown, the mobile device 1 includes a global planner unit 153, a local planner unit 154, an obstacle detection unit 155, and a steering control unit 156 to provide functions related to the speed control of the mobile device 1.

[0142] Global planner unit 153 performs global planning to output a sequence of reachable path points WP(n) from the starting point to the destination. Path points are a set of points on the travel route of mobile device 1. In addition to position (x, y, z), path points also have information about direction and velocity (Vmax).

[0143] Based on the path point (WP(n)) information obtained from the global planner unit 153, the local planner unit 154 actually plans the path connecting the path points and calculates the target direction (WP vector), target position coordinates Pct(n), and velocity (Vmax).

[0144] The obstacle detection unit 155 performs obstacle detection or determination based on the detection results of distance sensors 112 and 113. The obstacle detection unit 155 includes a movement amount calculation unit 155a that calculates the movement amount of the mobile device 1. The obstacle detection unit 155 also includes a height map generation unit 155b that generates a height map. The height map stores height information from the driving surface based on the distance to the driving surface and height direction data measured by distance sensor 113. The obstacle detection unit 155 also includes a height history map generation unit 155c that generates a height history map (height history data). Even though distance sensor 113 fails to measure distances in blind spot areas, the height history map stores height information even for blind spot areas by shifting the movement amount of the mobile device 1 from the height information stored in the height map. In other words, the height history map generation unit 155c acts as a height history data generation unit to generate height history data based on the amount of travel of the mobile device 1 during a predetermined time period before the mobile device 1 stops, supplemented by the height information saved before the mobile device 1 stops. The height information indicates the height above the road surface on which the mobile device 1 travels. A predetermined time period can be stored in memory, for example, by the manufacturer based on experience data. The obstacle detection unit 155 outputs an avoidance direction (θ), a speed command (V), and a stop command (V) to the steering control unit 156 based on detected obstacles. The obstacle detection unit 155 also uses the results of obstacle detection to generate an obstacle map of the road surface on which the mobile device 1 travels.

[0145] The steering control unit 156 calculates the speed and angular velocity of the moving device 1, and calculates the speeds (VL and VR) of the right track moving body 11a and the left track moving body 11b for steering control. The steering control unit 156 transmits the speeds (VL and VR) of the right track moving body 11a and the left track moving body 11b to the serial interface (I / F) 157 and the motor drivers 122a and 122b to control the travel motors 132a and 132b.

[0146] The following describes the functional configuration related to speed control performed by the steering control unit 156. Figure 6 This is a block diagram showing the functional configuration related to speed control performed by the steering control unit 156.

[0147] Mobile device 1 is a tracked robot that independently drives the right tracked mobile body 11a and the left tracked mobile body 11b. Mobile device 1 rotates by the speed difference between the right tracked mobile body 11a and the left tracked mobile body 11b. In order to travel stably on various road surfaces with different driving loads, the left tracked mobile body 11a and the right tracked mobile body 11b rotate at a specified speed, regardless of the load.

[0148] Therefore, such as Figure 6 As shown, the steering control unit 156 performs driving control of the moving device 1 in separate loops: a main loop for controlling steering and a secondary loop for controlling the speed of the left tracked moving body 11a and the right tracked moving body 11b.

[0149] like Figure 6 As shown, the steering control unit 156 includes a position proportional-integral-derivative (PID) unit 161, speed PID units 162a and 162b, and speed calculation units 163a and 163b.

[0150] The position PID unit 161 calculates the operating quantities of the left track moving body 11a and the right track moving body 11b for steering control based on the difference (position deviation) between the set value and the measured value.

[0151] Speed ​​PID units 162a and 162b obtain the change in the operating quantity from the speed deviation.

[0152] Speed ​​calculation units 163a and 163b determine the motor speed based on signals (motor pulses) from Hall sensors attached to drive motors 132a and 132b.

[0153] In the main loop, the steering control unit 156 calculates the difference (position deviation) between the self-localization result and the target position (Pset) and the target direction (θset) to the target point (Pset). To obtain the self-localization result, the odometer unit 151 uses motor pulses from the drive motors 132a and 132b to perform odometer calculation, and the self-localization unit 152 uses position information obtained from the navigation satellite system 114, including RTK-GNSS, and attitude changes (θgps) and velocity changes (Vgps) measured by the IMU 115 to correct the odometer result.

[0154] More specifically, the steering control unit 156 outputs the calculated difference (position deviation) to the position PID unit 161.

[0155] The position PID unit 161 calculates the operating quantities VLset and VRset (indicating speed V and rotation angle ω of travel motors 132a and 132b) of the left tracked moving body 11a and the right tracked moving body 11b based on the difference (position deviation) between the set value and the measured value for steering control.

[0156] The mobile device 1 can travel stably via the left tracked mobile body 11a and the right tracked mobile body 11b controlled by the operation quantities VLset and VRset from the main circuit.

[0157] Furthermore, in the secondary loop, the steering control unit 156 uses speed calculation units 163a and 163b to determine the motor speed based on signals (motor pulses) from Hall sensors attached to the drive motors 132a and 132b. Then, based on the difference (speed deviation) between the speed determination result from the speed calculation units 163a and 163b and the target speed, the steering control unit 156 uses speed PID units 162a and 162b to control the voltage applied to the drive motors 132a and 132b to perform speed feedback control. The purpose of this speed feedback control in the secondary loop is to minimize speed fluctuations caused by disturbance torque on the drive motors 132a and 132b.

[0158] Obstacle detection by mobile devices

[0159] The following describes the detection of obstacles on the road surface on which the mobile device 1 travels under the control of autonomous driving of the mobile device 1.

[0160] The detection of obstacles on the road surface on which the mobile device 1 travels is summarized below. The obstacle detection unit 155 of the mobile device 1 performs vibration compensation based on planar detection of data obtained from a distance sensor 113. The distance sensor 113 is a 3D LiDAR sensor mounted at an angle downwards. The obstacle detection unit 155 then overlays the corrected data over time to obtain detailed road surface data. The mobile device 1 converts the vibration-corrected detailed road surface data into a two-dimensional height map, where the maximum height is stored as height information in the unit, allowing for the determination of height relationships in the front-back, left-right, and tilt directions. Obstacles are then identified from the height differences in the height map. The height map is a two-dimensional map that records height information of the road surface on which the mobile device 1 (such as a vehicle or robot) travels.

[0161] Figure 7 This is a diagram showing the outline of the curb detection. The moving device 1 will obtain the data from the distance sensor 113. Figure 7 The raw data from the 3D LiDAR shown in (a) is accumulated for 2 seconds, and then generated. Figure 7 The elevation map shown in (b) is used by mobile device 1 to determine whether a protrusion is a curb based on differences in elevation and smoothness. More specifically, when in... Figure 7 When the following conditions are met on the height map in (b) of the mobile device 1, the obstacle detection unit 155 will detect the protruding part as a curb. The conditions are: a) the height difference from the ground is equal to or greater than a threshold (detection based on the height obstacle map); and b) the difference value from adjacent cells on the two-dimensional map is equal to or greater than a threshold (detection based on the difference obstacle map).

[0162] Figure 8This is a diagram showing the outline of the ditch detection. The moving device 1 will obtain the data from the distance sensor 113. Figure 8 The raw data from the 3D LiDAR shown in (a) is accumulated for 2 seconds, and then generated. Figure 8 The height map is shown in (b). The obstacle detection unit 155 uses the height map data to cluster two-dimensional map cells whose height difference with the ground is equal to or less than a threshold to determine whether a depression is a ditch. Specifically, the obstacle detection unit 155 of the mobile device 1 clusters two-dimensional map cells whose height difference with the ground is equal to or less than a threshold, and determines the cluster of cells in the two-dimensional map cells whose difference value relative to adjacent cells is equal to or greater than the threshold as a ditch. When a ditch candidate cluster has one or more high edge information, the entire ditch candidate cluster is determined to be a ditch.

[0163] This process allows aerial objects to be detected as obstacles, in addition to road surface obstacles such as curbs and ditches. Furthermore, obstacles in front of the mobile device 1 or obstacles at steep approach angles are detected. Essentially, the obstacle detection unit 155 determines the presence of obstacles based on the generated height map, but performs different processes for protruding obstacles such as curbs and recessed obstacles such as ditches. In the following description, protruding obstacles are referred to as curb obstacles, while recessed obstacles are referred to as ditch obstacles.

[0164] Figure 9A and Figure 9B These are diagrams showing the results of obstacle detection on the road surface. Figure 9A The results of the curb detection are shown, and Figure 9B The results of the ditch inspection are shown. Figure 9A The results of curb detection at an approach angle of 45 degrees are shown, and Figure 9B The results of the groove detection are shown. Black cells on the 2D map indicate road obstacles.

[0165] Different processes are executed to detect protruding and dented obstacles, and the processing results are integrated to detect both protruding and dented obstacles on the road surface. The detailed process of obstacle detection is described below with reference to the flowchart.

[0166] Figure 10 This is a flowchart of the process of detecting obstacles on the road surface performed by the mobile device 1.

[0167] like Figure 10As shown, the obstacle detection unit 155 sequentially executes the following steps to detect obstacles: storing point cloud data (step S1), generating a height map (step S2), processing blind spots (step S3), generating a height obstacle map (step S4), generating a differential obstacle map (step S5), generating a ditch obstacle marker map (step S6), and generating an obstacle map (step S7). Each process will be described in detail below.

[0168] The process of storing point cloud data is described below (step S1). Figure 11 This is a flowchart of the process of storing point cloud data.

[0169] refer to Figure 11 In step S11, the obstacle detection unit 155 begins to be scanned by the distance sensor 113 to detect tilt direction.

[0170] In step S12, the obstacle detection unit 155 converts the point cloud data obtained by the distance sensor 113 into robot coordinates of the mobile device 1 and generates road surface point cloud data. In this way, based on the data obtained from the three-dimensional distance sensor, the obstacle detection unit 155 generates road surface data, which is three-dimensional data about the mobile device 1 on the road surface.

[0171] In step S13, the obstacle detection unit 155 uses mileage information to move the stored road surface point cloud data. For example, when the mobile device 1 is not moving, mileage information may not be available.

[0172] In step S14, the obstacle detection unit 155 performs planar detection using the acquired road surface point cloud data and calculates the ground tilt based on the planar detection results. Specifically, the obstacle detection unit 155 extracts the tilt within a predetermined rectangular range from the acquired road surface point cloud data (e.g., vertical length x is 1.0 meter and horizontal length y is 1.4 meters), and performs planar detection using the extracted data. The obstacle detection unit 155 calculates the pitch angle based on the detected plane and outputs the pitch angle as the ground tilt.

[0173] In step S15, the obstacle detection unit 155 uses the calculated ground tilt to correct the tilt of the obtained road surface point cloud data.

[0174] In step S16, the obstacle detection unit 155 removes the road point cloud data within the rectangle of the mobile device 1 from the obtained road point cloud data that has been corrected for tilt, so as to exclude the sensor data corresponding to the mobile device 1 itself.

[0175] In step S17, the obstacle detection unit 155 stores the road surface point cloud data that has undergone tilt correction and the aforementioned removal. The obstacle detection unit 155 stores the data in the stored road surface point cloud data, but only saves the most recent 2 seconds of road surface point cloud data and deletes older data.

[0176] Using the above process, the back-and-forth swaying of the mobile device 1 traveling on rough ground is corrected, and corrected road surface data as detailed road point cloud data can be obtained.

[0177] The process of generating an elevation map is described below (step S2). Figure 12 This is a flowchart of the process of generating a height map. As described above, the height map generation unit 155b of the obstacle detection unit 155 generates a height map that stores height information from the driving surface based on the distance to the driving surface and the height orientation data measured by the distance sensor 113. Figure 12 As shown, using the stored road surface point cloud data generated as described above, an elevation map is generated in the following sequence.

[0178] like Figure 12 As shown, in step S21, the height map generation unit 155b of the obstacle detection unit 155 uses a pass-through filter to extract road surface point cloud data within the detection range.

[0179] When viewed from the front of bumper 16 (see...) Figure 1 The detection range is a rectangular area with a vertical length x of 2.0 meters and a horizontal length y of 2.8 meters from the center of the robot.

[0180] In step 22, the obstacle detection unit 155 generates voxel mesh data using road surface point cloud data within the detection range. The resolution is 2.5 cm.

[0181] In step 23, the obstacle detection unit 155 performs noise removal using a radius outlier filter. Figure 13 This is a diagram illustrating the overview of noise removal. Specifically, if, for example, the number of other data points within a radius d meters of a data point is less than n, then the obstacle detection unit 155 removes that data point, such as... Figure 13 As shown. The removed data points are in Figure 13 The middle is indicated by a cross mark.

[0182] In step 24, obstacle detection unit 155 generates a height map using noise-removed voxel mesh data. The resolution is 5.0 cm in the xy plane and 2.5 cm in the height (z). For example, the maximum value of the corresponding z-value of the point cloud data included in a voxel is stored as the height value of that voxel. Obstacle detection unit 155 generates a map in which the maximum value of the corresponding z-value of multiple voxels with certain xy values ​​is stored as the height value of the location indicated by the xy values. The resulting map is a height map in which the maximum height above the driving surface is stored on a two-dimensional map.

[0183] The above process produces a height map from which noise has been removed. Due to the reduction in the number of point clouds, this height map has the effect of reducing processing load and increasing processing speed, and because of its relatively coarse resolution, it allows the mobile device 1 to travel on the fence.

[0184] When navigating outdoors, mobile device 1 needs to detect ditches as road obstacles to avoid the risk of falling. However, in many locations, a mesh ditch grid (fence) is installed to allow mobile devices to travel. Since mobile device 1 can travel on the fence, it is desirable to detect the fence as a drivable area that is not a road obstacle. If the point cloud is observed closely, the mesh fence has flat sections and sections that are incorrectly detected as ditches. Incorrect detection of ditches can prevent mobile device 1 from traveling on the fence. When the 2D map is converted to a height map with a relatively coarse cell resolution, the point groups are not locally observed, which allows the fence to be detected as a drivable road surface without being incorrectly detected as a ditch. For example, considering the size of the tracks of mobile device 1, the resolution of the height map is set to 5.0 cm.

[0185] The noise removal in step S23 will be described in detail below.

[0186] Figure 14A and Figure 14B These are graphs showing the results of noise removal. In rain, the 3D LiDAR sensor, acting as distance sensor 113, may erroneously detect raindrops. As rainfall increases, such as Figure 14A As shown, point cloud data scattered in the air is incorrectly detected even though there are no obstacles in the air. If the elevation map is converted as is, raindrops may be detected as obstacles, and the mobile device 1 may stop or take an avoidance action even when there are no obstacles.

[0187] Therefore, execution Figure 13 The noise removal shown is used to remove raindrops as noise, so that obstacle detection can function even in the rain.

[0188] like Figure 13The noise removal shown is the process of removing data points when the number of data points within the radius d meters of the target point is less than n. Figure 14B The following is an example of the operation performed with a radius d of 0.06 meters and a data point count n of 2. Figure 13 The result of noise removal. Figure 14B In the results, with Figure 14A Compared to the results in the previous example, raindrops in the air are removed. By removing noise to eliminate erroneous detection data caused by raindrops, the mobile device 1 can cope with rain, that is, perform obstacle detection in the rain.

[0189] Figure 15A and Figure 15B Each shows the result of Figure 1 The diagram shows the height map obtained by the mobile device 1 performing the above-described process. Figure 15A An altitude map obtained by the stationary mobile device 1 is shown, and Figure 15B An altitude map obtained by the moving device 1 is shown.

[0190] Because the distance sensor 113 obtains 360-degree point cloud data, such as Figure 15A and Figure 15B As shown, the detection coverage DR of the obstacle is therefore (see Figure 3 The distance sensor 113 acquires point cloud data of the entire surrounding environment of the mobile device 1 and can detect obstacles on the side or behind. Therefore, taking into account the trajectory when an obstacle is detected on the lateral side or rear, the mobile device 1 can safely avoid the obstacle or stop.

[0191] In other words, since the mobile device 1 detects obstacles on the lateral side or rear, the mobile device 1 can perform an avoidance operation on the lateral side or rear, thereby improving safety.

[0192] As described above, the height map is a two-dimensional graph representing the height relationships in the forward / backward, left / right, and tilt directions. The height map stores the maximum height obtained from the distance sensor 113, which functions as a 3D LiDAR, in cells with a specific resolution. Figure 15A and Figure 15B An elevation map generated using data acquired from distance sensor 113 and accumulated over 2 seconds is shown.

[0193] Figure 15A The radial black area in the middle represents Figure 3 The blind spot area is shown in the diagram. Such a blind spot exists when the mobile device 1 stops. Conversely, since the mobile device 1 is moving while accumulating data from the distance sensor 113 for 2 seconds, therefore... Figure 15B As shown, blind spot areas have been largely eliminated.

[0194] like Figure 15B As shown, the mobile device 1 detects an obstacle in the blind spot area while driving. However, when the mobile device 1 stops, the detection state returns to normal. Figure 15A The state described in the document. Then, the blind spot area reappears, and no obstacle in the blind spot area is detected. If an obstacle exists in the blind spot area in this state, the obstacle is not detected. Then, when it is determined that there is no obstacle in the blind spot area, the moving device 1 may collide with the obstacle.

[0195] To prevent this, mobile device 1 handles blind spots in obstacle detection by storing data spatially rather than temporally. The handling of blind spots in obstacle detection by mobile device 1 will be described below.

[0196] Figure 16 This is a diagram illustrating an overview of blind spot handling in obstacle detection. (As shown) Figure 16 As shown, the mobile device 1 shifts previously acquired sensor data by the amount of movement of the mobile device 1, and stores only the history of sensor data in blind spots. The mobile device 1 uses an elevation history map that includes the history of the stored sensor data to perform obstacle detection. The elevation history map will be described later. In this way, obstacle detection can be performed in blind spot areas taking into account the surrounding environment, and the detection accuracy is improved.

[0197] When only stationary objects need to be detected, but moving objects such as people or vehicles are also present in the outdoor environment, blind spot processing in obstacle detection can be omitted. When a moving object moves back and forth in a blind spot, residual images of the moving object may still remain in the blind spot area. If detection is performed in this state, the detection may be erroneous. To prevent this, the mobile device 1 performs identification and tracking of the moving object, erases only historical data in the moving object area, and uses real-time tracking data in the moving object area as a countermeasure against residual moving objects.

[0198] For cases involving residual images, a client-server system is used that allows invocation of an altitude data history clearing service from external devices. When the client-server system allows invocation of the procedure for clearing altitude data history, countermeasures against residual images in blind spots can be executed from external devices.

[0199] The above process addresses blind spots in obstacle detection to prevent erroneous detections in blind spot areas. The mobile device 1 performs blind spot processing in obstacle detection by preparing the following two processes ( Figure 10 Step S3 in the process.

[0200] [Requirement 1]

[0201] Always as a process of algorithm execution

[0202] [Requirement 2]

[0203] The process of calling a client-server system from an external device

[0204] The following flowchart will be used to describe these two processes.

[0205] Figure 17 It is always a flowchart of the process of algorithm execution.

[0206] like Figure 17 As shown, the obstacle detection unit 155 uses three data points—a height map generated based on data from a distance sensor 113 (which is a 3D LiDAR), a height history map storing height data, and moving object recognition data—to sequentially perform the generation of the height history map in step S101 and the removal of the moving object region in step S102.

[0207] The process of generating the height history map in step S101 is described below. The obstacle detection unit 155 includes a height history map generation unit 155c. Taking into account the height information stored in the height map by the amount of movement of the mobile device 1, the height history map stores height information even for blind spot areas where the distance sensor 113 failed to measure the distance.

[0208] Figure 18 This is a flowchart of the process for generating historical elevation maps. For example... Figure 18 As shown, the height history map generation unit 155c of the obstacle detection unit 155 performs the height history map generation process using the height history map data in the following order.

[0209] like Figure 18 As shown, in step S1011, the obstacle detection unit 155 uses the mileage information (the amount of movement of the mobile device 1) calculated by the movement calculation unit 155a to move the altitude history map data by the amount of mileage. The obstacle detection unit 155 excludes data outside the driving area. However, initially, the data is not moved because there is no data in the altitude history map. When the data is in the altitude history map, the data is moved a second time and subsequently.

[0210] In step S1012, the obstacle detection unit 155 stores the current height map in the height history map, thereby updating the height history map. More specifically, when height data is stored in a cell of the height map, the obstacle detection unit 155 stores the current height data in a cell with the same cell number to update the height history map. Using the update process, the obstacle detection unit 155 generates the height history map, in which the height data is stored in the blind spot area.

[0211] The process of removing the moving object region in step S102 will be described below.

[0212] Figure 19 This is a flowchart of the process of removing a moved object region. For example... Figure 19 As shown, the obstacle detection unit 155 performs moving object region removal processing on the height history map in the following order: it removes the height data of the moving object region based on the identification and tracking results of the moving object. Therefore, the obstacle detection unit 155 eliminates the influence of blind spots caused by moving objects.

[0213] like Figure 19 As shown, in step S1021, the obstacle detection unit 155 uses the height history map generated in step S101 and the moving object identification data to perform moving object region removal processing. More specifically, the obstacle detection unit 155 compares the rectangular region of the identified moving object with the height history map and removes the height information from the rectangular region of the moving object.

[0214] In other words, obstacle detection unit 155 performs moving object region removal processing to remove obstacles that are residual images of moving objects in blind spots from the elevation history map. Therefore, obstacle detection unit 155 minimizes the impact of moving objects on the elevation history map.

[0215] The following describes the process of calling a client-server system from an external device.

[0216] The client-server system described herein includes an information processing device (external device) used as a client for a mobile device 1 with a remote control system.

[0217] Figure 20 This is a flowchart of the client-server process for using the height history map clearing service.

[0218] like Figure 20 As shown, the obstacle detection unit 155 performs height history map clearing processing in the following order using height history map data in client-server processing.

[0219] like Figure 20 As shown, in step S201, the obstacle detection unit 155 clears (deletes) all data in the height history map according to the instruction from the client (information processing device).

[0220] In step S202, the obstacle detection unit 155 stores the processing completion status as a response result to the client (information processing device).

[0221] As described above, in the event of any remaining residual images, a process for clearing the history of altitude data is invoked from an external device. Therefore, the erasure of data from the historical altitude map is performed from an external device, rather than within the algorithm itself. This is a countermeasure against residual images of obstacles remaining in blind spots during operation. The service is described below.

[0222] Figure 21 This is a sequence diagram illustrating the process of invoking the historical elevation map clearing service. Figure 21 During the process shown, the user can invoke the height history map clearing service at any desired time.

[0223] Figure 21 The process described herein involves the operation of a mobile device 1 equipped with a remote control system. The mobile device 1 allows the user to send various commands via a remote control system application (hereinafter referred to as a remote control application). At this time, such as... Figure 21 As shown, users can also invoke the altitude history map clearing service via a remote control application.

[0224] like Figure 21 As shown, in step S301, the user presses the altitude history map clearing button displayed on the remote control application. In step S302, the remote control application sends an altitude history map clearing command to the mobile device 1.

[0225] In step S303, the altitude history map clearing service is invoked from an external source (remote control application), and mobile device 1 executes the altitude history map clearing process. After the altitude history map clearing process is completed, in step S304, mobile device 1 sends a completion notification to the remote control application. This allows the user to recognize the completion of the altitude history map clearing process.

[0226] This configuration allows users to perform a process of clearing historical map data from an external source (remote control application) at any desired time when the remaining residual imagery is available.

[0227] For example, suppose a user manually drives mobile device 1 via remote control. Suppose that during manual driving, mobile device 1 detects a moving object in obstacle detection, including blind spot handling, and safely stops manual driving without colliding with the obstacle due to the influence of the moving object in the blind spot.

[0228] When a moving object is moved away and safety is confirmed, a residual image of the moving object may still remain in the blind spot area. If an obstacle is detected due to the influence of the residual image, the mobile device 1 will not move even if a driving command is sent. In this case, when the user confirms that there are no obstacles in the blind spot area, the altitude history map can be cleared by invoking the aforementioned service via a remote control application. Manual driving can then resume when safety is confirmed in the blind spot area.

[0229] In the following description, mobile device 1 responds to a remote control application that issues commands for autonomous driving of mobile device 1. When a user issues commands related to autonomous driving, a command to clear the altitude history map can be automatically sent before the commands are sent to mobile device 1.

[0230] Figure 22 This is a sequence diagram of another process that invokes the historical map cleanup service. Figure 22 During the process shown, the system automatically invokes the altitude history map clearing service. Figure 22 It shows a series of actions from, for example, the start of autonomous driving to the interruption, restart, and completion of autonomous driving.

[0231] like Figure 22 As shown, in step S401, when the user presses the autonomous driving start button of the remote control application (web application), in step S402, the remote control application sends an autonomous driving start command to the mobile device 1. In step S403, the mobile device 1 starts autonomous driving and sends an autonomous driving start notification to the remote control application.

[0232] When the destination is reached in step S404, in step S405, the mobile device 1 performs obstacle detection, including processing of blind spots.

[0233] When mobile device 1 stops for a period of time due to road surface obstacles such as curbs or ditches, in step S406, mobile device 1 itself interrupts autonomous driving. In step S407, mobile device 1 sends an interruption notification to the remote control application. This allows the user to recognize the interruption of autonomous driving of mobile device 1.

[0234] In step S408, the user presses the manual driving command button on the remote control application. In step S409, the remote control application sends a manual driving command to the mobile device 1. The user then manually moves the mobile device 1 to a safe location where there are no obstacles in the blind spot of the mobile device 1.

[0235] In step S410, when the mobile device 1 moves to a safe position with no obstacles in its blind spot, the user presses the restart autonomous driving button on the remote control application. In step S411, the remote control application sends an altitude history map clearing command to the mobile device 1 before sending the autonomous driving restart command.

[0236] In step S412, mobile device 1 executes an altitude history map clearing process in response to the altitude history map clearing command. After completing the altitude history map clearing process, in step S413, mobile device 1 sends a completion notification to the remote control application.

[0237] In response to the completion notification, in S414, the remote control application sends an autonomous driving restart command to the mobile device 1 to control the mobile device 1 to restart autonomous driving.

[0238] When the device moves to the destination in step S415, in step S416 the moving device 1 performs obstacle detection, including processing of blind spots.

[0239] When mobile device 1 reaches its destination in step S417, it transmits a notification of completion of autonomous driving to the remote control application in step S418. This allows the user to recognize that mobile device 1 has reached its destination and that autonomous driving has been completed.

[0240] Advantageously, after the user moves the mobile device 1 to a safe location free of obstacles in the blind spot and before resuming autonomous driving, a height history map clearing command can be sent, as in the case described above. Specifically, even if the residual image of the moved object still acts as an obstacle in the blind spot area, the height history map can be cleared, and autonomous driving can resume while ensuring safety in the blind spot area. In this case, the system automatically invokes the service, which is effective because the user can use the service without being aware of it.

[0241] The above process is in Figure 10 The blind spot processing in obstacle detection performed by the mobile device 1 in step S3 shown is executed.

[0242] The process of generating a height obstacle map in step S4 is described below. Figure 23 This is a flowchart of the process of generating a map of height obstacles. For example... Figure 23 As shown, the following processing is performed using the height history map during the generation of the height obstacle map.

[0243] like Figure 23 As shown, in step S41, obstacle detection unit 155 sets a threshold of ±6 cm and generates a height obstacle map. More specifically, obstacle detection unit 155 sets a value of 100 for cells above the +6 cm threshold as ordinary obstacles in the 2D map, sets a value of 99 for cells below -6 cm as ditch obstacles in the 2D map, sets a value of -1 for cells without point cloud data as so-called unknown obstacles, and sets a value of 0 for other cells as drivable cells. This process produces a height obstacle map where cells are classified into four types: ordinary obstacle cells, ditch obstacle cells, unknown obstacle cells, and drivable area cells. Conditional branches of the process can be used to assign values ​​to cells in the 2D map.

[0244] The process of generating a differential obstacle map in step S5 is described below. Figure 24 This is a flowchart of the process of generating a differential obstacle map. For example... Figure 24 As shown, during the generation of the differential obstacle map, the height history map is used to perform the processing in the following order.

[0245] like Figure 24 As shown, in step S51, the obstacle detection unit 155 corrects unknown obstacle units in the height history map. The height history map contains so-called unknown obstacle units, which lack point cloud data due to factors such as occlusion by obstacles. The obstacle detection unit 155 stores the average height of four adjacent units on the two-dimensional map in each unknown obstacle unit to correct for the unknown obstacle unit. When all four adjacent units are unknown obstacle units, the obstacle detection unit 155 stores a height value of 0.

[0246] In step S52, the obstacle detection unit 155 performs median filtering on the corrected height history map.

[0247] In step S53, the obstacle detection unit 155 performs Laplace filtering on the height history map that has already undergone median filtering correction. Therefore, a difference map is generated.

[0248] In step S54, the obstacle detection unit 155 sets the threshold to ±0.20 and uses a differential map generated by a Laplace filter to generate a differential obstacle map. On the two-dimensional map, cells with values ​​greater than the threshold +0.20 or with values ​​less than the threshold -0.20 are assigned the value 100 as differential obstacle cells, and other cells are assigned the value 0 as drivable area cells.

[0249] The above process generates a differential obstacle map, where cells are classified into two types: differential obstacle cells and drivable area cells. The differential map is a graph in which the height difference with neighboring cells on the height history map is represented as a numerical value. The greater the height difference with neighboring cells, the larger the value assigned to that cell. For example, in areas where neighboring cells have abrupt height differences (such as curbs, i.e., areas with vertical edges), the value is larger. As a result, the smoothness of the height history map is identified.

[0250] The correction in step S51 is used to perform median filtering, and the median filtering in step S52 is used to smooth the height changes in the height history map caused by noise from the distance sensor 113.

[0251] The result obtained by classifying cells in a differential map with thresholds is called a differential obstacle map. Cells with values ​​equal to or greater than the threshold indicate sections such as curbs with upright edges. Such sections are candidates for obstacles.

[0252] The process of generating a ditch obstacle marker map in step S6 is described below. Figure 25 This is a flowchart of the process of generating a map showing the markings of obstacles in the ditch. For example... Figure 25 As shown, in the process of generating a ditch obstacle marker map, the height obstacle map data is processed in the following order.

[0253] like Figure 25 As shown, in step 61, the obstacle detection unit 155 performs an extension process on the height obstacle map. More specifically, when the eight neighboring units around an unknown obstacle unit include ditch obstacle units on the height obstacle map, the obstacle detection unit 155 converts the target unknown obstacle unit into a ditch obstacle unit.

[0254] In step S62, obstacle detection unit 155 generates a ditch obstacle marking map using a height obstacle map in which the ditch obstacle extends into the unknown obstacle unit. More specifically, obstacle detection unit 155 marks eight adjacent units of the ditch obstacle unit.

[0255] The above process generates markers (clustering) of ditch obstacles on the height obstacle map.

[0256] The extension process in step S61 is performed under the assumption that the ditch obstacle unit and its adjacent unknown obstacle unit indicate the same ditch, because when the 3D LiDAR sensor detects a ditch, the cause of the unknown obstacle unit is usually a blockage. Therefore, the ditch obstacle extends into the unknown obstacle unit.

[0257] The process of generating an obstacle map in step S7 is described below. Figure 26 This is a flowchart of the process of generating an obstacle map. For example... Figure 26 As shown, in the process of generating the obstacle map, data from three maps (i.e., the height obstacle map, the differential obstacle map, and the ditch obstacle marker map) obtained through the process so far are processed in the following order.

[0258] like Figure 26 As shown, in step S71, the obstacle detection unit 155 integrates the height obstacle map data, the differential obstacle map data, and the ditch obstacle marker map data to generate an obstacle map. More specifically, the obstacle detection unit 155 observes the same cells in the differential obstacle map and the height obstacle map. When a cell has a value of 99 (ditch obstacle) or 100 (curb obstacle) in both maps, the obstacle detection unit 155 performs an AND operation to set the cell as an obstacle cell. At this time, in the case of ditch obstacles, all cells marked as obstacles are converted into obstacle cells.

[0259] In step S72, the obstacle detection unit 155 performs additional processing on the obstacle map to detect ditch obstacles. The additional processing detects ditch obstacles based on the height difference between the target cell and adjacent cells on the historical height map.

[0260] Figure 27 This is a diagram showing an overview of the additional processing applied to the obstacle map. Figure 28A This is a diagram showing the results of obstacle detection without additional processing of the obstacle map. Figure 28B This is a diagram showing the results of obstacle detection, including additional processing. The obstacle detection unit 155 calculates the average height of three units (i.e., the target unit, the left unit, and the right unit) in the lateral direction (Y direction) of the moving device 1. The obstacle detection unit 155 uses the calculated average height to compare the average heights of the three units in the travel direction (X direction) of the moving device 1. Figure 27 As shown, when the difference between the maximum and minimum average heights of the three units is greater than, for example, 6 centimeters, the obstacle detection unit 155 determines that the unit is an obstacle unit.

[0261] When only the above processing is performed, such as Figure 28A As shown, the sensitivity for detecting obstacles in the ditch ahead is low. Further processing of the obstacle map in step S72 is performed to increase the detection sensitivity. (See figure) Figure 28B As shown, additional processing of the obstacle map enables high-sensitivity detection of ditch obstacles ahead.

[0262] The above process generates an obstacle map that includes both ditch obstacles and curb obstacles.

[0263] In step S71, since obstacles are determined based on height difference and smoothness, the height obstacle map corresponding to the height difference and the differential obstacle map corresponding to the smoothness are integrated. In some cases, ditch obstacles may not be detected by AND operation of height difference and smoothness. Therefore, labeled (clustered) map data is used for ditch obstacles. When any cell on the same labeled map is subjected to an AND operation with an edge cell on the differential obstacle map, all cells with that label are identified as ditch obstacle cells.

[0264] As described above, when the mobile device 1 stops, the height map is supplemented with height information based on the amount of movement made by the mobile device 1 during a predetermined time period until it stops. This process preserves height information even in blind spots where the 3D distance sensor fails to measure distance. This enables the detection of obstacles in blind spots caused by obstructions, such as those caused by the mobile device's housing, or in areas outside the sensor's field of view, and allows the mobile device to avoid obstacles. The phrase "when the mobile device stops" refers not only to when the mobile device stops, but also to when the mobile device begins to move from a stopped state to resume operation.

[0265] Refer to the above Figures 1 to 28B In the description, the obstacle detection system is applied to a mobile device 1, which includes a relatively small tracked mobile body that travels on rough ground and is used for equipment inspection in facilities such as factories.

[0266] In the following description, the obstacle detection system is applied to a vehicle with autonomous driving capabilities. References are omitted in the following description. Figures 1 to 28B The description includes elements that are the same as or similar to the elements, and also describes the differences.

[0267] In autonomous driving, onboard stereo cameras are typically used to capture images of the surrounding environment in the forward or rearward directions to detect obstacles or anomalies. Autonomous driving also aims to detect obstacles or anomalies in the lateral direction. One approach would be to install numerous sensors on the sides of the car to eliminate blind spots, but this increases costs.

[0268] Figure 29 This is a diagram showing a car 200 as an obstacle detection system. (See diagram for reference.) Figure 29 As shown, the automobile 200, as a mobile device, includes a sensor mount 18 attached to the roof 201 via a rod 19. A distance sensor 112 for horizontal detection and a distance sensor 113 for tilt detection are mounted on the sensor mount 18.

[0269] The vehicle 200, as an obstacle detection system, has the following effect: In autonomous driving, even when the vehicle 200 is stopped, it performs obstacle detection in areas outside the blind spot or the observation range of the sensors due to obstructions such as those caused by the vehicle's housing.

[0270] The application of obstacle detection systems is not limited to cars with autonomous driving capabilities, but also to mobile devices with autonomous driving capabilities, such as various vehicles, aircraft, ships and robots.

[0271] Refer to the above Figures 1 to 28BIn the description, measures for blind spot areas are taken by only one mobile device 1, which includes a tracked moving body.

[0272] Alternatively, accumulated data such as elevation maps and historical elevation maps can be stored on cloud servers and shared with other mobile devices such as tracked robots and vehicles, as described below. In the following description, references to [other technologies / methods] are omitted. Figures 1 to 29 The description includes elements that are the same as or similar to the elements, and also describes the differences.

[0273] Figure 30 This is a diagram showing the configuration of the obstacle detection system 300. (As shown) Figure 30 As shown, the obstacle detection system 300 includes multiple mobile devices 301 and a cloud server 303 connected to the mobile devices 301 via a network 302 such as the Internet. Examples of mobile devices 301 include a mobile device (tracked robot) 1 and a car 200.

[0274] The cloud server 303 stores accumulated data, such as height maps and historical height maps based on road surface data obtained from the distance sensor 113 of the mobile device 301. Even for blind spots where the distance sensor 113 fails to measure distance, the historical height map stores height information.

[0275] For example, an obstacle detection unit 155 of one mobile device 301 sequentially stores accumulated data, such as a height map based on road surface data obtained from a distance sensor 113 and a historical height map that stores height information even for blind spot areas where the distance sensor 113 failed to measure distance, in a cloud server 303 via a network 302. Therefore, the data accumulated in the cloud server 303 is shared and kept up-to-date. An obstacle detection unit 155 of another mobile device 301 shares the accumulated data stored in the cloud server 303 via the network 302.

[0276] Furthermore, for example, the obstacle detection unit 155 of one mobile device 301 stores data on blind spot areas where the distance sensor 113 failed to measure distance in a cloud server 303 via a network 302, for sharing with another mobile device 301. The obstacle detection unit 155 of the other mobile device 301 supplements the blind spot area data by sharing the blind spot area data stored in the cloud server 303 via the network 302.

[0277] In other words, the obstacle detection unit 155 of one of the mobile devices 301 stores at least one of the elevation map, the historical elevation map, and the blind spot area data in the cloud server 303 for sharing with the other mobile device 301. When the obstacle detection unit 155 of one of the mobile devices 301 stores at least one of the elevation map and the historical elevation map as accumulated data other than the blind spot area data in the cloud server 303, the obstacle detection unit 155 of the other mobile device 301 can obtain only the blind spot area data to reduce the load on the network 302.

[0278] The aspects of this disclosure are as follows, for example.

[0279] According to a first aspect, an obstacle detection system for detecting obstacles includes: a three-dimensional distance sensor for measuring distances in three dimensions to measure distances to a road surface on which a mobile device travels and for measuring distances in the height direction; and an obstacle detection unit for detecting obstacles on the road surface when the mobile device stops.

[0280] The obstacle detection unit includes a height history data generation unit, which generates height history data based on the distance traveled by the mobile device during a predetermined period until the mobile device stops, supplemented by height information stored until the mobile device stops. The height information represents the height above the road surface.

[0281] According to the second aspect, in the obstacle detection system according to the first aspect, the height history data generation unit compares the rectangular area of ​​the identified moving object with the height history data, and removes the height information from the rectangular area of ​​the identified moving object.

[0282] According to the third aspect, in the obstacle detection system according to the first aspect, the height history data generation unit deletes height information from the height history data in response to an instruction from an external device.

[0283] According to the fourth aspect, in the obstacle detection system according to any one of the first to third aspects, a three-dimensional distance sensor references the moving device to generate three-dimensional road surface data as a measurement result; and the obstacle detection unit generates height map data based on the road surface data, the height map data including the maximum height from the road surface as height information, and provides the height map data to the height history data generation unit.

[0284] According to the fifth aspect, in the obstacle detection system according to the fourth aspect, road surface data is vibrated by planar detection and superimposed over time to generate corrected road surface data, and the obstacle detection unit generates height map data based on the corrected road surface data.

[0285] According to the sixth aspect, in the obstacle detection system according to the fourth aspect, a three-dimensional distance sensor is configured to detect the height and tilt angle of the road surface up to the front of the mobile device in the direction of travel of the mobile device.

[0286] According to the seventh aspect, in the obstacle detection system according to the fourth aspect, the obstacle detection unit determines whether there is a height obstacle on the road surface based on height difference and smoothness features, and determines whether there is a ditch obstacle on the road surface based on the height difference in the neighborhood.

[0287] According to the eighth aspect, in the obstacle detection system according to the fifth aspect, the obstacle detection unit generates a voxel grid based on the vibration correction results to generate height map data.

[0288] According to the ninth aspect, in the obstacle detection system according to the fifth aspect, the obstacle detection unit performs noise removal based on the point cloud distribution of the vibration correction result.

[0289] According to the tenth aspect, in the obstacle detection system according to the fourth aspect, the mobile device includes a tracked driving device with a certain ground contact length, and the obstacle detection unit detects obstacles based on judgment parameters matched with the driving device.

[0290] According to the eleventh aspect, in the obstacle detection system according to any one of the fourth to tenth aspects, the three-dimensional distance sensor obtains 360-degree three-dimensional road surface data.

[0291] According to the twelfth aspect, the obstacle detection system according to the fourth aspect includes a cloud server connected to a network. The obstacle detection unit stores at least one of elevation map data, historical elevation data, and data on blind spot areas where the three-dimensional distance sensor failed to perform measurements in a shareable manner on the cloud server.

[0292] According to the thirteenth aspect, in the obstacle detection system according to the twelfth aspect, when at least one of the height map data and the height history data, in addition to the blind spot area data, is also stored in the cloud server, the obstacle detection unit only obtains the blind spot area data.

[0293] According to the fourteenth aspect, a mobile device includes: a driving device that travels on a road surface; a three-dimensional distance sensor for measuring distances in three dimensions to measure the distance to the road surface and for measuring distances in the height direction; and an obstacle detection unit for detecting obstacles on the road surface when the driving device stops.

[0294] The obstacle detection unit includes a height history data generation unit, which generates height history data based on the amount of travel made by the driving device within a predetermined time period before the driving device stops. This height history data is supplemented with height information stored before the driving device stopped. The height information represents the height above the road surface.

[0295] According to the fifteenth aspect, the recording medium carries computer-readable code for controlling a computer system to execute a method. The method includes detecting obstacles on a road surface when the mobile device stops.

[0296] Obstacle detection includes generating historical height data, supplementing it with height information stored up to the point of stopping, based on the amount of travel made by the vehicle over a predetermined period until it stops. The height information represents the distance from the road surface.

[0297] According to the sixteenth aspect, an obstacle detection method includes detecting obstacles on a road surface when a moving device stops.

[0298] Obstacle detection includes generating historical height data, supplementing it with height information stored up to the point of stopping, based on the amount of travel made by the vehicle over a predetermined period until it stops. The height information represents the distance from the road surface.

[0299] The above embodiments are illustrative and do not limit the invention. Therefore, many additional modifications and variations are possible based on the above teachings. For example, within the scope of the invention, elements and / or features of different illustrative embodiments may be combined with and / or substituted for each other. Any of the above operations may be performed in various other ways, for example, in an order different from that described above.

[0300] This invention can be implemented in any convenient form, such as using dedicated hardware or a combination of dedicated hardware and software. This invention can be implemented as computer software implemented by one or more networked processing devices. Processing devices include any suitably programmed device, such as a general-purpose computer, a personal digital assistant, a Wireless Application Protocol (WAP) or third-generation (3G) compatible mobile phone, etc. Since this invention can be implemented as software, each aspect of this invention includes computer software that can be implemented on a programmable device. The computer software can be provided to the programmable device using any conventional carrier medium (carrier device). Carrier media include transient carrier media, such as electrical, optical, microwave, acoustic, or radio frequency signals carrying computer code. An example of such transient media is Transmission Control Protocol / Internet Protocol (TCP / IP) signals carrying computer code over an IP network such as the Internet. The carrier medium may also include storage media for storing processor-readable code, such as floppy disks, hard disks, compact optical disc read-only memory (CD-ROM), magnetic tape devices, or solid-state storage devices.

[0301] The functionality of the elements disclosed herein can be implemented using circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, application-specific integrated circuits (ASICs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), and / or combinations thereof, configured or programmed to perform the disclosed functions using one or more programs stored in one or more memories. Processors are considered processing circuitry or circuit systems because they include transistors and other circuitry. In this disclosure, circuits, units, or devices are hardware that performs or is programmed to perform the described functions. The hardware can be any hardware disclosed herein that is programmed or configured to perform the described functions.

[0302] There exists a memory that stores a computer program, including computer instructions. These computer instructions provide logic and routines that enable hardware (e.g., processing circuitry or circuitry) to perform the methods disclosed herein. The computer program may be implemented in known formats as a computer-readable storage medium, a computer program product, a memory device, a recording medium such as a CD-ROM or DVD, and / or a memory of an FPGA or ASIC.

[0303] This patent application is based on and claims priority to Japanese Patent Application No. 2024-045771 filed with the Japan Patent Office on March 21, 2024 and Japanese Patent Application No. 2024-221881 filed on December 18, 2024, the entire disclosure of which is incorporated herein by reference.

[0304] [List of reference numerals]

[0305] 1. Mobile device (obstacle detection system)

[0306] 11a, 11b Tracked moving body (travel device)

[0307] 113 Three-dimensional distance sensor

[0308] 155 obstacle detection units

[0309] 155c Height Historical Data Generation Unit

[0310] 200 Mobile devices (obstacle detection systems)

[0311] 300 Obstacle Detection System

[0312] 301 Mobile device.

Claims

1. An obstacle detection system, comprising: A three-dimensional distance sensor is used to measure the distance to the road surface on which a mobile device travels and to measure the distance in the vertical direction. as well as An obstacle detection unit is configured to detect obstacles on the road surface when the mobile device stops, wherein... The obstacle detection unit includes a height history data generation unit, which is configured to generate height history data supplemented by height information stored until the mobile device stops. The height information represents the height above the road surface and is based on the amount of travel made by the mobile device over a predetermined period of time until the mobile device stops.

2. The obstacle detection system according to claim 1, wherein... The height historical data generation unit is configured as follows: The rectangular region of the identified moving object is compared with the historical height data; and the height information is removed from the rectangular region of the identified moving object.

3. The obstacle detection system according to claim 1, wherein... The height history data generation unit is configured to delete the height information from the height history data in response to an instruction from an external device outside the obstacle detection system.

4. The obstacle detection system according to any one of claims 1 to 3, wherein The obstacle detection unit is configured as follows: Based on the measurement results obtained from the three-dimensional distance sensor, three-dimensional road surface data is generated with reference to the mobile device; Elevation map data is generated based on the road surface data, the elevation map data including the maximum elevation from the road surface as the elevation information; and The altitude map data is provided to the altitude history data generation unit.

5. The obstacle detection system according to claim 4, wherein... The obstacle detection unit is configured as follows: Vibration correction is performed based on planar detection of the road surface data; Over time, corrected road surface data are superimposed; and The elevation map data is generated based on the corrected road surface data.

6. The obstacle detection system according to claim 4, wherein The three-dimensional distance sensor is configured to detect the height and tilt angle of the road surface up to the front of the mobile device in the direction of travel of the mobile device.

7. The obstacle detection device according to claim 4, wherein, The obstacle detection unit is configured as follows: The presence of height obstacles on the road surface is determined based on height difference and smoothness characteristics. as well as Based on the height difference in the neighborhood of the height map data, it is determined whether there are any depressions or obstacles in the road surface.

8. The obstacle detection system according to claim 5, wherein The obstacle detection unit is configured to generate a voxel grid based on the vibration correction results to generate the height map data.

9. The obstacle detection system according to claim 5, wherein The obstacle detection unit is configured to perform noise removal based on the point cloud distribution resulting from the vibration correction.

10. The obstacle detection system according to claim 4, wherein The mobile device includes a tracked travel device with a ground contact length, and The obstacle detection unit is configured to detect obstacles based on decision parameters matched with the driving device.

11. The obstacle detection system according to any one of claims 4 to 10, wherein The three-dimensional distance sensor acquires the three-dimensional road surface data in 360 degrees.

12. The obstacle detection system according to claim 4 further includes a cloud server connected to a network, wherein... The obstacle detection unit is configured to store at least one of the height map data, the height history data, and blind spot area data in the cloud server for sharing with another mobile device, wherein the blind spot area is the area that the three-dimensional distance sensor failed to measure.

13. The obstacle detection system according to claim 12, wherein... The obstacle detection unit is configured to obtain only the data of the blind spot area, even when the cloud server stores at least one of the height map data and the height history data in addition to the data of the blind spot area.

14. A mobile device, comprising: A driving device that travels on a road surface; A three-dimensional distance sensor, the three-dimensional distance sensor being used to measure the distance to the road surface and to measure the distance in the height direction; as well as An obstacle detection unit is configured to detect obstacles on the road surface when the driving device stops, wherein... The obstacle detection unit includes a height history data generation unit, which is configured to generate height history data supplemented by height information stored until the driving device stops. The height information represents the height above the road surface and is based on the amount of travel made by the driving device during a predetermined time period until the driving device stops.

15. A recording medium carrying computer-readable code, said computer-readable code, when executed by a computer system, causing the computer system to perform a method, said method comprising: When the mobile device stops, it detects obstacles on the road surface, among which... Detecting the obstacle includes generating historical height data, which is supplemented with height information saved until the mobile device stops. The height information represents the height above the road surface and is based on the amount of travel made by the mobile device over a predetermined period of time until the mobile device stops.

16. An obstacle detection method, comprising: When the mobile device stops, it detects obstacles on the road surface, among which... Detecting the obstacle includes generating historical height data, which is supplemented with height information saved until the mobile device stops. The height information represents the height above the road surface and is based on the amount of travel made by the mobile device over a predetermined period of time until the mobile device stops.

Images