Obstacle Detection Device and Obstacle Detection Method

Abstract

To provide an obstacle detection device and an obstacle detection method capable of reducing the detected result error of an obstacle position due to the inclination of a camera.SOLUTION: An obstacle detection device comprises a camera, a detection unit to detect an obstacle position from an image captured by the camera, and an estimation unit to estimate a camera inclination. The estimation unit performs straight line acquisition processing to acquire straight lines from the captured image, straight line selection processing to select a straight line capable of forming vanishing points in the captured image among the acquired multiple straight lines, vanishing point estimation processing to estimate the position of the vanishing point in the captured image based on the selected multiple straight lines, and inclination calculation processing to calculate the camera inclination from a distance between the estimated vanishing point and the center of the captured image, and from a camera focal length. The detection unit detects the obstacle position considering the camera inclination estimated by the estimation unit.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an obstacle detection device and an obstacle detection method.

Background Art

[0002] Patent Document 1 discloses an obstacle detection device including a camera and a detection unit that detects the position of an obstacle from an image captured by the camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In such an obstacle detection device, if the camera is tilted, an error may occur in the detection result of the position of the obstacle.

Means for Solving the Problems

[0005] An obstacle detection device for solving the above problems is an obstacle detection device including a camera and a detection unit that detects the position of an obstacle from a captured image of the camera, the device including an estimation unit that estimates the tilt of the camera, the estimation unit performing a straight line acquisition process of acquiring a straight line from the captured image, a straight line selection process of selecting, from among the plurality of acquired straight lines, a straight line that can form a vanishing point in the captured image, a vanishing point estimation process of estimating the position of the vanishing point in the captured image based on the selected plurality of straight lines, and a tilt calculation process of calculating the tilt of the camera from the distance between the estimated vanishing point and the center of the captured image and the focal length of the camera, and the detection unit detecting the position of the obstacle in consideration of the tilt of the camera estimated by the estimation unit. Among the plurality of acquired straight lines, a straight line selection process of selecting a straight line that can form a vanishing point in the captured image, a vanishing point estimation process of estimating the position of the vanishing point in the captured image based on the selected plurality of straight lines, and a tilt calculation process of calculating the tilt of the camera from the distance between the estimated vanishing point and the center of the captured image and the focal length of the camera are performed, and the detection unit detects the position of the obstacle in consideration of the tilt of the camera estimated by the estimation unit.

[0006] The estimation unit estimates the tilt of the camera. The detection unit detects the position of the obstacle in consideration of the tilt of the camera estimated by the estimation unit. Therefore, an error in the detection result of the position of the obstacle due to the tilt of the camera can be reduced.

[0007] In addition, by performing the straight line selection process in the estimation unit, the processing speed of the vanishing point estimation process can be increased as compared with the case of estimating the vanishing point based on all the straight lines obtained by the straight line acquisition process.

[0008] Furthermore, the estimation unit estimates the tilt of the camera by using the captured image used for the detection of the obstacle. Therefore, a sensor or the like for detecting the tilt of the camera is not required. In the obstacle detection device, the estimation unit performs an intersection calculation process for calculating an intersection point at which the plurality of straight lines selected by the straight line selection process intersect, and among the plurality of calculated intersection points, an intersection point within a predetermined range from the center of the captured image And in the vanishing point estimation process, the average value or the median value of the positions of the plurality of intersection points selected in the intersection point selection process may be used as the position of the vanishing point.

[0009] By performing the intersection point selection process in the estimation unit, the estimation accuracy of the position of the vanishing point can be increased as compared with the case of estimating the vanishing point based on all the intersection points obtained by the intersection point acquisition process. Further, when the estimation unit sets the average value or the median value of the positions of the intersection points as the position of the vanishing point, the estimation accuracy of the position of the vanishing point can be increased as compared with the case of setting the position of one of the plurality of intersection points selected by the intersection point selection process as the position of the vanishing point.

[0010] In the obstacle detection device, the estimation unit performs an intersection calculation process for calculating an intersection point at which the plurality of straight lines selected by the straight line selection process intersect, and the estimation unit has a learned model that is machine-learned to output the position of the vanishing point when the plurality of calculated intersection points are input, The estimation unit may perform the vanishing point estimation process using the learned model.

[0011] Depending on the degree of learning of the learned model, the estimation accuracy of the position of the vanishing point can be improved. In the obstacle detection device, the estimation unit has a learned model that is machine-learned to output the position of the vanishing point when the plurality of lines selected by the line selection process are input, and the estimation unit may perform the vanishing point estimation process using the learned model.

[0012] The estimation unit can estimate the vanishing point without performing an intersection calculation process or an intersection selection process. Depending on the degree of learning of the learned model, the estimation accuracy of the position of the vanishing point can be improved.

[0013] In the obstacle detection device, the camera is a fisheye camera having a fisheye lens, the estimation unit performs a distortion correction process for correcting the distortion of the captured image, and in the straight line acquisition process, the straight line may be acquired from the captured image after the distortion correction process.

[0014] Since the angle of view of the camera is wide, the detection of obstacles can be performed over a wide range. In addition, by the estimation unit performing a distortion correction process, an inclination calculation process becomes possible. In the obstacle detection device, the estimation unit may perform image processing on the captured image before performing the straight line acquisition process.

[0015] By the estimation unit performing image processing on the captured image, the subsequent straight line acquisition process becomes easier. In the obstacle detection device, the estimation unit performs semantic segmentation to grasp an obstacle region including an image of the obstacle from the captured image, and in the straight line selection process, the straight line located in the obstacle region may be selected as a straight line that can form the vanishing point.

[0016] An obstacle may have a linear edge. Therefore, the vanishing point can be estimated by using the linear edge of the obstacle. In the above-described obstacle detection device, the obstacle detection device is mounted on a vehicle. The estimation unit grasps a host vehicle region including an image of the vehicle on which the obstacle detection device is mounted from the captured image by performing semantic segmentation. In the straight line selection process, the straight line located in the host vehicle region may be selected as a straight line that can form the vanishing point.

[0017] A vehicle has components having straight edges. For example, a forklift as a vehicle has pillars and masts as components having straight edges. Therefore, the vanishing point can be estimated by using the straight edges of the vehicle. In this case, the estimation unit can estimate the tilt of the camera with respect to the vehicle.

[0018] An obstacle detection method for solving the above problems includes an imaging step of imaging by a camera, a detection step of detecting the position of an obstacle from the captured image of the camera by a detection unit connected to the camera, and is an obstacle detection method having a tilt estimation step of estimating the tilt of the camera by an estimation unit connected to the camera. The tilt estimation step includes a straight line acquisition step of the estimation unit acquiring a straight line from the captured image, a straight line selection step of the estimation unit selecting a straight line that can form a vanishing point in the captured image from among the plurality of straight lines acquired by the estimation unit, a vanishing point estimation step of estimating the position of the vanishing point in the captured image based on the plurality of straight lines selected by the estimation unit, and a tilt calculation step of calculating the tilt of the camera from the distance between the vanishing point estimated by the estimation unit and the center of the captured image and the focal length of the camera. In the detection step, the gist is that the detection unit detects the position of the obstacle in consideration of the tilt of the camera estimated in the tilt estimation step.

[0019] In the tilt estimation step, the estimation unit estimates the tilt of the camera. In the detection step, the detection unit detects the position of the obstacle in consideration of the tilt of the camera estimated in the tilt estimation step. Therefore, the error in the detection result of the position of the obstacle due to the tilt of the camera can be reduced.

[0020] Also, by the straight line selection step, the processing speed of the vanishing point estimation step can be increased as compared with the case of estimating the vanishing point based on all the straight lines acquired in the straight line acquisition step.

[0021] Furthermore, in the tilt estimation step, the tilt of the camera is estimated using the captured image used in the detection step. Therefore, a sensor or the like for detecting the tilt of the camera is not required.

Advantages of the Invention

[0022] According to the present invention, the error in the detection result of the position of the obstacle due to the tilt of the camera can be reduced.

Brief Description of the Drawings

[0023]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Mode for Carrying Out the Invention

[0024] Hereinafter, embodiments embodying the obstacle detection device and the obstacle detection method will be described with reference to FIGS. 1 to 14. The obstacle detection device and the obstacle detection method of the present embodiment are applied to a forklift as a vehicle. The obstacle detection device and the obstacle detection method of the present embodiment detect the positions of obstacles existing around the forklift.

[0025] <Forklift> As shown in FIG. 1, the forklift 10 includes a machine base 11, two drive wheels 12a, two steering wheels 12b, and a cargo handling device 13. The two drive wheels 12a are provided at the front lower part of the machine base 11. The two steering wheels 12b are provided at the rear lower part of the machine base 11.

[0026] The machine 11 has a head guard 14, two front pillars 15, and two rear pillars 16. Of the two front pillars 15, one front pillar 15 stands upright from the left front part of the machine 11, and the other front pillar 15 stands upright from the right front part of the machine 11. The rear pillar 16 is provided behind the front pillar 15. Of the two rear pillars 16, one rear pillar 16 stands upright from the left rear part of the machine 11, and the other rear pillar 16 stands upright from the right rear part of the machine 11. The front pillar 15 and the rear pillar 16 extend linearly along the vertical direction. The head guard 14 is supported by the two front pillars 15 and the two rear pillars 16.

[0027] The handling device 13 is provided in front of the machine 11. The handling device 13 has a mast 17 and forks 18. The mast 17 has an outer mast and an inner mast. The inner mast is provided so as to be able to move up and down with respect to the outer mast. The mast 17 extends linearly along the vertical direction. The forks 18 are attached to the inner mast via a lift bracket (not shown). The forks 18 can move up and down together with the inner mast.

[0028] The forklift 10 of the present embodiment travels and performs handling operations by the handling device 13 in a factory or a warehouse. In this case, obstacles that can hinder the traveling and handling operations of the forklift 10 are objects placed on the floor of the factory or warehouse and people present on the floor. Examples of objects that can become obstacles in a factory or warehouse include pallets, triangular cones, shelves, pillars, and other forklifts 10.

[0029] <Obstacle Detection Device> As shown in FIG. 2, the obstacle detection device 20 has a camera 21 and a detection unit 22 connected to the camera 21. The obstacle detection device 20 of the present embodiment is mounted on the forklift 10.

[0030] <Camera> As shown in FIG. 1, the camera 21 is attached to the forklift 10. The camera 21 of the present embodiment is attached to the center in the left - right direction at the rear part of the head guard 14. The camera 21 is arranged to face downward in the vertical direction. The camera 21 mainly images the floors of factories and warehouses. The camera 21 of the present embodiment is a fish - eye camera having a fish - eye lens.

[0031] The attachment of the camera 21 to the forklift 10 is performed by an operator. In the present embodiment, the operator attaches the camera 21 to the forklift 10 so that the optical axis of the camera 21 extends along the vertical direction. At this time, it is preferable that the direction in which the optical axis of the camera 21 extends completely coincides with the vertical direction, but the direction in which the optical axis of the camera 21 extends may be slightly inclined with respect to the vertical direction. The allowable angle range of the inclination of the optical axis of the camera 21 with respect to the vertical direction is set in advance. The operator attaches the camera 21 to the forklift 10 so that the angle of the optical axis of the camera 21 with respect to the vertical direction falls within the allowable angle range.

[0032] Note that the inclination of the camera 21 occurs not only when the camera 21 is attached to the forklift 10 as described above, but also in the following cases. For example, when the drive wheels 12a or the steering wheels 12b are worn, or when the forklift 10 is carrying a load, the forklift 10 may tilt with respect to the floor. Then, the camera 21 mounted on the forklift 10 also tilts. Also, the camera 21 may tilt due to vibrations, impacts, etc. that occur after the camera 21 is attached to the forklift 10.

[0033] <Detection unit> As shown in FIG. 2, the detection unit 22 includes a processor 23 and a storage unit 24. As the processor 23, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor) is used. The storage unit 24 includes a RAM (Random Access Memory) and a ROM (Read Only Memory). A program for operating the obstacle detection device 20 is stored in the storage unit 24. It can be said that the storage unit 24 stores program codes or instructions configured to cause the processor 23 to execute processing. The storage unit 24, that is, the computer-readable medium, includes any available medium accessible by a general-purpose or dedicated computer. The detection unit 22 may be configured by a hardware circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The detection unit 22, which is a processing circuit, may include one or more processors operating according to a computer program, one or more hardware circuits such as an ASIC or an FPGA, or a combination thereof.

[0034] <Vanishing point> Generally, in a pinhole camera without distortion, lines that are parallel in the real world intersect at a point on the image plane. This point is called the vanishing point. When the camera 21 is directed vertically downward as in this embodiment, that is, when the camera 21 images the floor, lines extending perpendicular to the floor form the vanishing point. Hereinafter, the respective vanishing points when the camera 21 is not tilted and when the camera 21 is tilted will be described in detail.

[0035] As shown in FIG. 3, the real-world coordinate system has an X-axis, a Y-axis, and a Z-axis. The Z-axis is an axis along the vertical direction. The X-axis is an axis orthogonal to the Z-axis. The Y-axis is an axis orthogonal to both the X-axis and the Z-axis. The position of the camera 21 in the real-world coordinate system is set as the origin O. A plane that extends along the X-axis and the Y-axis and is orthogonal to the Z-axis, and exists at a position that is separated from the origin O by a distance H in the direction along the Z-axis is defined as the real-world plane P. The real-world plane P in the present embodiment is the floor. Note that the distance H can also be said to be the height of the camera 21 from the floor.

[0036] The camera coordinate system is a coordinate system based on the camera 21. The camera coordinate system has an Xc-axis, a Yc-axis, and a Zc-axis. The Zc-axis is an axis that coincides with the optical axis of the camera 21. The Xc-axis is an axis orthogonal to the Zc-axis. The Yc-axis is an axis orthogonal to both the Xc-axis and the Zc-axis. The position of the camera 21 in the camera coordinate system is set as the origin O. A plane that extends along the Xc-axis and the Yc-axis and is orthogonal to the Zc-axis, and exists at a position that is separated from the origin O by the focal length f of the camera 21 in the direction along the Zc-axis is defined as the image plane Pc.

[0037] FIG. 3 shows the real-world plane P and the image plane Pc when the camera 21 is not tilted. The case where the camera 21 is not tilted means the case where the Zc-axis and the Z-axis coincide. In other words, the case where the camera 21 is not tilted means the case where the direction in which the optical axis of the camera 21 extends coincides with the vertical direction and extends perpendicular to the floor. When the camera 21 is not tilted, the real-world plane P and the image plane Pc are in a parallel relationship.

[0038] As shown in FIG. 4, when the camera 21 is not tilted, a straight line L that extends perpendicular to the real-world plane P intersects at the center C of the image plane Pc. That is, the vanishing point V is formed at the center C of the image plane Pc.

[0039] FIG. 5 shows the real-world plane P and the image plane Pc when the camera 21 is tilted. The case where the camera 21 is tilted means the case where the Zc axis is tilted with respect to the Z axis. In other words, the case where the camera 21 is tilted means the case where the direction in which the optical axis of the camera 21 extends is tilted with respect to the vertical direction. When the camera 21 is tilted, the image plane Pc is also tilted with respect to the real-world plane P.

[0040] As shown in FIG. 6, when the camera 21 is tilted, a straight line L extending perpendicular to the real-world plane P intersects at a position shifted from the center C of the image plane Pc. That is, the vanishing point V is formed at a position shifted from the center C of the image plane Pc.

[0041] The unit vector from the origin O to the vanishing point V coincides with the unit vector in the Z-axis direction with respect to the camera 21 in the real world. Therefore, from the position of the vanishing point V, the inclination of the Zc axis with respect to the Z axis can be obtained. That is, the inclination of the optical axis of the camera 21 with respect to the vertical direction can be obtained.

[0042] As shown in FIGS. 5 and 6, for example, when the camera 21 is tilted by θx [°] about the X axis, the Y axis and the Z axis are each tilted by θx [°] with respect to the Yc axis and the Zc axis. At this time, the position of the vanishing point V is shifted in the direction along the Yc axis from the center C of the image plane Pc. Let the distance from the center C of the image plane Pc to the vanishing point V in the direction along the Yc axis be y v Then, θx = tan -1 (y v / f).

[0043] Similarly, when the camera 21 is tilted by θy [°] about the Y axis, the X axis and the Z axis are each tilted by θy [°] with respect to the Xc axis and the Zc axis. At this time, the position of the vanishing point V is shifted in the direction along the Xc axis from the center C of the image plane Pc. Let the distance from the center C of the image plane Pc to the vanishing point V in the direction along the Xc axis be x v Then, θy = tan -1 (x v / f).

[0044] <Obstacle Detection Method> As shown in FIG. 7, the obstacle detection method includes an imaging step S11, an inclination estimation step S12, and a detection step S13. The imaging step S11 is a step in which the camera 21 performs imaging. The inclination estimation step S12 is a step in which the detection unit 22 estimates the inclination of the camera 21 from the image captured by the camera 21. In the inclination estimation step S12 of the present embodiment, the inclination of the camera 21 with respect to the vertical direction is estimated. The detection step S13 is a step in which the detection unit 22 detects the position of an obstacle from the image captured by the camera 21. Therefore, the detection unit 22 of the present embodiment is a detection unit that detects the position of an obstacle and also an estimation unit that estimates the inclination of the camera 21.

[0045] The inclination estimation step S12 of the present embodiment includes a distortion correction step S21, a preprocessing step S22, a straight line acquisition step S23, a straight line selection step S24, an intersection calculation step S25, an intersection selection step S26, a vanishing point estimation step S27, and an inclination calculation step S28.

[0046] <Imaging step> In the imaging step S11 of the present embodiment, the camera 21 images downward in the vertical direction. Hereinafter, the image captured by the camera 21 is referred to as an imaging image I.

[0047] FIG. 8 is an example of the imaging image I. As described above, the camera 21 of the present embodiment is a fisheye camera having a fisheye lens. Therefore, the imaging image I is a fisheye image in which the subject is imaged in a distorted state. The imaging image I of the present embodiment includes an image of the forklift 10 (hereinafter referred to as the "own vehicle") on which the obstacle detection device 20 is mounted, an image of the obstacle S, and an image of the road surface R. The image of the obstacle S includes an image of a shelf. The image of the road surface R includes an image of the floor and an image of a line drawn on the floor such as an outer lane line. The image of the own vehicle includes images of two front pillars 15, images of two rear pillars 16, and an image of the mast 17.

[0048] <Distortion correction step> As shown in FIG. 9, in the distortion correction step S21, the detection unit 22 performs a distortion correction process for correcting the distortion of the captured image I which is a fisheye image.

[0049] An example of a method for correcting the distortion of a fisheye image will be described. After the light rays incident from the spatial range of the hemisphere pass through the fisheye lens of the camera 21, the fisheye image is captured by being projected onto the imaging surface of the two-dimensional image sensor. Therefore, when correcting the distortion of the fisheye image, a virtual spherical model of the fisheye lens is used. The detection unit 22 first sets a plane tangent to an arbitrary point on the virtual sphere as a screen. Next, the detection unit 22 converts the three-dimensional coordinate values on the virtual sphere corresponding to the two-dimensional coordinate values of the fisheye image into the three-dimensional coordinate values on the screen. When converting the three-dimensional coordinate values on the virtual sphere into the three-dimensional coordinate values on the screen, a geometric conversion formula is used. In this way, the detection unit 22 reproduces the fisheye image on the screen. The image reproduced on the screen is an image without distortion caused by the fisheye lens. The detection unit 22 creates an image with corrected distortion using the image reproduced on the screen. Note that the method for correcting the distortion of the fisheye image is not limited to the above method. Other well-known methods may be adopted for the method for correcting the distortion of the fisheye image.

[0050] Note that the captured image I before distortion correction shown in FIG. 8 and the captured image I after distortion correction shown in FIG. 9 are actually color images. <Preprocessing Step> In the preprocessing step S22, the detection unit 22 performs image processing on the captured image I after distortion correction. This image processing is preprocessing for facilitating the processing in the straight line acquisition step S23 that the detection unit 22 will perform later.

[0051] In the present embodiment, as preprocessing, the detection unit 22 performs grayscale conversion on the captured image I after distortion correction. Also, in the present embodiment, as preprocessing, the detection unit 22 detects edges by using Canny conversion on the captured image I after grayscale conversion.

[0052] FIG. 10 shows the detection result of the edges for the captured image I. In FIG. 10, the detected edges are indicated by white lines. The detected edges include a large number of edges of various shapes, such as linear edges and curved edges.

[0053] <Straight line acquisition step> As shown in FIG. 11, in the straight line acquisition step S23, the detection unit 22 performs a straight line acquisition process of acquiring a straight line L from the captured image I after distortion correction. The detection unit 22 of the present embodiment acquires the straight line L from the edges detected in the preprocessing step S22 by using the Hough transform or the probabilistic Hough transform. The linear edges are, for example, the edges of the outside lane line or the edges of obstacles S such as shelves and pillars. The edges of the outside lane line extend parallel to the floor. The edges of obstacles S such as shelves and pillars include edges extending perpendicular to the floor. In the straight line acquisition step S23, a large number of straight lines L are acquired.

[0054] <Straight line selection step> As shown in FIG. 12, in the straight line selection step S24, the detection unit 22 performs a straight line selection process of selecting a straight line L that can form a vanishing point V in the captured image I from among the large number of straight lines L acquired in the straight line acquisition step S23.

[0055] As described above, in the present embodiment, a straight line L extending perpendicular to the floor forms the vanishing point V. Therefore, in the present embodiment, the straight line L that can form the vanishing point V is a straight line L extending perpendicular to the floor.

[0056] When the camera 21 is not tilted, the vanishing point V appears at the center C of the captured image I. In other words, the straight line L extending vertically with respect to the floor intersects at the center C of the captured image I. Therefore, it can be said that the straight line L extending vertically with respect to the floor passes through the center C of the captured image I. Also, when the camera 21 is tilted, the vanishing point V appears near the center C of the captured image I. In other words, the straight line L extending vertically with respect to the floor intersects near the center C of the captured image I. Therefore, it can be said that the straight line L extending vertically with respect to the floor passes near the center C of the captured image I. From the above, the detection unit 22 selects, from among the plurality of straight lines L, the straight line L passing near the center C of the captured image I, thereby selecting the straight line L extending vertically with respect to the floor, that is, the straight line L that can form the vanishing point V.

[0057] Specifically, the detection unit 22 calculates the distance D from the center C of the captured image I to the straight line L. The distance D from the center C of the captured image I to the straight line L can be calculated by [Equation 1] from the coordinates (x0, y0) of the center C of the captured image I and the equation ax + by + c = 0 of the straight line L.

[0058]

Equation

[0059] The detection unit 22 determines whether or not the straight line L passes near the center C of the captured image I based on whether or not the calculated distance D is less than or equal to a predetermined distance Dth. The predetermined distance Dth is calculated from the maximum angle allowed as the tilt of the camera 21. When the calculated distance D is less than or equal to the predetermined distance Dth, the detection unit 22 selects it as the straight line L that can form the vanishing point V. When the calculated distance D is longer than the predetermined distance Dth, the detection unit 22 does not select it as the straight line L that can form the vanishing point V. In the straight line selection step S24, a plurality of straight lines L that can form the vanishing point V are selected.

[0060] <Intersection point calculation step> As shown in FIG. 13, in the intersection calculation step S25, the detection unit 22 performs an intersection calculation process of calculating an intersection Q from a plurality of straight lines L selected in the straight line selection step S24. Note that the intersection Q is a point where two or more straight lines L among the plurality of straight lines L selected in the straight line selection step S24 intersect. In the intersection calculation step S25, a large number of intersections Q are calculated.

[0061] <Intersection selection step> In the intersection selection step S26, the detection unit 22 performs an intersection selection process of selecting an intersection Q located near the center C of the captured image I from the plurality of intersections Q calculated in the intersection calculation step S25. In the present embodiment, the detection unit 22 selects an intersection Q within a predetermined range from the center C of the captured image I as an intersection Q located near the center C of the captured image I. The predetermined range is set from the maximum angle allowed as the inclination of the camera 21. In the intersection selection step S26, a plurality of intersections Q are calculated.

[0062] <Vanishing point estimation step> As shown in FIG. 14, in the vanishing point estimation step S27, the detection unit 22 performs a vanishing point estimation process of estimating the position of the vanishing point V based on the intersection Q selected in the intersection selection step S26. In the present embodiment, the detection unit 22 calculates the average value of the positions of the intersections Q in the captured image I using the plurality of intersections Q selected in the intersection selection step S26. Then, the detection unit 22 sets the average value of the calculated positions of the intersections Q as the position of the vanishing point V.

[0063] <Inclination calculation step> In the inclination calculation step S28, the detection unit 22 performs an inclination calculation process of calculating the inclination of the camera 21 from the distance between the vanishing point V estimated in the vanishing point estimation step S27 and the center C of the captured image I, and the focal length f of the camera 21.

[0064] The detection unit 22 of the present embodiment calculates the inclination θx around the X axis and the inclination θy around the Y axis, respectively. Specifically, the detection unit 22 determines the distance y between the center C and the vanishing point V of the captured image I in the direction along the Yc axis from the position of the vanishing point V in the direction along the Yc axis vCalculate it. The detection unit 22 calculates the inclination θx around the X axis from the calculated distance y v and the focal length f. Similarly, the detection unit 22 calculates the distance x between the center C of the captured image I in the Xc axis direction and the vanishing point V from the position of the vanishing point V in the direction along the Xc axis v Calculate it. The detection unit 22 calculates the inclination θx around the Y axis from the calculated distance x v and the focal length f.

[0065] <Detection step> In the detection step S13, the detection unit 22 detects the position of the obstacle S in consideration of the estimated inclination of the camera 21. Note that "considering the inclination of the camera 21" means, for example, using the inclination of the camera 21 when calculating the position of the obstacle S or correcting the detection result of the position of the obstacle S according to the inclination of the camera 21.

[0066] [Effects of this embodiment] The operations and effects of this embodiment will be described. (1) The detection unit 22 estimates the inclination of the camera 21 in the inclination estimation step S12. In the detection step S13, the detection unit 22 detects the position of the obstacle S in consideration of the inclination of the camera 21 estimated in the inclination estimation step S12. Therefore, the error in the detection result of the position of the obstacle S due to the inclination of the camera 21 can be reduced.

[0067] (2) In the straight line selection step S24, the detection unit 22 selects a straight line L that can form a vanishing point V in the captured image I from among the straight lines L obtained in the straight line acquisition step S23. Therefore, the processing speed in the vanishing point estimation step S27 can be increased as compared with the case of estimating the vanishing point V based on all the straight lines L obtained in the straight line acquisition step S23.

[0068] (3) The detection unit 22 estimates the inclination of the camera 21 by using the captured image I used for detecting the obstacle S. Therefore, a sensor or the like for detecting the inclination of the camera 21 is not required.

[0069] (4) The detection unit 22 calculates the intersection point Q where the plurality of straight lines L selected in the straight line selection step S24 intersect in the intersection point calculation step S25. In the intersection point selection step S26, the detection unit 22 selects the intersection point Q within a predetermined range from the center C of the captured image I among the intersection points Q calculated in the intersection point calculation step S25. In the vanishing point estimation step S27, the detection unit 22 sets the average value of the positions of the intersection points Q selected in the intersection point selection step S26 as the position of the vanishing point V.

[0070] Thereby, the estimation accuracy of the position of the vanishing point V can be improved as compared with the case of estimating the vanishing point V based on all the intersection points Q obtained in the intersection point calculation step S25. Also, the estimation accuracy of the position of the vanishing point V can be improved as compared with the case of setting the position of one of the intersection points Q selected in the intersection point selection step S26 as the position of the vanishing point V.

[0071] (5) The camera 21 is a fisheye camera having a fisheye lens. In this case, since the angle of view of the camera 21 is widened, the detection of the obstacle S can be performed over a wide range. Also, the detection unit 22 corrects the distortion of the captured image I in the distortion correction step S21. Thereby, the calculation of the inclination of the camera 21 in the inclination calculation step S28 becomes possible.

[0072] (6) The detection unit 22 performs grayscale conversion and edge detection on the captured image I in the preprocessing step S22. Thus, the straight line acquisition process in the straight line acquisition step S23 after the preprocessing step S22 becomes easy.

[0073] (7) The obstacle detection device 20 and the obstacle detection method of the present embodiment are applied to the forklift 10 used in a factory or a warehouse. There are a plurality of obstacles S having edges extending linearly along the vertical direction on the floor of the factory or the warehouse. Therefore, since the camera 21 is attached downward in the vertical direction so as to image the floor, the number of straight lines L that can form the vanishing point V increases. Thus, it is easy to estimate the position of the vanishing point V and, consequently, the inclination of the camera 21.

[0074] [Modification Example] In addition, each of the above embodiments can be implemented with the following modifications. Each of the above embodiments and the following modification examples can be implemented in combination with each other within a technically consistent range.

[0075] ○ The obstacle detection device 20 may include a plurality of cameras 21. In this case, since the imaging range becomes wider, the detectable range of the obstacle S also becomes wider. The detection unit 22 may detect the inclination for each of the plurality of cameras 21.

[0076] ○ In the obstacle detection device 20, an estimation unit that estimates the inclination of the camera 21 and a detection unit that detects the position of the obstacle S may be provided separately. In this case, the estimation unit outputs the estimated inclination of the camera 21 to the detection unit. The detection unit detects the position of the obstacle S in consideration of the inclination of the camera 21 input from the estimation unit.

[0077] ○ The camera 21 may be attached to the forklift 10 so that the optical axis is intentionally inclined with respect to the vertical direction. The angle of the optical axis with respect to the vertical direction when the camera 21 is intentionally inclined is defined as the reference angle. In this case, the storage unit 24 of the detection unit 22 stores the reference angle. The detection unit 22 corrects the calculation result of the inclination of the camera 21 using the stored reference angle.

[0078] ○ The camera 21 may be attached so that the optical axis extends in the horizontal direction. In this case, the real-world plane P is the wall surface of a factory or a warehouse. Also, the straight line L that can form the vanishing point V is a straight line L extending in the horizontal direction.

[0079] ○ The camera 21 does not have to be a fish-eye camera. In this case, the distortion correction step S21 becomes unnecessary. ○ In the preprocessing step S22, the detection unit 22 may perform only one of edge detection using grayscale conversion and Canny conversion.

[0080] ○ In the preprocessing step S22, the detection unit 22 may perform preprocessing other than edge detection using grayscale conversion and Canny conversion. For example, instead of grayscale conversion, the detection unit 22 may decompose the captured color image I into RGB components.

[0081] ○ The obstacle detection method may not have the preprocessing step S22. ○ In the straight line acquisition step S23, the detection unit 22 may perform the straight line acquisition process as follows.

[0082] As shown in FIG. 15, the storage unit 24 of the detection unit 22 stores the learned model M. The learned model M in this modified example is machine-learned to output a straight line in the input image. Specifically, the learned model M is learned by supervised learning. The teacher data has the captured image I and the correct data of the straight line L in the captured image I. The detection unit 22 acquires the straight line L from the captured image I after distortion correction by the learned model M.

[0083] In addition to outputting a straight line from the input image, the learned model M may be learned to classify the output straight line. For example, the learned model M may classify whether the straight line in the image is a straight line of the host vehicle, a straight line of the obstacle S, or a straight line of the road surface R. In this case, the teacher data has the captured image I, the correct data of the straight line L in the captured image I, and the correct data of the class corresponding to the straight line L.

[0084] The classification result of the straight line L can be used in the straight line selection step S24. For example, the detection unit 22 selects, as the straight line L that can form the vanishing point V, the straight line L of the obstacle S among the acquired straight lines L.

[0085] ○ In the straight line selection step S24, the detection unit 22 may perform the straight line selection process as follows. The storage unit 24 of the detection unit 22 stores the learned model M. The learned model M in this modification example is machine-learned to classify each of the plurality of pixels constituting the input image. For example, the learned model M is learned to classify each pixel constituting the input image as a pixel representing the host vehicle, a pixel representing the obstacle S, or a pixel representing the road surface R. Specifically, the learned model M is learned by supervised learning. The teacher data has the captured image I and the correct classification data for each pixel constituting the captured image I.

[0086] The detection unit 22 performs semantic segmentation on the captured image I after distortion correction using the learned model M. That is, the detection unit 22 classifies each pixel constituting the captured image I as a pixel representing the host vehicle, a pixel representing the obstacle S, or a pixel representing the road surface R using the learned model M.

[0087] As shown in FIG. 16, the detection unit 22 can identify from the result of semantic segmentation a host vehicle region that is a region showing an image of the host vehicle in the captured image I, an obstacle region that is a region showing an image of the obstacle S, and a road surface region that is a region showing an image of the road surface R. Then, the detection unit 22 selects, as a line L that can form the vanishing point V, a line L among the large number of lines L acquired in the straight line acquisition step S23, at least a part of which is located in the obstacle region.

[0088] If a misjudgment occurs in the classification of each pixel constituting the captured image I, the detection unit 22 cannot accurately identify the outlines of the host vehicle region, the obstacle region, and the road surface region. Therefore, the detection unit 22 may expand the obstacle region by morphological transformation or the like.

[0089] In addition, when the straight line L is obtained using the Hough transform in the straight line acquisition step S23, the end points of the straight line L are not known. Therefore, for example, even if the straight line L is extracted from the edge of the outside line of the road lane, a part of it may be located in the obstacle area, and thus it may be selected as the straight line L that can form the vanishing point V. For this reason, in the straight line acquisition step S23, it is preferable to obtain the straight line L using the probabilistic Hough transform. When the probabilistic Hough transform is used, the end points of the straight line L are known. Therefore, it is possible to more accurately determine whether the straight line L is located in the obstacle area.

[0090] ○ In the straight line selection step S24, the detection unit 22 may select the straight line L by combining the method of comparing the distance D between the straight line L shown in the above embodiment and the center C of the captured image I with a predetermined distance Dth and the method using the above learned model M. That is, the detection unit 22 may select, as the straight line L that can form the vanishing point V, the straight line L that passes near the center C of the captured image I and is located in the obstacle area.

[0091] ○ When it is clear that the inclination of the camera 21 does not change suddenly, the detection unit 22 may not detect the inclination of the camera 21 every time the camera 21 captures an image. The detection unit 22 may detect the inclination of the camera 21 after the camera 21 has captured images a plurality of times.

[0092] Specifically, in the imaging step S11, the camera 21 repeatedly captures images at a predetermined interval. Thereby, captured images I of a plurality of frames are obtained. As an example, the detection unit 22 may estimate the inclination of the camera 21 using one captured image I out of the captured images I of a plurality of frames.

[0093] As another example, the detection unit 22 may estimate the inclination of the camera 21 using the captured images I of a plurality of frames. For example, for the captured image I of each frame, the detection unit 22 performs a series of processes from the distortion correction step S21 to the intersection selection step S26. Then, in the vanishing point estimation step S27, the detection unit 22 calculates the average value or the median value of the positions of the intersections Q from the intersections Q of the captured images I of the plurality of frames. The detection unit 22 sets the calculated average value or median value of the position of the intersection Q as the position of the vanishing point V.

[0094] ○ The detection unit 22 may estimate the inclination of the camera 21 with respect to the forklift 10. The forklift 10 has components including linearly extending edges along the vertical direction. Components including linearly extending edges along the vertical direction are, for example, the front pillar 15, the rear pillar 16, and the mast 17. Therefore, when estimating the vanishing point V, instead of the linearly extending edges of the obstacle S, the linearly extending edges of the forklift 10 can be used. Specifically, the detection unit 22 estimates the inclination of the camera 21 with respect to the forklift 10 as follows.

[0095] The storage unit 24 of the detection unit 22 stores the learned model M. The learned model M in this modification example is learned to classify each pixel constituting the input image as a pixel displaying the host vehicle, a pixel displaying the obstacle S, or a pixel displaying the road surface R. The detection unit 22 performs semantic segmentation on the captured image I after distortion correction using the learned model M. The detection unit 22 can grasp the host vehicle region, the obstacle region, and the road surface region in the captured image I from the result of the semantic segmentation.

[0096] As shown in FIG. 17, the detection unit 22 selects, as a line that can form the vanishing point V, the line L located in the host vehicle region among the numerous lines L acquired in the line acquisition step S23. In addition, when the vertical direction of the forklift 10 coincides with the vertical direction, the inclination of the camera 21 with respect to the forklift 10 is the same as the inclination of the camera 21 with respect to the vertical direction. Therefore, the detection unit 22 may estimate the inclination of the camera 21 with respect to the vertical direction by estimating the inclination of the camera 21 with respect to the forklift 10.

[0097] ○ In the vanishing point estimation step S27, the detection unit 22 may use the median of the positions of the plurality of intersection points Q in the captured image I as the position of the vanishing point V. ○ In the vanishing point estimation step S27, the detection unit 22 may estimate the vanishing point V as follows.

[0098] The storage unit 24 of the detection unit 22 stores the learned model M. The learned model M in this modification example is machine-learned to output the vanishing point V from the input plurality of straight lines L. Specifically, the learned model M is learned by supervised learning. The teacher data has data of a plurality of straight lines L and correct answer data of the vanishing point V corresponding to the plurality of straight lines L. The detection unit 22 estimates the vanishing point V from the plurality of straight lines L by the learned model M. Note that the plurality of straight lines L are the straight lines L selected in the straight line selection step S24. In this case, the intersection calculation step S25 and the intersection selection step S26 become unnecessary.

[0099] ○ In the vanishing point estimation step S27, the detection unit 22 may estimate the vanishing point V as follows. The storage unit 24 of the detection unit 22 stores the learned model M. The learned model M in this modification example is machine-learned to output the vanishing point V from the input plurality of intersection points Q. Specifically, the learned model M is learned by supervised learning. The teacher data has data of a plurality of intersection points Q and correct answer data of the vanishing point V corresponding to the plurality of intersection points Q. The detection unit 22 estimates the vanishing point V from the plurality of intersection points Q by the learned model M. Note that the plurality of intersection points Q are preferably the intersection points Q selected in the intersection selection step S26, but may also be the intersection points Q calculated in the intersection calculation step S25.

[0100] ○ The obstacle detection device 20 and the obstacle detection method may be applied to industrial vehicles other than the forklift 10, for example, towing tractors. ○ The obstacle detection device 20 and the obstacle detection method may be applied to vehicles other than industrial vehicles. Examples of vehicles other than industrial vehicles include automobiles and trucks. When the obstacle detection device 20 and the obstacle detection method are applied to an automobile or a truck, it is preferable to use components having linear edges such as pillars and window frames for estimating the vanishing point V.

[0101] ○ The obstacle detection device 20 and the obstacle detection method may be applied to devices other than vehicles. For example, the obstacle detection device 20 may be applied to a drone. In this case, the camera 21 is mounted on the drone. The detection unit 22 may be mounted on the drone together with the camera 21, or may be on the ground. When the detection unit 22 is on the ground, the detection unit 22 acquires the captured image I from the camera 21 by wireless communication with the camera 21.

[0102] For example, the obstacle detection device 20 may be applied to a monitoring device. The monitoring device includes the obstacle detection device 20 and a notification device connected to the obstacle detection device 20. The camera 21 is attached, for example, to the upper part of a shelf or the ceiling of a factory. The camera 21 images vertically downward. The detection unit 22 transmits the detection result of the obstacle S to the notification device. The notification device receives the detection result of the obstacle S from the detection unit 22. Based on the detection result of the obstacle S, the notification device activates, for example, a buzzer or an alarm lamp, or warns the obstacle S by wireless communication with the obstacle S.

Explanation of Reference Numerals

[0103] 10... Forklift as a vehicle, 20... Obstacle detection device, 21... Camera, 22... Detection unit as an estimation unit, S11... Imaging step, S12... Inclination estimation step, S13... Detection step, I... Captured image, L... Straight line, Q... Intersection point, V... Vanishing point, S... Obstacle, M... Learned model.

Claims

1. An obstacle detection device comprising a camera and a detection unit that detects the position of an obstacle from a captured image of the camera, comprising an estimation unit that estimates the tilt of the camera, The estimation unit, a straight line acquisition process for acquiring a straight line from the captured image, a straight line selection process for selecting, from among the plurality of acquired straight lines, a straight line that can form a vanishing point in the captured image, a vanishing point estimation process for estimating the position of the vanishing point in the captured image based on the plurality of selected straight lines, a tilt calculation process for calculating the tilt of the camera from the distance between the estimated vanishing point and the center of the captured image and the focal length of the camera, performs, The detection unit detects the position of the obstacle in consideration of the tilt of the camera estimated by the estimation unit. An obstacle detection device characterized by this.

2. The estimation unit, an intersection calculation process for calculating an intersection point where the plurality of straight lines selected by the straight line selection process intersect, an intersection point selection process for selecting, from among the plurality of calculated intersection points, an intersection point within a predetermined range from the center of the captured image, performs, In the vanishing point estimation process, the obstacle detection device according to claim 1, wherein the average value or median value of the positions of the plurality of intersection points selected in the intersection point selection process is set as the position of the vanishing point.

3. The estimation unit performs an intersection calculation process for calculating an intersection point where the plurality of straight lines selected by the straight line selection process intersect, The estimation unit has a learned model that has been machine-learned to output the position of the vanishing point when the plurality of calculated intersection points are input, The estimation unit performs the vanishing point estimation process using the learned model. The obstacle detection device according to claim 1.

4. The estimation unit has a learned model that has been machine-learned to output the position of the vanishing point when the plurality of straight lines selected by the straight line selection process are input, The estimation unit performs the vanishing point estimation process using the learned model. The obstacle detection device according to claim 1.

5. The camera is a fisheye camera having a fisheye lens, The estimation unit, performs a distortion correction process for correcting the distortion of the captured image, In the straight line acquisition process, the straight line is acquired from the captured image after the distortion correction process. The obstacle detection device according to any one of claims 1 to 4.

6. The estimation unit performs image processing on the captured image before performing the straight line acquisition process. The obstacle detection device according to any one of claims 1 to 5.

7. The estimation unit grasps an obstacle region including an image of the obstacle from the captured image by performing semantic segmentation, The obstacle detection device according to any one of claims 1 to 6, wherein in the straight line selection process, the straight line located in the obstacle region is selected as a straight line that can form the vanishing point.

8. The obstacle detection device is mounted on a vehicle, The estimation unit grasps a host vehicle region including an image of the vehicle on which the obstacle detection device is mounted from the captured image by performing semantic segmentation, The obstacle detection device according to any one of claims 1 to 6, wherein in the straight line selection process, the straight line located in the host vehicle region is selected as a straight line that can form the vanishing point.

9. An imaging step of imaging by a camera, A detection step of detecting the position of an obstacle from the captured image of the camera by a detection unit connected to the camera, An obstacle detection method having: The obstacle detection method having an inclination estimation step of estimating an inclination of the camera by an estimation unit connected to the camera, The inclination estimation step includes: A straight line acquisition step of the estimation unit acquiring a straight line from the captured image, A straight line selection step of selecting, from among the plurality of straight lines acquired by the estimation unit, a straight line that can form a vanishing point in the captured image, A vanishing point estimation step of estimating the position of the vanishing point in the captured image based on the plurality of straight lines selected by the estimation unit, An inclination calculation step of calculating the inclination of the camera from the distance between the vanishing point estimated by the estimation unit and the center of the captured image, and the focal length of the camera, and having In the detection step, the detection unit detects the position of the obstacle in consideration of the inclination of the camera estimated in the inclination estimation step.

Images