Obstacle detection device and obstacle detection method

Abstract

To provide an obstacle detection device and an obstacle detection method, capable of detecting a position of an obstacle in a real world.SOLUTION: An obstacle detection device includes a camera attached to a fork lift and a detection unit that detects a person as an obstacle from a captured image by the camera. The camera is arranged so that an optical axis is inclined to a road surface as a reference surface where a distance from the camera is known. A portion located on the reference surface of the obstacle is taken as a target portion. When the obstacle is the person and the reference surface is the road surface, the target portion is the feet. The detection unit estimates coordinates of the feet in a captured image coordinate system. The detection unit converts the coordinates of the feet in the captured image coordinate system into those in a development image coordinate system as a coordinate system on a development image parallel to the road surface. The detection unit uses a relational expression between the coordinates of the feet in the development image coordinate system and a world coordinate system as a real world coordinate system centering on the camera, and the distance between the camera and the road surface to calculate a position of the feet on the road surface.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an obstacle detection device and an obstacle detection method. [Background technology]

[0002] Patent Document 1 discloses a work machine periphery monitoring device that monitors the periphery of a work machine. The work machine periphery monitoring device includes a camera and a processor. The camera captures images of the periphery of the work machine. The processor extracts a person image from captured image data that represents an image of the periphery of the work machine captured by the camera. The processor measures the distance between the work machine and the person based on the extracted person image and installation parameters of the camera. Specifically, the processor acquires foot coordinates, which are two-dimensional coordinates of the feet of the person image in the captured image data. The processor measures the distance between the work machine and the person based on the acquired foot coordinates and the installation parameters of the camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6940906 Summary of the Invention [Problem to be solved by the invention]

[0004] The perimeter monitoring device of Patent Document 1 may be able to obtain the distance between the work machine and a person, but cannot obtain the position of the person on the ground. [Means for solving the problem]

[0005] An obstacle detection device for solving the above problems is an obstacle detection device comprising a camera attached to an industrial vehicle and a detection unit that detects obstacles from images captured by the camera, wherein the camera is positioned so that its optical axis is inclined with respect to a reference plane whose distance from the camera is known, and when a portion of the obstacle located on the reference plane is treated as a target portion, the detection unit estimates the coordinates of the target portion in a captured image coordinate system, which is a coordinate system on the captured image, and coordinate-transforms the coordinates of the target portion from the captured image coordinate system to an unfolded image coordinate system, which is a coordinate system on an unfolded image parallel to the reference plane, and calculates the position of the target portion on the reference plane using a relationship between the coordinates of the target portion in the unfolded image coordinate system and the coordinates of the target part in a world coordinate system, which is a real world coordinate system centered on the camera, and the distance between the camera and the reference plane.

[0006] According to the above configuration, the detection unit converts the coordinates of the target part from the photographed image coordinate system to the unfolded image coordinate system. In addition, the distance between the camera and the reference plane is known. Therefore, the detection unit can calculate the position of the target part on the reference plane using a relational expression between the coordinates of the target part in the unfolded image coordinate system and the coordinates of the target part in the world coordinate system. Therefore, the position of an obstacle in the real world can be detected.

[0007] In the above obstacle detection device, the camera may be a fisheye camera or a wide-angle camera, and the detection unit may correct distortion of the captured image when transforming the coordinates of the target area from the captured image coordinate system to the unfolded image coordinate system.

[0008] According to the above configuration, the camera is a fisheye camera or a wide-angle camera with a wide angle of view. This allows obstacles to be detected over a wide range. In addition, the detection unit corrects distortion of the captured image when converting the coordinates of the target area from the captured image coordinate system to the unfolded image coordinate system. This makes it easier to detect the position of an obstacle in the real world.

[0009] In the above obstacle detection device, the detection unit has a trained model that is generated by machine learning using a teacher image including the target part and teacher data having coordinates of the target part in the teacher image, and when an image is input, outputs the coordinates of the target part in the image, and the image input to the teacher image and the trained model may be the captured image before distortion is corrected.

[0010] When converting a captured image into an unfolded image, the angle of view is limited, so that depending on the position of an obstacle in the fisheye image, the obstacle may not fit in the unfolded image. Also, the obstacle may be distorted in the unfolded image. According to the above configuration, the image input to the teacher image and the trained model is the captured image before the distortion is corrected. This makes it possible to generate the trained model more effectively and improve the estimation accuracy of the coordinates of the target part compared to when the image input to the teacher image and the trained model is the unfolded image.

[0011] The obstacle detection device may further include an angle detection means for detecting an angle of the camera, and the detection unit may perform coordinate conversion of the coordinates of the target portion from the captured image coordinate system to the unfolded image coordinate system using the angle of the camera detected by the angle detection means.

[0012] According to the above configuration, the detection unit converts the coordinates of the target part using the camera angle detected by the angle detection unit, so that even if the camera angle changes, it is possible to prevent a decrease in detection accuracy of the position of an obstacle in the real world.

[0013] In the obstacle detection device, the obstacle may be a person, the target part may be the person's feet, and the reference surface may be a road surface. According to the above configuration, the installation height of the camera from the road surface is the distance between the camera and the reference surface, so it is easy to accurately grasp the distance between the camera and the reference surface. In addition, the feet of a person are in contact with the road surface, so they are suitable as target parts. Therefore, the detection accuracy of the obstacle position can be improved.

[0014] An obstacle detection method for solving the above-mentioned problems is an obstacle detection method in which a detection unit detects an obstacle from an image captured by a camera attached to an industrial vehicle, the camera being positioned so that its optical axis is inclined with respect to a reference plane whose distance from the camera is known, and when a portion of the obstacle located on the reference plane is a target portion, the detection unit estimates the coordinates of the target portion in a captured image coordinate system, which is a coordinate system on the captured image, the detection unit converts the coordinates of the target portion from the captured image coordinate system to an unfolded image coordinate system, which is a coordinate system on an unfolded image parallel to the reference plane, and the detection unit calculates the position of the target portion on the reference plane using a relationship equation between the coordinates of the target portion in the unfolded image coordinate system and the coordinates of the target part in a world coordinate system, which is a real world coordinate system centered on the camera, and the distance between the camera and the reference plane.

[0015] According to the above method, the detection unit converts the coordinates of the target part from the photographed image coordinate system to the unfolded image coordinate system. In addition, the distance between the camera and the reference plane is known. Therefore, the detection unit can calculate the position of the target part on the reference plane using a relational expression between the coordinates of the target part in the unfolded image coordinate system and the coordinates of the target part in the world coordinate system. Therefore, the position of an obstacle in the real world can be detected. Effect of the Invention

[0016] According to the present invention, the position of an obstacle in the real world can be detected. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a plan view of a forklift. [Diagram 2] FIG. 2 is a block diagram showing the configuration of a forklift. [Diagram 3] FIG. 3 is a schematic diagram for explaining the fisheye image coordinate system, the developed image coordinate system, and the world coordinate system. [Figure 4]FIG. 4 is a flow chart showing an obstacle detection method. [Diagram 5] FIG. 5 is a diagram showing a captured image. [Figure 6] FIG. 6 is a diagram showing an expanded image. [Figure 7] FIG. 7 is a diagram showing a modified example of the method for estimating the coordinates of feet in a captured image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] An embodiment of an obstacle detection device and an obstacle detection method will be described below with reference to Figs. 1 to 6. The obstacle detection device and the obstacle detection method of this embodiment are applied to a forklift as an industrial vehicle. In the following description, "front", "rear", "left", "right", "up" and "down" refer to the forward movement direction of the forklift. The left-right direction coincides with the vehicle width direction.

[0019] <Forklift> As shown in Fig. 1, the forklift 10 includes a vehicle body 11 and two drive wheels 12. The vehicle body 11 includes a head guard 13 and four pillars 14. The head guard 13 is supported by the four pillars 14.

[0020] As shown in FIG. 2, the forklift 10 is equipped with a travel motor 15. The travel motor 15 drives the drive wheels 12 to rotate, thereby causing the forklift 10 to travel. The forklift 10 is equipped with a warning device 16. The warning device 16 issues a warning when an obstacle is present around the forklift 10. Examples of the warning device 16 include a warning buzzer and a warning light. The forklift 10 is equipped with a control device 17. The control device 17 controls the forklift 10. Specifically, the control device 17 controls the travel motor 15 to control the travel operation of the forklift 10. The control device 17 controls the warning operation of the warning device 16.

[0021] The control device 17 has a processor 17a and a storage unit 17b. The processor 17a may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). The storage unit 17b includes a random access memory (RAM) and a read only memory (ROM). It can be said that the storage unit 17b stores program codes or instructions configured to cause the processor 17a to execute processing. The storage unit 17b, i.e., the computer readable medium, includes any available medium accessible by a general-purpose or dedicated computer. The control device 17 may be configured by a hardware circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The control device 17, which is a processing circuit, may include one or more processors operating according to a computer program, one or more hardware circuits such as an ASIC or an FPGA, or a combination thereof.

[0022] <Obstacle detection device> The forklift 10 is equipped with an obstacle detection device 20. The obstacle detection device 20 detects obstacles present around the forklift 10. In this embodiment, the obstacle is a person P such as a worker. The obstacle detection device 20 has a camera 21, a detection unit 22, and an angle detection means 23. The camera 21 and the angle detection means 23 are each connected to the detection unit 22.

[0023] The camera 21 is a digital camera. The camera 21 has a lens and an image sensor. The camera 21 is a monocular camera. The lens in this embodiment is a fisheye lens. Therefore, the camera 21 in this embodiment is a fisheye camera. Moreover, an image If captured by the camera 21 (see FIG. 5) is a fisheye image in which the subject is captured in a distorted state. The image sensor is, for example, a CCD image sensor (Charge Coupled Device image sensor) or a CMOS image sensor (Complementary Metal Oxide Semiconductor image sensor). The image sensor is composed of a plurality of pixels (picture elements).

[0024] As shown in Fig. 1, the camera 21 is attached to the forklift 10. In this embodiment, the cameras 21 are attached to both sides of the head guard 13 in the vehicle width direction. In other words, two cameras 21 are attached to the forklift 10. The camera 21 in this embodiment captures images of the sides of the forklift 10. Therefore, the obstacle detection device 20 in this embodiment detects obstacles present on the sides of the forklift 10. The camera 21 repeatedly captures images at a predetermined capture period.

[0025] As shown in Fig. 3, a real world coordinate system with the camera 21 at its center is defined as a world coordinate system. The world coordinate system is a three-dimensional coordinate system in which the X-axis, the Y-axis, and the Z-axis are mutually orthogonal. In this embodiment, the X-axis extends along the left-right direction of the forklift 10. The Y-axis extends along the front-rear direction of the forklift 10. The Z-axis extends along the up-down direction of the forklift 10. The road surface R on which the forklift 10 travels is a plane perpendicular to the Z-axis.

[0026] In this embodiment, the camera 21 is installed at a height H from the road surface R. In other words, the installation height of the camera 21 from the road surface R is H. Therefore, the road surface R is a reference plane whose distance from the camera 21 is known. A part of the obstacle located on the reference plane is the target part. In this embodiment, the part of the person P located on the road surface R is the feet Pf. Therefore, the target part in this embodiment is the feet Pf.

[0027] The camera 21 is disposed so that its optical axis is inclined with respect to a reference plane. That is, the camera 21 is disposed so that its optical axis is inclined with respect to the road surface R. In this embodiment, the camera 21 rotates around the Y axis by a pitch angle θ so as to capture an image of the side and below the forklift 10. Therefore, the optical axis of the camera 21 is inclined by the pitch angle θ with respect to the Z axis. The pitch angle θ is set to, for example, 30 degrees. In this embodiment, the camera 21 does not rotate around the X axis or the Z axis. Therefore, the roll angle φ and yaw angle ψ of the camera 21 are 0 degrees.

[0028] The detection unit 22 detects an obstacle from the image If captured by the camera 21. As shown in FIG. 2, the detection unit 22 has a processor 22a and a storage unit 22b. The processor 22a may be, for example, a CPU, a GPU, or a DSP. The storage unit 22b includes a RAM and a ROM. It can be said that the storage unit 22b stores program code or instructions configured to cause the processor 22a to execute processing. The storage unit 22b, i.e., the computer-readable medium, includes any available medium accessible by a general-purpose or dedicated computer. The detection unit 22 may be configured by a hardware circuit such as an ASIC or an FPGA. The detection unit 22, which is a processing circuit, may include one or more processors operating according to a computer program, one or more hardware circuits such as an ASIC or an FPGA, or a combination thereof.

[0029] The memory unit 22b stores a trained model M. The trained model M of this embodiment is a model that simultaneously performs object detection and keypoint detection. Specifically, when an image is input, the trained model M outputs an obstacle area in the image in a rectangular frame B, and outputs the coordinates of a target part in the image. In this embodiment, when an image is input, the trained model M outputs a person P area in the image in a rectangular frame B, and outputs the coordinates of feet Pf in the image.

[0030] The trained model M is generated by machine learning using a plurality of teacher data. Each teacher data has a teacher image including a target part of an obstacle, a rectangular frame B showing the area of ​​the obstacle in the teacher image, and the coordinates of the target part in the teacher image. In this embodiment, each teacher data has a teacher image including the feet Pf of a person P, a rectangular frame B showing the area of ​​the person P in the teacher image, and the coordinates of the feet Pf in the teacher image. The captured image If, ​​which is a fisheye image, is used as the teacher image. That is, the captured image If before the distortion correction described later is performed is used as the teacher image.

[0031] The angle detection means 23 detects the angle of the camera 21. The angles of the camera 21 are the pitch angle θ, roll angle φ, and yaw angle ψ of the camera 21. The angle detection means 23 has an IMU (Inertial Measurement Unit).

[0032] <Obstacle detection method> The obstacle detection process performed by the obstacle detection device 20 will now be described together with an obstacle detection method. The obstacle detection device 20 of this embodiment performs the following process each time the camera 21 captures an image.

[0033] As shown in FIG. 4, the detection unit 22 acquires a captured image If from the camera 21 in step S11. Fig. 5 shows an image If captured by the camera 21. As described above, since the camera 21 in this embodiment is a fisheye camera, the captured image If is a fisheye image. As shown in Fig. 1, when a person P is present to the side of the forklift 10, the person P is included in the captured image If.

[0034] 4, in step S12, the detection unit 22 estimates the coordinates of the target part in the captured image coordinate system from the captured image If. The detection unit 22 of this embodiment estimates the coordinates of the target part in the captured image coordinate system and at the same time estimates the area of ​​the obstacle in the captured image If using a rectangular frame B. That is, the detection unit 22 estimates the coordinates of the feet Pf in the captured image coordinate system and at the same time estimates the area of ​​the person P in the captured image If using a rectangular frame B.

[0035] As shown in Figures 3 and 5, the captured image coordinate system is a coordinate system on the captured image If. The origin of the captured image coordinate system is the upper left corner of the captured image If. The captured image coordinate system is a two-dimensional coordinate system in which the Uf axis and the Vf axis are orthogonal to each other. The Uf axis extends along the left-right direction of the captured image If. The Vf axis extends along the up-down direction of the captured image If. The Uf axis and the Vf axis are orthogonal to the optical axis.

[0036] In this embodiment, the detection unit 22 uses the trained model M to estimate the area of ​​the person P in the captured image If using a rectangular frame B, and also estimates the coordinates (Uff, Vff) of the feet Pf in the captured image coordinate system. In detail, the captured image If before distortion correction, which will be described later, is input to the trained model M. Then, the trained model M outputs the area of ​​the person P in the input captured image If using a rectangular frame B. In addition, the trained model M outputs the coordinates (Uff, Vff) of the feet Pf in the input captured image If.

[0037] As shown in FIG. 4, in step S13, the detection unit 22 converts the coordinates of the feet Pf from the captured image coordinate system to the developed image coordinate system. As shown in Figures 3 and 6, the exfoliated image coordinate system is a coordinate system on the exfoliated image Ie that is parallel to the reference plane. The exfoliated image Ie is a plane perpendicular to the Z axis. The origin of the exfoliated image coordinate system is the upper left corner of the exfoliated image Ie. The exfoliated image coordinate system is a two-dimensional coordinate system in which the Ue axis and the Ve axis are perpendicular to each other. The Ue axis extends along the left-right direction of the exfoliated image Ie. The Ve axis extends along the up-down direction of the exfoliated image Ie. The Ue axis and the Ve axis are perpendicular to the Z axis.

[0038] For convenience of explanation, the exfoliated image Ie is illustrated, but in actual processing, the detection unit 22 does not convert the captured image If to the exfoliated image Ie, and therefore the exfoliated image Ie is not generated. As will be described later, as long as the coordinates in the captured image coordinate system can be converted to coordinates in the exfoliated image coordinate system parallel to the reference plane (as long as the coordinate conversion is set in advance), the image conversion from the captured image If to the exfoliated image Ie is arbitrary.

[0039] The detection unit 22 performs coordinate conversion from the coordinates (Uff, Vff) of the feet Pf in the captured image coordinate system to the coordinates (Uef, Vef) of the feet Pf in the unfolded image coordinate system. In detail, the detection unit 22 first corrects the distortion of the captured image If. Next, the detection unit 22 performs coordinate conversion from the coordinates (Uff, Vff) of the feet Pf in the fisheye image coordinate system in which the distortion has been corrected to the coordinates (Uef, Vef) of the feet Pf in the unfolded image coordinate system. A rotation matrix RM is used for this coordinate conversion. The rotation matrix RM is generated based on the angle of the camera 21 detected by the angle detection means 23. That is, the detection unit 22 performs coordinate conversion using the angle of the camera 21 detected by the angle detection means 23. In addition, the camera parameter K of the formula (1) is used for the coordinate conversion.

[0040]

number

[0041] 4, in step S14, the detection unit 22 calculates the position of the target part on the reference surface. In this embodiment, the detection unit 22 calculates the position of the feet Pf on the road surface R. The detection unit 22 calculates the position of the feet Pf on the road surface R using a relational expression between the coordinates (Uef, Vef) of the feet Pf in the developed image coordinate system and the coordinates (Xf, Yf, Zf) of the feet Pf in the world coordinate system, and the distance between the camera 21 and the road surface R, which is the reference surface.

[0042] The relation between the coordinates (Uef, Vef) of the feet Pf in the developed image coordinate system and the coordinates (Xf, Yf, Zf) of the feet Pf in the world coordinate system is expressed by the formula (2). Here, s is the scale. Furthermore, the formula (3) is obtained by transforming the formula (2). The inverse matrix K of the camera parameter K in the formula (3) is -1 is expressed by equation (4).

[0043]

number

[0044]

number

[0045] The control device 17 controls the forklift 10 based on the detection result by the detection unit 22. The control device 17 of this embodiment controls the forklift 10 based on the position of the person P in the real world detected by the detection unit 22. Specifically, the control device 17 controls the forklift 10 differently depending on whether the person P is present on the traveling path of the forklift 10 or not. For example, when the person P is not present on the traveling path of the forklift 10, the control device 17 lowers the warning level of the warning device 16 compared to when the person P is present on the traveling path of the forklift 10. For example, when the person P is present on the traveling path of the forklift 10, the control device 17 controls the travel motor 15 to decelerate the forklift 10 or restrict the start of the forklift 10.

[0046] [Actions and Effects of the Present Embodiment] The operation and effects of this embodiment will be described. (1) The camera 21 is disposed so that its optical axis is inclined with respect to the road surface R, the distance from the camera 21 being known. The feet Pf of the person P are located on the road surface R. The detection unit 22 estimates the coordinates of the feet Pf in a captured image coordinate system, which is a coordinate system on the captured image If. The detection unit 22 converts the coordinates of the feet Pf in the captured image coordinate system into coordinates of the feet Pf in an expanded image coordinate system, which is a coordinate system on the expanded image Ie parallel to the road surface R. The detection unit 22 calculates the position of the feet Pf on the road surface R using a relational expression between the coordinates of the feet Pf in the expanded image coordinate system and the coordinates of the feet Pf in a world coordinate system, which is a real world coordinate system centered on the camera 21, and the distance between the camera 21 and the road surface R.

[0047] According to this configuration, the detection unit 22 converts the coordinates of the feet Pf from the captured image coordinate system to the unfolded image coordinate system. In addition, the distance between the camera 21 and the road surface R is known. Therefore, the detection unit 22 can calculate the position of the feet Pf on the road surface R using a relational expression between the coordinates of the feet Pf in the unfolded image coordinate system and the coordinates of the feet Pf in the world coordinate system. Therefore, the detection unit 22 can detect the position of the person P in the real world.

[0048] (2) The detection unit 22 converts the coordinates of the feet Pf from the captured image coordinate system to the unfolded image coordinate system using not only the pitch angle θ but also the roll angle φ and the yaw angle ψ. Therefore, the detection unit 22 detects the position of the person P in the real world, taking into account not only the pitch angle θ but also the roll angle φ and the yaw angle ψ. This increases the degree of freedom in the mounting orientation of the camera 21.

[0049] (3) The camera 21 is a fisheye camera. This widens the angle of view of the camera 21, allowing the person P to be detected over a wide range. In addition, the detection unit 22 corrects distortion of the captured image If when converting the coordinates of the feet Pf from the captured image coordinate system to the developed image coordinate system. This makes it easier to detect the position of the person P in the real world.

[0050] (4) When the captured image If is converted into the unfolded image Ie, the angle of view is limited, and depending on the position of the person P in the captured image If, ​​the person P may not fit within the unfolded image Ie. Also, as shown in FIG. 6, the person P is distorted in the unfolded image Ie. In this embodiment, the image input to the teacher image and the trained model M is the captured image If before the distortion is corrected. This makes it possible to generate the trained model M more effectively and improve the estimation accuracy of the coordinates of the feet Pf than when the image input to the teacher image and the trained model M is the unfolded image Ie.

[0051] (5) The angle of the camera 21 may change. For example, when the forklift 10 turns, the forklift 10 may tilt left and right, causing a change in the pitch angle θ. Also, when the forklift 10 holds a load or travels on a slope, the forklift 10 may tilt forward and backward, causing a change in the roll angle φ. The obstacle detection device 20 of this embodiment includes an angle detection means 23 that detects the angle of the camera 21. The detection unit 22 converts the coordinates of the feet Pf using the angle of the camera 21 detected by the angle detection means 23. Therefore, even if the angle of the camera 21 changes, it is possible to suppress a decrease in the detection accuracy of the position of the person P in the real world.

[0052] (6) In this embodiment, the obstacle is person P. The target part is the feet Pf of person P. The reference plane is the road surface R. Since the installation height H of camera 21 from the road surface R is the distance between camera 21 and the reference plane, it is easy to accurately grasp the distance between camera 21 and the reference plane. In addition, since the feet Pf of person P are in contact with the road surface R, they are suitable as the target part. Therefore, the detection accuracy of the position of person P can be improved.

[0053] (7) The obstacle detection device 20 can detect the position of the person P in the real world. Therefore, the control device 17 can control the traveling operation of the forklift 10 and the warning operation of the warning device 16 in a more detailed manner than when the obstacle detection device 20 can detect only the distance from the forklift 10 to the person P, for example.

[0054] (8) The camera 21 is a monocular camera. Therefore, the cost of detecting the person P can be reduced compared to the case where an expensive camera such as a stereo camera is used. [Example of change] The above embodiment can be modified as follows: The above embodiment and the following modifications can be combined with each other to the extent that no technical contradiction occurs.

[0055] The obstacle detected by the obstacle detection device 20 and the obstacle detection method is not limited to the person P and may be an object. The camera 21 is not limited to a fisheye camera. The camera 21 may be a wide-angle camera, or a camera with a narrower angle of view than a fisheye camera or a wide-angle camera.

[0056] The number of cameras 21 attached to the forklift 10 is not limited to two and may be changed as appropriate. The location where the camera 21 is attached to the forklift 10 is not limited to the head guard 13 and may be changed as appropriate depending on the required detection range of obstacles.

[0057] The camera 21 may capture images in front of or behind the forklift 10. In this case, the obstacle detection device 20 detects obstacles present in front of or behind the forklift 10. In the above embodiment, for ease of explanation, the camera 21 is arranged in a state rotated only around the Y axis, but it may be arranged in a state rotated around the X axis or around the Z axis.

[0058] In the above embodiment, the detection unit 22 estimates the coordinates of the feet Pf in the captured image coordinate system and at the same time estimates the area of ​​the person P in the captured image If using the rectangular frame B. However, this is not limited to this. The detection unit 22 may estimate only the coordinates of the feet Pf in the captured image coordinate system without estimating the area of ​​the person P in the captured image If using the rectangular frame B. In this case, the trained model M does not need to be trained to detect objects as long as it has been trained to detect key points.

[0059] The reference plane is not limited to the road surface R. For example, the distance from the camera 21 to the head of the person P standing on the road surface R can be known by subtracting the average height of the person P from the installation height H of the camera 21 from the road surface R. Therefore, the reference plane may be a plane located above the road surface R by the average height of the person P. In this case, the target part is the head or helmet.

[0060] In the above embodiment, the detection unit 22 estimates the coordinates of the feet Pf in the captured image coordinate system by key point detection. However, the detection unit 22 may estimate the coordinates of the feet Pf in the captured image coordinate system by object detection.

[0061] 7, the detection unit 22 may estimate the coordinates of the intersection of a line L passing through an arbitrary point Ifp in the captured image If and the center Bc of the rectangular frame B with the rectangular frame B as the coordinates of the feet Pf in the captured image coordinate system. In the example shown in FIG. 7, the arbitrary point Ifp in the captured image If is a point located at the center of the lower end of the captured image If. The arbitrary point Ifp in the captured image If may be changed as appropriate depending on the angle of the camera 21.

[0062] In this case, the trained model M does not need to be trained to output the coordinates of the feet Pf in the captured image coordinate system as long as it has been trained to output the area of ​​the person P in the captured image If in a rectangular frame B. This is therefore effective when there is already a trained model M trained to output the area of ​​the person P in the captured image If in a rectangular frame B, or when there is no teacher data for training the trained model M to output the coordinates of the feet Pf in the captured image coordinate system. In addition, the coordinates of the feet Pf in the captured image If may be output based on the pixels of the imaging element of the camera 21.

[0063] If it is expected that the angle of the camera 21 will hardly change, the obstacle detection device 20 does not need to include the angle detection means 23. In this case, the detection unit 22 converts the coordinates of the feet Pf from the photographed image coordinate system to the unfolded image coordinate system using the angle of the camera 21 that is stored in advance.

[0064] The industrial vehicle is not limited to the forklift 10. The industrial vehicle may be another type of industrial vehicle, such as a towing tractor. [Note] The technical ideas that can be understood from the above-described embodiment and modified examples will be described below.

[0065] <Appendix 1> 1. An obstacle detection device comprising a camera attached to an industrial vehicle and a detection unit that detects an obstacle from an image captured by the camera, wherein the camera is positioned so that its optical axis is inclined with respect to a reference plane whose distance from the camera is known, and when a portion of the obstacle located on the reference plane is designated as a target portion, the detection unit estimates the coordinates of the target portion in a captured image coordinate system, which is a coordinate system on the captured image, and coordinate transforms the coordinates of the target portion from the captured image coordinate system to an unfolded image coordinate system, which is a coordinate system on an unfolded image parallel to the reference plane, and calculates a position of the target portion on the reference plane using a relationship equation between the coordinates of the target part in the unfolded image coordinate system and the coordinates of the target part in a world coordinate system, which is a real world coordinate system centered on the camera, and the distance between the camera and the reference plane.

[0066] <Appendix 2> 2. The obstacle detection device according to claim 1, wherein the camera is a fisheye camera or a wide-angle camera, and the detection unit corrects distortion of the captured image when converting the coordinates of the target area from the captured image coordinate system to the unfolded image coordinate system.

[0067] <Appendix 3> The obstacle detection device described in Appendix 2, wherein the detection unit has a trained model that is generated by machine learning using a teacher image including the target part and teacher data having coordinates of the target part in the teacher image, and when an image is input, outputs the coordinates of the target part in the image, and the image input to the teacher image and the trained model is the captured image before distortion is corrected.

[0068] <Appendix 4> 4. The obstacle detection device according to any one of claims 1 to 3, further comprising an angle detection means for detecting an angle of the camera, wherein the detection unit converts coordinates of the target portion from the captured image coordinate system to the unfolded image coordinate system using the angle of the camera detected by the angle detection means.

[0069] <Appendix 5> 5. The obstacle detection device according to any one of claims 1 to 4, wherein the obstacle is a person, the target part is the person's feet, and the reference surface is a road surface. [Explanation of symbols]

[0070] 10...forklift as industrial vehicle, 20...obstacle detection device, 21...camera, 22...detection unit, 23...angle detection means, If...captured image, Ie...unfolded image, M...trained model, P...person as obstacle, Pf...feet as target part, R...road surface as reference surface.

Claims

1. An obstacle detection device including a camera attached to an industrial vehicle and a detection unit that detects an obstacle from an image captured by the camera, the camera is disposed such that its optical axis is inclined with respect to a reference plane whose distance from the camera is known; When a portion of the obstacle located on the reference plane is a target portion, The detection unit is Estimating coordinates of the target region in a captured image coordinate system, which is a coordinate system on the captured image; A coordinate conversion is performed on the coordinates of the target portion from the photographed image coordinate system to a developed image coordinate system, which is a coordinate system on a developed image parallel to the reference plane; The position of the target part on the reference plane is calculated using a relational expression between the coordinates of the target part in the developed image coordinate system and the coordinates of the target part in a world coordinate system, which is a real world coordinate system centered on the camera, and a distance between the camera and the reference plane. An obstacle detection device comprising:

2. The camera is a fisheye camera or a wide-angle camera, The obstacle detection device according to claim 1 , wherein the detection unit corrects distortion of the captured image when converting the coordinates of the target portion from the captured image coordinate system to the unfolded image coordinate system.

3. the detection unit has a trained model that is generated by machine learning using teacher data having a teacher image including the target part and coordinates of the target part in the teacher image, and when an image is input, outputs the coordinates of the target part in the image; The obstacle detection device according to claim 2 , wherein the teacher image and the image input to the trained model are the captured images before distortion is corrected.

4. An angle detection means for detecting an angle of the camera is provided, 2. The obstacle detection device according to claim 1, wherein the detection unit converts coordinates of the target portion from the photographed image coordinate system to the unfolded image coordinate system using the camera angle detected by the angle detection means.

5. 2. The obstacle detection device according to claim 1, wherein the obstacle is a person, the target portion is the feet of the person, and the reference surface is a road surface.

6. An obstacle detection method in which a detection unit detects an obstacle from an image captured by a camera attached to an industrial vehicle, the camera is disposed such that an optical axis thereof is inclined with respect to a reference plane whose distance from the camera is known; When a portion of the obstacle located on the reference plane is a target portion, a step of estimating coordinates of the target portion in a captured image coordinate system, which is a coordinate system on the captured image, by the detection unit; a step of converting coordinates of the target site from the photographed image coordinate system to an expanded image coordinate system, which is a coordinate system on an expanded image parallel to the reference plane, by the detection unit; a step in which the detection unit calculates a position of the target part on the reference plane using a relational expression between coordinates of the target part in the expanded image coordinate system and coordinates of the target part in a world coordinate system, which is a real world coordinate system centered on the camera, and a distance between the camera and the reference plane; 11. An obstacle detection method comprising the steps of:

Images