Obstacle detection device and obstacle detection method

Abstract

Provided are an obstacle detection device and an obstacle detection method capable of detecting the position of an obstacle in the real world. An obstacle detection device is provided with a camera attached to a forklift, and a detection unit for detecting a person as an obstacle from an image captured by the camera. The camera is configured such that the optical axis is inclined with respect to a road surface that is a reference surface whose distance from the camera is known. A portion of the obstacle located on the reference surface is used as a target portion. When the obstacle is a person and the reference surface is a road surface, the target part is a foot part. The detection unit estimates the coordinates of the foot part in the captured image coordinate system. The detection unit converts the coordinates of the leg from a captured image coordinate system to an expanded image coordinate system, which is a coordinate system on an expanded image parallel to the road surface. The detection unit calculates the position of the foot on the road surface using a relational expression between the coordinates of the foot in the expanded image coordinate system and the coordinates of the foot in a world coordinate system, which is a real world coordinate system centered on the camera, and the distance between the camera and the road surface.

Description

Technical Field

[0001] The invention relates to an obstacle detection device and an obstacle detection method. Background Art

[0002] Patent document 1 discloses a peripheral monitoring device for a working machine that monitors the periphery of the working machine. The peripheral monitoring device for a working machine includes a photographing device and a processor. The photographing device photographs the periphery of the working machine. The processor extracts a person image from captured image data representing the peripheral image of the working machine captured by the photographing device. The processor measures the distance between the working machine and the person based on the extracted person image and the setting parameters of the photographing device. Specifically, the processor obtains the two-dimensional coordinates of the feet of the person image in the camera image data, namely, the foot coordinates. The processor measures the distance between the working machine and the person based on the obtained foot coordinates and the setting parameters of the photographing device.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent No. 6940906 Summary of the invention

[0006] Problem that the invention aims to solve

[0007] In the surrounding monitoring device of Patent Document 1, even if the distance between the working machine and the person can be acquired, the position of the person on the ground cannot be acquired.

[0008] Solutions for solving problems

[0009] An obstacle detection device for solving the above-mentioned problem is an obstacle detection device comprising a camera mounted on an industrial vehicle and a detection unit for detecting an obstacle from an image captured by the camera, wherein the camera is configured so that an optical axis is inclined relative to a reference plane whose distance from the camera is known, and when a portion of the obstacle located on the reference plane is taken as an object portion, the detection unit estimates the coordinates of the object portion in a captured image coordinate system, which is a coordinate system on the captured image, and performs coordinate conversion of the coordinates of the object portion from the captured image coordinate system to a developed image coordinate system, which is a coordinate system on the developed image parallel to the reference plane, and uses a relationship between the coordinates of the object portion in the developed image coordinate system and the coordinates of the object portion in a world coordinate system, as well as the distance between the camera and the reference plane, to calculate the position of the object portion on the reference plane, and the world coordinate system is a real world coordinate system centered on the camera.

[0010] According to the above configuration, the detection unit converts the coordinates of the object part from the captured image coordinate system to the expanded image coordinate system. In addition, the distance between the camera and the reference plane is known. Therefore, the detection unit can calculate the position of the object part on the reference plane using the relationship between the coordinates of the object part in the expanded image coordinate system and the coordinates of the object part in the world coordinate system. Therefore, the position of the obstacle in the real world can be detected.

[0011] In the obstacle detection device, the camera may be a fisheye camera or a wide-angle camera, and the detection unit may correct distortion of the captured image when converting the coordinates of the target part from the captured image coordinate system to the developed image coordinate system.

[0012] According to the above configuration, the camera is a fisheye camera or a wide-angle camera with a wide viewing angle. Thus, obstacles can be detected in a wide range. In addition, when the detection unit converts the coordinates of the object part from the captured image coordinate system to the expanded image coordinate system, the distortion of the captured image is corrected. Thus, the position of obstacles in the real world can be easily detected.

[0013] In the above-mentioned obstacle detection device, it may also be that the detection unit has a learning completion model, which is generated by performing machine learning using teacher data, and when an image is input, the coordinates of the object part in the image are output, the teacher data has a teacher image containing the object part and the coordinates of the object part in the teacher image, and the teacher image and the image input to the learning completion model are the captured images before the distortion is corrected.

[0014] When image conversion is performed from a captured image to an expanded image, the viewing angle is limited. Therefore, depending on the position of the obstacle in the fisheye image, the obstacle may not be included in the expanded image. In addition, the obstacle may be distorted in the expanded image. According to the above configuration, the teacher image and the image input to the learned model are captured images before the distortion is corrected. Thus, compared with the case where the teacher image and the image input to the learned model are expanded images, the learned model can be generated more efficiently, and the estimation accuracy of the coordinates of the object part can be improved.

[0015] The obstacle detection device may include an angle detection unit for detecting the angle of the camera, and the detection unit may convert the coordinates of the object part from the captured image coordinate system to the expanded image coordinate system using the angle of the camera detected by the angle detection unit.

[0016] According to the above configuration, the detection unit performs coordinate conversion on the coordinates of the target part using the angle of the camera detected by the angle detection unit. Therefore, even if the angle of the camera changes, it is possible to suppress a decrease in the detection accuracy of the position of the obstacle in the real world.

[0017] In the above obstacle detection device, the obstacle may be a person, the target part may be a foot of the person, and the reference plane may be a road surface.

[0018] According to the above configuration, since the installation height of the camera from the road surface becomes the distance between the camera and the reference surface, it is easy to accurately grasp the distance between the camera and the reference surface. In addition, since the human foot is in contact with the road surface, it is suitable as the target part. Therefore, the detection accuracy of the position of the obstacle can be improved.

[0019] An obstacle detection method for solving the above-mentioned problem is an obstacle detection method in which a detection unit detects an obstacle from an image captured by a camera mounted on an industrial vehicle, the gist of which is that the camera is configured so that an optical axis is inclined relative to a reference plane whose distance from the camera is known, and when a portion of the obstacle located on the reference plane is taken as an object portion, the obstacle detection method comprises: a step in which the detection unit estimates the coordinates of the object portion in a captured image coordinate system, wherein the captured image coordinate system is a coordinate system on the captured image; a step in which the detection unit performs coordinate conversion on the coordinates of the object portion from the captured image coordinate system to a developed image coordinate system, wherein the developed image coordinate system is a coordinate system on the developed image parallel to the reference plane; and a step in which the detection unit calculates the position of the object portion on the reference plane using a relational expression between the coordinates of the object portion in the developed image coordinate system and the coordinates of the object portion in a world coordinate system, and the distance between the camera and the reference plane, wherein the world coordinate system is a real world coordinate system centered on the camera.

[0020] According to the above method, the detection unit converts the coordinates of the object part from the captured image coordinate system to the expanded image coordinate system. In addition, the distance between the camera and the reference plane is known. Therefore, the detection unit can calculate the position of the object part on the reference plane using the relationship between the coordinates of the object part in the expanded image coordinate system and the coordinates of the object part in the world coordinate system. Therefore, the position of the obstacle in the real world can be detected.

[0021] Effects of the Invention

[0022] According to the present invention, the position of an obstacle in the real world can be detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a top view of the forklift.

[0024] Figure 2 It is a block diagram showing the structure of a forklift.

[0025] Figure 3 This is a schematic diagram for explaining the captured image coordinate system, the developed image coordinate system, and the world coordinate system.

[0026] Figure 4 is a flow chart showing an obstacle detection method.

[0027] Figure 5 It is a figure which shows the photographed image.

[0028] Figure 6 is a diagram showing an expanded image.

[0029] Figure 7 It is a diagram showing a modified example of the method of estimating the coordinates of the foot in the captured image.

[0030] Description of Reference Numerals

[0031] 10…a forklift as an industrial vehicle, 20…an obstacle detection device, 21…a camera, 22…a detection unit, 23…an angle detection unit, If…a captured image, Ie…an expanded image, M…a learned model, P…a person as an obstacle, Pf…a foot as an object part, and R…a road surface as a reference plane. DETAILED DESCRIPTION

[0032] Below, according to Figure 1 to Figure 6 An embodiment of the obstacle detection device and the obstacle detection method is described below. The obstacle detection device and the obstacle detection method of this embodiment are applied to a forklift as an industrial vehicle. In the following description, "front", "rear", "left", "right", "up", and "down" are "front", "rear", "left", "right", "up", and "down" when the forklift is used as a reference. The left-right direction is consistent with the vehicle width direction.

[0033] <Forklift>

[0034] like Figure 1 As shown, the forklift 10 includes a vehicle body 11 and two drive wheels 12. The vehicle body 11 has a roof guard 13 and four pillars 14. The roof guard 13 is supported by the four pillars 14.

[0035] like Figure 2As shown, the forklift 10 includes a travel motor 15. The drive wheel 12 is rotated by the drive of the travel motor 15, so that the forklift 10 performs a travel action. The forklift 10 includes a warning device 16. The warning device 16 warns when there is an obstacle around the forklift 10. As the warning device 16, for example, a warning buzzer or a warning light can be cited. The forklift 10 includes a control device 17. The control device 17 controls the forklift 10. Specifically, the control device 17 controls the travel action of the forklift 10 by controlling the travel motor 15. The control device 17 controls the warning action of the warning device 16.

[0036] The control device 17 has a processor 17a and a storage unit 17b. As the processor 17a, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) is used. The storage unit 17b includes a RAM (Random Access Memory) and a ROM (Read Only Memory). It can be said that the storage unit 17b stores program codes or instructions configured to cause the processor 17a to execute processing. The storage unit 17b, that is, the computer-readable medium, includes any available medium that can be accessed by a general or special computer. The control device 17 can also be composed of hardware circuits such as ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). The control device 17 as a processing circuit can include one or more processors that act according to a computer program, one or more hardware circuits such as ASIC or FPGA, or a combination thereof.

[0037] <Obstacle detection device>

[0038] The forklift 10 is equipped with an obstacle detection device 20. The obstacle detection device 20 detects obstacles existing around the forklift 10. The obstacle in this embodiment is a worker or other person P. The obstacle detection device 20 includes a camera 21, a detection unit 22, and an angle detection unit 23. The camera 21 and the angle detection unit 23 are connected to the detection unit 22, respectively.

[0039] The camera 21 is a digital camera. The camera 21 has a lens and an imaging element. The camera 21 is a single-lens reflex camera. The lens of this embodiment is a fisheye lens. Therefore, the camera 21 of this embodiment is a fisheye camera. In addition, the image If captured by the camera 21 (refer to Figure 5) is a fisheye image of a subject in a distorted state. The imaging element is, for example, a CCD image sensor (Charge Coupled Device image sensor) or a CMOS image sensor (Complementary Metal Oxide Semiconductor image sensor). The imaging element is composed of a plurality of picture elements (pixels).

[0040] like Figure 1 As shown, the camera 21 is mounted on the forklift 10. In the present embodiment, the camera 21 is mounted on both sides of the roof guard 13 in the vehicle width direction. That is, two cameras 21 are mounted on the forklift 10. The camera 21 of the present embodiment takes a picture of the side of the forklift 10. Therefore, the obstacle detection device 20 of the present embodiment detects obstacles existing on the side of the forklift 10. The camera 21 repeatedly takes pictures at a predetermined shooting cycle.

[0041] like Figure 3 As shown, the real world coordinate system centered on the camera 21 is set as the world coordinate system. The world coordinate system is a three-dimensional coordinate system in which the X-axis, the Y-axis, and the Z-axis are orthogonal to each other. In the present embodiment, the X-axis extends along the left-right direction of the forklift 10. The Y-axis extends along the front-back direction of the forklift 10. The Z-axis extends along the up-down direction of the forklift 10. The road surface R on which the forklift 10 travels is a plane orthogonal to the Z-axis.

[0042] In the present embodiment, the camera 21 is set at a height H from the road surface R. In other words, the setting height of the camera 21 from the road surface R is H. Therefore, the road surface R is a reference plane whose distance from the camera 21 is known. The part of the obstacle located on the reference plane is taken as the target part. In the present embodiment, the part of the person P located on the road surface R is the foot Pf. Therefore, the target part of the present embodiment is the foot Pf.

[0043] The camera 21 is configured so that the optical axis is tilted relative to the reference plane. That is, the camera 21 is configured so that the optical axis is tilted relative to the road surface R. In the present embodiment, the camera 21 is rotated around the Y axis at a pitch angle θ in order to capture images of the side and bottom of the forklift 10. Therefore, the optical axis of the camera 21 is tilted at a pitch angle θ relative to the Z axis. The pitch angle θ is set to 30 degrees, for example. In the present embodiment, the camera 21 is not rotated around the X axis and around the Z axis. Therefore, the roll angle of the camera 21 is and the yaw angle ψ is 0 degrees.

[0044] The detection unit 22 detects an obstacle from the captured image If of the camera 21 .

[0045] like Figure 2As shown, the detection unit 22 has a processor 22a and a storage unit 22b. As the processor 22a, for example, a CPU, a GPU or a DSP is used. The storage unit 22b includes a RAM and a ROM. It can be said that the storage unit 22b stores program codes or instructions configured to enable the processor 22a to perform processing. The storage unit 22b, that is, the computer-readable medium includes any available medium that can be accessed by a general or special computer. The detection unit 22 can also be composed of a hardware circuit such as an ASIC or FPGA. The detection unit 22 as a processing circuit can include one or more processors that act according to a computer program, one or more hardware circuits such as an ASIC or FPGA, or a combination thereof.

[0046] The storage unit 22b stores the learned model M. The learned model M of the present embodiment is a model that performs object detection and key point detection simultaneously. Specifically, when an image is input, the learned model M outputs the area of ​​the obstacle in the image as a rectangular frame B, and outputs the coordinates of the object part in the image. In the present embodiment, when an image is input, the learned model M outputs the area of ​​the person P in the image as a rectangular frame B, and outputs the coordinates of the foot Pf in the image.

[0047] The learning completion model M is generated by performing machine learning using a plurality of teacher data. Each teacher data has: a teacher image including an object part of an obstacle; a rectangular frame B showing an area of ​​the obstacle in the teacher image; and the coordinates of the object part in the teacher image. In the present embodiment, each teacher data has: a teacher image including a foot Pf of a person P; a rectangular frame B showing an area of ​​the person P in the teacher image; and the coordinates of the foot Pf in the teacher image. The teacher image uses a captured image If that is a fisheye image. That is, the teacher image uses a captured image If before the distortion correction described later is performed.

[0048] The angle detection unit 23 detects the angle of the camera 21. The angle of the camera 21 refers to the pitch angle θ, the roll angle θ, and the and a yaw angle ψ. The angle detection unit 23 includes an IMU (Inertial Measurement Unit).

[0049] <Obstacle detection method>

[0050] The obstacle detection process performed by the obstacle detection device 20 will be described together with the obstacle detection method. The obstacle detection device 20 of the present embodiment performs the following process every time the camera 21 performs an image.

[0051] like Figure 4 As shown, in step S11, the detection unit 22 obtains the captured image If from the camera 21.

[0052] Figure 5 FIG. 2 shows the captured image If of the camera 21. As described above, the camera 21 of the present embodiment is a fisheye camera, and therefore, the captured image If is a fisheye image. Figure 1 As shown in FIG. 1 , when a person P exists on the side of the forklift 10 , the captured image If includes the person P.

[0053] like Figure 4 As shown in FIG. 1 , in step S12, the detection unit 22 estimates the coordinates of the object part in the captured image coordinate system from the captured image If. In addition, the detection unit 22 of the present embodiment estimates the region of the obstacle in the captured image If with the rectangular frame B while estimating the coordinates of the object part in the captured image coordinate system. That is, the detection unit 22 estimates the region of the person P in the captured image If with the rectangular frame B while estimating the coordinates of the foot Pf in the captured image coordinate system.

[0054] like Figure 3 and Figure 5 As shown, the captured image coordinate system refers to the coordinate system on the captured image If. The origin of the captured image coordinate system is the upper left part of the captured image If. The captured image coordinate system is a two-dimensional coordinate system in which the Uf axis and the Vf axis are orthogonal to each other. The Uf axis extends along the left-right direction of the captured image If. The Vf axis extends along the up-down direction of the captured image If. The Uf axis and the Vf axis are orthogonal to the optical axis.

[0055] In the present embodiment, the detection unit 22 estimates the region of the person P in the captured image If with a rectangular frame B, and estimates the coordinates (Uff, Vff) of the foot Pf in the captured image coordinate system using the learned model M. Specifically, the captured image If before the distortion correction described later is input to the learned model M. Then, the learned model M outputs the region of the person P in the input captured image If with a rectangular frame B. In addition, the learned model M outputs the coordinates (Uff, Vff) of the foot Pf in the input captured image If.

[0056] like Figure 4 As shown, in step S13 , the detection unit 22 performs coordinate conversion of the coordinates of the foot Pf from the captured image coordinate system to the developed image coordinate system.

[0057] like Figure 3 and Figure 6As shown, the expanded image coordinate system refers to the coordinate system on the expanded image Ie parallel to the reference plane. The expanded image Ie is a plane orthogonal to the Z axis. The origin of the expanded image coordinate system is the upper left part of the expanded image Ie. The expanded image coordinate system is a two-dimensional coordinate system in which the Ue axis and the Ve axis are orthogonal to each other. The Ue axis extends along the left-right direction of the expanded image Ie. The Ve axis extends along the up-down direction of the expanded image Ie. The Ue axis and the Ve axis are orthogonal to the Z axis.

[0058] In addition, for the sake of convenience of explanation, the expanded image Ie is shown in the figure, but in actual processing, the detection unit 22 does not perform image conversion from the captured image If to the expanded image Ie, so the expanded image Ie is not generated. As long as the coordinates in the captured image coordinate system can be converted to the coordinates in the expanded image coordinate system parallel to the reference plane as described later (as long as it is set in advance that the coordinate conversion can be performed), the image conversion from the captured image If to the expanded image Ie is arbitrary.

[0059] The detection unit 22 performs coordinate conversion of the coordinates (Uff, Vff) of the foot Pf in the captured image coordinate system to the coordinates (Uef, Vef) of the foot Pf in the expanded image coordinate system. In detail, the detection unit 22 first corrects the distortion of the captured image If. Next, the detection unit 22 performs coordinate conversion of the coordinates (Uff, Vff) of the foot Pf in the captured image coordinate system after the distortion is corrected to the coordinates (Uef, Vef) of the foot Pf in the expanded image coordinate system. The coordinate conversion uses the rotation matrix RM. The rotation matrix RM is generated based on the angle of the camera 21 detected by the angle detection unit 23. That is, the detection unit 22 uses the angle of the camera 21 detected by the angle detection unit 23 to perform coordinate conversion. In addition, in the coordinate conversion, the camera parameter K of formula (1) is used.

[0060] [Mathematical formula 1]

[0061]

[0062] Among them, fx and fy are the distances from the camera 21 to the expanded image plane. fx and fy may be different from the focal length of the camera 21. cx and cy are the center coordinates of the expanded image Ie, respectively. In addition, the camera parameter K may be any parameter. For example, the larger the value of fx and fy, the larger the distance from the camera 21 to the expanded image plane. That is, the expanded image plane can be set at any distance from the camera 21 as long as it is a plane parallel to the road surface R.

[0063] like Figure 4As shown, in step S14, the detection unit 22 calculates the position of the target part on the reference plane. In the present embodiment, the detection unit 22 calculates the position of the foot Pf on the road surface R. The detection unit 22 calculates the position of the foot Pf on the road surface R using the relationship between the coordinates (Uef, Vef) of the foot Pf in the developed image coordinate system and the coordinates (Xf, Yf, Zf) of the foot Pf in the world coordinate system, and the distance between the camera 21 and the road surface R as the reference plane.

[0064] The relationship between the coordinates (Uef, Vef) of the foot Pf in the expanded image coordinate system and the coordinates (Xf, Yf, Zf) of the foot Pf in the world coordinate system is expressed by equation (2). In addition, s is the scale. In addition, by transforming equation (2), equation (3) is obtained. The inverse matrix K of the camera parameter K in equation (3) is -1 It is expressed by formula (4).

[0065] [Mathematical formula 2]

[0066]

[0067] As described above, the distance between the camera 21 and the road surface R is the installation height H of the camera 21 from the road surface R. Therefore, it can be assumed that Zf=-H. In addition, Zf=s. Therefore, s=-H. The position of the foot Pf on the road surface R is calculated from equation (3) and s=-H as in equations (5) and (6).

[0068] [Mathematical formula 3]

[0069]

[0070] In this way, the detection unit 22 detects the position of the foot Pf of the person P in the real world. The detection unit 22 may further calculate the distance from the camera 21 to the foot Pf from the position of the foot Pf in the real world.

[0071] The control device 17 controls the forklift 10 based on the detection result of the detection unit 22. The control device 17 of the present embodiment controls the forklift 10 based on the position of the person P in the real world detected by the detection unit 22. Specifically, the control device 17 controls the forklift 10 differently when the person P is on the travel path of the forklift 10 and when the person P is not on the travel path of the forklift 10. For example, when the person P is not on the travel path of the forklift 10, the control device 17 lowers the warning level of the warning device 16 compared to when the person P is on the travel path of the forklift 10. For example, when the person P is on the travel path of the forklift 10, the control device 17 controls the travel motor 15 to decelerate the forklift 10 or restrict the start.

[0072] [Functions and Effects of the Present Embodiment]

[0073] The functions and effects of this embodiment will be described.

[0074] (1) The camera 21 is configured so that the optical axis is inclined relative to a road surface R whose distance from the camera 21 is known. The foot Pf of the person P is located on the road surface R. The detection unit 22 estimates the coordinates of the foot Pf in the captured image coordinate system, which is a coordinate system on the captured image If. The detection unit 22 performs coordinate conversion on the coordinates of the foot Pf in the captured image coordinate system to the coordinates of the foot Pf in the expanded image coordinate system, which is a coordinate system on the expanded image Ie parallel to the road surface R. The detection unit 22 calculates the position of the foot Pf on the road surface R using the relationship between the coordinates of the foot Pf in the expanded image coordinate system and the coordinates of the foot Pf in the world coordinate system, as well as the distance between the camera 21 and the road surface R. The world coordinate system is a real world coordinate system centered on the camera 21.

[0075] According to this configuration, the detection unit 22 performs coordinate conversion of the coordinates of the foot Pf from the captured image coordinate system to the expanded image coordinate system. In addition, the distance between the camera 21 and the road surface R is known. Therefore, the detection unit 22 can calculate the position of the foot Pf on the road surface R using the relationship between the coordinates of the foot Pf in the expanded image coordinate system and the coordinates of the foot Pf in the world coordinate system. Therefore, the detection unit 22 can detect the position of the person P in the real world.

[0076] (2) The detection unit 22 uses not only the pitch angle θ but also the roll angle The coordinates of the foot Pf are converted from the captured image coordinate system to the developed image coordinate system by using the yaw angle ψ and the pitch angle θ. The position of the person P in the real world is detected by using the yaw angle ψ. Therefore, the degree of freedom of the installation posture of the camera 21 is improved.

[0077] (3) The camera 21 is a fisheye camera. As a result, the viewing angle of the camera 21 becomes wider, and the person P can be detected in a wide range. In addition, the detection unit 22 corrects the distortion of the captured image If when converting the coordinates of the foot Pf from the captured image coordinate system to the developed image coordinate system. As a result, the position of the person P in the real world can be easily detected.

[0078] (4) When the image is converted from the captured image If to the expanded image Ie, the viewing angle is limited. Therefore, depending on the position of the person P in the captured image If, ​​the person P may not be included in the expanded image Ie. Figure 6As shown, in the expanded image Ie, the person P is distorted. In the present embodiment, the teacher image and the image input to the learned model M are the captured image If before the distortion is corrected. Thus, compared with the case where the teacher image and the image input to the learned model M are the expanded image Ie, the learned model M can be efficiently generated, and the estimation accuracy of the coordinates of the foot Pf can be improved.

[0079] (5) The angle of the camera 21 may change. For example, when the forklift 10 turns, the forklift 10 tilts to the left or right, so that the pitch angle θ may change. In addition, when the forklift 10 is carrying cargo or driving on a slope, the forklift 10 tilts to the front and back, so that the roll angle θ may change. Sometimes it changes. The obstacle detection device 20 of this embodiment includes an angle detection unit 23 for detecting the angle of the camera 21. The detection unit 22 performs coordinate conversion on the coordinates of the foot Pf using the angle of the camera 21 detected by the angle detection unit 23. Therefore, even if the angle of the camera 21 changes, the detection accuracy of the position of the person P in the real world can be suppressed from decreasing.

[0080] (6) In the present embodiment, the obstacle is a person P. The target part is the foot Pf of the person P. The reference plane is the road surface R. Since the installation height H of the camera 21 from the road surface R becomes the distance between the camera 21 and the reference plane, it is easy to accurately grasp the distance between the camera 21 and the reference plane. In addition, since the foot Pf of the person P is in contact with the road surface R, it is suitable as the target part. Therefore, the detection accuracy of the position of the person P can be improved.

[0081] (7) The obstacle detection device 20 can detect the position of the person P in the real world. Therefore, compared with the case where the obstacle detection device 20 can only detect the distance from the forklift 10 to the person P, for example, the control device 17 can control the travel action of the forklift 10 or the warning action of the warning device 16 in a more detailed manner.

[0082] (8) The camera 21 is a single-lens reflex camera. Therefore, the cost of detecting the person P can be reduced compared to the case where an expensive camera such as a stereo camera is used.

[0083] [Change example]

[0084] In addition, the above-mentioned embodiment can be implemented by the following modifications. The above-mentioned embodiment and the following modifications can be implemented in combination with each other within the range that there is no technical contradiction.

[0085] The obstacles detected by the obstacle detection device 20 and the obstacle detection method are not limited to the person P, but may also be objects.

[0086] The camera 21 is not limited to a fisheye camera. The camera 21 may be a wide-angle camera or a camera having a narrower viewing angle than a fisheye camera or a wide-angle camera.

[0087] The number of cameras 21 mounted on the forklift 10 is not limited to two, and may be changed as appropriate.

[0088] The mounting position of the camera 21 on the forklift 10 is not limited to the roof guard 13 and can be appropriately changed according to the required obstacle detection range.

[0089] The camera 21 may also capture images of the front or rear of the forklift 10 . In this case, the obstacle detection device 20 detects obstacles that exist in front of or behind the forklift 10 .

[0090] In the above embodiment, for convenience of description, the camera 21 is arranged in a state rotated only around the Y axis, but it may be arranged in a state rotated around the X axis or in a state rotated around the Z axis.

[0091] ○ In the above embodiment, the detection unit 22 estimates the coordinates of the foot Pf in the captured image coordinate system and estimates the region of the person P in the captured image If with the rectangular frame B, but the present invention is not limited thereto. The detection unit 22 may estimate only the coordinates of the foot Pf in the captured image coordinate system instead of estimating the region of the person P in the captured image If with the rectangular frame B. In this case, the learned model M may be learned to perform key point detection or may not be learned to perform object detection.

[0092] ○ The reference plane is not limited to the road surface R. For example, if the height of the person P is subtracted from the installation height H of the camera 21 from the road surface R, the distance from the camera 21 to the head of the person P standing on the road surface R becomes known. Therefore, the reference plane may also be a plane located above the road surface R by the height of the person P. In this case, the target part is the head or the helmet.

[0093] In the above embodiment, the detection unit 22 estimates the coordinates of the foot Pf in the captured image coordinate system by key point detection. However, the coordinates of the foot Pf in the captured image coordinate system may be estimated by using object detection.

[0094] like Figure 7 As shown, the detection unit 22 may estimate the coordinates of the intersection of the line L passing through an arbitrary point Ifp of the captured image If and the center Bc of the rectangular frame B and the rectangular frame B as the coordinates of the foot Pf in the captured image coordinate system. Figure 7 In the example shown, the arbitrary point Ifp in the captured image If is a point located at the center of the lower end of the captured image If. The arbitrary point Ifp in the captured image If can be changed as appropriate according to the angle of the camera 21.

[0095] In this case, the learned model M only needs to be the area of ​​the person P in the captured image If that has been learned to output the coordinates of the foot Pf in the captured image coordinate system as the rectangular frame B, and may not be learned to output the coordinates of the foot Pf in the captured image coordinate system. Therefore, it is effective in the case where there is already a learned model M that has been learned to output the area of ​​the person P in the captured image If as the rectangular frame B, or in the case where there is no teacher data for learning the learned model M to output the coordinates of the foot Pf in the captured image coordinate system. In addition, the coordinates of the foot Pf in the captured image If may also be output based on the pixels of the imaging element of the camera 21.

[0096] When the angle of the camera 21 is expected to hardly change, the obstacle detection device 20 may not include the angle detection unit 23. In this case, the detection unit 22 uses the pre-stored angle of the camera 21 to convert the coordinates of the foot Pf from the captured image coordinate system to the developed image coordinate system.

[0097] The industrial vehicle is not limited to the forklift 10. The industrial vehicle may be another industrial vehicle such as a tractor.

[0098] [Note]

[0099] The following describes technical ideas that can be grasped from the above-mentioned embodiments and modified examples.

[0100] <Note 1>

[0101] An obstacle detection device is an obstacle detection device that includes a camera mounted on an industrial vehicle and a detection unit that detects obstacles from an image captured by the camera, characterized in that the camera is configured so that the optical axis is inclined relative to a reference plane whose distance from the camera is known, and when a portion of the obstacle located on the reference plane is taken as an object portion, the detection unit estimates the coordinates of the object portion in a captured image coordinate system, which is a coordinate system on the captured image, and performs coordinate conversion on the coordinates of the object portion from the captured image coordinate system to a developed image coordinate system, which is a coordinate system on the developed image parallel to the reference plane, and uses a relationship between the coordinates of the object portion in the developed image coordinate system and the coordinates of the object portion in a world coordinate system, as well as the distance between the camera and the reference plane, to calculate the position of the object portion on the reference plane, and the world coordinate system is a real world coordinate system centered on the camera.

[0102] <Note 2>

[0103] According to the obstacle detection device described in Appendix 1, wherein the camera is a fisheye camera or a wide-angle camera, the detection unit corrects the distortion of the captured image when converting the coordinates of the object part from the captured image coordinate system to the expanded image coordinate system.

[0104] <Note 3>

[0105] An obstacle detection device according to Note 2, wherein the detection unit has a learning completion model, the learning completion model is generated by performing machine learning using teacher data, and when an image is input, the coordinates of the object part in the image are output, the teacher data has a teacher image containing the object part and the coordinates of the object part in the teacher image, and the teacher image and the image input to the learning completion model are the captured images before distortion is corrected.

[0106] <Note 4>

[0107] An obstacle detection device according to any one of Notes 1 to 3, wherein the device comprises an angle detection unit for detecting the angle of the camera, and the detection unit uses the angle of the camera detected by the angle detection unit to convert the coordinates of the object part from the captured image coordinate system to the expanded image coordinate system.

[0108] <Note 5>

[0109] The obstacle detection device according to any one of appendices 1 to 4, wherein the obstacle is a person, the target part is a foot of the person, and the reference plane is a road surface.

Claims

1. An obstacle detection device comprising a camera mounted on an industrial vehicle and a detection unit for detecting obstacles from an image captured by the camera, wherein: The camera is configured such that the optical axis is tilted relative to a reference plane at a known distance from the camera, When the portion of the obstacle located on the reference plane is taken as the target portion, The detection unit estimating coordinates of the object part in a captured image coordinate system, the captured image coordinate system being a coordinate system on the captured image, converting the coordinates of the target part from the captured image coordinate system to a developed image coordinate system, wherein the developed image coordinate system is a coordinate system on the developed image parallel to the reference plane; The position of the object part on the reference plane is calculated using the relationship between the coordinates of the object part in the expanded image coordinate system and the coordinates of the object part in the world coordinate system, as well as the distance between the camera and the reference plane. The world coordinate system is a real world coordinate system centered on the camera.

2. The obstacle detection device according to claim 1, wherein: The camera is a fisheye camera or a wide-angle camera, The detection unit corrects distortion of the captured image when converting the coordinates of the target site from the captured image coordinate system to the developed image coordinate system.

3. The obstacle detection device according to claim 2, wherein: The detection unit has a learning completion model, which is generated by machine learning using teacher data, and when an image is input, the coordinates of the object part in the image are output, and the teacher data has a teacher image containing the object part and the coordinates of the object part in the teacher image, The teacher image and the image input to the learning completed model are the captured images before distortion is corrected.

4. The obstacle detection device according to any one of claims 1 to 3, wherein: An angle detection unit is provided to detect the angle of the camera. The detection unit converts the coordinates of the target part from the captured image coordinate system to the developed image coordinate system using the angle of the camera detected by the angle detection unit.

5. The obstacle detection device according to any one of claims 1 to 4, wherein: The obstacle is a person, the target part is the foot of the person, and the reference plane is a road surface.

6. An obstacle detection method, wherein a detection unit detects an obstacle from an image captured by a camera mounted on an industrial vehicle, wherein: The camera is configured such that the optical axis is tilted relative to a reference plane at a known distance from the camera, When the portion of the obstacle located on the reference plane is taken as the target portion, The obstacle detection method has the following features: a step of the detection unit estimating the coordinates of the target part in a captured image coordinate system, wherein the captured image coordinate system is a coordinate system on the captured image; The detection unit converts the coordinates of the target part from the captured image coordinate system to a developed image coordinate system, wherein the developed image coordinate system is a coordinate system on the developed image parallel to the reference plane; and The detection unit calculates the position of the object part on the reference plane using a relationship between the coordinates of the object part in the expanded image coordinate system and the coordinates of the object part in the world coordinate system, and a distance between the camera and the reference plane, wherein the world coordinate system is a real world coordinate system centered on the camera.

Images