Obstacle detection device and obstacle detection method

Abstract

An obstacle detection device (30) is mounted on a forklift (10). The obstacle detection device (30) includes a stereo camera (31) for detecting an obstacle and a position detection device (41) for detecting a position of the obstacle based on a detection result of the stereo camera (31). A non-detection area (NA1, NA2) is set in advance in a detectable area (CA). The non-detection area (NA1, NA2) is set at a position where a counterweight (15) of the forklift (10) exists. The position detection device (41) determines that there is no obstacle in the non-detection area (NA1, NA2). The position detection device (41) detects a position of an obstacle existing in a detection area (DA) that is an area different from the non-detection area (NA1, NA2) in the detectable area (CA).

Description

Technical Field

[0001] This disclosure relates to obstacle detection devices and methods. Background Technology

[0002] An obstacle detection device is installed on a moving body such as a vehicle to detect obstacles. Patent Document 1 discloses an obstacle detection device comprising a sensor for detecting obstacles and a position detection unit for detecting the position of obstacles based on the sensor's detection results. The position detection unit detects the position of obstacles existing within the detectable area of ​​the sensor. A stereo camera is used as the sensor. The position detection unit derives a parallax image from the image captured by the stereo camera and detects the position of the obstacle based on the parallax image.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2016-206801 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] Depending on the sensor's location, a portion of a moving object may sometimes enter the sensor's detectable area. Therefore, the obstacle detection device may detect a portion of the moving object as an obstacle.

[0008] The purpose of this disclosure is to provide an obstacle detection device and method capable of preventing a portion of a moving body from being detected as an obstacle.

[0009] Solution for solving the problem

[0010] An obstacle detection device for solving the above problems is mounted on a mobile body and includes: a sensor for detecting obstacles; and a position detection unit for detecting the position of the obstacle based on the detection result of the sensor. The position detection unit includes: a non-detection unit that, when a region that is a pre-defined detectable area where the obstacle can be detected by the sensor and where a part of the mobile body exists is designated as a non-detection area, determines that the obstacle does not exist in the non-detection area regardless of the detection result of the sensor; and a detection unit that detects the position of the obstacle existing in a detection area that is a region different from the non-detection area within the detectable area.

[0011] A non-detection area is pre-defined within the detectable area. Even if an obstacle exists in the non-detection area, the non-detection unit determines that no obstacle exists in the non-detection area. Since a part of the moving body exists in the non-detection area, by determining that no obstacle exists in the non-detection area, it is possible to prevent a part of the moving body from being detected as an obstacle by the obstacle detection device.

[0012] Alternatively, the obstacle detection device described above can be a forklift, and the non-detection area can be set at the location where the counterweight of the forklift is located.

[0013] Alternatively, the obstacle detection device described above may include a position detection unit that derives coordinates of the obstacle in a coordinate system in actual space, with an axis in one horizontal direction as the X-axis, an axis in the horizontal direction orthogonal to the X-axis as the Y-axis, and an axis in the direction orthogonal to both the X-axis and the Y-axis as the Z-axis.

[0014] Alternatively, the non-detection area can be defined by the three-dimensional coordinates of the region representing a part of the moving body in the coordinate system of the actual space.

[0015] Alternatively, the obstacle detection method for solving the above problem is an obstacle detection device mounted on a moving body and equipped with a sensor and a position detection unit that detects the position of an obstacle. This method includes: a step where the position detection unit obtains the detection result of the sensor; a step where, when a pre-defined detectable area where the obstacle can be detected by the sensor and a part of the moving body exists is designated as a non-detectable area, the position detection unit determines that the obstacle does not exist in the non-detectable area, regardless of the sensor's detection result; and a step where the position detection unit detects the position of the obstacle existing in a detection area within the detectable area, which is a different area from the non-detectable area.

[0016] Since a portion of the moving object exists in the non-detection area, by determining that there is no obstacle in the non-detection area, it is possible to suppress the detection of a portion of the moving object as an obstacle.

[0017] Invention Effects

[0018] According to the present invention, it is possible to suppress the detection of a portion of a moving body as an obstacle. Attached Figure Description

[0019] Figure 1 This is a side view of the forklift in the first embodiment.

[0020] Figure 2This is a top view of the forklift in the first embodiment.

[0021] Figure 3 This is a schematic diagram of the forklift and obstacle detection device in the first embodiment.

[0022] Figure 4 This is a diagram showing an example of the first image taken by a stereo camera.

[0023] Figure 5 This is a flowchart illustrating the obstacle detection process performed by the position detection device.

[0024] Figure 6 It is a diagram used to illustrate the detectable area, the non-detectable area, and the detectable area.

[0025] Figure 7 This is a schematic diagram showing the position of obstacles in the XY plane of the world coordinate system.

[0026] Figure 8 This is a side view of the forklift in the second embodiment.

[0027] Figure 9 This is a diagram showing an example of the first image taken by a stereo camera. Detailed Implementation

[0028] (First Embodiment)

[0029] The first embodiment of the obstacle detection device and the obstacle detection method will be described below.

[0030] like Figure 1 and Figure 2 As shown, the forklift 10, as a mobile vehicle, includes: a body 11; drive wheels 12 disposed at the lower front of the body 11; steering wheels 13 disposed at the lower rear of the body 11; and a cargo loading / unloading device 17. The body 11 includes: a top guard 14 disposed above the driver's seat; and a counterweight 15 used to balance the cargo loaded on the cargo loading / unloading device 17. The counterweight 15 is disposed at the rear of the body 11. The forklift 10 can be operated by the rider, can be operated automatically, and can switch between manual and automatic operation. In the following description, left and right refer to left and right relative to the forward direction of the forklift 10.

[0031] like Figure 3As shown, the forklift 10 includes a main control unit 20, a travel motor M1, a travel control unit 23 for controlling the travel motor M1, and a speed sensor 24. The main control unit 20 performs control related to travel and loading / unloading operations. The main control unit 20 includes a processor 21 and a storage unit 22. The processor 21 can be, for example, a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). The storage unit 22 includes random access memory (RAM) and read-only memory (ROM). The storage unit 22 stores programs for operating the forklift 10. In other words, the storage unit 22 stores program code or instructions configured to cause the processor 21 to perform processing. The storage unit 22, i.e., the computer-readable medium, includes all usable media accessible by a general-purpose or special-purpose computer. The main control device 20 can be constructed from hardware circuits such as application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs). The main control device 20, as a processing circuit, can include one or more processors, ASICs, or FPGAs, or combinations thereof, that operate according to a computer program.

[0032] The main control unit 20 provides the travel control unit 23 with a command regarding the rotational speed of the travel motor M1, so that the forklift 10's speed becomes the target speed. In this embodiment, the travel control unit 23 is a motor driver. The speed sensor 24 outputs the rotational speed of the travel motor M1 to the travel control unit 23. The travel control unit 23 controls the travel motor M1 based on the command from the main control unit 20, ensuring that the rotational speed of the travel motor M1 matches the command.

[0033] The forklift 10 is equipped with an obstacle detection device 30. The obstacle detection device 30 includes a stereo camera 31, which acts as a sensor, and a position detection device 41, which detects the position of obstacles based on images captured by the stereo camera 31. The stereo camera 31 is configured to view the road surface through which the forklift 10 travels from above. In this embodiment, the stereo camera 31 captures images of the area behind the forklift 10. Therefore, obstacles detected by the position detection device 41 become obstacles behind the forklift 10.

[0034] like Figure 1As shown, the stereo camera 31 is, for example, mounted on the roof 14. Figure 2 As shown, the stereo camera 31 is positioned offset from the center position CP in the width direction of the forklift 10. In this embodiment, the stereo camera 31 is positioned offset to the left from the center position CP in the width direction of the forklift 10.

[0035] The stereo camera 31 captures images within a range determined by the horizontal and vertical viewing angles. The counterweight 15 is included within the vertical viewing angle range. Therefore, the counterweight 15, which is part of the forklift 10, is always included in the image captured by the stereo camera 31.

[0036] like Figure 3 As shown, the stereo camera 31 includes a first camera 32 and a second camera 33. Examples of cameras using CCD image sensors or CMOS image sensors include the first camera 32 and the second camera 33. The first camera 32 and the second camera 33 are configured with their optical axes parallel to each other. In this embodiment, the first camera 32 and the second camera 33 are arranged horizontally relative to each other. When the image captured by the first camera 32 is designated as the first image and the image captured by the second camera 33 is designated as the second image, the same obstacle is presented in a horizontally offset manner in both the first and second images. Specifically, when the same obstacle is captured, the obstacle presented in the first image and the obstacle presented in the second image will be misaligned in the horizontal pixel dimension [px] corresponding to the distance between the first camera 32 and the second camera 33. The first image and the second image have the same number of pixels, for example, using a 640×480 [px] = VGA image. The first image and the second image are, for example, images represented using RGB signals.

[0037] The position detection device 41 includes a processor 42 and a storage unit 43. The processor 42 may be, for example, a CPU, GPU, or DSP. The storage unit 43 includes RAM and ROM. The storage unit 43 stores various programs for detecting obstacles based on images captured by the stereo camera 31. In essence, the storage unit 43 stores program code or instructions configured to cause the processor 42 to perform processing. The storage unit 43, i.e., the computer-readable medium, includes all usable media accessible by a general-purpose or special-purpose computer. The position detection device 41 may be constructed from hardware circuits such as ASICs or FPGAs. The position detection device 41, as a processing circuit, may include one or more hardware circuits such as processors, ASICs, or FPGAs, or combinations thereof, that operate according to a computer program.

[0038] The obstacle detection process performed by the position detection device 41 and the obstacle detection method will be described below together. The obstacle detection process is performed by executing the program stored in the storage unit 43 by the processor 42. The obstacle detection process is repeated at a predetermined control cycle.

[0039] In the following description, as an example, an image taken by stereo camera 31 is shown. Figure 4 The obstacle detection and handling procedures in the illustrated environment are explained. Figure 4 The first image I1 is obtained by taking a picture of the rear of the forklift 10. As can be determined from the first image I1, there is a person or an obstacle other than a person behind the forklift 10. A portion of the counterweight 15 is included in the first image I1. Furthermore, for ease of explanation, the coordinates of the obstacle in the first image I1 are shown by boxes A1, A2, A3, and A4, but boxes A1, A2, A3, and A4 do not exist in the actual first image I1.

[0040] like Figure 5 As shown, in step S1, the position detection device 41 obtains a first image I1 and a second image of the same frame from the image captured by the stereo camera 31. The first image I1 and the second image are the detection results of the stereo camera 31.

[0041] Next, in step S2, the position detection device 41 obtains a disparity image by performing stereo processing. The disparity image is an image that maps disparity [px] to pixels. Disparity can be obtained by comparing the first image I1 and the second image and calculating the difference in the number of pixels between the first image I1 and the second image for the same feature point presented in each image. Furthermore, feature points refer to identifiable boundaries such as the edges of obstacles. Feature points can be detected from information such as brightness.

[0042] The position detection device 41 uses RAM that temporarily stores each image to perform the conversion from RGB to YCrCb. Furthermore, the position detection device 41 can also perform distortion correction, edge enhancement processing, etc. The position detection device 41 performs stereo processing to calculate disparity by comparing the similarity of each pixel of the first image I1 with each pixel of the second image. Furthermore, as stereo processing, a method of calculating disparity per pixel can be used, or a block matching method can be used to divide each image into blocks containing multiple pixels and calculate the disparity of each block. The position detection device 41 obtains a disparity image using the first image I1 as the reference image and the second image as the comparison image. The position detection device 41 extracts the pixel of the second image that is most similar to each pixel of the first image I1, and calculates the disparity as the difference in the horizontal pixel count between a pixel of the first image I1 and the pixel most similar to that pixel. Thus, a disparity image that corresponds to each pixel of the first image I1, which serves as the reference image, can be obtained. The parallax image does not necessarily need to be displayed; instead, it represents data that maps the parallax to the individual pixels in the parallax image. Furthermore, the position detection device 41 can also perform processing to remove the road surface parallax from the parallax image.

[0043] Next, in step S3, the position detection device 41 derives the coordinates of the feature points in the world coordinate system. First, the position detection device 41 derives the coordinates of the feature points in the camera coordinate system. The camera coordinate system is a coordinate system with the stereo camera 31 as the origin. The camera coordinate system is a three-axis orthogonal coordinate system with the optical axis as the Z-axis and two axes orthogonal to the optical axis as the X-axis and Y-axis, respectively. The coordinates of the feature points in the camera coordinate system can be represented by the Z-coordinate Zc, X-coordinate Xc, and Y-coordinate Yc in the camera coordinate system. The Z-coordinate Zc, X-coordinate Xc, and Y-coordinate Yc can be derived using the following equations (1) to (3), respectively.

[0044] [Mathematical Expression 1]

[0045]

[0046] [Mathematical Expression 2]

[0047]

[0048] [Mathematical Expression 3]

[0049]

[0050] In equations (1) to (3), B is the baseline length [mm], f is the focal length [mm], and d is the parallax [px]. xp is any X coordinate in the parallax image, and x′ is the X coordinate of the center coordinate of the parallax image. yp is any Y coordinate in the parallax image, and y′ is the Y coordinate of the center coordinate of the parallax image.

[0051] By setting xp as the X coordinate of a feature point in the parallax image, yp as the Y coordinate of a feature point in the parallax image, and d as the parallax corresponding to the coordinates of the feature point, the coordinates of the feature point in the camera coordinate system can be derived.

[0052] Here, with the forklift 10 positioned horizontally, a three-axis orthogonal coordinate system is defined as the world coordinate system, which serves as the coordinate system in actual space. The x-axis is the axis extending horizontally along the width of the forklift 10; the y-axis is the axis extending horizontally in a direction orthogonal to the x-axis; and the z-axis is the axis orthogonal to both the x-axis and y-axis. The y-axis of the world coordinate system can also be described as the axis extending in the forward / backward direction of the forklift 10, which is its direction of travel. The z-axis of the world coordinate system can also be described as the axis extending vertically. The coordinates of feature points in the world coordinate system can be represented using the x-coordinate (Xw), y-coordinate (Yw), and z-coordinate (Zw) of the world coordinate system.

[0053] The position detection device 41 uses the following equation (4) to perform a world coordinate transformation, converting the camera coordinates to world coordinates. World coordinates refer to coordinates in the world coordinate system.

[0054] [Mathematical Expression 4]

[0055]

[0056] Here, H in equation (4) is the height of the stereo camera 31 in the world coordinate system [mm], and θ is the angle +90° between the optical axes of the first camera 32 and the second camera 33 and the horizontal plane.

[0057] In this embodiment, the origin of the world coordinate system is defined by the X-coordinate Xw and Y-coordinate Yw for the position of the stereo camera 31, and the Z-coordinate Zw for the coordinates of the road surface. The position of the stereo camera 31 is, for example, the midpoint between the lens of the first camera 32 and the lens of the second camera 33.

[0058] The X-coordinate Xw obtained through world coordinate transformation represents the distance from the origin to the feature point relative to the width of the forklift 10. The Y-coordinate Yw represents the distance from the origin to the feature point relative to the travel direction of the forklift 10. The Z-coordinate Zw represents the height from the road surface to the feature point. The feature point is a point representing a portion of an obstacle. Furthermore, in the diagram, arrow X represents the X-axis of the world coordinate system, arrow Y represents the Y-axis, and arrow Z represents the Z-axis.

[0059] like Figure 6As shown, in the world coordinate system, the area where world coordinates can be obtained is the detectable region CA, which is the area where obstacles can be detected. The detectable region CA is determined, for example, by the shooting range of the stereo camera 31. Through the processing in step S3, the position detection device 41 functions as a coordinate derivation unit.

[0060] Here, a non-detection area NA1 is pre-defined within the detectable area CA of the stereo camera 31. The non-detection area NA1 is the area determined to be free of obstacles regardless of whether the stereo camera 31 has captured an image of the obstacle. The area within the detectable area CA that differs from the non-detection area NA1 is designated as the detection area DA. The detection area DA is the area where obstacle detection is performed. Therefore, it can be said that when the stereo camera 31 captures an image of an obstacle and the obstacle exists within the detection area DA, the position detection device 41 detects the obstacle.

[0061] like Figure 5 As shown, in step S4, the position detection device 41 deletes feature points in the non-detection area NA1 as unwanted feature points. The non-detection area NA1 is defined as the location in the detectable area CA where a portion of the forklift 10 is present. In this embodiment, the location where the counterweight 15 is present is the non-detection area NA1. The unwanted feature points can also be described as feature points generated because the counterweight 15 was captured.

[0062] Unnecessary feature points can be exported according to vehicle specifications. The vehicle specifications used to export unnecessary feature points are stored, for example, in the storage unit 43 of the position detection device 41.

[0063] like Figure 1 and Figure 2 As shown, information representing the width W1 of the counterweight 15, the height H1 of the counterweight 15, the distance L1 from the stereo camera 31 to the rear end of the counterweight 15 in the front-rear direction, and the distance W2 between the center position CP of the forklift 10 and the width direction of the stereo camera 31 are stored as vehicle specifications.

[0064] The width W1 of the counterweight 15 refers to the dimension of the counterweight 15 in the vehicle width direction. The width W1 of the counterweight 15 can also be described as the dimension of the counterweight 15 in the X-axis direction of the world coordinate system. In this embodiment, the counterweight 15 captured by the stereo camera 31 has a fixed width. Therefore, the width W1 of the counterweight 15 can be set to a fixed value. When the width W1 of the counterweight 15 is not fixed, the width of the counterweight 15 can be stored based on its position in the front-rear direction. That is, the width of the counterweight 15 can also be stored by corresponding to its Y-coordinate Yw, so that the width of the counterweight 15 can be known even when it is not fixed. Alternatively, the width of the counterweight 15 can be considered fixed even when it is not fixed. In this case, the maximum width of the counterweight 15 can be considered as its width.

[0065] The height H1 of counterweight 15 refers to the dimension from the road surface to the top of counterweight 15. Since the origin of the Z-axis in the world coordinate system is the road surface, the height H1 of counterweight 15 can also be considered as the Z-coordinate Zw of the top of counterweight 15 in the world coordinate system. Furthermore, when the height of counterweight 15 varies depending on its position in the front-rear direction and its position in the vehicle width direction, simply setting the highest position as the top of counterweight 15 is sufficient.

[0066] The distance L1 from the stereo camera 31 to the rear end of the counterweight 15 in the front-rear direction refers to the dimension along the Y-axis in the world coordinate system. Since the origin of the Y-axis in the world coordinate system is the stereo camera 31, the distance L1 from the stereo camera 31 to the rear end of the counterweight 15 in the front-rear direction can also be considered as the Y-coordinate Yw of the rear end of the counterweight 15 in the world coordinate system. Furthermore, when the position of the rear end of the counterweight 15 varies depending on its position in the front-rear direction and its position in the vehicle width direction, simply defining the rearmost point as the rear end of the counterweight 15 is sufficient.

[0067] The distance W2 between the center position CP of the forklift 10 and the stereo camera 31 in the width direction refers to the dimension along the X-axis from the center position CP of the forklift 10 to the stereo camera 31 in the world coordinate system. Since the origin of the X-axis in the world coordinate system is the stereo camera 31, the distance W2 between the center position CP of the forklift 10 and the stereo camera 31 in the width direction can also be said to be the X-coordinate Xw of the center position CP of the forklift 10 in the world coordinate system.

[0068] The position detection device 41 deletes feature points that are consistent with all of the following conditions 1, 2 and 3 according to the vehicle specifications described above, as unwanted feature points.

[0069] Condition 1…-(W1 / 2+W2)≤Xw≤(W1 / 2-W2)

[0070] Condition 2…0≤Yw≤L1

[0071] Condition 3…0≤Zw≤H1

[0072] The first condition can be described as extracting feature points within a range equal to half the width W1 of the counterweight 15 on each side of the X-axis in the world coordinate system, starting from the center position CP of the forklift 10 in the width direction. In this embodiment, since the center position CP of the forklift 10 is offset by a distance W2 from the origin of the X-axis in the world coordinate system, it can be said that by shifting the X-coordinate Xw to the right by a distance W2, the range of the X-coordinate Xw is shifted to the range based on the center position CP of the forklift 10.

[0073] The second condition can be described as extracting feature points that exist in the range from the rear end of the stereo camera 31 to the counterweight 15.

[0074] The third condition can be described as extracting feature points from the road surface to the upper end of counterweight 15.

[0075] Each condition indicates the range of three-dimensional coordinates in the world coordinate system. Specifically, the cuboid region with X-coordinate Xw ranging from -(W1 / 2+W2) to (W1 / 2-W2), Y-coordinate Yw ranging from 0 to L1, and Z-coordinate Zw ranging from 0 to H1 is the non-detection region NA1 where feature points are deleted. By deleting feature points that are identical to all three conditions (condition 1, condition 2, and condition 3), the feature points in the non-detection region NA1 are deleted.

[0076] like Figure 6 As shown, the non-detection region NA1 can be considered as the area enclosed by coordinates P1 to P8 in the world coordinate system. When the three-dimensional coordinates in the world coordinate system are represented by coordinates (Xw, Yw, Zw), coordinate P1 can be represented by (-(W1 / 2+W2), 0, H1), coordinate P2 by (W1 / 2-W2, 0, H1), coordinate P3 by (-(W1 / 2+W2), L1, H1), and coordinate P4 by (W1 / 2-W2, L1, H1). Similarly, coordinate P5 can be represented by (-(W1 / 2+W2), 0, 0), coordinate P6 by (W1 / 2-W2, 0, 0), coordinate P7 by (-(W1 / 2+W2), L1, 0), and coordinate P8 by (W1 / 2-W2, L1, 0). The non-detection region NA1 is defined by the three-dimensional coordinates of the region containing the weight 15 in the world coordinate system.

[0077] Furthermore, whether a coordinate is a "+" or a "-" in world coordinates indicates its direction relative to the origin of the world coordinate system, and can be arbitrarily set for each coordinate axis. For the X-coordinate (Xw), a coordinate to the left of the origin is considered a "+" coordinate, and a coordinate to the right is considered a "-" coordinate. For the Y-coordinate (Yw), a coordinate behind the origin is considered a "+" coordinate, and a coordinate in front of the origin is considered a "-" coordinate. For the Z-coordinate (Zw), a coordinate above the origin is considered a "+" coordinate, and a coordinate below the origin is considered a "-" coordinate.

[0078] like Figure 5 As shown, in step S5, the position detection device 41 extracts obstacles existing in the world coordinate system. The position detection device 41 treats a set of feature points representing a portion of an obstacle, assuming they represent the same obstacle, as a point cluster, and extracts this point cluster as an obstacle. For example, the position detection device 41 clusters feature points within a specified range into a point cluster based on the world coordinates of the feature points derived in step S3. The position detection device 41 then treats the clustered point cluster as an obstacle. In step S4, feature points in the non-detection region NA1 have been deleted; therefore, the obstacle extracted in step S5 can be considered an obstacle existing in the detection region DA, which is a region different from the non-detection region NA1. The non-detection region NA1 is determined to be free of obstacles regardless of the detection results of the stereo camera 31, in other words, regardless of the presence or absence of obstacles. Furthermore, the clustering of feature points performed in step S5 can be performed using various methods. That is, clustering can be performed by any method as long as multiple feature points can be treated as a point cluster and treated as an obstacle.

[0079] Next, in step S6, the position detection device 41 outputs the position of the obstacle extracted in step S5. In this embodiment, the position of the obstacle refers to the coordinates of the obstacle in the XY plane of the world coordinate system. The position detection device 41 can identify the world coordinates of the obstacle based on the world coordinates of the feature points constituting the clustered point group. For example, the X coordinates (Xw), Y coordinates (Yw), and Z coordinates (Zw) of multiple feature points located at the edge of the clustered point group can be set as the X coordinates (Xw), Y coordinates (Yw), and Z coordinates (Zw) of the obstacle, or the X coordinates (Xw), Y coordinates (Yw), and Z coordinates (Zw) of the feature point that will become the center of the point group can be set as the X coordinates (Xw), Y coordinates (Yw), and Z coordinates (Zw) of the obstacle. That is, the coordinates of the obstacle in the world coordinate system can represent the entire obstacle or a single point of the obstacle.

[0080] like Figure 7As shown, the position detection device 41 derives the X-coordinates Xw and Y-coordinates Yw of the obstacle on the XY plane of the world coordinate system by projecting the X-coordinates Xw, Y-coordinates Yw, and Z-coordinates Zw of the obstacle onto the XY plane of the world coordinate system. That is, the position detection device 41 derives the X-coordinates Xw and Y-coordinates Yw of the obstacle in the horizontal direction by subtracting the Z-coordinate Zw from the X-coordinates Xw, Y-coordinates Yw, and Z-coordinate Zw of the obstacle.

[0081] Figure 7 The obstacles O1 to O4 shown are obstacles detected from the first image I1 and the second image through the processing of steps S1 to S6. Obstacle O1 is an obstacle present in frame A1. Obstacle O2 is an obstacle present in frame A2. Obstacle O3 is an obstacle present in frame A3. Obstacle O4 is an obstacle present in frame A4.

[0082] If the feature points in the non-detection area NA1 are not deleted, the position detection device 41 will extract the obstacle O5 corresponding to the counterweight 15. In this embodiment, by deleting the feature points in the non-detection area NA1 and determining that there is no obstacle in the non-detection area NA1, the extraction of obstacle O5 is suppressed. Through the processing in step S4, the position detection device 41 functions as a non-detection unit. Through the processing in steps S5 and S6, the position detection device 41 functions as a detection unit. The position detection device 41 functions as a position detection unit.

[0083] Furthermore, the "deletion of feature points" in step S4 means that feature points in the non-detection region NA1 are not used in the obstacle extraction performed in step S5. That is, "deletion of feature points" includes not only the case where the world coordinates of feature points in the non-detection region NA1 are deleted from the RAM of the position detection device 41, but also the case where the world coordinates of feature points in the non-detection region NA1 are not deleted from the RAM of the position detection device 41, but rather the case where the feature points in the non-detection region NA1 are not used for obstacle extraction.

[0084] By performing obstacle detection processing by the position detection device 41, the horizontal positional relationship between the forklift 10 and the obstacle can be determined. The main control device 20 determines the horizontal positional relationship between the forklift 10 and the obstacle by obtaining the detection results from the position detection device 41. The main control device 20 performs control corresponding to the positional relationship between the forklift 10 and the obstacle. For example, the main control device 20 limits the vehicle speed or issues an alarm when the distance between the forklift 10 and the obstacle is below a threshold.

[0085] The function of the first embodiment will be explained.

[0086] A non-detection area NA1 is pre-defined within the detectable area CA. The position detection device 41 deletes feature points existing in the non-detection area NA1. Therefore, even if an obstacle exists in the non-detection area NA1, the position detection device 41 determines that no obstacle exists there. The non-detection area NA1 is the area where the counterweight 15 exists. Since the positional relationship between the stereo camera 31 and the counterweight 15 is fixed, the counterweight 15 will always enter the imaging range of the stereo camera 31.

[0087] When the main control unit 20 sets speed limits or alarms based on the distance to an obstacle, it is possible that speed limits or alarms will be set because the counterweight 15 is detected as an obstacle. Since the counterweight 15 always enters the detectable zone CA, speed limits or alarms may always be set. In this case, the operating efficiency of the forklift 10 may deteriorate. In addition, because alarms are always set, it may be impossible to determine whether an obstacle is approaching the forklift 10.

[0088] In this respect, in the first embodiment, since the counterweight 15 is not detected as an obstacle, the speed limit or alarm caused by the counterweight 15 being photographed by the stereo camera 31 is suppressed.

[0089] The effects of the first embodiment will be explained.

[0090] (1-1) A non-detection area NA1 is pre-defined in the detectable area CA of the stereo camera 31. The position detection device 41 determines that there is no obstacle in the non-detection area NA1 by deleting feature points of the non-detection area NA1. It can suppress the detection of the counterweight 15 present in the non-detection area NA1 as an obstacle.

[0091] (1-2) In the forklift 10, a counterweight 15 is positioned at the rear of the vehicle body 11 to achieve balance with the goods loaded on the loading / unloading device 17. Therefore, the counterweight 15 easily enters the detectable area CA of the stereo camera 31 that captures images from the rear. Furthermore, due to configuration limitations, it is sometimes difficult to configure the stereo camera 31 so that the counterweight 15 does not enter the detectable area CA. By setting the area where the counterweight 15 exists as the non-detectable area NA1, even when the counterweight 15 enters the detectable area CA of the stereo camera 31, it is possible to both suppress the detection of the counterweight 15 as an obstacle and detect obstacles in the detection area DA.

[0092] (1-3) The non-detection region NA1 is defined by the three-dimensional coordinates of the world coordinate system. Alternatively, the non-detection region NA1 can be defined using the X-coordinates Xw and Y-coordinates Yw of the world coordinate system, deleting feature points independently of the Z-coordinate Zw. In this case, even if an obstacle is placed on the counterweight 15, the obstacle will be included in the non-detection region NA1. Therefore, even if an obstacle exists on the counterweight 15, it is considered non-existent. By defining the non-detection region NA1 using three-dimensional coordinates, obstacles existing on the counterweight 15 can be detected.

[0093] (1-4) The non-detection area NA1 is a pre-defined area. When a part of the moving body enters the detectable area CA due to the movement of its movable component, the position detection device 41 needs to set the area containing the movable component as a non-detection area to prevent this part of the moving body from being detected as an obstacle. Since the movable component can move, it is impossible to pre-define the non-detection area; the position detection device 41 needs to detect the position of the movable component and set that position as a non-detection area. In this embodiment, a non-detection area NA1 is provided corresponding to the counterweight 15, which has a fixed positional relationship with the stereo camera 31. Since the position of the counterweight 15 in the detectable area CA is fixed, the non-detection area NA1 can be pre-defined. Compared to detecting the position of the movable component and setting the non-detection area accordingly, the load on the position detection device 41 can be reduced.

[0094] (1-5) An obstacle detection method is performed by the obstacle detection device 30, thereby assuming that there is no obstacle in the non-detection area NA1. It is possible to suppress the detection of the counterweight 15 present in the non-detection area NA1 as an obstacle.

[0095] (Second Implementation)

[0096] The obstacle detection device and obstacle detection method of the second embodiment will be described below. In the following description, the parts that are the same as those in the first embodiment will be omitted.

[0097] like Figure 8 As shown, the forklift 10 includes a mirror 18 and a support 19 for supporting the mirror 18. The support 19 extends rearward toward the vehicle body 11. The mirror 18 and the support 19 are located within the vertical field of view of the stereo camera 31. The mirror 18 and the support 19 are part of the forklift 10.

[0098] like Figure 9 As shown, mirror 18 and support 19 are captured in the first image I1 by stereo camera 31. In the second embodiment, obstacle detection processing is performed in such a way that mirror 18 and support 19, except for counterweight 15, are not detected as obstacles.

[0099] like Figure 8 As shown, the height H2 of the mirror 18 is stored in the storage unit 43 of the position detection device 41 as a vehicle specification. The height H2 of the mirror 18 refers to the dimension from the road surface to the bottom of the mirror 18. Since the origin of the Z-axis in the world coordinate system is the road surface, the height H2 of the mirror 18 can also be described as the Z-coordinate Zw of the bottom of the mirror 18 in the world coordinate system. Furthermore, the support unit 19 is generally located above the bottom of the mirror 18.

[0100] By modifying the third condition of the first embodiment as shown below, the position detection device 41 can delete feature points generated by the mirror 18 and the support 19 as unwanted feature points, in addition to the counterweight 15. The position detection device 41 deletes feature points that are consistent with all three conditions (first, second, and third) as unwanted feature points.

[0101] The third condition… 0 ≤ Zw ≤ H1 or Zw ≥ H2

[0102] In addition to the third condition of the first embodiment, Zw≥H2 is added as an OR condition. Therefore, feature points that correspond to 0≤Zw≤H1 in the first, second, and third conditions, as well as feature points that correspond to Zw≥H2 in the first, second, and third conditions, are deleted as unwanted feature points. It can be said that the non-detection region NA2 defined by Zw≥H2 in the first, second, and third conditions is a region where the X coordinate Xw ranges from -(W1 / 2+W2) to (W1 / 2-W2), the Y coordinate Yw ranges from 0 to L1, and the Z coordinate Zw is H2 or higher.

[0103] By changing the third condition to the conditions described above, the mirror 18 and the support 19 are no longer identified as obstacles. Furthermore, since the first and second conditions are set to be the same as in the first embodiment, feature points within the same range as the counterweight 15 are deleted for the X-coordinate Xw and Y-coordinate Yw. Depending on the size of the mirror 18 and the support 19, the ranges of the X-coordinate Xw and Y-coordinate Yw of the non-detection area NA2 may sometimes be too large or too small relative to the mirror 18 and the support 19. In this case, each condition can be set separately for the non-detection area NA1 for the counterweight 15 and the non-detection area NA2 for the mirror 18 and the support 19.

[0104] The effects of the second embodiment will be explained.

[0105] (2-1) In addition to the counterweight 15, the mirror 18 and the support 19 are also prevented from being detected as obstacles. Even when multiple components enter the detectable area CA, by setting non-detection areas NA1 and NA2 for multiple components, it is possible not to regard each component as an obstacle, and to detect obstacles in the detection area DA.

[0106] Each implementation can be modified as shown below. The various implementations and the following variations can be combined and implemented within a technically compatible framework.

[0107] Alternatively, in each embodiment, the non-detection region NA1 can be defined using two-dimensional coordinates representing the coordinates on the XY plane of the world coordinate system. That is, the third condition in each embodiment can be deleted, and feature points consistent with the first and second conditions can be deleted. In this case, feature points existing in the non-detection region defined by the X coordinate Xw and Y coordinate Yw will be deleted as unwanted feature points, regardless of the Z coordinate Zw.

[0108] Alternatively, in various embodiments, the position of the obstacle derived in step S6 is a three-dimensional coordinate in the world coordinate system. It can be said that the position detection device 41 may not project the obstacle onto the XY plane of the world coordinate system.

[0109] Alternatively, in various embodiments, the obstacle detection device 30 may use a sensor other than the stereo camera 31, which is capable of acquiring three-dimensional coordinates in the world coordinate system, as a sensor. Examples of such sensors include Laser Imaging Detection and Ranging (LIDAR), millimeter-wave radar, and Time-of-Flight (TOF) cameras. A LIDAR is a distance meter that identifies the surrounding environment by illuminating a laser while changing the illumination angle and receiving the reflected light partially reflected from the laser beam. Millimeter-wave radar is a radar that identifies the surrounding environment by illuminating the surroundings with radio waves of a specified frequency band. A TOF camera includes a camera and a light source for illumination, and derives the depth direction distance based on the time until the reflected light from the light source is received, per pixel of the image captured by the camera. A combination of the above-mentioned sensors may also be used as the sensor.

[0110] Alternatively, in various embodiments, the obstacle detection device 30 may include a two-dimensional LiDAR as a sensor that illuminates a laser while changing the illumination angle in the horizontal direction. The LiDAR illuminates the laser while changing the illumination angle within a range of illuminating angles. The illuminating angle is, for example, 270 degrees relative to the horizontal direction. The detectable area CA of the two-dimensional LiDAR refers to the range defined by the illuminating angle and the measurable distance. When the portion illuminated by the laser is designated as the illumination point, the two-dimensional LiDAR can measure the distance to the illumination point by relating it to the illumination angle. In other words, the two-dimensional LiDAR can measure the two-dimensional coordinates of the illumination point with the two-dimensional LiDAR as the origin. The two-dimensional coordinates measured by the two-dimensional LiDAR are coordinates in a world coordinate system with one direction in the horizontal direction as the X-axis and the direction in the horizontal direction orthogonal to the X-axis as the Y-axis. In this case, the non-detectable area is defined by the two-dimensional coordinates.

[0111] Alternatively, in each embodiment, the location of the stereo camera 31 can be appropriately changed. For example, the stereo camera 31 can also be located at the center position CP. In this case, since the origin of the X-axis of the world coordinate system coincides with the center position CP, the first condition can be changed as follows.

[0112] Condition 1…-W1 / 2≤Xw≤W1 / 2

[0113] Thus, when the coordinate axes of the world coordinate system are changed in the implementation method by changing the setting position of the stereo camera 31, the conditions are changed accordingly.

[0114] Alternatively, in each embodiment, the non-detection area is set within the image captured by the stereo camera 31. Taking the first image I1 as an example, the coordinates of the presentation weight 15 in the first image I1 can be predetermined based on the setting position and angle of the stereo camera 31. Setting the coordinates of the presentation weight 15 in the first image I1 as the non-detection area prevents parallax from being calculated for the non-detection area. The non-detection area only needs to be set in at least one of the first image I1 and the second image. Since feature points cannot be obtained at the position of the presentation weight 15, the same effect as in each embodiment can be achieved. Similarly, the coordinates of the mirror 18 and the support 19 can also be set as non-detection areas in the image. When a non-detection area is set in the image, the detectable area CA becomes the range within the image captured by the stereo camera 31. More specifically, it is the range within the image captured by the stereo camera 31 where a parallax image can be obtained.

[0115] In each embodiment, the non-detection areas NA1 and NA2 only need to include a portion of the area where the forklift 10 is located, or they can be larger than the area where the forklift 10 is located. That is, the non-detection areas NA1 and NA2 can also include areas with margin.

[0116] Alternatively, in each embodiment, after clustering feature points to extract obstacles in step S5, the position detection device 41 determines whether each obstacle exists in the non-detection area NA1. The position detection device 41 considers obstacles existing in the non-detection area NA1 as non-existent. For obstacles that span both inside and outside the non-detection area NA1, the position detection device 41 may consider them as existing in the non-detection area NA1 or as existing outside the non-detection area NA1. If obstacles exist that span both inside and outside the non-detection area NA1, the position detection device 41 may only consider the portion existing outside the non-detection area NA1 as an obstacle.

[0117] In each embodiment, the entire area outside the non-detectable areas NA1 and NA2 in the detectable area CA can be designated as the detection area DA, or a portion of the area outside the non-detectable areas NA1 and NA2 in the detectable area CA can be designated as the detection area DA.

[0118] Alternatively, in various embodiments, after processing in step S6, the position detection device 41 performs a process to determine whether the detected obstacle is a person or an object other than a person. The determination of whether an obstacle is a person can be performed using various methods. For example, the position detection device 41 determines whether an obstacle is a person by performing human detection processing on an image captured by either of the two cameras 32 and 33 of the stereo camera 31. The position detection device 41 converts the coordinates of the obstacle in the world coordinate system obtained in step S6 into camera coordinates, and then converts these camera coordinates into the coordinates of the images captured by cameras 32 and 33. For example, the position detection device 41 converts the coordinates of the obstacle in the world coordinate system into the coordinates of the first image I1. The position detection device 41 performs human detection processing on the coordinates of the obstacle in the first image I1. For example, it uses feature extraction and a human detector that has undergone prior machine learning to perform human detection processing. As for feature extraction, methods for extracting features from local regions of an image, such as Histogram of Oriented Gradients (HOG) features and Haar-like features, can be listed. As for human determiners, devices that have undergone machine learning through supervised learning models can be used, for example. Supervised learning models can be, for example, support vector machines, neural networks, naive Bayes, deep learning, decision trees, etc. As supervised data for machine learning, inherent image components such as shape features and appearance features extracted from the image can be used. Shape features include, for example, the size or outline of a person. Appearance features include, for example, light source information, texture information, camera information, etc. Light source information includes information related to reflectivity, shadows, etc. Texture information includes color information, etc. Camera information includes information related to image quality, resolution, and viewpoint, etc.

[0119] Since human detection processing is time-consuming, when detecting people from an image, the coordinates of any obstacles are determined, and human detection is performed on those coordinates. Performing human detection by specifying coordinates reduces the processing time compared to performing human detection on the entire image area. Since a portion of the forklift 10, such as the counterweight 15, is not considered an obstacle, human detection is not performed on the coordinates of that portion of the forklift 10 in the image. Therefore, compared to the case where a portion of the forklift 10 is detected as an obstacle and human detection is performed on the coordinates of that obstacle, processing time is reduced.

[0120] In various embodiments, the counterweight 15 located behind the stereo camera 31 is designated as the non-detection area NA1, but the non-detection area NA1 can also be defined based on the range captured by the stereo camera 31. For example, according to... Figure 1 and Figure 2 As can be determined, depending on the placement of the stereo camera 31 or its vertical viewing angle, even a counterweight 15 located further back than the stereo camera 31 may sometimes have a portion that does not enter the shooting range of the stereo camera 31. Therefore, the portion that does not enter the shooting range of the stereo camera 31 may not be included in the non-detection area NA1. For example, if a second condition is set as in the embodiment, the lower limit of the Y-coordinate Yw of the second condition may be set to a value greater than 0.

[0121] Alternatively, in each embodiment, instead of storing vehicle specifications, the coordinates of the designated non-detection area are stored in the storage unit 43 of the position detection device 41. If it is a non-detection area NA1, then it is sufficient to pre-store the coordinates P1 to P8.

[0122] Alternatively, in various embodiments, the obstacle detection device 30 detects obstacles located in front of the forklift 10. In this case, the stereo camera 31 is configured to capture images of the front of the forklift 10. Even when the stereo camera 31 captures images of the front of the forklift 10, depending on the placement of the stereo camera 31, a portion of the forklift 10 may sometimes enter the detectable area CA of the stereo camera 31. A non-detection area is then defined corresponding to the portion of the forklift 10 that enters the detectable area CA. Furthermore, the obstacle detection device 30 can also detect obstacles on both the front and rear sides of the forklift 10. In this case, the stereo camera 31 includes both a stereo camera capturing images of the front of the forklift 10 and a stereo camera capturing images of the rear of the forklift 10.

[0123] ○ In each implementation, the world coordinate system is not limited to an orthogonal coordinate system, but may also be set as a polar coordinate system.

[0124] Alternatively, in each embodiment, the position detection unit may be composed of multiple devices. For example, the position detection unit may individually include: a device that functions as a non-detection unit, a device that functions as a detection unit, and a device that functions as a coordinate derivation unit.

[0125] Alternatively, in various embodiments, the conversion from camera coordinates to world coordinates can be performed using tabular data. This tabular data includes data that maps the combination of Y-coordinates Yc and Z-coordinates Zc to Y-coordinate Yw, and data that maps the combination of Y-coordinates Yc and Z-coordinates Zc to Z-coordinate Zw. By pre-storing this tabular data in the storage unit 43 of the position detection device 41, the Y-coordinates Yw and Z-coordinates Zw in the world coordinate system can be calculated from the Y-coordinates Yc and Z-coordinates Zc in the camera coordinate system. Furthermore, in this embodiment, since the X-coordinate Xc in the camera coordinate system is the same as the X-coordinate Xw in the world coordinate system, tabular data for calculating the X-coordinate Xw is not stored.

[0126] Alternatively, in various embodiments, the first camera 32 and the second camera 33 may be arranged vertically.

[0127] Alternatively, in various embodiments, the obstacle detection device 30 may include an auxiliary storage device configured to store various information, such as information stored in the storage unit 43 of the location detection device 41. As an auxiliary storage device, for example, a non-volatile storage device capable of rewriting data, such as a hard disk drive, a solid-state drive, or an electrically erasable programmable read-only memory (EEPROM), can be used.

[0128] Alternatively, in various embodiments, the stereo camera 31 may have three or more cameras.

[0129] Alternatively, in each embodiment, the stereo camera 31 may be installed at any position, such as the cargo loading and unloading device 17.

[0130] Alternatively, in various embodiments, the forklift 10 is a forklift driven by an engine. In this case, the travel control device becomes a device for controlling the amount of fuel injected into the engine, etc.

[0131] In various embodiments, a part of the forklift 10 may be any part other than the counterweight 15, mirror 18, and support 19, as long as it is part of the forklift 10 and enters the detectable area CA.

[0132] In various embodiments, the obstacle detection device 30 can be mounted on various mobile bodies other than construction machinery, automated guided vehicles, trucks, and forklifts 10, such as industrial vehicles, passenger cars, and aircraft.

[0133] Explanation of reference numerals in the attached figures

[0134] CA…Detectable Area

[0135] DA… detection area

[0136] NA1, NA2... non-detection areas

[0137] 10… Forklifts as moving vehicles

[0138] 15… Counterweight as part of the forklift

[0139] 18…Mirror as part of a forklift

[0140] 19… Support section as part of the forklift

[0141] 30… Obstacle detection device

[0142] 31… Stereo cameras as sensors

[0143] 41…A position detection device that serves as a position detection unit, a non-detection unit, a detection unit, and a coordinate derivation unit.

Claims

1. An obstacle detection device, mounted on a moving body, characterized in that, have: Sensors used to detect obstacles; and The position detection unit, based on the detection results of the sensor, obtains feature points in a coordinate system in actual space, with one direction in the horizontal direction as the X-axis, an axis in the horizontal direction orthogonal to the X-axis as the Y-axis, and an axis in the direction orthogonal to both the X-axis and the Y-axis as the Z-axis, and detects the position of the obstacle represented by the feature points. The position detection unit includes: The non-detection unit, when setting a pre-defined detectable area where the obstacle can be detected by the sensor and where a part of the moving body exists as a non-detectable area, determines that the obstacle does not exist in the non-detectable area regardless of the sensor's detection result; as well as The detection unit detects the position of the obstacle in the detection area, which is a detection area different from the non-detection area, existing within the detectable area. If the coordinates of the feature point in the coordinate system of the non-detection unit in the actual space are included in the non-detection area, it is determined that there is no obstacle represented by those coordinates.

2. The obstacle detection device according to claim 1, wherein, The mobile vehicle is a forklift. The non-detection area is set at the location where the counterweight of the forklift is located.

3. The obstacle detection device according to claim 1, wherein, The non-detection area is defined by the three-dimensional coordinates of the region where the portion exists in the coordinate system of the actual space.

4. The obstacle detection device according to claim 1, wherein, The sensor is a stereo camera. The position detection unit The disparity is calculated by comparing a first image obtained from the stereo camera as the detection result with a second image, and the coordinates of the feature point in the coordinate system of the actual space are obtained based on the disparity. The location of the obstacle is detected by clustering the feature points that are excluded from the feature points determined by the non-detection unit to be included in the non-detection region.

5. An obstacle detection method, comprising an obstacle detection device mounted on a moving body and equipped with a sensor and a position detection unit, for detecting the position of an obstacle, characterized in that, Include: The step of the position detection unit obtaining the detection result of the sensor; The step of the position detection unit acquiring feature points in a coordinate system in actual space with one direction in the horizontal direction as the X-axis, the axis in the horizontal direction orthogonal to the X-axis as the Y-axis, and the axis in the direction orthogonal to the X-axis and Y-axis as the Z-axis, and detecting the position of the obstacle represented by the feature points; When a pre-defined detectable area where the obstacle can be detected by the sensor and where a part of the moving body exists is set as a non-detectable area, the position detection unit determines that the obstacle does not exist in the non-detectable area regardless of the sensor's detection result; as well as The step of the position detection unit detecting the position of the obstacle in the detection area, which is a detection area different from the non-detection area, existing in the detectable area. The step of determining that the obstacle does not exist in the non-detection area, regardless of the sensor's detection result, is the step of determining that the obstacle represented by the coordinates does not exist if the coordinates of the feature point in the coordinate system of the actual space are contained in the non-detection area.

Images