Visibility assistance system, industrial vehicle, visibility assistance method and program

Abstract

To provide a visual field support system which can display, for a driver, a predicted route and whether or not there is any obstacle on the predicted route.SOLUTION: A visual field support system according to the present invention includes: a measurement unit for estimating a position of an industrial vehicle; a route prediction unit for predicting a route of an object moving together with the industrial vehicle or a part thereof with reference to the estimated position and calculating an interference position of an obstacle with the route predicted based on map data including positional information of the obstacle; a transparent display unit that is attached to a person's head and is capable of displaying image data; and a control unit for displaying image data of the route starting from the object or a part thereof in the estimated position of the industrial vehicle and ending at the interference position so as to be superimposed on the route of the actual world seen through the display unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a visibility assistance system, an industrial vehicle, and a visibility assistance method and program. [Background technology]

[0002] Patent Document 1 discloses a system that calculates the driving operation required for parking based on parking-related data such as the initial position of the vehicle and the size of the parking lot, and the positioning result of the vehicle, and notifies the driver. With this system, for example, the trajectory of the rear wheels when the vehicle is reversed and parked can be grasped. When driving a forklift, especially when transporting a wide load, it is necessary to check not only the trajectory of the tires but also the side of the vehicle.

[0003] Patent Document 2 discloses a display device that takes pictures of the surroundings of the vehicle using cameras arranged around the vehicle body, creates an overhead image that shows an image of the surroundings with the vehicle at the center based on the taken images, and displays an area in real space where an object may exist within the predicted path of the vehicle by superimposing it on the overhead image. In general, the overhead view does not correctly reproduce the positional relationship of objects that are higher (or lower) than a reference plane (for example, the ground), making it difficult to judge whether the predicted path displayed in accordance with the reference plane interferes with a three-dimensional obstacle.

[0004] Using AR (Augmented Reality) glasses, it is possible to display a virtual object of the predicted path by AR superimposed on the real space. However, if a virtual object is simply displayed, the virtual object itself is displayed in front of the obstacle, making it difficult to understand the three-dimensional positional relationship between the predicted path and the obstacle (whether it is in front or interfering), and it is difficult to judge whether the predicted path interferes with the obstacle. In response to this, it is possible to recognize the surrounding environment of the AR glasses, judge the overlap between the predicted path and the obstacle, and display the predicted path by partially blocking it, thereby expressing interference (occlusion), but this requires a sensor and increases the processing load. In addition, in the case of a forklift, blocking occurs due to overlap with structures equipped on the vehicle. For example, the predicted path of the tip of the fork is blocked by a mast that is in front of the driver. In response to this, it is possible to implement a function that recognizes the vehicle's surrounding environment by installing multiple sensors around the vehicle and blocks the predicted path of the fork tips when they encounter obstacles but does not block structures on the vehicle (e.g., masts). However, since the viewpoints of the sensors and the AR glasses (driver) are different, this complicates the processing.

[0005] As a related technique, Patent Document 3 discloses a support device that detects the tire angle of a forklift, calculates a predicted trajectory of the tips of the forks or the tips of a transported object supported by the forks when the forklift moves forward at that tire angle, and displays the calculated predicted trajectory on a monitor in the driver's seat by superimposing it on an image of the front including the tips of the forks. Patent Document 4 discloses a technique for estimating the self-position of a forklift. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 59-201082 [Patent Document 2] JP 2006-303985 A [Patent Document 3] JP 2006-96457 A [Patent Document 4] JP 2022-34861 A Summary of the Invention [Problem to be solved by the invention]

[0007] There is a demand for technology that can clearly display a predicted path and whether or not there is interference with obstacles on that path.

[0008] The present disclosure provides a visibility assistance system, an industrial vehicle, a visibility assistance method, and a program that can solve the above problems. [Means for solving the problem]

[0009] The visibility assistance system of the present disclosure includes a positioning unit that estimates a position of an industrial vehicle, a path prediction unit that predicts a path of a part of the industrial vehicle or an object moving together with the industrial vehicle based on the estimated position and calculates an interference position where the path interferes with the obstacle based on map data including position information of the obstacle, a transparent display unit that is worn on a person's head and is capable of displaying image data, and a control unit that displays image data of the path starting from the part of the industrial vehicle or the object related to the prediction of the path at the estimated position to the interference position, superimposed on the position of the path in the real world as seen through the display unit. When the path interferes with the obstacle, the control unit cuts off the path at the interference position and displays image data of the path up to the interference position by displaying the obstacle blocking the path beyond the interference position.

[0010] The industrial vehicle of the present disclosure is equipped with the above-mentioned visibility assistance system.

[0011] The visibility assistance method of the present disclosure includes a step of estimating a position of an industrial vehicle; a step of predicting a course of a part of the industrial vehicle or an object moving together with the industrial vehicle based on the estimated position, and calculating an interference position where the course interferes with the obstacle based on map data including position information of the obstacle; and a step of superimposing and displaying image data of the course starting from the part of the industrial vehicle or the object related to the prediction of the course at the estimated position to the interference position on the position of the course in the real world as seen through a transparent display unit worn on a person's head and capable of displaying image data. If, in the displaying step, the path interferes with the obstacle, the path is cut off at the interference position, and the image data of the path up to the interference position is displayed by displaying the obstacle blocking the path beyond the interference position.

[0012] The program disclosed herein also includes the steps of: estimating a position of an industrial vehicle; predicting a path of a part of the industrial vehicle or an object moving together with the industrial vehicle based on the estimated position, and calculating an interference position where the path interferes with the obstacle based on map data including position information of the obstacle; and displaying image data of the path starting from the part of the industrial vehicle or the object related to the prediction of the path at the estimated position to the interference position, superimposed on the position of the path in the real world as seen through a transparent display unit worn on a person's head and capable of displaying image data. and in the display step, when the path interferes with the obstacle, a process of cutting the path at the interference position and displaying the path so that the obstacle blocks the path ahead of the interference position, thereby displaying image data of the path up to the interference position. Execute the command. Effect of the Invention

[0013] According to the visibility assistance system, industrial vehicle, visibility assistance method, and program described above, the predicted path of the industrial vehicle and the presence or absence of interference with an obstacle on that predicted path can be displayed in an easily understandable manner. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a forklift according to an embodiment. [Diagram 2] 1 is a block diagram showing an example of a visibility assistance system according to an embodiment. [Diagram 3]FIG. 4 is a first diagram illustrating calculation of a predicted path and interference with an obstacle according to an embodiment. [Figure 4A] FIG. 2 is a first diagram showing a display example of a predicted virtual course according to the embodiment. [Figure 4B] FIG. 11 is a first diagram showing a display example in which the position of interference with an obstacle is not displayed. [Figure 5A] FIG. 2 is a second diagram showing a display example of a predicted virtual course according to the embodiment. [Figure 5B] FIG. 11 is a second diagram showing a display example in which the position of interference with an obstacle is not displayed. [Figure 6] FIG. 13 is a third diagram showing a display example of a predicted virtual course according to the embodiment. [Figure 7] FIG. 11 is a second diagram illustrating calculation of a predicted path and interference with an obstacle according to the embodiment. [Figure 8] FIG. 4 is a fourth diagram showing an example of a display of a predicted virtual course according to the embodiment. [Figure 9] 11 is a flowchart illustrating an example of a display process of a predicted virtual course according to the embodiment. [Figure 10] 1 is a diagram illustrating an example of a hardware configuration of a visibility assistance system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] <Embodiment> Hereinafter, the visibility assistance system of the present disclosure will be described with reference to the drawings. (composition) FIG. 1 is a schematic diagram showing an example of a forklift according to each embodiment. The forklift 1 includes a vehicle body 10, a loading device 11, a path calculation device 16, and a display device 20. The vehicle body 10 includes a driving device and a steering device (not shown) and tires 14, 15 driven by them, and drives the forklift 1. The loading device 11 includes a fork 12 and a mast 13 for raising and lowering the fork 12, and performs loading and unloading operations by raising and lowering the load loaded on the fork 12. The path calculation device 16 predicts the path of the forklift 1, checks whether there is interference with an obstacle present on the path, and calculates the interference position on the predicted path if interference occurs. The path calculation device 16 transmits information such as the predicted path and the interference position with the obstacle to the display device 20. The display device 20 is an AR glass or an MR (Mixed Reality) glass. The driver 100 drives the forklift 1 wearing the display device 20. The display device 20 displays the predicted path and interference position calculated by the path calculation device 16 by superimposing them on the real world. In this specification, the direction in which the loading device 11 of the forklift 1 is provided as shown in Fig. 1 is called the front, the opposite direction is called the rear, and the left and right directions when facing forward are called the left side and the right side, respectively.

[0016] FIG. 2 is a block diagram illustrating an example of a visibility assistance system according to the embodiment. The visibility assistance system 200 includes a path calculation device 16 and a display device 20. The path calculation device 16 and the display device 20 are communicatively connected. The path calculation device 16 is configured by a computer. The display device 20 is an AR glass or an MR glass configured to include a display device and a computer (hereinafter, both will be referred to as AR glasses).

[0017] (Configuration of the route calculation device) As shown in the figure, the course calculation device 16 includes a positioning unit 161 , a course prediction unit 162 , a transmission / reception unit 163 , and a storage unit 164 . The positioning unit 161 includes positioning sensors such as a GPS (Global Positioning System), an odometer, an IMU (Inertial Measurement Unit), and a camera (not shown) mounted on the vehicle body 10, and estimates the position of the forklift 1 based on information detected by the sensors. Any method of estimating the position may be used. For example, as described in Patent Document 4, the positioning unit 161 may estimate the position by matching an image captured by a camera with a marker in a warehouse, may estimate the position using a V-SLAM (Visual Simultaneous Localization and Mapping) technology, or may estimate the position by odometry. The positioning unit 161 does not need to include all of the above sensors, and may estimate the position using a sensor other than the above sensors. The positioning unit 161 estimates the position every moment as the vehicle travels.

[0018] The path prediction unit 162 acquires information such as the angle of the tires 14 and 15, the height of the fork 12, and the vertical and horizontal sizes of the cargo loaded on the fork 12, and calculates a predicted path of the forklift 1 when it proceeds in the same direction (forward, backward, left turn, right turn, etc.) based on the information and the distance from the tire 14 to the tire 15. For example, the path prediction unit 162 calculates a predicted path of the tip of the fork 12, a predicted path of the left and right sides of the cargo loaded on the fork 12, a predicted path of the vehicle body 10 or the tires 14 and 15, etc. Although the details of the calculation process of the predicted path are not described, various conventionally provided methods can be used. The path prediction unit 162 updates the predicted path from moment to moment as the vehicle proceeds. In addition, the path prediction unit 162 superimposes the predicted path on map data created in advance, and determines whether or not the predicted path will interfere with an obstacle if the predicted path proceeds, and if an obstacle will interfere with the obstacle, calculates the interference position of the obstacle on the predicted path. The interference includes collision, contact, and intersection in a three-dimensional manner. FIG. 3 shows a diagram in which the map data 30 and the predicted path of the forklift 1 are superimposed. The map data 30 includes position information of obstacles such as walls, shelves, and luggage in the warehouse where the forklift 1 is operating, and is registered in the storage unit 164 in advance. In addition, the map data 30 preferably includes the latest position information of luggage and information on the size of each luggage, taking into consideration the possibility that the luggage may become an obstacle when the luggage is temporarily placed in an aisle or an empty space, or when the luggage is placed on a shelf and is large in size and protrudes into the aisle. For example, the warehouse management system may manage and update the map data 30 to which the latest position information of luggage and information on the height and width of each luggage are added, and the path calculation device 16 may acquire the latest map data 30 from the warehouse management system. When the forklift 1 advances in the direction of the arrow 31 as shown in FIG. 3, the path prediction unit 162 calculates the predicted paths of the left tire 15 and the right tire 15. The predicted path 32 is the predicted path of the left tire 15, and the predicted path 33 is the predicted path of the right tire 15. The path prediction unit 162 calculates whether or not there is an obstacle such as a wall or luggage on the predicted path, and if an obstacle is present, calculates the interference position between the predicted path and the obstacle.The shape of the interference position varies depending on the shape and traveling direction of the interfering objects, and may be a point, a line, a surface, or the like. In the example of FIG. 3, the course prediction unit 162 calculates an interference position 35 between the left tire 15 and the wall 34, and an interference position 36 between the right tire 15 and the wall 34. The course prediction unit 162 transmits information on the calculated predicted courses 32, 33 and the interference positions 35, 36 to the display device 20 through the transmission / reception unit 163. In the example of FIG. 3, the course prediction unit 162 transmits, for example, information specifying the start points of the predicted courses 32, 33 (for example, position information on the left and right tires 15 based on a predetermined position of the forklift 1), information representing the shape of the curves of the predicted courses 32, 33 (for example, a curve function), and position information on the interference positions 35, 36 (for example, coordinate information in a two-dimensional coordinate system with the start point of the predicted course 32 as the origin).

[0019] The transmitting / receiving unit 163 transmits and receives data to and from other devices. For example, the transmitting / receiving unit 23 transmits route information and the like to the display device 20 by wireless communication such as WiFi (registered trademark) or Bluetooth (registered trademark). The transmitting / receiving unit 163 also acquires information detected by various sensors (not shown) such as a GPS provided in the positioning unit 161, and sensors that detect the height of the fork 12 and the angles of the tires 14, 15. For example, when the driver 100 reads information from a two-dimensional code attached to the luggage by a barcode or the like when loading luggage onto the forklift 1, the transmitting / receiving unit 163 receives size information of the luggage from the warehouse management system in a case where the fact that the luggage has been loaded onto the forklift 1 and the size of each luggage are managed by a warehouse management system.

[0020] The memory unit 164 stores map data 30 including position information of obstacles, information acquired by the transmission / reception unit 163, information on the size of the forklift 1 used for route prediction, such as the distance between tires 14 and 15, the distance from a reference position to the tip of the fork 12, the width of the vehicle body 10, the height of the mast 13, and information such as the vertical and horizontal sizes of the cargo being transported acquired from a warehouse management system or the like.

[0021] (Display Device Configuration) As shown in the figure, the display device 20 includes a sensor unit 21, a display unit 22, a transmitting / receiving unit 23, a control unit 24, and a storage unit 25.

[0022] The sensor unit 21 includes a sensor for head tracking of the driver 100 (eg, a plurality of visible light cameras or an IMU), a sensor for eye tracking of the driver 100 (eg, an infrared camera), a depth sensor, a microphone, and the like.

[0023] The display unit 22 constitutes a part of the lens (screen) of the AR glasses, and is constituted by, for example, a transparent holographic display (the outside world is visible through the lens). In a state where a virtual object is not projected onto the lens, the driver 100 can see the real world as it is. The virtual object is image data projected onto the display unit 22. When a virtual object is displayed on the display unit 22, the driver 100 sees the virtual object superimposed on the real world.

[0024] The transmission / reception unit 23 transmits and receives data to and from other devices. For example, the transmission / reception unit 23 receives route information and the like from the route calculation device 16 via wireless communication such as WiFi (registered trademark) or Bluetooth (registered trademark).

[0025] The control unit 24 performs display control of the display unit 22, etc. The control unit 24 recognizes the spatial shape around the driver 100 based on the information detected by the sensor unit 21, creates three-dimensional map information of the surrounding space, and generates a virtual space. The control unit 24 places a virtual object at a desired position in the virtual space based on the created map information. Then, the virtual object is displayed to the driver 100 superimposed on an object existing at that position in the real world. For example, the virtual object can be displayed at an arbitrary position based on a predetermined position of the forklift 1. When placing a virtual object at the tip of the fork 12, the positional relationship between the driver 100 and the forklift 1 can be calculated from the positional relationship between the driver 100 and the forklift 1 and the positional relationship between the center of the front axle of the forklift 1 and the tip of the fork 12, assuming that the center of the front axle of the forklift 1 is used as a reference. The positional relationship between the driver 100 and the forklift 1 can be calculated by calibration (described later). The positional relationship between the center of the front axle of the forklift 1 and the tip of the fork 12 can be calculated from the design information of the forklift 1. When the control unit 24 places a virtual object at a position in the virtual space corresponding to the position of the tip of the fork 12 for the driver 100 calculated in this manner, the virtual object is recognized by the driver 100 as if it actually exists at the tip of the fork 12. For example, when the predicted path of the tip of the fork 12 is displayed as a virtual object from the tip of the fork 12, the driver 100 can confirm the trajectory that the tip of the fork 12 will take in the future while driving, and can visually check whether the tip will hit an obstacle or the like. In addition, the control unit 24 calculates the position of the head of the driver 100 when the driver 100 takes a posture such as bending or leaning out of the driver's seat to the left or right based on the information detected by the sensor unit 21. Then, the control unit 24 generates and places a virtual object based on the positional relationship between the head of the driver 100 and the virtual object so that the virtual object looks the same as when it actually exists in the real world. As a result, even if the driver 100 changes his / her posture while driving, the driver 100 can see the virtual object of the predicted path placed in the direction that the driver 100 looks.For example, if both the predicted path of the tips of the forks 12 and the predicted path of the tires 15 (rear wheels) are placed as virtual objects, when the driver 100 faces forward, the predicted path of the tips of the forks 12 is visible, but when the driver looks back, the predicted path of the tips of the forks 12 is not visible, and only the predicted path of the tires 15 is visible.

[0026] The AR glasses have the functions of freely arranging virtual objects, arranging a virtual object at a certain position in a virtual space so that the virtual object appears to exist at that position, and changing the appearance of the virtual object according to the posture of the driver 100. The user sets the desired virtual object in the display device 20 (a developer uses an AR development tool to develop a virtual object and implements the developed program in the display device 20). The control unit 24 then places the virtual object at the desired position according to the setting. In this embodiment, the control unit 24 is configured to display a virtual object of a predicted path indicated by the path information when it acquires the path information from the path calculation device 16. For example, when displaying a predicted path of the tip of the fork 12, assuming that the forklift 1 is moving along a straight line without an obstacle, the path calculation device 16 transmits information specifying the tip of the fork 12, which is the starting point of the predicted path (for example, information indicating the positional relationship between the center of the front wheel axle of the forklift 1 and the tip of the fork 12), and transmits information indicating the shape of the predicted path, including a straight line and a direction and angle (for example, forward) of the straight line. When the control unit 24 acquires this information, it specifies a position in the virtual space corresponding to the tip of the real fork 12 as the starting point, and places a straight-line virtual object in the forward direction from there. This allows the driver 100 to recognize the predicted path of the tip of the fork 12 and that there is no obstacle ahead. In a similar situation, if there is an obstacle 5 m ahead on the right side, the path calculation device 16 provides information specifying the tip of the fork 12 as the starting point, a straight line as the shape of the predicted path, and coordinate information 5 m ahead as the interference position on the right front. When the control unit 24 acquires this information, it places a straight-line virtual object in the forward direction from the tip of the left fork 12 as the starting point, and places a straight-line virtual object in the forward direction limited to 5 m ahead from the tip of the right fork 12. From the driver 100's perspective, the straight line of the predicted path for the left fork 12 extends all the way ahead, and for the right fork 12, it appears to extend to the obstacle 5 m ahead and be cut off there.This indicates that an obstacle exists before the cut-off path, and the path of the right fork 12 is blocked by the obstacle. In this manner, in this embodiment, when the predicted path interferes with an obstacle, the predicted path is cut at the interference position, and a display is performed in which the path is blocked (pseudo occlusion). This allows the driver 100 to recognize that if the fork 12 is moved forward as is, it will interfere with an obstacle in front of the right side. An example of a virtual object of the predicted path as seen by the driver 100 wearing the display device 20 is shown in Figs. 4A to 6. The virtual object of the predicted path is referred to as a "predicted virtual path."

[0027] 4A shows a display example of the predicted virtual paths of the tires 15. A predicted virtual path 41A is the predicted path of the left tire 15, and a predicted virtual path 42A is the predicted path of the right tire 15. An interference position 43A is the interference position of the left tire 15 and a wall 40A, and an interference position 44A is the interference position of the right tire 15 and a wall.

[0028] FIG. 4B shows an example of a display of a predicted virtual path when the interference position is not displayed even though there is interference between the predicted path and an obstacle. A predicted virtual path 41B is the predicted path of the left tire 15, and a predicted virtual path 42B is the predicted path of the right tire 15. Referring to FIG. 4B, the predicted virtual paths of the left and right tires 15 are displayed superimposed on a wall 40B, making it difficult for the driver 100 to determine whether or not there is interference between the left and right tires 15 and the wall 40B. In contrast, as shown in FIG. 4A, by displaying the predicted virtual paths 41A and 42A cut off at interference positions 43A and 44A, the driver 100 can easily recognize interference with an obstacle.

[0029] FIG. 5A shows an example of a display of a predicted virtual path of the tip of the fork 12. FIG. 5A shows a scene where the tip of the fork 12 is inserted into a hole in a pallet 50A. If the pallet 50A to be worked on is regarded as an obstacle, displaying the predicted path of the tip of the fork 12 can be useful in determining whether the current path is acceptable. A predicted virtual path 51A is a predicted path of the tip of the left fork 12, and a predicted virtual path 52A is a predicted path of the tip of the right fork 12. An interference position 53A is an interference position between the tip of the left fork 12 and the pallet 50A, and an interference position 54A is an interference position between the tip of the right fork 12 and the pallet 50A. In FIG. 5A and FIG. 5B and FIG. 6 described later, the fork 12 is displayed in front of the mast 13 without using the occlusion function in order to make the fork 12 easier to recognize, but this is not related to the function of this embodiment.

[0030] FIG. 5B shows an example of a display of a predicted virtual path when interference with an obstacle (pallet 50B) is not displayed even though the interference position is present. A predicted virtual path 51B is a predicted path of the tip of the left fork 12, and a predicted virtual path 52B is a predicted path of the tip of the right fork 12. Referring to FIG. 5B, the predicted paths of the tips of the left and right forks 12 are displayed superimposed on the pallet 50B, making it difficult for the driver 100 to determine whether the fork 12 can be inserted into the hole of the pallet 50B or whether the fork 12 will pass above the pallet 50B. In contrast, as shown in FIG. 5A, by displaying the predicted virtual paths 51A and 52A cut off at the interference positions 53A and 54A, the driver 100 can easily confirm whether the tip of the fork 12 can be inserted into the hole of the pallet 50A.

[0031] FIG. 6 shows a display example in which which of the left and right forks 12 will interfere with an obstacle first is displayed as a virtual object. As in FIG. 5A, predicted virtual paths 61 and 62 are predicted paths of the tips of the left and right forks 12, respectively, and interference positions 63 and 64 are interference positions between the tips of the left and right forks 12 and the pallet 60, respectively. The control unit 24 displays an auxiliary line 65, which is a perpendicular line from the interference position 63, which is closer to the driver 100, to the predicted virtual path 62 among the interference positions 63 and 64. The auxiliary line 65 is a virtual object. This allows the driver 100 to know which of the tips of the left and right forks 12 will interfere with the pallet 60 first (the left one interferes first). In addition, by displaying the auxiliary line 65, the driver 100 can know the distance 66 in the depth direction between the interference position 63 and the interference position 64, making it easier to know how directly facing the pallet 60 the driver 100 is.

[0032] In this way, when there are multiple interference positions, the control unit 24 may perform a display that clarifies the position that will interfere first. As another display example, the control unit 24 may change the color of the interference position 63 in FIG. 6, make it blink, indicate the interference position 63 with an arrow or the like, or display characters such as "interference" near the interference position 63.

[0033] So far, examples of displaying the predicted virtual path of the tip of the fork 12 and the predicted virtual path of the tire 15 have been described. The display target of the predicted virtual path is not limited to these, and the predicted virtual path can be displayed for any other part of the forklift 1 (for example, the top end of the mast 13 or the left and right sides of the vehicle body 10). In addition, the predicted virtual path of the load carried by the forklift 1 may be displayed, not limited to a part of the forklift 1. For example, when a wide container 70 is transported as shown in FIG. 7, the container 70 may interfere with an obstacle 71 even if the forklift 1 itself does not interfere. The path prediction unit 162 of the path calculation device 16 obtains information on the height and width of the container 70 from, for example, a warehouse management system, and calculates a predicted path 72 of the left side and a predicted path 73 of the right side of the container 70 based on the position estimated by the positioning unit 161. The path prediction unit 162 also checks the calculated predicted paths 72, 73 against the map data 30 to check whether the predicted paths 72, 73 interfere with an obstacle 71 such as a wall or baggage included in the map data 30. If there is interference, the path prediction unit 162 calculates, for example, the coordinate position of an interference position 74 based on a predetermined position of the forklift 1. The display device 20 displays the predicted virtual path on the display unit 22 based on the predicted paths 72, 73 and the interference position 74 calculated by the path prediction unit 162. Since the predicted virtual path (not shown) is displayed in front, the driver can recognize the predicted path of the container 70 even when the driver's view is blocked by the wide container 70. Furthermore, since the predicted virtual path is cut off at the interference position 74 with the obstacle 71 for the right predicted path 73, the driver 100 can confirm the interference with the obstacle 71. FIG. 7 illustrates an example of a forklift 1 loaded with a container 70 moving forward, but the same process can also be used when the forklift 1 is moving backwards, whereby the predicted course and interference position can be calculated and the predicted virtual course can be displayed.

[0034] Up to now, we have dealt with the predicted course in two dimensions and the interference with obstacles. However, when the forks 12 are raised to perform loading and unloading work, it is desirable to display the predicted course at the height position of the forks 12 and the presence or absence of interference with obstacles. For this purpose, position information of obstacles by height is added to the map data 30, and the course prediction unit 162 recognizes the height of the forks 12, for example, by a lifting height sensor or the like provided on the mast 13, and reads the position of the obstacle at the recognized height from the map data 30. Then, the course prediction unit 162 calculates the predicted course at the height of the forks 12 and the interference position with the obstacle in the same manner as in the two-dimensional case, using the angles of the tires 14 and 15, the size of the luggage, and the like. The display device 20 displays the predicted virtual course on the display unit 22 based on the predicted course at the height of the forks 12 and the interference position calculated by the course prediction unit 162. FIG. 8 shows an example of visual information that can be seen when a driver wearing the display device 20 leans to the right of the driver's seat and looks up when the forks 12 are raised and inserted into holes in a pallet 80 loaded with luggage 81. Predicted virtual paths 82 and 83 are the predicted paths of the tips of the left and right forks 12, respectively, and interference positions 84 and 85 are the interference positions of the tips of the left and right forks 12 and the pallet 80, respectively. By displaying the predicted virtual path taking the height into consideration, the driver 100 can perform loading and unloading work while visually checking the predicted path of the forks 12 at the work target position and interference with the pallet 80 and luggage 81. In addition, by displaying the predicted virtual path taking the height into consideration, it is possible to prevent a phenomenon that the display of the predicted virtual path is cut off when the forks 12 are raised and moving and only obstacles that are actually shorter than the height of the forks 12 are present (for example, when the forks 12 pass over flat-laid luggage).

[0035] The storage unit 25 stores information detected by the sensor unit 21, information received by the transmission / reception unit 23, CAD information of the forklift 1 or various types of forklifts, and the like.

[0036] (operation) Next, the flow of the display processing for the predicted virtual course will be described with reference to FIG. FIG. 9 is a flowchart illustrating an example of display control of the virtual monitor according to the embodiment. When the driver 100 wearing the display device 20 sits in the driver's seat of the forklift 1, the driver starts up the display device 20 and performs calibration (step S1). In the calibration, the positional relationship between the driver 100 and the forklift 1 is calculated. For example, the control unit 24 recognizes the handle and other structures of the forklift 1 using the visible light camera of the sensor unit 21, and estimates the type of vehicle of the forklift 1 from the shape and appearance of the photographed handle, etc. The control unit 24 reads out CAD information corresponding to the estimated type of vehicle from the CAD information recorded in the storage unit 25, and grasps the overall shape of the forklift 1. In addition, for example, the control unit 24 calculates the positional relationship between the driver 100 and the steering wheel, that is, where the driver 100 is sitting in the driver's seat, how high the driver 100's head is, or the positional relationship between the driver 100 and each part of the forklift 1 (horizontal distance, vertical distance, etc.) from the read CAD information of the forklift 1 by analyzing an image of the steering wheel captured by the visible light camera of the sensor unit 21 or measuring the distance to the steering wheel by the depth sensor of the sensor unit 21. This makes it possible to grasp the distance between the driver 100 and the tips of the forks 12, the left and right tires 15, and both sides of the cargo (the size of the cargo is also required for the distance to both sides of the cargo), and to place the predicted virtual route in an accurate position (in accordance with the position of the tips of the forks 12, etc. in the real world). In addition, for example, a two-dimensional marker may be provided at a predetermined position on the forklift 1 (for example, on the driver's side of the mast 13), the two-dimensional marker may be recognized by the sensor unit 21, and the positional relationship between the driver 100 and each part of the forklift 1 may be calculated in a similar manner to that described above based on the positional relationship with the two-dimensional marker, the type of forklift 1 indicated by the two-dimensional marker, and CAD information for that type of vehicle.

[0037] When the calibration is completed, the driver 100 performs a predetermined operation on the display device 20 to start a function for displaying a predicted virtual path. The driver 100 also performs a predetermined operation to start the path calculation device 16. When the path calculation device 16 is started, the positioning unit 161 estimates the position (predetermined reference position) of the forklift 1 (step S2). Next, the path prediction unit 162 calculates the interference position between the predicted path and an obstacle (step S3). Next, the path prediction unit 162 transmits information such as the predicted path to the display device 20 through the transmission / reception unit 163 (step S4). For example, in the case of the predicted path of the tire 15 shown in FIG. 3, if the predicted paths 32 and 33 are each a part of a circular arc, the path prediction unit 162 transmits information specifying the left and right tires 15 as the starting points, information on the predicted path such as the radius, center position, and angle of each circular arc, and coordinate information on the interference positions 35 and 36 to the display device 20. In the display device 20, the control unit 24 acquires information such as the predicted course transmitted by the course prediction unit 162 through the transmission / reception unit 23. The control unit 24 displays the predicted virtual course on the display unit 22 (step S5). For example, the control unit 24 generates a virtual object of a specified arc for the left tire 15, and places the generated virtual object of the arc with a starting point at a position corresponding to the left tire 15 in the virtual space. The virtual object may be a fan-shaped virtual object that transparently represents the inside of a circle surrounded by the arc, even if it is only the arc portion. The control unit 24 similarly generates a virtual object for the right tire 15, and places the virtual object along the predicted course indicated by the course information transmitted from the course calculation device 16 with a starting point at the right tire 15. This allows the driver 100 to confirm the predicted virtual courses 41A and 42A exemplified in FIG. 4A.

[0038] Next, the control unit 24 judges whether to end the display of the predicted virtual path (step S6). For example, if the driver 100 performs an operation to end the function of displaying the predicted virtual path, the control unit 24 judges that the function of displaying the predicted virtual path is to be ended, and if not, judges that the function is not to be ended. If it is judged that the function is not to be ended (step S6; No), the processing from step S2 onwards is repeated. The positioning unit 161 estimates the position of the forklift 1 from moment to moment, and the path prediction unit 162 updates the predicted path of the forklift 1 based on the tire angle at each moment. The display device 20 arranges the predicted virtual path on the virtual section based on the predicted path calculated by the path prediction unit 162. This allows the driver 100 to drive while checking the future path of the forklift 1 that changes due to his / her driving. On the other hand, if it is judged that the function is to be ended (step S6; Yes), the processing flow of FIG. 8 is ended.

[0039] In the above embodiment, the visibility assistance system 200 is applied to the forklift 1, but the visibility assistance system 200 can be applied to industrial vehicles other than the forklift, such as an excavator, a bulldozer, a crane, and a reach stacker. The fork 12 of the forklift 1, the boom and bucket of an excavator, the bucket and arm of a bulldozer, and the boom, jib, and hook of a crane are examples of working devices that are equipped on an industrial vehicle. Furthermore, objects such as luggage, earth and sand, and rubble that are transported by these working devices are examples of objects that move together with the industrial vehicle. Furthermore, in the above embodiment, the path calculation device 16 is provided on the forklift 1, but functions and configurations other than the positioning unit 161 may be implemented in an external computer or the display device 20. When implemented in the display device 20, the positioning unit 161 may be included in the display device 20.

[0040] (effect) In this embodiment, the driver wears AR glasses (display device 20) and the predicted virtual route is displayed on the virtual section. Furthermore, the location of obstacles is grasped based on map data of the warehouse, and the predicted virtual route is blocked by determining whether the predicted virtual route overlaps with the obstacle. This makes it possible to clearly display to the driver the predicted route and whether there is interference between the predicted route and the obstacle. Furthermore, by displaying the predicted route superimposed on the real world using the AR glasses, the driver 100 does not need to check the monitor in the driver's seat, and can check the predicted route with a natural line of sight.

[0041] 10 is a diagram showing an example of the hardware configuration of the visual field assistance system according to the embodiment. A computer 900 includes a CPU 901, a main memory device 902, an auxiliary memory device 903, an input / output interface 904, and a communication interface 905. The above-mentioned course calculation device 16 and display device 20 are implemented in a computer 900. The above-mentioned functions are stored in the auxiliary storage device 903 in the form of a program. The CPU 901 reads the program from the auxiliary storage device 903, loads it in the main storage device 902, and executes the above-mentioned processing in accordance with the program. The CPU 901 also reserves a storage area in the main storage device 902 in accordance with the program. The CPU 901 also reserves a storage area in the auxiliary storage device 903 for storing data being processed in accordance with the program.

[0042] A program for realizing all or part of the functions of the course calculation device 16 and the display device 20 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform processing by each functional unit. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. In addition, if a WWW system is used, the term "computer system" also includes a homepage providing environment (or display environment). In addition, the term "computer-readable recording medium" refers to a portable medium such as a CD, DVD, or USB, or a storage device such as a hard disk built into a computer system. In addition, if the program is distributed to the computer 900 via a communication line, the computer 900 that receives the program may expand the program into the main storage device 902 and execute the above processing. In addition, the above program may be for realizing part of the above-mentioned functions, or may be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

[0043] As described above, several embodiments according to the present disclosure have been described, but all of these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and gist of the invention.

[0044] <Additional Notes> The visibility assistance system, the industrial vehicle, the visibility assistance method, and the program described in each embodiment can be understood, for example, as follows.

[0045] (1) A visibility assistance system according to a first aspect includes a positioning unit that estimates a position of an industrial vehicle; a path prediction unit that predicts a path of a part of the industrial vehicle or an object moving together with the industrial vehicle based on the estimated position and calculates an interference position where the path interferes with the obstacle based on map data including position information of the obstacle; a transparent display unit that is worn on a person's head and is capable of displaying image data; and a control unit that displays image data of the path starting from the part of the industrial vehicle or the object related to the prediction of the path at the estimated position to the interference position, superimposed on the position of the path in the real world as seen through the display unit. The predicted path of the industrial vehicle and whether or not there is interference with an obstacle on that path can be displayed in an easily understandable manner.

[0046] (2) A visibility assistance system according to a second aspect is the visibility assistance system of (1), in which the path prediction unit predicts the path of a work device provided on the industrial vehicle and calculates the interference position. In industrial vehicles, the working equipment (forks of a forklift) may protrude from the body, and care must be taken when driving to avoid interference between the working equipment and obstacles. The system can calculate the predicted course and interference position of the working equipment of the industrial vehicle.

[0047] (3) A visibility assistance system according to a third aspect is the visibility assistance system of (1) to (2), wherein the path prediction unit predicts the path of a work object transported by the industrial vehicle and calculates the interference position. This makes it possible to calculate the predicted path and interference position on both sides of a load, for example, when a forklift is carrying a load that is wider than the vehicle body.

[0048] (4) A visibility assistance system according to a fourth aspect is a visibility assistance system according to any one of (1) to (3), wherein the path prediction unit predicts the path of a part of the industrial vehicle or the object related to the path prediction at a predetermined height, and calculates the interference position at that height. This makes it possible to calculate a predicted course and interference position according to height, even if the shape and size of the industrial vehicle and the distribution of obstacles vary depending on the height.

[0049] (5) A fifth aspect of the visibility assistance system is the visibility assistance system of (4), wherein the course prediction unit calculates the interference position at the height based on the map data including position information of obstacles by height. This makes it possible to calculate the interference position at various heights.

[0050] (6) A visibility assistance system according to a sixth aspect is a visibility assistance system including a course calculation device and a display device, wherein the course calculation device includes a positioning unit that estimates a position of an industrial vehicle, a course prediction unit that predicts a course of a part of the industrial vehicle or an object moving together with the industrial vehicle based on the estimated position, and calculates an interference position where the obstacle interferes with the course based on map data including position information of the obstacle, and a transmission unit that transmits information identifying the part of the industrial vehicle or the object related to the predicted course and information on the course to the interference position to the display device, wherein the display device includes a receiving unit that receives the identifying information and information on the course to the interference position, a transparent display unit that is worn on a person's head and is capable of displaying image data, and a control unit that displays image data of the course to the interference position starting from the part of the industrial vehicle or the object indicated by the identifying information at the estimated position, superimposed on the position of the course in the real world as seen through the display unit. This makes it possible to clearly display the predicted course and whether or not there is interference with an obstacle on that predicted course.

[0051] (7) An industrial vehicle according to a seventh aspect includes a visibility assistance system including (1) to (6).

[0052] (8) A visibility assistance method according to an eighth aspect includes the steps of: estimating a position of an industrial vehicle; predicting a path of a part of the industrial vehicle or an object moving together with the industrial vehicle based on the estimated position; and calculating an interference position where the path interferes with the obstacle based on map data including position information of the obstacle; and displaying image data of the path starting from the part of the industrial vehicle or the object related to the prediction of the path at the estimated position to the interference position, superimposed on the position of the path in the real world as seen through a transparent display unit worn on a person's head and capable of displaying image data.

[0053] (9) A program according to a ninth aspect causes a computer to execute the steps of estimating a position of an industrial vehicle; predicting a course of a part of the industrial vehicle or an object moving together with the industrial vehicle based on the estimated position, and calculating an interference position where the course interferes with the obstacle based on map data including position information of the obstacle; and displaying image data of the course starting from the part of the industrial vehicle or the object related to the prediction of the course at the estimated position to the interference position, superimposed on the position of the course in the real world as seen through a transparent display unit worn on a person's head and capable of displaying image data. [Explanation of symbols]

[0054] 1. Forklift 10. Body 11. Cargo handling equipment 12···Fork 13 Mast 14, 15... Tires 16. Path calculation device 161 Positioning unit 162 ···Career Prediction Section 163 Transmitter / receiver 164...Storage section 20...Display device 21 Sensor section 22...Display section 23 Transmitter / receiver 24 Control section 25...Storage section 100 Driver 200 Visibility aid system 900...Computer 901··CPU 902...Main memory 903...Auxiliary storage device 904 Input / Output Interface 905 Communication Interface

Claims

1. A positioning unit that estimates a position of the industrial vehicle; a path prediction unit that predicts a path of a part of the industrial vehicle or an object moving together with the industrial vehicle based on the estimated position, and calculates an interference position where the path interferes with the obstacle based on map data including position information of the obstacle; a transparent display unit that is attached to a person's head and is capable of displaying image data; a control unit that displays image data of the path starting from a part of the industrial vehicle or the object, which is related to the prediction of the path at the estimated position, to the interference position, superimposed on the position of the path in the real world as seen through the display unit; Equipped with When the path interferes with the obstacle, the control unit cuts the path at the interference position and displays the image data of the path up to the interference position by displaying a portion beyond the interference position being blocked by the obstacle. Vision aid system.

2. The path prediction unit predicts the path of a work device provided on the industrial vehicle and calculates the interference position. The vision aid system according to claim 1 .

3. The path prediction unit predicts the path of the work object transported by the industrial vehicle and calculates the interference position. The visibility assistance system according to claim 1 or 2.

4. the path prediction unit predicts a path of the part of the industrial vehicle or the object related to the prediction of the path at a predetermined height, and calculates the interference position at the height. The visibility assistance system according to claim 1 or 2.

5. the course prediction unit calculates the interference position at the height based on the map data including position information of obstacles by height; The vision assistance system according to claim 4.

6. A visibility assistance system including a course calculation device and a display device, The course calculation device is A positioning unit that estimates a position of the industrial vehicle; a path prediction unit that predicts a path of a part of the industrial vehicle or an object moving together with the industrial vehicle based on the estimated position, and calculates an interference position where the path interferes with the obstacle based on map data including position information of the obstacle; a transmission unit that transmits information that identifies a part of the industrial vehicle or the object related to the predicted path and information on the path up to the interference position to the display device, The display device includes: A receiving unit that receives the identifying information and information on the path to the interference position; a transparent display unit that is attached to a person's head and is capable of displaying image data; a control unit that displays image data of the path starting from a part of the industrial vehicle or the object indicated by the identifying information at the estimated position to the interference position, superimposed on the position of the path in the real world as seen through the display unit. Vision aid system.

7. The visibility assistance system according to claim 1 or 6, An industrial vehicle equipped with

8. estimating a position of the industrial vehicle; a step of predicting a course of a part of the industrial vehicle or an object moving together with the industrial vehicle based on the estimated position, and calculating an interference position where the course interferes with the obstacle based on map data including position information of the obstacle; a step of displaying image data of the path starting from a part of the industrial vehicle or the object related to the prediction of the path at the estimated position to the interference position, superimposed on the position of the path in the real world as seen through a transparent display unit that is worn on a person's head and is capable of displaying image data; having In the display step, if the path interferes with the obstacle, the path is cut at the interference position, and the image data of the path up to the interference position is displayed by displaying the path as being blocked by the obstacle beyond the interference position. Visual aid method.

9. On the computer, estimating a position of the industrial vehicle; a step of predicting a course of a part of the industrial vehicle or an object moving together with the industrial vehicle based on the estimated position, and calculating an interference position where the course interferes with the obstacle based on map data including position information of the obstacle; a step of displaying image data of the path starting from a part of the industrial vehicle or the object related to the prediction of the path at the estimated position to the interference position, superimposed on the position of the path in the real world as seen through a transparent display unit that is worn on a person's head and is capable of displaying image data; having a process of displaying image data of the route up to the interference position by cutting the route at the interference position and displaying the route so that the obstacle blocks the route beyond the interference position in the step of displaying the image data of the route up to the interference position; A program that executes the following.

Images