Work Machine and Environment Recognition System

Abstract

To provide a work machine and an environment recognition system in which obstacle information acquired and accumulated by existing work machines is provided for work machines operated by operators who are unfamiliar with factories.SOLUTION: A work machine according to the present invention includes: a sensor for observing surrounding environment; a self-position estimation unit; a normal operation recording unit of the work machine; an abnormal operation detection unit of the work machine; a sensor data recording unit for recording an observation result of the sensor at an abnormal operation position; and an obstacle recognition unit for generating a three-dimensional shape of the surrounding environment of the work machine by using the abnormal operation position and the observation result of the sensor, and the abnormal operation position and the observation result of the sensor detected in the other work machine and recognizing the obstacle in the work area from the three-dimensional shape.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a work machine and an environment recognition system. [Background technology]

[0002] In recent years, work machines operating in factories and logistics warehouses have begun to be equipped with obstacle detection sensors (cameras, LIDAR, millimeter waves, etc.) that are installed in automobiles, making it possible to safely avoid obstacles by using these sensors.

[0003] The detection performance of these sensors mainly consists of detection range and detection accuracy (resolution), but there is a trade-off between these two performance characteristics. In other words, if you try to widen the detection range, the detection accuracy will decrease, and conversely, if you increase the detection accuracy, the detection range will become narrower. To eliminate this trade-off, it is possible to equip the work machine with two types of sensors, one with a wide detection range and one with high detection accuracy, but this would increase costs. As such, there are limits to obstacle detection that relies solely on the work machine's sensors.

[0004] Patent document 1 describes a system in which a moving vehicle detects objects on a road from images captured by a camera that captures images including the road, and if the detected object is determined to be an obstacle, stores information about the obstacle and transmits that information to a central management device, which then safely and quickly notifies other vehicles of the information about the obstacle present on the road. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2007-323117 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, obstacles to be detected within a factory include small obstacles such as steps and wiring, and a high-resolution sensor is required to detect them.

[0007] The present invention has been made in consideration of this point, and has as its object to provide a work machine and environment recognition system that utilizes obstacle information acquired and accumulated by existing work machines, for work machines operated by operators who are unfamiliar with the work site, or for work machines that have been newly introduced to the site. [Means for solving the problem]

[0008] In order to solve the above problems, for example, the configurations described in the claims are adopted. The present invention includes a plurality of means for solving the above problems. To give an example of such means, A work machine traveling within a work area, A sensor that observes the surrounding environment of the work machine; a self-location estimation unit that estimates a self-location; a normal operation recording unit that records normal operations of the work machine; an abnormal operation detection unit that detects abnormal operation of the work machine by comparing the normal operation recorded in the normal operation recording unit; a sensor data recording unit that identifies an abnormal operation position where the work machine has detected an abnormal operation based on the history of the self-position, and records an observation result of the sensor at the abnormal operation position where the abnormal operation has been detected; an obstacle recognition unit that generates a three-dimensional shape of the surrounding environment of the work machine using the abnormal operation position and the observation result of the sensor at the abnormal operation position, and the abnormal operation position detected in another work machine and the observation result of the sensor at the abnormal operation position, and recognizes obstacles within the work area from the three-dimensional shape; has. Effect of the Invention

[0009] According to the present invention, it is possible to provide obstacle information acquired and accumulated by existing work machines to work machines operated by operators who are unfamiliar with the work site where the work will be performed, and to work machines newly introduced to the site. Further features related to the present invention will become apparent from the description of this specification and the attached drawings. Furthermore, problems, configurations and effects other than those described above will become apparent from the description of the embodiments below. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of a work machine according to the present invention. [Diagram 2] FIG. 1 is a diagram showing an example of a factory in which a plurality of work machines are operated. [Diagram 3] FIG. 1 is a diagram showing an example of a system configuration of an environment recognition system according to a first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a flow of accumulation and sharing of sensor data of the surrounding environment of the work machine in the first embodiment. [Diagram 5] FIG. 4 is a diagram showing an example of a processing flow for obstacle detection of the work machine according to the first embodiment. [Figure 6] FIG. 4 is a diagram showing an example of sensor data acquisition when the work machine of the first embodiment makes a sharp turn. [Figure 7] FIG. 4 is a diagram showing an example of sensor data accumulation when the work machine of the first embodiment makes a sharp turn. [Figure 8] FIG. 4 is a diagram showing a modified example of the system configuration of the work machine in the first embodiment. [Figure 9] FIG. 4 is a diagram showing a modified example of the system configuration of the work machine in the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] [Example 1] First, the overall configuration of the environment recognition system and each of its components will be described. FIG. 1 is a diagram showing an example of a work machine of the present invention, FIG. 2 is a diagram showing an example of a work site where a plurality of work machines are operating, and FIG. 3 is a diagram showing an example of a system configuration of an environment recognition system in Example 1.

[0013] In this embodiment, as shown in Fig. 2, a case will be described as an example in which a plurality of work machines 100 travel and perform work within a predetermined work area 10, such as a factory or warehouse. The plurality of work machines 100 include manned forklifts operated by an operator and unmanned transport devices that are automatically operated based on a program, as shown in Fig. 1, for transporting cargo 11. In Fig. 2, work machines 100A to 100C are work machines that have been used conventionally, and work machine 100N is a work machine that will be newly introduced into work area 10.

[0014] As shown in Fig. 3, the environment recognition system of this embodiment has a plurality of work machines 100 and a sensor data server 150 which transmits and receives information via wireless communication between the plurality of work machines 100. The work machine 100 is equipped with a plurality of microcomputers. The work machine 100 is equipped with a work machine control unit 110, a position / posture / state estimation unit 120, an environment recognition unit 130, and a notification unit 140 as internal functions of the microcomputer.

[0015] The work machine control unit 110 has a work machine control calculation unit 111, a braking / driving control unit 113, a steering control unit 115, a work operation control unit 117, a braking / driving device 112, a steering device 114, and a work device 116. The work machine control calculation unit 111 calculates control amounts based on braking / driving instructions, steering instructions, and work operation instructions such as lifting for the work machine 100 from the operator, and the braking / driving control unit 113, steering control unit 115, and work operation control unit 117 control the braking / driving device 112, steering device 114, and work device 116 of the work machine, respectively, based on the control amounts calculated by the work machine control calculation unit 111.

[0016] The position / attitude / state estimation unit 120 calculates the position / attitude, angular velocity, and acceleration of the work machine 100. For example, this is a system that estimates the position / attitude of the work machine 100 by odometry based on the amount of wheel rotation and steering amount acquired from the work machine control unit 110, or by scan matching between the surrounding environment observation data acquired by the surrounding environment observation unit 131 and a surrounding environment map defined in an external coordinate system, and measures, using a gyroscope and an accelerometer installed on the work machine, an impact caused by the work machine 100 coming into contact with surrounding objects, pitching caused by going over a step, and the like.

[0017] The environment recognition unit 130 includes a surrounding environment observation unit 131 , a sensor data recording unit 132 , a communication unit 133 , an abnormal operation detection unit 134 , a normal operation recording unit 135 , a 3D shape generation unit 136 , and an obstacle recognition unit 137 .

[0018] The surrounding environment observing unit 131 is installed in front of the work machine 100 and is a sensor that acquires the position, shape, color, reflection intensity, etc. of objects around the work machine 100, and is composed of a camera and laser radar. The surrounding environment observing unit 131 is attached to each work machine 100 at a different height position so that the observation accuracy differs for each work machine 100. Furthermore, the surrounding environment observing unit 131 attached to a manned work machine 100 has a narrower observation range and higher observation accuracy than the surrounding environment observing unit 131 attached to an unmanned work machine 100.

[0019] When abnormal operation is detected by the abnormal operation detection unit 134, the sensor data recording unit 132 records the sensor data acquired by the surrounding environment observation unit 131. The sensor data recording unit 132 identifies the abnormal operation position where the work machine 100 of its own detected abnormal operation based on the position history of the work machine 100 of its own, and records the observation results of the surrounding environment observation unit 131 at the abnormal operation position where the abnormal operation was detected.

[0020] The communication unit 133 transmits the sensor data recorded and accumulated in the sensor data recording unit 132 to the sensor data server 150. The sensor data transmitted to the sensor data server 150 is shared with other work machines 100. The communication unit 133 transmits the abnormal operation position detected by the abnormal operation detection unit 134 and the observation results of the sensor at the abnormal operation position to other work machines, and receives the abnormal operation position detected in the other work machines 100 within the work area 10 and the observation results of the sensor at the abnormal operation position from the other work machines 100.

[0021] The abnormal operation detection unit 134 compares the changes in position, attitude, and state and the control amount acquired from the position / attitude / state estimation unit 120 and the work machine control unit 110 with those during normal operation recorded in the normal operation recording unit 135, and detects abnormal operation of its own work machine 100. The abnormal operation detection unit 134 judges whether or not an operation is abnormal by comparing it with threshold values ​​of the position, attitude, and state during normal operation recorded in the normal operation recording unit 135.

[0022] The abnormal operation detection unit 134 determines that abnormal operation has occurred, for example, when the route along which the work machine 100 travels in normal operation is set as a designated route, and the work machine 100 travels at a position deviating from the designated route, or when the work machine 100 travels backwards while traveling along the designated route. Also, when the actual acceleration or actual angular velocity of the work machine 100 deviates from the reference acceleration or reference angular velocity during normal operation of the work machine 100 by a threshold value or more, it determines that abnormal operation has occurred.

[0023] The normal operation recording unit 135 records the normal operation of the work machine, and updates the thresholds for the position, attitude, and state during normal operation based on the stored data.

[0024] The 3D shape generator 136 generates a three-dimensional shape of the surrounding environment including obstacles based on the accumulated sensor observation results (sensor data). Specifically, the 3D shape generator 136 generates a three-dimensional shape of the surrounding environment of the work machine using the abnormal operation position and the sensor observation results at the abnormal operation position, and the abnormal operation position detected in another work machine and the sensor observation results at the abnormal operation position.

[0025] The obstacle recognition unit 137 recognizes obstacles that may hinder the operation of the work machine 100 from the three-dimensional shape of the surrounding environment generated by the 3D shape generation unit 136. Information about the obstacles recognized by the obstacle recognition unit 137 is output to the notification unit 140.

[0026] The notification unit 140 has a notification management unit 141, a speaker 142, and a display 143. The notification management unit 141 performs control such as generating a warning sound from the speaker 142 and displaying an obstacle on the display 143 based on the recognition result acquired from the environment recognition unit 130, and notifies the operator of the presence of an obstacle.

[0027] The sensor data server 150 is composed of hardware and software such as a processor and a memory, and has a communication unit 151, a sensor data integration processing unit 152, and a sensor data integration recording unit 153 as internal functions. The sensor data server 150 acquires the abnormal operation position and the sensor observation results at the abnormal operation position recorded in the sensor data recording unit 132 of each work machine 100 from the communication unit 151, and registers them in the sensor data integration recording unit 153 by the sensor data integration processing unit 152. Furthermore, in response to a request from the work machine 100, the abnormal operation position and the sensor observation results at the abnormal operation position registered in the sensor data integration recording unit 153 are provided to the work machine 100 that issued the request from the communication unit 151. The sensor data integration processing unit 152 then updates data, deletes old data, and removes outliers by statistical processing.

[0028] Next, the operation of the environment recognition unit 130 will be described. FIG. 4 is a flowchart showing an example of a processing procedure executed by the environment recognition unit 130, FIG. 6 is a diagram showing an example of sensor data acquisition when the work machine of the first embodiment makes a sharp turn, and FIG. 7 is a diagram showing an example of sensor data accumulation when the work machine of the first embodiment makes a sharp turn.

[0029] In steps S101 to S106, sensor data during abnormal operation is accumulated and shared. This process is executed at each measurement cycle of the surrounding environment observing unit 131.

[0030] In step S101, the position, attitude, and state (acceleration, angular velocity, etc.) and control variables (braking / driving, steering, work operation) are acquired from the position / attitude / state estimation unit 120 and the work machine control unit 110.

[0031] In step S102, the abnormal operation detection unit 134 compares the position, posture, state, and control amount with data from the normal operation recording unit 135 to determine whether or not the operation is abnormal. Abnormal operations include abnormal operations by the operator such as sudden braking and sharp turns, deviation from a designated route in the factory, and retreating midway through the transport route. Machine learning may also be used to extract abnormal operations that the operator is not clearly aware of.

[0032] By observing the state quantity rather than the control quantity, significant changes in acceleration and angular velocity can be captured, and abnormal operation of the work machine 100 such as contact with surrounding objects or climbing over small steps can be detected.

[0033] 6 shows a state in which an obstacle 12 is detected within the detection range 22 of the surrounding environment observation unit 131 while the work machine 100 is traveling, and the operator of the work machine 100 takes a sharp turn trajectory 21 to avoid the obstacle 12. Here, in the case of a sharp turn as shown in FIG. 6, the area in which the obstacle exists may be outside the detection range of the sensor (blind spot) at the time of abnormal operation detection. In particular, manned work machines use sensors with narrower detection ranges than unmanned work machines, and therefore the obstacle is more likely to go outside the detection range when making a sharp turn.

[0034] Therefore, as shown in FIG. 7, it is advisable to accumulate sensor data from the time (Time_A) when the position of the abnormal operation detection time (Time_B) falls within the detection range 22 of the sensor to the time when normal operation is restored (Time_C).

[0035] If the operation is not determined to be abnormal but is assumed to be normal in step S103, it is recorded as a normal operation in normal operation recording section 135 (step S104).

[0036] On the other hand, if it is determined in step S103 that the operation is abnormal, the sensor data acquired by the surrounding environment observing unit 131 is recorded in the sensor data recording unit 132 (step S105).

[0037] Next, in step S106, the data in the sensor data recording unit 132 is sent to the sensor data server 150 via the communication unit 133, and shared with the other work machines 100.

[0038] FIG. 5 is a diagram illustrating an example of a processing flow of obstacle detection by the work machine according to the first embodiment. In steps S107 to S110, processing is performed for obstacle detection by the work machine 100. This processing is executed when abnormal operation is detected in step S103 or when sensor data of the surrounding environment is updated by another work machine 100 in step S106.

[0039] In step S107, a three-dimensional shape of the surrounding environment is generated from the sensor data around the work machine 100 recorded in the sensor data recording unit 132. In step S108, obstacles are recognized from the three-dimensional shape of the surrounding environment by the obstacle recognition unit 137. A publicly known method can be used to recognize obstacles from the three-dimensional shape of the surrounding environment.

[0040] In order to obtain a highly accurate 3D shape of the surrounding environment, it is desirable to accumulate sensor data from as many different sensor positions as possible. For this reason, in this embodiment, the sensors are arranged in different positions, particularly in the height direction, in each work machine 100. Therefore, even if each work machine 100 travels along the same route, sensor data can be obtained from different sensor positions, and a highly accurate 3D shape of the surrounding environment can be obtained in a short period of time.

[0041] When it is determined in step S109 that an obstacle has been detected, the notification unit 140 notifies the operator of the obstacle along with its position and shape (step S110). When the work machine 100 is in a state in which it is about to pass near an obstacle, the notification unit 140 notifies the operator.

[0042] According to the work machine 100 of this embodiment, the abnormal operation position detected in its own work machine and the sensor observation results at the abnormal operation position, and the abnormal operation position detected in another work machine and the sensor observation results at the abnormal operation position are used to recognize obstacles for its own work machine, thereby greatly improving the obstacle detection performance of the work machine 100. Furthermore, obstacle information acquired and accumulated by existing work machines 100 can be supplied to work machines 100 operated by operators who are unfamiliar with the work site or to work machines 100 that have been newly introduced to the site.

[0043] In the above-described embodiment, a configuration of an environment recognition system in which information is sent and received between a plurality of work machines 100 via the sensor data server 150 has been described. However, as a variation of this embodiment, it is also possible to omit the sensor data server 150 and configure the plurality of work machines 100 to exchange data directly between them.

[0044] FIG. 8 is a diagram showing the configuration of a work machine 100 that directly exchanges data with other work machines 100. As shown in FIG.

[0045] When a request is received from another work machine 100, the communication unit 133 directly transmits the abnormal operation position detected by its own work machine 100 and the sensor observation results at the abnormal operation position. Also, it is possible to request and receive from the other work machines 100 the abnormal operation position detected in the other work machines 100 within the work area 10 and the sensor observation results at the abnormal operation position.

[0046] In this modification, a sensor data integration processing unit 138 is provided between the sensor data recording unit 132 and the communication unit 133 . When an abnormal operation is detected by the abnormal operation detection unit 134, the sensor data recording unit 132 records the sensor data acquired by the surrounding environment observation unit 131, and the sensor data accumulated there is shared with other work machines by the communication unit 133 through the sensor data integration processing unit 138. Specifically, the sensor data integration processing unit 138 records information on the abnormal operation position and the sensor observation results at the abnormal operation position received by the communication unit 133 from the other work machines 100. In addition, the abnormal operation position and the sensor observation results at the abnormal operation position registered in the sensor data recording unit 132 of the own work machine 100 are provided to the other work machines 100 from the communication unit 133. Then, the sensor data integration processing unit 138 updates the data, deletes old data, and removes outliers by statistical processing.

[0047] The obstacle recognition unit 137 recognizes obstacles within the work area 10 using the abnormal operation position detected by its own work machine 100 and the sensor observation results at the abnormal operation position, and the abnormal operation position detected in another work machine 100 and directly received by the communication unit 133 and the sensor observation results at the abnormal operation position. Therefore, the obstacle detection performance of its own work machine 100 can be greatly improved.

[0048] Furthermore, in each of the above-mentioned embodiments, a description has been given of a case in which there are a plurality of work machines 100. However, as a modification of the present embodiments, the present embodiments can also be applied to a single work machine 100.

[0049] FIG. 9 is a diagram showing the configuration of a system made up of one work machine. The work machine 100 has a sensor data management unit 139 instead of the communication unit 133 in the above-described embodiment. When an abnormal operation is detected by the abnormal operation detection unit 134, the sensor data recording unit 132 records the sensor data acquired by the surrounding environment observation unit 131, and the sensor data accumulated therein is managed by the sensor data management unit 139. The sensor data management unit 139 manages the abnormal operation position recorded in the sensor data recording unit 132 and the sensor observation results at the abnormal operation position. The sensor data management unit 139 updates data, deletes old data, and removes outliers by statistical processing.

[0050] The obstacle recognition unit 137 recognizes obstacles within the work area 10 using abnormal operation positions detected by its own work machine 100 and stored in the sensor data recording unit 132, and the sensor observation results at the abnormal operation positions. Therefore, the obstacle detection performance of its own work machine 100 can be greatly improved.

[0051] Although the embodiments of the present invention have been described above in detail, the present invention is not limited to the above-described embodiments, and various design changes can be made without departing from the spirit of the present invention described in the claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. [Explanation of symbols]

[0052] 100...work machine, 110...work machine control unit, 111...work machine control calculation unit, 112...braking / driving device, 113...braking / driving control unit, 114...steering device, 115...steering control unit, 116...work device, 117...work operation control unit, 120...position / posture / state estimation unit, 130...environment recognition unit, 131...surrounding environment observation unit (sensor), 132...sensor data recording unit, 133...communication unit, 134...abnormal operation detection unit, 135...normal operation recording unit, 136...3D shape generation unit, 137...obstacle recognition unit, 140...alarm unit, 141...alarm management unit, 142...speaker, 143...display

Claims

[Claim 1] An environment recognition system comprising a plurality of work machines traveling within a work area and a server that transmits and receives information between the plurality of work machines, The plurality of work machines include A sensor that observes the surrounding environment of the work machine; A position / attitude / state estimation unit that estimates a self-position and an angular velocity; a normal operation recording unit that records normal operations of the work machine; an abnormal operation detection unit that detects abnormal operation of the work machine when an actual angular velocity of the work machine estimated by the position / posture / state estimation unit deviates by a threshold value or more from a reference angular velocity in normal operation recorded in the normal operation recording unit; a sensor data recording unit that identifies an abnormal operation position where the abnormal operation detection unit detects an abnormal operation based on the self-location history and records an observation result of the sensor at the abnormal operation position where the abnormal operation is detected; a work machine communication unit that transmits the abnormal operation position and the observation results of the sensor at the abnormal operation position to the server, and receives the abnormal operation position detected in the work machine itself or another work machine within the work area and the observation results of the sensor at the abnormal operation position from the server; an obstacle recognition unit that generates a three-dimensional shape of the surrounding environment of the work machine using the abnormal operation position and the observation result of the sensor at the abnormal operation position, and the abnormal operation position detected in the other work machine and the observation result of the sensor at the abnormal operation position, and recognizes an obstacle from the three-dimensional shape; The server, An integrated record section; A server communication unit, the integrated recording unit records the abnormal operation positions detected in the own work machine or the other work machines in the work area and the observation results of the sensors at the abnormal operation positions, which are received from the work machine by the server communication unit; the server communication unit provides the abnormal operation position recorded in the integrated recording unit and the observation results of the sensor at the abnormal operation position to the work machine; The sensors of the plurality of work machines are attached at positions in a height direction different from each other, The plurality of work machines include manned work machines operated by an operator and unmanned work machines that are automatically operated based on a program, An environment recognition system for a work machine, wherein the sensor attached to the manned work machine has a narrower observation range than the sensor attached to the unmanned work machine.

Images