ASSISTANCE DEVICE AND SYSTEM INCLUDING THE ASSISTANCE DEVICE

Abstract

To improve visibility of images captured by an imaging device mounted on an unmanned air vehicle.SOLUTION: A support device supports a work machine based on image information from an imaging device mounted on an unmanned air vehicle and includes: a state information acquisition part which acquires state information indicating a state of the work machine; a determination part which determines a target position of the unmanned air vehicle and a moving mode of the unmanned air vehicle to the target position; and a control unit which controls movement of the unmanned air vehicle to the target position based on the moving mode determined by the determination part. The determination part determines the moving mode based on the state information so as to avoid interference with the work machine.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an assistive device and a system including the assistive device. [Background technology]

[0002] There is known a technique for using an imaging device attached to an autonomous flying vehicle to capture images of a space that cannot be captured by an imaging device attached to the upper rotating body of a shovel (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. WO2017 / 131194 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology described above, the position of the aircraft is constantly changed to follow the movement of the shovel, which can easily reduce the visibility of the images captured by the imaging device due to the shaking of the aircraft (vibrations caused by acceleration, etc.).

[0005] Therefore, in one aspect, the present invention aims to improve the visibility of images captured by an imaging device mounted on an unmanned aerial vehicle. [Means for solving the problem]

[0006] In one aspect, the following solution is provided.

[0007] (1) A support device that supports a work machine based on image information from an imaging device mounted on an unmanned aerial vehicle, a status information acquisition unit that acquires status information representing the status of the work machine; A determination unit that determines a target position of the unmanned aerial vehicle and a movement mode of the unmanned aerial vehicle to the target position; a control unit that controls movement of the unmanned aerial vehicle to the target position based on the movement mode determined by the determination unit, The determination unit determines the movement mode based on the state information so as to avoid interference with the work machine.

[0008] (2) In the configuration of (1) above, the work machine includes a lower traveling body, an upper rotating body mounted on the lower traveling body, and a work mechanism provided on the upper rotating body, The status information includes attitude information that indicates the attitude of at least one of the work machine and the work mechanism.

[0009] (3) In the configuration of (2) above, a calculation unit is further provided that calculates, based on the status information, a value of a parameter that indicates the possibility of interference between the unmanned aerial vehicle and the work machine when the unmanned aerial vehicle moves in a straight line from its current position to the target position, The determining unit determines the movement mode based on the value of the parameter.

[0010] (4) In the configuration of (2) or (3) above, the determining unit determines the target position so that the imaging device can capture an image of the working mechanism.

[0011] (5) In the configuration of (2) or (3) above, the determining unit determines the target position so that the imaging device can capture an image of a part of the working mechanism or the work target object.

[0012] (6) In the configuration of (4) or (5) above, the target position includes a left position where the imaging device can capture an image of the working mechanism from the left side of the working mechanism, and a right position where the imaging device can capture an image of the working mechanism from the right side of the working mechanism, The determination unit determines the movement mode when the target position changes between the left position and the right position.

[0013] (7) In any of the configurations (1) to (6) above, when the target position changes, the determination unit is characterized in that it determines, as the movement mode, one selected from a first movement mode in which the target position moves linearly from the original target position to the new target position, and a second movement mode in which the target position moves from the original target position to the new target position via a waypoint.

[0014] (8) In the configuration of (7) above, the second movement mode is characterized by including linear movement from the original target position to the waypoint and linear movement from the waypoint to the new target position.

[0015] (9) In the configuration of (7) or (8) above, the determination unit is characterized in that, when the second movement mode is selected, the determination unit determines the waypoint based on the state information.

[0016] (10) In the configuration of (9) above, the determination unit is characterized in that it determines the intermediate point so that the shortest distance between the unmanned aerial vehicle and the work machine when moving according to the second movement mode is longer than the shortest distance when moving according to the first movement mode.

[0017] (11) In any of the configurations (1) to (10) above, a surrounding environment information acquisition unit is further provided that acquires surrounding environment information that represents the surrounding environment of the work machine, The determining unit determines the movement mode based on the state information and the surrounding environment information.

[0018] (12) In any of the configurations (1) to (11) above, the device is characterized by further comprising an information output unit that outputs predetermined information when the determination unit cannot determine the movement mode.

[0019] (13) A system including any one of the support devices (1) to (12) above and at least one of the work machine and the unmanned aerial vehicle. [Effects of the Invention]

[0020] According to the present disclosure, it is possible to improve the visibility of images captured by an imaging device mounted on an unmanned aerial vehicle. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a diagram illustrating a configuration of a system including a support device according to an embodiment of the present invention. [Figure 2] 1 is a diagram showing the functional configuration of a support device according to an embodiment of the present invention; [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration related to a control system of a work machine. [Figure 4] 4 is a flowchart showing the operation of the flight control device. [Figure 5] 10A and 10B are top views of the left and right positions. [Figure 5A] FIG. 1 illustrates a flight path through two waypoints. [Figure 5B] FIG. 1 illustrates a flight path via one waypoint. [Figure 5C] FIG. 10 is a diagram showing an example of a flight path set at the start of work. [Figure 6] A diagram illustrating a flight path when a cable is connected to an unmanned aerial vehicle. [Figure 6A] A diagram illustrating a flight path when a cable is connected to an unmanned aerial vehicle. DETAILED DESCRIPTION OF THE INVENTION

[0022] Each embodiment will be described in detail below with reference to the accompanying drawings.

[0023] FIG. 1 is a diagram illustrating the configuration of a system including a support device of this embodiment, and FIG. 2 is a diagram illustrating the functional configuration of the support device of this embodiment.

[0024] As shown in FIG. 1, a system including the assistance device of this embodiment is configured to include, for example, a flight control device 1, an unmanned aerial vehicle 2, and a work machine 3. In FIG. 1, the flight control device 1 and the unmanned aerial vehicle 2, and the flight control device 1 and the work machine 3 are connected via a network. In this case, the flight control device 1 can be configured, for example, by a server (server computer). In this case, the network configuration is arbitrary, but the network may include a wireless communication network, the Internet, a VPN (Virtual Private Network), a WAN (Wide Area Network), a wired network, or any combination of these.

[0025] The support device of this embodiment is a device that supports work on the work machine 3 based on image information from an imaging device 22 mounted on the unmanned aerial vehicle 2. In this embodiment, the functions of the support device are provided in the flight control device 1. However, all of the functions of the support device may be provided in either the unmanned aerial vehicle 2 or the work machine 3. The functions of the support device may also be provided separately in any two or three of the flight control device 1, unmanned aerial vehicle 2, and work machine 3.

[0026] Furthermore, the work machine 3 may be equipped with a device that has some or all of the functions of the support device.

[0027] As shown in Figure 2, the flight control device 1 constituting the assistance device of this embodiment comprises a status information acquisition unit 11 that acquires status information representing the status of the work machine 3, a determination unit 12 that determines the target position of the unmanned aerial vehicle 2 and the movement mode of the unmanned aerial vehicle 2 to this target position, a control unit 13 that controls the movement of the unmanned aerial vehicle 2 to the target position based on the movement mode determined by the determination unit 12, a calculation unit 14 that calculates the value of a parameter that represents the possibility of interference between the unmanned aerial vehicle 2 and the work machine 3 when the unmanned aerial vehicle 2 moves in a straight line from its current position to the target position based on the status information acquired by the status information acquisition unit 11, a surrounding environment information acquisition unit 15 that acquires surrounding environment information representing the surrounding environment of the work machine 3, and an information output unit 16 that outputs specified information when the determination unit 12 cannot determine the above-mentioned movement mode.

[0028] The unmanned aerial vehicle 2 is, for example, a rotorcraft, and in this case, is equipped with a plurality of rotatable blades, an electric motor (actuator) for rotating the plurality of blades, a battery for supplying power to the electric motor, etc. Note that instead of or in addition to such a battery, a power supply line may be connected to the unmanned aerial vehicle 2 from the ground.

[0029] The unmanned aerial vehicle 2 also includes a control device 21 and an imaging device 22.

[0030] The control device 21 controls the flight state (forward, backward, ascending, descending, hovering, etc.) of the unmanned aerial vehicle 2 in accordance with control information from the flight control device 1, and guides the unmanned aerial vehicle 2 to a target position. The target position is expressed, for example, by latitude, longitude, and altitude. The control device 21 also controls the attitude of the unmanned aerial vehicle 2 at the target position.

[0031] The control device 21 acquires aircraft information that indicates various conditions related to the airframe of the unmanned aerial vehicle 2, and controls the flight state of the unmanned aerial vehicle 2 based on this aircraft information. The aircraft information includes position information of the unmanned aerial vehicle 2 and attitude information of the unmanned aerial vehicle 2, for example. The position information of the unmanned aerial vehicle 2 is expressed, for example, by latitude, longitude, and altitude. Such position information of the unmanned aerial vehicle 2 can be acquired from a GPS sensor. The attitude information of the unmanned aerial vehicle 2 includes, for example, information regarding rotation around the yaw axis, roll axis, and pitch axis of the unmanned aerial vehicle 2. Such attitude information of the unmanned aerial vehicle 2 can be acquired from a sensor such as an inertial measurement unit (IMU) mounted on the unmanned aerial vehicle 2.

[0032] The control device 21 also has a transmitting function for transmitting images acquired by the imaging device 22 to the work machine 3 or the flight control device 1 via a transmitting unit (not shown), and a receiving function for receiving control information transmitted from the flight control device 1 via a receiving unit (not shown).

[0033] The imaging device 22 is configured to include a camera attached to the unmanned aerial vehicle 2. The type of camera is arbitrary, and may be, for example, a wide-angle camera. The imaging device 22 acquires images of the forward environment in front of the unmanned aerial vehicle 2 using an imaging element such as a CCD (charge-coupled device) or a CMOS (complementary metal oxide semiconductor). The imaging device 22 may, for example, acquire images of the forward environment in real time and supply them to the control device 21 in a stream format at a predetermined frame rate.

[0034] The imaging device 22 preferably includes a gimbal (not shown). The gimbal functions to keep the optical axis of the imaging device 22 in a constant direction (for example, a predetermined direction in a horizontal plane) even if the attitude of the unmanned aerial vehicle 2 changes.

[0035] The work machine 3 carries out predetermined work in cooperation with the unmanned aerial vehicle 2. The work machine 3 is, for example, a construction machine equipped with a crusher suitable for demolition work, and includes a crawler-type lower track body 31 and an upper rotating body 32 that is rotatably mounted on the lower track body 31. A cab (operator's compartment) 34 is provided on the front left side of the upper rotating body 32. A work mechanism 35 is provided in the front center of the upper rotating body 32. The work machine 3 may be in the form of a construction machine equipped with a crane function, or in another form such as a crawler crane.

[0036] The working mechanism 35 includes a boom 35a that is mounted on the upper rotating body 32 so that it can be raised and lowered, an arm 35b that is rotatably connected to the tip of the boom 35a, and a crusher 35c that is attached to the tip of the arm 35b. As is well known, the boom 35a, the arm 35b, and the crusher 35c are driven by hydraulic cylinders or the like that are provided as part of the working mechanism 35.

[0037] The work machine 3 may also be provided with a basement from which the unmanned aerial vehicle 2 takes off and lands.

[0038] FIG. 3 is a diagram illustrating an example of a hardware configuration related to a control system of a work machine.

[0039] As shown in FIG. 3, the work machine 3 includes a control device 40 and peripheral equipment 50.

[0040] The control device 40 includes a CPU (Central Processing Unit) 41, a RAM (Random Access Memory) 42, a ROM (Read Only Memory) 43, an auxiliary storage device 44, a drive device 45, and a communication interface 47, all connected by a bus B, as well as a wired transceiver unit 48 and a wireless transceiver unit 49 connected to the communication interface 47.

[0041] The auxiliary storage device 44 is, for example, a hard disk drive (HDD) or a solid state drive (SSD), and is a storage device that stores data related to application software and the like.

[0042] The wired transceiver 48 includes a transceiver capable of communicating over a wired network. Peripheral devices 50 are connected to the wired transceiver 48. However, some or all of the peripheral devices 50 may be connected to the bus B or may be connected to the wireless transceiver 49.

[0043] The wireless transceiver 49 is a transceiver capable of communicating using a wireless network. The wireless network may include a wireless communication network for mobile phones, the Internet, VPN, WAN, etc. The wireless transceiver 49 may also include a near field communication (NFC) unit, a Bluetooth (registered trademark) communication unit, a Wi-Fi (Wireless-Fidelity (registered trademark)) transceiver, an infrared transceiver, etc. The wireless transceiver 49 can communicate with the flight control device 1 in the form of a server. The wireless transceiver 49 also receives images acquired by the imaging device 22 and transmitted by the control device 21.

[0044] The peripheral equipment 50 includes electronically controllable devices, various sensors, an operation unit, etc. that are mounted on the work machine 3. The peripheral equipment 50 may include, for example, an image output device 51, a buzzer, an audio output device (not shown), a hydraulic pressure generating device (not shown) that activates the work mechanism 35, various sensors 52 that detect the operating state of various operating members, and an operation unit 53 that accepts operations. The hydraulic pressure generating device may be a hydraulic pump driven by an engine and / or an electric motor. When a hydraulic pump driven by an electric motor is used, the hydraulic pressure generating device may include an inverter for driving the electric motor.

[0045] The various sensors 52 include a gyro sensor, a GPS (Global Positioning System) sensor, various angle sensors, and an acceleration sensor (tilt sensor). The GPS sensor acquires position information of the work machine 3. The position information of the work machine 3 is expressed as latitude, longitude, and altitude. The GPS sensor includes a GPS receiver, and calculates latitude, longitude, and altitude by interferometric positioning or the like based on radio waves from satellites.

[0046] The various sensors 52 also include various sensors that acquire parameters related to the attitude of the work machine 3, and acquire these parameters as attitude information of the work machine 3. In this case, the various sensors that acquire the parameters related to the attitude may be, for example, a boom angle sensor, an arm angle sensor, a bucket angle sensor, a machine body inclination sensor, etc. The boom angle sensor is a sensor that acquires the boom angle, and includes, for example, a rotation angle sensor that detects the rotation angle of the boom foot pin, a stroke sensor that detects the stroke amount of the boom cylinder, and a tilt (acceleration) sensor that detects the tilt angle of the boom 35a. The same applies to the arm angle sensor and the bucket angle sensor. The machine body inclination sensor is a sensor that acquires the machine body inclination angle, and detects, for example, the inclination angle of the upper rotating body 32 with respect to the horizontal plane.

[0047] The various sensors 52 also include various sensors that acquire parameters related to the rotation angle of the upper rotating body 32 relative to the lower traveling body 31 as attitude information of the work machine 3, and these sensors acquire rotation angle information that indicates the rotation angle of the upper rotating body 32. In this case, the various sensors that acquire the parameters related to the rotation angle of the upper rotating body 32 may be, for example, a geomagnetic sensor, a rotation angle sensor (e.g., a resolver, etc.) that detects the rotation angle about the rotation axis of a rotation mechanism that rotates the upper rotating body 32 relative to the lower traveling body 31, a gyro sensor, etc.

[0048] The posture information includes fixed (known) parameters of the work machine 3 other than the parameters obtained by the various sensors 52, such as the boom length, boom foot position, jib length, and arm length, as well as parameters that change during work, such as the boom angle, jib angle, and arm angle, which are obtained by the various sensors 52.

[0049] The position information of the work machine 3 and the attitude information of the work machine 3 correspond to the status information of the work machine 3.

[0050] The image output device 51 is provided inside the cab 34 so that it can be seen by the operator of the work machine 3. The image output device 51 displays images acquired by the imaging device 22 and received by the wireless transceiver unit 49. This allows the operator of the work machine 3 to understand, for example, the situation of the work site that cannot be seen with a direct view, from the forward environmental image on the image output device 51.

[0051] The configuration of the image output device 51 is arbitrary, and may be, for example, a liquid crystal display, an organic EL (Electro-Luminescence) display, etc. In a modified example, the image output device 51 may be a portable device (for example, a tablet terminal, etc.) that can be brought into the cab 34 by the operator of the work machine 3.

[0052] Furthermore, the image output from the image output device 51 may be displayed on another display device instead of on a display device provided in the image output device 51. In this case, the output destination of the image output device 51 is arbitrary, and may be, for example, a display device such as a cluster provided in the cab of the work machine, a terminal device such as a tablet carried by the operator, or a management screen managed by the site supervisor.

[0053] The operation unit 53 is provided inside the cab 34 and receives instructions from the operator of the work machine 3. These instructions can include instructions to the flight control device 1, and the instructions to the flight control device 1 are transmitted to the flight control device 1 via the wireless transceiver unit 49.

[0054] The control device 40 may be connectable to a recording medium 46. The recording medium 46 stores a predetermined program. The program stored in the recording medium 46 is installed in the auxiliary storage device 44 of the control device 40 via the drive device 45. The installed predetermined program can be executed by the CPU 41 of the control device 40. For example, the recording medium 46 may be a recording medium that records information optically, electrically, or magnetically, such as a CD (Compact Disc)-ROM, a flexible disk, or a magneto-optical disk, or a semiconductor memory that records information electrically, such as a ROM or a flash memory. The recording medium 46 does not include a carrier wave.

[0055] FIG. 4 is a flowchart showing the operation of the flight control device.

[0056] In step S102 of Figure 4, the flight control device 1 determines whether a new target position for the unmanned aerial vehicle 2 has been set, and waits for the determination to be affirmative before proceeding to step S104. Here, the determination in step S102 is affirmative when, for example, an instruction operation to the operation unit 53 is received that instructs the user to change the target position.

[0057] Setting a new target position includes setting a new target position when the unmanned aerial vehicle 2 takes off. Setting a new target position also includes setting a new target position when the target position is changed while photographing is being performed by the unmanned aerial vehicle 2. Setting the landing point as the target position when the unmanned aerial vehicle 2 lands also includes setting the new target position.

[0058] In step S104, the flight control device 1, as a function of the status information acquisition unit 11, acquires status information that represents the status of the work machine 3. As described above, the status information of the work machine 3 corresponds to the position information of the work machine 3 and the attitude information of the work machine 3.

[0059] In step S106, the flight control device 1 acquires the current position of the unmanned aerial vehicle 2 as a function of the calculation unit 14. This current position corresponds to, for example, the initial position of the unmanned aerial vehicle 2 before the unmanned aerial vehicle 2 starts taking pictures, or the target position before the change if the target position is changed while the unmanned aerial vehicle 2 is taking pictures. Alternatively, this current position corresponds to the final shooting position (target position) when shooting by the unmanned aerial vehicle 2 is terminated. The current position may be determined based on aircraft information transmitted from the control device 21.

[0060] In step S108, the flight control device 1, as a function of the calculation unit 14, calculates the value of a parameter that represents the possibility of interference between the unmanned aerial vehicle 2 and the work machine 3 when the unmanned aerial vehicle 2 moves in a straight line from its current position to a new target position, based on the status information of the work machine 3 acquired in step S104. Here, for example, the parameter that can be calculated is the shortest distance between the path that the unmanned aerial vehicle 2 takes when it moves in a straight line from its current position to a new target position and the work mechanism 35 (the distance between the path and the work mechanism 35 when the unmanned aerial vehicle 2 is closest to the work mechanism 35).

[0061] Here, the position (attitude) of the work mechanism 35 is calculated based on the status information of the work machine 3, i.e., the position information of the work machine 3 and the attitude information of the work machine 3. Therefore, for example, the position (attitude) of the work mechanism 35 can be determined in association with the latitude, longitude, and altitude, just like the flight path, without relying on image processing of images captured by the imaging device 22. This reduces the processing load for calculating the above parameters.

[0062] In step S110, the flight control device 1, as a function of the calculation unit 14, determines whether the parameter value (shortest distance) calculated in step S108 is greater than or equal to a predetermined threshold value, and if the determination is positive, proceeds to step S112, and if the determination is negative, proceeds to step S114.

[0063] In step S112, the flight control device 1, as a function of the determination unit 12, selects the path along which the unmanned aerial vehicle 2 moves in a straight line from the current position to the new target position (the path in the first movement mode) as a tentative flight path, and proceeds to step S116.

[0064] In step S114, the flight control device 1, as a function of the determination unit 12, searches for and extracts a route along which the unmanned aerial vehicle 2 will move from its current position to a new target position based on the status information acquired in step S104, and which avoids interference between the unmanned aerial vehicle 2 and the working mechanism 35. In addition, the flight control device 1 selects the route determined to be optimal from the extracted flight routes as a tentative flight route (route in the second movement mode), and proceeds to step S116.

[0065] In step S116, the flight control device 1 acquires surrounding environment information that represents the surrounding environment of the work machine 3 as a function of the surrounding environment information acquisition unit 15. Here, for example, the flight control device 1 transmits images acquired by the imaging device 22 transmitted by the control device 21 to the image processing device 10 (FIG. 1), and can acquire surrounding environment information based on the results of image processing in the image processing device 10. The surrounding environment information includes the positions (coordinates) of obstacles that exist in the vicinity of the work machine 3 and that may interfere with the flight of the unmanned aerial vehicle 2. The image processing device 10 may be installed in any location, and may be installed in the flight control device 1, the unmanned aerial vehicle 2, or the work machine 3, for example.

[0066] In order to acquire information about the surrounding environment, a radar, sonar sensor, or the like may be provided in the unmanned aerial vehicle 2 instead of or in addition to the image processing device 10. In this case, the position (coordinates) of the obstacle can be detected by performing analysis based on the radio waves or sound waves reflected by the obstacle in a processing device provided in, for example, the unmanned aerial vehicle 2, the flight control device 1, or the work machine 3.

[0067] In step S118, the flight control device 1, as a function of the decision unit 12, determines, based on the surrounding environment information acquired in step S116, whether there is an obstacle on the tentative flight path selected in step S112 or step S114 that may interfere with the flight of the unmanned aerial vehicle 2, and if the determination is positive, proceeds to step S120, and if the determination is negative, proceeds to step S124.

[0068] In step S120, the flight control device 1, as a function of the decision unit 12, searches for and extracts a flight path that avoids interference between the unmanned aerial vehicle 2 and the work machine 3 and between the unmanned aerial vehicle 2 and obstacles, based on the status information acquired in step S104 and the surrounding environment information acquired in step S116.

[0069] In step S122, the flight control device 1, as a function of the decision unit 12, determines whether the flight path search in step S120 has extracted a flight path that avoids interference between the unmanned aerial vehicle 2 and the work machine 3 and between the unmanned aerial vehicle 2 and an obstacle, and if the determination is positive, proceeds to step S124, and if the determination is negative, proceeds to step S126.

[0070] In step S124, the flight control device 1, as a function of the determination unit 12, sets the tentative flight path selected in step S112 or step S114, or the flight path selected from the flight paths extracted in step S120, as the final flight path, and proceeds to step S130.

[0071] Meanwhile, in step S126, the flight control device 1 outputs, as a function of the information output unit 16, information indicating that there is no flight route to be set, and proceeds to step S102. The information output from the information output unit 16 may include information on the reason why a flight route cannot be set, for example, information that a flight route cannot be selected that keeps the distance from the work machine 3 within a predetermined range due to the presence of an obstacle. The information output from the information output unit 16 may be displayed on the image output device 51, for example, and may also include audio output information.

[0072] In step S130, the flight control device 1, as a function of the control unit 13, transmits control information specifying the final flight path set in step S124 to the unmanned aerial vehicle 2, and proceeds to step S102. The transmitted control information is received by the control device 21. The unmanned aerial vehicle 2 flies along the specified flight path in accordance with the control of the control device 21, heading toward the new target position.

[0073] Thus, in this embodiment, the unmanned aerial vehicle 2 remains at the target position before the update (until the determination in step S102 is affirmative), and photographing continues at this target position using the imaging device 22. By maintaining a hovering state at the target position, the unmanned aerial vehicle 2 can photograph the status of the work (e.g., demolition work) using the imaging device 22 from an appropriate photographing position, and the captured images can be viewed by the operator of the work machine 3. Therefore, there is no degradation in image quality due to the shaking of the unmanned aerial vehicle 2, and highly visible captured images can be presented to the operator, effectively supporting the work by the work machine 3.

[0074] Furthermore, if linear movement to the new target position is possible, a flight path that moves the unmanned aerial vehicle 2 in a straight line is selected, thereby minimizing the flight path and flight time.

[0075] Furthermore, if linear movement to the new target position cannot be selected due to interference with the working mechanism 35, a flight path that bypasses the working mechanism 35 is automatically selected. This allows the unmanned aerial vehicle 2 to be moved to the new target position without imposing a burden on the operator.

[0076] Furthermore, in this embodiment, a flight path that avoids obstacles is automatically selected, so the unmanned aerial vehicle 2 can be moved to a new target position without imposing a burden on the operator.

[0077] Next, the current position (step S106) and the target position (step S108) will be described.

[0078] First, before the unmanned aerial vehicle 2 takes off, the current position is the initial position where the unmanned aerial vehicle 2 is placed, and the new target position is the initial shooting position.

[0079] When the photographing position is changed after the imaging device 22 starts photographing, the current position is the photographing position before the change, and the new target position is the photographing position after the change.

[0080] Furthermore, for example, when work is completed, the unmanned aerial vehicle 2 flies from the last photographed position to a predetermined position, for example, the initial position, and lands. In this case, the current position is the last photographed position, and the new target position is the position where the unmanned aerial vehicle 2 will land, for example, the initial position.

[0081] The photographing position is defined as a position where the state of the work mechanism 35 can be photographed, or a position where at least a part of the work object, for example, a building that is the subject of demolition work, can be photographed. A position where both the state of the work mechanism 35 and the work object can be photographed may also be defined as a photographing position.

[0082] As the photographing positions, a left position where the imaging device 22 can photograph the working mechanism 35 from the left side of the working mechanism 35, and a right position where the imaging device 22 can photograph the working mechanism 35 from the right side of the working mechanism 35 can be set. One of the left position and the right position is selected as the target position (photographing position).

[0083] The set of left and right positions can be set based on the state (position, direction) of the working mechanism 35 at the time when the left and right positions are set. In this case, the left and right positions may be set to positions symmetrical to the left and right of the working mechanism 35.

[0084] Furthermore, the set of left and right positions can be set based on the state (position, direction) of the lower traveling body 31 or the upper rotating body 32. In this case, the left and right positions may be set to positions symmetrical to the left and right with respect to the lower traveling body 31 or the upper rotating body 32.

[0085] The pair of left and right positions may also be determined according to an instruction from the operator.

[0086] Figure 5 shows the left and right positions as viewed from above. In Figure 5, the left and right positions 2L and 2R are located forward of the rotation axis of the boom 35a, i.e., in the direction in which the working mechanism 35 extends, and are symmetrical with respect to the working mechanism 35 at the start of work.

[0087] The left and right positions may be prepared as a set of multiple positions. In this case, one set may be selected from the multiple sets of positions in response to an instruction from the operator or in response to the state of the work machine 3 (the positions and directions of each part of the work machine 3).

[0088] Furthermore, the set of left and right positions may be changed as needed depending on the state (position, direction) of the work mechanism 35 or the state (position, direction) of the lower traveling body 31 or the upper rotating body 32. In this case, an appropriate photographing position according to the state of the work machine 3 can be secured.

[0089] Either the left position or the right position can be selected as the shooting position. For example, either position can be selected in accordance with instructions from the operator. In this case, the operator of the work machine 3 can select a more effective image for understanding, for example, the situation at the work site that cannot be seen with a direct view. For example, in order to obtain a more effective image, the operator can switch the shooting position from the left position to the right position. Furthermore, the shooting position may be initially set to the left position or the right position, or the shooting position set in the initial setting may be switchable in accordance with instructions from the operator.

[0090] In addition, either the left position or the right position may be selected as appropriate depending on the state (position, direction) of the working mechanism 35 or the state (position, direction) of the lower traveling body 31 or the upper rotating body 32.

[0091] The shooting positions are not limited to the left and right positions, and can be set to any positions instead of or in addition to the left and right positions. The number of shooting positions is also arbitrary, and three or more shooting positions may be provided and selected as appropriate in response to instructions from the operator or in response to the state of the work machine 3 (the position and direction of each part of the work machine 3).

[0092] Next, the value of the parameter indicating the possibility of interference between the unmanned aerial vehicle 2 and the work machine 3 (step S108) will be described.

[0093] When calculating the values ​​of the above parameters, the flight control device 1 determines the shortest distance (minimum separation distance) between the path that the unmanned aerial vehicle 2 will take if it moves in a straight line from its current position to a new target position, and the work mechanism 35. The state of the work mechanism 35 is calculated based on the above-mentioned state information of the work machine 3, i.e., the position information and attitude information of the work machine 3. As described above, the attitude information of the work machine 3 includes fixed (known) parameters such as the boom length, boom foot position, jib length, and arm length of the work machine 3, as well as parameters that change during work, such as the boom angle, jib angle, and arm angle.

[0094] If this shortest distance is less than a predetermined threshold, there is a possibility that the unmanned aerial vehicle 2 will come into contact with the working mechanism 35 while moving in a straight line from the current position to the new target position. Therefore, in such a case (if the determination in step S108 is negative), a straight-line path is not selected, and a route that avoids and detours the working mechanism 35 is searched for (step S114). On the other hand, if the shortest distance is equal to or greater than the predetermined threshold (if the determination in step S108 is positive), there is no possibility that the unmanned aerial vehicle 2 will come into contact with the working mechanism 35 while moving in a straight line, and therefore a straight-line path from the current position to the new target position is selected as the tentative flight path (step S112).

[0095] In this embodiment, the shortest distance is calculated as the value of the parameter. However, the parameter may be a binary parameter indicating whether linear movement is possible. For example, without calculating the shortest distance, the value of the parameter (one of two values) may be calculated using an algorithm that determines whether interference will occur between the unmanned aerial vehicle 2 flying on a linear path and the work machine 3. In this case, the time required to select a flight path can be reduced, particularly when linear movement is possible.

[0096] Next, we will explain how to select a flight path (tentative flight path) (step S114) that avoids interference between the unmanned aerial vehicle 2 and the work machine 3 when a straight-line movement path cannot be selected (when the judgment in step S110 is negative).

[0097] A flight path that avoids interference between the unmanned aerial vehicle 2 and the work machine 3 is extracted and selected as a path that ensures a certain distance from the work mechanism 35. In this case, the shortest distance between the flight path and the work mechanism 35 can be, for example, the same as or longer than the above-mentioned predetermined threshold. Also, different lower limit values ​​for the shortest distance to the work mechanism 35 may be set according to different parts, such as the boom 35a and the arm 35b, and a flight path may be selected such that the shortest distance for each part exceeds the lower limit value. For example, because the arm 35b (crusher 35c) typically moves faster than the boom 35a, the lower limit value for the shortest distance to the arm 35b (crusher 35c) may be set to a value greater than the lower limit value for the shortest distance to the boom 35a.

[0098] The trajectory of the flight path can be, for example, a path that includes waypoints and connects the current position, the waypoints, and the target position in a straight line. One or more waypoints can be set.

[0099] Figure 5A is a diagram showing a flight path via two waypoints. The example of Figure 5A shows a flight path in which the unmanned aerial vehicle 2 moves from left position 2L to right position 2R, sequentially passing through waypoints 201 and 202. In this case, the unmanned aerial vehicle 2 rises in a straight line from left position 2L to waypoint 201, then flies horizontally in a straight line to waypoint 202, and then descends in a straight line to right position 2R, thereby avoiding interference with arm 53b.

[0100] Figure 5B is a diagram showing a flight path via one waypoint. The example of Figure 5B shows a flight path in which the unmanned aerial vehicle 2 moves from the left position 2L to the right position via waypoint 203. In this case, the unmanned aerial vehicle 2 flies horizontally in a straight line from the left position 2L to the waypoint 203, and then flies horizontally in a straight line to the right position, thereby avoiding interference with the boom 53a.

[0101] Furthermore, the flight path can also be set to a curved, for example, arc-shaped path so that the entire flight path maintains a certain distance or more from the working mechanism 35. For example, an arc-shaped path can be set that avoids the working mechanism 35, as in the path 101 shown in Figure 5. The path 101 shown in Figure 5 indicates the path that the unmanned aerial vehicle 2 flies horizontally from the left position 2L to the right position 2R.

[0102] FIG. 5C is a diagram showing an example of a flight path set at the start of work. FIG. 5C shows a flight path 102 in which the unmanned aerial vehicle 2, which is currently at the initial position 301, moves to the left position 2L, which is the target position. In this example, the unmanned aerial vehicle 2 ascends from the initial position 301 to the waypoint 204, and then flies horizontally to the waypoint 205. The unmanned aerial vehicle 2 then ascends from the waypoint 205 to the left position 2L. In this case, if the unmanned aerial vehicle 2 were to move in a straight line from the initial position 301 to the left position 2L, there is a possibility that it would interfere with the boom 53a. However, by selecting the flight path 102 that passes through the waypoints 204 and 205, the unmanned aerial vehicle 2 can be prevented from contacting the boom 53a. In this case, a distance Δ greater than or equal to the minimum distance threshold (predetermined threshold) set to avoid contact is secured between the selected flight path 102 and the boom 53a.

[0103] Although multiple flight routes can be selected as flight routes that avoid interference between the unmanned aerial vehicle 2 and the work machine 3, it is desirable to select a route with a short flight distance or a route that shortens travel time.

[0104] One possible selection method is to select the shortest flight route among those that avoid interference between the unmanned aerial vehicle 2 and the work machine 3. However, for example, when setting a route that sequentially connects the current position, waypoints, and the target position in a straight line, increasing the number of waypoints can shorten the flight distance. However, on the other hand, increasing the number of waypoints increases the number of turns and stops at the waypoints, which increases the travel time. For this reason, it is desirable to select a flight route that meets the required conditions. For example, if travel time is a priority, a flight route that minimizes travel time can be selected among those that avoid interference between the unmanned aerial vehicle 2 and the work machine 3.

[0105] Next, a method for setting a flight path when there is an obstacle on the tentative flight path that may interfere with the flight of the unmanned aerial vehicle 2 (when the determination in step S118 is positive) will be described.

[0106] In this case, the flight path is set so that interference between the unmanned aerial vehicle 2 and the work machine 3 can be avoided and the shortest distance between the flight path and the obstacle is equal to or greater than a predetermined threshold. This eliminates the risk of the unmanned aerial vehicle 2 coming into contact with the obstacle. The flight path may also be set, for example, to shorten the flight distance, for example, to minimize the flight distance, or to shorten the flight time, for example, to minimize the flight time. The flight path may be a combination of straight paths connecting waypoints, or may be composed of curves, for example, arc-shaped curves.

[0107] Next, the flight path when a cable is connected to the unmanned aerial vehicle 2 will be described.

[0108] 6 and 6A are diagrams illustrating flight paths when a cable is connected to the unmanned aerial vehicle. The cable 400 is, for example, a cable used to supply power to the unmanned aerial vehicle 2 or a cable used for communication between the unmanned aerial vehicle 2 and a work machine 3. This also includes cases where the cable 400 is used as a wire to tether the unmanned aerial vehicle 2 and limit its flight range.

[0109] The example in Figure 6 shows a flight path in which the unmanned aerial vehicle 2 moves from the initial position 301 to the right-hand position 2R, which is the first shooting position. In this case, the unmanned aerial vehicle 2 rises in a straight line from the initial position 301 to the waypoint 207, then flies horizontally in a straight line to the right-hand position 2R and reaches the right-hand position 2R. In this case, the waypoint 207 is set so that not only the unmanned aerial vehicle 2 but also the cable 400 extending from the unmanned aerial vehicle 2 does not come into contact with and become tangled in the boom 53a or arm 53b on the flight path.

[0110] The example of Figure 6A shows a flight path in which the unmanned aerial vehicle 2 moves from left position 2L to right position 2R. In this case, the unmanned aerial vehicle 2 flies horizontally in a straight line from left position 2L to waypoint 208, then flies horizontally in a straight line to right position 2R to reach right position 2R. In this case, waypoint 208 is set at a position on the flight path so that not only the unmanned aerial vehicle 2 but also the cable 400 extending from the unmanned aerial vehicle 2 does not come into contact with and become tangled with the boom 53a or arm 53b.

[0111] In this way, when the cable 400 is connected to the unmanned aerial vehicle 2, even if there is no possibility that the unmanned aerial vehicle 2 itself will interfere with the working mechanism 35 when the unmanned aerial vehicle 2 moves in a straight line from its current position to its target position, if there is a possibility that the cable 400 will come into contact with the working mechanism 35, a flight path will be selected that takes a detour from the current position to the target position to avoid contact.

[0112] In addition, in order to prevent excess slack in the cable 400, a mechanism such as a reel that winds up the cable 400 with a constant tension may be provided. In this case, the cable 400 will have a shape that is close to a straight line, and therefore, it is possible to eliminate the risk that the cable 400, which is blown by wind or the like, will inadvertently come into contact with the working mechanism 35.

[0113] As described above, in this embodiment, the unmanned aerial vehicle 2 remains at the target position before the update until the target position is updated, and photography continues at this target position using the imaging device 22. As a result, there is no degradation in image quality due to shaking of the unmanned aerial vehicle 2, and highly visible captured images can be presented to the operator, effectively supporting work by the work machine 3.

[0114] Furthermore, if linear movement to the new target position is possible, a flight path that moves the unmanned aerial vehicle 2 in a straight line is selected, thereby minimizing the flight path and flight time. Furthermore, if linear movement to the new target position cannot be selected due to interference with the work mechanism 35, a flight path that bypasses the work mechanism 35 is automatically selected. This allows the unmanned aerial vehicle 2 to be moved to the new target position without imposing a burden on the operator.

[0115] Although each embodiment has been described in detail above, it is not limited to the specific embodiment, and various modifications and changes are possible within the scope of the claims. It is also possible to combine all or a plurality of components of the above-described embodiments.

[0116] For example, the above-described embodiment relates to a configuration for assisting work on the work machine 1, but it can also be applied to inspecting the work machine 1 itself, or to assisting in the travel and other operations of the work machine 1. For example, it can be applied to cases where it is necessary to assist in the inspection of tall objects such as attachments when the machine is stopped, or to assist in the monitoring of the surroundings while traveling in order to move the work machine 1. In this way, the content of the above-described embodiment can be applied to various types of support related to the work machine 1. [Explanation of symbols]

[0117] 1 Flight control unit 2 Unmanned Aerial Vehicles 3. Work machinery 11 Status information acquisition unit 12 Decision Section 13 Control Unit 14 Calculation section 15 Surrounding environment information acquisition unit 16 Information output section

Claims

1. An assistance device that assists a work machine based on an image from an imaging device mounted on an unmanned aerial vehicle, a status information acquisition unit that acquires status information representing the status of the work machine; a determination unit that determines a target position of the unmanned aerial vehicle and a flight path of the unmanned aerial vehicle to the target position; a control unit that controls movement of the unmanned aerial vehicle to the target position based on the flight path determined by the determination unit, the determination unit determines the flight path based on the status information so as to avoid interference with the work machine; the status information is posture information and position information of the work machine, the target position is an imaging position where the image is captured or a predetermined position from the imaging position where the image is captured, the work machine includes a lower traveling body, an upper rotating body mounted on the lower traveling body, and a work mechanism provided on the upper rotating body, the status information includes posture information representing a posture of at least one of the work machine and the work mechanism, The system further includes a calculation unit that determines the possibility of interference between the unmanned aerial vehicle and the work machine by calculating the shortest distance between the unmanned aerial vehicle and the work mechanism to determine the possibility of interference between the unmanned aerial vehicle and the work machine when the unmanned aerial vehicle moves linearly from its current position to the target position based on the status information, or a binary value indicating whether or not the linear movement is possible using an algorithm that determines whether or not there is interference with the work machine, The determination unit determines the flight path based on the judgment of the calculation unit.

2. The support device according to claim 1 , wherein the determination unit determines the target position so that the imaging device can capture an image of the working mechanism.

3. The support device according to claim 1 , wherein the determination unit determines the target position so that the imaging device can capture an image of a part of the working mechanism or the work target object.

4. An assistance device that assists a work machine based on an image from an imaging device mounted on an unmanned aerial vehicle, a status information acquisition unit that acquires status information representing the status of the work machine; a determination unit that determines a target position of the unmanned aerial vehicle and a flight path of the unmanned aerial vehicle to the target position; a control unit that controls movement of the unmanned aerial vehicle to the target position based on the flight path determined by the determination unit, the determination unit determines the flight path based on the status information so as to avoid interference with the work machine; the status information is posture information and position information of the work machine, the work machine includes a lower traveling body, an upper rotating body mounted on the lower traveling body, and a work mechanism provided on the upper rotating body, the state information includes posture information representing a posture of the working mechanism, the target positions include a left-side position at which the imaging device can capture an image of the preset working mechanism from the left side of the working mechanism, and a right-side position at which the imaging device can capture an image of the preset working mechanism from the right side of the working mechanism, The determination unit determines the flight path when the target position changes due to an operator selecting either the left position or the right position.

5. An assistance device that assists a work machine based on an image from an imaging device mounted on an unmanned aerial vehicle, a status information acquisition unit that acquires status information representing the status of the work machine; a determination unit that determines a target position of the unmanned aerial vehicle and a flight path of the unmanned aerial vehicle to the target position; a control unit that controls movement of the unmanned aerial vehicle to the target position based on the flight path determined by the determination unit, the determination unit determines the flight path based on the status information so as to avoid interference with the work machine; the status information is posture information and position information of the work machine, the target position is an imaging position where the image is captured or a predetermined position from the imaging position where the image is captured, When the target position changes, the determination unit determines, as the flight path, one of a first flight path that moves linearly from the original target position to the new target position, and a second flight path that moves from the original target position to the new target position via a waypoint.

6. The assistance device of claim 5 , wherein the second flight path includes a linear movement from the original target position to the waypoint and a linear movement from the waypoint to the new target position.

7. The assistance device according to claim 5 or 6, wherein the determination unit determines the waypoints based on the state information when selecting the second flight route.

8. The assistance device described in claim 7, wherein the determination unit determines the intermediate point so that the shortest distance between the unmanned aerial vehicle and the work machine when moving along the second flight path is longer than the shortest distance when moving along the first flight path.

9. a surrounding environment information acquisition unit that acquires surrounding environment information that represents the surrounding environment of the work machine, The support device according to claim 1 , wherein the determination unit determines the flight path based on the state information and the surrounding environment information.

10. The support device according to claim 1 , further comprising an information output unit that outputs predetermined information when the flight path cannot be determined by the determination unit.

11. A system comprising an assistance device described in any one of claims 1 to 10 and at least one of the work machine and the unmanned aerial vehicle.

Images