Work Support Device

Abstract

To provide a work support device allowing to avoid inconvenience when continuously imaging a place under work.SOLUTION: A work support device has: an information acquiring part acquiring information showing the state of a work machine and the state of a flying body; an imaging object position calculation part calculating an imaging object position related to the work; an imaging parameter setting part setting an imaging angle or an imaging direction of an imaging device to image the imaging object position; and a flight control part controlling a flying state of the flying body, on the basis of the information acquired by the information acquiring part. In a situation of the flight control part performing hovering keeping control for not varying a target flying position or a target flying attitude of the flying body, the imaging parameter setting part varies the imaging angle or the imaging direction of the imaging device in order to continue the hovering keeping control, when the imaging object position calculated by the imaging object position calculation part is displaced from the imaging angle of the imaging device, or when the displacement is expected.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to a work assistance device. [Background technology]

[0002] There is known a technique for capturing images of a space that cannot be captured by an imaging device attached to an upper rotating body of a shovel, using an imaging device attached to an autonomous flying vehicle (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. WO2017 / 131194 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above-mentioned conventional technology, when the actual work location is continuously imaged, the target flight position of the aircraft needs to be constantly changed according to the movement of the work location. In this case, there is a risk of an increase in the processing load required for control and a shortened life of the battery mounted on the aircraft.

[0005] Therefore, in one aspect, the present invention aims to effectively avoid the inconvenience that occurs when a place where work is being carried out is continuously targeted for imaging. [Means for solving the problem]

[0006] On one side, A work support device that controls a positional relationship between a work machine and an aircraft equipped with an imaging device, The work machine includes a lower traveling body, an upper rotating body mounted on the lower traveling body, and a display device that displays an image captured by the imaging device, The work support device includes: an information acquisition unit that acquires information representing a state of the work machine and a state of the aircraft; an imaging target position calculation unit that calculates a position of an imaging target related to the work; an imaging parameter setting unit that sets an imaging angle of view or an imaging direction of the imaging device so that the position of the imaging target is imaged; A flight control unit that controls a flight state of the aircraft based on the information acquired by the information acquisition unit, A work support device is provided in which, when the flight control unit is executing hovering maintenance control that does not change the target flight position or target flight attitude of the aircraft, if the position of the imaging target calculated by the imaging target position calculation unit deviates from the imaging angle of view of the imaging device, or is predicted to deviate from it, the imaging parameter setting unit changes the imaging angle of view or imaging direction of the imaging device so that the hovering maintenance control is continued. Effect of the Invention

[0007] According to one aspect of the present invention, it is possible to effectively avoid the inconvenience that occurs when a place where work is being carried out is continuously targeted for imaging. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram relating to the configuration of a work machine and an unmanned airplane according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration related to a control system of a work machine. [Diagram 3] FIG. 2 is a diagram illustrating functions realized by various control devices. [Figure 4] 11A and 11B are diagrams illustrating the relationship between the angle of view of the imaging device and the imaging range of the imaging device in the case of hovering maintenance control. [Figure 4A] 11A and 11B are diagrams illustrating the relationship between the imaging direction of the imaging device and the imaging range of the imaging device in the case of hovering maintenance control. [Diagram 5] 4 is a schematic flowchart showing an example of processing executed by a flight control device. [Figure 6] 6 is a schematic flowchart showing another example of processing executed by the flight control device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Each embodiment will be described in detail below with reference to the accompanying drawings.

[0010] [Example 1] Fig. 1 is an explanatory diagram relating to the configuration of a work machine 1 and an unmanned airplane 40 according to one embodiment. In addition to the work machine 1 and the unmanned airplane 40, Fig. 1 also illustrates a flight control device 50 (an example of a work support device) and a remote control device 52.

[0011] The work machine 1 performs a predetermined task in cooperation with the unmanned aerial vehicle 40. The work machine 1 is, for example, a construction machine equipped with a crusher 145 suitable for demolition work and the like, and is equipped with a crawler-type lower traveling body 110, an upper rotating body 120 rotatably mounted on the lower traveling body 110 via a rotating mechanism 130, and a work mechanism 140. A cab (operator's compartment) 122 is provided on the front left side of the upper rotating body 120. The work mechanism 140 is provided in the front center of the upper rotating body 120, and the crusher 145 is provided at the tip of the work mechanism 140. The work machine 1 may also be equipped with a basement from which the unmanned aerial vehicle 40 takes off and lands.

[0012] The working mechanism 140 includes a boom 141 that is mounted on the upper rotating body 120 so as to be able to rise and fall, an arm 143 that is rotatably connected to the tip of the boom 141, and a crusher 145 attached to the tip of the arm 143. The working mechanism 140 is equipped with a boom cylinder 142, an arm cylinder 144, and a bucket cylinder 146 that are formed by extendable and retractable hydraulic cylinders. Instead of the crusher 145, another tip attachment such as a bucket may be attached to the tip of the arm 143.

[0013] The boom cylinder 142 is interposed between the boom 141 and the upper rotating body 120 so as to extend and retract when supplied with hydraulic oil to rotate the boom 141 in the hoisting direction. The arm cylinder 144 is interposed between the arm 143 and the boom 141 so as to extend and retract when supplied with hydraulic oil to rotate the arm 143 about a horizontal axis relative to the boom 141. The bucket cylinder 146 is interposed between the crusher 145 and the arm 143 so as to extend and retract when supplied with hydraulic oil to rotate the crusher 145 about a horizontal axis relative to the arm 143. The crusher cylinder 147 is provided in the crusher 145 so as to extend and retract when supplied with hydraulic oil to open and close the crusher 145.

[0014] The unmanned airplane 40 is a rotorcraft equipped with a plurality of blades (for example, 4, 6, or 8), a battery that supplies power to an electric motor (actuator) for rotating the plurality of blades, etc. Note that, instead of or in addition to such a battery, the unmanned airplane 40 may be connected to a power supply line from the ground.

[0015] The unmanned airplane 40 includes a control device 400 and an imaging device 410 .

[0016] The control device 400 realizes various flight states (forward state, backward state, ascending state, descending state, hovering, etc.) of the unmanned airplane 40 according to control information (commands) from the flight control device 50 described later and operation information from the remote control device 52. In addition, the control device 400 transmits an image (forward environment image) captured by the imaging device 410 to the work machine 1.

[0017] The imaging device 410 includes a camera. The type of camera is arbitrary, and may be, for example, a wide-angle camera. The imaging device 410 may be removably attached to the unmanned airplane 40, or may be firmly fixed to the unmanned airplane 40. The imaging device 410 acquires a forward environment image in front of the body of the unmanned airplane 40 using an imaging element such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The imaging device 410 may acquire a forward environment image in real time, and supply the image to the control device 400 in a stream format with a predetermined frame period, for example.

[0018] The imaging device 410 preferably includes a gimbal (not shown). The gimbal functions to keep the optical axis of the imaging device 410 in a constant direction (e.g., a predetermined direction in a horizontal plane) even if the attitude of the unmanned aerial vehicle 40 changes.

[0019] The flight control device 50 executes various controls of the unmanned airplane 40. In one embodiment, the flight control device 50 is realized by a server (server computer), and in this case, the flight control device 50 is connected to the work machine 1 and the unmanned airplane 40 via a network (not shown). In this case, the network may include a wireless communication network, the Internet, a Virtual Private Network (VPN), a Wide Area Network (WAN), a wired network, or any combination of these. In another embodiment, the flight control device 50 may be realized by the control device 10 of the work machine 1. In another embodiment, the flight control device 50 may be realized by the control device 400 of the unmanned airplane 40. Alternatively, in another embodiment, the functions of the flight control device 50 may be realized by a combination of any two or all of the server, the control device 10, and the control device 400 in cooperation with each other. Details of the flight control device 50 will be described later.

[0020] The remote operation device 52 may be in the form of a remote controller, for example, and may be operated by a user (for example, the operator of the work machine 1 or a worker different from the operator). If the user is the operator of the work machine 1, the remote operation device 52 may be brought into the cab 122. The remote operation device 52 is capable of wireless communication with the unmanned airplane 40, and transmits an operation signal to the unmanned airplane 40 in response to the user's operation. In this case, when the unmanned airplane 40 receives operation information from the remote operation device 52, the control device 400 of the unmanned airplane 40 realizes the movement of the unmanned airplane 40 (forward, backward, up and down, etc.) in response to the operation information. In a modified example, the remote operation device 52 may be omitted. The remote operation device 52 may also be realized by a smartphone or the like.

[0021] FIG. 2 is a diagram showing an example of a hardware configuration related to a control system of the work machine 1. As shown in FIG.

[0022] As shown in FIG. 2, the work machine 1 includes a peripheral device 8 and a control device 10.

[0023] The peripheral devices 8 are electronically controllable devices and various sensors mounted on the work machine 1. The peripheral devices 8 may include, for example, an image output device 80 (one example of a display device), a buzzer, an audio output device (not shown), a hydraulic pressure generating device (not shown) that operates the work mechanism 140, and various sensors 82 that detect the operating states of various operating members. The hydraulic pressure generating device may be a hydraulic pump driven by an engine and / or an electric motor. When a hydraulic pump driven by an electric motor is used, the hydraulic pressure generating device may include an inverter for driving the electric motor. The various sensors 82 may include a gyro sensor, a GPS (Global Positioning System) sensor, various angle sensors, an acceleration sensor (tilt sensor), a hydraulic pressure sensor that detects hydraulic pressure at a predetermined location of a hydraulic line (not shown) applied by the hydraulic pressure generating device, and the like.

[0024] The image output device 80 is provided in the cab 122 so as to be visible to the operator of the work machine 1. The image output device 80 is arbitrary, and may be, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display. In a modified example, the image output device 80 may be a portable device (for example, a tablet terminal, etc.) that can be brought into the cab 122 by the operator of the work machine 1.

[0025] The control device 10 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, an auxiliary storage device 14, a drive device 15, and a communication interface 17, all connected by a bus 19, as well as a wired transceiver unit 25 and a wireless transceiver unit 26 connected to the communication interface 17.

[0026] The auxiliary storage device 14 is, for example, a hard disk drive (HDD) or a solid state drive (SSD), and is a storage device that stores data related to application software and the like.

[0027] The wired transceiver 25 includes a transceiver capable of communicating using a wired network. Peripheral devices 8 are connected to the wired transceiver 25. However, some or all of the peripheral devices 8 may be connected to the bus 19 or may be connected to the wireless transceiver 26.

[0028] The wireless transceiver 26 is a transceiver capable of communicating using a wireless network. The wireless network may include a wireless communication network for mobile phones, the Internet, a VPN, a WAN, and the like. The wireless transceiver 26 may also include a near field communication (NFC) unit, a Bluetooth (registered trademark) communication unit, a Wi-Fi (Wireless-Fidelity (registered trademark)) transceiver unit, an infrared transceiver unit, and the like. For example, the wireless transceiver 26 can realize communication with a flight control device 50 in the form of a server.

[0029] The control device 10 may be connectable to a recording medium 16. The recording medium 16 stores a predetermined program. The program stored in the recording medium 16 is installed in the auxiliary storage device 14 of the control device 10 via the drive device 15. The installed predetermined program can be executed by the CPU 11 of the control device 10. For example, the recording medium 16 may be a recording medium that records information optically, electrically or magnetically, such as a CD (Compact Disc)-ROM, a flexible disk, or a magneto-optical disk, or a semiconductor memory that records information electrically, such as a ROM or a flash memory. The recording medium 16 does not include a carrier wave.

[0030] Here, the control system of the work machine 1 has been described with reference to Fig. 2, but the control system of the unmanned airplane 40 may be substantially similar, except for the configuration related to the peripheral device 8. For example, in the case of the control system of the unmanned airplane 40, the hardware configuration of the control device 400 may be similar to that of the control device 10. In addition, peripheral devices corresponding to the peripheral device 8 include an imaging device 410 (see Fig. 1) and various sensors.

[0031] Furthermore, the hardware configuration of the flight control device 50 may be substantially similar to the hardware configuration of the control device 10 shown in FIG.

[0032] Next, the flight control device 50 will be described in detail with reference to FIG. 3 onward, along with the control device 10 and the control device 400.

[0033] FIG. 3 is a diagram for explaining functions realized by the control device 10, the control device 400, and the flight control device 50.

[0034] 3, the control device 10 includes a position information acquisition unit 150, a posture information acquisition unit 151, an orientation information acquisition unit 152, a communication processing unit 153, and an image output processing unit 154. Each functional unit such as the position information acquisition unit 150 can be realized by the CPU 11 shown in FIG. 2 executing a program in a storage device such as the ROM 13 shown in FIG. 2.

[0035] The position information acquisition unit 150 acquires position information of the work machine 1 from a GPS sensor (not shown) among the sensors 82. The position information of the work machine 1 is expressed by latitude, longitude, and altitude. The GPS sensor includes a GPS receiver, and calculates latitude, longitude, and altitude by interference positioning or the like based on radio waves from satellites.

[0036] The attitude information acquisition unit 151 acquires attitude information of the work machine 1 based on various sensors among the sensors 82 that acquire parameters related to the attitude of the work machine 1. In this case, the various sensors that acquire the parameters related to the attitude may be, for example, a boom angle sensor, an arm angle sensor, a bucket angle sensor, a machine body inclination sensor, etc. The boom angle sensor is a sensor that acquires the boom angle, and includes, for example, a rotation angle sensor that detects the rotation angle of the boom foot pin, a stroke sensor that detects the stroke amount of the boom cylinder 142, and a tilt (acceleration) sensor that detects the tilt angle of the boom 141. The same applies to the arm angle sensor and the bucket angle sensor. The machine body inclination sensor is a sensor that acquires the machine body inclination angle, and detects, for example, the inclination angle of the upper rotating body 120 with respect to the horizontal plane.

[0037] The orientation information acquisition unit 152 acquires orientation information indicating the orientation of the upper rotating body 120 based on various sensors that acquire parameters related to the orientation of the upper rotating body 120 relative to the lower running body 110 (hereinafter also simply referred to as the "orientation of the upper rotating body 120") among the sensors 82. In this case, the various sensors that acquire the parameters related to the orientation of the upper rotating body 120 may be, for example, a geomagnetic sensor, a rotation angle sensor (e.g., a resolver, etc.) that detects the rotation angle around the rotation axis of the rotation mechanism 130, a gyro sensor, etc.

[0038] The communication processing unit 153 transmits various pieces of information acquired by the position information acquisition unit 150, the attitude information acquisition unit 151, and the orientation information acquisition unit 152 to the flight control device 50. For example, the communication processing unit 153 may transmit the latest information to the flight control device 50 at predetermined intervals in response to a request from the flight control device 50.

[0039] Furthermore, the communication processing unit 153 receives image data from the unmanned airplane 40. The image data is data of a forward environment image captured by the imaging device 410.

[0040] The image output processing unit 154 outputs a forward environment image onto the image output device 80 based on the image data acquired by the communication processing unit 153. This enables the operator of the work machine 1 to grasp, for example, the situation of the work site that cannot be seen with a direct view, from the forward environment image on the image output device 80.

[0041] 3, the control device 400 includes an aircraft information acquisition unit 401, a target flight state setting unit 402, an aircraft control unit 403, and a communication processing unit 404. Each functional unit such as the aircraft information acquisition unit 401 can be realized by a CPU such as the CPU 11 shown in FIG. 2 executing a program in a storage device such as the ROM 13 shown in FIG. 2.

[0042] The aircraft information acquisition unit 401 acquires aircraft information that indicates various conditions related to the aircraft of the unmanned airplane 40. The aircraft information may include position information of the unmanned airplane 40, attitude information of the unmanned airplane 40, and the like. The position information of the unmanned airplane 40 may be expressed, for example, by latitude, longitude, and altitude. Note that such position information of the unmanned airplane 40 can be acquired from a GPS sensor. The attitude information of the unmanned airplane 40 may include information on rotation around each of the yaw axis, roll axis, and pitch axis of the unmanned airplane 40. Note that such attitude information of the unmanned airplane 40 can be acquired from a sensor such as an inertial measurement unit (IMU) mounted on the unmanned airplane 40.

[0043] The target flight state setting unit 402 sets a target flight state of the unmanned airplane 40 based on control information (commands) from the flight control device 50. The target flight state includes a target flight position and a target flight attitude. Note that, if the control information includes a target flight position and a target flight attitude, the target flight state setting unit 402 may use the target flight position and the target flight attitude as they are.

[0044] The target flight position may be expressed, for example, in terms of latitude, longitude, and altitude.

[0045] In this embodiment, a "state tracking control" that dynamically changes the target flight position of the unmanned airplane 40 by tracking the change in the position of the image capture target, and a "hovering maintenance control" that continuously maintains the flight position of the unmanned airplane 40 at a fixed position are selectively executed. The state tracking control will be described below. The hovering maintenance control will be described later.

[0046] In the case of state tracking control, the latitude and longitude of the target flight position are set to values ​​that make it easy to capture an image of the location where work is being performed, for example, the position of the tip attachment (the position of the image capture target). The latitude and longitude of the target flight position may be variable by the user. These values ​​may also be automatically changed depending on the work mode, etc.

[0047] The altitude of the target flight position may be constant (for example, within a range of 30 m to 40 m). However, the altitude of the target flight position may be variable by the user or may be automatically changed depending on the operation mode, etc.

[0048] The target flight attitude may be expressed by parameters related to rotations around the yaw axis, the roll axis, and the pitch axis, for example. The parameters related to the target flight attitude may be variable by a user, or may be automatically changed depending on the operation mode, etc.

[0049] The aircraft control unit 403 controls the unmanned airplane 40 so as to achieve the target flight state set by the target flight state setting unit 402, based on the aircraft information acquired by the aircraft information acquisition unit 401. Note that the control method for the unmanned airplane 40 is arbitrary, and may be achieved by, for example, PID (Proportional Integral Differential) control or the like.

[0050] The communication processing unit 404 transmits the aircraft information and the like acquired by the aircraft information acquisition unit 401 to the flight control device 50. For example, the communication processing unit 404 may transmit the latest aircraft information to the flight control device 50 at predetermined intervals in response to a request from the flight control device 50.

[0051] Furthermore, the communication processing unit 404 transmits data of the forward environment image captured by the imaging device 410 to the work machine 1. For example, the communication processing unit 404 may transmit the data of the forward environment image to the flight control device 50 at predetermined intervals in response to a request from the flight control device 50.

[0052] 3, the flight control device 50 includes an information acquisition unit 510, an attachment position calculation unit (an example of an image capture target position calculation unit) 512, an image capture parameter setting unit 513, and a flight control unit 514. Each functional unit such as the information acquisition unit 510 can be realized by a CPU such as the CPU 11 shown in FIG. 2 executing a program in a storage device such as the ROM 13 shown in FIG. 2.

[0053] The information acquisition unit 510 acquires various information necessary for various controls of the flight control unit 514. In this embodiment, as an example, the information acquisition unit 510 acquires position information, attitude information, and orientation information of the work machine 1, and aircraft information of the unmanned airplane 40. The position information, attitude information, and orientation information of the work machine 1 can be acquired by communication from the communication processing unit 153 of the control device 10 of the work machine 1. Furthermore, the aircraft information of the unmanned airplane 40 can be acquired from the communication processing unit 404 of the control device 400 of the unmanned airplane 40.

[0054] The attachment position calculation unit 512 calculates the position of the end attachment corresponding to the image capture target, for example, the position of a predetermined part of the end attachment or the position of the end of the end attachment, based on the position information, attitude information, and orientation information of the work machine 1 acquired by the information acquisition unit 510. The position of the end attachment calculated by the attachment position calculation unit 512 may be the position of the image capture target related to the work, or a position from which the position of the image capture target can be derived. The position of the end attachment can be expressed, for example, by latitude, longitude, and altitude.

[0055] The imaging parameter setting unit 513 sets the imaging angle of view and the imaging direction of the imaging device 410 .

[0056] The imaging parameter setting unit 513 can optically set the imaging angle of view of the imaging device 410. In this case, for example, the imaging parameter setting unit 513 can set the imaging angle of view of the imaging device 410 by operating the optical system of the imaging device 410 via communication with the communication processing unit 404.

[0057] The imaging parameter setting unit 513 can set the imaging angle of view of the imaging device 410 by electronic processing of the captured image of the imaging device 410. In this case, for example, the imaging parameter setting unit 513 can set the imaging angle of view of the imaging device 410 by operating the image output processing unit 154 via communication with the communication processing unit 404. For example, in a state where the entire captured image of the imaging device 410 is output from the image output processing unit 154, the imaging angle of view is maximum, and in a state where a part of the captured image of the imaging device 410 (for example, an image corresponding to pixels in the center) is output, the imaging angle of view becomes smaller. In this way, the apparent angle of view can be electronically manipulated without manipulating the optical system of the imaging device 410.

[0058] The imaging parameter setting section 513 can also set the imaging angle of view by combining optical means and electronic processing of the captured image.

[0059] Next, the imaging parameter setting unit 513 can set the imaging direction of the imaging device 410 by rotating the imaging device 410. In this case, for example, the imaging parameter setting unit 513 can set the flight attitude (direction) of the unmanned airplane 40 via the flight control unit 514, thereby setting the imaging direction of the imaging device 410.

[0060] The imaging parameter setting unit 513 can set the imaging direction of the imaging device 410 by electronic processing of the captured image of the imaging device 410. In this case, for example, the imaging parameter setting unit 513 can set the imaging angle of view of the imaging device 410 by operating the image output processing unit 154 via communication with the communication processing unit 404. For example, the imaging angle can be switched by selectively outputting a part of the captured image of the imaging device 410 (for example, an image corresponding to a part of pixels) from the image output processing unit 154. In this way, the apparent imaging angle can be electronically operated without operating the orientation of the imaging device 410.

[0061] The imaging parameter setting section 513 can also set the imaging angle by combining the rotation of the imaging device 410 and electronic processing of the captured image.

[0062] The flight control unit 514 generates control information (commands) to be transmitted to the unmanned airplane 40 based on the various information acquired by the information acquisition unit 510. As described above, the control information is information for causing the target flight state setting unit 402 of the unmanned airplane 40 to set a target flight state.

[0063] When the hovering start condition is satisfied, the flight control unit 514 generates control information so that the unmanned airplane 40 starts hovering. The hovering start condition is satisfied, for example, when the position of the unmanned airplane 40 reaches a target flight position. The hovering start condition is also satisfied, for example, when the imaging device 410 can continue to capture an image of the imaging target even if the flight position of the unmanned airplane 40 is maintained at a constant position. Whether the position of the unmanned airplane 40 has reached the target flight position can be determined, for example, based on the aircraft information acquired by the information acquisition unit 510. Also, as described later, whether the imaging device 410 can continue to capture an image of the imaging target can be determined based on the position of the tip attachment calculated by the attachment position calculation unit 512.

[0064] The flight control unit 514 may cause the control device 400 to start hovering, for example, by not changing the control information (and therefore not changing the target flight position of the unmanned airplane 40). Alternatively, the flight control unit 514 may cause the control device 400 to start hovering by sending a command to the unmanned airplane 40 to start hovering (for example, control information instructing a mode such as a hovering mode).

[0065] Note that the flight control unit 514 may realize the hovering maintenance control, for example, by not changing the control information (and therefore not changing the target flight position of the unmanned airplane 40). Alternatively, the flight control unit 514 may realize the hovering maintenance control by transmitting a command to the unmanned airplane 40 to maintain the hovering (for example, control information instructing a mode such as a hovering mode).

[0066] However, the movements of the work machine 1 can vary depending on the work content, but there are cases where the position of the image capture target related to the work (position of the tip attachment) is limited to a narrow range without moving significantly for a certain period of time. In such cases, if the flight position (target flight position) of the unmanned airplane 40 is frequently moved in small increments, there are concerns that problems such as an increase in the processing load in controlling the flight state of the unmanned airplane 40 and a shortened battery life due to increased power consumption of the unmanned airplane 40 may occur.

[0067] In contrast, according to this embodiment, even if the position of the imaging target of the imaging device 410 (position of the tip attachment) falls outside the imaging angle of view, or is predicted to fall outside, the flight control unit 514 executes hovering maintenance control to continuously maintain the flight position of the unmanned aircraft 40 if the changed position of the imaging target can be captured by the imaging angle of view or imaging direction of the imaging device 410 set by the imaging parameter setting unit 513.

[0068] Therefore, when the position of the imaging target related to the work does not move significantly for a certain period of time and is limited to a small range, there is no need to frequently move the flight position (target flight position) of the unmanned airplane 40 in small increments. This makes it possible to avoid an increase in the processing load in controlling the flight state of the unmanned airplane 40 and a shortened battery life due to an increase in the power consumption of the unmanned airplane 40.

[0069] Depending on the change in the position of the imaging target, if the hovering maintenance control is maintained, it may not be possible to capture the position of the imaging target no matter how the imaging angle of view or imaging direction of the imaging device 410 is changed. In this case, the flight control unit 514 at least temporarily shifts the control state of the unmanned airplane 40 to state tracking control. This makes it possible to continue capturing images of the position of the imaging target.

[0070] FIG. 4 is a diagram showing the relationship between the angle of view of the imaging device 410 and the imaging range of the imaging device 410 in the case of hovering maintenance control.

[0071] As shown in FIG. 4, when the flight state (flight position and flight attitude) of the unmanned airplane 40 is maintained by the hovering maintenance control, the tip attachment moves and the imaging target moves from position P1 to position P2 outside the imaging angle of view L1 as shown by the arrow R1 under the condition that the imaging device 410 captures the position of the imaging target (for example, the tip position of the tip attachment) P1 within the imaging angle of view L1. In this case, in this embodiment, the imaging angle of view is switched to a wider imaging angle of view while continuing the hovering maintenance control, so that the imaging target at position P2 can be continuously captured within the imaging angle of view L2. Therefore, it is possible to avoid an increase in the processing load in controlling the flight state of the unmanned airplane 40 and a shortened battery life due to an increase in the power consumption of the unmanned airplane 40.

[0072] The timing for switching the imaging angle of view of the imaging device 410 to a wider imaging angle of view may be changed. For example, even if an imaging target exists within the imaging angle of view L1, when it is predicted that the imaging target is highly likely to deviate from the imaging angle of view L1, the imaging angle of view of the imaging device 410 may be switched to a wide angle at that time. For example, when the position of the imaging target on the image enters a range of a predetermined number of pixels from the outer edge of the imaging angle of view L1, the imaging angle of view of the imaging device 410 may be switched to a wide angle to capture the imaging target within the imaging angle of view L2.

[0073] In addition, in the case where the imaging target moves from position P2 to position P1, the angle of view may be switched from a state in which the imaging target at position P2 was captured within the imaging angle of view L2 to a state in which the imaging target at position P1 is captured within the imaging angle of view L1.

[0074] FIG. 4A is a diagram showing the relationship between the imaging direction of the imaging device 410 and the imaging range of the imaging device 410 in the case of hovering maintenance control.

[0075] As shown in FIG. 4A, when the flight position of the unmanned airplane 40 is maintained by the hovering maintenance control, the image pickup device 410 captures the position of the image pickup target (for example, the tip position of the tip attachment) P3 within the image pickup angle L1, and the tip attachment moves, and the image pickup target moves from position P3 to position P4 outside the image pickup angle L1 as shown by the arrow R2. In this case, in this embodiment, while continuing the hovering maintenance control, the flight attitude is slightly changed while maintaining the flight position of the unmanned airplane 40, and the direction of the image pickup device 410 is changed. This allows the image pickup direction of the image pickup device 410 to be switched and the image pickup target at position P4 to be continuously captured within the image pickup angle L3. Therefore, it is possible to avoid an increase in the processing load in controlling the flight state of the unmanned airplane 40 and a shortened battery life due to an increase in the power consumption of the unmanned airplane 40.

[0076] The timing for switching the imaging angle of the imaging device 410 may be changed. For example, even if an imaging target exists within the imaging angle of view L1, when it is predicted that the imaging target is highly likely to move out of the imaging angle of view L1, the imaging angle of the imaging device 410 may be switched at that time. For example, when the position of the imaging target on the image enters a range of a predetermined number of pixels from the outer edge of the imaging angle of view L1, the imaging angle of the imaging device 410 may be switched to capture the imaging target within the imaging angle of view L3.

[0077] Next, an example of the operation of the flight control device 50 will be described with reference to Figure 5 onwards. In the process flow diagrams (flowcharts) that follow, the order of the processes in each step may be changed as long as the relationship between the input and output in each step is not lost.

[0078] Fig. 5 is a schematic flowchart showing an example of processing executed by the flight control device 50, and shows an example of controlling the imaging angle of view of the imaging device 410. The processing shown in Fig. 5 may be executed repeatedly at predetermined intervals, for example.

[0079] In step S30, the flight control device 50 acquires various pieces of control information (for example, position information, attitude information, and orientation information of the work machine 1, and aircraft information of the unmanned airplane 40). The method of acquiring the various pieces of control information is as described above.

[0080] In step S31, the flight control device 50 calculates the position of the end attachment based on the position information, attitude information, and orientation information of the work machine 1 acquired in step S30.

[0081] In step S32, the flight control device 50 determines whether the object to be imaged is within the angle of view of the imaging device 410 based on the aircraft information of the unmanned aircraft 40 acquired in step S30 and the position of the tip attachment calculated in step S31, and if the determination is positive, proceeds to step S34, and if the determination is negative, proceeds to step S38.

[0082] In step S34, the flight control device 50 generates control information for achieving hovering, and transmits the generated control information to the unmanned airplane 40. In this case, upon receiving such control information, the control device 400 of the unmanned airplane 40 controls the unmanned airplane 40 so that the hovering related to the control information is achieved or maintained. The control information for achieving hovering may be, for example, the same information as the control information of the previous cycle, or may be information indicating that.

[0083] In step S38, the flight control device 50 determines, based on the aircraft information of the unmanned airplane 40 acquired in step S30 and the position of the tip attachment calculated in step S31, whether the subject to be imaged will be positioned within the angle of view of the imaging device 410 after the change when the current angle of view of the imaging device 410 (for example, the angle of view at which L1 within the imaging angle of view in Figure 4 is imaged) is changed to the planned angle of view after the change (for example, the angle of view at which L2 within the imaging angle of view in Figure 4 is imaged), and if the determination is positive, proceeds to step S40, and if the determination is negative, proceeds to step S42.

[0084] In step S40, the flight control device 50 switches the angle of view of the imaging device 410 to the angle of view after the change for which the determination in step S38 was affirmative (the angle of view that was to be changed).

[0085] On the other hand, in step S42, the flight control device 50 calculates a target flight state for state tracking control. Specifically, the flight control device 50 calculates a target flight state (including a target flight position and a target flight attitude) based on the various information obtained in step S30, and generates control information (commands) that represent the target flight state.

[0086] Next, in step S44, the flight control device 50 transmits the control information generated in step S42 to the unmanned airplane 40 to set a new target flight state. In this case, upon receiving such control information, the control device 400 of the unmanned airplane 40 controls the unmanned airplane 40 so as to realize the target flight state related to the control information. Note that the hovering maintenance control instructed in step S34 or the like is maintained until a new target flight state is set in step S44.

[0087] Fig. 6 is a schematic flowchart showing another example of processing executed by the flight control device 50, and shows an example of controlling the imaging direction of the imaging device 410. The processing shown in Fig. 6 may be executed repeatedly at predetermined intervals, for example.

[0088] In step S30, the flight control device 50 acquires various pieces of control information (for example, position information, attitude information, and orientation information of the work machine 1, and aircraft information of the unmanned airplane 40). The method of acquiring the various pieces of control information is as described above.

[0089] In step S31, the flight control device 50 calculates the position of the end attachment based on the position information, attitude information, and orientation information of the work machine 1 acquired in step S30.

[0090] In step S32, the flight control device 50 determines whether the object to be imaged is within the angle of view of the imaging device 410 based on the aircraft information of the unmanned aircraft 40 acquired in step S30 and the position of the tip attachment calculated in step S31, and if the determination is positive, proceeds to step S34, and if the determination is negative, proceeds to step S38.

[0091] In step S34, the flight control device 50 generates control information for achieving hovering, and transmits the generated control information to the unmanned airplane 40. In this case, upon receiving such control information, the control device 400 of the unmanned airplane 40 controls the unmanned airplane 40 so that the hovering related to the control information is achieved or maintained. The control information for achieving hovering may be, for example, the same information as the control information of the previous cycle, or may be information indicating that.

[0092] In step S38, the flight control device 50 determines, based on the aircraft information of the unmanned airplane 40 acquired in step S30 and the position of the tip attachment calculated in step S31, whether the imaging target will be positioned within the changed angle of view (within the imaging range) of the imaging device 410 when the current imaging angle of the imaging device 410 (for example, the imaging angle at which L1 within the imaging angle of view of Figure 4A is captured) is changed to a planned changed imaging angle (for example, the angle of view at which L3 within the imaging angle of view of Figure 4A is captured), and if the determination is positive, proceeds to step S40, and if the determination is negative, proceeds to step S42.

[0093] In step S40, the flight control device 50 switches the imaging angle of the imaging device 410 to the imaging angle after the change for which the determination in step S38 was affirmative (the angle to which the imaging angle was to be changed).

[0094] On the other hand, in step S42, the flight control device 50 calculates a target flight state for state tracking control. Specifically, the flight control device 50 calculates a target flight state (including a target flight position and a target flight attitude) based on the various information obtained in step S30, and generates control information (commands) that represent the target flight state.

[0095] Next, in step S44, the flight control device 50 transmits the control information generated in step S42 to the unmanned airplane 40 to set a new target flight state. In this case, upon receiving such control information, the control device 400 of the unmanned airplane 40 controls the unmanned airplane 40 so as to realize the target flight state related to the control information. Note that the hovering maintenance control instructed in step S34 or the like is maintained until a new target flight state is set in step S44.

[0096] In the above embodiment, the attachment position calculation unit 512 calculates the position of the end attachment corresponding to the image capture target (position of the image capture target) based on the position information, attitude information, and orientation information of the work machine 1 acquired by the information acquisition unit 510. However, the position P1 of the image capture target may be calculated (recognized), for example, by image processing based on images of the current image capture device 410 and past image capture device 410. In this case, the attachment position calculation unit 512 can acquire an image of the image capture device 410 by communication processing via the communication processing unit 404, for example, and recognize the position of the end attachment included in the image of the image capture device 410 (position of the image capture target), or recognize the position of the end attachment not included in the image of the image capture device 410 (position of the image capture target). Specifically, for example, the position of the tip attachment (position of the object to be imaged) is recognized based on the image captured by the imaging device 410, the flight state (flight position and flight attitude) at the time of image capture by the imaging device 410 acquired from the flight control unit 514, and the imaging angle of view and imaging angle at the time of image capture by the imaging device 410 acquired from the imaging parameter setting unit 513.

[0097] In this way, a marker may be provided at position P1 when calculating position P1 of the imaging target by image processing based on the image of imaging device 410. The position, size, and form of the marker are arbitrary, but for example, a circle with a diameter of about 30 cm can be used as the marker.

[0098] Furthermore, when a marker is provided, the position P1 of the imaging target can be set to the upper surface position of the tip of the arm 143, and the marker can be provided at this position. In this case, there is an advantage that the marker can be easily captured by the imaging device 410 at all times.

[0099] For example, during building demolition work, the tip attachment such as the crusher 145 attached to the tip of the arm 143 repeatedly swings up and down, so if a marker is provided on the tip attachment, there is a high possibility that the imaging device 410 will not be able to capture the image during work. On the other hand, the arm 143 does not swing significantly as frequently as the tip attachment, so the imaging device 410 can easily capture the marker at all times. In addition, since the target flight position for hovering maintenance control of the unmanned airplane 40 during demolition work is assumed to be a position above the tip attachment and the arm 143, the upper surface of the arm 143 is a good position for the marker. Furthermore, in order to simultaneously capture the image of the target of the demolition work, i.e., the part being demolished, with the imaging device 410 at the same time as the marker, it is good to set the marker at the tip of the arm 143 close to the tip attachment.

[0100] As described above, according to the above embodiment, even if the position of the imaging target changes, if the changed position of the imaging target can be imaged using the imaging angle of view or imaging direction of the imaging device 410 set by the imaging parameter setting unit 513, hovering maintenance control is executed to continuously maintain the flight position of the aircraft 40.This effectively avoids inconveniences such as an increase in the processing load in controlling the flight state of the unmanned airplane 40 and a shortened battery life due to increased power consumption of the unmanned airplane 40.

[0101] Although each embodiment has been described above in detail, the present invention is not limited to the specific embodiment, and various modifications and changes are possible within the scope of the claims. In addition, it is also possible to combine all or a plurality of the components of the above-described embodiments. [Explanation of symbols]

[0102] 1. Work Machinery 8 Peripheral Devices 10 Control device 40 Unmanned Aircraft 50 Flight control device 52 Remote Control Device 80 Image output device 82 Sensors 110 Undercarriage 120 Upper rotating body 122 Cab (operator's compartment) 130 Swivel Mechanism 140 Working mechanism 141 Boom 142 Boom cylinder 143 Arm 144 Arm Cylinder 145 Crusher 146 Cylinder 150 Location information acquisition section 151 Attitude information acquisition unit 152 Orientation information acquisition unit 153 Communication processing unit 154 Image output processing section 400 Control device 401 Aircraft Information Acquisition Unit 402 Target Flight State Setting Unit 403 Aircraft Control Unit 404 Communication processing unit 410 Imaging device 510 Information Acquisition Department 512 Attachment position calculation unit 513 Imaging parameter setting unit 514 Flight Control Unit

Claims

1. A work support device that controls a positional relationship between a work machine and an aircraft equipped with an imaging device, The work machine includes a lower traveling body, an upper rotating body mounted on the lower traveling body, and a display device that displays an image captured by the imaging device, The work support device includes: an information acquisition unit that acquires information representing a state of the work machine and a state of the aircraft; an imaging target position calculation unit that calculates a position of an imaging target related to the work; an imaging parameter setting unit that sets an imaging angle of view of the imaging device so that the position of the imaging target is imaged; A flight control unit that controls a flight state of the aircraft based on the information acquired by the information acquisition unit, The flight control unit: A state tracking control that dynamically changes a target flight position of the flying object in response to a change in the position of the imaging target; a hovering maintenance control that does not change the target flight position or the target flight attitude of the aircraft, the hovering maintenance control being started when a hovering start condition is satisfied; A work support device in which, when the flight control unit is executing the hovering maintenance control, if the position of the imaging target calculated by the imaging target position calculation unit falls outside the imaging angle of view of the imaging device, or is predicted to fall outside the imaging angle of view, the imaging parameter setting unit switches the imaging angle of the imaging device to a wider imaging angle of view, thereby continuing the hovering maintenance control.

2. The work support device according to claim 1 , wherein the image capture target position calculation unit calculates the position of the image capture target based on information indicating a state of the work machine acquired by the information acquisition unit.

3. The work support apparatus according to claim 1 , wherein the imaging target position calculation unit calculates the position of the imaging target through image recognition based on the image captured by the imaging device.

4. 4. The work support device according to claim 1, wherein the imaging parameter setting unit sets an imaging angle of view of the imaging device by optical means.

5. 4. The work support device according to claim 1, wherein the imaging parameter setting unit sets an imaging angle of view of the imaging device by electronic processing of an image captured by the imaging device.

Images