Control method

Abstract

This invention provides a control method that allows for easy setting of the imaging position for capturing images to support work on machinery, and enables control to direct an unmanned aerial vehicle to capture images at that set imaging position. [Solution] The present invention provides a control method for an unmanned aerial vehicle 40 having an imaging device 41 that captures images of an imaging position to assist in work by a work machine 1, comprising: a fixed position acquisition step of acquiring a fixed imaging position to be captured by the imaging device 41; and a first imaging step of capturing images of the fixed imaging position with the imaging device 41 while the unmanned aerial vehicle 40 is in flight, wherein the fixed position acquisition step is a step of specifying a position by the operation of the work machine and acquiring the specified position as the fixed imaging position, and by having the fixed imaging position acquired in the fixed position acquisition step, the unmanned aerial vehicle 40 is made to capture images of the fixed imaging position in the first imaging step regardless of the operation of the work machine 1.

Description

Technical Field

[0001] The present invention relates to a method for controlling an unmanned aerial vehicle that has an imaging device and performs imaging at an imaging position for assisting work on a work machine.

Background Art

[0002] Patent Document 1 discloses an excavator having a lower traveling body, an upper revolving body rotatably mounted on the lower traveling body, a receiving device, an orientation detection device, a control device, and a display device attached to the upper revolving body. The receiving device receives an imaging image captured by a camera-mounted autonomous flying body, the orientation detection device detects the orientation of the excavator, the control device generates a target rotation angle of the camera-mounted autonomous flying body based on the orientation of the excavator that changes with the turning operation of the upper revolving body detected by the orientation detection device, and the display device displays the imaging image in the same orientation as the image that can be captured when the camera-mounted autonomous flying body rotates by the target rotation angle.

[0003] And it is explained that by performing the above control, for example, even when the upper revolving body turns, the imaging image captured by the camera-mounted autonomous flying body can capture the front of the upper revolving body, and it is possible to avoid confusing the operator who is viewing the captured image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, for example, when working with a work machine having a crusher or the like attached to a tip attachment, the position suitable for imaging to assist the work may not be the tip attachment of the work machine.

[0006] For example, when demolishing a building, instead of imaging the crusher attachment, it may be more beneficial to image the demolition site itself, without being limited by the movement of the crusher, to obtain images that are useful for the work.

[0007] However, the exact location where the demolition is taking place is unclear, as it is similar to the tip attachment of a work machine in that it is not precisely positioned relative to the machine's reference point.

[0008] Furthermore, if the location of the demolition site itself is surveyed, and the image position is input using a remote control device for the unmanned aerial vehicle based on the survey results, the work efficiency will be significantly reduced.

[0009] This invention has been made in view of these circumstances, and one of its objectives is to provide a control method that allows for easy setting of an imaging position for capturing images to support work with a work machine, and that enables control to cause an unmanned aerial vehicle to capture images at the set imaging position. [Means for solving the problem]

[0010] To achieve the above objective, the present invention is understood by the following configuration. The present invention provides a control method for an unmanned aerial vehicle that has an imaging device and performs imaging of an imaging position to assist in work by a work machine, A fixed position acquisition step for acquiring the fixed imaging position to be imaged by the imaging device, A first imaging step includes capturing an image of a fixed imaging position with the imaging device while the unmanned aerial vehicle is in flight, The fixed position acquisition step is a step of specifying a position by the operation of the work machine and acquiring that specified position as the fixed imaging position. By acquiring a fixed imaging position in the fixed position acquisition step, the unmanned aircraft is made to perform imaging of the fixed imaging position in the first imaging step, regardless of the operation of the work machine. [Effects of the Invention]

[0011] According to the present invention, a control method is provided that allows for easy setting of an imaging position for capturing images to support work on a work machine, and enables control to cause an unmanned aerial vehicle to capture images at the set imaging position. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic plan view showing an unmanned aerial vehicle according to the first embodiment of the present invention performing imaging to assist in the operation of a work machine. [Figure 2] This is a block diagram showing the hardware configuration of a control device according to the first embodiment of the present invention. [Figure 3] This is a flowchart showing a series of steps in the control method for an unmanned aerial vehicle according to the first embodiment of the present invention. [Figure 4] This figure shows the display on the display unit when setting the flight position of an unmanned aerial vehicle according to the first embodiment of the present invention. [Figure 5] This is a flowchart showing a series of steps in the control method for an unmanned aerial vehicle according to a second embodiment of the present invention. [Modes for carrying out the invention]

[0013] Hereinafter, embodiments for carrying out the present invention (hereinafter referred to as "embodiments") will be described in detail with reference to the attached drawings. Throughout the description of the embodiments, the same elements are assigned the same numbers or reference numerals.

[0014] However, please note that, for the sake of readability in the drawings, not all identical elements are assigned numbers or symbols, and some elements do not have numbers or symbols assigned.

[0015] <<First Embodiment>> This section describes a control method for an unmanned aerial vehicle 40 that performs imaging of an imaging position to assist in the work of a work machine 1 according to the first embodiment of the present invention.

[0016] FIG. 1 is a plan view schematically showing an imaging operation in which an unmanned aerial vehicle 40 according to a first embodiment of the present invention supports the operation of a work machine 1.

[0017] Note that FIG. 1 is a view of the left side of the work machine 1 when viewed from the rear side when the side where the cab 21 of the upper swing body 20 is located is the front and the opposite side is the rear.

[0018] Also, in the following description of the upper swing body 20, the side where the cab 21 is located is the front, the opposite side is the rear, and when viewed from the rear side, the left side when the upper swing body 20 is viewed is the left and the right side is the right.

[0019] Note that since the upper swing body 20 swings with respect to the lower traveling body 10, the definition of the direction of the upper swing body 20 and the definition of the direction of the lower traveling body 10 may not match.

[0020] Therefore, in the lower traveling body 10, hereinafter, the direction will be described based on the traveling direction in the normal traveling operation.

[0021] Specifically, in the normal traveling operation of the lower traveling body 10, the forward direction will be the forward traveling direction (front of the forward direction, front side of the forward direction, etc.), and conversely, the backward direction will be the backward traveling direction (front of the backward direction, front side of the backward direction, etc.) for the description.

[0022] Also, hereinafter, the case where the work performed by the work machine 1 is the demolition work of the building BL will be described as an example, but the work performed by the work machine 1 does not necessarily have to be limited to the demolition work.

[0023] Therefore, in the first embodiment, the case where the tip attachment 33 of the work machine 1 is a crusher is shown, but the tip attachment 33 may be appropriately selected according to the work performed by the work machine 1 and is not limited to the crusher.

[0024] [Work Machine 1] As shown in Figure 1, the work machine 1, which is assisted by the unmanned aerial vehicle 40, comprises a lower traveling body 10, an upper rotating body 20 that is rotatable relative to the lower traveling body 10, and a work mechanism 30 that is rotatable relative to the upper rotating body 20. In some cases, the term "main body of the work machine 1" refers to the combined portion of the lower traveling body 10 and the upper rotating body 20.

[0025] (Lower running body 10) As shown in Figure 1, the lower vehicle 10 is a crawler-type vehicle in the first embodiment, but it is not limited to a crawler-type vehicle.

[0026] (Upper rotating body 20) The upper rotating body 20 includes a driver's cab 21 where the operator sits to control the vehicle, and a machine room 22 located behind the driver's cab. Furthermore, the driver's cab 21 is equipped with, for example, an operating section (not shown) for the operator to perform various operations on the lower traveling body 10, the upper rotating body 20, and the work mechanism 30.

[0027] Furthermore, the driver's cab 21 is equipped with a communication device for communication (wired communication and wireless communication) with the control device 50 described later. This communication device includes a connection port for connecting a communication line for wired communication, and a wireless antenna for wireless communication. Regarding the radio antenna, it may be installed on the outside of the driver's cab 21.

[0028] Although not shown in the diagram, the upper rotating body 20 is equipped with various sensors such as a GNSS (GPS may also be used) and a compass, which allow for the acquisition of the latitude, longitude, altitude of the reference position of the upper rotating body 20 (for example, a predetermined position on the rotation center R in Figure 4), and the direction of the upper rotating body 20 to the front.

[0029] Furthermore, since the base end of the boom 31 (described later) is located at a fixed position on the upper slewing body 20, the latitude, longitude, and altitude of the base end of the boom 31 (described later) can be determined from the latitude, longitude, and altitude acquired by various sensors installed on the upper slewing body 20.

[0030] Furthermore, since the forward orientation of the upper slewing body 20 is also the forward orientation of the work mechanism 30 (boom 31, arm 32, and tip attachment 33), the forward orientation of the work mechanism 30 (boom 31, arm 32, and tip attachment 33) can be obtained from the forward orientation of the upper slewing body 20 acquired by various sensors installed on the upper slewing body 20.

[0031] The latitude, longitude, altitude of the reference position of the upper rotating body 20, as well as the forward direction of the upper rotating body 20, are transmitted to the control device 50 (described later) via wireless communication using the communication device installed in the driver's cab 21, or via wired communication.

[0032] In the first embodiment, the latitude, longitude, altitude of the reference position of the upper rotating body 20, and the forward direction of the upper rotating body 20 are transmitted to the control device 50 at regular intervals via the communication device installed in the driver's cab 21 described above. However, the data may be transmitted only when a transmission request is received from the control device 50.

[0033] (Working mechanism 30) The working mechanism 30 includes a boom 31 that is provided to be able to raise and lower relative to the upper rotating body 20, an arm 32 whose base end is provided to be able to rotate relative to the tip of the boom 31, and a tip attachment 33 (a crusher in this example) whose base end is provided to be able to rotate relative to the tip of the arm 32.

[0034] In the first embodiment, the working mechanism 30 is shown to include a boom 31 and an arm 32, but for example, a member for changing the working range may be provided between the boom 31 and the arm 32.

[0035] The aforementioned luffing and rotating movements are performed by hydraulic cylinders, and the crusher, which is the tip attachment 33, has its claws opened and closed by hydraulic cylinders that open and close the claws provided on the crusher.

[0036] Although not shown in the diagram, the boom 31, arm 32, and tip attachment 33 are each provided with angle sensors (boom angle sensor, arm angle sensor, and tip attachment angle sensor) for detecting the tilt angle.

[0037] Specifically, the boom angle sensor acquires the tilt angle of the boom 31 relative to the upper slewing body 20, the arm angle sensor acquires the tilt angle of the arm 32 relative to the boom 31, and the tip attachment angle sensor acquires the tilt angle of the tip attachment 33 relative to the arm 32.

[0038] As described above, the latitude, longitude, and altitude of the base end of the boom 31 (described later) can be determined, and the lengths from the base end to the tip of the boom 31, arm 32, and tip attachment 33 are known.

[0039] Furthermore, as described above, the orientations of the boom 31, arm 32, and tip attachment 33 (working mechanism 30) can also be determined, and from these, the position (latitude, longitude, and altitude, etc.) of the tip of the tip attachment 33 can be determined.

[0040] The lengths from the base end to the tip end of the boom 31, arm 32, and tip attachment 33 are registered in the control device 50, which will be described later.

[0041] [Unmanned aircraft 40] The unmanned aerial vehicle 40 is a so-called drone, etc., equipped with an imaging device 41 (e.g., a video camera) for taking images to assist in the work of the work machine 1.

[0042] Furthermore, the unmanned aerial vehicle 40 is equipped with various sensors, such as GNSS (GPS may also be used) and a compass, in order to obtain its current flight position and other information.

[0043] The imaging device 41 provided on the unmanned aerial vehicle 40 may be, for example, a 3-axis gimbal camera, in which case image blurring caused by shaking of the unmanned aerial vehicle 40 is suppressed, and panning and tilting can be performed.

[0044] The imaging device 41 determines how much panning or tilting to perform based on the positional information (latitude, longitude, altitude, etc.) measured by various sensors of the unmanned aircraft 40, and the imaging position specified by latitude, longitude, and altitude, and then performs panning or tilting so that the specified imaging position can be imaged.

[0045] However, the imaging device 41 is not limited to determining how much panning or tilting it should perform. Since the positional information (latitude, longitude, altitude, etc.) measured by the various sensors of the unmanned aerial vehicle 40 is transmitted to the control device 50 (described later) at regular intervals, the control device 50 may determine how much panning or tilting the imaging device 41 should perform and transmit this as a control command to the unmanned aerial vehicle 40.

[0046] Furthermore, the imaging device 41 is not limited to a gimbal camera; it may be a fixed type that does not perform panning or tilting.

[0047] In this case, the unmanned aircraft 40 itself can change its attitude or other characteristics so that the orientation of the imaging device 41 is directed towards the imaging position.

[0048] Furthermore, if the unmanned aerial vehicle 40 itself changes its attitude or other characteristics for imaging with the imaging device 41, the unmanned aerial vehicle 40 may determine how much its attitude should change, as in the case of a gimbal camera, or the control device 50, described later, may determine this and transmit it to the unmanned aerial vehicle 40 as a control command.

[0049] Furthermore, the unmanned aerial vehicle 40 is equipped with a wireless communication device (not shown) that communicates with the control device 50, which will be described later. The wireless communication device (not shown) receives various control commands transmitted from the control device 50, and the unmanned aerial vehicle 40 performs flight and imaging based on the received control commands.

[0050] Furthermore, in the first embodiment, the unmanned aerial vehicle 40 transmits flight information of the unmanned aerial vehicle 40 (flight position and direction such as latitude, longitude, and altitude, and flight attitude such as inclination) and image data captured by the imaging device 41 to the control device 50 at regular intervals via a wireless communication device (not shown). However, the transmission may be configured to occur when a transmission request is received from the control device 50.

[0051] Furthermore, the unmanned aerial vehicle 40 may be equipped with a wired communication device that supports wired communication, and communication may be enabled by connecting a communication line of a length that does not interfere with flight to the wired communication device.

[0052] For example, since the unmanned aerial vehicle 40 flies alongside the work machine 1, if communication is to be performed via wired communication, the communication line may be connected to the communication device (not shown) installed in the driver's cab 21 as described earlier, and communication with the control device 50 may be performed via the communication device (not shown) installed in the driver's cab 21.

[0053] Then, as will be explained later, the unmanned aerial vehicle 40 (specifically the imaging device 41) performs a first imaging step (also called the first imaging mode) in which the imaging device 41 continues to image the fixed imaging position acquired in the fixed position acquisition step, which acquires a fixed imaging position to be imaged by the imaging device 41, regardless of the operation of the work machine 1, during flight, and a second imaging step (also called the second imaging mode) in which the imaging device 41 images the imaging position in accordance with the rotation of the work machine 1 during flight.

[0054] As will be explained in more detail later, the operator controlling the work machine 1 makes a selection as to whether to have the unmanned aerial vehicle 40 perform the first imaging process or the second imaging process, and the unmanned aerial vehicle 40 performs either the first imaging process or the second imaging process according to the operator's selection.

[0055] More specifically, the control device 50, which controls the overall operation of the unmanned aerial vehicle 40, causes the imaging device 41 of the unmanned aerial vehicle 40 to perform either a first imaging process or a second imaging process, according to the selection of an operator or the like.

[0056] [Control device 50] Figure 2 is a block diagram showing the hardware configuration of the control device 50 according to the first embodiment of the present invention.

[0057] Furthermore, the control device 50 does not need to be separate from the work machine 1, as in the first embodiment, and may be configured as a controller attached to the work machine 1.

[0058] Alternatively, some functions of the control device 50 may be provided in a controller attached to the work machine 1, and the control device 50 may have the remaining functions.

[0059] Thus, the configuration of the control device 50 itself is not particularly limited, including which components of the control device 50 are attached to the work machine 1.

[0060] The control device 50 is responsible for the overall control of the unmanned aerial vehicle 40, including its flight status (autonomous flight and remotely controlled flight) and the imaging status (imaging status by the imaging device 41). As shown in Figure 2, it comprises a control unit 51, a storage unit 52, a communication unit 53, a remote control unit 54, an operation unit 55, and a display unit 56, which are connected to each other by a bus BS so that data can be transmitted between them.

[0061] (Control Unit 51) The control unit 51 sends control commands to the unmanned aerial vehicle 40, including the imaging device 41, according to the program stored in the memory unit 52 (described later), and is in charge of the overall control of the unmanned aerial vehicle 40, such as the flight state and imaging state of the unmanned aerial vehicle 40. Specifically, the control unit 51 is a CPU (Central Processing Unit), but it may also be an MPU and / or a GPU, or a VLSI that also has memory functionality.

[0062] (Storage unit 52) The memory unit 52 includes a non-volatile memory called ROM (Read Only Memory) for storing programs and parameters that do not require modification, and a volatile memory called RAM (Random Access Memory) for storing data that needs to be temporarily stored, generated by calculations performed by the control unit 51. Furthermore, the storage unit 52 may include a hard disk or the like that supports saving new parameters and data, as well as overwriting existing data.

[0063] The memory unit 52 stores data transmitted from the unmanned aerial vehicle 40 and the work machine 1. For example, the data transmitted from the unmanned aerial vehicle 40 and the work machine 1 may include flight information of the unmanned aerial vehicle 40, the latitude, longitude, and altitude of the reference position of the upper rotating body 20 (for example, a predetermined position on the rotation center R), and the forward direction of the upper rotating body 20.

[0064] Furthermore, the memory unit 52 stores the lengths from the base end to the tip end of each of the boom 31, arm 32, and tip attachment 33, as described earlier.

[0065] In the first embodiment, the case where the length is stored in the memory unit 52 is shown, but for example, the lengths of the boom 31, arm 32, and tip attachment 33 may be stored in external memory or the cloud, and the lengths may be retrieved via the communication unit 53 described later.

[0066] Furthermore, the length obtained from external memory or the cloud via the communication unit 53 (described later) may be stored in the storage unit 52 at a predetermined timing.

[0067] Furthermore, the lengths of the boom 31, arm 32, and tip attachment 33 may be stored in a portable memory device, such as a USB memory stick, which functions as a memory unit 52 when attached to the control device 50.

[0068] (Communications Section 53) The communication unit 53 includes a wireless communication unit for wireless communication with the unmanned aerial vehicle 40 and the work machine 1, and a wired communication unit for wired communication with the work machine 1. Furthermore, the communication unit 53 also includes connection ports for connecting communication lines when performing wired communication.

[0069] For example, since the control device 50 is portable, when it is brought into the driver's cab 21 of the upper rotating body 20 for use, a communication line may be connected to the communication device inside the driver's cab 21 for wired communication.

[0070] The communication unit 53 then receives data such as latitude, longitude, altitude, and direction acquired by various sensors installed on the upper rotating body 20 at regular intervals.

[0071] Furthermore, the communications unit 53 receives flight information of the unmanned aircraft 40 (flight position and direction such as latitude, longitude, and altitude, and flight attitude such as inclination) at regular intervals.

[0072] Furthermore, the communication unit 53 transmits data (also called imaging control information) to the unmanned aerial vehicle 40 at regular intervals, including the flight position (including attitude) in which the unmanned aerial vehicle 40 is to be flown and the direction in which the imaging device 41 is to take images.

[0073] (Remote control unit 54) The remote control unit 54 is an operating unit used when an operator or the like wants to directly remotely control the flight of the unmanned aircraft 40, and as shown in Figure 1, it is equipped with a pair of sticks 54A located on the left and right when the control device 50 is held in the hand. The operation of this pair of 54A sticks is similar to that of the sticks on a typical drone transmitter.

[0074] (Operation unit 55) The control unit 55 is an operation unit in which an operator or the like performs operations such as selecting and setting the flight status of the unmanned aerial vehicle 40 during autonomous flight (for example, the flight position relative to the work machine 1), inputting settings for imaging by the imaging device 41, and turning the power ON and OFF. As shown in Figure 1, it is equipped with a plurality of operation buttons 55A, etc. The control unit 55 is also operated when an operator or other person selects whether to remotely control the flight of the unmanned aircraft 40 or to perform autonomous flight.

[0075] (Display section 56) The display unit 56 is a so-called monitor, which displays, for example, images captured by the imaging device 41. Furthermore, the display unit 56 displays information (such as menus) necessary when an operator or other person operates the control unit 55 to perform the above-mentioned selections, setting inputs, etc.

[0076] Furthermore, the display unit 56 does not need to be limited to displaying information only; it may also have a touch panel function and, like the operation unit 55 described earlier, allow operators to select flight status and imaging status, input settings, and perform other operations.

[0077] Next, a series of steps for controlling an unmanned aerial vehicle 40 that performs imaging of an imaging position to support the work of the work machine 1 according to the first embodiment of the present invention will be described.

[0078] Figure 3 is a flowchart showing a series of steps in the control method for the unmanned aerial vehicle 40 according to the first embodiment of the present invention. It should be assumed that, prior to the start of the flowchart in Figure 3, the operator or other relevant personnel have already operated the control unit 55 of the control device 50 to select autonomous flight for the unmanned aircraft 40.

[0079] (S1) In S1, the flight position of the unmanned aircraft 40 is set by operating the control unit 55 by an operator or the like.

[0080] Figure 4 shows the display on the display unit 56 when setting the flight position of the unmanned aerial vehicle 40 according to the first embodiment of the present invention. Note that Figure 4 includes some added components for illustrative purposes, and therefore differs from the actual display on the display unit 56.

[0081] When an operator or the like operates the control unit 55 to bring up the flight position setting screen, the display unit 56 displays, as shown in Figure 4, a schematic drawing 1P of the work machine 1, and the names of the pre-set positions of the unmanned aircraft 40 (in this example, A, B, and C), along with a schematic drawing 40P of the unmanned aircraft 40 indicating its position.

[0082] In the first embodiment, as can be seen in Figure 4, three locations with different latitudes and longitudes (location A, location B, and location C) are pre-set as the locations where the unmanned aerial vehicle 40 will fly.

[0083] Specifically, position A is located on the rear side of the upper rotating body 20, at a distance Y from the rotation center R of the upper rotating body 20 (see Figure 1), and in a top view, it is in a position that coincides with the counterweight of the work machine 1 (see Figure 1).

[0084] Furthermore, position B is located on the rear side of the upper rotating body 20, at a distance Y from the rotation center R of the upper rotating body 20 (see Figure 1), and at a distance X to the left of the upper rotating body 20, in a direction perpendicular to the straight line (also called the front-rear axis) that extends from the front and rear of the upper rotating body 20 through the rotation center R.

[0085] Furthermore, position C is positioned symmetrically to position B, with respect to position A. Therefore, position C is located on the rear side of the upper rotating body 20, at a distance Y from the rotation center R of the upper rotating body 20 (see Figure 1), and at a distance X to the right of the upper rotating body 20, in a direction perpendicular to the straight line (also called the front-rear axis) that extends from the front and rear of the upper rotating body 20 through the rotation center R.

[0086] Furthermore, the flight location of the pre-configured unmanned aircraft 40 is not limited to positions A, B, and C described above; it can be determined according to the nature of the work, etc.

[0087] For example, instead of position B being the rear side of the upper rotating body 20, it could be a position directly beside the upper rotating body 20, located at a distance X from the rotation center R of the upper rotating body 20 (see Figure 1), and to the left of the upper rotating body 20.

[0088] Furthermore, in the first embodiment, three positions (position A, position B, and position C) are set in advance as the positions where the unmanned aircraft 40 will fly, but the number of positions set may be less than three or four or more.

[0089] Then, through the operation of the control unit 55 by an operator, as shown in Figure 4, the flight position of the unmanned aerial vehicle 40 is set by selecting which of positions A, B, and C will be the horizontal position to which the unmanned aerial vehicle 40 will fly (in this example, position B) and by selecting the vertical position to which the unmanned aerial vehicle 40 will fly (in this example, an altitude of 15m).

[0090] In the following explanation, we will assume that the flight position for the unmanned aircraft 40 is set to position B and the altitude to 15m.

[0091] Once the flight position of the unmanned aerial vehicle 40 is set, the control device 50 performs a flight control process to control the flight position of the unmanned aerial vehicle 40 in synchronization with the movement of the work machine 1, so that the unmanned aerial vehicle 40 flies at the set flight position.

[0092] For example, when the work machine 1 moves forward due to the drive of the lower traveling body 10, the control device 50 performs a flight position determination process to determine the left rearward position (latitude, longitude, etc.) of the upper rotating body 20 described above again based on the position information of the work machine 1, and then performs a flight control process to fly the unmanned aerial vehicle 40 in the forward direction so that the unmanned aerial vehicle 40 is positioned at the determined position.

[0093] Furthermore, if, for example, the work machine 1 itself rotates (rotation by the lower traveling body 10) and the upper rotating body 20 rotates, the flight position determination process is performed again to determine the left rearward position (latitude, longitude, etc.) of the upper rotating body 20 as described above, based on the rotation information of the work machine 1 (information on rotation by the lower traveling body 10 and rotation of the upper rotating body 20), and a flight control process is performed to make the unmanned aerial vehicle 40 rotate so that it is positioned at the determined position.

[0094] Furthermore, if the attitude of the work machine 1 changes due to, for example, the slope of the ground, the control device 50 performs a flight position determination process to determine a flight position in which the attitude of the unmanned aerial vehicle 40 will change by the amount of the change in the attitude of the work machine 1, and then performs a flight control process to change the attitude of the unmanned aerial vehicle 40 so that it is positioned at the determined position.

[0095] Regarding the flight position in terms of attitude, for example, if it is tilted upward, the flight position will be such that the center of the unmanned aircraft 40 maintains position B, while the front of the unmanned aircraft 40 is positioned above position B, and the rear of the unmanned aircraft 40 is positioned below position B.

[0096] (S2) In S2, the control device 50 calculates the latitude and longitude of the flight position set in S1 (in this example, position B, altitude 15m), and sends a control command to the unmanned aircraft 40 to fly to the position at the calculated latitude and longitude, where the altitude is 15m. The unmanned aircraft 40 then flies toward the flight position specified by the control command. Furthermore, once the unmanned aircraft 40 reaches its flight position, it maintains that position until it receives instructions for a new flight position from the control device 50.

[0097] More precisely, the control device 50 transmits flight position instructions at regular intervals, and as the same flight position instruction continues to be transmitted until it becomes necessary to change the flight position, the unmanned aircraft 40 maintains flight at the same position.

[0098] (S3) As shown in Figure 3, in S3, the operator or other person operates the control unit 55 to set whether the imaging performed by the unmanned aerial vehicle 40 will be in the first imaging mode or the second imaging mode.

[0099] Then, as will be described later, once the first imaging mode is set, the control device 50 causes the imaging device 41 of the unmanned aerial vehicle 40 to perform a first imaging process to capture images at a fixed imaging position, regardless of the operation of the work machine 1.

[0100] Furthermore, as described later, when the second imaging mode is set, the control device 50 causes the imaging device 41 of the unmanned aircraft 40 to perform a second imaging process in which it captures an image at an imaging position that matches the rotation of the work machine 1.

[0101] (S4) In S4, the control device 50 determines whether the first imaging mode is set. If the first imaging mode is set, the determination is YES and the process proceeds to S5. If the first imaging mode is not set, the determination is NO and the process proceeds to S8.

[0102] Furthermore, if the first imaging mode is set, it is preferable that the imaging device 41 of the unmanned aerial vehicle 40 be configured to image the area around the tip attachment 33 until step S6, described later, is started.

[0103] (S5) In S5, the control device 50 performs a fixed position acquisition process to acquire a fixed imaging position to be imaged by the imaging device 41 of the unmanned aircraft 40.

[0104] Specifically, the operator operates the work mechanism 30 of the work machine 1 to adjust the position (tip position) of the tip attachment 33 of the work machine 1 to the position PT (see Figure 1) that they want to be the fixed imaging position. Then, the operator or the like operates the control unit 55 to instruct the control device 50 to acquire that position as the fixed imaging position.

[0105] As explained earlier in step S4, the imaging device 41 of the unmanned aerial vehicle 40 is configured to image the area around the tip attachment 33. Therefore, the image displayed on the display unit 56 shows the position of the tip attachment 33 of the work machine 1 (tip position) to the desired fixed imaging position PT (see Figure 1), making it easier for the operator to align the tip attachment 33.

[0106] Then, in accordance with the instruction to acquire a fixed imaging position, the control device 50 determines (calculates) the latitude, longitude, and altitude of the tip attachment 33 of the work machine 1, and acquires the determined latitude, longitude, and altitude as a fixed imaging position. The control device 50 then performs a process to register (store) the acquired fixed imaging position in the storage unit 52.

[0107] Thus, the fixed position acquisition process involves specifying the position PT (see Figure 1) by aligning the tip attachment 33 to the position PT that is to be used as the fixed imaging position, and then acquiring that specified position PT as the fixed imaging position.

[0108] In other words, in the first embodiment, the position PT is specified by aligning the tip attachment 33 with the position PT that is to be fixed as the imaging position.

[0109] (S6) In S6, the control device 50 transmits a control command to the unmanned aerial vehicle 40 to start imaging the fixed imaging position acquired in the fixed position acquisition step of S5, causing the imaging device 41 of the unmanned aerial vehicle 40 to execute a first imaging step to image the fixed imaging position.

[0110] Specifically, the control device 50 transmits the latitude, longitude, and altitude of a fixed imaging position to the unmanned aerial vehicle 40, and the imaging device 41 starts imaging by panning, tilting, etc., so that the imaging center (see the dashed line CA in Figure 1) is positioned at that location. Furthermore, as mentioned above, it is not necessary to limit the imaging device 41 itself to determining its orientation and tilt angle so that it faces a fixed imaging position, and to performing imaging at that fixed imaging position.

[0111] For example, the control device 50 may determine how much the imaging device 41 needs to pan or tilt to fix the imaging center to a fixed imaging position, and then transmit a control command to the unmanned aerial vehicle 40 to instruct the imaging device 41 to pan or tilt, and the imaging device 41 may perform the pan or tilt according to that control command.

[0112] Furthermore, even while this first imaging process is being performed, the control device 50 transmits flight position instructions at regular intervals.

[0113] As explained earlier, the control device 50 performs a flight position determination process to determine a new flight position that corresponds to the change in the state of the work machine 1, based on the position information, rotation information, and attitude information of the work machine 1, and transmits this new flight position to the unmanned aircraft 40.

[0114] In other words, the flight position is set as a relative position based on the work machine 1, and therefore changes in accordance with the operation of the work machine 1.

[0115] For example, in this example, position B is set (selected) by an operator, etc., but if the upper rotating body 20 rotates clockwise when viewed from above, position B shown in Figure 4 will also change to a position rotated clockwise by the rotation angle around the rotation center R.

[0116] In this case, as described above, the control device 50 performs a flight position determination process to determine a new flight position that has changed clockwise, and transmits that new flight position to the unmanned aircraft 40.

[0117] As a result, the unmanned aircraft 40 will receive a new flight position, and flight control will be performed to fly at that newly received flight position.

[0118] On the other hand, when the aircraft changes to a new flight position, it may become impossible to capture images at a fixed position unless the imaging settings are adjusted.

[0119] In other words, if the direction in which the imaging device 41 is imaging remains the same as before, there may be cases where, at the new flight position, there is no fixed imaging position within the imaging range.

[0120] Therefore, regardless of changes in the state (operation) of the work machine 1, the imaging device 41 maintains a state in which it can always capture images at a fixed imaging position by panning, tilting, etc., even while the unmanned aircraft 40 is flying toward a new flight position.

[0121] Therefore, imaging of the fixed imaging position is continued while the unmanned aircraft 40 is flying toward the new flight position, and even after it has reached the new flight position.

[0122] In this way, during the first imaging process, the imaging process continues to capture images at the fixed imaging position acquired in S5, regardless of the operation of the work machine 1.

[0123] (S7) In S7, the control device 50 determines whether an operator or the like has performed an operation to terminate the first imaging process by operating the operation unit 55.

[0124] If the operation to terminate the first imaging process has not been performed, the determination is set to NO, the system returns to S6, and the control device 50 continues the first imaging process.

[0125] On the other hand, if the operation to terminate the first imaging process has been performed, the processing of the series of flowcharts will be completed, and the system will once again wait for the next instruction to be given by the operator or the like by operating the control unit 55.

[0126] For example, while the flight control of the unmanned aircraft 40 to fly at the flight position set in S1 continues, the control of the imaging device 41 to capture images at a fixed imaging position is terminated.

[0127] In other words, the imaging device 41 terminates by simply releasing the controls that perform panning, tilting, etc., in order to capture an image at a fixed imaging position.

[0128] Furthermore, if, after the S7 determination is YES, the next operation performed by the operator or other personnel is again autonomous flight, the process will return to the beginning of the flowchart in Figure 3.

[0129] (S8) In S8, the control device 50 performs a second imaging step in which it images an imaging position predetermined based on the work machine 1.

[0130] As a specific example, a predetermined imaging position is set in advance at a predetermined distance from the rotation center R to the front of the upper rotating body 20 and at a predetermined altitude, and this predetermined imaging position is stored in the memory unit 52.

[0131] Furthermore, since this predetermined imaging position is a relative position with respect to the upper rotating body 20, it changes due to movements such as rotation and travel of the work machine 1.

[0132] For example, if the upper rotating body 20 rotates clockwise when viewed from above, the predetermined imaging position also changes clockwise. Therefore, the imaging device 41 performs panning, tilting, etc., to image the predetermined imaging position that has changed clockwise.

[0133] Specifically, the control device 50 determines a predetermined imaging position (latitude, longitude, etc.) that changes in accordance with the movement of the work machine 1, and transmits the determined predetermined imaging position (latitude, longitude, etc.) to the unmanned aerial vehicle 40.

[0134] Then, the unmanned aerial vehicle 40 receives this information about the change in imaging position in real time, and the imaging device 41 performs actions such as panning and tilting to image the new predetermined imaging position received by the unmanned aerial vehicle 40.

[0135] Furthermore, since the second imaging mode is often selected when it is desired to capture a wide area in front of the upper rotating body 20, a wider angle of view may be selected than that used during the first imaging process.

[0136] Thus, in the second imaging step, the imaging device 41 captures images of the imaging position that changes in accordance with the rotation of the work machine 1 (for example, rotation by the lower traveling body 10, rotation by the upper rotating body 20, etc.) while the unmanned aerial vehicle 40 is in flight.

[0137] In the second imaging process, similar to the first imaging process, the control device 50 performs a flight position determination process to determine a new flight position that corresponds to the change in the state of the work machine 1, based on the position information, rotation information, and attitude information of the work machine 1. This new flight position is transmitted to the unmanned aerial vehicle 40, and upon receiving the new flight position, the unmanned aerial vehicle 40 performs flight control to fly at the received new flight position.

[0138] Furthermore, while the unmanned aircraft 40 is flying toward this new flight position, the imaging device 41 maintains a state where it can constantly capture images at the predetermined imaging position by panning, tilting, etc.

[0139] The predetermined imaging position captured in the second imaging step may be the tip attachment 33 or the like, and multiple predetermined imaging positions may be provided in advance.

[0140] Thus, if multiple predetermined imaging positions for imaging in the second imaging step are provided in advance, a step (process) may be added before S8 in which an operator or the like operates the control unit 55 to set (select) which of the multiple predetermined imaging positions provided in advance will be used as the predetermined imaging position for imaging in the second imaging step.

[0141] Furthermore, if multiple predetermined imaging positions are set in advance, they may be set in association with the flight position, and imaging of these associated predetermined imaging positions may be performed in the second imaging step.

[0142] For example, if the flight position is position A as shown in Figure 4, the predetermined imaging position is a predetermined position at a predetermined altitude, a predetermined distance in front of the upper rotating body 20 from the rotation center R as described earlier. If the flight position is positions B and C as shown in Figure 4, the tip attachment 33 may be set as the predetermined imaging position.

[0143] (S9) In S9, the control device 50 determines whether an operator or the like has performed an operation to terminate the second imaging process by operating the operation unit 55.

[0144] If the operation to terminate the second imaging process has not been performed, the determination is set to NO, the system returns to S8, and the control device 50 continues the second imaging process.

[0145] On the other hand, if the operation to terminate the second imaging process has been performed, the processing of the series of flowcharts will be completed, and the system will once again wait for the next instruction to be given by the operator or the like by operating the control unit 55.

[0146] For example, while the flight control of the unmanned aircraft 40 to fly at the flight position set in S1 continues, the control of the imaging device 41 to capture images at a predetermined imaging position (an imaging position that changes in accordance with the movement of the work machine 1) is terminated.

[0147] In other words, the control that allows the imaging device 41 to pan, tilt, etc., in order to capture an image at a predetermined imaging position is simply released, and the process ends.

[0148] According to the control method for the unmanned aerial vehicle 40, which has the imaging device 41 of the first embodiment described above and performs imaging of an imaging position to support work on the work machine 1, the fixed imaging position is acquired by a fixed position acquisition step in which the position (tip position) of the tip attachment 33 of the work machine 1 is aligned with the position PT (see Figure 1) that is to be a fixed imaging position, and that position is acquired as a fixed imaging position. Therefore, there is no need to perform surveying or other work to determine that position, and a fixed imaging position can be easily acquired.

[0149] Furthermore, by acquiring a fixed imaging position in the fixed position acquisition process, the unmanned aerial vehicle 40 is made to take images of the fixed imaging position in the first imaging process, regardless of the operation of the work machine 1. This makes it easy to take images of the area being crushed by the crusher, which is the tip attachment 33, rather than the tip attachment 33 itself.

[0150] Specifically, when you want to check how far the crushing process has progressed, if the imaging position is set to the tip attachment 33, the tip attachment 33 will always be in the image, making it difficult to check the crushing status.

[0151] However, with the control method of the first embodiment, the location of the crushing can be specified as a fixed imaging position without being restricted by the movement of the tip attachment 33, so that the crushing situation can be confirmed without the tip attachment 33 appearing in the image.

[0152] Furthermore, according to the control method of the first embodiment, the control device 50 performs a flight position determination process in which it determines the flight position of the unmanned aerial vehicle 40 using at least one of the position information, rotation information, and attitude information of the work machine 1, and transmits the flight position in synchronization with the movement of the work machine 1, thereby controlling the flight of the unmanned aerial vehicle 40.

[0153] In this way, a flight control process is implemented to control the flight position of the unmanned aerial vehicle 40 in synchronization with the movement of the work machine 1, so the unmanned aerial vehicle 40 flies while maintaining its relative position relative to the work machine 1.

[0154] Therefore, the unmanned aerial vehicle 40 is designed to fly autonomously without interfering with any part of the work machine 1 (for example, the work mechanism 30, etc.).

[0155] <<Second Embodiment>> Next, with reference to Figure 5, a control method for an unmanned aerial vehicle 40 that performs imaging of an imaging position to support the work of the work machine 1 of the second embodiment according to the present invention will be described.

[0156] Figure 5 is a flowchart showing a series of steps in the control method for the unmanned aerial vehicle 40 according to the second embodiment of the present invention, and corresponds to Figure 3.

[0157] The control method of the second embodiment has the same basic configuration as the control method of the first embodiment, and in Figure 5, the same step numbers as in Figure 3 are assigned to the steps that perform the same processing. Therefore, explanations regarding points similar to those in the first embodiment may be omitted.

[0158] The second embodiment differs from the first embodiment primarily in that it allows for the setting of multiple fixed imaging positions.

[0159] In other words, the fixed position acquisition step described in S5 of the first embodiment, which involves acquiring a fixed imaging position, can be performed multiple times, making it possible to acquire multiple fixed imaging positions.

[0160] For example, in the case of crushing operations, there is a demand to frequently check not only the area where the crushing is taking place, but also the area where the crushed material held by the crusher is unloaded.

[0161] Therefore, it is preferable to register the location where the crushed material is unloaded as a fixed imaging position, and to allow the imaging position to be changed by a switching operation. The control method of the second embodiment is designed to accommodate such imaging.

[0162] (T1) In T1, the control device 50 determines whether an operator or the like has performed an operation to terminate the process of acquiring a fixed imaging position by operating the operation unit 55. If the operation to terminate the acquisition of a fixed imaging position has not been performed, the determination is set to NO, and the process returns to S5, where the acquisition of the next fixed imaging position is performed. The procedure for acquiring the next fixed imaging position, which is performed by returning to S5, is the same as that described in the first embodiment.

[0163] In other words, the operator operates the work mechanism 30 of the work machine 1 to adjust the position of the tip attachment 33 of the work machine 1 to the desired fixed imaging position, and then the operator instructs the control device 50 to acquire that position as a fixed imaging position by operating the control unit 55.

[0164] The control device 50 then determines (calculates) the latitude, longitude, and altitude of the tip attachment 33 of the work machine 1, and acquires the determined latitude, longitude, and altitude as a fixed imaging position.

[0165] The control device 50 then takes the acquired fixed imaging position, assigns it a different number or other identifier (iteration) from the previously acquired fixed imaging position, and registers (stores) it in the storage unit 52 as the next fixed imaging position.

[0166] Therefore, operators can determine which fixed imaging position to use for imaging simply by operating the control unit 55 and selecting the number (iteration) assigned to the fixed imaging position.

[0167] Alternatively, instead of using numbers (iterations), images of the fixed imaging positions may be acquired, and when selecting a fixed imaging position, the acquired images of multiple fixed imaging positions may be displayed on the display unit 56, allowing the user to select the image they want to use as the fixed imaging position, thereby determining which fixed imaging position to use for imaging.

[0168] On the other hand, if the control device 50 determines that an operator or the like has performed an operation to terminate the process of acquiring a fixed imaging position by operating the operation unit 55, it will determine that the process is YES and proceed to T2.

[0169] As mentioned above, if the location to be used as a fixed imaging position is the place where the crushed material is unloaded, instead of specifying it with the tip attachment 33, the main body of the work machine 1 (the part consisting of the lower traveling body 10 and the upper rotating body 20) may be positioned at the unloading location, and that position may be acquired as the fixed imaging position.

[0170] In other words, the fixed position acquisition process performed in S5 may be a process in which the work machine 1 specifies position PT (see Figure 1) by aligning the main body of the work machine 1 with the position PT that is to be used as a fixed imaging position, and then acquires the specified position PT as a fixed imaging position. In this case, the latitude, longitude, altitude, etc. of the fixed imaging position can be determined based on the latitude, longitude, altitude, etc. measured by the GNSS installed on the upper rotating body 20.

[0171] (T2) In T2, the operator or other person operates the control unit 55 to select which of the multiple fixed imaging positions acquired will be used as the fixed imaging position to be captured.

[0172] In other words, a selection process is performed to select a fixed imaging position from among the multiple fixed imaging positions that have been acquired, and to have the imaging device 41 take an image.

[0173] Then, the control device 50 uses the fixed imaging position selected in T2 as the fixed imaging position to be imaged in the first imaging step, and executes the process of S6 described in the first embodiment.

[0174] (T3) In T3, the control device 50 determines whether an operator or the like has operated the control unit 55 to instruct a change in the imaging position from the currently imaging fixed position to another fixed imaging position.

[0175] Then, if no instruction has been given to change to another fixed imaging position, the control device 50 determines NO and proceeds to T4, and if an instruction to change has been given, it determines YES and returns to T2.

[0176] Furthermore, the content of T2, which returns when the determination is YES, is the same as T2 as explained earlier. By operating the control unit 55, a selection is made as to which of the multiple fixed imaging positions acquired will be used as the fixed imaging position to be imaged from now on.

[0177] Then, when the imaging position is changed to a different fixed imaging position, the control device 50 uses the changed fixed imaging position as the fixed imaging position to be used for imaging and executes the first imaging step in S6.

[0178] In other words, the control device 50 transmits a control command to the unmanned aerial vehicle 40 specifying a new fixed imaging position, and the imaging device 41 performs actions such as panning and tilting to image the new fixed imaging position received by the unmanned aerial vehicle 40.

[0179] Thus, in the second embodiment, the unmanned aerial vehicle 40 receives information about the change in the fixed imaging position in real time, and imaging of the fixed imaging position is performed according to the information about the change in the fixed imaging position.

[0180] (T4) The control device 50 determines whether an operator or the like has instructed the operation unit 55 to acquire a new fixed imaging position that is different from a fixed imaging position that has already been registered. If no instruction for new acquisition has been given, the judgment is NO and the process proceeds to S7. If an instruction for new acquisition has been given, the judgment is YES and the process returns to S5.

[0181] If step T4 is YES, step S5 will be performed. Therefore, in order to facilitate the acquisition of a fixed imaging position, as described in step S5 of the first embodiment, it is preferable that the imaging device 41 of the unmanned aerial vehicle 40 be set to image the area around the tip attachment 33 until step S6 is started.

[0182] Furthermore, the content of S5, which returns when the judgment is YES, is the same as that of S5 explained earlier.

[0183] In other words, the operator operates the work mechanism 30 of the work machine 1 to adjust the position of the tip attachment 33 of the work machine 1 to the desired fixed imaging position, and then the operator instructs the control device 50 to acquire that position as a fixed imaging position by operating the control unit 55.

[0184] The control device 50 then determines (calculates) the latitude, longitude, and altitude of the tip attachment 33 of the work machine 1, and acquires the determined latitude, longitude, and altitude as a fixed imaging position.

[0185] The control device 50 then takes the acquired fixed imaging position, assigns it a different number or other identifier (iteration) from the previously acquired fixed imaging positions, and registers (stores) it in the storage unit 52 as a new fixed imaging position.

[0186] As described above, according to the control method of the second embodiment, multiple fixed imaging positions can be registered, so the imaging location can be easily switched as needed.

[0187] Furthermore, it also supports setting new fixed imaging positions during operation, making it easy to add fixed imaging positions.

[0188] Although the present invention has been described above based on specific embodiments, the present invention is not limited to the above embodiments.

[0189] For example, machine 1 may be a machine that performs tasks such as a forklift. In this case, the fixed position acquisition process can be defined as a process in which the forklift (work machine) operates to align the lift's position with the desired fixed imaging position, thereby specifying the position, and acquiring that specified position as the fixed imaging position.

[0190] Furthermore, for example, the work machine 1 may be an industrial robot or other machine that performs tasks in a factory or the like. In this case, the fixed position acquisition process can be defined as a process in which the industrial robot (working machine) specifies the position by aligning the tip of the industrial robot's arm with the desired fixed imaging position, and then acquires that specified position as the fixed imaging position.

[0191] Therefore, the work machine 1 can be any machine that performs work to which the control method disclosed in the embodiment can be applied, and is not limited to the work machine 1 disclosed in the embodiment.

[0192] Thus, the present invention also includes modifications and improvements to the above embodiments, which will be clear to those skilled in the art from the claims. [Explanation of Symbols]

[0193] 1...Working machine, 10...Lower traveling body, 20...Upper rotating body, 21...Operator's cab, 22...Machine room, 30...Working mechanism, 31...Boom, 32...Arm, 33...Tip attachment, 40...Unmanned aircraft, 41...Imaging device, 50...Control device, 51...Control unit, 52...Memory unit, 53...Communication unit, 54...Remote control unit, 55...Operation unit, 56...Display unit, BS...Bus, PT...Position

Claims

1. A control method for an unmanned aerial vehicle that has an imaging device and performs imaging of an imaging position to assist in work by a work machine, A fixed position acquisition step for acquiring the fixed imaging position to be imaged by the imaging device, A first imaging step includes imaging a fixed imaging position with the imaging device while the unmanned aerial vehicle is in flight, The fixed position acquisition step is a step of specifying a position by the operation of the work machine and acquiring that specified position as the fixed imaging position. A control method that causes the unmanned aircraft to perform imaging of the fixed imaging position in the first imaging step, regardless of the operation of the work machine, by acquiring the fixed imaging position in the fixed position acquisition step.

2. The control method according to claim 1, wherein the aforementioned designation is performed by aligning the tip attachment to the desired position for the fixed imaging position.

3. The control method according to claim 2, which includes a flight position determination step of determining the flight position of the unmanned aerial vehicle using at least one of the position information, rotation information, and attitude information of the work machine.

4. The control method according to claim 3, which includes a flight control step of controlling the flight position of the unmanned aircraft in synchronization with the movement of the work machine.

5. The fixed position acquisition step is a step that can acquire multiple fixed imaging positions, A control method according to any one of claims 1 to 4, comprising a selection step of selecting a fixed imaging position from among a plurality of fixed imaging positions to be imaged by the imaging device.

6. The control method according to any one of claims 2 to 4, further comprising a second imaging step of using the imaging device to capture the imaging position aligned with at least one of the rotation or travel of the work machine while the unmanned aerial vehicle is in flight.

Images