Remote operation support device and remote operation support system

Abstract

To provide a remote operation assisting device capable of using, as a fixed camera, a camera mounted on an unmanned aerial vehicle.SOLUTION: In order to achieve such an object, the present invention includes a first assist processing element and a second assist processing element. The first assist processing element transmit, to the unmanned aerial vehicle, a landing command for landing at one of a predetermined position in a work machine and a predetermined position in a work site. The second assist processing element acquires an environment image captured by a camera of the unmanned aerial vehicle and executes a process for causing a remote operation device to output the environment image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a remote operation support device and a remote operation support system that support remote work performed by an operator operating a work machine by operating a remote control device. [Background technology]

[0002] BACKGROUND ART Conventionally, it is known to use a camera mounted on an unmanned aerial vehicle to remotely control a work machine (see, for example, Patent Document 1).

[0003] It is also known to use a camera mounted on an unmanned aerial vehicle to manage the progress of work (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-179226 [Patent Document 2] International Publication No. 2017 / 170651 Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, in order to remotely operate a work machine, image data of the work site captured by an imaging device mounted on an unmanned aircraft and an imaging device installed on a structure at the work site can be displayed.

[0006] Furthermore, in the technique of Patent Document 2, a camera mounted on an unmanned aerial vehicle is used to manage the progress of work, so that the progress of work can be easily managed.

[0007] However, while using a camera mounted on an unmanned aircraft is convenient because it allows an operator to capture images of a location they want to view (e.g., an excavation target) from various angles, unmanned aircraft are battery-powered and therefore have a limited flight time. In other words, the time that a camera mounted on an unmanned aircraft can capture images is limited. Furthermore, when operators and work managers install multiple cameras at a work site to capture images of the work site, some cameras may become unnecessary depending on the location of the work, the content of the work, and the imaging direction of the fixed cameras. In other words, the cost of cameras that are not used by operators and work managers is wasted.

[0008] In view of the above, an object of the present invention is to provide a remote operation support device and a remote operation support system that are capable of long-term imaging and do not waste costs.

[0009] Furthermore, the present invention aims to provide technology that allows an operator who remotely controls a work machine to use a real-machine imaging device (for example, a camera mounted inside the cab of the work machine, a camera fixed at the work site, etc.) as the main camera, and also allows a camera mounted on an unmanned aerial vehicle to be used as an auxiliary camera. [Means for solving the problem]

[0010] (1) In order to achieve the above object, the present invention provides: A remote operation support device for supporting an operator in remotely operating a work machine (e.g., a hydraulic excavator) using a remote operation device, The remote operation support device includes: a first support processing element; a second supporting processing element; The first support processing element transmits a landing command to an unmanned aerial vehicle (e.g., a drone) equipped with a camera in response to a designated operation on a remote input interface provided on the remote control device, to cause the unmanned aerial vehicle to land at one of a predetermined position on the work machine (e.g., a predetermined position on the top surface of the cab) and a predetermined position on the work site (e.g., a predetermined position on the ground, a predetermined position on a pole installed at the work site, a building being demolished that is the subject of demolition work, etc.); the second support processing element acquires an environmental image showing the surroundings of the work machine captured by the camera mounted on the unmanned airplane when the unmanned airplane has landed at the predetermined position, and transmits the environmental image to the remote control device so that the image output device of the remote control device can output the environmental image; It is characterized by:

[0011] (Action and effect) This allows the unmanned aircraft equipped with a camera to land at either a predetermined position on the work machine or a predetermined position at the work site, allowing the operator to continue remotely operating the work machine without worrying about the remaining battery life of the unmanned aircraft.

[0012] Furthermore, because the unmanned aircraft equipped with a camera lands at either a predetermined position on the work machine or a predetermined position at the work site, those who manage work costs can reduce the number of fixed cameras that are permanently installed at the work site, i.e., those who manage work costs can reduce the costs associated with installing fixed cameras.

[0013] Furthermore, when an unmanned aircraft equipped with a camera lands at a predetermined position on the work machine, the operator can view environmental images captured by the camera mounted on the work machine, even if the work machine is used at a work site where it is difficult to install a fixed camera.

[0014] In addition, an unmanned airplane equipped with a camera lands at one of a predetermined position on the work machine and a predetermined position at the work site.

[0015] Furthermore, since the unmanned aerial vehicle can land not only on its own vehicle (the work machine that the operator is remotely controlling) but also at a designated location at the work site, if the operator wants to view an area that is likely to be in the blind spot of the main camera (for example, a camera mounted inside the cab of the work machine) (for example, the front right based on the orientation of the seat inside the cab), the operator can use the camera mounted on the unmanned aerial vehicle as an auxiliary camera to view the area that is likely to be in the blind spot of the main camera.

[0016] (2) Furthermore, it is preferable that the first support processing element of the present invention recognizes the operating speed of the work machine, and when the operating speed of the work machine is below a predetermined value, transmits a landing command to land the unmanned aircraft at the predetermined position on the work machine.

[0017] (Action and effect) According to the present invention, the first assistance processing element recognizes the operating speed of the work machine, and if the operating speed of the work machine is below a predetermined value, sends a landing command to land the unmanned aircraft at the predetermined position on the work machine.

[0018] This allows the operator to land the unmanned aircraft at a predetermined position on the work machine when the work machine is moving at a low speed, thereby ensuring that the operator can land the unmanned aircraft on the work machine.

[0019] (3) Furthermore, it is preferable that the first support processing element of the present invention recognizes whether the operation of the work machine has stopped, and when it recognizes that the specified operation has been performed and the operation of the work machine has stopped, it sends a landing command to the unmanned aircraft to land it at the specified position on the work machine.

[0020] (Action and effect) According to the present invention, the first assistance processing element recognizes whether the operation of the work machine has stopped, and if it recognizes that the specified operation has been performed and the operation of the work machine has stopped, it sends a landing command to the unmanned aircraft to land it at the specified position on the work machine.

[0021] This allows the operator to land the unmanned aircraft at a predetermined position on the work machine when the work machine is stopped, thereby enabling the operator to land the unmanned aircraft at the predetermined position reliably and quickly.

[0022] (4) Furthermore, it is preferable that the first support processing element of the present invention recognizes one landing position designated by the operator from among multiple landing positions on the work machine using the remote input interface as a designated landing position based on communication with the remote control device, and transmits a landing command to the unmanned aircraft to land the unmanned aircraft at the designated landing position.

[0023] (Action and effect) According to the present invention, based on communication with the remote control device, the first support processing element recognizes one landing position specified by the operator using the remote input interface from among multiple landing positions on the work machine as a designated landing position, and transmits a landing command to the unmanned aircraft to land the unmanned aircraft at the designated landing position.

[0024] This allows the operator to land the unmanned aircraft at a landing location specified by the operator, allowing the operator to view any direction they wish to check when remotely operating the work machine.

[0025] (5) Furthermore, it is preferable that the first support processing element transmits to the unmanned aircraft a landing command including a command to capture an image in a predetermined imaging direction (e.g., to the left of the work machine, based on the orientation of the seat inside the cab) determined according to the specified landing position (e.g., the left end of the cab's top surface, based on the orientation of the seat inside the cab).

[0026] (Action and effect) As a result, the unmanned aircraft lands so as to capture images in a predetermined imaging direction determined according to the designated landing position, allowing the operator to visually confirm the predetermined imaging direction determined according to the designated landing position.

[0027] For example, if the specified landing position is the rightmost part of the cab zenith surface, but the unmanned aircraft lands so that it takes images to the left based on the orientation of the seat installed inside the cab, most of the captured image will be of the cab zenith surface, and the operator may find himself or herself unable to see much of the environmental image that he or she wants to see.

[0028] However, according to the present invention, when the operator operates the remote control device so that the unmanned aircraft lands on the left edge of the cab's top surface, the unmanned aircraft lands so as to capture images to the left of the work machine.

[0029] This allows the operator to see without being obstructed by the zenith surface parts (front part, right end part, rear part, central part, etc.) of the cab zenith surface except for the left end part of the cab zenith surface.

[0030] Although the top surface of the cab has been cited as an example of a portion that blocks the operator's view, examples of portions that block the operator's view are not limited to this example.

[0031] Other examples of parts that block the operator's view include attachments (boom, arm, bucket), the top surface of the upper rotating body, and exhaust equipment.

[0032] (6) Furthermore, it is preferable that the first support processing element recognizes the position of each of the work machines at the work site based on communication with the plurality of work machines present at the work site, and transmits a landing command to the unmanned aircraft to land at either the predetermined position of a designated work machine, which is one of the plurality of work machines present at the work site and which is designated by the operator via the remote input interface, or the predetermined position at the work site.

[0033] (Action and effect) As a result, the unmanned aircraft equipped with the camera lands at either the predetermined position on the designated work machine specified by the operator via the remote input interface or the predetermined position at the work site, so the operator can send a landing command to the unmanned aircraft even if the designated work machine among the multiple work machines present at the work site is switched from one work machine to another.

[0034] (7) Furthermore, it is preferable that the first support processing element recognizes the position information of the work machine in response to operation on the remote input interface provided on the remote control device, recognizes the landing position of the unmanned airplane based on communication with the unmanned airplane, and if the landing position of the unmanned airplane is not the specified position on the work machine (for example, if the landing position of the unmanned airplane is a pole installed at the work site), transmits a landing command to the unmanned airplane including a command to capture an image in the direction in which the work machine is located as seen from the camera mounted on the unmanned airplane.

[0035] (Action and effect) As a result, if the landing position of the unmanned aircraft is not the specified position on the work machine (for example, if the landing position of the unmanned aircraft is a pole installed at the work site), the camera mounted on the unmanned aircraft will capture an image in the direction in which the work machine is located, and the captured image will include an image of the work machine and its surrounding environment.

[0036] For example, the first support processing element recognizes, as the position information of the work machine, position information of the work machine in a world coordinate system and information relating to the altitude at which the work machine is located.

[0037] Furthermore, if the landing position of the unmanned airplane is a pole installed at the work site, the first support processing element recognizes the landing position of the unmanned airplane.

[0038] In this case, the first support processing element also acquires information about the altitude of the unmanned airplane as information about the landing position of the unmanned airplane.

[0039] Then, when the first support processing element recognizes the landing position of the unmanned aircraft, it sends a landing command to the unmanned aircraft, which includes a command to capture an image in the direction in which the work machine is located, as seen from the camera mounted on the unmanned aircraft.

[0040] This means that even if there is a difference in height between the landing position of the unmanned aircraft and the location of the work machine, the operator can see an environmental image that includes the work machine he or she is remotely operating and its surroundings, allowing the operator to quickly recognize the situation around the work machine.

[0041] Furthermore, the first support processing element acquires landing position information including information regarding the altitude of the unmanned aircraft, so that even if there is a depression or slope at the landing position of the unmanned aircraft, the operator can quickly recognize the situation around the work machine by taking into account information regarding this terrain.

[0042] Furthermore, since the work machine position information also includes information about the altitude at which the work machine is located, even if there are depressions or slopes at the location where the work machine is located, the operator can quickly recognize the situation around the work machine by taking into account this information about the terrain.

[0043] Therefore, according to the present invention, the operator can visually recognize an environmental image that includes the work machine that he or she is remotely operating and the surrounding area, thereby enabling the operator to quickly recognize the surrounding area of ​​the work machine.

[0044] (8) Furthermore, it is preferable that the second support processing element of the present invention recognizes, based on communication with the unmanned aircraft, the result of a judgment as to whether the remaining battery charge in the unmanned aircraft is below a predetermined value, and if it recognizes that the judgment result is positive, transmits to the remote control device information regarding a landing position that has a charging device capable of charging the battery installed in the unmanned aircraft when the unmanned aircraft lands, from among multiple landing positions on the work machine.

[0045] (Action and effect) This allows the operator to land the unmanned aircraft at a landing position that has a charging device when the remaining battery power of the unmanned aircraft is low, The operator can remotely operate the work machine for a longer period of time while charging the unmanned aerial vehicle and viewing the environmental images captured by the camera mounted on the unmanned aerial vehicle.

[0046] (9) Furthermore, it is preferable that the second support processing element of the present invention recognizes, based on communication with the work machine, whether or not there is a positioning mechanism for fixing the unmanned aircraft at a landing position on the work machine, and if the determination regarding whether or not there is a positioning mechanism for fixing the unmanned aircraft at a landing position on the work machine is positive, transmits position information regarding the landing position having the positioning mechanism and / or the landing position not having the positioning mechanism to the remote control device.

[0047] (Action and effect) This allows the operator, when landing the unmanned aircraft on a work machine, to land the unmanned aircraft at a landing position after recognizing whether or not a positioning mechanism is present, and allows the operator to select how secure the unmanned aircraft is as needed. For example, if the operator wants to land the unmanned aircraft on the work machine more quickly and the unmanned aircraft does not need to be firmly secured to the positioning mechanism, the operator can land the unmanned aircraft on the work machine more quickly. [Brief explanation of the drawings]

[0048] [Figure 1] 1 is an explanatory diagram illustrating a configuration of a remote operation support system as an embodiment of a remote operation support device according to the present invention; [Figure 2] FIG. 2 is an explanatory diagram illustrating the configuration of a remote control device. [Figure 3] FIG. 2 is an explanatory diagram relating to the configuration of a work machine. [Figure 4] 1 is an explanatory diagram illustrating a configuration of an embodiment of a positioning mechanism according to the present invention. [Figure 5] 1 is an explanatory diagram relating to the configuration of one embodiment of a landing position provided on a work machine according to the present invention; [Figure 6] 1 is a bird's-eye view of a work site according to an embodiment of the present invention; [Figure 7] FIG. 1 is an explanatory diagram showing the configuration of one embodiment of an implantation port provided on the top of a fixed structure according to the present invention. [Figure 8] FIG. 2 is an explanatory diagram of a processing flow for remote control according to an embodiment of the present invention. [Figure 9] FIG. 1 is an explanatory diagram of a processing flow including a first support process and a second support process according to an embodiment of the present invention. [Figure 10] FIG. 10 is an explanatory diagram of a processing flow including an example of a second support process according to another embodiment of the present invention. [Figure 11] FIG. 10 is an explanatory diagram of a processing flow including an example of a second support process according to another embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0049] (Configuration of remote operation support system) The remote operation support system shown in Figure 1 as one embodiment of a remote operation support device 100 configured according to the present invention is configured with a remote operation support server 10 having the remote operation support device 100 for supporting an operator in remotely operating a work machine using the remote operation device, a remote operation device 20 for remotely operating a work machine 40 remotely operated by the operator, and an unmanned aerial vehicle 50 having a UAV imaging device 512 (e.g., a camera) for capturing environmental images showing the state of the work site. The remote operation support server 10, the remote operation device 20, the work machine 40, and the unmanned aerial vehicle 50 are configured to be able to communicate with each other via a network. The mutual communication network between the remote operation support server 10 and the remote operation device 20, the mutual communication network between the remote operation support server 10 and the work machine 40, and the mutual communication network between the remote operation support server 10 and the unmanned aerial vehicle 50 may be the same or different.

[0050] Furthermore, the number of work machines 40 may be one or more. Furthermore, when there are more than one work machine 40, there may be a first work machine 40A that is the main target of remote operation by the operator, and a second work machine 40B that becomes the target of remote operation when the operator suspends remote operation of the first work machine 40A and switches the target of remote operation from the first work machine 40A. It is assumed that the first work machine 40A and the second work machine 40B are located at the same work site. Furthermore, the types of work machines 40 may be the same type of work machine, or different types of work machines.

[0051] Further, remote operation is a concept that means that the operator operates the work machine 40 from a position remote from the work machine 40 without being on board the work machine 40 .

[0052] Moreover, the term "operator" is a concept that refers to a person who operates the remote control device 20 to control the work machine 40.

[0053] (Configuration of remote operation support server) The remote operation support server 10 includes a remote operation support device 100, a database 110, and a server wireless communication device 122. The remote operation support device 100 includes a first support processing element 101 and a second support processing element 102. Each support processing element is configured by an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing (described below) on the data in accordance with the software.

[0054] The database 110 has the function of storing and retaining operation information, which is information relating to the operating status of the work machine 40. For example, when the work machine 40 is traveling, the database 110 stores and retains information relating to its traveling speed. Also, for example, when the upper rotating body 460 (see FIG. 3) of the work machine 40 is rotating, the database 110 stores and retains information relating to its rotation speed.

[0055] The database 110 may also have the function of storing and retaining information regarding the size of the work machine 40 (for example, information that the hydraulic excavator in operation is a 7-ton excavator, or information that the hydraulic excavator in operation is a 13-ton excavator, etc.).

[0056] Furthermore, the database 110 has a function to read out operation information, operation information, etc. of the work machine 40 in response to a request from the operator. Furthermore, the database 110 has a function to automatically write operation information, operation information, etc. by acquiring operation information, operation information, etc. from the work machine 40.

[0057] The database 110 also has the function of storing and retaining work position information including information about the location where the work machine 40 is performing work. For example, the database 110 uses a GNSS receiver (GNNS: Global Navigation Satellite System) to acquire information about the coordinates where the work machine 40 is performing work, thereby storing and retaining the work position information of the work machine 40. The information about the coordinates may be information about world coordinates, or may be information about local coordinates at the work site where the work machine 40 is performing work.

[0058] Furthermore, the database 110 has the function of storing and retaining landing position information, including information about positions at which the unmanned aerial vehicle 50 can land, at the work site where the work machine 40 is performing work. For example, the database 110 stores and retains information about the position of the work machine 40 as landing position information. For example, the database 110 may store and retain information about whether or not the work machine 40 has a positioning mechanism for landing the unmanned aerial vehicle 50. Furthermore, if the work machine 40 has a positioning mechanism for landing the unmanned aerial vehicle 50, the database 110 may store and retain information about the position at which the positioning mechanism is installed on the work machine 40. Furthermore, if the work machine 40 has a positioning mechanism for landing the unmanned aerial vehicle 50, the database 110 may store and retain information about how many positioning mechanisms are installed on the work machine 40. The information may be stored in memory.

[0059] Furthermore, the database 110 stores and holds information on the installation position of the landing structure 60 (for example, a pole) installed at the work site as landing position information.

[0060] Furthermore, the database 110 may store and hold information relating to the height (for example, altitude) of the landing structure 60 (for example, pole) installed at the work site as the landing position information.

[0061] The database 110 may also store information regarding whether the landing structure 60 installed at the work site has a battery for charging the unmanned aerial vehicle 50.

[0062] Database 110 may also store information regarding whether or not landing structure 60 installed at the work site has a magnet for securing unmanned aerial vehicle 50.

[0063] The database 110 also stores and retains information relating to the attributes of the work machine 40, the operation schedule of the work machine 40, the name, location, size, etc. of the work site, and environmental images showing the state of the work site where the work machine 40 is operating.

[0064] Furthermore, the database 110 may store and retain information that the work machine 40 is a hydraulic excavator, information relating to the manufacturer's name, information relating to the serial number, etc., as attributes of the work machine 40. Furthermore, if there are multiple work machines 40 within the same work site, the database 110 stores and retains identification information for identifying each work machine 40 (for example, a management number for each work machine 40 determined in advance by the work site manager).

[0065] In addition, the server wireless communication equipment 122 has the function of transmitting and receiving command signals between the remote control device 20, the work machine 40, and the unmanned aerial vehicle 50, which are used by the operator to remotely operate the work machine 40 while viewing an environmental image including the state of the work site around the work machine 40 captured by the UAV imaging device 512 possessed by the unmanned aerial vehicle 50.

[0066] (Configuration of remote control device) The remote operation device 20 comprises a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is configured with a processing device (a single-core processor or a multi-core processor, or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data in accordance with the software. The remote input interface 210 comprises a remote operation mechanism 211. The remote output interface 220 comprises an image output device 221 and a remote wireless communication device 222.

[0067] Remote input interface 210 includes a landing operation device for landing unmanned airplane 50. For example, the landing operation device is a push button, and when the operator presses the push button, unmanned airplane 50 starts landing control.

[0068] Although a push button has been given as an example of a landing control device, examples of the landing control device are not limited to push buttons. Another example of a landing control device is a dial. The operator may turn the dial to cause unmanned airplane 50 to start landing control.

[0069] Another example of a landing operation device is an operation lever included in remote operation mechanism 211. The operator may operate the operation lever to cause unmanned airplane 50 to start landing control.

[0070] Another example of a landing operation device is a foot pedal included in remote operation mechanism 211. When the operator steps on the foot pedal, unmanned airplane 50 may start landing control.

[0071] Another example of a landing operation device is one that uses remote output interface 220. In this case, the operator may tap an icon displayed on remote output interface 220, causing unmanned airplane 50 to start landing control.

[0072] Although the tap operation has been given as an example of an operator manipulating an icon, examples of an operator manipulating an icon are not limited to the tap operation. Other examples of an operator manipulating an icon include a double tap operation, a long tap operation, a swipe operation, a pinch operation, a drag operation, and a flick operation.

[0073] Another example of using the remote output interface 220 as a landing operation device is shaking the remote output interface 220. For example, if the remote output interface 220 is a tablet terminal device, the operator may shake the tablet terminal device to cause the unmanned airplane 50 to start landing control.

[0074] The remote operation mechanism 211 includes a travel operation device, a swing operation device, a boom operation device, an arm operation device, and a bucket operation device. Each operation device has an operation lever that is rotated. The operation lever (travel lever) of the travel operation device is operated to move the undercarriage 450. The travel lever may also serve as a travel pedal. For example, a travel pedal may be fixed to the base or lower end of the travel lever. The operation lever (swing lever) of the swing operation device is operated to operate the hydraulic swing motor that constitutes the swing mechanism 430. The operation lever (boom lever) of the boom operation device is operated to operate the boom cylinder 442 of the work machine 40. The operation lever (arm lever) of the arm operation device is operated to operate the arm cylinder 444 of the work machine 40. The operation lever (bucket lever) of the bucket operation device is operated to operate the bucket cylinder 446 of the work machine 40.

[0075] The operating levers constituting the remote control mechanism 211 are arranged around a seat St on which the operator sits, as shown in Fig. 2, for example. The seat St is in the form of a high-back chair with armrests. The seat St may be in the form of a low-back chair without a headrest, or a chair without a backrest, or any other form of seating portion on which the operator can sit.

[0076] A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operating lever may serve as multiple operating levers. For example, the left operating lever 2111 provided in front of the left frame of the seat St shown in FIG. 3 may function as an arm lever when operated in the forward / backward direction, and as a swing lever when operated in the left / right direction. Similarly, the right operating lever 2112 provided in front of the right frame of the seat St shown in FIG. 3 may function as a boom lever when operated in the forward / backward direction, and as a bucket lever when operated in the left / right direction. The lever pattern may be changed as desired by an operator's operating command.

[0077] 3, the image output device 221 is composed of a central image output device 2210, a left image output device 2211, and a right image output device 2212, each having a substantially rectangular screen, and arranged respectively in front of, diagonally forward left of, and diagonally forward right of the seat St. The shapes and sizes of the screens (image display areas) of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be the same or different.

[0078] 2, the right edge of the left image output device 2211 is adjacent to the left edge of the central image output device 2210 so that the screens of the central image output device 2210 and left image output device 2211 form an inclination angle θ1 (e.g., 120°≦θ1≦150°). As shown in Fig. 3, the left edge of the right image output device 2212 is adjacent to the right edge of the central image output device 2210 so that the screens of the central image output device 2210 and right image output device 2212 form an inclination angle θ2 (e.g., 120°≦θ2≦150°). The inclination angles θ1 and θ2 may be the same or different.

[0079] The screens of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be parallel to the vertical direction or may be inclined relative to the vertical direction. At least one of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be configured as a plurality of divided image output devices. For example, the central image output device 2210 may be configured as a pair of image output devices adjacent to each other above and below, each having a substantially rectangular screen. The image output devices 221 (the central image output device 2210, the left image output device 2211, and the right image output device 2212) may further include speakers (audio output devices).

[0080] The remote wireless communication device 222 included in the remote output interface 220 has the function of transmitting and receiving command signals input by an operator to remotely operate the work machine 40 .

[0081] Remote wireless communication device 222 also has the function of transmitting command signals input by the operator to remotely operate unmanned airplane 50. The command signals input to remotely operate unmanned airplane 50 include commands to land unmanned airplane 50 at a predetermined position (e.g., positioning mechanism 480) designated by the operator.

[0082] The remote wireless communication device 222 also has the function of receiving, via the remote operation support server 10, images captured by an actual machine imaging device 412 mounted on the work machine 40, which will be described later.

[0083] The remote wireless communication device 222 also has a function of receiving, via the remote operation support server 10, images captured by a UAV imaging device 512 mounted on an unmanned airplane 50, which will be described later.

[0084] (Work machine configuration) The work machine 40 refers to a work vehicle that can operate at a work site. The work machine 40 is, for example, a hydraulic excavator with a bucket.

[0085] Although a hydraulic excavator has been given as an example of the work machine 40, the example of the work machine 40 is not limited to this example. Other examples of the work machine 40 include a hybrid excavator, a crane, a lifting magnet, a cargo handling machine having a grappler, a demolition machine having a crushing device, a bulldozer having a blade, and the like.

[0086] The work machine 40 is, for example, a crawler excavator (construction machine), and as shown in Fig. 3, includes a crawler-type lower traveling body 450 and an upper rotating body 460 that is rotatably mounted on the lower traveling body 450 via a rotating mechanism 430. A cab 470 (operator's compartment) is provided on the front left side of the upper rotating body 460. An operating mechanism 440 is provided in the front center of the upper rotating body 460.

[0087] The work machine 40 is also provided with a landing marker. The landing marker is a sheet-like member that can be stretched horizontally with its outer edge fixed to the upper rotating body 460. The landing marker is, for example, a sticker that can be attached to the upper rotating body 460 and peeled off.

[0088] The landing marker has a plurality of AR markers on its upper surface that can be recognized by the unmanned aerial vehicle 50.

[0089] An AR marker is a sign used to specify a position where additional information is to be displayed in an image recognition-based AR system, and is, for example, a simple, clear black-and-white figure.

[0090] For example, multiple AR markers are arranged in combination with large and small AR markers to achieve highly accurate implantation.

[0091] In this example, the image recognition device mounted on the unmanned aircraft 50 recognizes the small AR marker as it approaches the work machine 40 from a distance, allowing it to land accurately at the center position of the landing marker.

[0092] The actual machine input interface 410 includes an actual machine operation mechanism 411, an actual machine imaging device 412, and an actual machine positioning device 414. The actual machine operation mechanism 411 includes multiple operation levers arranged around a seat inside the cab 470 in the same manner as the remote operation mechanism 211. A drive mechanism or robot is provided in the cab 470 that receives signals corresponding to the operation modes of the remote operation levers and moves the actual machine operation levers based on the received signals. The actual machine imaging device 412 is installed, for example, inside the cab 470, and captures images of the environment including at least a portion of the operating mechanism 440 through the front window and a pair of left and right side windows partitioned by a pair of left and right pillars (the symbols "L" and "R" are included in the symbols when distinguishing between left and right) on the front side of the cab 470. Some or all of the front window and side windows may be omitted. The actual machine positioning device 414 is a device that detects the position of the work machine 40 and is configured, for example, by a GNSS receiver.

[0093] The actual machine positioning device 414 has the function of acquiring information relating to the altitude at which the work machine 40 is located. For example, the actual machine positioning device 414 includes a known air pressure sensor, and acquires information relating to the altitude at which the work machine 40 is located by measuring the air pressure related to the work machine 40.

[0094] Although a barometric pressure sensor has been given as an example in which the actual machine positioning device 414 acquires information relating to the altitude at which the work machine 40 is located, the example in which information relating to the altitude at which the work machine 40 is located is not limited to this.

[0095] Another example of the actual machine positioning device 414 acquiring information regarding the altitude at which the work machine 40 is located is to acquire information regarding the altitude at which the work machine 40 is located from altitude data included in map data in GNSS.

[0096] The actual machine output interface 420 is equipped with an actual machine wireless communication device 422. Information relating to the position of the work machine 40 detected by the actual machine positioning device 414 is transmitted to the remote operation support server 10 via the actual machine wireless communication device 422.

[0097] The working attachment serving as the operating mechanism 440 includes a boom 441 mounted on the upper rotating body 460 so as to be able to be raised and lowered, an arm 443 rotatably connected to the tip of the boom 441, and a bucket 445 rotatably connected to the tip of the arm 443. The operating mechanism 440 is equipped with a boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446, each of which is made up of extendable and retractable hydraulic cylinders.

[0098] The boom cylinder 442 is interposed between the boom 441 and the upper rotating body 460 so as to extend and retract by receiving a supply of hydraulic oil to rotate the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to extend and retract by receiving a supply of hydraulic oil to rotate the arm 443 about a horizontal axis relative to the boom 441. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to extend and retract by receiving a supply of hydraulic oil to rotate the bucket 445 about a horizontal axis relative to the arm 443.

[0099] Positioning mechanism 480 (for example, landing port) has the function of fixing landed unmanned airplane 50, which will be described later.

[0100] Being fixed means that unmanned airplane 50 will not move from its landing position even if a force of a predetermined value or less is applied to unmanned airplane 50 in a predetermined direction.

[0101] For example, the positioning mechanism 480 functions as an electromagnet and secures the unmanned airplane 50 by magnetically attaching to the landing mechanism 580 of the unmanned airplane 50 .

[0102] Although magnetic adhesion has been given as an example of the function of positioning mechanism 480 to secure unmanned airplane 50, examples of the function of positioning mechanism 480 to secure unmanned airplane 50 are not limited to magnetic adhesion.

[0103] Another example of the function of the positioning mechanism 480 to secure the unmanned airplane 50 is when the positioning mechanism 480 has a recess and the landing mechanism 580 (e.g., landing legs) of the unmanned airplane 50 has a protrusion, and the protrusion fits into the recess, thereby causing the positioning mechanism 480 to secure the unmanned airplane 50.

[0104] As an example of the function of the positioning mechanism 480 to secure the unmanned airplane 50, an example has been shown in which a convex portion in the landing mechanism 580 fits into a concave portion in the positioning mechanism 480, but the function of the positioning mechanism 480 to secure the unmanned airplane 50 is not limited to this example.

[0105] Another example of the function of the positioning mechanism 480 to fix the unmanned airplane 50 is, for example, when the positioning mechanism 480 has a recess and has a landing mechanism 580 for the unmanned airplane 50, the landing mechanism 580 fits into the recess.

[0106] Positioning mechanism 480 may also have battery 4810 for charging unmanned airplane 50 (see FIG. 4). This allows battery 4810 to charge unmanned airplane 50 when the remaining battery charge of unmanned airplane 50 is below a predetermined value. An example of a method for charging unmanned airplane 50 is to use a known wireless power supply.

[0107] Furthermore, the positioning mechanism 480 may be installed at one location on the work machine 40, or at multiple locations on the work machine 40.

[0108] When the positioning mechanism 480 is provided at multiple locations on the work machine 40, as shown in FIG. 5, a front positioning mechanism 481 may be provided at the front part of the work machine 40 (for example, the front part of the top surface of the cab 470), a left positioning mechanism 482 may be provided at the left side of the work machine 40 (for example, the left side of the top surface of the cab 470), a right positioning mechanism 483 may be provided at the right side of the work machine 40 (for example, the right side of the upper rotating body 460), and a rear positioning mechanism 484 may be provided at the rear of the work machine 40 (for example, the rear of the upper rotating body 460).

[0109] 5, the symbol F indicates the front side (front side) of the work machine 40. The symbol L indicates the left side of the work machine 40. The symbol R indicates the right side of the work machine 40. The symbol B indicates the rear side (back side) of the work machine 40.

[0110] Furthermore, although an example in which the front positioning mechanism 481, the left positioning mechanism 482, the right positioning mechanism 483, and the rear positioning mechanism 484 are used has been described as an example in which the positioning mechanism 480 is installed, the example in which a plurality of positioning mechanisms 480 are installed is not limited to this example. For example, other examples in which the positioning mechanism 480 is installed include an example in which the positioning mechanism 480 is installed on the left front side of the cab 470, an example in which the positioning mechanism 480 is installed on the left rear side of the upper rotating body 460, an example in which the positioning mechanism 480 is installed on the right front side of the upper rotating body 460, and an example in which the positioning mechanism 480 is installed on the right rear side of the upper rotating body 460.

[0111] (Unmanned aircraft configuration) The unmanned aerial vehicle 50 is an aerial vehicle that cooperates with the work machine 40. For example, the unmanned aerial vehicle 50 is a known drone equipped with a UAV imaging device 512 (e.g., a camera). The unmanned aerial vehicle 50 has a UAV control device 500, a UAV input interface 510, a UAV imaging device 512, a UAV output interface 520, a UAV wireless communication device 522, a landing mechanism 580 (e.g., landing legs), and an altitude sensor (not shown).

[0112] Furthermore, unmanned airplane 50 may have an angle adjustment mechanism for adjusting the imaging angle of UAV imaging device 512. For example, an example of the angle adjustment mechanism is a ball head in which a camera mount attached to the base of unmanned airplane 50 and a camera mount head are connected by a ball joint.

[0113] The ball head also has the ability to adjust the camera's imaging angle by electrically operating the ball joint with pan (approximately 360-degree rotation), tilt (approximately 90-degree rotation), and roll (approximately 180-degree rotation).

[0114] Although a ball head has been given as an example of an angle adjustment mechanism, examples of angle adjustment mechanisms are not limited to ball heads. Other examples of angle adjustment mechanisms include known pan heads that adjust the imaging direction (yaw direction, pitch direction) of a camera.

[0115] In addition, the unmanned aircraft 50 may be remotely operated by an operator who remotely operates the work machine 40 using the remote control device 20, or by a different operator (for example, a dedicated operator for the unmanned aircraft 50).

[0116] Furthermore, the unmanned airplane 50 may fly autonomously according to a predetermined program.

[0117] The UAV imaging device 512 included in the UAV input interface 510 has the function of capturing an environmental image, which is an image showing the state of the work site including at least a part of the work machine 40.

[0118] The UAV wireless communication device 522 included in the UAV output interface 520 has the function of transmitting and receiving information including the position information of the unmanned airplane 50 to and from the remote operation support server 10.

[0119] Landing mechanism 580 has the function of landing unmanned airplane 50. As a function of landing unmanned airplane 50, landing mechanism 580 may have a magnet. If landing mechanism 580 has a magnet, unmanned aerial vehicle 50 can land on a ferrous landing surface.

[0120] For example, when the unmanned airplane 50 lands on the work machine 40, even if the work machine 40 does not have a positioning mechanism 480, the unmanned airplane 50 can land by magnetically attaching a landing mechanism 580 having a magnet to an iron part of the work machine 40 (for example, the top surface of the upper rotating body 460).

[0121] Although an example in which the landing mechanism 580 is magnetically attached to the upper surface of the upper rotating body 460 has been given as an example in which the unmanned airplane 50 lands on the work machine 40, the example in which the unmanned airplane 50 lands on the work machine 40 is not limited to this example. For example, examples include an example in which the landing mechanism 580 is magnetically attached to the side of the upper rotating body 460, an example in which the unmanned airplane 50 is magnetically attached to an iron landing structure 60, an example in which the unmanned airplane 50 is magnetically attached to an iron plate laid on the ground at the work site, etc.

[0122] The altitude sensor has a function of measuring the altitude at which the unmanned airplane 50 flies. The altitude sensor also has a function of measuring the altitude of the landing position of the unmanned airplane 50.

[0123] The altitude sensor is, for example, a barometric pressure sensor. The barometric pressure sensor measures the barometric pressure acting on the unmanned airplane 50, allowing the UAV control device 500 to obtain information regarding the altitude at which the unmanned airplane 50 flies and information regarding the altitude of the landing position of the unmanned airplane 50.

[0124] (Configuration of implantation structure) The landing structure 60 is a structure used for landing of the unmanned aerial vehicle 50. An example of the landing structure 60 is a pole installed at a work site.

[0125] Although a pole installed at a work site has been given as an example of the landing structure 60, the example of the landing structure 60 is not limited to this. Other examples of the landing structure 60 include the roof of an office installed at a work site, a gate installed at a work site, a building that is the target of demolition work at a work site for building demolition work, a building built at a height that allows a bird's-eye view of the work site, etc.

[0126] Furthermore, the landing structure 60 may be installed in one location on the work site, or may be installed in multiple locations on the work site (see FIG. 6).

[0127] Furthermore, the landing structure 60 only needs to be installed at a height that allows the UAV imaging device 512 of the unmanned airplane 50 to capture an overhead image of the work site. Therefore, the landing structure 60 does not necessarily need to be located inside the work site.

[0128] Furthermore, the landing structure 60 may have a landing port 680 on the upper surface of the landing structure 60, which is used for the unmanned aerial vehicle 50 to land (see FIG. 7).

[0129] Furthermore, although an example in which the landing structure 60 has the landing port 680 is described above in which the landing structure 60 has the landing port 680 installed on its upper surface, the example in which the landing structure 60 has the landing port 680 is not limited to this example. For example, the landing port 680 may be attached to the top of a pole installed at the work site so that the bottom surface of the landing port 680 faces the outer circumferential surface of the pole. In this case, the unmanned airplane 50 can land in a direction perpendicular to the extension direction of the pole. Therefore, in this case, even if there is no space to install the landing port 680 on the top surface of the pole, the unmanned airplane 50 can land on the landing port 680 installed on the top of the pole. In other words, the UAV imaging device 512 provided on the unmanned airplane 50 can capture images of the work site from a bird's-eye view.

[0130] As an example of attaching the landing port 680 to the top of a pole installed at a work site, an example has been given in which the bottom surface of the landing port 680 is attached so as to face the outer peripheral surface of the pole, but examples of attaching the landing port 680 to the top of a pole installed at a work site are not limited to this example.

[0131] For example, another example of attaching landing port 680 to the top of a pole installed at a work site is to attach landing port 680 so that the outer circumferential surface of landing port 680 faces the outer circumferential surface of the pole. If landing port 680 is attached to the pole in this manner, unmanned aircraft 50 can land on landing port 680 installed at the top of the pole even if there is no space on the top surface of the pole to install landing port 680.

[0132] Landing port 680 may have the function of fixing landed unmanned airplane 50, which will be described later. Landing port 680 functions as an electromagnet, and fixes unmanned airplane 50 by magnetically attaching to landing mechanism 580 of unmanned airplane 50.

[0133] The landing port 680 may also have a charging device (for example, a power source connected to the work site, a portable battery, etc. (not shown)) for charging the unmanned airplane 50. This allows the charging device to charge the unmanned airplane 50 when the remaining battery power of the unmanned airplane 50 is below a predetermined value. An example of a method for charging the unmanned airplane 50 is to use a known wireless power supply.

[0134] (function) The functions of the remote operation support system configured as described above will be explained using the flowcharts shown in Figures 8 to 11. In these flowcharts, a block "C●" is used for the sake of simplicity, and represents the transmission and / or reception of data, and represents a conditional branch in which processing in the branching direction is executed on the condition that the data is transmitted and / or received.

[0135] Using the flowchart shown in Figure 8, the process for storing the operation information, landing position information, position information, positioning mechanism information, etc. of the work machine 40 after the operator starts up the system according to the present invention in this embodiment will be described.

[0136] In the remote operation device 20, the remote control device 200 determines whether or not a system startup operation has been performed (FIG. 8 / STEP 201).

[0137] The "system activation operation" is the concept of an operator pushing the remote input interface 210 to specify the work machine 40 that he or she intends to remotely operate. Although a push operation has been given as an example of a system activation operation, examples of system activation operations are not limited to this example. Other examples of system activation operations include a tap operation, a double tap operation, a long tap operation, a swipe operation, a pinch operation, a drag operation, a flick operation, and a shake operation.

[0138] Furthermore, when a tablet terminal device is used for the remote output interface 220, other examples of system startup operations include pushing, tapping, double-tapping, long-tapping, swiping, pinching, dragging, flicking, shaking, etc., on the tablet terminal device in order for the operator to specify the work machine 40 that he or she intends to remotely operate.

[0139] If the determination result is negative (FIG. 8 / STEP 201...NO), the processing ends. On the other hand, if the determination result is positive (FIG. 8 / STEP 201...YES), the remote control device 200 transmits an environment confirmation request, which is a request to confirm the environment of the work site around the work machine 40 specified by the operator through the system startup operation, to the remote operation support server 10 via the remote wireless communication device 222 (FIG. 8 / STEP 202).

[0140] When an environment confirmation request is received by the remote operation support server 10 via the server wireless communication device 122 (FIG. 8 / C10), the remote operation support server 10 transmits the environment confirmation request to the work machine 40 specified by the operator via the server wireless communication device 122 (FIG. 8 / STEP 101).

[0141] When an environment confirmation request is received in the work machine 40 via the actual machine wireless communication device 422 (FIG. 8 / C40), the actual machine control device 400 executes processing to acquire environmental image data (FIG. 8 / STEP 401).

[0142] Environmental image data is a concept of data that includes an image that shows the surroundings of the work machine. The environmental image data may include not only an image that shows the surroundings of the work machine, but also information such as the time that the environmental image was captured.

[0143] In addition, the environmental image may be captured by the actual machine imaging device 412 or by the UAV imaging device 512.

[0144] When the real device control device 400 acquires the environmental image data, it transmits the environmental image data to the remote operation support server 10 via the real device wireless communication device 422 (FIG. 8 / STEP 402).

[0145] When the remote operation support server 10 receives environmental image data through the server wireless communication device 122 (Figure 8 / C11), the remote operation support server 10 stores the environmental image data in the database 110 and transmits the environmental image data to the remote operation device 20 through the server wireless communication device 122 (Figure 8 / STEP 102).

[0146] When environmental image data is received in the remote operation device 20 through the remote wireless communication device 222 (FIG. 8 / C21), the remote control device 200 executes processing to control the output mode for outputting the environmental image data to the image output device 221 (FIG. 8 / STEP 203).

[0147] For example, an example of STEP 203 is where the remote control device 200 executes control to split and display the environmental image included in the environmental image data on the central image output device 2210, the left image output device 2211, and the right image output device 2212.

[0148] In the remote operation device 20, the remote control device 200 recognizes the operation mode of the remote operation mechanism 211 (FIG. 8 / STEP 204), and the remote control device 200 transmits a remote operation command corresponding to the operation mode to the remote operation support server 10 via the remote wireless communication device 222 (FIG. 8 / STEP 205).

[0149] When the remote operation support server 10 receives the remote operation command through the server wireless communication device 122 (FIG. 8 / C12), the remote operation support server 10 transmits the remote operation command to the work machine 40 (FIG. 8 / STEP 103).

[0150] In the work machine 40, when the actual machine control device 400 receives the remote operation command via the actual machine wireless communication device 422 (FIG. 8 / C42), the actual machine control device 400 executes processing for controlling the operation of the operating mechanism 440, etc. (FIG. 8 / STEP 403). For example, the actual machine control device 400 executes processing for scooping up soil in front of the work machine 40 with the bucket 445, rotating the upper rotating body 460, and then dropping the soil from the bucket 445.

[0151] In the work machine 40, the actual machine control device 400 recognizes the operation information, which is information relating to the operation of the work machine, and transmits it to the remote operation support server 10 via the actual machine wireless communication device 422 (FIG. 8 / STEP 404).

[0152] In addition, in this case, in addition to operation information, the actual machine control device 400 may also transmit, via the actual machine wireless communication device 422, landing position information regarding the position at which the unmanned airplane 50 can land on the work machine 40, position information regarding the location at which the work machine 40 is located, and positioning mechanism information, which is information regarding whether the work machine 40 has a positioning mechanism 480, to the remote operation support server 10.

[0153] In addition, the positioning mechanism information may include not only information regarding whether the work machine 40 has a positioning mechanism 480, but also information regarding whether the positioning mechanism 480 has a battery 4810 used to charge the unmanned aerial vehicle 50.

[0154] When the remote operation support server 10 receives operation information, etc. through the remote wireless communication device 122 (Figure 8 / C13), the remote operation support server 10 stores the operation information, etc. in the database 110 (Figure 8 / STEP 104).

[0155] The first support process and the second support process in this embodiment will be described with reference to the flowchart shown in FIG.

[0156] In the remote operation device 20, the remote control device 200 determines whether the operator has performed a designation operation through the remote input interface 210 (FIG. 9 / STEP 211). The "designation operation" refers to the concept of the operator operating the remote input interface 210 to land the unmanned airplane 50. If the determination result is negative (FIG. 9 / STEP 211...NO), the processing ends. On the other hand, if the determination result is positive (FIG. 9 / STEP 211...YES), the remote control device 200 transmits a landing command including a command signal to land the unmanned airplane 50 to the remote operation support server 10 through the remote wireless communication device 222 (FIG. 9 / STEP 212).

[0157] The implantation command may also include a command signal that recognizes the types of different types of implantation markers and causes implantation on an implantation marker of the type specified by the operator.

[0158] In this case, the unmanned aircraft 50 lands on the type of landing marker specified by the operator, so even if location information regarding the landing position cannot be obtained, the unmanned aircraft 50 can land at any location specified by the operator.

[0159] In addition, if the landing control of the unmanned aircraft 50 is controlled by autonomous control or semi-autonomous control using autonomous control and operator operation, the unmanned aircraft 50 may start landing control by detecting a landing marker.

[0160] In addition, when the landing control of the unmanned aircraft 50 is controlled by autonomous control or semi-autonomous control using autonomous control and operator operation, the unmanned aircraft 50 may start landing control to determine the landing position by recognizing the difference in light emission intensity of a light-emitting device (e.g., an LED light) that emits light at the landing position.

[0161] In the remote operation support server 10, when the remote operation support processing element 101 included in the remote operation support device 100 receives a landing command via the server wireless communication device 122 (FIG. 9 / C14), it recognizes operation information, etc. of the work machine 40 (FIG. 9 / STEP 111). Operation information is a concept that includes information related to the operation of the work machine 40. For example, the remote operation support server 10 recognizes the operation information, etc. by referring to information, etc. related to the operation of the work machine 40 stored in the database 110 (FIG. 9 / STEP 111).

[0162] The remote operation support device 100 determines whether the operating speed of the work machine 40 is equal to or less than a predetermined value based on the acquired operation information (FIG. 9 / STEP 112). For example, if the traveling speed of the work machine 40 is equal to or less than 5 kilometers per hour, the remote operation support device 100 may determine that the operating speed of the work machine 40 is equal to or less than a predetermined value. If the determination is negative (FIG. 9 / STEP 112...NO), the remote operation support device 100 executes processing to return to the flow immediately before the determination. On the other hand, if the determination is positive (FIG. 9 / STEP 112...YES), the remote operation support device 100 transmits a landing command to the unmanned airplane 50 via the server wireless communication device 122, the landing command including a command signal for landing the unmanned airplane 50 (FIG. 9 / STEP 113).

[0163] Here, a description will be given of the arithmetic processing performed by the remote operation support device 100 in STEP 113. The arithmetic processing for the remote operation support device 100 to transmit a landing command is performed by a first support processing element 101 included in the remote operation support device 100.

[0164] In STEP 113, the first support processing element 101 sends a landing command to the unmanned aircraft 50 to land at least one of a predetermined position on the work machine 40 (e.g., the cab 470, etc.) or a predetermined position at the work site (e.g., the landing structure 60, etc.) in accordance with a specified operation on the remote input interface 210 provided on the remote control device 20.

[0165] In addition, in STEP 113, the first support processing element 101 may recognize whether the operation of the work machine 40 has stopped or not, and if it recognizes that a specified operation has been performed and that the operation of the work machine 40 has stopped, it may execute a process of sending a landing command to the unmanned airplane 50 to land at a predetermined position on the work machine 40 (e.g., the cab 470, etc.).

[0166] In addition, in STEP 113, the first support processing element 101 may, based on communication with the remote control device 20, recognize one landing position (e.g., the forward positioning mechanism 481) designated by the operator from among multiple landing positions (e.g., the forward positioning mechanism 481, the left positioning mechanism 482, the right positioning mechanism 483, the rear positioning mechanism 484, etc.) on the work machine 40 using the remote input interface 210 as a designated landing position, and perform a process of transmitting a landing command to the unmanned aircraft 50 to land the unmanned aircraft 50 at the designated landing position.

[0167] In addition, in STEP 113, the first support processing element 101 may recognize the position of each work machine 40 at the work site based on communication with the multiple work machines 40 present at the work site, and may execute a process of sending a landing command to the unmanned aircraft 50 to land the unmanned aircraft 50 on a designated work machine 40, which is one of the multiple work machines 40 present at the work site that the operator has designated via the remote input interface 210.

[0168] In addition, in STEP 113, the first support processing element 101 may recognize the position information of the work machine 40 in response to operation on the remote input interface 210 provided on the remote control device 20, recognize the landing position of the unmanned airplane 50 based on communication with the unmanned airplane 50, and if the landing position of the unmanned airplane 50 is not a predetermined position on the work machine 40 (for example, if the landing position of the unmanned airplane is a landing structure 60 installed at the work site), execute a process of transmitting a landing command to the unmanned airplane 50, including a command to capture an image in the direction in which the work machine 40 is located, as viewed from a camera (for example, a UAV imaging device 512) mounted on the unmanned airplane 50.

[0169] Here, the direction in which the work machine 40 is located does not only refer to the front, back, left, and right directions at the work site. It refers to three-dimensional directions including the height direction at the work site. For example, first, the first support processing element 101 acquires information regarding the X and Y coordinates of the work machine 40 in the world coordinate system (i.e., information regarding the X and Y coordinates at which the work machine 40 is located at the work site) based on the position information of the work machine 40 measured by the actual machine positioning device 414 (for example, the GNSS included in the actual machine positioning device 414). Furthermore, the first support processing element 101 acquires height information regarding the altitude at the position of the work machine 40 based on information regarding the X and Y coordinates of the work machine 40 as well as information regarding the atmospheric pressure detected by a barometric pressure sensor included in the actual machine positioning device 414.

[0170] An example in which the first support processing element 101 uses information related to atmospheric pressure detected by a barometric pressure sensor to acquire height information related to the altitude at the position of the work machine 40 has been shown, but examples of acquiring height information related to the altitude at the position of the work machine 40 (i.e., information related to the Z coordinate at which the work machine 40 is located at the work site) are not limited to this example. Another example of acquiring height information related to the altitude at the position of the work machine 40 is acquiring height information related to the altitude at the position of the work machine 40 based on map information stored and held in advance in the database 110.

[0171] Next, the first support processing element 101 acquires information about the X and Y coordinates of the unmanned airplane 50 in the world coordinate system (i.e., information about the X and Y coordinates at which the unmanned airplane 50 is located at the work site) based on position information of the unmanned airplane 50 measured by a UAV positioning device (not shown) (e.g., a GNSS included in the UAV positioning device) mounted on the unmanned airplane 50. In addition to the information about the X and Y coordinates of the unmanned airplane 50, the first support processing element 101 acquires height information about the altitude at the position of the unmanned airplane 50 (i.e., information about the Z coordinate at which the unmanned airplane 50 is located at the work site) based on information about the atmospheric pressure detected by a barometric pressure sensor included in the UAV positioning device.

[0172] Although an example in which the first support processing element 101 uses information about atmospheric pressure detected by a barometric pressure sensor to acquire height information about the altitude at the position of the unmanned airplane 50 has been shown, examples of acquiring height information about the altitude at the position of the unmanned airplane 50 (i.e., information about the Z coordinate at which the unmanned airplane 50 is located at the work site) are not limited to this example. Another example of acquiring height information about the altitude at the position of the unmanned airplane 50 is acquiring height information about the altitude at the position of the landing structure 60 where the unmanned airplane 50 has landed (for example, information about the height of a pole installed at the work site, information about the height of a building where the unmanned airplane 50 has landed, information about the height of another work machine 40 where the unmanned airplane 50 has landed, etc.) based on map information stored and held in advance in the database 110.

[0173] Then, based on the acquired position information including the altitude at which the unmanned airplane 50 is located, the first support processing element 101 determines whether the landing position of the unmanned airplane 50 is a predetermined position on the work machine 40, and if the landing position of the unmanned airplane 50 is not a predetermined position on the work machine, the first support processing element 101 calculates the imaging angle for imaging the work machine 40 by performing calculations related to known trigonometric functions based on the position information of the work machine 40 and the position information of the unmanned airplane 50, and sends a landing command to the unmanned airplane 50 including a command to capture an image in the direction in which the work machine 40 is located as seen from the camera mounted on the unmanned airplane 50.

[0174] The first support processing element 101 may also send a landing command to the unmanned airplane 50 that includes a command signal for zooming in or out the camera mounted on the unmanned airplane 50 after landing.

[0175] Furthermore, for example, the first support processing element 101 may obtain information relating to the height from the undercarriage 450 to the cab 470 of the work machine 40 to be imaged by referring to the database 110.

[0176] In this case, the first support processing element 101 takes into account the altitude at which the work machine 40 is located and the height from the undercarriage 450 to the cab 470 of the work machine 40, thereby being able to focus the camera mounted on the unmanned aerial vehicle 50 on the vicinity of the cab 470 of the work machine 40 to be imaged, thereby enabling a clear image of the work machine 40 and its surroundings to be captured.

[0177] In the unmanned airplane 50, when the UAV control device 500 receives a landing command via the UAV wireless communication device 522 (Figure 9 / C50), it executes control to land the unmanned airplane 50 (Figure 9 / STEP 511).

[0178] When the unmanned airplane 50 lands, the UAV control device 500 acquires an environmental image representing the surroundings of the work machine 40 by capturing images of the surroundings of the work machine 40 using the UAV imaging device 512 (Figure 9 / STEP 512).

[0179] When the UAV control device 500 acquires the environmental image, it transmits environmental image data including the environmental image to the remote operation support server 10 via the UAV wireless communication device 522 (FIG. 9 / STEP 513).

[0180] In the remote operation support server 10, when the remote operation support element 102 included in the remote operation support device 100 receives a landing command through the server wireless communication device 122 (Figure 9 / C15), it transmits environmental image data to the remote operation device 20 (Figure 9 / STEP 114).

[0181] Here, a description will be given of the arithmetic processing performed by the remote operation support apparatus 100 in STEP 114. The arithmetic processing for the remote operation support apparatus 100 to transmit the environmental image data is performed by the second support processing element 102 included in the remote operation support apparatus 100.

[0182] In STEP 114, the second support processing element 102 acquires an environmental image showing the surroundings of the work machine 40 captured by a camera (e.g., UAV imaging device 512) of the unmanned aircraft 50 that has landed at a predetermined position (e.g., cab 470), and performs processing to transmit the environmental image to the remote control device 20 so that the image output device 221 of the remote control device 20 can output the environmental image.

[0183] In addition, the second support processing element 102 may recognize, based on communication with the work machine 40, whether or not there is a positioning mechanism (e.g., forward positioning mechanism 481) for fixing the unmanned aircraft 50 at a landing position (e.g., cab 470) on the work machine 40, and if the determination is positive, may execute a process to transmit to the remote control device 20 position information regarding a landing position (e.g., the front part of the ceiling of the cab 470) that has a positioning mechanism (e.g., forward positioning mechanism 481) and / or a landing position (e.g., the boarding door part of the cab 470) that does not have a positioning mechanism (e.g., positioning mechanism 480).

[0184] In addition, the second support processing element 102 may acquire an environmental image that represents the surroundings of the work machine 40 captured by a camera (e.g., UAV imaging device 512) of the unmanned airplane 50 that has landed at a predetermined position (e.g., the left positioning mechanism 482), and that represents the state in a direction (to the left of the work machine 40) corresponding to the predetermined position (e.g., the left positioning mechanism 482) on the work machine 40, and that includes a part of the work machine 40 (e.g., the left edge portion of the ceiling of the cab 470), and may perform processing to transmit the environmental image to the remote control device 20 so that the image output device 221 of the remote control device 20 can output the environmental image. In this case, for example, when the landing position is at forward positioning mechanism 481, unmanned aerial vehicle 50 lands so as to capture images of the front of work machine 40, when the landing position is at left positioning mechanism 482, unmanned aerial vehicle 50 lands so as to capture images of the left of work machine 40, and when the landing position is at rear positioning mechanism 484, unmanned aerial vehicle 50 lands so as to capture images of the rear of work machine 40. This allows the operator to see the surroundings of work machine 40 without their view being obstructed by cab 470, operating mechanism 440, the top surface of upper rotating body 460, the exhaust device, etc.

[0185] In addition, the second support processing element 102 may recognize the landing position of the unmanned airplane 50 and the captured image taken by a camera (e.g., UAV imaging device 512) possessed by the unmanned airplane 50, and if the unmanned airplane 50 has not landed on the work machine 40 and the captured image has not acquired an environmental image that represents the surroundings of the work machine 40 and includes part or all of the work machine 40, it may execute a process of sending a camera control command to the unmanned airplane 50, which is a signal including a command signal for controlling the operation of the camera (e.g., UAV imaging device 512) to acquire an environmental image.

[0186] In the remote operation device 20, when the remote control device 200 included in the remote operation device 20 receives environmental image data through the remote wireless communication device 222 (Fig. 9 / C22), it executes control to output the environmental image data to the image output device 221 in the remote output interface 220 (Fig. 9 / STEP213).

[0187] The second support processing element 102 may also acquire information about the remaining battery charge of the unmanned airplane 50 and execute a process of transmitting the information about the remaining battery charge to the remote control device 20 via the server wireless communication device 122.

[0188] For example, as shown in Fig. 10, in the remote operation device 20, the remote control device 200 determines whether a flight operation, which is an operation for flying the unmanned airplane 50, has been performed (Fig. 10 / STEP 221). If the determination is negative (Fig. 10 / STEP 221...NO), the remote control device 200 terminates the calculation process. On the other hand, if the determination is positive (Fig. 10 / STEP 221...YES), the remote control device 200 transmits a flight command including a command signal for flying the unmanned airplane 50 to the remote operation support server 10 via the remote wireless communication device 222 (Fig. 10 / STEP 222).

[0189] In the remote operation support server 10, when the remote operation support device 100 receives a flight command through the server wireless communication device 122 (Figure 10 / C16), the remote operation support device 100 transmits the flight command to the unmanned airplane 50 through the server wireless communication device 122 (Figure 10 / STEP 121).

[0190] In the unmanned airplane 50, when the UAV control device 500 receives a flight command through the UAV wireless communication device 522 (Figure 10 / C51), it performs flight control to fly the unmanned airplane 50 so that the flight behavior corresponds to the operation behavior of the operating device of the unmanned airplane 50 (Figure 10 / STEP521).

[0191] The remote control device 20 may be used as the operating device for the unmanned airplane 50, or a device dedicated to operating the unmanned airplane 50 (for example, a remote control) may be used. Furthermore, flight control of the unmanned airplane 50 may be independent, without relying on an operating device, by executing control for flying in a command mode based on pre-programmed command signals.

[0192] Furthermore, the UAV control device 500 acquires remaining battery level data, which is information including information relating to the remaining level of the battery that is the power source of the unmanned airplane 50 (FIG. 10 / STEP 522).

[0193] Furthermore, when the UAV control device 500 acquires the remaining battery level data, it transmits the remaining battery level data to the remote operation support server 10 via the UAV wireless communication device 522 (FIG. 10 / STEP 523).

[0194] In the remote operation support server 10, when the remote operation support element 102 included in the remote operation support device 100 receives remaining battery power data through the server wireless communication device 122 (FIG. 10 / C17), it determines whether the remaining battery power or the flight time of the unmanned airplane 50 based on the remaining battery power is equal to or less than a predetermined value based on the received remaining battery power data (FIG. 10 / STEP 122). If the determination is negative (FIG. 10 / STEP 122...NO), the remote operation support element 102 terminates the remote operation support process. On the other hand, if the determination is positive (FIG. 10 / STEP 122...YES), the remote operation support element 102 transmits information about a landing position (e.g., a forward positioning mechanism 481 having a battery 4810, a landing structure 60 having a battery 4810) having a charging device (e.g., a battery 4810) to the remote operation device 20 through the server wireless communication device 122 (FIG. 10 / STEP 123).

[0195] In the remote operation device 20, when the remote control device 200 receives battery remaining capacity data through the remote wireless communication device 222 (Figure 10 / C22), the remote control device 200 executes processing to control the output mode for outputting information regarding the landing position (e.g., forward positioning mechanism 481 having battery 4810, landing structure 60 having battery 4810) having the charging device (e.g., battery 4810) to the image output device 221 (Figure 8 / STEP223).

[0196] Also, for example, as shown in FIG. 11, the remote operation support element 102 may perform control to acquire an environmental image by sending a camera control command to the unmanned airplane 50 even before the unmanned airplane 50 lands on the work machine.

[0197] In the remote operation device 20, similar to STEP 211, the remote control device 200 determines whether or not the operator has performed a designation operation through the remote input interface 210 (FIG. 11 / STEP 231). If the determination result is negative (FIG. 11 / STEP 231...NO), the processing ends. On the other hand, if the determination result is positive (FIG. 11 / STEP 231...YES), the remote control device 200 transmits a landing command including a command signal for landing the unmanned airplane 50 to the remote operation support server 10 through the remote wireless communication device 222 (FIG. 11 / STEP 232).

[0198] When the remote operation support server 10 receives a landing command through the server wireless communication device 122 (Figure 10 / C18), the first support processing element 101 sends a landing command to the unmanned airplane 50 to land it at a predetermined position at the work site (e.g., landing structure 60) in accordance with a specified operation on the remote input interface 210 provided on the remote operation device 20 (Figure 11 / STEP 131).

[0199] In the unmanned airplane 50, when the UAV control device 500 receives the landing command through the UAV wireless communication device 522 (Figure 11 / C52), the UAV control device 500 executes a control process to land the unmanned airplane 50 at a predetermined position at the work site (e.g., landing structure 60) (Figure 11 / STEP531).

[0200] The UAV control device 500 acquires landing position data, which is information regarding the location where the unmanned aircraft 50 has landed, using a landing position information acquisition means (e.g., GNSS, etc.) that is also used to acquire information including the location where the unmanned aircraft 50 has landed (Figure 11 / STEP 532).

[0201] When the unmanned aircraft 50 lands, the UAV control device 500 activates the UAV imaging device 512 to capture images of the area around the landing location and acquires captured image data, which is information including images showing the area around the landing location (Figure 11 / STEP 533).

[0202] When the UAV control device 500 acquires the landing position data and the captured image data, it transmits the landing position data and the captured image data to the remote operation support server 10 via the UAV wireless communication device 522 (Figure 11 / STEP 534).

[0203] In the remote operation support server 10, when the second support processing element 102 receives landing position data and captured image data via the remote wireless communication device 122 (Figure 11 / C19), it performs a determination as to whether the unmanned airplane 50 has landed on the work machine 40 (Figure 11 / STEP 132).

[0204] If the determination result is positive (FIG. 11 / STEP 132...YES), the second support processing element 102 ends the calculation process. On the other hand, if the determination result is negative (FIG. 11 / STEP 132...NO), the remote control device 200 determines whether the captured image represents the surroundings of the work machine 40 and whether an environmental image has been acquired that is an image that includes all or part of the work machine 40 (FIG. 11 / STEP 133).

[0205] If the determination result is positive (FIG. 11 / STEP 133...YES), the second support processing element 102 terminates the calculation process. On the other hand, if the determination result is negative (FIG. 11 / STEP 132...NO), the second support processing element 102 transmits a camera control command including a command signal for controlling the operation of a camera (e.g., UAV imaging device 512) possessed by the unmanned airplane 50 to the unmanned airplane 50 via the server wireless communication device 122 (FIG. 11 / STEP 134).

[0206] An example of a camera control command is one that includes a command signal to change the direction of the camera (e.g., UAV imaging device 512) possessed by the unmanned aerial vehicle 50 in the direction of the work machine 40 that is to be remotely controlled by the operator when the image captured by the camera (e.g., UAV imaging device 512) possessed by the unmanned aerial vehicle 50 does not include the work machine 40 that is to be remotely controlled by the operator (i.e., when an environmental image cannot be obtained).

[0207] For example, examples of camera direction changing operations include panning, tilting, and rolling.

[0208] Although an example of a camera control command has been given in which the direction of a camera (e.g., UAV imaging device 512) carried by the unmanned aerial vehicle 50 is changed has been given, examples of camera control commands are not limited to this example. For example, if a work machine 40 that is the target of remote control by the operator is included in an image captured by a camera (e.g., UAV imaging device 512) carried by the unmanned aerial vehicle 50, but the image is out of focus and the operator is unable to recognize the work machine 40 that he or she is remotely operating (i.e., unable to acquire an environmental image), an example of the command command includes a command signal for zooming in or out the camera (e.g., UAV imaging device 512) carried by the unmanned aerial vehicle 50 to focus and acquire an environmental image. For example, if the work machine 40 to be imaged is a large 13-ton excavator that is relatively tall, an example of the command signal is to zoom out the camera carried by the unmanned aerial vehicle 50 to focus. For example, if the work machine 40 to be imaged is a 7-ton mini-shovel, and the height of the shovel is relatively low, the camera mounted on the unmanned airplane 50 may be zoomed in to focus.

[0209] That is, the second support processing element 102 recognizes the landing position of the unmanned airplane 50 and the captured image taken by the camera (e.g., UAV imaging device 512) possessed by the unmanned airplane 50, and if the unmanned airplane 50 has not landed on the work machine 40 and the captured image has not acquired an environmental image that represents the surroundings of the work machine 40 and includes part or all of the work machine 40, it executes a process of sending a camera control command to the unmanned airplane 50, which is a signal including a command signal for controlling the operation of the camera to acquire the environmental image.

[0210] An example of the second support process executed by the second support process element 102 is steps 132 to 134.

[0211] In the unmanned airplane 50, when the UAV control device 500 receives a camera control command via the UAV wireless communication device 522 (FIG. 11 / C53), it controls the operation of the camera (FIG. 11 / STEP 535).

[0212] When the UAV control device 500 controls the operation of a camera (e.g., UAV imaging device 512) possessed by the unmanned aircraft 50, the captured image acquires environmental image data, which is information including an environmental image that shows the surroundings of the work machine 40 and includes part or all of the work machine 40 (Figure 11 / STEP 536).

[0213] When the UAV control device 500 acquires the environmental image data, it transmits the environmental image data to the remote operation support server 10 via the UAV wireless communication device 522 (FIG. 11 / STEP 537).

[0214] In the remote operation support server 10, when the remote operation support apparatus 100 receives the environmental image data through the server wireless communication device 122 (FIG. 11 / C100), The environmental image data is transmitted to the remote operation device 20 via the server wireless communication device 122 (FIG. 11 / STEP 135).

[0215] In the remote operation device 20, when the remote control device 200 receives environmental image data through the remote wireless communication device 222 (Figure 11 / C23), the remote control device 200 executes processing to control the output mode for outputting the environmental image data to the image output device 221 (Figure 11 / STEP233).

[0216] (Other embodiments) It is preferable that the second support processing element of the present invention performs processing to orient the imaging direction of the UAV imaging device 512 mounted on the unmanned aircraft 50 fixed to the positioning mechanism 480 in a direction corresponding to the operation manner of the remote input interface 210 operated by the operator.

[0217] (Action and effect) According to the present invention, after the unmanned aerial vehicle 50 is fixed to the positioning mechanism 480, the second support processing element 102 performs processing to point the UAV imaging device 512 in a direction according to the operation manner of the remote input interface 210 operated by the operator.

[0218] This means that when the unmanned airplane 50 is fixed to the positioning mechanism 480, even if the imaging direction of the UAV imaging device 512 is in a direction that the operator does not want, the imaging direction of the UAV imaging device 512 can be adjusted after the unmanned airplane 50 is fixed to the positioning mechanism 480.

[0219] Therefore, after the operator fixes the unmanned airplane 50 to the positioning mechanism 480, the operator can point the imaging direction of the UAV imaging device 512 in the direction he or she wants to see.

[0220] In addition, if there are multiple unmanned airplanes 50 and multiple work machines 40, the remote operation support device 100 may acquire an ID (identifier) ​​to identify each unmanned airplane 50 and each work machine 40.

[0221] Regarding the computational processing performed by the first support processing element, an example has been given in which, when the landing position of the unmanned airplane 50 is not a predetermined position on the work machine 40, a landing command including a command to capture an image in the direction in which the work machine 40 is located as seen from the camera mounted on the unmanned airplane 50 is sent to the unmanned airplane 50, using known trigonometric functions.However, examples in which the first support processing element sends a landing command to the unmanned airplane 50 including a command to capture an image in the direction in which the work machine 40 is located as seen from the camera mounted on the unmanned airplane 50 are not limited to this example.

[0222] For example, the first support processing element may use a known image recognition function to send a landing command to the unmanned aircraft 50 that includes a control command to drive an angle adjustment mechanism for adjusting the imaging angle of the UAV imaging device 512 until the work machine 40 to be imaged is found. [Explanation of symbols]

[0223] 10...remote operation support server, 20...remote operation device, 40...work machine, 50...unmanned aerial vehicle, 100...remote operation support device, 101...first support processing element, 102...second support processing element, 210...remote output interface, 211...remote operation mechanism, 220...remote output interface, 221 image output device.

Claims

1. a work machine including a lower traveling body and an upper rotating body rotatably mounted on the lower traveling body; an unmanned aerial vehicle having an imaging device; an assistance processing element configured outside the work machine, capable of communicating with each of the work machine and the unmanned aerial vehicle, and configured to send a landing command to the unmanned aerial vehicle to cause it to land at a predetermined position on the work machine; an actual machine control device that is mounted on the work machine and recognizes operation information that is information about the operation of the work machine, including information about the swing speed of the upper swing body; an actual machine wireless communication device that is mounted on the work machine and has a function of transmitting the operation information recognized by the actual machine control device to the assistance processing element, the actual device control device transmits the operation information to the support processing element via the actual device wireless communication device; the support processing element determines whether or not the operating speed of the work machine is equal to or less than a predetermined value based on the operation information transmitted by the actual machine control device to the support processing element via the actual machine wireless communication device; If the determination is affirmative, a process is executed to land the unmanned airplane at the predetermined position of the work machine in response to the landing command. system.

2. 10. The system of claim 1, A process is executed to cause the imaging device to capture an image when the unmanned airplane has landed at the predetermined position. system.

3. 3. The system of claim 2, When the unmanned airplane lands at the predetermined position, a process is executed to output an environmental image captured by the imaging device to an image output device. system.

4. The system according to any one of claims 1 to 3, The predetermined position is located on the upper rotating body. system.

5. 5. The system of claim 4, The predetermined position is located on the top surface of a cab provided on the upper rotating body. system.

6. The system according to any one of claims 1 to 5, the support processing element determines, when a designation operation for designating the predetermined position on the work machine is performed, whether or not the operating speed of the work machine is equal to or less than a predetermined value based on the operation information; If the determination is affirmative, a process is executed to land the unmanned airplane at the predetermined position designated in the designation operation. system.

7. The system according to any one of claims 1 to 6, the support processing element recognizes whether or not operation of the work machine has stopped based on the operation information; When the support processing element recognizes that the operation of the work machine has stopped, a process is executed to land the unmanned aerial vehicle at the predetermined position. system.

8. The system according to any one of claims 1 to 7, the support processing element is capable of communicating with a plurality of the work machines present at the work site; In response to designation of one of the plurality of work machines present at the work site, a process is executed to land the unmanned aircraft on the designated work machine as the one work machine. system.

9. a processing device configured outside the work machine and capable of communicating with a work machine having a lower traveling body and an upper rotating body rotatably mounted on the lower traveling body, and an unmanned aircraft having an imaging device, and configured to transmit a landing command to the unmanned aircraft to land the work machine at a predetermined position, The information is information about the operation of the work machine, including information about the rotation speed of the upper rotating body recognized by an actual machine control device mounted on the work machine, and the information is transmitted to the arithmetic processing device by the actual machine control device via an actual machine wireless communication device mounted on the work machine and having the function of transmitting the information to the arithmetic processing device, and based on the operation information, if the operation speed of the work machine is equal to or less than a predetermined value, the landing command is transmitted to the unmanned aerial vehicle. Processing unit.

Images