Work machine control system

Abstract

To provide a work machine control system that enables accurate operation to be maintained for a work machine (robot or the like) and a movable body (work floor or the like) when the work machine such as a robot is mounted in the movable body such as a work floor.SOLUTION: A work machine control system 100 includes: a work machine WM (work robot 10 or the like); a movable body MB (work vehicle 20 or the like) that moves the work machine WM; a switching unit SW that switches a mode between operation of the work machine WM and operation of the movable body MB; and a controller 52 that performs the operation of the work machine WM and the operation of the movable body MB according to switching of the mode by the switching unit SW.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a work machine control system for controlling the operation of a work machine such as a remote-controlled work robot. [Background technology]

[0002] For example, as remote control techniques, there are known techniques for improving the workability in operating a slave control device (see Patent Document 1) and for operating a crane device equipped with a boom or the like (see Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 8-257948 [Patent Document 2] JP 2019-172409 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned Patent Documents 1 and 2 are silent about specific operation methods in a case where, for example, a remote-controlled robot is placed on the work platform of an aerial work vehicle that operates a boom or the like.

[0005] The present invention has been made in consideration of the above-mentioned points, and aims to provide a work machine control system that enables accurate operation of a work machine (robot, etc.) and a moving body (work floor, etc.) when the work machine, such as a robot, is mounted on a moving body such as a work floor. [Means for solving the problem]

[0006] To achieve the above objective, a work machine control system includes a work machine, a moving body that moves the work machine, a switching unit that switches modes between operation of the work machine and operation of the moving body, and a controller that operates the work machine and the moving body in accordance with the mode switching by the switching unit.

[0007] In the above-mentioned work machine control system, the controller operates the work machine and the mobile body in accordance with mode switching by the switching unit, making it possible to maintain accurate operation when work is performed while moving the work machine using the mobile body.

[0008] In a specific aspect of the present invention, the work machine is a work robot, and the mobile body is a work vehicle that has the function of extending, contracting, and rotating with the work robot mounted thereon. In this case, appropriate switching between the operation of the work robot and the operation of the work vehicle becomes possible, and each operation can be accurately maintained by the controller.

[0009] In another aspect of the present invention, the switching unit switches modes using the on / off of a foot switch as a trigger. In this case, by using the foot switch, the operation can be switched safely and reliably.

[0010] In yet another aspect of the present invention, the controller is a hand controller that an operator holds with both hands and is operable when the operator holds the controller. In this case, the controller is held with both hands by the operator and the operation is started when the operator detects that the controller is held with both hands, so that erroneous operation can be suppressed.

[0011] In yet another aspect of the present invention, when the switching unit detects that the operator is holding the controller, the switching unit prioritizes continuation of the current operation and does not switch modes. In this case, while the operator is holding the controller, the operation is prioritized and mode switching is prevented, so that a situation such as a sudden change to an unintended mode can be prevented.

[0012] In yet another aspect of the present invention, an imaging unit is provided in the work machine to capture an image of the outside world, and a head-mounted display unit is provided to display an image captured by the imaging unit. In this case, an operator can perform an operation while visually checking the outside world from the work machine by wearing the head-mounted display unit.

[0013] In yet another aspect of the present invention, the head mounted display unit displays an image relating to the operation of the work machine and the operation of the moving object by superimposing the image on the captured image. In this case, the operation content can be confirmed while visually confirming the outside world.

[0014] In yet another aspect of the present invention, the head mounted display unit has an attitude detection unit, and the work machine changes the imaging range of the imaging unit in accordance with the detection result of the attitude detection unit. In this case, it is possible to provide an outside world image from the work machine in accordance with a change in the attitude of the operator wearing the head mounted display unit. [Brief description of the drawings]

[0015] [Figure 1] 1 is a perspective view for explaining an overview of an example of a work machine control system according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a perspective view showing an example of a working robot as a working machine. [Diagram 3] 2 is a conceptual diagram showing an example of each part constituting the work machine control system. FIG. [Figure 4] 1(A) and 1(B) are conceptual front views showing an example of the movement of an arm of a working robot. [Diagram 5] 1A and 1B are diagrams for explaining an overview of the operation of a camera mounted on a working robot. [Figure 6] 5(A) to 5(C) are diagrams for explaining details regarding the operation of a camera mounted on a working robot. [Figure 7] 1A and 1B are block diagrams for explaining an outline of the operation of an example of a work machine control system. [Figure 8] 1(A) to 1(D) are a front view and a side view showing an example of a controller (hand controller). [Figure 9] 13A to 13C are conceptual image diagrams showing an example of a display mode during operation. [Figure 10] (A) is a figure for explaining an image relating to the operation of a moving object superimposed on a captured image in a moving object operation mode, (B) is a figure for explaining an image relating to the operation of a work machine superimposed on a captured image in a work machine operation mode, and (C) is a conceptual image diagram showing another example of the display mode of an image relating to the operation of a work machine. [Figure 11] 4 is a flowchart for explaining an example of an operation in the work machine control system. [Figure 12] 1 is a conceptual diagram showing an outline of a configuration of a work machine control system. [Figure 13] 10 is a flowchart for explaining another example of the operation of the work machine control system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] An example of a work machine control system according to an embodiment will be described below with reference to Fig. 1 etc. Fig. 1 is a perspective view for explaining a work machine control system 100 which is an example of the work machine control system according to the embodiment, and Fig. 2 is a perspective view showing an example of a work robot (heavy machinery robot) 10 as a work machine WM constituting the work machine control system 100.

[0017] As shown in Figures 1 and 2, the work machine control system 100 of this embodiment includes a work robot 10, which is a work machine WM, a work vehicle (aerial work vehicle) 20, which is a mobile body MB that moves the work robot 10 (work machine WM), and a cockpit CO for operating these.

[0018] As shown in FIG. 2, the working robot 10 has a structure that imitates the upper body of a human. More specifically, the working robot 10 has a body 11 that is rotatably mounted on a platform PP of a work vehicle 20, a pair of arms 12R, 12L that are mounted on the left and right sides of the body 11, a neck 13 that is mounted on the upper part of the body 11, and a head 14 that is mounted on the neck 13. The above-mentioned parts that make up the working robot 10 are rotatable about axes at various points, and can change their posture in a variety of ways, just like a human. Therefore, by moving the arms 12R, 12L, etc. by remote control, the working robot 10 can perform various tasks on the platform PP in place of a human. In addition, a pair of cameras CA, CA are provided on the head 14 at positions that correspond to human eyes, and it is possible to capture continuous images of the surroundings of the working robot 10. In other words, the pair of cameras CA, CA are provided on the working robot 10 (working machine WM) and function as an imaging unit that captures images of the outside world.

[0019] Also, it is assumed here that it is possible to measure the distance to objects around the working robot 10 (perform distance measurement) by utilizing the parallax of the pair of cameras CA, CA or by providing a separate distance measurement device. This makes it possible, for example, to grasp the distance from the working robot 10 to the nearest object. Note that various information indicating the environment of the working robot 10 acquired by the above-mentioned imaging and distance measurement is transmitted to the cockpit CO side, i.e., the side performing the operation, and is used, for example, as feedback information.

[0020] The work vehicle 20 is an aerial work vehicle consisting of a vehicle having wheels, a driver's seat, etc., as shown in FIG. 1 as an overall appearance, and as shown in FIG. 2, a platform PP is provided at the tip of a boom BM, and as described above, with the work robot 10 mounted and fixed on the platform PP, it can be moved to a desired height.

[0021] In the above configuration, the work machine control system 100 is capable of changing the posture of the working robot 10, moving the arms, etc., and extending and revolving the boom BM of the work vehicle 20, which is a vehicle for working at height, in accordance with commands from the cockpit CO. In this embodiment, as shown in Fig. 2, the working robot 10 is placed on the boom BM of the work vehicle 20 (on a placement part PP provided at the tip of the boom BM), and by moving the boom BM of the work vehicle 20, the working robot 10 can be moved to a desired height or position to perform a desired task. In other words, the work vehicle 20 (boom BM) has the function of extending and revolving while the working robot 10 is placed on the placement part PP.

[0022] In particular, in this embodiment, the operation mode can be switched between the operation of the working robot 10 (working machine WM) and the operation of the moving body MB (working vehicle 20) on the operating side. This makes it possible to maintain accurate operation of the working robot 10 (working machine WM) and the moving body MB (working vehicle 20).

[0023] Returning to Fig. 1, as shown in a partially enlarged perspective view of an example enclosed by a dashed line in the figure, a control unit 50 composed of various equipment is provided in the cockpit CO so that the above-mentioned operations can be performed by the operator HU. Specifically, the control unit 50 includes a control device 51, a controller 52, a foot switch 53, an HMD (head mounted display) 54 which is a head mounted display unit, and a power supply unit 55. The controller 52 is composed of a pair of left and right hand controllers 52R, 52L which the operator HU holds with both hands, and in the illustrated example, the hand controllers 52R, 52L are attached to a movable unit 52M.

[0024] Of the control unit 50, the control device 51 is composed of, for example, a PC or various control circuits, and is connected to various parts such as the controller 52 (not shown), and performs various processes related to the operation of the working robot 10 and the work vehicle 20 based on input operations from the controller 52 or foot switch 53. The control device 51 is also connected to the working robot 10 and the work vehicle 20 that are located outside the cockpit CO (not shown), and communicates information with them, issues various commands, and obtains information from the working robot 10 and the work vehicle 20.

[0025] As described above, the controller 52 is composed of hand controllers 52R, 52L and a movable part 52M, and can be operated by the operator HU holding the right hand controller 52R and the left hand controller 52L with the left and right hands. When remotely controlling the working robot 10, the right arm 12R (see FIG. 2) of the working robot 10 is moved based on a command from the right hand controller 52R, and the left arm 12L (see FIG. 2) of the working robot 10 is moved based on a command from the left hand controller 52L. In the example shown in FIG. 1, the movable part 52M has a pair of arm-shaped parts MR, ML, and the arm-shaped parts MR, ML can slide in the up-down direction and can expand and contract in the front-rear direction, and can rotate in multiple axial directions at the tip side. The hand controllers 52R, 52L are attached to the tip parts of the arm-shaped parts MR, ML, respectively, so that the operator HU can freely move the hand controllers 52R, 52L. Furthermore, since movable section 52M is configured as described above, bilateral control is performed in which not only commands from the operator HU to the working robot 10 but also movements in the opposite direction are transmitted. More specifically, when the operator HU applies force to hand controllers 52R, 52L, the working robot 10 operates according to the magnitude and direction of the force, while force-propagating bilateral control is performed in which the movement of hand sections HR, HL (see FIG. 2) forming the tips of arms 12R, 12L of the working robot 10 is fed back to the positions of hand controllers 52R, 52L. A more specific configuration example of hand controllers 52R, 52L will be described later with reference to FIG. 8.

[0026] The foot switch 53 functions as a switching unit SW or a part thereof that switches modes between operating the work robot 10 and operating the boom of the work vehicle 20. In this example, turning the foot switch 53 on and off is used as a trigger to switch modes between a mode for operating the work robot 10 and a mode for operating the work vehicle 20. More specifically, when the foot switch 53 is on, i.e., when the foot switch 53 is stepped on, the boom of the work vehicle 20 can be operated, and when the foot switch 53 is off, i.e., when the foot switch 53 is not stepped on, the work robot 10 can be operated.

[0027] In response to mode switching by foot switch 53 as a switching unit SW as described above, controller 52 operates the work robot 10 and the boom of the work vehicle 20. For this reason, there are two ways to use the controller 52: one for operating the work robot 10 and one for operating the boom of the work vehicle 20. Looking at it from another perspective, one controller 52 is used for both operating the work robot 10 and operating the boom of the work vehicle 20, which are different operations.

[0028] As shown in the figure, the HMD 54 is a head-mounted display device, and when worn by the operator HU, an image is displayed in a location covering the front of the eyes, allowing the operator HU to view the image. Here, by displaying images captured by a pair of cameras CA, CA attached to the head 14 of the working robot 10, it is possible to capture the outside world from the viewpoint of the working robot 10. In other words, the HMD 54 as a head-mounted display unit displays images captured by the camera CA, which is an imaging unit.

[0029] Furthermore, in this embodiment, a sensor SE is provided for capturing the posture and line of sight of the operator HU. That is, the sensor SE includes a posture detection sensor PD as a posture detection unit composed of a gyro sensor and an acceleration sensor for capturing the inclination of the HMD 54, and a line of sight detection sensor GD composed of an infrared sensor for capturing the line of sight of the operator HU. Information about the posture of the operator HU detected by the posture detection sensor PD (head posture information) and information about the line of sight of the operator HU detected by the line of sight detection sensor GD (line of sight information) are managed by the control device 51. Also, the display on the HMD 54 is in a display mode that allows the user to know whether the working robot 10 is being operated or the boom of the work vehicle 20 is being operated. This will be described in detail later with reference to FIG. 9 and the like.

[0030] The power supply unit 55 is appropriately provided in a location within the cockpit CO that does not interfere with investigation by the operator HU, in order to ensure the power necessary to operate each of the above-mentioned components.

[0031] In addition to the above-described control unit 50, a box BO that forms a space, a chair CH for an operator HU to sit on, and the like are provided inside the cockpit CO. In addition, the cockpit CO may be provided with other components, such as a window WI for checking the outside world from the cockpit CO and for taking in outside light.

[0032] An example of the above-mentioned components constituting the work machine control system 100 and their relationships will be described below with reference to the conceptual diagram shown in Fig. 3. In particular, the operation associated with switching of the operation mode by the switching unit SW will be described here.

[0033] First, the work machine control system 100 is considered to be roughly composed of three component groups. Specifically, as shown in the figure, which is divided into three areas enclosed by dashed lines, the system is divided into a component group related to the work robot 10 (work machine WM), a component group related to the work vehicle 20 (mobile body MB), and a component group related to the control unit 50 that issues operation commands to these.

[0034] First, in the control unit 50, the control device 51 receives an on / off signal from the foot switch 53 in the switching circuit 51s and switches between various devices. For example, when the switching circuit 51s receives a signal indicating that the foot switch 53 is in an on state, the control device 51 performs processing in the control unit 50 to make the contents of the input signal received as an operation command from the controller 52 (hand controllers 52R, 52L) and the display content on the HMD 54 compatible with boom operation, so as to enable boom operation. In addition, with regard to external output, the output destination is switched so that various commands (operation commands) relating to operation from the controller 52 are output to the work vehicle 20 (mobile body MB).

[0035] On the other hand, when the switching circuit 51s receives a signal from the foot switch 53 indicating that it is in the off state, the control device 51 performs processing within the control unit 50 to make the controller 52 and HMD 54 compatible for robot operation so as to enable operation of the work robot 10, and for external output, switches the output destination to the work robot 10 (work machine WM).

[0036] In the above cases, it can also be considered that the foot switch 53 and the switching circuit 51s function as a switching unit SW that switches modes between operating the work machine WM (work robot 10) and the mobile body MB (work vehicle 20).

[0037] On the work vehicle 20 side, when the switching unit SW is in the ON state, the operation command transmitted from the control device 51 is received by the boom control unit 25, and the operation of each unit is controlled in accordance with the operation command, thereby performing boom operation. In other words, under the control of the boom control unit 25, the boom BM and the like can be moved to move the work robot 10 on the platform PP to a desired position.

[0038] On the working robot 10 side, when the switching unit SW is in the OFF state, the robot control unit 15 receives the operation command sent from the control device 51, and the operation of each unit is controlled in accordance with the operation command, thereby operating the robot. That is, under the control of the robot control unit 15, the torso 11 and the arms 12R, 12L are moved to perform various desired operations.

[0039] Under the control of the robot control unit 15, the neck unit 13 and head unit 14 are moved to capture (obtain) images and measure distances to the target position and direction using the camera CA. As described above, the detection results of the sensor SE of the HMD 54 are transmitted to the robot control unit 15 via the control device 51, and based on this, the neck unit 13 and head unit 14 are operated, and the acquired image data is displayed on the HMD 54, and this is repeated. In other words, as a result, as shown by the dashed-dotted bidirectional arrow AA1 in the figure, feedback display operation processing is performed between the neck unit 13 and head unit 14 (camera CA) on the robot side and the HMD 54 on the operator side. As a result, the head posture of the operator, i.e., the person wearing the HMD 54, is reflected in the imaging direction of the working robot 10. In addition, in this case, the viewpoint is not fixed, so visibility is easily ensured, for example, at night or during rain.

[0040] An example of the movements of the body 11 and arms 12R, 12L constituting the working robot 10 will be described below with reference to Fig. 4. Fig. 4(A) and Fig. 4(B) are conceptual front views showing an example of the movements of the arms 12R, 12L of the working robot, Fig. 4(A) shows the arms 12R, 12L in a lowered state, and Fig. 4(B) shows the arms 12R, 12L in a spread-out state. In each figure, the left-right direction in which the arms 12R, 12L are lined up is defined as the x-direction, the front-rear direction (depth direction and front direction) among the directions perpendicular to this is defined as the y-direction, and the up-down direction perpendicular to the x-direction and y-direction is defined as the z-direction. As described above, the working robot 10 in this example has a body portion 11 that can rotate about an axis extending in the z direction on the placement part PP (see FIG. 2, etc.) (typically, the body portion 11 can rotate about its central axis), and it is considered that the body portion 11 as a whole is hardly displaced from its current position. On the other hand, the arms 12R, 12L are maximum displaced downward (-z direction) in the state shown in FIG. 4(A), and maximum displaced left and right (±x directions) in the state shown in FIG. 4(B). Although not shown, the arms 12R, 12L are maximum displaced upward (+z direction) when they are raised, opposite to the state shown in FIG. 4(A). By considering the above, a distance necessary for the working robot 10 to avoid unintended contact with other objects in the vicinity can be set. As shown in the figure, the arms 12R, 12L have multiple joints and hand portions HR, HL, and are capable of rotating about an axis at each location, enabling various movements other than those illustrated in Figure 4(A) and the like.

[0041] Of the above, the operation of the cameras CA, CA mounted on the head 14 of the working robot 10 will be described below with reference to Fig. 5 etc. Fig. 5(A) and Fig. 5(B) are diagrams for explaining an overview of the operation of related parts including the cameras CA, CA, and Fig. 6(A) to Fig. 6(C) are diagrams (plan view, front view, and side view) for explaining a detailed example of the movement of the cameras CA, CA.

[0042] First, as shown in FIG. 5(A), a command (gaze direction command) for the gaze direction is given from the operation side to the cameras CA, CA. To explain this step by step, first, as described above, for the HMD 54 having the gaze detection sensor GD as a posture detection unit as part of the sensor SE, the posture and gaze are detected by the sensor SE. That is, head posture information is output by the posture detection sensor PD of the sensor SE, and gaze information is output by the gaze detection sensor GD. The head posture information and gaze information are output to the robot control unit 15, which is a control unit on the robot side, via the control unit 51, which is a control unit on the operation side. The robot control unit 15 outputs a drive signal of a neck posture command to the neck unit 13 based on the head posture information out of the head posture information and gaze information accepted as an operation command, and operates the neck unit 13 to position the head 14 provided on the neck unit 13 according to the head posture information. Also, the robot control unit 15 outputs a drive signal of a gaze direction command to the cameras CA, CA based on the gaze information, and operates the cameras CA, CA to position them according to the gaze information. That is, on the robot side, the imaging range of the cameras CA, CA is changed according to the detection results of the line of sight detection sensor GD and line of sight detection sensor GD.

[0043] According to the above-mentioned embodiment, the cameras CA, CA set at the desired positions and directions capture images, and output image information as the captured images to the robot control unit 15, as shown in Fig. 5(B). The robot control unit 15 outputs the received image information to the control device 51, which is the control device on the operation side, and the control device 51, which receives this information, performs image processing and display operations on the HMD 54. In this way, it becomes possible to provide images according to changes in the posture and line of sight of the operator HU (see Fig. 1), who is the wearer of the HMD 54.

[0044] In addition, in the image processing at this time, the control device 51 forms a superimposed image based on overlay display information. As the overlay display information here, display information showing an operation status according to the mode at the time of operation is assumed. The superimposed image will be described later with reference to FIG. 9 etc.

[0045] Among the above-mentioned operations, the cameras CA and CA can be rotated as shown in the plan view, front view, and side view of Fig. 6(A), Fig. 6(B), and Fig. 6(C), respectively. This enables the posture to be changed (the line of sight to be changed) in response to a line of sight direction command.

[0046] Of the various operations in the work machine control system 100 described above, an example of an operation associated with switching of the operation mode will be described with reference to a block diagram of some of the components of the work machine control system 100 shown in Fig. 7. Fig. 7(A) shows the operation in the work machine (robot; heavy equipment robot) operation mode in which the switching unit SW is in the OFF state, and Fig. 7(B) shows the operation in the moving body (boom) operation mode in which the switching unit SW is in the ON state.

[0047] For example, in the case of the working machine operation mode shown in FIG. 7(A), the controller 52 functions as a robot (heavy machinery robot) operation unit OPr in response to switching by the switching unit SW, and outputs a robot operation signal to the robot control unit 15. The robot control unit 15 outputs a robot control signal to drive each part of the main body of the working robot 10, i.e., the main parts of the operation, such as the torso part 11 and the arms 12R, 12L, in accordance with the robot operation signal received as an operation command from the operation side, and receives actuator information from these as feedback. The robot control unit 15 generates various data indicating the operating status of the working robot 10 based on the robot operation signal, the robot control signal, and further the actuator information. The robot control unit 15 outputs the generated information as overlay display information to a camera image processing unit 51G described later.

[0048] On the other hand, the robot control unit 15 operates the cameras CA, CA housed in the neck unit 13 and head unit 14 of the main body of the working robot 10. That is, as described above, image information is obtained as the result of imaging by the cameras CA, CA, and a camera image signal corresponding to the image information is output to the operating side. Here, it is assumed that the control device 51 on the operating side is provided with a camera image processing unit 51G that receives and processes the camera image signal.

[0049] In addition to the camera image signals from cameras CA, CA, the camera image processing unit 51G receives various data indicative of the operating status of the working robot 10 generated by the robot control unit 15 as overlay display information. The camera image processing unit 51G visualizes various information indicative of the operating status of the working robot 10 as graphs, figures, etc., onto the image information capturing an image of the surroundings of the working robot 10 as a camera image signal, and superimposes this information as overlay display information that can be superimposed on the camera image signal to form a superimposed image. The image signal corresponding to the formed superimposed image is output to the HMD 54.

[0050] On the other hand, in the case of the mobile object operation mode shown in Fig. 7(B), in response to switching at the switching unit SW, the controller 52 functions as a boom operation unit OPb and outputs a boom operation signal to the boom control unit 25. The boom control unit 25 outputs a boom control signal to drive the boom BM according to the boom operation signal received as an operation command from the operating side, and receives actuator information from these as feedback, and based on this, generates various data indicating the operating status of the boom BM and outputs it to the camera image processing unit 51G as overlay display information.

[0051] The cameras CA, CA of the work robot 10 capture images even when operated in the mobile object operation mode, and camera image signals corresponding to the image information resulting from the capture are output to the operating side, and the camera image processing unit 51G receives the camera image signals.

[0052] The camera image processing unit 51G visualizes various data indicating the operating status of the boom BM generated in the boom control unit 25 in graphs, figures, etc., and superimposes the visualized data on the camera image signals from the cameras CA and CA as overlay display information that can be superimposed on the camera image signals to form a superimposed image. The image signal corresponding to the formed superimposed image is output to the HMD 54.

[0053] In this way, during both operation in the work machine operation mode and operation in the mobile object operation mode, the HMD 54 can display a superimposed image, that is, an image of the surroundings of the work robot 10 superimposed with display information showing the operation content. In other words, the HMD 54 displays an image relating to the operation of the work machine WM and the operation of the mobile object MB by superimposing it on the captured image captured by the cameras CA, CA.

[0054] A specific example of controller 52 (hand controllers 52R, 52L) will be described below with reference to Fig. 8. Fig. 8(A) and Fig. 8(B) are a front view and a side view showing an example of hand controller 52L for the left hand, of controller 52, and Fig. 8(C) and Fig. 8(D) are a side view and a front view showing an example of hand controller 52R for the right hand.

[0055] As shown in the figure, the hand controllers 52R, 52L include grip portions RRg, LLg, grip sensors RRs, LLs, trigger portions RRt, LLt, and operation button portions RRb, LLb.

[0056] The hand controllers 52R, 52L are configured so that the operator HU (see FIG. 1) holds the grip parts RRg, LLg with the left and right hands, and are activated when the grip sensors RRs, LLs provided on the grip parts RRg, LLg detect that the operator HU holds the hand controllers 52R, 52L with both hands. In other words, the controller 52 becomes operable when the operator HU holds the hand controllers 52R, 52L with both hands.

[0057] The trigger parts RRt, LLt are provided at positions where the index fingers of the operator HU can be placed, and in the work machine operation mode, the operator HU pulls and releases the trigger parts RRt, LLt to open and close the hand parts HR, HL (see Figure 2, etc.) that form the tips of the arms 12R, 12L of the work robot 10.

[0058] The operation button sections RRb, LLb are composed of a plurality of operation buttons, and the instructions given to them differ between the working machine operation mode and the mobile body operation mode. That is, even if the same button is operated, the instructions given to the buttons differ depending on the mode switching in the switching section SW (see FIG. 3, etc.). For example, if one button is a button for determining whether the trigger sections RRt, LLt are enabled or disabled in the working machine operation mode, that is, a button for determining whether the hand sections HR and HL can be opened or closed, the same button switches to function as a button for extending and retracting the boom BM in the mobile body operation mode.

[0059] Hereinafter, with reference to the image diagram shown in FIG. 9, an example of a display mode on the HMD 54 during the above-mentioned operation will be described.

[0060] For example, when an image GG as shown in Fig. 9(A) is obtained by imaging with cameras CA, CA of the work robot 10 (see Fig. 5, etc.), in the moving object operation mode, a superimposed image GGb in which information on the operating status of the boom BM is superimposed as overlay display information is displayed on the HMD 54 as shown in Fig. 9(B). On the other hand, in the work machine operation mode, a superimposed image GGr in which information on the operating status of the arms 12R, 12L of the work robot 10 is superimposed as overlay display information is displayed on the HMD 54 as shown in Fig. 9(C).

[0061] Hereinafter, the contents of an example of the overlay display information displayed on the superimposed image GGb and the superimposed image GGr will be described with reference to Fig. 10(A) and Fig. 10(B). In Fig. 10(A), for ease of understanding, the parts of the superimposed image GGb in the moving object operation mode shown in Fig. 9(B) that correspond to the image GG in Fig. 9(A) are shown by dotted lines, and the parts that correspond to the overlay display information are shown by solid lines. In Fig. 10(B), the parts of the superimposed image GGr in the working machine operation mode shown in Fig. 9(C) that correspond to the image GG in Fig. 9(A) are shown by dotted lines, and the parts that correspond to the overlay display information are shown by solid lines.

[0062] In the example shown in Fig. 10(A), as a display reflecting the operation contents in the moving body operation mode, selection marks MC1 to MC3 indicating which of rotation, telescopic, and elevation is selected, and arrow marks MA1 to MA4 indicating the movement direction are displayed. Here, rotation means the operation of rotating (rotating) the placement part PP (see Fig. 2, etc.) provided at the tip of the boom BM, and the marks MA2 and MA3 indicating left and right indicate whether it is a left rotation or a right rotation. Telescopic and elevation mean that the boom BM itself is telescopic and elevation, and the marks MA1 and MA4 indicating up and down indicate extension and contraction or rising and falling. In the example shown, left and right arrows are also displayed next to rotation, and up and down arrows are displayed next to telescopic and elevation.

[0063] Of the selection marks MC1-MC3 and the arrow marks MA1-MA4, for example, the one currently selected is displayed in a different color (hatched in the figure) from the unselected ones. In the illustrated example, the selection mark MC1 and the arrow mark MA2 are displayed in a different color (hatched), indicating that they are turning counterclockwise.

[0064] In an example shown in FIG. 10(B), as a display reflecting the operation contents in the work machine operation mode, operation feasibility marks ML1, MR1 indicating whether the left and right arms 12R, 12L (see FIG. 2, etc.) of the working robot 10 can be operated, bar graphs ML2, MR2 indicating the gripping force of the left and right hand parts HR, HL, and a distance DD from the head 14 to the closest object within the imaging range (in front of the cameras CA, CA) are displayed. Here, the operation feasibility marks ML1, MR1 indicate whether they are in an operable state or not depending on whether there is an X mark in a round mark, and the characters "operable" or "not operable" are displayed next to the operation feasibility marks ML1, MR1. In the illustrated example, the left indicates that the operation is not possible and the right indicates that the operation is possible. In addition, the length of the bar graphs ML2, MR2 extending in the vertical direction indicates the magnitude of the gripping force. The distance DD is displayed in front of the screen together with a specific numerical value of the distance (0.8 m in the illustrated example).

[0065] As described above, by displaying the corresponding operation content according to the operation mode, it is possible to switch between moving body operation and work machine operation and to grasp the operation status in each operation mode while keeping the HMD 54 on. In particular, in this embodiment, one controller 52 can be used for both moving body operation and work machine operation, so that there is no need to change the controller when switching the operation mode.

[0066] The above is an example of the display mode, and the displayed contents and display modes can be changed in various ways. For example, the display of the operation contents in the work machine operation mode may be as shown in FIG. 10(C) in addition to the example shown in FIG. 10(B). That is, the left and right operation availability marks ML1 and MR1 may be expressed by different colors (different hatching patterns in the figure), or in addition to or instead of the text display of "operable" or "not operable", the left and right hands may be displayed in text whether they are in a "locked" state or a "free" state (unlocked state). Also, it is possible to provide a scale on the bar graphs ML2 and MR2. In addition, the distance DD may be displayed by indicating the range in which the distance is measured on the image, or the measured distance value may be displayed in mm units.

[0067] An example of a series of operations relating to mode switching in the work machine control system 100 will be described below with reference to a flowchart shown in FIG.

[0068] When the work machine control system 100 is started, the control device 51 first checks the input state of the foot switch 53 (step S101). In step S101, if the foot switch 53 is stepped on and in an on state, and the switching circuit 51s receives this information as the switching unit SW, the control device 51 switches the mode to enable boom operation of the work vehicle 20 (step S102). That is, the control device 51 switches to issue an operation command to the work vehicle 20 (boom BM). In addition, as an internal mode switch on the operating side in this case, the control device 51 first performs various settings to accept the mode setting of the controller 52, that is, button operation on the controller 52, as the contents of boom operation (step S103). In addition, the control device 51 performs display setting to make the display mode on the HMD 54 correspond to the boom operation (step S104). That is, the control device 51 performs image processing setting to form a superimposed image in a mode as exemplified in FIG. 9(B).

[0069] After the processing of steps S103 and S104, the control device 51 checks whether the input of the foot switch 53 has been switched (step S105). If no switching has been performed (step S105: No), the control device 51 operates the boom BM in accordance with a command (input operation) from the controller 52 based on the setting in step S103 (step S106). If no switching has been performed in step S105 (step S105: No), the control device 51 maintains the operation in that state.

[0070] On the other hand, when the input state of the foot switch 53 is confirmed in step S101 and the foot switch 53 is not stepped on and is in an off state, when the switching circuit 51s receives this information as the switching unit SW, the control device 51 switches the mode to enable the operation of the working robot 10 (step S107). That is, the control device 51 switches to issue an operation command to the working robot 10. In addition, as an internal mode switch on the operating side in this case, the control device 51 first performs various settings to accept the mode setting of the controller 52, that is, button operations on the controller 52, as the contents of the robot operation (step S108), and further performs display settings to make the display mode of the HMD 54 correspond to the robot operation (step S109). That is, the control device 51 performs image processing settings to form a superimposed image in a mode as exemplified in FIG. 9(C).

[0071] After the processing of steps S108 and S109, a check is made to see whether the input of the foot switch 53 has been switched (step S110). If no switching has been performed (step S110: No), the control device 51 operates each part of the work robot 10 in accordance with the commands (input operations) from the controller 52 based on the settings made in step S108 (step S111). If no switching has been performed in step S110 (step S110: No), the operation is maintained in that state.

[0072] On the other hand, if a switch has been made in step S105 (step S105: Yes), the process proceeds to step S107, where a series of processes are performed to switch the mode from boom operation to robot operation. Similarly, if a switch has been made in step S110 (step S110: Yes), the process proceeds to step S102, where a series of processes are performed to switch the mode from robot operation to boom operation.

[0073] The above-described series of operations continues until the operations are terminated or interrupted, for example, when the power supply to the work machine control system 100 is turned off.

[0074] Hereinafter, an outline of the configuration of the work machine control system 100 of this embodiment will be described with reference to a conceptual diagram shown in FIG.

[0075] As described above, in the working machine control system 100 of the present embodiment, the working machine WM, such as the working robot (heavy machine robot) 10, is placed on the moving body MB, such as the working vehicle 20 or the boom BM of the working vehicle 20, to make it movable, and has a switching unit SW, and while switching between the operation of the working machine WM and the operation of the moving body MB, the controller 52 operates according to the mode switching. That is, the working machine WM is moved by the moving body MB, and work is performed using the working machine WM. Typically, the working machine WM is remotely operated to perform work. In this embodiment, when such a work method is performed, the operation of the working machine WM and the operation of the moving body MB are appropriately switched, so that accurate operation can be maintained.

[0076] In the above-described embodiment, the working robot (heavy machinery robot) 10 is given as an example of the working machine WM, and the working vehicle (aerial work vehicle) 20 or the boom BM of the working vehicle (aerial work vehicle) 20 is given as an example of the moving body MB, but the working machine WM and the moving body MB in this embodiment are not limited to these and can be various. Also, the direction and manner of movement of the moving body MB are not limited to the exemplified turning, telescopic, and undulating, and can be various.

[0077] As described above, the work machine control system 100 of this embodiment includes a work machine WM (work robot 10, etc.), a mobile body MB (work vehicle 20 or boom BM of work vehicle 20, etc.) that moves the work machine WM, a switching unit SW that switches between modes for operating the work machine WM and operating the mobile body MB, and a controller 52 that operates the work machine WM and the mobile body MB in response to mode switching by the switching unit SW. In the work machine control system 100, the controller 52 operates the work robot 10 as the work machine WM and the boom BM of the work vehicle 20 as the mobile body MB in response to mode switching by the switching unit SW, making it possible to maintain accurate operation in a mode in which work is performed while moving the work machine WM by the mobile body MB.

[0078] 〔others〕 The present invention is not limited to the above-described embodiment, and can be embodied in various forms without departing from the spirit and scope of the present invention.

[0079] First, among the above, the mode switching between boom operation and robot operation is not limited to the one mode described with reference to the flowchart of FIG. 11, and various modes can be used. For example, when the grip of the controller 52 by the operator HU is detected, that is, when the grip sensors RRs and LLs of the controller 52 detect that the operator HU is holding the controller 52, the operator HU may be assumed to intend to continue the current operation, and the mode may be switched without priority given to the continuation of the currently performed operation even if the on / off switching of the foot switch 53 is detected in the switching unit SW. FIG. 13 is a flowchart showing an example of the mode switching mode as described above, and corresponds to FIG. 11. In FIG. 13, steps S201 to S206 and S210 to S214 correspond to steps S101 to S106 and S107 to S111 in FIG. 11.

[0080] In Fig. 13, as in the example shown in Fig. 11, in principle, the mode is switched between boom operation and robot operation in response to the on / off switching of the foot switch 53, but when determining whether to execute the mode switching, the grip of the controller 52 by the operator HU is detected (steps S207, S215), and if detected, the mode switching is prevented (steps S208, S216), which is different from the example shown in Fig. 11. Specifically, for example, in step S205 (corresponding to step S105 in Fig. 11) of the boom operation, if the foot switch 53 is switched from on to off (step S205: Yes), the grip of the controller 52 by the operator HU is further detected (step S207). That is, the detection by the grip sensors RRs, LLs is confirmed. In step S207, if gripping is not detected (step S207: No), the process proceeds to step S210 (corresponding to step S107 in FIG. 11), where a series of processes are performed to switch the mode from boom operation to robot operation. On the other hand, in step S207, if gripping is detected (step S207: Yes), even if there is a command for switching based on the foot switch 53, this is treated as invalid, and the boom operation is maintained (step S208). When it is decided to maintain in step S208, it is detected whether or not the foot switch 53 has been switched again, that is, switched from off to on (step S209), and if not detected (step S209: No), the process returns to the process from step S207. In other words, the process from gripping detection of the controller 52 is repeated, and the boom operation mode is maintained until gripping is no longer detected. On the other hand, in step S209, if it is detected that the foot switch 53 has been switched again (step S209: Yes), the process returns to step S205.

[0081] During the robot operation, the same processing as in the boom operation is performed (steps S213 to S217). That is, even if the foot switch 53 is switched from off to on, if the grip of the controller 52 by the operator HU is detected, the command for switching based on the foot switch 53 is treated as invalid, and the robot operation mode is maintained until the grip of the controller 52 is no longer detected.

[0082] In the above cases, while the operator HU is performing a gripping operation indicating his / her intention to maintain the current operation mode, that operation takes priority, so that it is possible to prevent a situation in which the mode suddenly changes from boom operation to robot operation, or conversely, from robot operation to boom operation, due to unintended operation of the foot switch 53.

[0083] In addition to the above, other methods of determining whether to execute mode switching may be used, such as, for example, suppressing unintentional transition to boom operation by disabling both the operation of the foot switch 53 and the operation of the controller 52 when both are on, or preventing switching to another operation mode by the foot switch 53 unless the operator HU intentionally ends the active operation mode. In the example shown in Fig. 13, it was confirmed that the gripping operation by the operator HU was continuing based on the detection by the grip sensors RRs and LLs, but the method is not limited to this and various methods can be applied as long as it can be confirmed that the operator HU intends to maintain the current operation.

[0084] In addition, in the above description, remote control is performed using a wired connection. However, if sufficient support is available in terms of communication, wireless communication may be used in part or in whole.

[0085] Furthermore, in the above, a heavy machinery robot is shown as an example of the working robot 10, but the working robot 10 can be in various forms, and its role can also be in various forms. For example, in addition to a form in which the robot performs work alone, it can also be considered to perform work together with humans (assisting human work). The shape of the robot is also not limited to a humanoid shape as shown in the figure, and can be in various shapes, etc. Furthermore, the content of the work can be various matters such as tree felling, painting, etc., in addition to maintenance of various facilities at high places and installation of various devices.

[0086] In the above description, image information is acquired by the HMD 54, but this is not limiting and other methods may be used as long as necessary information, such as the surrounding environment of the working robot 10, can be acquired.

[0087] In the above embodiment, the foot switch 53 is turned on during boom operation, which has a large range of movement and requires a greater degree of safety confirmation, and is turned off during robot operation, which has a limited range of motion and has other methods for ensuring safety in the surroundings, such as distance measurement, but the relationship between the switching of the switch and the selection of the operation mode is not limited to the above and can be in various other embodiments. In the above embodiment, the foot switch 53 is used for mode switching, taking into consideration that the controller 52 is used to be operated using both hands, but other switch mechanisms may be used for switching. [Explanation of symbols]

[0088] 10...working robot, 11...torso part, 12R, 12L...arm part, 13...neck part, 14...head part, 15...robot control part, 20...work vehicle (high altitude work vehicle), 25...boom control part, 50...control unit, 51...control device, 51G...camera image processing part, 51s...switching circuit, 52...controller, 52M...moving part, 52R, 52L...hand controller, 53...foot switch, 54...HMD (head mounted display), 55...power supply part, 100...working machine control system, AA1...bidirectional arrow, BM...boom, BO...box, CA...camera, CA...head 14 (camera, CH...chair, CO...cockpit , DD...distance, GD...line of sight detection sensor, GG...image, GGb,GGr...overlay image, HR,HL...hand part, HU...operator, MA1~MA4...arrow mark, MB...moving object, MC1~MC3...selection mark, ML1,MR1...operation possible / not possible mark, ML2,MR2...bar graph, MR,ML...arm-shaped part, OPb...boom operation part, OPr...robot (heavy equipment robot) operation part, PD...posture detection sensor, PP...placement part, RRb,LLb...operation button part, RRg,LLg...grip part, RRs,LLs...grip sensor, RRt,LLt...trigger part, SE...sensor, SW...switching part, WI...window, WM...work machine

Claims

1. A working machine, A moving body that moves the working machine; A switching unit that switches between a mode for operating the work machine and a mode for operating the movable body; a controller that operates the work machine and the moving body in response to the mode switching by the switching unit; A work machine control system comprising:

2. the working machine is a working robot, The work machine control system according to claim 1 , wherein the mobile body is a work vehicle having a function of extending, contracting and rotating with the work robot mounted thereon.

3. The work machine control system according to claim 1 , wherein the switching unit switches the modes when a foot switch is turned on or off as a trigger.

4. The work machine control system according to claim 1 , wherein the controller is a hand controller that an operator holds with both hands and that becomes operable when the operator holds the hand controller.

5. The work machine control system according to claim 1 , wherein the switching unit, when detecting that an operator is holding the controller, prioritizes continuation of the current operation and does not switch the mode.

6. An imaging unit provided in the work machine for imaging an external environment; a head-mounted display unit that displays an image captured by the imaging unit; The work machine control system according to claim 1 .

7. The work machine control system according to claim 6 , wherein the head mounted display unit displays an image relating to the operation of the work machine and the operation of the moving object by superimposing the image on the captured image.

8. The head mounted display unit has an attitude detection unit, The work machine control system according to claim 6 , wherein the work machine changes an imaging range of the imaging unit in accordance with a detection result by the attitude detection unit.

Images