Work machine control method, work machine control program, work machine control system, and work machine

Abstract

To provide a work machine control method, a work machine control program, a work machine control system, and a work machine, which make it easy for an operator to intuitively grasp an operation state of a detector.SOLUTION: A work machine control method includes: acquiring a captured image Im100 of a monitoring area around the work machine; and displaying a display screen Dp1 that includes the captured image Im100 on a display device. The work machine control method further includes: when a detection processor for detecting a detection target Ob1 in the monitoring area is enabled, displaying a valid object X1 that moves in at least one of a circumferential direction and a radial direction of a virtual circle centered on a reference point set for the captured image Im100 overlapping the captured image Im100 on the display screen Dp1.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a control method for a working machine, a control program for a working machine, a control system for a working machine, and a working machine, which are used for a working machine having a function of detecting an object to be detected in a surrounding monitoring area.

Background Art

[0002] As related art, a working machine (excavator) capable of detecting an object existing around is known (see, for example, Patent Document 1). The working machine according to the related art includes a camera attached to an upper swing body, a display device, and an object detection device. The object detection device detects a predetermined object within a predetermined detection range set around the excavator. In this working machine, the image displayed on the display device includes a first area for displaying a camera image and a second area for displaying an operating state of the object detection function.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above related art, the amount of information displayed on the display device is large, and there is a possibility that the operator may overlook the operating state of the detection processing unit (object detection function), or it is necessary to pay attention to the display device so as not to overlook the operating state of the detection device.

[0005] An object of the present invention is to provide a control method for a working machine, a control program for a working machine, a control system for a working machine, and a working machine, in which the operator can easily intuitively grasp the operating state of the detection processing unit.

Means for Solving the Problems

[0006] A control method for a working machine according to an aspect of the present invention includes: acquiring a captured image of a monitoring area around the working machine; causing a display device to display a display screen including the captured image; and when a detection processing unit for detecting a detection target in the monitoring area is valid, displaying an effective object that moves in at least one of the circumferential direction and the radial direction of a virtual circle centered on a reference point set for the captured image, overlaid on the captured image in the display screen.

[0007] A control program for a working machine according to an aspect of the present invention is a program for causing one or more processors to execute the control method for the working machine.

[0008] A control system for a working machine according to an aspect of the present invention includes an image acquisition unit and a display processing unit. The image acquisition unit acquires a captured image of a monitoring area around the working machine. The display processing unit causes a display device to display a display screen including the captured image. When a detection processing unit for detecting a detection target in the monitoring area is valid, the display processing unit displays an effective object that moves in at least one of the circumferential direction and the radial direction of a virtual circle centered on a reference point set for the captured image, overlaid on the captured image in the display screen.

[0009] A working machine according to an aspect of the present invention includes the control system for the working machine and a machine body on which the display device is mounted.

Advantages of the Invention

[0010] According to the present invention, it is possible to provide a control method for a working machine, a control program for a working machine, a control system for a working machine, and a working machine, which enable an operator to intuitively grasp the operating state of the detection processing unit.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

BEST MODE FOR CARRYING OUT THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The following embodiments are an example of embodying the present invention and are not intended to limit the technical scope of the present invention.

[0013] (Embodiment 1) [1] Overall Configuration As shown in FIG. 1, the working machine 3 according to this embodiment includes a traveling unit 31, a slewing unit 32, and a working unit 33 on the machine body 30. Further, as shown in FIG. 2, the working machine 3 further includes a control system 1 for working machines (hereinafter also simply referred to as "control system 1"). In addition, the machine body 30 further includes a display device 2 and an operating device, etc.

[0014] "Working machine" as used in this disclosure means various working machines. As an example, it is a work vehicle such as a backhoe (including hydraulic excavators, mini excavators, etc.), a wheel loader, and a carrier. The working machine 3 includes a working unit 33 configured to be able to execute one or more operations including at least lifting operations. The working machine 3 is not limited to a "vehicle", and may be, for example, a working flying object such as a working ship, a drone, or a multicopter. Further, the working machine 3 is not limited to construction machinery (construction machines), and may be, for example, agricultural machinery (agricultural machines) such as a rice transplanter, a tractor, or a combine. In this embodiment, unless otherwise specified, the working machine 3 is a backhoe with a lifting function (crane function), and taking as an example the case where in addition to lifting operations, excavation operations, leveling operations, trench excavation operations, or loading operations, etc. can be executed as operations for explanation.

[0015] Also, in this embodiment, for convenience of explanation, the vertical direction in the state where the working machine 3 can be used is defined as the up-down direction D1. Further, in the non-slewing state of the slewing unit 32, the front-rear direction D2 and the left-right direction D3 are defined based on the direction seen from the user (operator) boarding the working machine 3 (the operating unit 321). In other words, each direction used in this embodiment is a direction defined with reference to the machine body 30 of the working machine 3, and the direction in which the machine body 30 moves when the working machine 3 moves forward is "forward", and the direction in which the machine body 30 moves when the working machine 3 moves backward is "backward". Similarly, the direction in which the front end of the machine body 30 moves when the working machine 3 slews to the right is "right", and the direction in which the front end of the machine body 30 moves when the working machine 3 slews to the left is "left". However, these directions are not intended to limit the usage direction (direction during use) of the working machine 3.

[0016] The working machine 3 is equipped with an engine as a power source. In the working machine 3, for example, the hydraulic pump 41 (see Fig. 2) is driven by the engine, and hydraulic oil is supplied from the hydraulic pump 41 to the hydraulic actuators (including the hydraulic motor 43 and the hydraulic cylinder 44, etc.) of each part of the machine body 30, so that the machine body 30 is driven. Further, the working machine 3 is controlled, for example, by a user (operator) boarding the operation part 321 of the machine body 30 operating an operation lever or the like of the operation device.

[0017] In the present embodiment, since it is assumed that the working machine 3 is a ride-on type backhoe as described above, the working part 33 is driven according to the operation of a user (operator) boarding the operation part 321, and performs operations such as excavation work. The operation part 321 where the user boards is provided on the slewing part 32.

[0018] The traveling part 31 has a traveling function and is configured to be able to travel (including slewing) on the ground. The traveling part 31 has, for example, a pair of left and right crawlers 311 and a blade 312, etc. The traveling part 31 further has a hydraulic motor 43 (hydraulic actuator) for traveling to drive the crawler 311.

[0019] The slewing part 32 is located above the traveling part 31 and is configured to be able to slew around a rotation axis along the vertical direction with respect to the traveling part 31. The slewing part 32 has a hydraulic motor (hydraulic actuator) for slewing, etc. The slewing part 32 is equipped with, in addition to the operation part 321, an engine and a hydraulic pump 41, etc. Further, at the front end of the slewing part 32, a boom bracket 322 to which the working part 33 is attached is provided.

[0020] The working part 33 is configured to be able to perform operations including lifting work. The working part 33 is supported by the boom bracket 322 of the slewing part 32 and performs operations. The working part 33 has a bucket 331, a boom 332, an arm 333, etc. The working part 33 further has a hydraulic actuator (including a hydraulic cylinder 44 and a hydraulic motor, etc.) for driving each part.

[0021] The bucket 331 is a type of attachment (working tool) attached to the body 30 of the working machine 3, and is composed of an arbitrary tool selected according to the content of the work from among a plurality of types of attachments. The bucket 331 is, as an example, detachably attached to the body 30 and is exchanged according to the content of the work. As attachments for the working machine 3, for example, in addition to the bucket 331, there are various tools such as a breaker, an auger, a crusher, a fork, a fork clamp, a steel cutter, an asphalt cutter, a lawn mower, a ripper, a multi-purpose tool, a tilt rotator, and a tamper. The working unit 33 executes work by driving the bucket 331 with power from a driving device.

[0022] The boom 332 is rotatably supported by a boom bracket 322 of the slewing unit 32. Specifically, the boom 332 is rotatably supported by the boom bracket 322 about a rotation axis along the horizontal direction. The boom 332 has a shape extending upward from a base end portion supported by the boom bracket 322. The arm 333 is connected to the tip of the boom 332. The arm 333 is rotatably supported by the boom 332 about a rotation axis along the horizontal direction. The bucket 331 is attached to the tip of the arm 333.

[0023] The working unit 33 operates by receiving power from an engine as a power source. Specifically, the hydraulic pump 41 is driven by the engine, and hydraulic oil is supplied from the hydraulic pump 41 to the hydraulic actuators (such as the hydraulic cylinder 44) of the working unit 33, whereby each part (the bucket 331, the boom 332, and the arm 333) of the working unit 33 operates.

[0024] In particular, in the present embodiment, the working unit 33 has a multi-joint type structure in which the boom 332 and the arm 333 are individually rotatable. That is, by each of the boom 332 and the arm 333 rotating about a rotation axis along the horizontal direction, for example, the multi-joint type working unit 33 including the boom 332 and the arm 333 can perform operations such as extending and folding as a whole.

[0025] For each of the traveling unit 31 and the slewing unit 32, similar to the working unit 33, it operates by receiving power from the engine as a power source. That is, the hydraulic motor 43 of the traveling unit 31 and the hydraulic motor of the slewing unit 32 etc. are supplied with hydraulic oil from the hydraulic pump 41, whereby the slewing unit 32 and the traveling unit 31 operate.

[0026] The engine functions as a power source for supplying power to each part as described above. Here, the engine is mounted on the slewing unit 32 together with the hydraulic pump 41 etc. In this embodiment, as an example, the engine is a diesel engine. The engine is driven by fuel (here, light oil) being supplied from the fuel tank.

[0027] Here, the machine body 30 is provided with various sensors (including cameras) for detecting a detection object Ob1 (see FIG. 3) in the monitoring area A1 (see FIG. 3) around the working machine 3, such as a camera for imaging the periphery of the machine body 30. In this embodiment, as an example, as shown in FIG. 3, a plurality (here, three) of cameras (imaging devices) including a left camera 341, a right camera 342, and a rear camera 343 are mounted on the slewing unit 32 of the machine body 30. Further, as shown in FIG. 3, a detection device 5 for detecting the detection object Ob1 in the monitoring area A1 is mounted on the slewing unit 32 of the machine body 30.

[0028] The left camera 341, the right camera 342, and the rear camera 343 are connected to the control system 1 and output the images captured by each to the control system 1. FIG. 3 is a plan view of the working machine 3 seen from above, schematically showing the monitoring area A1 set around the working machine 3, the detection object Ob1, and the machine body 30 (including the left camera 341, the right camera 342, and the rear camera 343) of the working machine 3.

[0029] The left camera 341, the right camera 342, and the rear camera 343 are installed to face left, right, and rear with respect to the operation unit 321 so as to be able to image the monitoring area A1 that is to the left, right, and rear as seen from the operator who has boarded the operation unit 321 of the swivel unit 32. That is, as shown in FIG. 3, the monitoring area A1 includes a plurality (here, three) of small areas A11, A12, and A13. The left camera 341 images the small area A11 (left area) that is to the left as seen from the operator who has boarded the operation unit 321. Similarly, the right camera 342 images the small area A12 (right area) that is to the right as seen from the operator who has boarded the operation unit 321, and the rear camera 343 images the small area A13 (rear area) that is to the rear as seen from the operator who has boarded the operation unit 321. Thereby, it becomes possible to cover the sides (left and right) and the rear, which are likely to be blind spots for the operator, with the left camera 341, the right camera 342, and the rear camera 343.

[0030] The detection device 5 is connected to the control system 1 and outputs the detection result of the detection target object Ob1 in the monitoring area A1 to the control system 1. Here, as an example, the detection device 5 is installed to face rearward so as to be able to detect the detection target object Ob1 within the monitoring area A1, which is the monitoring area A1 that can be photographed by the left camera 341, the right camera 342, and the rear camera 343. That is, the detection range of the detection target object Ob1 by the detection device 5 is common to the photographable range (monitoring area A1) of the left camera 341, the right camera 342, and the rear camera 343.

[0031] In this embodiment, the detection device 5 includes a sensor 51 (see FIG. 2) and an imaging unit 52 (see FIG. 2). The sensor 51 is a three-dimensional sensor that measures the distance to the detection target object Ob1 by the TOF (Time Of Flight) method that measures the distance to the measurement point based on the round-trip time until radio waves, light, sound, etc. reach the measurement point and return. The sensor 51 is a ranging sensor that uses radio waves as a medium, for example, like a millimeter-wave radar, to specify the distance to the detection target object Ob1, the azimuth where the detection target object Ob1 exists, and the like. The imaging unit 52 is a camera (including an image sensor and an optical element) that captures an image of the detection area (monitoring area A1) of the sensor 51. Thus, according to the detection device 5 that combines the sensor 51 and the imaging unit 52, when the detection target object Ob1 exists in the monitoring area A1, the three-dimensional position and attributes (shape, size, color, movement, etc.) of the detection target object Ob1 can be specified.

[0032] That is, the detection result of the detection device 5 may include the presence or absence of the detection target object Ob1 in the monitoring area A1, the position of the detection target object Ob1 in the monitoring area A1 when the detection target object Ob1 exists in the monitoring area A1, the attributes of the detection target object Ob1, and the like.

[0033] In short, the detection device 5 detects the detection target object Ob1 in the monitoring area A1 around the working machine 3. The detection device 5 determines the presence or absence (existence or non-existence) of the detection target object Ob1 in the monitoring area A1 and outputs a detection result indicating whether the detection target object Ob1 exists in the monitoring area A1. In this embodiment, as an example, the detection target object Ob1 is a "person". That is, as a result of the working machine 3 moving or the "person" around the working machine 3 moving, when a "person" enters the monitoring area A1 around the working machine 3, the detection device 5 detects the "person" as the detection target object Ob1. When there are a plurality of detection target objects Ob1 in the monitoring area A1, the detection device 5 may also detect including the number (number of people) of the detection target objects Ob1.

[0034] In FIG. 2, the hydraulic circuit and the electric circuit (electrical connection relationship) of the work machine 3 according to the present embodiment are schematically shown. In FIG. 2, solid lines represent high-pressure oil passages (for hydraulic oil), dotted lines represent low-pressure oil passages (for pilot oil), and the arrow of the dashed-dotted line indicates the path of the electrical signal.

[0035] As shown in FIG. 2, in addition to the hydraulic pump 41, the hydraulic motor 43 (not shown in FIG. 2), and the hydraulic cylinder 44, the work machine 3 includes a pilot pump 42, a remote control valve 45, a control valve 46, and a direction change valve (control valve) 47 and the like.

[0036] The hydraulic oil from the hydraulic pump 41 driven by the engine is supplied to the hydraulic motor 43 of the traveling unit 31, the hydraulic motor of the slewing unit 32, and the hydraulic cylinder 44 of the working unit 33 and the like. Thereby, hydraulic actuators such as the hydraulic motor 43 and the hydraulic cylinder 44 are driven.

[0037] A pilot type direction change valve 47 capable of switching the direction and flow rate of the hydraulic oil from the hydraulic pump 41 is provided in the hydraulic actuators such as the hydraulic motor 43 and the hydraulic cylinder 44. The direction change valve 47 is driven by being supplied with pilot oil serving as an input command from the pilot pump 42.

[0038] Here, for example, a remote control valve 45 is provided in the supply path of the pilot oil to the direction change valve 47 corresponding to the hydraulic cylinder 44 of the working unit 33. The remote control valve 45 outputs a work operation command for the working unit 33 according to the operation of the operation lever. The work operation command instructs the deployment operation and the contraction operation and the like of the working unit 33. Further, an electromagnetic control valve 46 (solenoid valve) is inserted between the remote control valve 45 and the pilot pump 42. The control valve 46 is connected to the power source 351 via the cutoff relay 352 and the cutoff switch 353, and operates according to the supply current from the power source 351.

[0039] Similarly, a remote control valve is also provided in the pilot oil supply passage to the direction switching valve corresponding to the hydraulic motor 43 of the traveling unit 31. This remote control valve outputs a traveling operation command for the traveling unit 31 according to the operation of the operation lever. The traveling operation command instructs the traveling operation (such as forward or backward) of the traveling unit 31. Further, a remote control valve is also provided in the pilot oil supply passage to the direction switching valve corresponding to the hydraulic motor of the slewing unit 32. This remote control valve outputs a slewing operation command for the slewing unit 32 according to the operation of the operation lever. The slewing operation command instructs the slewing operation (such as left slewing or right slewing) of the slewing unit 32. And an electromagnetic control valve 46 (solenoid valve) is inserted between these remote control valves and the pilot pump 42. The control valve 46 is connected to the power supply 351 via the cutoff relay 352 and the cutoff switch 353, and operates according to the supply current from the power supply 351.

[0040] The control valve 46 opens the pilot oil flow path from the pilot pump 42 to the remote control valve 45 in the energized state, that is, the state where current is supplied, and shuts off the pilot oil flow path in the non-energized state, that is, the state where the supply current is cut off. Therefore, when the supply current to the control valve 46 is cut off, the hydraulic actuator corresponding to the remote control valve 45 becomes inoperable, and the output of the hydraulic actuator is forcibly stopped regardless of the operation of the operation lever.

[0041] Here, the cutoff relay 352 is connected to the control system 1 and switches between on and off according to a control signal (electrical signal) from the control system 1. The cutoff switch 353 switches between on and off according to the operation of a cutoff lever. For example, it turns on when the cutoff lever is operated downward. Therefore, when both the cutoff relay 352 and the cutoff switch 353 are on, the control valve 46 is energized, and the flow path of the pilot oil from the pilot pump 42 to the remote control valve 45 is opened, so the hydraulic actuator is driven according to the operation of the operation lever. On the other hand, when at least one of the cutoff relay 352 and the cutoff switch 353 is off, the control valve 46 is de-energized, and the flow path of the pilot oil is blocked, so the hydraulic actuator cannot be driven.

[0042] For example, when at least one of the cutoff relay 352 and the cutoff switch 353 connected to the control valve 46 inserted between the remote control valve corresponding to the hydraulic motor of the swing unit 32 and the pilot pump 42 is off, the hydraulic motor of the swing unit 32 cannot be driven. In this state, the output of the hydraulic actuator (hydraulic motor of the swing unit 32) is forcibly stopped regardless of the operation of the operation lever, so the swinging operation of the swing unit 32 is prohibited.

[0043] The control system 1 mainly consists of a computer system having one or more processors such as a CPU (Central Processing Unit) and one or more memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and executes various processes (information processing). In the present embodiment, the control system 1 is an integrated controller that controls the entire working machine 3 and is composed of, for example, an electronic control unit (ECU: Electronic Control Unit). However, the control system 1 may be provided separately from the integrated controller. The control system 1 will be described in detail in the section "[2] Configuration of the Control System".

[0044] The display device 2 is arranged in the operation unit 321 of the aircraft 30, and is a user interface for receiving operation inputs by the user (operator) and outputting various information to the user. The display device 2 receives various operations by the user, for example, by outputting an electrical signal according to the user's operation. As a result, the user (operator) can visually recognize the display screen Dp1 (see FIG. 4) displayed on the display device 2, and can operate the display device 2 as necessary.

[0045] As shown in FIG. 2, the display device 2 includes a control unit 21, an operation unit 22, and a display unit 23. The display device 2 is configured to be communicable with the control system 1, and data can be exchanged between the display device 2 and the control system 1. In this embodiment, as an example, the display device 2 is a dedicated device used for the working machine 3.

[0046] The control unit 21 controls the display device 2 according to the data from the control system 1. Specifically, the control unit 21 outputs an electrical signal according to the user operation received by the operation unit 22, or causes the display unit 23 to display the display screen Dp1 generated by the control system 1.

[0047] The operation unit 22 is a user interface for receiving operation inputs by the user (operator) with respect to the display screen Dp1 displayed on the display unit 23. The operation unit 22 receives various operations by the user U1, for example, by outputting an electrical signal according to the operation of the user U1 (see FIG. 4). In this embodiment, as an example, the operation unit 22 includes a plurality (here, six) of mechanical push-button switches 221 to 226 as shown in FIG. 4. These plurality of push-button switches 221 to 226 are arranged close to the display area (below in the example of FIG. 4) along the periphery of the display area of the display unit 23. These plurality of push-button switches 221 to 226 are associated with the items displayed on the display screen Dp1 described later, and when any one of the plurality of push-button switches 221 to 226 is operated, any one of the items on the display screen Dp1 is operated (selected).

[0048] Further, the operation unit 22 may include a touch panel, an operation dial, and the like. Even in this case, any item on the display screen Dp1 will be operated (selected) by an operation on the operation unit 22.

[0049] The display unit 23 is a user interface for presenting information to the user U1 (operator), such as a liquid crystal display or an organic EL display that displays various types of information. The display unit 23 presents various types of information to the user by display. In this embodiment, as an example, the display unit 23 is a full-color liquid crystal display with a backlight, and as shown in FIG. 4, it has a horizontally long "landscape" display area.

[0050] The display device 2 presents various types of information on the display screen Dp1 to the user U1 (operator) who operates the working machine 3. That is, the user U1 who operates the working machine 3 can visually obtain various types of information related to the working machine 3 by looking at the display screen Dp1 displayed on the display device 2. As an example, by displaying information related to the operating state of the working machine 3, such as the coolant water temperature and the operating oil temperature, on the display device 2, the user U1 can confirm, on the display device 2, the information related to the operating state of the working machine 3 necessary for operating the working machine 3. Further, the display device 2 can also display, on the display screen Dp1, the peripheral images of the working machine 3 (images of the monitoring area A1) captured by the left camera 341, the right camera 342, and the rear camera 343. Thereby, when operating the working machine 3, the user U1 (operator) can confirm, on the display screen Dp1 displayed on the display device 2, the situations on the sides and rear of the working machine 3, which are likely to be blind spots from the driver's cab 321, for example.

[0051] Furthermore, the working machine 3 is provided with a sound output unit 36 (see FIG. 2) that outputs sound (including voice) to the user U1 (operator). The sound output unit 36 includes a buzzer, a speaker, or the like, and outputs sound upon receiving an electrical signal. The sound output unit 36 is connected to the control system 1 and outputs sound such as a beep sound or voice according to a sound control signal from the control system 1. In the present embodiment, the sound output unit 36 is provided in the operation unit 321 of the machine body 30 in the same manner as the display device 2. The sound output unit 36 may be provided integrally with the display device 2.

[0052] In addition to the above-described configuration, the machine body 30 further includes an operation lever, a cut-off lever, a communication terminal, a fuel tank, a battery, and the like. Furthermore, the machine body 30 is provided with sensors for monitoring the operating state of the machine body 30, such as a coolant water temperature sensor, an operating oil temperature sensor, a tachometer for measuring the engine speed, and an hour meter for measuring the operating time. In addition, the machine body 30 is also provided with sensors for detecting the states of a cut-off lever, a starter key switch, and the like.

[0053] [2] Configuration of the control system Next, the configuration of the control system 1 according to the present embodiment will be described with reference to FIG. 2. The control system 1 controls the display device 2 to display a display screen Dp1 on the display device 2. In the present embodiment, the display device 2 is mounted on the machine body 30 of the working machine 3 as described above. The control system 1 is a component of the working machine 3 and constitutes the working machine 3 together with the machine body 30 and the like. In other words, the working machine 3 according to the present embodiment includes at least the control system 1 and the machine body 30 (including the traveling unit 31, the slewing unit 32, and the working unit 33) on which the display device 2 is mounted.

[0054] As used in the present disclosure, the "screen" such as the display screen Dp1 means a video (image) displayed on the display device 2 and includes images, graphics, photographs, text, videos, and the like. That is, the control system 1 can cause the display device 2 to display a display screen Dp1 including, for example, an image representing information regarding the operating state of the working machine 3 such as the coolant water temperature and the operating oil temperature. Here, when the display screen Dp1 includes a video or the like, the display screen Dp1 includes a video that changes moment by moment rather than a fixed video.

[0055] As shown in FIG. 2, the control system 1 includes a display processing unit 11, a restraint processing unit 12, a switching processing unit 13, an image acquisition unit 14, and a detection processing unit 15. In the present embodiment, as an example, the control system 1 is mainly configured by a computer system having one or more processors. By executing a control program for a working machine by one or more processors, these multiple functional units (display processing unit 11, etc.) are realized. These multiple functional units included in the control system 1 may be distributed and provided in multiple enclosures, or may be provided in one enclosure.

[0056] The control system 1 is configured to be communicable with devices provided in each part of the machine body 30. That is, at least the display device 2, the detection device 5, the sound output unit 36, the cutoff relay 352, the left camera 341, the right camera 342, and the rear camera 343 are connected to the control system 1. Thereby, the control system 1 can control the display device 2, the sound output unit 36, etc., control the control valve 46 by controlling the cutoff relay 352, and acquire the detection results of the detection device 5 and the captured images of the left camera 341, the right camera 342, the rear camera 343, and the like. Here, the control system 1 may directly exchange various information (data) with each device, or may indirectly exchange information through a repeater or the like.

[0057] The image acquisition unit 14 executes an image acquisition process for acquiring a captured image of the monitoring area A1 around the working machine 3. In the present embodiment, the image acquisition unit 14 periodically or irregularly acquires the outputs of the left camera 341, the right camera 342, and the rear camera 343 from the left camera 341, the right camera 342, and the rear camera 343. That is, the image acquisition unit 14 acquires image data (captured images) of the monitoring area A1 (each small area A11, A12, A13) around the working machine 3. The data acquired by the image acquisition unit 14 is stored, for example, in a memory or the like.

[0058] The detection processing unit 15 executes a detection process for detecting a detection target object Ob1 in the monitoring area A1. In the present embodiment, since the detection device 5 detects the detection target object Ob1 in the monitoring area A1, the detection processing unit 15 detects the detection target object Ob1 in the monitoring area A1 by acquiring the detection result of the detection device 5 from the detection device 5.

[0059] The restraint processing unit 12 executes a restraint process for restraining the operation of the working machine 3 based on the detection result of the detection device 5. In the present embodiment, when the detection result of the detection device 5 indicates the presence of a detection target object Ob1 (here, a person) in the monitoring area A1, the restraint processing unit 12 executes the restraint process. The "restraint process" as used in the present disclosure means a process that acts in a direction of suppressing something with respect to the operation of the working machine 3. As an example, the restraint process includes a process of indirectly restraining the operation of the working machine 3 by warning the user U1 (operator) who operates the working machine 3 by sound or light (including display). Further, the restraint process includes a process of directly restraining the operation of the working machine 3 by controlling the traveling unit 31, the turning unit 32, the working unit 33, etc. of the working machine 3.

[0060] In the present embodiment, the restraint processing unit 12 includes a sound output processing unit 121 and a restriction processing unit 122.

[0061] When the detection target object Ob1 exists in the monitoring area A1, the sound output processing unit 121 controls the sound output unit 36 to output a notification sound from the sound output unit 36. That is, in this embodiment, the deterrence process includes a sound output process for outputting a notification sound. The notification sound may be a simple beep sound or a voice such as "Please be careful". Furthermore, the notification sound may change according to the detection result of the detection device 5 (such as the distance from the aircraft 30 to the detection target object Ob1). Thereby, by warning the user U1 (operator) who operates the work machine 3 with the notification sound, the operation of the work machine 3 can be indirectly deterred, so the degree of freedom of operation of the work machine 3 is high. That is, by the user U1 paying attention to the detection target object Ob1 and operating the work machine 3, the operation of the work machine 3 can be continued while avoiding contact with the detection target object Ob1.

[0062] When the detection target object Ob1 exists in the monitoring area A1, the restriction processing unit 122 controls the cutoff relay 352 to turn off the cutoff relay 352. As a result, the control valve 46 connected to the power supply 351 via the cutoff relay 352 becomes de-energized, and the output of the hydraulic actuator corresponding to the control valve 46 is forcibly stopped. That is, in this embodiment, the deterrence process includes a restriction process for restricting the operation of the work machine 3. The "restriction process" referred to in the present disclosure means a process that acts in a direction of restricting the operation of the work machine 3 in some way. As an example, the restriction process includes a process of prohibiting the traveling operation of the traveling unit 31 (making the traveling operation impossible), a process of prohibiting the turning operation of the turning unit 32 (making the turning operation impossible), and a process of prohibiting the operation of the working unit 33 (making the work impossible), etc. Thereby, the operation of the work machine 3 can be forcibly restricted regardless of the operation of the user U1 (operator). That is, contact between the aircraft 30 and the detection target object Ob1 caused by the operation of the work machine 3 can be avoided.

[0063] Here, the restriction process executed by the restriction processing unit 122 includes at least a process of restricting the turning operation of the turning unit 32. Specifically, the restriction processing unit 122 is configured to be able to control a cutoff relay 352 connected to a control valve 46 corresponding to the hydraulic motor of the turning unit 32, and when a detection target object Ob1 exists in the monitoring area A1, turns off the cutoff relay 352. As a result, when a detection target object Ob1 exists in the monitoring area A1, the hydraulic motor of the turning unit 32 becomes inoperable. If the turning unit 32 is in the turning operation, the turning unit 32 makes an emergency stop, and if the turning unit 32 is not in the turning operation, the turning operation of the turning unit 32 is prohibited. That is, in the present embodiment, the working machine 3 includes a traveling unit 31 and a turning unit 32 that can turn with respect to the traveling unit 31. The restriction process restricts at least the turning operation of the turning unit 32. Thereby, when a detection target object Ob1 exists in the monitoring area A1, which is a blind spot for the user U1 (operator), contact between the machine body 30 and the detection target object Ob1 due to the turning of the turning unit 32 can be avoided.

[0064] The switching processing unit 13 switches between enabling and disabling the detection processing unit 15. That is, the switching processing unit 13 switches between enabling and disabling the function related to the detection process of the detection target object Ob1 in the monitoring area A1 by the detection processing unit 15. In short, the detection processing unit 15 is not always enabled, and it is possible to switch between enabled / disabled. If the detection processing unit 15 is enabled, when a detection target object Ob1 exists in the monitoring area A1, the detection target object Ob1 is detected by the detection processing unit 15, so the deterrence process by the deterrence processing unit 12 is executed. On the other hand, if the detection processing unit 15 is disabled, even when a detection target object Ob1 exists in the monitoring area A1, the detection target object Ob1 is not detected by the detection processing unit 15, so the deterrence process by the deterrence processing unit 12 is not executed.

[0065] In this embodiment, the switching processing unit 13 switches between enabling and disabling the detection processing unit 15 depending on whether or not to operate the detection device 5. That is, when the switching processing unit 13 enables the detection processing unit 15, it operates the detection device 5, and when it disables the detection processing unit 15, it does not operate the detection device 5. Thereby, power consumption by the detection device 5 can be suppressed when the detection processing unit 15 is disabled.

[0066] However, not limited to this configuration, the switching processing unit 13 may disable the detection processing unit 15, for example, by invalidating the detection processing (acquisition of detection results from the detection device 5) of the detection processing unit 15 or invalidating the acquired detection results. In this case, when the detection processing unit 15 is disabled, although the detection device 5 itself operates, even if the detection target object Ob1 exists in the monitoring area A1, the deterrence processing by the deterrence processing unit 12 is not executed. In short, the switching processing unit 13 may disable the detection processing unit 15 by invalidating the processing using the detection results of the detection device 5.

[0067] In this embodiment, as an example, the switching between enabling and disabling the detection processing unit 15 is performed by the user U1 (operator) operating the display device 2. That is, when the user U1 operates the operation unit 22 of the display device 2 to enable the detection processing unit 15, the switching processing unit 13 receives this operation and enables the detection processing unit 15. On the other hand, when the user U1 operates the operation unit 22 of the display device 2 to disable the detection processing unit 15, the switching processing unit 13 receives this operation and disables the detection processing unit 15.

[0068] In addition, in the present embodiment, the deterrence process performed by the deterrence processing unit 12 includes a sound output process executed by the sound output processing unit 121 and a restriction process executed by the restriction processing unit 122. Thus, the deterrence process includes a plurality of specific processes (such as sound output process and restriction process) for deterring the operation of the work machine 3. Here, the deterrence process may be individually switchable between effective and ineffective for each specific process. That is, the switching processing unit 13 may be individually switchable between effective and ineffective for the sound output processing unit 121 and the restriction processing unit 122 in the deterrence processing unit 12. As an example, it is possible to enable the sound output processing unit 121 and disable the restriction processing unit 122, or disable the sound output processing unit 121 and enable the restriction processing unit 122. Thereby, only the necessary specific processes can be enabled according to the situation, and the degree of freedom of the deterrence process is improved.

[0069] The display processing unit 11 executes a display process for causing at least the display screen Dp1 to be displayed on the display device 2. Specifically, the display processing unit 11 generates the display screen Dp1 based on data obtained by the image acquisition unit 14 and controls the display device 2 to display the display screen Dp1 on the display unit 23 of the display device 2. Further, the display processing unit 11 operates according to an operation received by the operation unit 22 of the display device 2. The display processing unit 11, for example, causes the captured images Im100 (see FIG. 5) captured by the left camera 341, the right camera 342, and the rear camera 343 to be displayed on the display screen Dp1. That is, the display processing unit 11 causes the image acquisition unit 14 to display an image of the monitoring area A1 around the work machine 3 on the display device 2.

[0070] Here, in addition to the captured image Im100, the display processing unit 11 can display on the display screen Dp1 a valid object X1 (see FIG. 5) indicating that the detection processing unit 15 is valid. When the detection processing unit 15 is valid, the display processing unit 11 displays the valid object X1 overlaid on the captured image Im100 on the display screen Dp1. When the detection processing unit 15 is invalid, the display processing unit 11 does not display the valid object X1. The "object" such as the valid object X1 in the present disclosure includes marks, images, figures, photos, texts, videos, etc. displayed in the display screen Dp1, or combinations thereof.

[0071] Thereby, the user U1 (operator) can grasp that the detection processing unit 15 is valid by the fact that the valid object X1 is displayed on the display screen Dp1. That is, the user U1 can visually confirm whether the detection processing unit 15 is valid or invalid, and can operate the working machine 3 after knowing whether the detection processing unit 15 is valid or invalid.

[0072] Incidentally, in the present embodiment, the detection processing unit 15 acquires the detection result of the detection device 5 outside the control system 1 and executes the detection processing of the detection target object Ob1 based on the detection result, but the configuration is not limited thereto. For example, the detection processing unit 15 may perform the detection processing of the detection target object Ob1 in the monitoring area A1 based on the output of the sensor 51 and / or the imaging unit 52 outside the control system 1.

[0073] [3] Control method of working machine Hereinafter, with reference to FIGS. 5 to 9, an example of a control method (hereinafter simply referred to as "control method") of the working machine 3 mainly executed by the control system 1 will be described.

[0074] Since the control method according to this embodiment is executed by a control system 1 mainly composed of a computer system, in other words, it is embodied in a control program for a working machine (hereinafter simply referred to as the "control program"). That is, the control program according to this embodiment is a computer program for causing one or more processors to execute each process related to the control method. Such a control program may be executed in cooperation by, for example, the control system 1 and the display device 2.

[0075] Here, when a preset specific start operation for executing the control program is performed, the control system 1 executes the following various processes related to the control method. The start operation is, for example, a start operation of the engine of the working machine 3. On the other hand, when a preset specific end operation is performed, the control system 1 ends the following various processes related to the control method. The end operation is, for example, a stop operation of the engine of the working machine 3.

[0076] [3.1] Display Screen Here, first, the configuration of the display screen Dp1 displayed on the display unit 23 of the display device 2 by the control method according to this embodiment will be described. In the drawing showing the display screen Dp1 displayed on the display unit 23 of the display device 2 such as FIG. 5, the dashed-dotted line, the leader line, and the reference numerals representing the regions are only attached for the purpose of explanation and are not actually displayed on the display device 2.

[0077] The display screen Dp1 shown in FIG. 5 is the home screen that is first displayed by the control method. The home screen is the basic display screen Dp1 that will be first displayed on the display device 2 during the operation of the working machine 3. The display screen Dp1 can be transitioned from the home screen to various display screens Dp1 including a menu screen, a crane screen, a mode screen, a PTO screen, etc. according to the operation on the operation unit 22.

[0078] As shown in FIG. 5, the display screen Dp1 includes a first region R1, a second region R2, a third region R3, a fourth region R4, a fifth region R5, a sixth region R6, a seventh region R7, an eighth region R8, a ninth region R9, and a tenth region R10. In the control method according to the present embodiment, as an example, in the second region R2 that occupies most of the display screen Dp1, the captured image Im100 of the monitoring area A1, the valid object X1, etc. are displayed.

[0079] Specifically, the display screen Dp1 is divided into four regions in the vertical direction (up and down direction). And the upper three regions are each further divided into three regions in the horizontal direction (left and right direction). As a result, the display screen Dp1 is divided into a total of ten regions. And the regions in the second row from the top are, in order from the left, the first region R1, the second region R2, and the third region R3. The lowermost region is the fourth region R4. Further, the regions in the third row from the top are, in order from the left, the fifth region R5, the sixth region R6, and the seventh region R7, and the uppermost region is, in order from the left, the eighth region R8, the ninth region R9, and the tenth region R10. In terms of the vertical size, among the four regions divided in the vertical direction, the regions in the second row from the top (the first region R1, the second region R2, and the third region R3) are the widest. In terms of the horizontal size, among the three regions divided in the horizontal direction, the middle region (the second region R2, the sixth region R6, and the ninth region R9) is the widest.

[0080] However, the arrangement and size of each of these regions are only examples and can be changed as appropriate. Also, it is not essential that each region is clearly divided by a boundary line. For example, even in the example of FIG. 5, the second region R2 and the third region R3 are clearly divided by a boundary line, while there is no boundary line between the first region R1 and the second region R2. Of course, the first region R1 and the second region R2 may be clearly divided by a boundary line.

[0081] The first region R1 is a vertically long rectangular region. In the first region R1, for example, the remaining amount information G1 regarding the remaining amount of the fuel (e.g., light oil) of the engine is displayed. The display processing unit 11 generates the remaining amount information G1 in the display screen Dp1 based on the output (sensor signal) of the remaining amount sensor, etc.

[0082] The second area R2 is a horizontally long rectangular area. In the second area R2, the captured image Im100 of the monitoring area A1, the valid object X1, etc. are displayed. In the second area R2, the aircraft object Im10, the captured image Im200 of the rear camera 343, etc. are further displayed.

[0083] The captured image Im100 is an overhead image generated by coordinate-transforming and synthesizing the captured images of the left camera 341, the right camera 342, and the rear camera 343. That is, the captured image Im100 is an image synthesized from the image of the small area A11 on the left side of the driving unit 321 captured by the left camera 341, the image of the small area A12 on the right side of the driving unit 321 captured by the right camera 342, and the image of the small area A13 behind the driving unit 321 captured by the rear camera 343, and is an image that displays the monitoring area A1 as seen from above the aircraft 30. The display processing unit 11 synthesizes the captured images acquired by the image acquisition unit 14 and displays them in real time.

[0084] The valid object X1 is an object indicating that the detection processing unit 15 is valid, and is displayed superimposed on the captured image Im100 on the display screen Dp1. Details of the valid object X1 will be described in the "[3.2 Details]" column.

[0085] The aircraft object Im10 is displayed within the 11th area R11 set in the central part of the second area R2. In this embodiment, as an example, the aircraft object Im10 is an image (icon) imitating the aircraft 30 as seen from above. The captured image Im200 is an image of the small area A13 behind the driving unit 321 captured by the rear camera 343.

[0086] The third region R3 is a vertically long rectangular region. An image (icon) Im1 corresponding to the operating state of each part of the working machine 3 is displayed in the third region R3. A plurality of images Im1 can be displayed in the third region R3, and the design (pattern) of each individual image Im1 indicates which state, for example, battery, seat belt, coolant temperature, hydraulic oil temperature, etc. Here, each image Im1 indicates the operating state by a display mode such as display color or size. The display processing unit 11 determines the state of each part of the working machine 3 using the outputs of various sensors (including a coolant temperature sensor and a hydraulic oil temperature sensor) that detect the operating state of each part of the working machine 3. When an abnormal value is detected at any part, the display processing unit 11 performs a warning display, for example, by changing the display mode such as the display color of the image Im1 of that part.

[0087] The fourth region R4 is a strip-shaped region extending across the entire width of the display screen Dp1. Each item for operating the display screen Dp1 is displayed in the fourth region R4. In FIG. 5, as an example, six items of "Menu", "Crane", "Mode", "Camera", "PTO", and "Switch" are arranged side by side in this order from the left in the fourth region R4. Six push button switches 221 to 226 of the operation unit 22 located directly below these six items are respectively associated with them. For example, the push button switch 221 is associated with the "Menu" item, and the push button switch 222 is associated with the "Crane" item. Therefore, for example, when the push button switch 224 corresponding to the "Camera" item is operated by the user U1 (see FIG. 4), the "Camera" item is operated (selected).

[0088] Furthermore, in the present embodiment, in the fourth region R4, any one of the items is highlighted so as to also correspond to operations such as the operation dial (or cursor keys) of the operation unit 22. In the example of FIG. 5, the item "Menu" is highlighted, and the highlighted item is switched by an operation such as an operation dial (or cursor keys). In the case of the user U1, it is possible to select a desired item by operating the determination button with the desired item highlighted. Therefore, for example, when the determination button is operated with the highlight moved to the item "Camera", the item "Camera" is operated (selected). Further, when the operation unit 22 includes a touch panel, the user U1 can select a desired item by touching the desired item on the display screen Dp1.

[0089] In the fifth region R5, an image (icon) for warning display indicating that an abnormal value has been detected by various sensors (including a coolant temperature sensor and an operating oil temperature sensor) is displayed. In the sixth region R6, for example, information regarding the working unit 33 that is operating in the working machine 3 is displayed. In the seventh region R7, for example, information regarding the operating state of the working machine 3 such as the engine speed is displayed. In the eighth region R8, for example, the current time is displayed. In the ninth region R9, for example, information indicating the item to which the currently displayed display screen Dp1 belongs is displayed. In the tenth region R10, for example, information regarding the operating time (hour meter) of the working machine 3 is displayed.

[0090] [3.2] Details Next, the details of the control method according to the present embodiment will be described.

[0091] The control method according to this embodiment includes acquiring a captured image Im100 of a monitoring area A1 around the work machine 3 (image acquisition process), and causing a display device 2 to display a display screen Dp1 including the captured image Im100 (display process). In the display process, when a detection processing unit 15 that detects a detection target object in the monitoring area A1 is valid, the control method further includes displaying a valid object X1 overlaid on the captured image Im100 in the display screen Dp1. The valid object X1 moves in at least one of the circumferential direction and the radial direction of a virtual circle C1 (see FIG. 6) centered on a reference point P1 (see FIG. 6) set for the captured image Im100.

[0092] In short, when the display screen Dp1 including the captured image Im100 is displayed on the display device 2, the user U1 can check the presence or absence of the detection target object Ob1 in the monitoring area A1 in the captured image Im100 on the display screen Dp1. Here, if the detection processing unit 15 is valid, a valid object X1 that moves in at least one of the circumferential direction and the radial direction of the virtual circle C1 centered on the reference point P1 is displayed overlaid on the captured image Im100 in the display screen Dp1. Therefore, the user U1 can grasp whether the detection processing unit 15 is valid or not by the valid object X1 that is dynamically displayed at a position that naturally enters the field of view when viewing the captured image Im100. Therefore, according to the control method according to this embodiment, it becomes easier for the operator to intuitively grasp the operating state of the detection device 5.

[0093] Hereinafter, with reference to FIGS. 6 to 8, the display of the second area R2 in the display screen Dp1 will be described in more detail. FIGS. 6 to 8 show an example of the display content of the second area R2 in the display screen Dp1. In FIGS. 6 to 8, only the periphery of the second area R2 in the display screen Dp1 is illustrated, and other illustrations are omitted. Further, the reference point P1 and the virtual circle C1 shown in FIGS. 6 and 7 are both virtual points or circles and are not actually displayed on the display device 2.

[0094] As shown in FIG. 6, the captured image Im100 is a fan-shaped image with an arc-shaped outer contour on the side far from the aircraft 30. The "arc shape" as used in the present disclosure includes not only an arc that forms a part of a perfect circle but also all curved lines that are convex outward. Therefore, as shown in FIG. 6, even if the outer contour line of the captured image Im100 consists of a curved line that is convex outward as a whole, it can be said to be a fan-shaped image with an arc-shaped outer contour. Here, the captured image Im100 is a fan shape obtained by cutting out the upper side of the display screen Dp1 from the arc, and the left, right, and lower sides thereof are images of the monitoring areas A1 (each small area A11, A12, A13) corresponding to the left, right, and rear of the aircraft 30, respectively. That is, on the display screen Dp1, the captured image Im100 is displayed in a state where the outer contours of at least the monitoring areas A1 on the left, right, and rear of the working machine 3 are formed in an arc shape.

[0095] More specifically, when a virtual circle C1 centered on a reference point P1 located near the center of the second region R2 is set, the captured image Im100 is a fan shape lacking an exclusion range from a first end E101 to a second end E102, which is a part of the circumferential direction of the virtual circle C1. In the example of FIG. 6, the captured image Im100 has a linear first end E101 extending from the reference point P1 to the upper left and a linear second end E102 extending from the reference point P1 to the upper right on the display screen Dp1.

[0096] Also, as shown in FIG. 6, the aircraft object Im10 is arranged at the apex angle portion of the captured image Im100. That is, the aircraft object Im10 is arranged at a position on the reference point P1 corresponding to the apex angle of the fan-shaped captured image Im100. In other words, the control method according to the present embodiment further includes displaying the aircraft object Im10 at the reference point P1 on the display screen Dp1.

[0097] In this embodiment, the upper part of the aircraft object Im10 corresponds to the front of the aircraft 30 in the real space, the lower part of the aircraft object Im10 corresponds to the rear of the aircraft 30 in the real space, the right part of the aircraft object Im10 corresponds to the right side of the aircraft 30 in the real space, and the left part of the aircraft object Im10 corresponds to the left side of the aircraft 30 in the real space. Specifically, the lower part of the aircraft object Im10 is configured in an arc shape imitating the rear part (counterweight) of the slewing unit 32, and the upper part of the aircraft object Im10 is configured in a shape imitating the front part (working unit 33) of the slewing unit 32. In this embodiment, the working unit 33 is arranged at a position offset to the right with respect to the center of the slewing unit 32 in the left-right direction D3, and the driving unit 321 is arranged at a position offset to the left with respect to the center of the slewing unit 32 in the left-right direction D3. Therefore, also in the aircraft object Im10, a figure imitating the working unit 33 is located in the upper right part, and a figure imitating the driving unit 321 is located in the left part.

[0098] Also, in this embodiment, the aircraft object Im10 is displayed based on the orientation of the slewing unit 32 on which the driving unit 321 is mounted, rather than the orientation of the traveling unit 31. That is, when the slewing unit 32 rotates relative to the traveling unit 31, the orientation in the real space corresponding to the upper part of the aircraft object Im10 changes. As a result, the upper part of the aircraft object Im10 always corresponds to the front of the slewing unit 32, which is in front as seen from the operator boarding the driving unit 321.

[0099] Here, the orientation of the aircraft 30 represented by the aircraft object Im10 is made to match the orientation of the captured image Im100 as seen from the aircraft 30. That is, both the front of the aircraft 30 imitated by the aircraft object Im10 and the front in the captured image Im100 are directed upward on the display screen Dp1. As a result, it is easy for the operator to intuitively grasp the position of the detection target object Ob1 by looking at the captured image Im100. However, not limited to this example, the aircraft object Im10 may be arranged at a position other than the apex angle of the captured image Im100, for example, below the captured image Im100.

[0100] In addition, the captured image Im200 of the rear camera 343 is disposed within the exclusion range between the first end E101 and the second end E102 in the circumferential direction of the virtual circle C1. That is, the captured image Im200 is disposed above the captured image Im100 on the display screen Dp1. In this embodiment, as an example, the captured image Im200 has an outer shape in a pentagon shape including two sides along the first end E101 and the second end E102. Thereby, using the dead space in the second region R2, the captured image Im200 of the small area A13 behind the driving unit 321 captured by the rear camera 343 can be displayed.

[0101] Furthermore, in this embodiment, since the detection target object Ob1 in the monitoring area A1 is the detection target by the detection device 5, when the detection target object Ob1 exists in the monitoring area A1, naturally, the detection target object Ob1 will be reflected in the captured image Im100 of the monitoring area A1. Thus, when the detection result indicates the existence of the detection target object Ob1 in the monitoring area A1, at least a part of the detection target object is reflected in the captured image Im100 on the display screen Dp1. Thereby, for the user U1, it becomes possible to confirm the detection target object Ob1 on the captured image Im100. That is, the operator (user U1) can confirm the situations such as the sides and the rear of the working machine 3 which are likely to be blind spots from the driving unit 321 on the display screen Dp1 displayed on the display device 2. Therefore, when the detection target object Ob1 exists within the monitoring area A1, it becomes easier to grasp the situation of the detection target object Ob1 in detail on the display screen Dp1.

[0102] By the way, when the detection processing unit 15 is effective, on the display screen Dp1, the effective object X1 is displayed in a form overlapping the captured image Im100. The effective object X1 moves in at least one of the circumferential direction and the radial direction of the virtual circle C1 centered on the reference point P1. In this embodiment, the effective object X1 is to move continuously in the circumferential direction of the virtual circle C1.

[0103] As shown in FIGS. 6 and 7, this valid object X1 has a length along the radius of the virtual circle C1 and is an image (object) that moves in the circumferential direction of the virtual circle C1. Specifically, as illustrated in FIG. 6, the valid object X1 is an elongated object extending from a reference point P1, which is the center of the virtual circle C1, toward the arc-shaped outer shape (outer periphery) of the captured image Im100. Such a valid object X1 moves so as to rotate in the clockwise direction (the direction indicated by the dashed arrow in FIG. 7) along the circumferential direction of the virtual circle C1 with the reference point P1 as the center. Thereby, it becomes easier for the valid object X1 to enter the field of view of the user U1, and for the user U1, it becomes easier to grasp whether the detection processing unit 15 is valid or not.

[0104] Here, as described above, the captured image Im100 has a shape lacking an exclusion range from a first end E101 to a second end E102, which is a part of the circumferential direction of the virtual circle C1. And the valid object X1 moves from the first end E101 to the second end E102 skipping the exclusion range as shown in FIG. 8. That is, as shown in the upper part of FIG. 8, when the valid object X1 moves in the clockwise direction (the direction indicated by the dashed arrow in FIG. 8) and reaches the first end E101 of the captured image Im100, it moves to the second end E102 of the captured image Im100 as shown in the lower part of FIG. 8.

[0105] Specifically, the valid object X1 moves continuously on the captured image Im100 in the circumferential direction of the virtual circle C1, and skips the exclusion range (the range from the first end E101 to the second end E102) where the captured image Im100 is interrupted, thereby moving discontinuously from the first end E101 to the second end E102. As a result, the valid object X1 is superimposed only on the captured image Im100 without being superimposed on the captured image Im200 of the rear camera 343. Therefore, for the user U1, it becomes easier to grasp whether the detection processing unit 15 is valid or not.

[0106] Also, in this embodiment, the effective object X1 is transparent. That is, while the effective object X1 is displayed overlapping the captured image Im100, the part of the captured image Im100 where the effective object X1 overlaps is also displayed through the effective object X1. Therefore, for the user U1, the entire captured image Im100 can be visually recognized through the effective object X1, and the visibility of the captured image Im100 is improved compared to the case where the captured image Im100 is partially blocked by the effective object X1.

[0107] Furthermore, the captured image Im100 is an overhead image. That is, the effective object X1 is displayed overlapping the captured image Im100 so as to move on the captured image Im100 which is an overhead image looking down on the monitoring area A1 from above. Therefore, for the user U1, the effective object X1 seems to be like a radar scanning the monitoring area A1. Thus, it becomes easier for the user U1 to intuitively understand that the effective object X1 represents the effectiveness of the detection processing unit 15.

[0108] Also, in the control method according to this embodiment, it is preferable to change the display mode of the effective object X1 according to the detection result of the detection device 5. In this case, since the display mode of the effective object X1 changes according to the detection result of the detection device 5, there are at least two cases: the "non-detection" when the detection target object Ob1 does not exist in the monitoring area A1 and the "detection" when the detection target object Ob1 exists in the monitoring area A1. Furthermore, the display mode of the effective object X1 preferably also changes according to the distance from the working machine 3 to the detection target object Ob1 and / or the orientation of the detection target object Ob1 (that is, the position of the detection target object Ob1). Therefore, even among the "detection" cases where the detection target object Ob1 exists in the monitoring area A1, the display mode of the effective object X1 is further different between the case where the distance from the working machine 3 to the detection target object Ob1 is far and the case where the distance from the working machine 3 to the detection target object Ob1 is near.

[0109] The display mode of the valid object X1 that changes according to the detection result of the detection device 5 includes, for example, the display color of the valid object X1. In this case, the display color of the valid object X1 on the display screen Dp1 will change according to the detection result of the detection device 5. For example, the closer the detection target object Ob1 is to the aircraft 30, the more prominent the display mode of the valid object X1 will be changed to a display color. As an example, when the detection target object Ob1 approaches the aircraft 30, the display color of the valid object X1 changes from yellow to red.

[0110] In addition, the display mode of the valid object X1 that changes according to the detection result of the detection device 5 may include the moving speed, moving range, and / or moving direction of the valid object X1. For example, the closer the detection target object Ob1 is to the aircraft 30, the display mode of the valid object X1 is changed so that the moving speed of the valid object X1 becomes faster.

[0111] In this way, by changing the display mode of the valid object X1 according to the detection result of the detection device 5, for the user U1, it becomes easier to intuitively grasp the presence of the detection target object Ob1 (for example, a person) in the monitoring area A1 based on the display mode of the valid object X1.

[0112] Here, there may be a plurality of detection target objects Ob1 in the monitoring area A1 at the same time. In this case, it is preferable that the display mode of the valid object X1 is determined according to the position of the detection target object Ob1 closest to the aircraft 30 of the working machine 3 among the plurality of detection target objects Ob1. For example, when a detection target object Ob1 existing at a position within the threshold distance from the working machine 3 and a detection target object Ob1 existing at a position farther than the threshold distance from the working machine 3 are detected at the same time, the display mode of the valid object X1 is determined based on the detection target object Ob1 existing at a position within the threshold distance from the working machine 3. Thereby, since the display mode of the valid object X1 is determined according to the detection result of the detection target object Ob1 with the highest urgency, it becomes easier for the operator to intuitively grasp the detection result.

[0113] [3.3] Overall Processing Next, the overall flow of the process related to the control method will be described with reference to FIG. 9. FIG. 9 is a flowchart showing an example of the process related to the control method.

[0114] As shown in FIG. 9, the image acquisition unit 14 of the control system 1 acquires the captured image from the left camera 341, the right camera 342, and the rear camera 343 (S1). Then, the display processing unit 11 of the control system 1 causes the display device 2 to display a display screen Dp1 including the captured image Im100 (S2).

[0115] Here, the display processing unit 11 of the control system 1 determines whether the detection processing unit 15 is valid or invalid by the switching processing unit 13 (S3). When the detection processing unit 15 is valid (S3: Yes), the display processing unit 11 superimposes and displays a valid object X1 that moves in the circumferential direction of the virtual circle C1 on the captured image Im100 in the display screen Dp1 (S4). On the other hand, when the detection processing unit 15 is invalid (S3: No), the display processing unit 11 makes the valid object X1 non-displayed (S5).

[0116] The control system 1 repeatedly executes the processes of steps S1 to S5 above. As a result, the display screen Dp1 is displayed on the display device 2 at any time. However, the flowchart shown in FIG. 9 is only an example, and processes may be added or omitted as appropriate, or the order of the processes may be changed as appropriate.

[0117] [4] Modification Hereinafter, modifications of Embodiment 1 will be listed. The modifications described below can be applied in appropriate combinations.

[0118] The control system 1 in the present disclosure includes a computer system. The computer system mainly consists of one or more processors as hardware and one or more memories. By the processor executing a program recorded in the memory of the computer system, the functions as the control system 1 in the present disclosure are realized. The program may be pre-recorded in the memory of the computer system, may be provided through a telecommunication line, or may be provided by being recorded in a non-transitory recording medium such as a memory card, an optical disk, or a hard disk drive that can be read by the computer system. Also, some or all of the functional units included in the control system 1 may be composed of electronic circuits.

[0119] Moreover, it is not an essential configuration of the control system 1 that at least some of the functions of the control system 1 are integrated in one housing, and the components of the control system 1 may be provided dispersedly in a plurality of housings. On the contrary, in Embodiment 1, functions dispersed in a plurality of devices (for example, the control system 1 and the display device 2) may be integrated in one housing. Furthermore, at least some of the functions of the control system 1 may be realized by a cloud (cloud computing) or the like.

[0120] Also, the power source of the working machine 3 is not limited to a diesel engine, and for example, it may be an engine other than a diesel engine, a motor (electric motor), or a hybrid power source including an engine and a motor (electric motor).

[0121] Also, the display device 2 is not limited to a dedicated device, and for example, it may be a general-purpose terminal such as a laptop computer, a tablet terminal, or a smartphone. Furthermore, the display unit 23 is not limited to a mode of directly displaying a display screen, such as a liquid crystal display or an organic EL display, and for example, it may be a configuration that displays a display screen by projection, such as a projector.

[0122] In addition, as a mode of inputting information of the operation unit 22, modes other than the push button switch, the touch panel, and the operation dial may be adopted. For example, the operation unit 22 may adopt modes such as a keyboard, a pointing device such as a mouse, voice input, gesture input, or input of an operation signal from another terminal.

[0123] In addition, the restriction process executed by the restriction processing unit 122 only needs to be a process for restricting the operation of the working machine 3, and is not limited to a process of prohibiting (making inoperable) the operation (such as a turning operation) of the working machine 3. The restriction process may be, for example, a process of decelerating the speed of the operation (such as a turning operation) of the working machine 3, a process of narrowing the operation range (such as a turning angle) of the working machine 3, or a process of restricting the allowable area of the operation of the working machine 3.

[0124] In addition, it is not essential for the deterrence process to include a plurality of specific processes (such as sound output processing and restriction processing) for deterring the operation of the working machine 3. Furthermore, even when the deterrence process includes a plurality of specific processes, it is not essential that the validity and invalidity can be switched individually for each specific process, and the validity and invalidity may be switchable collectively for the plurality of specific processes.

[0125] In addition, the function related to the deterrence process by the deterrence processing unit 12 is not essential, and the deterrence processing unit 12 can be omitted as appropriate.

[0126] In addition, the detection device 5 for detecting the detection target object Ob1 in the monitoring area A1 around the working machine 3 may include sensors such as a human sensor, a sonar sensor, a radar, or a LiDAR (Light Detection and Ranging) in addition to or instead of the sensor 51 and the imaging unit 52.

[0127] Further, the detection processing unit 15 may detect a detection target object Ob1 in the monitoring area A1 based on the outputs (image data) of, for example, the left camera 341, the right camera 342, and the rear camera 343. Specifically, the detection processing unit 15 performs image processing on the image data acquired by the image acquisition unit 14 to extract feature amounts in the image, and based on the feature amounts, determines whether or not the detection target object Ob1 (a "person" in the present embodiment) is captured in the image. Here, when the detection target object Ob1 is captured in the image, the detection processing unit 15 determines in which image captured by any of the left camera 341, the right camera 342, and the rear camera 343 the detection target object Ob1 is captured. That is, the detection processing unit 15 distinguishes in which of the small area A11 captured by the left camera 341, the small area A12 captured by the right camera 342, and the small area A13 captured by the rear camera 343 the detection target object Ob1 exists, and performs detection of the detection target object Ob1.

[0128] Further, the captured image Im100 displayed on the display screen Dp1 is not limited to an overhead image generated by coordinate transformation and synthesis of the captured images of the left camera 341, the right camera 342, and the rear camera 343. The captured image displayed on the display screen Dp1 may be, for example, an image as it is captured by at least one of the left camera 341, the right camera 342, and the rear camera 343 (that is, an un-synthesized image).

[0129] Further, the camera that captures the captured image Im100 is not limited to the left camera 341, the right camera 342, and the rear camera 343, and may include one, two, or four or more cameras (image sensors). Further, for example, an omnidirectional camera (360-degree camera) that can capture the entire area as viewed from the working machine 3 may be used to capture the captured image.

[0130] Further, in addition to or instead of a "person", the detection target object Ob1 may include a moving body such as a vehicle (including other working machines), a structure such as a wall and a pillar, a plant, an animal, a step, a groove, or other obstacles.

[0131] Further, it is not essential to reflect the positional relationship of the monitoring area A1 with respect to the working machine 3 in the real space in the positional relationship of the captured image Im100 with respect to the aircraft object Im10 on the display screen Dp1. Further, it is also not essential to reflect the positional relationship of the monitoring area A1 with respect to the turning unit 32 in the real space in the positional relationship of the captured image Im100 with respect to the aircraft object Im10 on the display screen Dp1.

[0132] Also, it is not essential that the display mode of the valid object X1 changes according to the detection result of the detection device 5.

[0133] (Embodiment 2) As shown in FIG. 10, the working machine 3 according to the present embodiment is different from the working machine 3 according to the first embodiment in the display content of the second area R2 of the display screen Dp1. Hereinafter, the same components as those in the first embodiment will be denoted by common reference numerals and the description thereof will be omitted as appropriate. In FIG. 10, only the second area R2 of the display screen Dp1 is shown, and the illustration of the areas other than the second area R2 is omitted.

[0134] In the present embodiment, the valid object X1 moves in the radial direction of the virtual circle C1 (see FIG. 6) from the reference point P1 side toward the outer peripheral side of the virtual circle C1. Specifically, as illustrated in FIG. 10, the valid object X1 is an arc-shaped (annular) object centered on the reference point P1 which is the center of the virtual circle C1. That is, the valid object X1 moves from the state shown in the upper part of FIG. 10 to the state shown in the lower part of FIG. 10 so as to spread outward of the captured image Im100 like ripples centered on the reference point P1 (in the direction indicated by the dashed arrow in FIG. 10). Such a valid object X1 moves along the radial direction of the virtual circle C1 toward the outer peripheral side (that is, in the direction away from the reference point P1) with the reference point P1 as the center. Thereby, the valid object X1 is likely to enter the field of view of the user U1, and it becomes easier for the user U1 to grasp whether the detection processing unit 15 is valid.

[0135] Also, even in this embodiment, the valid object X1 is not displayed in the exclusion range from the first end E101 to the second end E102 of the captured image Im100. Therefore, the valid object X1 has a shape lacking the exclusion range from the first end E101 to the second end E102, similar to the captured image Im100.

[0136] Furthermore, when the valid object X1 moves to the outermost end of the movable range, it moves to the innermost end of the movable range. That is, the valid object X1 moves from the reference point P1 side to the outer periphery of the captured image Im100, which is the outermost end of the movable range, and then moves to the inner periphery of the captured image Im100, which is the innermost end of the movable range.

[0137] Specifically, in the radial direction of the virtual circle C1, the valid object X1 moves continuously on the captured image Im100. When it moves to the outermost end where the captured image Im100 is interrupted, it moves discontinuously from the outer periphery to the inner periphery of the captured image Im100. As a result, the valid object X1 repeatedly moves on the captured image Im100, making it easier for the user U1 to grasp whether the detection processing unit 15 is effective.

[0138] The configuration according to Embodiment 2 can be adopted in appropriate combination with various configurations (including modified examples) described in Embodiment 1.

[0139] 〔Supplementary Note of the Invention〕 Hereinafter, the outline of the invention extracted from the above-described embodiments will be appended. Note that each configuration and each processing function described in the following supplementary note can be arbitrarily selected and combined.

[0140] <Supplementary Note 1> Obtaining a captured image of a monitoring area around a working machine, Causing a display device to display a display screen including the captured image, When the detection processing unit that detects the object to be detected in the monitoring area is valid, an effective object that moves in at least one of the circumferential direction and the radial direction of a virtual circle centered on the reference point set for the captured image is overlaid and displayed on the captured image on the display screen. Control method for a working machine.

[0141] <Appendix 2> It further includes displaying an aircraft object at the reference point on the display screen. The control method for a working machine according to Appendix 1.

[0142] <Appendix 3> According to the detection result of the detection processing unit, change the display mode of the effective object. The control method for a working machine according to Appendix 1 or 2.

[0143] <Appendix 4> The effective object has transparency. The control method for a working machine according to any one of Appendices 1 to 3.

[0144] <Appendix 5> The captured image is an aerial view image. The control method for a working machine according to any one of Appendices 1 to 4.

[0145] <Appendix 6> The effective object has a length along the radius of the virtual circle and moves in the circumferential direction of the virtual circle. The control method for a working machine according to any one of Appendices 1 to 5.

[0146] <Appendix 7> The captured image has a shape missing an exclusion range from a first end to a second end, which is a part of the circumferential direction of the virtual circle. The effective object skips the exclusion range and moves from the first end to the second end. The control method for a working machine according to Appendix 6.

[0147] <Supplementary Note 8> The effective object moves in the radial direction of the virtual circle from the reference point side toward the outer peripheral side of the virtual circle. The control method for a working machine according to any one of Supplementary Notes 1 to 5.

[0148] <Supplementary Note 9> When the effective object moves to the outermost end of the movable range, it moves to the innermost end of the movable range. The control method for a working machine according to Supplementary Note 8.

[0149] <Supplementary Note 10> The control method for a working machine according to any one of Supplementary Notes 1 to 9, A control program for a working machine for causing one or more processors to execute.

Explanation of Signs

[0150] 1 Control system for a working machine 2 Display device 3 Working machine 5 Detection device 11 Display processing unit 14 Image acquisition unit 30 Machine body A1 Monitoring area C1 Virtual circle E101 First end E102 Second end Dp1 Display screen Im10 Machine body object Im100 Captured image Ob1 Detection target P1 Reference point X1 Effective object

Claims

1. Obtaining a captured image of a monitoring area around a work machine; Causing a display device to display a display screen including the captured image; When a detection processing unit for detecting a detection target object in the monitoring area is valid, displaying an effective object that moves in at least one of the circumferential direction and the radial direction of a virtual circle centered on a reference point set for the captured image, superimposed on the captured image in the display screen; and A control method for a work machine.

2. Further including displaying an aircraft object at the reference point in the display screen; The control method for a work machine according to Claim 1.

3. Changing a display mode of the effective object according to a detection result of the detection processing unit; The control method for a work machine according to Claim 1 or 2.

4. The effective object has transparency; The control method for a work machine according to Claim 1 or 2.

5. The captured image is an overhead image; The control method for a work machine according to Claim 1 or 2.

6. The effective object has a length along the radius of the virtual circle and moves in the circumferential direction of the virtual circle; The control method for a work machine according to Claim 1 or 2.

7. The captured image has a shape lacking an exclusion range from a first end to a second end, which is a part of the circumferential direction of the virtual circle; The effective object skips the exclusion range and moves from the first end to the second end; The control method for a work machine according to Claim 6.

8. The effective object moves in the radial direction of the virtual circle from the reference point side toward the outer peripheral side of the virtual circle; The control method for a work machine according to Claim 1 or 2.

9. When the effective object moves to the outermost end of the movable range, it moves to the innermost end of the movable range; The control method for a work machine according to Claim 8.

10. A control program for a work machine for causing one or more processors to execute the control method for a work machine according to Claim 1 or 2.

11. An image acquisition unit that acquires a captured image of a monitoring area around a work machine; and A display processing unit that causes a display device to display a display screen including the captured image. ​ When the detection processing unit that detects the object to be detected in the monitoring area is valid, the display processing unit overlays and displays an effective object that moves in at least one of the circumferential direction and the radial direction of a virtual circle centered on the reference point set for the captured image on the captured image on the display screen. Control system for a working machine.

12. The control system for a working machine according to claim 11, and a machine body on which the display device is mounted. Working machine.

Images