Work machine control method, work machine control program, work machine control system, and work machine

Abstract

To provide a work machine control method, a work machine control program, a work machine control system, and a work machine, in which a visual recognition property of a display screen is hardly degraded.SOLUTION: A work machine control method includes: detecting a detection target Ob1 in a monitoring area around the work machine; displaying a display screen Dp1 on a display device; and displaying a detection object X10, whose display mode is changed according to a detection result of the detection target Ob1, on the display screen Dp1. The detection object X10 is arranged along an outer peripheral edge E10 of the display screen Dp1.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a work machine control method, a work machine control program, a work machine control system, and a work machine, which are used in a work machine having a function of detecting an object to be detected in a surrounding monitoring area. [Background technology]

[0002] As a related art, a work machine (work vehicle) capable of detecting surrounding objects is known (see, for example, Patent Document 1). The work machine according to the related art displays an image of the surroundings of the work machine on a display unit. When a detection device (detection device) detects an object present around the work machine, a bordered image in which a border has been applied to the surrounding image is displayed on the display unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-175822 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the related art described above, the area on the display screen in which information can actually be displayed becomes smaller due to the border image being displayed, which may reduce the visibility of status information relating to the status of the work machine, for example.

[0005] An object of the present invention is to provide a work machine control method, a work machine control program, a work machine control system, and a work machine in which visibility of a display screen is less likely to decrease. [Means for solving the problem]

[0006] A method for controlling a work machine according to one aspect of the present invention includes detecting a detection object in a monitoring area around the work machine, displaying a display screen on a display device, and displaying a detection object on the display screen, the display mode of which changes depending on a detection result of the detection object. The detection object is arranged along an outer periphery of the display screen.

[0007] A work machine control program according to one aspect of the present invention is a program for causing one or more processors to execute the work machine control method.

[0008] A work machine control system according to one aspect of the present invention includes a detection processing unit and a display processing unit. The detection processing unit detects a detection object in a monitoring area around the work machine. The display processing unit displays a display screen on a display device. The display processing unit displays a detection object on the display screen, the display mode of which changes depending on the detection result of the detection object. The detection object is arranged along the outer periphery of the display screen.

[0009] A work machine according to one aspect of the present invention includes the work machine control system and a body on which the display device is mounted. Effect of the Invention

[0010] According to the present invention, it is possible to provide a work machine control method, a work machine control program, a work machine control system, and a work machine in which visibility of the display screen is less likely to decrease. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic perspective view showing the overall configuration of a work machine according to a first embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing a hydraulic circuit and the like of the work machine according to the first embodiment. [Diagram 3] FIG. 3 is a schematic plan view showing a monitoring area set around the work machine according to the first embodiment, as viewed from above. [Figure 4] FIG. 4 is a schematic external view of a display device on which a display screen is displayed by the work machine control system according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of a display screen displayed by the work machine control system according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a display screen displayed by the work machine control system according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an example of a display screen displayed by the work machine control system according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a display screen displayed by the work machine control system according to the first embodiment. [Figure 9] FIG. 9 is a flowchart showing an operation example of the work machine control system according to the first embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a display screen displayed by the work machine control system according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a display screen displayed by the work machine control system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. The following embodiment is an example of the present invention, and is not intended to limit the technical scope of the present invention.

[0013] (Embodiment 1) [1] Overall structure As shown in Fig. 1, the work machine 3 according to this embodiment is equipped with a traveling section 31, a rotating section 32, and a working section 33 on a body 30. The work machine 3 is further equipped with a work machine control system 1 (hereinafter also simply referred to as the "control system 1") as shown in Fig. 2. In addition, the body 30 is further equipped with a display device 2, an operating device, etc.

[0014] The term "working machine" in this disclosure means a machine for various works, and examples thereof include working vehicles such as backhoes (including hydraulic excavators, mini excavators, etc.), wheel loaders, and carriers. The working machine 3 is equipped with a working unit 33 configured to be able to perform at least one work including lifting work. The working machine 3 is not limited to a "vehicle" and may be, for example, a work vessel, a work flying object such as a drone or a multicopter, etc. Furthermore, the working machine 3 is not limited to a construction machine (construction machinery), and may be, for example, an agricultural machine (farm machinery) such as a rice transplanter, a tractor, or a combine harvester. In this embodiment, unless otherwise specified, an example will be described in which the working machine 3 is a backhoe with a lifting function (with a crane function) and can perform excavation work, land leveling work, trench digging work, loading work, etc. as work in addition to lifting work.

[0015] Also, in this embodiment, for convenience of explanation, the vertical direction when the work machine 3 is in a usable state is defined as the up-down direction D1. Furthermore, when the swivel unit 32 is in a non-swivel state, the front-rear direction D2 and the left-right direction D3 are defined based on the direction as seen by the user (operator) riding on the work machine 3 (the driving unit 321) of the work machine 3. In other words, each direction used in this embodiment is a direction defined based on the body 30 of the work machine 3, and the direction in which the body 30 moves when the work machine 3 moves forward is "forward", and the direction in which the body 30 moves when the work machine 3 moves backward is "rear". Similarly, the direction in which the front end of the body 30 moves when the work machine 3 turns right is "rightward", and the direction in which the front end of the body 30 moves when the work machine 3 turns left is "leftward". However, these directions are not intended to limit the use direction of the work machine 3 (direction during use).

[0016] The work machine 3 is equipped with an engine that serves as a power source. In the work machine 3, for example, a hydraulic pump 41 (see FIG. 2) is driven by the engine, and hydraulic oil is supplied from the hydraulic pump 41 to hydraulic actuators (including a hydraulic motor 43, a hydraulic cylinder 44, etc.) of each part of the machine body 30, thereby driving the machine body 30. In addition, the work machine 3 is controlled, for example, by a user (operator) who sits on the driving section 321 of the machine body 30 operating an operating lever or the like of an operating device.

[0017] In this embodiment, since it is assumed that the work machine 3 is a riding type backhoe as described above, the working unit 33 is driven according to the operation of a user (operator) riding in the driving unit 321 to perform work such as excavation work. The driving unit 321 on which the user rides is provided on the rotating unit 32.

[0018] The traveling unit 31 has a traveling function and is configured to be able to travel (including turn) on the ground. The traveling unit 31 has, for example, a pair of left and right crawlers 311 and a blade 312. The traveling unit 31 further has a hydraulic motor 43 (hydraulic actuator) for driving the crawlers 311 and the like.

[0019] The swivel unit 32 is located above the travel unit 31, and is configured to be rotatable around a rotation axis along the vertical direction with respect to the travel unit 31. The swivel unit 32 has a hydraulic motor (hydraulic actuator) for rotation, etc. In addition to a driving unit 321, an engine, a hydraulic pump 41, etc. are mounted on the swivel unit 32. Furthermore, a boom bracket 322 to which the working unit 33 is attached is provided at the front end of the swivel unit 32.

[0020] The working unit 33 is configured to be able to perform work including lifting work. The working unit 33 is supported by a boom bracket 322 of the swivel unit 32 and performs work. The working unit 33 has a bucket 331, a boom 332, an arm 333, and the like. The working unit 33 further has a hydraulic actuator (including a hydraulic cylinder 44, a hydraulic motor, and the like) for driving each part.

[0021] The bucket 331 is a type of attachment (working tool) that is attached to the body 30 of the work machine 3, and is any tool selected from multiple types of attachments depending on the content of the work. As an example, the bucket 331 is removably attached to the body 30, and is replaced depending on the content of the work. In addition to the bucket 331, attachments for the work machine 3 include various tools such as a breaker, an auger, a crusher, a fork, a fork claw, a steel frame cutter, an asphalt cutter, a grass cutter, a ripper, a mulcher, a tilt rotator, and a tamper. The working unit 33 performs work by driving the bucket 331 with power from a drive device.

[0022] The boom 332 is rotatably supported by the boom bracket 322 of the swivel section 32. Specifically, the boom 332 is supported by the boom bracket 322 so as to be rotatable about a rotation axis along the horizontal direction. The boom 332 has a shape that extends upward from a base end supported by the boom bracket 322. The arm 333 is connected to the tip of the boom 332. The arm 333 is supported relative to the boom 332 so as to be rotatable about a rotation axis along the horizontal direction. A bucket 331 is attached to the tip of the arm 333.

[0023] The working unit 33 operates by receiving power from an engine as a power source. Specifically, a hydraulic pump 41 is driven by the engine, and hydraulic oil is supplied from the hydraulic pump 41 to hydraulic actuators (hydraulic cylinder 44, etc.) of the working unit 33, thereby operating each part of the working unit 33 (bucket 331, boom 332, and arm 333).

[0024] Particularly in this embodiment, the working unit 33 has a multi-joint structure in which the boom 332 and the arm 333 are configured to be rotatable individually. In other words, by each of the boom 332 and the arm 333 rotating about a rotation axis along the horizontal direction, the multi-joint working unit 33 including the boom 332 and the arm 333 can, for example, be extended or folded as a whole.

[0025] Similarly to the working unit 33, each of the traveling unit 31 and the swivel unit 32 receives power from an engine as a power source and operates. That is, the swivel unit 32 and the traveling unit 31 operate when hydraulic oil is supplied from the hydraulic pump 41 to the hydraulic motor 43 of the traveling unit 31 and the hydraulic motor of the swivel unit 32, etc.

[0026] As described above, the engine functions as a power source that supplies power to each part. Here, the engine is mounted on the slewing part 32 together with the hydraulic pump 41 and the like. In this embodiment, as an example, the engine is a diesel engine. The engine is driven by fuel (here, diesel) supplied from a fuel tank.

[0027] Here, the machine body 30 is equipped with various sensors (including cameras) for detecting a detection object Ob1 (see FIG. 3) in a monitoring area A1 (see FIG. 3) around the work machine 3, such as a camera that images the periphery of the machine body 30. As an example in this embodiment, as shown in FIG. 3, a plurality of cameras (imaging devices) (three in this case) including a left camera 341, a right camera 342, and a rear camera 343 are mounted on the rotating section 32 of the machine body 30. Furthermore, as shown in FIG. 3, a detection device 5 that detects a detection object Ob1 in the monitoring area A1 is mounted on the rotating section 32 of the machine body 30.

[0028] The left camera 341, the right camera 342, and the rear camera 343 are connected to the control system 1, and output images captured by each of them to the control system 1. Fig. 3 is a plan view of the work machine 3 seen from above, and diagrammatically shows a monitoring area A1 set around the work machine 3, a detection target Ob1, and the body 30 of the work machine 3 (including the left camera 341, the right camera 342, and the rear camera 343).

[0029] The left camera 341, the right camera 342, and the rear camera 343 are installed facing left, right, and rear with respect to the driving unit 321 so as to capture images of the monitoring area A1, which is to the left, right, and rear as seen by the operator riding in the driving unit 321 of the swivel unit 32. That is, the monitoring area A1 includes a plurality of (three in this case) small areas A11, A12, and A13 as shown in FIG. 3, and the left camera 341 captures the small area A11 (left area) which is to the left as seen by the operator riding in the driving unit 321. Similarly, the right camera 342 captures the small area A12 (right area) which is to the right as seen by the operator riding in the driving unit 321, and the rear camera 343 captures the small area A13 (rear area) which is to the rear as seen by the operator riding in the driving unit 321. This makes it possible to cover the sides (left and right) and rear, which are likely to be blind spots for the operator, with the left camera 341, the right camera 342, and the rear camera 343.

[0030] The detection device 5 is connected to the control system 1, and outputs a detection result of a detection object Ob1 in the monitoring area A1 to the control system 1. In this example, the detection device 5 is installed facing backward so as to detect the detection object Ob1 in the monitoring area A1 that can be photographed by the left camera 341, the right camera 342, and the rear camera 343. In other words, the detection range of the detection object Ob1 by the detection device 5 is the same as the photographable range (monitoring area A1) of the left camera 341, the right camera 342, and the rear camera 343.

[0031] In this embodiment, the detection device 5 has a sensor 51 (see FIG. 2) and an imaging unit 52 (see FIG. 2). The sensor 51 is a three-dimensional sensor that measures the distance to the detection object Ob1 by a TOF (Time Of Flight) method that measures the distance to the distance measurement point based on the round-trip time it takes for radio waves, light, sound, etc. to reach the distance measurement point and return. The sensor 51 is a distance measurement sensor that uses radio waves as a medium, such as a millimeter wave radar, to identify the distance to the detection object Ob1, the direction in which the detection object Ob1 exists, etc. The imaging unit 52 is a camera (including an image sensor and an optical element) that captures an image of the detection area (monitoring area A1) of the sensor 51. In this way, according to the detection device 5 that combines the sensor 51 and the imaging unit 52, when the detection object Ob1 exists in the monitoring area A1, it is possible to identify the three-dimensional position and attributes (shape, size, color, movement, etc.) of the detection object Ob1.

[0032] In other words, the detection results of the detection device 5 may include the presence or absence of the detection object Ob1 in the monitoring area A1, the position of the detection object Ob1 in the monitoring area A1 if the detection object Ob1 is present in the monitoring area A1, the attributes of the detection object Ob1, etc.

[0033] In short, the detection device 5 detects a detection object Ob1 in the monitoring area A1 around the work machine 3. The detection device 5 determines the presence or absence (presence) of the detection object Ob1 in the monitoring area A1, and outputs a detection result indicating whether or not the detection object Ob1 exists in the monitoring area A1. In this embodiment, as an example, the detection object Ob1 is a "person." In other words, when a "person" enters the monitoring area A1 around the work machine 3 as a result of the movement of the work machine 3 or the movement of a "person" around the work machine 3, the detection device 5 detects the "person" as the detection object Ob1. When multiple detection objects Ob1 exist in the monitoring area A1, the detection device 5 may also detect the number (number of people) of detection objects Ob1.

[0034] Fig. 2 shows a schematic diagram of the hydraulic circuit and the electric circuit (electrical connection relationship) of the work machine 3 according to this embodiment. In Fig. 2, solid lines indicate high-pressure oil passages (for hydraulic oil), dotted lines indicate low-pressure oil passages (for pilot oil), and dashed arrows indicate paths of electric signals.

[0035] As shown in FIG. 2, the work machine 3 is equipped with a hydraulic pump 41, a hydraulic motor 43 (not shown in FIG. 2), and a hydraulic cylinder 44, as well as a pilot pump 42, a remote control valve 45, a control valve 46, and a directional control valve (control valve) 47.

[0036] Hydraulic oil from a hydraulic pump 41 driven by the engine is supplied to a hydraulic motor 43 of the traveling unit 31, a hydraulic motor of the rotating unit 32, a hydraulic cylinder 44 of the working unit 33, etc. This drives hydraulic actuators such as the hydraulic motor 43 and the hydraulic cylinder 44.

[0037] Hydraulic actuators such as the hydraulic motor 43 and the hydraulic cylinder 44 are provided with a pilot-type directional control valve 47 capable of switching the direction and flow rate of hydraulic oil from the hydraulic pump 41. The directional control valve 47 is driven by the supply of pilot oil, which serves as an input command, from the pilot pump 42.

[0038] Here, for example, a remote control valve 45 is provided in a supply path of pilot oil to a directional control valve 47 corresponding to the hydraulic cylinder 44 of the working unit 33. The remote control valve 45 outputs a work operation command for the working unit 33 in response to the operation of an operating lever. The work operation command instructs the deployment operation, retraction operation, etc. of the working unit 33. In addition, an electromagnetic control valve 46 (solenoid valve) is inserted between the remote control valve 45 and the pilot pump 42. The control valve 46 is connected to a power source 351 via a cutoff relay 352 and a cutoff switch 353, and operates in response to the current supplied from the power source 351.

[0039] Similarly, a remote control valve is provided in a supply path of pilot oil to a directional control valve corresponding to the hydraulic motor 43 of the traveling unit 31. This remote control valve outputs a traveling operation command for the traveling unit 31 in response to the operation of an operating lever. The traveling operation command instructs the traveling operation (forward or backward, etc.) of the traveling unit 31. Furthermore, a remote control valve is provided in a supply path of pilot oil to a directional control valve corresponding to the hydraulic motor of the slewing unit 32. This remote control valve outputs a swing operation command for the slewing unit 32 in response to the operation of an operating lever. The swing operation command instructs the swing operation (left swing or right swing, etc.) of the slewing unit 32. An electromagnetic control valve 46 (solenoid valve) is also inserted between these remote control valves and the pilot pump 42. The control valve 46 is connected to a power source 351 via a cutoff relay 352 and a cutoff switch 353, and operates in response to the supply current from the power source 351.

[0040] The control valve 46 opens the flow path of pilot oil from the pilot pump 42 to the remote control valve 45 when it is energized, i.e., when current is being supplied, and blocks the flow path of pilot oil when it is de-energized, i.e., when the supply current to the control valve 46 is cut off. Therefore, when the supply current to the control valve 46 is cut off, the hydraulic actuator corresponding to the remote control valve 45 cannot be driven, and the output of the hydraulic actuator is forcibly stopped regardless of the operation of the operating lever.

[0041] Here, the cutoff relay 352 is connected to the control system 1 and is switched on / off in response to a control signal (electrical signal) from the control system 1. The cutoff switch 353 is switched on / off in response to the operation of the cutoff lever 354, and is turned on, for example, when the cutoff lever 354 is operated downward. Therefore, when both the cutoff relay 352 and the cutoff switch 353 are on, the control valve 46 is in an energized state and the flow path of the pilot oil from the pilot pump 42 to the remote control valve 45 is opened, so that the hydraulic actuator is driven in response to the operation of the operation lever. On the other hand, when at least one of the cutoff relay 352 and the cutoff switch 353 is in an off state, the control valve 46 is in a non-energized state and the flow path of the pilot oil is blocked, so that the hydraulic actuator cannot be driven.

[0042] For example, when at least one of the cutoff relay 352 and the cutoff switch 353 connected to the control valve 46 inserted between the remote control valve corresponding to the hydraulic motor of the swivel unit 32 and the pilot pump 42 is in an off state, the hydraulic motor of the swivel unit 32 cannot be driven. In this state, the output of the hydraulic actuator (hydraulic motor of the swivel unit 32) is forcibly stopped regardless of the operation of the operation lever, so that the swivel operation of the swivel unit 32 is prohibited.

[0043] The cutoff lever 354 is disposed in the driving section 321 of the machine body 30, and receives operation input from a user (operator). In this embodiment, as an example, the cutoff lever 354 can be operated in the up-down direction D1. When the cutoff lever 354 is in the "up position", which is the upper end position of the movable range, the cutoff switch 353 is "off", and when the cutoff lever 354 is in the "down position", which is the lower end position of the movable range, the cutoff switch 353 is "on".

[0044] Therefore, when the cutoff lever 354 is in the "down position", the hydraulic actuator (hydraulic motor of the swivel section 32) is driven by operating the operation lever. In contrast, when the cutoff lever 354 is in the "up position", the output of the hydraulic actuator is forcibly stopped regardless of the operation of the operation lever. Therefore, to drive the hydraulic actuator, the user (operator) needs to operate the cutoff lever 354 to the "down position".

[0045] In this embodiment, the state of the cutoff lever 354 when it is in the "up position", i.e., when the work machine 3 cannot be operated, is defined as the "locked state". On the other hand, the state of the cutoff lever 354 when it is in the "down position", i.e., when the work machine 3 can be operated, is defined as the "unlocked state".

[0046] In short, when the cutoff switch 353 is off, it is in a "locked state" in which the operation of the work machine 3 is restricted (including prohibited), and when it is on, it is in an "unlocked state" in which the operation of the work machine 3 is not restricted. And when the cutoff lever 354 is in the "raised position" and the cutoff switch 353 is in a locked state (off), the operation of the work machine 3 is forcibly restricted without the operation of the operating device 35. The cutoff lever 354 is a lever that is operated when locking the operation of the work machine 3 in this way, and is synonymous with a gate lock lever.

[0047] The control system 1 mainly comprises a computer system having one or more processors such as a CPU (Central Processing Unit) and one or more memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and executes various processes (information processing). In this embodiment, the control system 1 is an integrated controller that controls the entire work machine 3, and is composed of, for example, an electronic control unit (ECU). However, the control system 1 may be provided separately from the integrated controller. The control system 1 will be explained in detail in the section "[2] Configuration of the control system".

[0048] The display device 2 is disposed in the driving section 321 of the machine body 30, and is a user interface for receiving operation inputs by a user (operator) and outputting various information to the user. The display device 2 receives various operations by the user, for example, by outputting an electrical signal in response to the user's operation. This allows the user (operator) to visually check the display screen Dp1 (see FIG. 4) displayed on the display device 2, and also allows the user (operator) to operate the display device 2 as necessary.

[0049] 2, the display device 2 includes a control unit 21, an operation unit 22, and a display unit 23. The display device 2 is configured to be able to communicate with the control system 1, and is able to send and receive data between the display device 2 and the control system 1. In this embodiment, as an example, the display device 2 is a dedicated device used in the work machine 3.

[0050] The control unit 21 controls the display device 2 in accordance with data from the control system 1. Specifically, the control unit 21 outputs an electrical signal in response to a user's operation received by the operation unit 22, and displays a display screen Dp1 generated by the control system 1 on the display unit 23.

[0051] The operation unit 22 is a user interface for accepting operation input by a user (operator) to the display screen Dp1 displayed on the display unit 23. The operation unit 22 accepts various operations by the user U1 (see FIG. 4), for example, by outputting an electric signal according to the operation of the user U1. As an example in the present embodiment, the operation unit 22 includes a plurality of (six here) mechanical push button switches 221-226 as shown in FIG. 4. These plurality of push button switches 221-226 are arranged close to the display area (lower in the example of FIG. 4) so ​​as to follow the periphery of the display area of ​​the display unit 23. These plurality of push button switches 221-226 are associated with items displayed on the display screen Dp1 described later, and by operating any one of the plurality of push button switches 221-226, any one of the items on the display screen Dp1 is operated (selected).

[0052] Furthermore, the operation unit 22 may include a touch panel, an operation dial, etc. In this case as well, an operation on the operation unit 22 causes an operation (selection) of any item on the display screen Dp1.

[0053] The display unit 23 is a user interface for presenting information to the user U1 (operator), such as a liquid crystal display or an organic EL display that displays various information. The display unit 23 presents various information to the user by display. In the present embodiment, as an example, the display unit 23 is a full-color liquid crystal display with a backlight, and has a "landscape-long" display area that is long in the horizontal direction, as shown in FIG. 4.

[0054] The display device 2 presents various information on the display screen Dp1 to the user U1 (operator) who operates the work machine 3. In other words, the user U1 who operates the work machine 3 can visually obtain various information related to the work machine 3 by looking at the display screen Dp1 displayed on the display device 2. As an example, information on the operating state of the work machine 3, such as the cooling water temperature and the hydraulic oil temperature, is displayed on the display device 2, so that the user U1 can check the information on the operating state of the work machine 3 required for operating the work machine 3 on the display device 2. In addition, the display device 2 can also display, on the display screen Dp1, images of the surroundings of the work machine 3 (images of the monitoring area A1) captured by the left camera 341, the right camera 342, and the rear camera 343. This allows the user U1 (operator) to check, for example, the situation on the side and rear of the work machine 3, which are likely to be blind spots from the driving unit 321, on the display screen Dp1 displayed on the display device 2 when operating the work machine 3.

[0055] Furthermore, the work machine 3 is equipped with a sound output unit 36 ​​(see FIG. 2) that outputs sound (including voice) to the user U1 (operator). The sound output unit 36 ​​includes a buzzer, a speaker, or the like, and outputs sound upon receiving an electric signal. The sound output unit 36 ​​is connected to the control system 1, and outputs sound such as a beep or voice in response to a sound control signal from the control system 1. In this embodiment, the sound output unit 36 ​​is provided in the driving unit 321 of the machine body 30, similar to the display device 2. The sound output unit 36 ​​may be provided integrally with the display device 2.

[0056] In addition to the above-mentioned configuration, the machine 30 further includes an operating lever, a communication terminal, a fuel tank, a battery, etc. Furthermore, the machine 30 is equipped with sensors for monitoring the operating state of the machine 30, such as a coolant temperature sensor, a hydraulic oil temperature sensor, a tachometer for measuring the engine RPM, and an hour meter for measuring the operating time, etc. The machine 30 is also equipped with sensors for detecting the states of the cut-off lever 354 and the starter key switch, etc.

[0057] [2] Control system configuration Next, the configuration of the control system 1 according to this embodiment will be described with reference to Fig. 2. The control system 1 controls the display device 2 to cause the display device 2 to display the display screen Dp1. In this embodiment, the display device 2 is mounted on the body 30 of the work machine 3 as described above. The control system 1 is a component of the work machine 3, and constitutes the work machine 3 together with the body 30 and the like. In other words, the work machine 3 according to this embodiment comprises at least the control system 1 and the body 30 (including a traveling section 31, a rotating section 32 and a working section 33) on which the display device 2 is mounted.

[0058] In the present disclosure, a "screen" such as the display screen Dp1 means a video (image) displayed on the display device 2, and includes icons, figures, photographs, text, video, and the like. That is, the control system 1 is capable of causing the display device 2 to display the display screen Dp1 including icons and the like showing information relating to the operating state of the work machine 3, such as the cooling water temperature and hydraulic oil temperature. Here, when the display screen Dp1 includes a video, and the like, the display screen Dp1 does not include a fixed image, but includes an image that changes from moment to moment.

[0059] As shown in Fig. 2, the control system 1 includes a display processing unit 11, a check processing unit 12, a switching processing unit 13, an image acquisition unit 14, and a detection processing unit 15. In this embodiment, as an example, the control system 1 is mainly configured as a computer system having one or more processors, and these multiple functional units (display processing unit 11, etc.) are realized by the one or more processors executing a work machine control program. These multiple functional units included in the control system 1 may be provided in a distributed manner across multiple housings, or may be provided in a single housing.

[0060] The control system 1 is configured to be able to communicate with devices provided in each part of the machine body 30. That is, at least the display device 2, the detection device 5, the sound output unit 36, the cutoff relay 352, the left camera 341, the right camera 342, and the rear camera 343 are connected to the control system 1. As a result, the control system 1 can control the display device 2 and the sound output unit 36, etc., control the cutoff relay 352 to control the control valve 46, and obtain the detection result of the detection device 5 and the captured images of the left camera 341, the right camera 342, and the rear camera 343, etc. Here, the control system 1 may directly exchange various information (data) with each device, or indirectly via a repeater or the like.

[0061] The image acquisition unit 14 executes image acquisition processing to acquire captured images of the monitoring area A1 around the work machine 3. In this embodiment, the image acquisition unit 14 acquires the outputs of the left camera 341, the right camera 342 and the rear camera 343 periodically or irregularly from the left camera 341, the right camera 342 and the rear camera 343. In other words, the image acquisition unit 14 acquires image data (captured images) of the monitoring area A1 (each of the small areas A11, A12, A13) around the work machine 3. The data acquired by the image acquisition unit 14 is stored, for example, in a memory or the like.

[0062] The detection processing unit 15 executes a detection process to detect a detection object Ob1 in the monitoring area A1. In this embodiment, since the detection device 5 detects the detection object Ob1 in the monitoring area A1, the detection processing unit 15 detects the detection object Ob1 in the monitoring area A1 by acquiring the detection result of the detection device 5 from the detection device 5.

[0063] The restraint processing unit 12 executes restraint processing to restrain the operation of the work machine 3 based on the detection result of the detection device 5. In this embodiment, when the detection result of the detection device 5 indicates the presence of a detection target Ob1 (here, a person) in the monitoring area A1, the restraint processing unit 12 executes restraint processing. In this disclosure, "restraint processing" refers to processing that acts in some way to suppress the operation of the work machine 3. As an example, the restraint processing includes processing to indirectly restrain the operation of the work machine 3 by warning the user U1 (operator) who operates the work machine 3 by sound or light (including display). Furthermore, the restraint processing includes processing to directly restrain the operation of the work machine 3 by controlling the traveling unit 31, the rotating unit 32, the working unit 33, etc. of the work machine 3.

[0064] In this embodiment, the restraint processing unit 12 includes a sound output processing unit 121 and a restriction processing unit 122.

[0065] When a detection object Ob1 is present in the monitoring area A1, the sound output processing unit 121 controls the sound output unit 36 ​​to output an alarm sound from the sound output unit 36. That is, in this embodiment, the restraint processing includes a sound output processing for outputting an alarm sound. The alarm sound may be a simple beep sound or a voice such as "please be careful." Furthermore, the alarm sound may change according to the detection result of the detection device 5 (such as the distance from the machine body 30 to the detection object Ob1). As a result, the operation of the work machine 3 can be indirectly restrained by warning the user U1 (operator) who operates the work machine 3 with the alarm sound, so that the degree of freedom in operation of the work machine 3 is high. That is, by the user U1 operating the work machine 3 while paying attention to the detection object Ob1, the operation of the work machine 3 can be continued while avoiding contact with the detection object Ob1.

[0066] When a detection object Ob1 is present in the monitoring area A1, the restriction processing unit 122 controls the cutoff relay 352 to turn off the cutoff relay 352. As a result, the control valve 46 connected to the power source 351 via the cutoff relay 352 is de-energized, and the output of the hydraulic actuator corresponding to the control valve 46 is forcibly stopped. That is, in this embodiment, the restraint processing includes a restriction processing that restricts the operation of the work machine 3. The "restriction processing" in this disclosure means a processing that acts in some way to restrict the operation of the work machine 3. As an example, the restriction processing includes a processing that prohibits the traveling operation of the traveling unit 31 (making the traveling operation impossible), a processing that prohibits the turning operation of the turning unit 32 (making the turning operation impossible), and a processing that prohibits the operation of the working unit 33 (making the work impossible). This makes it possible to forcibly restrict the operation of the work machine 3 without the operation of the user U1 (operator). That is, it is possible to avoid contact between the machine body 30 and the detection object Ob1 caused by the operation of the work machine 3.

[0067] Here, the restriction process executed by the restriction processing unit 122 includes at least a process of restricting the rotational movement of the rotating unit 32. Specifically, the restriction processing unit 122 is configured to be able to control a cutoff relay 352 connected to a control valve 46 corresponding to the hydraulic motor of the rotating unit 32, and turns off the cutoff relay 352 when a detection object Ob1 is present in the monitoring area A1. As a result, when a detection object Ob1 is present in the monitoring area A1, the hydraulic motor of the rotating unit 32 becomes inoperable, the rotating unit 32 is brought to an emergency stop if the rotating unit 32 is in the middle of a rotational movement, and the rotating movement of the rotating unit 32 is prohibited if the rotating unit 32 is not in the middle of a rotational movement. That is, in this embodiment, the work machine 3 includes a traveling unit 31 and a rotating unit 32 that is capable of rotating with respect to the traveling unit 31. The restriction process restricts at least the rotational movement of the rotating unit 32. As a result, when a detection object Ob1 is present in a monitoring area A1 that is a blind spot for a user U1 (operator), contact between the machine body 30 and the detection object Ob1 caused by the rotation of the swivel unit 32 can be avoided.

[0068] The switching processing unit 13 switches between enabling and disabling the detection processing unit 15. In other words, the switching processing unit 13 switches between enabling and disabling a function related to the detection process of the detection object Ob1 in the monitoring area A1 by the detection processing unit 15. In short, the detection processing unit 15 is not always enabled, but can be switched between enabled and disabled. If the detection processing unit 15 is enabled, when the detection object Ob1 exists in the monitoring area A1, the detection object Ob1 is detected by the detection processing unit 15, and the check processing unit 12 executes the check processing. On the other hand, if the detection processing unit 15 is disabled, even if the detection object Ob1 exists in the monitoring area A1, the detection object Ob1 is not detected by the detection processing unit 15, and the check processing unit 12 does not execute the check processing.

[0069] In this embodiment, the switching processing unit 13 switches between enabled and disabled states of the detection processing unit 15 depending on whether or not to activate the detection device 5. That is, the switching processing unit 13 activates the detection device 5 when the detection processing unit 15 is enabled, and does not activate the detection device 5 when the detection processing unit 15 is disabled. This makes it possible to reduce power consumption by the detection device 5 when the detection processing unit 15 is disabled.

[0070] However, this configuration is not limiting, and the switching processing unit 13 may disable the detection processing unit 15, for example, by disabling the detection processing of the detection processing unit 15 (acquisition of the detection result from the detection device 5) or by disabling the acquired detection result. In this case, when the detection processing unit 15 is disabled, the detection device 5 itself will operate, but the check processing unit 12 will not execute the check processing even if a detection target Ob1 is present in the monitoring area A1. In short, the switching processing unit 13 may disable the detection processing unit 15 by disabling the processing that uses the detection result of the detection device 5.

[0071] In this embodiment, as an example, the detection processing unit 15 is switched between enabled and disabled by a user U1 (operator) operating the display device 2. That is, when the user U1 operates the operation unit 22 of the display device 2 to enable the detection processing unit 15, the switching processing unit 13 receives this operation and enables the detection processing unit 15. On the other hand, when the user U1 operates the operation unit 22 of the display device 2 to disable the detection processing unit 15, the switching processing unit 13 receives this operation and disables the detection processing unit 15.

[0072] In addition, in this embodiment, the checking process performed by the checking process unit 12 includes a sound output process executed by the sound output process unit 121 and a restriction process executed by the restriction process unit 122. In this way, the checking process includes a plurality of specific processes (sound output process, restriction process, etc.) for checking the operation of the work machine 3. Here, the checking process may be able to be switched between enabled and disabled for each specific process individually. In other words, the switching process unit 13 may be able to switch between enabled and disabled for the sound output process unit 121 and the restriction process unit 122 in the checking process unit 12 individually. As an example, it is possible to enable the sound output process unit 121 and disable the restriction process unit 122, or to disable the sound output process unit 121 and enable the restriction process unit 122. This makes it possible to enable only the specific processes that are required depending on the situation, improving the degree of freedom of the checking process.

[0073] The display processing unit 11 executes display processing for displaying at least the display screen Dp1 on the display device 2. Specifically, the display processing unit 11 generates the display screen Dp1 based on data acquired by the image acquisition unit 14 and controls the display device 2 to display the display screen Dp1 on the display unit 23 of the display device 2. Furthermore, the display processing unit 11 operates in response to an operation received by the operation unit 22 of the display device 2. The display processing unit 11 displays, for example, captured images Im100 (see FIG. 5) captured by the left camera 341, the right camera 342, and the rear camera 343 on the display screen Dp1. That is, the display processing unit 11 causes the image acquisition unit 14 to display an image of the monitoring area A1 around the work machine 3 on the display device 2.

[0074] Here, the display processing unit 11 displays on the display screen Dp1 a detection object X10 (see FIG. 5 ) whose display mode changes depending on the detection result of the detection target Ob1 by the detection processing unit 15. In the present disclosure, an "object" such as the detection object X10 includes a mark, an image, a figure, a photograph, a text, a video, or the like, or a combination of these, displayed on the display screen Dp1.

[0075] This allows the user U1 (operator) to grasp the detection result (presence or absence, etc.) of the detection object Ob1 in the monitoring area A1 from the detection object X10 displayed on the display screen Dp1. In other words, the user U1 can visually confirm the detection result of the detection object Ob1 and can operate the work machine 3 after knowing the presence or absence, etc., of the detection object Ob1 in the monitoring area A1.

[0076] Incidentally, in this embodiment, the detection processing unit 15 acquires the detection result of the detection device 5 outside the control system 1 and executes the detection process of the detection target Ob1 based on the detection result, but is not limited to this configuration. For example, the detection processing unit 15 may execute the detection process of the detection target Ob1 in the monitoring area A1 based on the output of the sensor 51 and / or the imaging unit 52 outside the control system 1.

[0077] [3] Control method for work machine An example of a control method for the work machine 3 executed mainly by the control system 1 (hereinafter simply referred to as a "control method") will be described below with reference to Figs. 5 to 9.

[0078] The control method according to this embodiment is executed by a control system 1 having a computer system as its main component, and in other words is embodied in a work machine control program (hereinafter simply referred to as a "control program"). In other words, the control program according to this embodiment is a computer program for causing one or more processors to execute each process related to the control method. Such a control program may be executed, for example, by the control system 1 and the display device 2 in cooperation with each other.

[0079] Here, the control system 1 executes the various processes described below related to the control method when a specific, preset start operation is performed to execute the control program. The start operation is, for example, an operation to start the engine of the work machine 3. On the other hand, the control system 1 ends the various processes described below related to the control method when a specific, preset end operation is performed. The end operation is, for example, an operation to stop the engine of the work machine 3.

[0080] [3.1] Display screen Here, first, a description will be given of the configuration of the display screen Dp1 displayed on the display unit 23 of the display device 2 by the control method according to this embodiment. In the drawings showing the display screen Dp1 displayed on the display unit 23 of the display device 2, such as Fig. 5, the dashed dotted lines, leader lines and reference symbols representing areas are all added merely for the purpose of explanation and are not actually displayed on the display device 2.

[0081] 5 is a home screen that is first displayed by the control method. The home screen is a basic display screen Dp1 that is first displayed on the display device 2 while the work machine 3 is in operation. The display screen Dp1 can transition from the home screen to various display screens Dp1 including a menu screen, a crane screen, a mode screen, a PTO screen, and the like, in accordance with an operation on the operation unit 22.

[0082] 5, the display screen Dp1 includes a first region R1, a second region R2, a third region R3, a fourth region R4, a fifth region R5, a sixth region R6, a seventh region R7, an eighth region R8, a ninth region R9, and a tenth region R10. In the control method according to the present embodiment, as an example, a captured image Im100 of the monitoring area A1, etc. are displayed in the second region R2 that occupies most of the display screen Dp1.

[0083] Specifically, the display screen Dp1 is divided into four regions in the vertical direction (up and down direction). The top three regions are further divided into three regions in the horizontal direction (left and right direction). As a result, the display screen Dp1 is divided into a total of ten regions. The regions in the second row from the top are, from the left, the first region R1, the second region R2, and the third region R3. The region in the bottom row is the fourth region R4. The regions in the third row from the top are, from the left, the fifth region R5, the sixth region R6, and the seventh region R7, and the regions in the top row are, from the left, the eighth region R8, the ninth region R9, and the tenth region R10. In terms of vertical size, the regions in the second row from the top (the first region R1, the second region R2, and the third region R3) are the largest among the four regions divided vertically. In terms of horizontal size, the middle region (second region R2, sixth region R6, ninth region R9) is the largest among the three regions divided horizontally.

[0084] However, the arrangement and size of each of these regions are merely examples and can be changed as appropriate. Also, it is not essential that each region is clearly divided by a boundary line. For example, in the example of FIG. 5, the second region R2 and the third region R3 are clearly divided by a boundary line, whereas there is no boundary line between the first region R1 and the second region R2. Of course, the first region R1 and the second region R2 may be clearly divided by a boundary line.

[0085] The first area R1 is a vertically long rectangular area. In the first area R1, for example, remaining amount information G1 regarding the remaining amount of fuel (e.g., diesel) for the engine is displayed. The display processing unit 11 generates the remaining amount information G1 in the display screen Dp1 based on the output (sensor signal) of the remaining amount sensor, etc.

[0086] The second area R2 is a rectangular area that is elongated in the horizontal direction. A captured image Im100 of the monitoring area A1 is displayed in the second area R2. A machine body object Im10, a captured image Im200 by the rear camera 343, and the like are further displayed in the second area R2.

[0087] The captured image Im100 is an overhead image generated by converting the coordinates of the captured images of the left camera 341, the right camera 342, and the rear camera 343 and synthesizing them. That is, the captured image Im100 is an image in which the image of the small area A11 to the left of the driving unit 321 captured by the left camera 341, the image of the small area A12 to the right of the driving unit 321 captured by the right camera 342, and the image of the small area A13 to the rear of the driving unit 321 captured by the rear camera 343 are synthesized, and the monitored area A1 is displayed looking down from above the aircraft 30. The display processing unit 11 synthesizes the captured images acquired by the image acquisition unit 14 and displays them in real time.

[0088] The machine object Im10 is displayed in an eleventh area R11 set in the center of the second area R2. In the present embodiment, as an example, the machine object Im10 is an icon that represents the machine 30 seen from above. The captured image Im200 is an image of a small area A13 behind the driver's unit 321 captured by the rear camera 343.

[0089] The third area R3 is a vertically long rectangular area. In the third area R3, an icon Im1 corresponding to the operating state of each part of the work machine 3 is displayed. In the third area R3, a plurality of icons Im1 can be displayed, and the design (pattern) of each icon Im1 indicates which state it represents, for example, the battery, seat belt, cooling water temperature, hydraulic oil temperature, etc. Here, each icon Im1 indicates the operating state by, for example, a display mode such as a display color or size. The display processing unit 11 judges the state of each part of the work machine 3 using the output of various sensors (including a cooling water temperature sensor and a hydraulic oil temperature sensor) that detect the operating state of each part of the work machine 3. Then, when an abnormal value is detected in any part, the display processing unit 11 displays a warning by changing the display mode, such as the display color, of the icon Im1 of that part.

[0090] The fourth region R4 is a strip-shaped region extending across the entire width of the display screen Dp1. In the fourth region R4, various items for operating the display screen Dp1 are displayed. In FIG. 5, as an example, the fourth region R4 has six items, "menu", "crane", "mode", "camera", "PTO", and "switch", arranged in this order from the left. These six items are associated with the six push button switches 221 to 226 of the operation unit 22 located directly below them. For example, the "menu" item is associated with the push button switch 221, and the "crane" item is associated with the push button switch 222. Therefore, for example, when the push button switch 224 corresponding to the "camera" item is operated by the user U1 (see FIG. 4), the "camera" item is operated (selected).

[0091] Furthermore, in this embodiment, in order to correspond to the operation of the operation dial (or cursor key) of the operation unit 22, any of the items is highlighted in the fourth region R4. In the example of FIG. 5, the item "Menu" is highlighted, and the highlighted item is switched by the operation of the operation dial (or cursor key). The user U1 can select the desired item by operating the decision button with the desired item highlighted. Therefore, for example, when the decision button is operated with the highlight moved to the "Camera" item, the "Camera" item is operated (selected). In addition, when the operation unit 22 includes a touch panel, the user U1 can select the desired item by touching the desired item on the display screen Dp1.

[0092] The fifth area R5 displays warning images (icons) indicating that abnormal values ​​have been detected by various sensors (including the cooling water temperature sensor and the hydraulic oil temperature sensor). The sixth area R6 displays, for example, information related to the working unit 33 currently operating in the work machine 3. The seventh area R7 displays, for example, information related to the operating state of the work machine 3, such as the engine RPM. The eighth area R8 displays, for example, the current time. The ninth area R9 displays, for example, information indicating the item to which the currently displayed display screen Dp1 belongs. The tenth area R10 displays, for example, information related to the operating time (hour meter) of the work machine 3.

[0093] Furthermore, on the display screen Dp1, a detection object X10 is displayed, the display mode of which changes according to the detection result of the detection target object Ob1 by the detection processing unit 15. Here, the detection object X10 is arranged along the outer periphery E10 of the display screen Dp1. The detection object X10 will be described in detail in the section "[3.2] Details."

[0094] [3.2] Details Next, the control method according to this embodiment will be described in detail.

[0095] The control method according to this embodiment includes detecting a detection object Ob1 in a monitoring area A1 around a work machine 3 (detection processing), displaying a display screen Dp1 on a display device 2 (display processing), and displaying a detection object X10 on the display screen Dp1, the display mode of which changes depending on the detection result of the detection object Ob1. The detection object X10 is arranged along an outer periphery E10 of the display screen Dp1.

[0096] In the present disclosure, "disposed along the outer circumferential edge E10" means that at least one of the position and shape of the detection object X10 is disposed along the outer circumferential edge E10. Specifically, regarding the position of the detection object X10, the detection object X10 is disposed along the outer circumferential edge E10 when the detection object X10 faces the outer circumferential edge E10 or when there is no other object between the detection object X10 and the outer circumferential edge E10. Furthermore, regarding the shape of the outer circumferential edge E10, the detection object X10 is disposed along the outer circumferential edge E10 when the outer circumferential edge E10 has a length in the longitudinal direction of the outer circumferential edge E10.

[0097] In short, by displaying the display screen Dp1 including the detection object X10 on the display device 2, the user U1 can check the presence or absence of the detection object Ob1 in the monitoring area A1 from the detection object X10 on the display screen Dp1. Here, the detection object X10 is arranged along the outer periphery E10 of the display screen Dp1. Therefore, while the detection object X10 is displayed in a state where it is unlikely to interfere with other information included in the display screen Dp1, the user U1 can grasp the detection result of the detection object Ob1 from the detection object X10 displayed at a position that is naturally in the field of view when looking at the display screen Dp1. Therefore, according to the control method of this embodiment, the visibility of the display screen Dp1 is unlikely to decrease.

[0098] Hereinafter, the display contents around the detection object X10 on the display screen Dp1 will be described in more detail with reference to FIGS.

[0099] 5, the detection object X10 includes a first object X11 and a second object X12. The first object X11 and the second object X12 are arranged symmetrically in at least one direction. In the present embodiment, as an example, the first object X11 and the second object X12 are arranged symmetrically in the left-right direction (the horizontal direction of the display screen Dp1) with respect to the center of the display screen Dp1. Here, the first object X11 and the second object X12 are both substantially linear objects having a length in the up-down direction (the vertical direction of the display screen Dp1), and are arranged in the center of the vertical direction of the display screen Dp1.

[0100] More specifically, the first object X11 and the second object X12 are formed in a substantially trapezoidal shape with the inner side in the horizontal direction of the display screen Dp1 as the short side and the outer side as the long side. That is, the first object X11 and the second object X12 are configured to be symmetrical in the left-right direction (the horizontal direction of the display screen Dp1) in terms of both position and shape. In this way, the first object X11 and the second object X12 that are symmetrical in one direction (the left-right direction) make it easy to improve the visibility of the detection object X10.

[0101] Here, the outer periphery E10 of the display screen Dp1 includes a first vertical side E1 and a second vertical side E2 that face each other in the left-right direction (horizontal direction) of the display screen Dp1. The first object X11 is disposed along the first vertical side E1, and the second object X12 is disposed along the second vertical side E2.

[0102] That is, the display screen Dp1, which is rectangular as a whole, includes a first vertical side E1 and a second vertical side E2 that face each other in the left-right direction (horizontal direction) and a first horizontal side E3 and a second horizontal side E4 that face each other in the up-down direction (vertical direction) on the outer periphery E10. In other words, the first vertical side E1 constitutes the right side of the display screen Dp1, the second vertical side E2 constitutes the left side of the display screen Dp1, the first horizontal side E3 constitutes the upper side of the display screen Dp1, and the second horizontal side E4 constitutes the lower side of the display screen Dp1. The first object X11 is disposed at the right end of the display screen Dp1 along the first vertical side E1, which is the right side of the display screen Dp1, and the second object X12 is disposed at the left end of the display screen Dp1 along the second vertical side E2, which is the left side of the display screen Dp1.

[0103] This improves the visibility of the first object X11 and the second object X12 while preventing the detection object X10 from interfering with other information included in the display screen Dp1.

[0104] The display screen Dp1 also has an information presentation region on at least one side of the detection object X10 in the vertical direction. In this embodiment, the first object X11 is disposed on the right side of the third region R3 on the display screen Dp1, and the second object X12 is disposed on the left side of the first region R1 on the display screen Dp1. Therefore, the tenth region R10 and the eighth region R8 are located above the first object X11 and the second object X12, and the seventh region R7 and the fifth region R5 are located below the first object X11 and the second object X12. Therefore, the fifth region R5, the seventh region R7, the eighth region R8, and the tenth region R10 are examples of information presentation regions located on both sides of the detection object X10 in the vertical direction.

[0105] With this configuration, when user U1 looks at various pieces of information displayed in the information presentation areas (fifth area R5, seventh area R7, eighth area R8, and tenth area R10), the detection object X10 naturally comes into view, making it easier to improve the visibility of the detection object X10.

[0106] The control method according to the present embodiment further includes displaying a captured image Im100 of the monitoring area A1 at a position between the first object X11 and the second object X12 on the display screen Dp1. That is, in the present embodiment, the captured image Im100 of the monitoring area A1 is displayed in the second region R2 located at the center of the display screen Dp1. This allows the user U1 to visually recognize the state of the monitoring area A1 on the display screen Dp1.

[0107] Here, as shown in FIG. 5, the captured image Im100 is a fan-shaped image with an arc-shaped outer shape on the side farther from the machine body 30. In the present disclosure, the term "arc-shaped" is not limited to an arc that constitutes a part of a perfect circle, but includes any curved line that is convex outward. Therefore, even if the captured image Im100 has an outer shape that is a curved line that is convex outward as a whole, it can be said to be a fan-shaped image with an arc-shaped outer shape. Here, the captured image Im100 is a fan-shaped image with the upper side of the arc cut out on the display screen Dp1, and the left, right, and lower sides are images of the monitoring area A1 (each of the small areas A11, A12, and A13) corresponding to the left, right, and rear of the machine body 30, respectively. That is, on the display screen Dp1, the captured image Im100 is displayed with the outer shape of at least the monitoring area A1 to the left, right, and rear of the work machine 3 formed into an arc-shaped shape.

[0108] More specifically, when a virtual circle is set with a reference point located near the center of the second region R2 as its center, the captured image Im100 has a sector shape lacking an exclusion range from a first end E101 to a second end E102, which is a part of the circumferential direction of the virtual circle. In the example of Fig. 5, the captured image Im100 has, on the display screen Dp1, a first end E101 that is a straight line extending from the reference point to the upper left and a second end E102 that is a straight line extending from the reference point to the upper right.

[0109] 5, the aircraft object Im10 is disposed at the apex of the captured image Im100. That is, the aircraft object Im10 is disposed at a position on the reference point that corresponds to the apex of the sector-shaped captured image Im100. In other words, the control method according to the present embodiment further includes displaying the aircraft object Im10 at the reference point on the display screen Dp1.

[0110] In this embodiment, the upper side of the machine body object Im10 corresponds to the front of the machine body 30 in the real space, the lower side of the machine body object Im10 corresponds to the rear of the machine body 30 in the real space, the right side of the machine body object Im10 corresponds to the right side of the machine body 30 in the real space, and the left side of the machine body object Im10 corresponds to the left side of the machine body 30 in the real space. Specifically, the lower part of the machine body object Im10 is configured in an arc shape imitating the rear part (counterweight) of the rotating part 32, and the upper part of the machine body object Im10 is configured in a shape imitating the front part (working part 33) of the rotating part 32. In this embodiment, the working part 33 is disposed at a position offset to the right with respect to the center of the rotating part 32 in the left-right direction D3, and the driving part 321 is disposed at a position offset to the left with respect to the center of the rotating part 32 in the left-right direction D3. Therefore, in the machine body object Im10, a figure imitating the working unit 33 is located in the upper right portion, and a figure imitating the driving unit 321 is located in the left portion.

[0111] Furthermore, in this embodiment, the machine body object Im10 is displayed based on the orientation of the turning unit 32 on which the driving unit 321 is mounted, rather than the orientation of the running unit 31. In other words, the orientation in real space corresponding to the upper part of the machine body object Im10 changes as the turning unit 32 turns relative to the running unit 31. As a result, the upper part of the machine body object Im10 corresponds to the front of the turning unit 32, which is always in front as seen by the operator riding in the driving unit 321.

[0112] Here, the orientation of the aircraft 30 represented by the aircraft object Im10 and the orientation of the captured image Im100 as viewed from the aircraft 30 are made to match. That is, the front of the aircraft 30 represented by the aircraft object Im10 and the front in the captured image Im100 are both directed upward on the display screen Dp1. This makes it easy for the operator to intuitively grasp the position of the detection target Ob1 by looking at the captured image Im100. However, this is not limited to the example, and the aircraft object Im10 may be disposed at a position other than the apex angle of the captured image Im100, for example, below the captured image Im100.

[0113] Moreover, the captured image Im200 of the rear camera 343 is disposed in an exclusion range between the first end E101 and the second end E102 in the circumferential direction of the virtual circle. That is, the captured image Im200 is disposed above the captured image Im100 on the display screen Dp1. As an example in this embodiment, the captured image Im200 has a pentagonal outer shape including two sides along the first end E101 and the second end E102. This allows the captured image Im200 of the small area A13 behind the driving section 321 captured by the rear camera 343 to be displayed by utilizing the dead space in the second region R2.

[0114] Furthermore, in this embodiment, since the detection object Ob1 in the monitoring area A1 is the detection object of the detection device 5, when the detection object Ob1 exists in the monitoring area A1, the detection object Ob1 will naturally be reflected in the captured image Im100 of the monitoring area A1. In this way, when the detection result indicates the presence of the detection object Ob1 in the monitoring area A1, at least a part of the detection object is reflected in the captured image Im100 on the display screen Dp1. This allows the user U1 to confirm the detection object Ob1 on the captured image Im100. In other words, the operator (user U1) can confirm the situation of the sides and rear of the work machine 3, which are likely to be blind spots from the driving unit 321, on the display screen Dp1 displayed on the display device 2. Therefore, when the detection object Ob1 exists in the monitoring area A1, the situation of the detection object Ob1 can be easily grasped in detail on the display screen Dp1.

[0115] Furthermore, the control method according to this embodiment further comprises displaying first status information relating to the status of the work machine 3 in a position between the first object X11 and the captured image Im100 on the display screen Dp1. In this embodiment, a third area R3 is located between the first object X11 and the captured image Im100. Therefore, the image Im1 according to the operating status of each part of the work machine 3 displayed in the third area R3 is an example of the first status information.

[0116] According to this configuration, when the user U1 looks at the first status information (image Im1), the first object X11 naturally comes into view, which makes it easier to improve the visibility of the detected object X10.

[0117] Furthermore, the control method according to this embodiment further comprises displaying second status information relating to the status of the work machine 3 at a position between the second object X12 and the captured image Im100 on the display screen Dp1. The second status information is various information relating to the status of the work machine 3, and is information different from the first status information (image Im1). In this embodiment, a first region R1 is located between the second object X12 and the captured image Im100. Therefore, for example, remaining amount information G1 relating to the remaining amount of fuel in the engine, which is displayed in the first region R1, is an example of the second status information.

[0118] According to this configuration, when the user U1 looks at the second status information (remaining amount information G1), the second object X12 naturally comes into view, which makes it easier to improve the visibility of the detected object X10.

[0119] Incidentally, the detection object X10 is an object whose display mode changes according to the detection result of the detection object Ob1. In this embodiment, the display mode of the detection object X10 includes at least one of the display color, size (dimension), and shape of the detection object X10. In other words, at least one of the display color, (display) size, and shape of the detection object X10 changes according to the detection result of the detection object Ob1.

[0120] Here, there are at least two display modes of the detection object X10: "non-detection" when the detection object Ob1 is not present in the monitoring area A1, and "detection" when the detection object Ob1 is present in the monitoring area A1. Furthermore, it is preferable that the display mode of the detection object X10 also changes depending on the distance from the work machine 3 to the detection object Ob1 and / or the direction of the detection object Ob1 (i.e., the position of the detection object Ob1). Therefore, even during "detection" when the detection object Ob1 is present in the monitoring area A1, the display mode of the detection object X10 further differs between when the distance from the work machine 3 to the detection object Ob1 is long and when the distance from the work machine 3 to the detection object Ob1 is short.

[0121] In the present embodiment, as an example, the display mode of the detection object X10 that changes according to the detection result of the detection device 5 includes, for example, the display color of the detection object X10. In this case, the display color of the detection object X10 on the display screen Dp1 changes according to the detection result of the detection device 5. For example, as shown in FIG. 5, when a detection target object Ob1 exists in the monitoring area A1, the display color of the detection object X10 becomes "yellow" or "red." On the other hand, as shown in FIG. 6, when a detection target object Ob1 does not exist in the monitoring area A1, the display color of the detection object X10 becomes "green."

[0122] In addition, the closer the detection object Ob1 is to the aircraft 30, the more the display color of the detection object X10 is changed to a more noticeable color. As an example, when the detection object Ob1 approaches the aircraft 30, the display color of the detection object X10 changes from yellow to red.

[0123] Furthermore, the display mode of the detection object X10 that changes according to the detection result of the detection device 5 may include at least one of the size and shape of the detection object X10 in addition to or instead of the display color of the detection object X10. For example, the display mode of the detection object X10 may change so that the size of the detection object X10 becomes larger as the detection target object Ob1 is closer to the aircraft body 30.

[0124] In this way, at least one of the display color, size, shape, and position of the detection object X10 changes depending on the detection result of the detection device 5, making it easier for the user U1 to intuitively grasp, from the display mode of the detection object X10, the presence of a detection target Ob1 (e.g., a person) in the monitoring area A1.

[0125] Here, multiple detection objects Ob1 may exist simultaneously in the monitoring area A1. In this case, it is preferable to determine the display mode of the detection object X10 according to the position of the detection object Ob1 closest to the body 30 of the work machine 3 among the multiple detection objects Ob1. For example, when a detection object Ob1 existing within a threshold distance from the work machine 3 and a detection object Ob1 existing farther away from the work machine 3 than the threshold distance are detected simultaneously, the display mode of the detection object X10 is determined based on the detection object Ob1 existing within the threshold distance from the work machine 3. As a result, the display mode of the detection object X10 is determined according to the detection result of the most urgent detection object Ob1, making it easier for the operator to intuitively grasp the detection result.

[0126] However, in this embodiment, the display position of the detection object X10 on the display screen Dp1 is fixed regardless of the detection result of the detection target object Ob1. In other words, the display position of the detection object X10 on the display screen Dp1 is invariable, and the detection object X10 is always disposed along the outer periphery E10 of the display screen Dp1. This improves the visibility of the detection object X10 for the user U1.

[0127] Furthermore, in this embodiment, when the detection target Ob1 is in a detectable state, the detection object X10 is displayed on the display screen Dp1. For example, when the detection processing unit 15 is enabled by the switching processing unit 13, the detection object X10 is displayed on the display screen Dp1 as shown in Figs. 5 and 6. On the other hand, when the detection processing unit 15 is disabled by the switching processing unit 13, the detection object X10 is not displayed on the display screen Dp1 as shown in Fig. 7. This allows the user U1 to intuitively know whether the detection processing unit 15 is enabled or not on the display screen Dp1.

[0128] Furthermore, in this embodiment, when the work machine 3 is in an inoperable state due to the cutoff lever 354, the detection object is hidden. In other words, not only when the detection processing unit 15 is disabled, but also when the cutoff lever 354 is in the "up position" and the output of the hydraulic actuator is forcibly stopped without operation of the operating lever, the detection object X10 is hidden on the display screen Dp1 as shown in Fig. 7. This allows the user U1 to intuitively grasp the operation state of the cutoff lever 354 on the display screen Dp1.

[0129] Furthermore, in this embodiment, when the "Camera" item is operated (selected) by operating the push button switch 224 corresponding to the "Camera" item while the display screen Dp1 as shown in Fig. 5 is displayed, the display mode of the display screen Dp1 transitions to a full screen mode as shown in Fig. 8. Furthermore, when any item is operated (selected) in the full screen mode, the display mode of the display screen Dp1 transitions from the full screen mode to a normal mode as shown in Fig. 5. However, the operations for transitioning the display screen Dp1 and the order of transitions are not limited to the above example and can be changed as appropriate.

[0130] In the full screen mode, the display screen Dp1 is a screen in which the second region R2 in which the captured image Im100 is displayed is enlarged compared to the display screen Dp1 shown in Fig. 5. In the example of Fig. 8, the display screen Dp1 includes only the second region R2. That is, in the full screen mode, the first region R1 and the third region R3 are not displayed, and the captured image Im100 is enlarged and displayed over the entire area of ​​the display screen Dp1. In this way, the display screen Dp1 has a plurality of display modes.

[0131] Here, the display mode of the detection object X10 does not change even if the display mode of the display screen Dp1 changes. That is, as described above, even if the display mode of the display screen Dp1 transitions between the normal mode (see FIG. 5) and the full screen mode (see FIG. 8), the display mode (including the display position) of the detection object X10 does not change. Therefore, the visibility of the detection object X10 is improved for the user U1.

[0132] [3.3] Overall processing Next, the overall flow of the process related to the control method will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the process related to the control method.

[0133] 9, the image acquisition unit 14 of the control system 1 acquires captured images from the left camera 341, the right camera 342, and the rear camera 343 (S1). Then, the display processing unit 11 of the control system 1 causes the display device 2 to display a display screen Dp1 including the captured image Im100 (S2).

[0134] Here, the display processing unit 11 of the control system 1 determines whether the detection processing unit 15 is enabled or disabled by the switching processing unit 13 (S3). If the detection processing unit 15 is enabled (S3: Yes), the display processing unit 11 displays the detection object X10 along the outer periphery E10 of the display screen Dp1 (S4). On the other hand, if the detection processing unit 15 is disabled (S3: No), the display processing unit 11 does not display the detection object X10 (S5).

[0135] The control system 1 repeatedly executes the processes of steps S1 to S5. As a result, the display screen Dp1 is constantly displayed on the display device 2. However, the flowchart shown in Fig. 9 is merely an example, and processes may be added or omitted as appropriate, and the order of processes may be changed as appropriate.

[0136] [4] Variations Below, we will list some modified examples of the embodiment 1. The modified examples explained below can be applied in appropriate combinations.

[0137] The control system 1 in the present disclosure includes a computer system. The computer system is mainly composed of one or more processors and one or more memories as hardware. The processor executes a program recorded in the memory of the computer system, thereby realizing the functions of the control system 1 in the present disclosure. The program may be pre-recorded in the memory of the computer system, may be provided through a telecommunication line, or may be recorded and provided in a non-transitory recording medium such as a memory card, an optical disk, or a hard disk drive that can be read by the computer system. In addition, some or all of the functional units included in the control system 1 may be configured with electronic circuits.

[0138] Furthermore, it is not essential for the control system 1 that at least some of the functions of the control system 1 are concentrated in one housing, and the components of the control system 1 may be distributed across multiple housings. Conversely, in the first embodiment, functions distributed across multiple devices (e.g., the control system 1 and the display device 2) may be concentrated in one housing. Furthermore, at least some of the functions of the control system 1 may be realized by the cloud (cloud computing) or the like.

[0139] Furthermore, the power source of the work machine 3 is not limited to a diesel engine, and may be, for example, an engine other than a diesel engine, or may be a motor (electric motor), or a hybrid power source including an engine and a motor (electric motor).

[0140] The display device 2 is not limited to a dedicated device, and may be, for example, a general-purpose terminal such as a laptop computer, a tablet terminal, or a smartphone. Furthermore, the display unit 23 is not limited to a mode that directly displays a display screen, such as a liquid crystal display or an organic EL display, and may be configured to display a display screen by projection, such as a projector.

[0141] Furthermore, a form other than a push button switch, a touch panel, or an operation dial may be adopted as the form of inputting information to the operation unit 22. For example, the operation unit 22 may adopt a form such as a keyboard, a pointing device such as a mouse, a voice input, a gesture input, or input of an operation signal from another terminal.

[0142] Furthermore, the restriction processing executed by the restriction processing unit 122 may be any processing that restricts the operation of the work machine 3, and is not limited to processing that prohibits (makes inoperable) the operation (turning operation, etc.) of the work machine 3. The restriction processing may be, for example, processing that reduces the speed of the operation (turning operation, etc.) of the work machine 3, processing that narrows the operating range (turning angle, etc.) of the work machine 3, processing that restricts the allowable area for operation of the work machine 3, or the like.

[0143] Furthermore, it is not essential that the restraint processing includes multiple specific processes (such as sound output processing and restriction processing) for restraining the operation of the work machine 3. Furthermore, even if the restraint processing includes multiple specific processes, it is not essential that each specific process can be switched between enabled and disabled individually, and multiple specific processes may be switched between enabled and disabled collectively.

[0144] Furthermore, the functions relating to check processing by the check processing unit 12 are not essential, and the check processing unit 12 can be omitted as appropriate.

[0145] In addition, the detection device 5 for detecting a detection object Ob1 in the monitoring area A1 around the work machine 3 may include, in addition to or instead of the sensor 51 and the imaging unit 52, a sensor such as a human presence sensor, a sonar sensor, a radar, or a LiDAR (Light Detection and Ranging).

[0146] Furthermore, the detection processing unit 15 may detect the detection object Ob1 in the monitoring area A1 based on, for example, the output (image data) of the left camera 341, the right camera 342, and the rear camera 343. Specifically, the detection processing unit 15 performs image processing on the image data acquired by the image acquisition unit 14 to extract a feature amount in the image, and determines whether or not the detection object Ob1 (in this embodiment, a "person") is captured in the image based on the feature amount. Here, if the detection object Ob1 is captured in the image, the detection processing unit 15 determines whether the detection object Ob1 is captured in an image captured by the left camera 341, the right camera 342, or the rear camera 343. In other words, the detection processing unit 15 detects the detection object Ob1 by distinguishing whether the detection object Ob1 is present in the small area A11 captured by the left camera 341, the small area A12 captured by the right camera 342, or the small area A13 captured by the rear camera 343.

[0147] The camera that captures the captured image Im100 is not limited to the left camera 341, the right camera 342, and the rear camera 343, and may include one, two, or four or more cameras (image sensors). Furthermore, the captured image may be captured by a camera that can capture images in all directions as seen from the work machine 3, such as a spherical camera (360-degree camera).

[0148] In addition, the detection object Ob1 may include, in addition to or instead of a "person," a moving object such as a vehicle (including other work machinery), structures such as walls and pillars, plants, animals, steps, ditches, or other obstacles.

[0149] In addition, it is not essential to reflect the positional relationship of the monitored area A1 to the work machine 3 in real space in the positional relationship of the captured image Im100 to the machine object Im10 on the display screen Dp1. Furthermore, it is not essential to reflect the positional relationship of the monitored area A1 to the rotating part 32 in real space in the positional relationship of the captured image Im100 to the machine object Im10 on the display screen Dp1.

[0150] Furthermore, it is not essential that the display mode of the detection object X10 changes according to the position of the detection target Ob1.

[0151] (Embodiment 2) 10 and 11, the display content of the second area R2 of the display screen Dp1 of the work machine 3 according to this embodiment is different from that of the work machine 3 according to embodiment 1. Hereinafter, the same components as those in embodiment 1 will be denoted by the same reference numerals and descriptions thereof will be omitted as appropriate.

[0152] In this embodiment, the captured image displayed in the second area R2 is, for example, an image captured by at least one of the left camera 341, the right camera 342, and the rear camera 343 as is (that is, an uncombined image).

[0153] 10, the second region R2 displays a captured image Im11 of a small area A11 captured by the left camera 341, a captured image Im12 of a small area A12 captured by the right camera 342, and a captured image Im13 of a small area A13 captured by the rear camera 343. In this example, the captured images Im11 and Im12 are arranged side by side in the left-right direction in the second region R2, and the captured image Im13 is arranged below the captured images Im11 and Im12 in the second region R2.

[0154] 11, the second region R2 displays a captured image Im13 of the small area A13 captured by the rear camera 343. In this example, the machine object Im10 is not displayed.

[0155] The configuration according to the second embodiment can be adopted in appropriate combination with the various configurations (including modified examples) described in the first embodiment.

[0156] [Appendix to the invention] The following will provide an overview of the invention extracted from the above-described embodiment. Note that the configurations and processing functions described in the following supplementary notes can be selected and combined as desired.

[0157] <Appendix 1> Detecting a detection object in a monitoring area around the work machine; Displaying the display screen on a display device; displaying, on the display screen, a detection object whose display mode changes according to a detection result of the detection target object; the detection object is disposed along the outer periphery of the display screen; A method for controlling a work machine.

[0158] <Appendix 2> When the detection object is in a state where it can be detected, the detection object is displayed on the display screen. 2. A method for controlling a work machine as described in claim 1.

[0159] <Appendix 3> When the work machine is in an inoperable state due to a cut-off lever, the detection object is hidden. 3. A method for controlling a work machine as described in appended claim 2.

[0160] <Appendix 4> Regardless of the detection result of the detection target object, the display position of the detection object on the display screen is fixed. A control method for a work machine according to any one of appendices 1 to 3.

[0161] <Appendix 5> The detection objects include a first object and a second object that are arranged symmetrically in at least one direction. A control method for a work machine according to any one of appendices 1 to 4.

[0162] <Appendix 6> the outer periphery includes a first vertical side and a second vertical side facing each other in a left-right direction of the display screen, the first object is disposed along the first vertical side; The second object is disposed along the second vertical side. 6. A method for controlling a work machine as described in appendix 5.

[0163] <Appendix 7> The display screen has an information presentation area on at least one side of the detection object in an up-down direction. 7. A method for controlling a work machine as described in appendix 6.

[0164] <Appendix 8> displaying a captured image of the monitoring area at a position between the first object and the second object on the display screen. A control method for a work machine according to any one of appendices 5 to 7.

[0165] <Appendix 9> displaying first status information regarding a status of the work machine at a position between the first object and the captured image on the display screen. 9. A method for controlling a work machine as described in appended claim 8.

[0166] <Appendix 10> and displaying second status information regarding a status of the work machine at a position between the second object and the captured image on the display screen. 10. A method for controlling a work machine according to claim 8 or 9.

[0167] <Appendix 11> The display mode of the detection object includes at least one of a display color, a size, and a shape of the detection object. A control method for a work machine according to any one of appendices 1 to 10.

[0168] <Appendix 12> the display manner of the detection object does not change even if the display mode of the display screen changes; A control method for a work machine according to any one of appendices 1 to 11.

[0169] <Appendix 13> A method for controlling a work machine according to any one of appendices 1 to 12, A control program for a work machine for execution by one or more processors. [Explanation of symbols]

[0170] 1. Control system for work machines 2 Display device 3. Work Machinery 11 Display processing section 15 Detection processing section 30 Aircraft A1 Surveillance Area Dp1 display screen E10 Outer edge E1 First vertical side E2 Second vertical side G1 remaining amount information (second status information) Im1 Iconography (First State Information) Im11, Im12, Im13, Im100 Images Ob1 Detected object R5 5th area (information presentation area) R7 7th area (information presentation area) R8 8th area (information presentation area) R10 10th area (information presentation area) X10 Detect Object X11 1st Object X12 Second Object

Claims

1. Detecting a detection object in a monitoring area around the work machine; Displaying the display screen on a display device; displaying, on the display screen, a detection object whose display mode changes according to a detection result of the detection target object; the detection object is disposed along the outer periphery of the display screen; A method for controlling a work machine.

2. When the detection object is in a state where it can be detected, the detection object is displayed on the display screen. A method for controlling a work machine according to claim 1.

3. When the work machine is in an inoperable state due to a cut-off lever, the detection object is hidden. A method for controlling a work machine according to claim 2.

4. Regardless of the detection result of the detection target object, the display position of the detection object on the display screen is fixed. A control method for a work machine according to any one of claims 1 to 3.

5. The detection object includes a first object and a second object that are arranged symmetrically in at least one direction. A control method for a work machine according to any one of claims 1 to 3.

6. the outer periphery includes a first vertical side and a second vertical side facing each other in a left-right direction of the display screen, the first object is disposed along the first vertical side; The second object is disposed along the second vertical side. A method for controlling a work machine according to claim 5.

7. The display screen has an information presentation area on at least one side of the detection object in an up-down direction. A method for controlling a work machine according to claim 6.

8. displaying a captured image of the monitoring area at a position between the first object and the second object on the display screen. A method for controlling a work machine according to claim 5.

9. displaying first status information regarding a status of the work machine at a position between the first object and the captured image on the display screen. A method for controlling a work machine according to claim 8.

10. and displaying second status information regarding a status of the work machine at a position between the second object and the captured image on the display screen. A method for controlling a work machine according to claim 8.

11. The display mode of the detection object includes at least one of a display color, a size, and a shape of the detection object. A control method for a work machine according to any one of claims 1 to 3.

12. the display manner of the detection object does not change even if the display mode of the display screen changes; A control method for a work machine according to any one of claims 1 to 3.

13. A control method for a work machine according to any one of claims 1 to 3, A control program for a work machine for execution by one or more processors.

14. a detection processing unit that detects a detection object in a monitoring area around the work machine; A display processing unit that displays a display screen on a display device, the display processing unit displays, on the display screen, a detection object whose display manner changes according to a detection result of the detection target object; the detection object is disposed along the outer periphery of the display screen; Control systems for work machines.

15. A control system for a work machine according to claim 14; A machine body on which the display device is mounted. Working machinery.

Images