Control method for work machines, control program for work machines, control system for work machines, and work machines

Abstract

The present invention provides a control method for a work machine, a control program for a work machine, a control system for a work machine, and a work machine that allow the operator to intuitively understand the presence of a target object in the monitoring area. [Solution] The control method for the work machine includes detecting multiple objects to be detected in a monitoring area around the work machine, displaying a display screen Dp1 on a display device, and determining the display mode of the detection results displayed on the display screen Dp1 according to the distance from the work machine to the object closest to it among the multiple objects to be detected.

Description

Technical Field

[0001] The present invention relates to a control method for a working machine, a control program for a working machine, a control system for a working machine, and a working machine, which are used for a working machine having a function of detecting a detection target in a surrounding monitoring area.

Background Art

[0002] As a related art, a working machine (excavator) equipped with a peripheral monitoring system for a working machine that monitors the surroundings of the working machine is known (for example, see Patent Document 1). The working machine according to the related art displays a peripheral monitoring image synthesized using the captured images of a plurality of imaging devices attached to the working machine on the screen of a display device, and relatively enlarges and displays a part of the peripheral monitoring image. Here, when a person is present in the monitoring area around the working machine, the peripheral monitoring system displays a yellow circle (or red circle) as a person detection marker at the position of a reference point corresponding to the person in the display screen (output image), and displays the area corresponding to that position in yellow (or red).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above related art, since the amount of information displayed on the display device is large, it may be difficult for an operator to intuitively grasp the presence of a detection target (for example, a person) in the monitoring area.

[0005] An object of the present invention is to provide a control method for a working machine, a control program for a working machine, a control system for a working machine, and a working machine, which make it easy for an operator to intuitively grasp the presence of a detection target in a monitoring area. [Means for solving the problem]

[0006] A control method for a work machine according to one aspect of the present invention includes detecting a plurality of objects to be detected in a monitoring area surrounding the work machine, displaying a display screen on a display device, and determining the display mode of the detection results to be displayed on the display screen according to the distance from the work machine to the object among the plurality of objects to be detected.

[0007] A control program for a work machine according to one aspect of the present invention is a program that causes one or more processors to execute a control method for the work machine.

[0008] A control system for a work machine according to one aspect of the present invention comprises a detection unit and a display processing unit. The detection unit detects a plurality of objects to be detected in a monitoring area surrounding the work machine. The display processing unit causes a display screen to be displayed on a display device. The display processing unit determines the display mode of the detection results to be displayed on the display screen according to the distance from the work machine to the object among the plurality of objects to be detected.

[0009] A work machine according to one aspect of the present invention comprises a control system for the work machine and a machine body on which the display device is mounted. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide a control method for a work machine, a control program for a work machine, a control system for a work machine, and a work machine that make it easy for an operator to intuitively grasp whether an object to be detected is present in the monitoring area. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a schematic perspective view showing the overall configuration of the work machine according to Embodiment 1. [Figure 2] Figure 2 is a schematic diagram showing the hydraulic circuit and other components of the work machine according to Embodiment 1. [Figure 3]Figure 3 is a schematic plan view of the work machine according to Embodiment 1, as seen from above, illustrating the monitoring area and other elements set around the work machine. [Figure 4] Figure 4 is a schematic external view of a display device whose display screen is shown by the control system for work machines according to Embodiment 1. [Figure 5] Figure 5 shows an example of a display screen shown by the control system for work machines according to Embodiment 1. [Figure 6] Figure 6 shows an example of how detected objects are displayed on the display screen shown by the control system for work machines according to Embodiment 1. [Figure 7] Figure 7 shows an example of the display of the second area of ​​the display screen shown by the control system for work machines according to Embodiment 1. [Figure 8] Figure 8 is a flowchart showing an example of the operation of the control system for a work machine according to Embodiment 1. [Figure 9] Figure 9 shows an example of the display of the second area of ​​the display screen shown by the control system for work machines according to Embodiment 2. [Figure 10] Figure 10 shows an example of the display of the second area of ​​the display screen shown by the control system for work machines according to Embodiment 2. [Modes for carrying out the invention]

[0012] The embodiments of the present invention will be described below with reference to the attached drawings. The following embodiments are examples that embody the present invention and are not intended to limit the technical scope of the present invention.

[0013] (Embodiment 1) [1] Overall structure As shown in Figure 1, the work machine 3 according to this embodiment is equipped with a traveling section 31, a rotating section 32, and a working section 33 on a machine body 30. Furthermore, as shown in Figure 2, the work machine 3 is equipped with a control system 1 for work machines (hereinafter also simply referred to as "control system 1"). In addition, the machine body 30 is equipped with a display device 2 and an operating device, etc.

[0014] As used in this disclosure, the "working machine" means various working machines. As an example, it is a working vehicle such as a backhoe (including a hydraulic excavator, a mini excavator, etc.), a wheel loader, and a carrier. The working machine 3 includes a working unit 33 configured to be capable of performing one or more operations including at least a lifting operation. The working machine 3 is not limited to a "vehicle", and may be, for example, a working flying object such as a working ship, a drone, or a multicopter. Furthermore, the working machine 3 is not limited to a construction machine, and may be, for example, an agricultural machine (agricultural implement) such as a rice transplanter, a tractor, or a combine. In the present embodiment, unless otherwise specified, the working machine 3 is a backhoe with a lifting function (a crane function), and the case where, in addition to the lifting operation, excavation work, leveling work, trench excavation work, loading work, or the like can be performed as work will be described as an example.

[0015] Also, in the present embodiment, for convenience of explanation, the vertical direction in the state where the working machine 3 can be used is defined as the up-and-down direction D1. Furthermore, in the non-rotating state of the rotating unit 32, the front-back direction D2 and the left-right direction D3 are defined based on the direction seen from the user (operator) boarding the working machine 3 (the driving unit 321). In other words, each direction used in the present embodiment is a direction defined based on the machine body 30 of the working machine 3, and the direction in which the machine body 30 moves when the working machine 3 moves forward is the "front", and the direction in which the machine body 30 moves when the working machine 3 moves backward is the "rear". Similarly, the direction in which the front end of the machine body 30 moves when the working machine 3 turns to the right is the "right", and the direction in which the front end of the machine body 30 moves when the working machine 3 turns to the left is the "left". However, these directions are not intended to limit the usage direction (direction during use) of the working machine 3.

[0016] The work machine 3 includes an engine as a power source. In the work machine 3, for example, the hydraulic pump 41 (see FIG. 2) is driven by the engine, and hydraulic oil is supplied from the hydraulic pump 41 to the hydraulic actuators (including the hydraulic motor 43 and the hydraulic cylinder 44, etc.) of each part of the machine body 30, so that the machine body 30 is driven. Further, the work machine 3 is controlled, for example, by a user (operator) boarding the operation unit 321 of the machine body 30 operating an operation lever or the like of the operation device.

[0017] In the present embodiment, since it is assumed that the work machine 3 is a ride-on type backhoe as described above, the work unit 33 is driven according to the operation of a user (operator) boarding the operation unit 321 and performs work such as excavation work. The operation unit 321 on which the user boards is provided on the swing unit 32.

[0018] The traveling unit 31 has a traveling function and is configured to be able to travel (including turning) on the ground. The traveling unit 31 has, for example, a pair of left and right crawlers 311 and a blade 312, etc. The traveling unit 31 further has a hydraulic motor 43 (hydraulic actuator) for traveling for driving the crawler 311.

[0019] The swing unit 32 is located above the traveling unit 31 and is configured to be able to swing about a rotation axis along the vertical direction with respect to the traveling unit 31. The swing unit 32 has a hydraulic motor (hydraulic actuator) for swinging, etc. In addition to the operation unit 321, the swing unit 32 is equipped with an engine and a hydraulic pump 41, etc. Further, a boom bracket 322 to which the work unit 33 is attached is provided at the front end of the swing unit 32. [[ID=一三]]

[0020] The work unit 33 is configured to be able to perform work including lifting work. The work unit 33 is supported by the boom bracket 322 of the swing unit 32 and performs work. The work unit 33 has a bucket 331, a boom <三三二> and an arm 333, etc. The work unit 33 further has hydraulic actuators (including hydraulic cylinders 44 and hydraulic motors, etc.) for driving each part.

[0021] The bucket 331 is a type of attachment (working tool) that is mounted on the machine body 30 of the work machine 3, and consists of any tool selected from several types of attachments according to the nature of the work. For example, the bucket 331 is detachably mounted to the machine body 30 and can be replaced according to the nature of the work. In addition to the bucket 331, various other tools can be used as attachments for the work machine 3, such as breakers, augers, crushers, forks, fork claws, steel frame cutters, asphalt milling machines, brush cutters, rippers, mulchers, tilt rotators, and tampers. The work unit 33 performs its work by driving the bucket 331 with power from the drive unit.

[0022] The boom 332 is rotatably supported by the boom bracket 322 of the slewing section 32. Specifically, the boom 332 is rotatably supported by the boom bracket 322 around a rotation axis along the horizontal direction. The boom 332 has a shape that extends upward from the base end supported by the boom bracket 322. The arm 333 is connected to the tip of the boom 332. The arm 333 is rotatably supported relative to the boom 332 around a rotation axis along the horizontal direction. A bucket 331 is attached to the tip of the arm 333.

[0023] The work unit 33 operates by receiving power from the engine, which serves as the power source. Specifically, the engine drives the hydraulic pump 41, and hydraulic fluid is supplied from the hydraulic pump 41 to the hydraulic actuators (hydraulic cylinders 44, etc.) of the work unit 33, thereby operating each part of the work unit 33 (bucket 331, boom 332, and arm 333).

[0024] In this embodiment, the work unit 33 has a multi-joint structure in which the boom 332 and the arm 333 are configured to rotate independently. That is, by each of the boom 332 and the arm 333 rotating around a rotation axis along the horizontal direction, the multi-joint work unit 33 including the boom 332 and the arm 333 can be extended or folded as a whole.

[0025] The traveling unit 31 and the slewing unit 32, like the working unit 33, operate by receiving power from the engine as a power source. In other words, the slewing unit 32 and the traveling unit 31 operate when hydraulic fluid is supplied from the hydraulic pump 41 to the hydraulic motor 43 of the traveling unit 31 and the hydraulic motor of the slewing unit 32, etc.

[0026] As described above, the engine functions as a power source that supplies power to each part. Here, the engine is mounted on the slewing section 32 together with the hydraulic pump 41, etc. In this embodiment, as an example, the engine is a diesel engine. The engine is driven by fuel (diesel in this case) supplied from the fuel tank.

[0027] Here, the machine body 30 is equipped with various sensors (including cameras) for detecting the object Ob1 (see Figure 3) in the monitoring area A1 (see Figure 3) surrounding the work machine 3, including a camera that takes images of the area around the machine body 30. In this embodiment, as an example, as shown in Figure 3, multiple cameras (three in this case), including a left camera 341, a right camera 342, and a rear camera 343, are mounted on the rotating section 32 of the machine body 30. The left camera 341, the right camera 342, and the rear camera 343 are connected to the control system 1 and output the images they capture to the control system 1. Figure 3 is a plan view of the work machine 3 seen from above, schematically representing the monitoring area A1 set around the work machine 3, the object Ob1, and the machine body 30 of the work machine 3 (including the left camera 341, the right camera 342, and the rear camera 343).

[0028] The left camera 341, the right camera 342, and the rear camera 343 are positioned to the left, right, and rear of the driver's unit 321, respectively, so that they can capture images of the monitoring area A1 that is to the left, right, and rear of the operator seated in the driver's unit 321 of the rotating unit 32. In other words, as shown in Figure 3, the monitoring area A1 includes several (in this case, three) sub-areas A11, A12, and A13, and the left camera 341 captures the sub-area A11 (left area) which is to the left of the operator seated in the driver's unit 321. Similarly, the right camera 342 captures the sub-area A12 (right area) which is to the right of the operator seated in the driver's unit 321, and the rear camera 343 captures the sub-area A13 (rear area) which is behind the operator seated in the driver's unit 321. This makes it possible to cover the sides (left and right) and rear, which are often blind spots for the operator, using the left camera 341, the right camera 342, and the rear camera 343.

[0029] Figure 2 schematically shows the hydraulic and electrical circuits (electrical connection relationships) of the work machine 3 according to this embodiment. In Figure 2, solid lines represent high-pressure (hydraulic oil) oil passages, dotted lines represent low-pressure (pilot oil) oil passages, and dashed arrows indicate the paths of electrical signals.

[0030] As shown in Figure 2, the work machine 3 is equipped with a hydraulic pump 41, a hydraulic motor 43 (not shown in Figure 2), and a hydraulic cylinder 44, as well as a pilot pump 42, a remote control valve 45, a control valve 46, and a directional control valve 47, etc.

[0031] Hydraulic fluid from the engine-driven hydraulic pump 41 is supplied to the hydraulic motor 43 of the travel section 31, the hydraulic motor of the slewing section 32, and the hydraulic cylinder 44 of the work section 33, etc. This drives the hydraulic actuators such as the hydraulic motor 43 and the hydraulic cylinder 44.

[0032] Hydraulic actuators such as the hydraulic motor 43 and hydraulic cylinder 44 are equipped with a pilot-operated directional control valve 47 that can switch the direction and flow rate of the hydraulic fluid from the hydraulic pump 41. The directional control valve 47 is driven by pilot oil supplied from the pilot pump 42, which serves as an input command.

[0033] Here, for example, a remote control valve 45 is provided in the supply path for pilot oil to the directional control valve 47 corresponding to the hydraulic cylinder 44 of the work unit 33. The remote control valve 45 outputs work operation commands for the work unit 33 in response to the operation of the operating lever. The work operation commands instruct the work unit 33 to perform operations such as extending and retracting. In addition, an electromagnetic control valve 46 (solenoid valve) is inserted between the remote control valve 45 and the pilot pump 42. The control valve 46 is connected to the power supply 351 via a cutoff relay 352 and a cutoff switch 353, and operates in accordance with the current supplied from the power supply 351.

[0034] Similarly, a remote control valve is also provided in the supply path for pilot oil to the directional control valve corresponding to the hydraulic motor 43 of the travel unit 31. This remote control valve outputs a travel operation command for the travel unit 31 in response to the operation of the operating lever. The travel operation command instructs the travel unit 31 to travel (forward or backward, etc.). Furthermore, a remote control valve is also provided in the supply path for pilot oil to the directional control valve corresponding to the hydraulic motor of the slewing unit 32. This remote control valve outputs a slewing operation command for the slewing unit 32 in response to the operation of the operating lever. The slewing operation command instructs the slewing unit 32 to slewing (left turn or right turn, etc.). An electromagnetic control valve 46 (solenoid valve) is also inserted between these remote control valves and the pilot pump 42. The control valve 46 is connected to the power supply 351 via a cutoff relay 352 and a cutoff switch 353, and operates in accordance with the current supplied from the power supply 351.

[0035] The control valve 46 opens the flow path of pilot oil from the pilot pump 42 to the remote control valve 45 when energized, i.e., when current is supplied, and closes the flow path of pilot oil when de-energized, i.e., when the supply current is cut off. As a result, when the current supplied to the control valve 46 is cut off, the hydraulic actuator corresponding to the remote control valve 45 becomes unable to drive, and the output of the hydraulic actuator is forcibly stopped regardless of the operation of the operating lever.

[0036] Here, the cutoff relay 352 is connected to the control system 1 and switches on / off in response to a control signal (electrical signal) from the control system 1. The cutoff switch 353 switches on / off in response to the operation of the cutoff lever, for example, it turns on when the cutoff lever is operated downwards. Therefore, when both the cutoff relay 352 and the cutoff switch 353 are on, the control valve 46 is energized, and the flow path of pilot oil from the pilot pump 42 to the remote control valve 45 is opened, so the hydraulic actuator is driven in response to the operation of the operating lever. Conversely, when at least one of the cutoff relay 352 and the cutoff switch 353 is off, the control valve 46 is de-energized, and the flow path of pilot oil is blocked, so the hydraulic actuator cannot be driven.

[0037] For example, if at least one of the cutoff relay 352 and cutoff switch 353, which are connected to a control valve 46 inserted between the remote control valve corresponding to the hydraulic motor of the slewing section 32 and the pilot pump 42, is in the off state, the hydraulic motor of the slewing section 32 becomes inoperable. In this state, the output of the hydraulic actuator (hydraulic motor of the slewing section 32) is forcibly stopped regardless of the operation of the operating lever, thus preventing the slewing operation of the slewing section 32.

[0038] The control system 1 primarily consists of a computer system having one or more processors such as a CPU (Central Processing Unit) and one or more memories such as ROM (Read Only Memory) and RAM (Random Access Memory), and performs various processes (information processing). In this embodiment, the control system 1 is an integrated controller that controls the entire work machine 3, and consists of, for example, an electronic control unit (ECU). However, the control system 1 may be provided separately from the integrated controller. The control system 1 will be explained in detail in the section "[2] Configuration of the Control System".

[0039] The display device 2 is located in the control unit 321 of the machine body 30 and is a user interface for receiving operation input from the user (operator) and outputting various information to the user. The display device 2 accepts various operations from the user by outputting electrical signals corresponding to the user's operations. As a result, the user (operator) can view the display screen Dp1 (see Figure 4) displayed on the display device 2 and operate the display device 2 as needed.

[0040] As shown in Figure 2, the display device 2 comprises a control unit 21, an operation unit 22, and a display unit 23. The display device 2 is configured to communicate with the control system 1 and can exchange data with the control system 1. In this embodiment, as an example, the display device 2 is a dedicated device used in the work machine 3.

[0041] The control unit 21 controls the display device 2 according to data from the control system 1. Specifically, the control unit 21 outputs electrical signals corresponding to user operations received by the operation unit 22, and displays the display screen Dp1 generated by the control system 1 on the display unit 23.

[0042] The operation unit 22 is a user interface for receiving operation input from a user (operator) to the display screen Dp1 displayed on the display unit 23. The operation unit 22 accepts various operations by user U1 (see Figure 4) by outputting electrical signals corresponding to the user U1's operations. In this embodiment, as an example, the operation unit 22 includes a plurality of (in this case, six) mechanical push-button switches 221 to 226, as shown in Figure 4. These plurality of push-button switches 221 to 226 are arranged close to the display area (below in the example of Figure 4) so ​​as to be along the periphery of the display area of ​​the display unit 23. These plurality of push-button switches 221 to 226 are associated with items displayed on the display screen Dp1, which will be described later, and when any of the plurality of push-button switches 221 to 226 are operated, any of the items on the display screen Dp1 are operated (selected).

[0043] Furthermore, the operation unit 22 may include a touch panel and an operation dial. In this case as well, an operation on the operation unit 22 will result in an operation (selection) of one of the items on the display screen Dp1.

[0044] The display unit 23 is a user interface for presenting information to the user U1 (operator), such as a liquid crystal display or an organic EL display that displays various types of information. The display unit 23 presents various types of information to the user through display. In this embodiment, as an example, the display unit 23 is a full-color liquid crystal display with a backlight, and as shown in Figure 4, it has a horizontally elongated "landscape" display area.

[0045] The display device 2 presents various information to the user U1 (operator) operating the work machine 3 via the display screen Dp1. In other words, the user U1 operating the work machine 3 can visually obtain various information related to the work machine 3 by looking at the display screen Dp1 displayed on the display device 2. For example, by displaying information related to the operating status of the work machine 3, such as the cooling water temperature and hydraulic oil temperature, on the display device 2, the user U1 can check the information related to the operating status of the work machine 3 necessary for operating the work machine 3 on the display device 2. In addition, the display device 2 can also display images of the surrounding area of ​​the work machine 3 (images of the monitoring area A1) captured by the left camera 341, the right camera 342, and the rear camera 343 on the display screen Dp1. This allows the user U1 (operator) to check the situation of the side and rear of the work machine 3, which are often blind spots from the driver's unit 321, on the display screen Dp1 displayed on the display device 2 when operating the work machine 3.

[0046] Furthermore, the work machine 3 is equipped with a sound output unit 36 ​​(see Figure 2) that outputs sound (including voice) to the user U1 (operator). The sound output unit 36 ​​includes a buzzer or speaker, etc., and outputs sound in response to an electrical signal. The sound output unit 36 ​​is connected to the control system 1 and outputs sounds such as beeps or voices in response to sound control signals from the control system 1. In this embodiment, the sound output unit 36 ​​is provided in the operating unit 321 of the machine body 30, similar to the display device 2. The sound output unit 36 ​​may be provided integrally with the display device 2.

[0047] In addition to the above-described configuration, the aircraft 30 is further equipped with an operating lever, a cutoff lever, a communication terminal, a fuel tank, and a battery. Furthermore, the aircraft 30 is equipped with sensors to monitor its operating status, such as a coolant temperature sensor, a hydraulic oil temperature sensor, a tachometer for measuring engine speed, and an hour meter for measuring operating time. The aircraft 30 is also equipped with sensors to detect the status of the cutoff lever and starter key switch, etc.

[0048] [2] Control system configuration Next, the configuration of the control system 1 according to this embodiment will be described with reference to Figure 2. The control system 1 controls the display device 2 to display the display screen Dp1 on the display device 2. In this embodiment, the display device 2 is mounted on the machine body 30 of the work machine 3 as described above. The control system 1 is a component of the work machine 3 and together with the machine body 30, etc., constitutes the work machine 3. In other words, the work machine 3 according to this embodiment comprises at least the control system 1 and the machine body 30 (including the traveling section 31, the rotating section 32, and the working section 33) on which the display device 2 is mounted.

[0049] In this disclosure, "screen" as used in display screen Dp1, etc., means the image (video) displayed on the display device 2, and includes illustrations, figures, photographs, text, and videos. That is, the control system 1 can display on the display device 2 a display screen Dp1 that includes illustrations, etc., representing information about the operating status of the work machine 3, such as the cooling water temperature and the hydraulic oil temperature. Here, if the display screen Dp1 includes videos, etc., the display screen Dp1 does not include a constant video, but rather a video that changes moment by moment.

[0050] As shown in Figure 2, the control system 1 includes a display processing unit 11, a control processing unit 12, a switching processing unit 13, an image acquisition unit 14, and a detection unit 15. In this embodiment, as an example, the control system 1 mainly consists of a computer system having one or more processors, so these multiple functional units (display processing unit 11, etc.) are realized by one or more processors executing a control program for the work machine. These multiple functional units included in the control system 1 may be distributed and provided in multiple housings, or they may be provided in a single housing.

[0051] The control system 1 is configured to communicate with devices provided in various parts of the aircraft body 30. Specifically, the control system 1 is connected to at least the display device 2, the sound output unit 36, the cutoff relay 352, the left camera 341, the right camera 342, and the rear camera 343. This allows the control system 1 to control the display device 2 and the sound output unit 36, control the cutoff relay 352 to control the control valve 46, and acquire images captured by the left camera 341, the right camera 342, and the rear camera 343. Here, the control system 1 may exchange various types of information (data) directly with each device, or indirectly via a relay or the like.

[0052] The image acquisition unit 14 performs image acquisition processing to acquire captured images of the monitoring area A1 surrounding the work machine 3. In this embodiment, the image acquisition unit 14 periodically or irregularly acquires the outputs of the left camera 341, the right camera 342, and the rear camera 343 from the left camera 341, the right camera 342, and the rear camera 343. In other words, the image acquisition unit 14 acquires image data (captured images) of the monitoring area A1 (each sub-area A11, A12, A13) surrounding the work machine 3. The data acquired by the image acquisition unit 14 is stored in, for example, memory.

[0053] The detection unit 15 detects the object to be detected Ob1 in the monitoring area A1 surrounding the work machine 3. In other words, the detection unit 15 determines the presence or absence of the object to be detected Ob1 in the monitoring area A1 and outputs a detection result indicating whether or not the object to be detected Ob1 exists in the monitoring area A1. In this embodiment, as an example, the object to be detected Ob1 is a "person". In other words, if the work machine 3 moves, or if a "person" moves around the work machine 3, and as a result a "person" enters the monitoring area A1 surrounding the work machine 3, the detection unit 15 detects the "person" as the object to be detected Ob1. If there are multiple objects to be detected Ob1 in the monitoring area A1, the detection unit 15 may also detect the number of objects to be detected Ob1 (number of people).

[0054] In this embodiment, the detection unit 15 detects the object to be detected Ob1 in the monitoring area A1 based on the output (image data) of the left camera 341, the right camera 342, and the rear camera 343. Specifically, the detection unit 15 extracts feature quantities from the image by performing image processing on the image data acquired by the image acquisition unit 14, and determines whether or not the object to be detected Ob1 (in this embodiment, "person") is captured in the image based on these feature quantities. If the object to be detected Ob1 is captured in the image, the detection unit 15 determines whether the object to be detected Ob1 is captured in the image captured by the left camera 341, the right camera 342, or the rear camera 343. In other words, the detection unit 15 distinguishes whether the object to be detected Ob1 is located in the small area A11 captured by the left camera 341, the small area A12 captured by the right camera 342, or the small area A13 captured by the rear camera 343, and then detects the object to be detected Ob1.

[0055] The deterrent processing unit 12 executes deterrent processing to deter the operation of the work machine 3 based on the detection result of the detection unit 15. In this embodiment, the deterrent processing unit 12 executes deterrent processing when the detection result of the detection unit 15 indicates the presence of a detected object Ob1 (in this case, a person) in the monitoring area A1. In this disclosure, "deterrent processing" means processing that acts in a direction that suppresses the operation of the work machine 3 in some way. As an example, the deterrent processing includes processing that indirectly deter the operation of the work machine 3 by issuing a warning to the user U1 (operator) operating the work machine 3 by sound or light (including display). Furthermore, the deterrent processing includes processing that directly deter the operation of the work machine 3 by controlling the travel unit 31, the swivel unit 32, the work unit 33, etc. of the work machine 3.

[0056] In this embodiment, the deterrent processing unit 12 includes a sound output processing unit 121 and a limiting processing unit 122.

[0057] The sound output processing unit 121 controls the sound output unit 36 ​​to output a notification sound when a detected object Ob1 is present in the monitoring area A1. In other words, in this embodiment, the deterrent process includes sound output processing that outputs a notification sound. The notification sound may be a simple beep or a voice message such as "Please be careful." Furthermore, the notification sound may change according to the detection result of the detection unit 15 (such as the distance from the machine 30 to the detected object Ob1). This allows for indirect deterrence of the operation of the work machine 3 by providing a warning sound to the user U1 (operator) operating the work machine 3, thus increasing the degree of freedom in operating the work machine 3. In other words, by operating the work machine 3 while paying attention to the detected object Ob1, the user U1 can continue operating the work machine 3 while avoiding contact with the detected object Ob1.

[0058] The limiting processing unit 122 controls the cutoff relay 352 to turn it off when the object Ob1 is present in the monitoring area A1. As a result, the control valve 46 connected to the power supply 351 via the cutoff relay 352 becomes de-energized, and the output of the hydraulic actuator corresponding to the control valve 46 is forcibly stopped. In other words, in this embodiment, the restraining process includes a limiting process that restricts the operation of the work machine 3. In this disclosure, "limiting process" means a process that acts in a direction that restricts the operation of the work machine 3 in some way. For example, the limiting process includes a process that prohibits the travel operation of the travel unit 31 (making it impossible to travel), a process that prohibits the rotation operation of the slewing unit 32 (making it impossible to rotate), and a process that prohibits the operation of the work unit 33 (making it impossible to work). As a result, the operation of the work machine 3 can be forcibly restricted without the operation of the user U1 (operator). In other words, contact between the machine body 30 and the object Ob1 caused by the operation of the work machine 3 can be avoided.

[0059] Here, the restriction processing performed by the restriction processing unit 122 includes at least processing to restrict the rotational movement of the slewing unit 32. Specifically, the restriction processing unit 122 is configured to control a cutoff relay 352 connected to a control valve 46 corresponding to the hydraulic motor of the slewing unit 32, and turns off the cutoff relay 352 when a detected object Ob1 is present in the monitoring area A1. As a result, when a detected object Ob1 is present in the monitoring area A1, the hydraulic motor of the slewing unit 32 becomes inoperable, the slewing unit 32 makes an emergency stop if it is in the process of rotating, and the rotational movement of the slewing unit 32 is prohibited if it is not in the process of rotating. In other words, in this embodiment, the work machine 3 comprises a traveling unit 31 and a slewing unit 32 that can rotate relative to the traveling unit 31. The restriction processing restricts at least the rotational movement of the slewing unit 32. This allows the rotating unit 32 to rotate and avoid contact between the machine 30 and the object Ob1 when the object Ob1 is located in the monitoring area A1, which is a blind spot for the user U1 (operator).

[0060] The switching processing unit 13 switches the deterrence processing unit 12 between enabled and disabled. In other words, the switching processing unit 13 switches the functions related to deterrence processing between enabled and disabled. In short, the deterrence processing unit 12 is not always enabled; it can be switched between enabled and disabled. If the deterrence processing unit 12 is enabled, the deterrence processing by the deterrence processing unit 12 is executed when the detected object Ob1 is present in the monitoring area A1. On the other hand, if the deterrence processing unit 12 is disabled, the deterrence processing by the deterrence processing unit 12 is not executed even if the detected object Ob1 is present in the monitoring area A1.

[0061] In this embodiment, as an example, the switching between enabling and disabling the function related to the restraining process (restraining processing unit 12) is performed by the user U1 (operator) operating the display device 2. That is, when user U1 operates the operation unit 22 of the display device 2 to enable the function related to the restraining process, the switching processing unit 13 receives this operation and enables the function related to the restraining process. On the other hand, when user U1 operates the operation unit 22 of the display device 2 to disable the function related to the restraining process, the switching processing unit 13 receives this operation and disables the function related to the restraining process.

[0062] Furthermore, in this embodiment, the restraint processing performed by the restraint processing unit 12 includes sound output processing performed by the sound output processing unit 121 and restriction processing performed by the restriction processing unit 122. Thus, the restraint processing includes a plurality of specific processing (sound output processing and restriction processing, etc.) for restraining the operation of the work machine 3. Here, the restraint processing can be individually switched between enabled and disabled for each specific processing. In other words, the switching processing unit 13 can individually switch between enabled and disabled for the sound output processing unit 121 and the restriction processing unit 122 in the restraint processing unit 12. For example, it is possible to enable the sound output processing unit 121 and disable the restriction processing unit 122, or to disable the sound output processing unit 121 and enable the restriction processing unit 122. This makes it possible to enable only the necessary specific processing depending on the situation, improving the flexibility of the restraint processing.

[0063] The display processing unit 11 performs display processing to display at least the display screen Dp1 on the display device 2. Specifically, the display processing unit 11 generates the display screen Dp1 based on data acquired by the image acquisition unit 14, and controls the display device 2 to display the display screen Dp1 on the display unit 23 of the display device 2. Furthermore, the display processing unit 11 operates in accordance with the operation received by the operation unit 22 of the display device 2. For example, the display processing unit 11 displays the captured images Im11, Im12, Im13 (see Figure 5) captured by the left camera 341, the right camera 342, and the rear camera 343 on the display screen Dp1. In other words, the display processing unit 11 and the image acquisition unit 14 display images of the monitoring area A1 (each sub-area A11, A12, A13) around the work machine 3 on the display device 2.

[0064] Here, the display processing unit 11 displays, in addition to the captured images Im11, Im12, and Im13, a detection object Im10 (see Figure 5) on the display screen Dp1, which represents the detection result of the detection unit 15 regarding the presence or absence of the object Ob1 in the monitoring area A1. The detection object Im10 is displayed in the same position on the display screen Dp1 regardless of the detection result of the detection unit 15. Therefore, the display position of the detection object Im10 is the same whether or not the object Ob1 is present in the monitoring area A1. In other words, the detection object Im10 is always displayed in a fixed position on the display screen Dp1, regardless of the detection result of the detection unit 15. On the other hand, the display mode of the detection object Im10 changes according to the detection result of the detection unit 15. Therefore, the display mode of the detection object Im10 is different depending on whether or not the object Ob1 is present in the monitoring area A1.

[0065] In other words, the display processing unit 11 displays a display screen Dp1, which includes the captured images Im11, Im12, Im13 and the detected object Im10, on the display device 2. The display processing unit 11 displays the detected object Im10 at the same position on the display screen Dp1 regardless of the detection result of the detection unit 15 that detects the object Ob1 in the monitoring area A1, and changes the display mode of the detected object Im10 according to the detection result. In this disclosure, "objects" such as the detected object Im10 include marks, images, figures, photographs, text or videos, etc., displayed on the display screen Dp1, or combinations thereof. In this disclosure, "display mode" includes the display color, size (including size, line thickness, etc.), shape, operating state (including animation, rotation / stillness, etc.), or display pattern (including blinking patterns, etc.), etc., of the object, or combinations thereof. In short, the display processing unit 11 changes the display mode, such as the display color of the detected object Im10, according to the detection result of the detection unit 15.

[0066] Furthermore, in this embodiment, the display processing unit 11 can display a display screen Dp1 on the display device 2, which includes restraint status information I1 (see Figure 5) indicating whether the restraint function is enabled or disabled. In other words, the enable / disable status of the restraint function is switched by the switching processing unit 13, and the current status of the restraint function (enabled / disabled) is displayed on the display screen Dp1 as restraint status information I1. This allows the user U1 (operator) to visually confirm whether the restraint function is enabled or disabled, and to operate the work machine 3 knowing whether the restraint function is enabled or disabled.

[0067] By the way, the detection unit 15 is not an essential component of the control system 1. For example, the control system 1 may be configured to acquire detection results from an external detection unit, and the restraining processing unit 12 may execute restraining processing based on those detection results.

[0068] [3] Control method for working machinery The following describes an example of a control method (hereinafter simply referred to as "control method") for the work machine 3, which is mainly performed by the control system 1, with reference to Figures 5 to 8.

[0069] The control method according to this embodiment is executed by a control system 1, which mainly consists of a computer system; in other words, it is embodied in a control program for a work machine (hereinafter simply referred to as the "control program"). That is, the control program according to this embodiment is a computer program that causes one or more processors to execute each process related to the control method. Such a control program may be executed in cooperation with, for example, the control system 1 and the display device 2.

[0070] Here, the control system 1 executes the following various processes related to the control method when a specific pre-set start operation is performed to execute the control program. The start operation is, for example, the operation to start the engine of the work machine 3. On the other hand, the control system 1 terminates the following various processes related to the control method when a specific pre-set end operation is performed. The end operation is, for example, the operation to stop the engine of the work machine 3.

[0071] [3.1] Display screen First, the configuration of the display screen Dp1 displayed on the display unit 23 of the display device 2 by the control method according to this embodiment will be described. In the drawings showing the display screen Dp1 displayed on the display unit 23 of the display device 2, such as Figure 5, the dashed lines, leader lines, and reference numerals representing areas are all there for illustrative purposes only and are not actually displayed on the display device 2.

[0072] The display screen Dp1 shown in Figure 5 is the home screen that is initially displayed depending on the control method. The home screen is the basic display screen Dp1 that will be displayed first on the display device 2 while the work machine 3 is in operation. From the home screen, the display screen Dp1 can transition to various display screens Dp1, including the menu screen, crane screen, mode screen, and PTO screen, according to operations performed on the control unit 22.

[0073] As shown in Figure 5, the display screen Dp1 includes the first region R1, the second region R2, the third region R3, the fourth region R4, the fifth region R5, the sixth region R6, the seventh region R7, the eighth region R8, the ninth region R9, and the tenth region R10. In the control method according to this embodiment, as an example, the second region R2, which occupies most of the display screen Dp1, displays the captured images Im11, Im12, Im13 of the monitoring area A1 (each sub-area A11, A12, A13), the detected object Im10, and the suppression status information I1, etc. The area in the second region R2 where the detected object Im10 is displayed is designated as the eleventh region R11.

[0074] Specifically, the display screen Dp1 is divided into four vertical (up and down) sections. The top three sections are then further divided into three horizontal (left and right) sections. This results in a total of 10 sections on the display screen Dp1. The second section from the top consists of three sections from left to right: Section 1 R1, Section 2 R2, and Section 3 R3. The bottom section is Section 4 R4. The third section from the top consists of three sections from left to right: Section 5 R5, Section 6 R6, and Section 7 R7. The top section consists of three sections from left to right: Section 8 R8, Section 9 R9, and Section 10 R10. In terms of vertical size, the second section from the top (Section 1 R1, Section 2 R2, and Section 3 R3) is the largest of the four vertically divided sections. In terms of horizontal size, the middle region (region 2 R2, region 6 R6, and region 9 R9) is the largest among the three regions divided horizontally.

[0075] However, the arrangement and size of these regions are merely examples and can be changed as needed. Furthermore, it is not necessary for each region to be clearly separated by a boundary line. For example, in the example in Figure 5, the second region R2 and the third region R3 are clearly separated by a boundary line, while there is no boundary line between the first region R1 and the second region R2. Of course, the first region R1 and the second region R2 may also be clearly separated by a boundary line.

[0076] The first region R1 is a rectangular area that is elongated vertically. In the first region R1, for example, remaining fuel information G1 regarding the remaining amount of engine fuel (e.g., diesel fuel) is displayed. The display processing unit 11 generates the remaining fuel information G1 on the display screen Dp1 based on the output of the fuel level sensor (sensor signal), etc.

[0077] The second region R2 is a horizontally elongated rectangular area. The second region R2 displays the captured images Im11, Im12, and Im13 of the monitoring area A1, the detected object Im10, and the deterrent status information I1, etc. Captured image Im11 is an image of the small area A11 to the left of the driver unit 321, captured by the left camera 341, and captured image Im12 is an image of the small area A12 to the right of the driver unit 321, captured by the right camera 342. Captured image Im13 is an image of the small area A13 behind the driver unit 321, captured by the rear camera 343. The display processing unit 11 displays the captured images Im11, Im12, and Im13 acquired by the image acquisition unit 14 in real time.

[0078] The detection object Im10 is displayed within the 11th region R11, which is set in the center of the second region R2. In this embodiment, as an example, the detection object Im10 is an icon that mimics the aircraft 30 as seen from above. Therefore, the top of the detection object Im10 corresponds to the front of the aircraft 30 in real space, the bottom of the detection object Im10 corresponds to the rear of the aircraft 30 in real space, the right side of the detection object Im10 corresponds to the right side of the aircraft 30 in real space, and the left side of the detection object Im10 corresponds to the left side of the aircraft 30 in real space. Specifically, the lower part of the detection object Im10 is configured in an arc shape that mimics the rear part (counterweight) of the swivel section 32, and the upper part of the detection object Im10 is configured in a shape that mimics the front part (work section 33) of the swivel section 32. In this embodiment, the work unit 33 is positioned offset to the right with respect to the center of the left-right direction D3 of the swivel unit 32, and the driving unit 321 is positioned offset to the left with respect to the center of the left-right direction D3 of the swivel unit 32. Therefore, in the detection object Im10, a figure resembling the work unit 33 is located in the upper right part, and a figure resembling the driving unit 321 is located in the left part.

[0079] In this embodiment, the captured images Im11, Im12, and Im13 have a rectangular cutout in the center of the second region R2. Therefore, the detection object Im10 displayed in the center of the second region R2 (11th region R11) is positioned to fit into the cutout portion of the captured images Im11, Im12, and Im13. As a result, the detection object Im10 is positioned adjacent to the captured images Im11, Im12, and Im13 on the display screen Dp1. Consequently, the operator can easily intuitively grasp the distance between the machine 30 and the object Ob1 by looking at the display screen Dp1.

[0080] Furthermore, in this embodiment, the captured images Im11, Im12, and Im13 are displayed so as to correspond to the positional relationship when the machine 30 is viewed from above, with captured image Im11 positioned to the upper left of the detected object Im10, captured image Im12 positioned to the upper right of the detected object Im10, and captured image Im13 positioned below the detected object Im10. In other words, the captured image Im11 of the small area A11 to the left of the machine 30 is positioned to the left of the detected object Im10, the captured image Im12 of the small area A12 to the right of the machine 30 is positioned to the right of the detected object Im10, and the captured image Im13 of the small area A13 behind the machine 30 is positioned below the detected object Im10. Thus, the control method according to this embodiment reflects the positional relationship of the monitoring area A1 to the work machine 3 in real space in relation to the detected object Im10 in the positional relationship of the captured images Im11, Im12, and Im13 to the detected object Im10 on the display screen Dp1. Therefore, operators can easily intuitively understand which part of monitoring area A1 each image (Im11, Im12, Im13) was captured by looking at the display screen Dp1.

[0081] Furthermore, in this embodiment, the detected object Im10 is displayed based on the orientation of the swivel unit 32 on which the driver unit 321 is mounted, rather than the orientation of the travel unit 31. In other words, as the swivel unit 32 rotates relative to the travel unit 31, the orientation in real space corresponding to the area above the detected object Im10 changes. As a result, the area above the detected object Im10 always corresponds to the area in front of the swivel unit 32, which is in front of the operator riding in the driver unit 321. That is, the work machine 3 comprises a travel unit 31 and a swivel unit 32 that can rotate relative to the travel unit 31. In this embodiment, the control method reflects the positional relationship of the captured images Im11, Im12, and Im13 with respect to the detected object Im10 on the display screen Dp1, as well as the positional relationship of the monitoring area A1 with respect to the swivel unit 32 in real space. Therefore, the operator can easily intuitively understand which part of the monitoring area A1 was captured by the captured images Im11, Im12, and Im13 by looking at the display screen Dp1.

[0082] The deterrence status information I1 is displayed in the lower right of the second area R2. As described above, the deterrence status information I1 is information that indicates whether the function related to the deterrence process is enabled or disabled, that is, the status (enabled / disabled) of the function related to the deterrence process selected by the switching processing unit 13. In this embodiment, the deterrence status information I1 includes an image (icon). The deterrence status information I1 indicates the status (enabled / disabled) of the function related to the deterrence process by display manner, such as the display color or size of the image. The display processing unit 11 determines the status (enabled / disabled) of the function related to the deterrence process based on the operating state of the switching processing unit 13. Then, the display processing unit 11 determines the display manner of the image in the deterrence status information I1 according to the status (enabled / disabled) of the function related to the deterrence process.

[0083] Here, the restraint process includes multiple specific processes (such as sound output processing and restriction processing), and each specific process can be individually enabled or disabled. The restraint status information I1 is displayed individually for each specific process. In the example in Figure 5, the restraint status information I1 includes a first restraint status information I11 and a second restraint status information I12, each corresponding to a specific process. The first restraint status information I11 indicates whether the function related to sound output processing is enabled or disabled, and the second restraint status information I12 indicates whether the function related to restriction processing is enabled or disabled. The design (pattern) of each image in the first and second restraint status information I11 indicates whether it corresponds to sound output processing or restriction processing.

[0084] In this embodiment, as an example, if the function related to the deterrence process is enabled, the deterrence status information I1 is displayed as active, and if the function related to the deterrence process is disabled, the deterrence status information I1 is displayed as inactive. In other words, the deterrence status information I1 that is displayed as active indicates that the function related to the deterrence process is enabled, and the deterrence status information I1 that is displayed as inactive indicates that the function related to the deterrence process is disabled. The deterrence status information I1 that is displayed as active is displayed in a high-contrast display color such as green, for example, while the deterrence status information I1 that is displayed as inactive is displayed in a low-contrast display color such as gray.

[0085] The third region R3 is a vertically elongated rectangular area. Icons (Im1) corresponding to the operating status of each part of the work machine 3 are displayed in the third region R3. Multiple icons (Im1) can be displayed in the third region R3, and the design (pattern) of each icon (Im1) indicates which state it represents, for example, the battery, seat belt, coolant temperature, hydraulic oil temperature, etc. Here, each icon (Im1) indicates its operating status by its display characteristics, such as its display color or size. The display processing unit 11 determines the status of each part of the work machine 3 using the outputs of various sensors (including a coolant temperature sensor and a hydraulic oil temperature sensor) that detect the operating status of each part of the work machine 3. If an abnormal value is detected in any part, the display processing unit 11 displays a warning by changing the display characteristics, such as the display color, of the icon (Im1) for that part.

[0086] The fourth area R4 is a band-shaped area that extends across the entire width of the display screen Dp1. The fourth area R4 displays various items for operating the display screen Dp1. As an example in Figure 5, the fourth area R4 has six items: "Menu," "Crane," "Mode," "Camera," "PTO," and "Switch," arranged from left to right in that order. These six items are associated with the six push-button switches 221 to 226 on the operation unit 22 located directly below them. For example, the push-button switch 221 is associated with the "Menu" item, and the push-button switch 222 is associated with the "Crane" item. Therefore, for example, when the push-button switch 224 corresponding to the "Camera" item is operated by user U1 (see Figure 4), the "Camera" item is operated (selected).

[0087] Furthermore, in this embodiment, in order to respond to operations such as the operation dial (or cursor keys) of the operation unit 22, one of the items is highlighted in the fourth area R4. In the example in Figure 5, the "Menu" item is highlighted, and the highlighted item is switched by operations such as the operation dial (or cursor keys). User U1 can select the desired item by operating the OK button while the desired item is highlighted. For example, if the OK button is operated while the highlight is moved to the "Camera" item, the "Camera" item will be operated (selected). Also, if the operation unit 22 includes a touch panel, user U1 can select the desired item by touching the desired item on the display screen Dp1.

[0088] Area 5 R5 displays warning icons indicating that abnormal values ​​have been detected by various sensors (including coolant temperature sensor and hydraulic oil temperature sensor). Area 6 R6 displays information about the working unit 33 currently operating in the working machine 3, for example. Area 7 R7 displays information about the operating status of the working machine 3, for example, the engine speed. Area 8 R8 displays the current time, for example. Area 9 R9 displays information indicating the item to which the currently displayed screen Dp1 belongs, for example. Area 10 R10 displays information about the operating time (hour meter) of the working machine 3, for example.

[0089] [3.2] Details Next, the control method according to this embodiment will be described in detail.

[0090] The control method according to this embodiment includes acquiring captured images Im11, Im12, and Im13 of the monitoring area A1 surrounding the work machine 3 (image acquisition process), and displaying a display screen Dp1 including the captured images Im11, Im12, and Im13 and the detected object Im10 on the display device 2 (display process). The control method further includes, in the display process, displaying the detected object Im10 at the same position on the display screen Dp1 regardless of the detection result of the detection unit 15 that detects the object to be detected Ob1 in the monitoring area A1, while changing the display manner of the detected object Im10 according to the detection result.

[0091] In short, by displaying the captured images Im11, Im12, Im13 and the detected object Im10 on the display screen Dp1 on the display device 2, the detection result of the object Ob1 in the monitoring area A1 can be represented by the display mode of the detected object Im10. Moreover, although the display mode of the detected object Im10 changes, its display position remains fixed on the display screen Dp1 regardless of the detection result of the detection unit 15, thus providing good visibility. Therefore, the operator (user U1) can grasp the detection result of the object Ob1 in the monitoring area A1 simply by looking at the detected object Im10 displayed at a fixed position on the display screen Dp1. Accordingly, according to the control method of this embodiment, the operator can intuitively grasp the presence of an object Ob1 (e.g., a person) in the monitoring area A1.

[0092] Figure 6 shows an example of the display mode of the detected object Im10. In Figure 6, only the area around the 11th region R11 on the display screen Dp1 is shown, and other parts are omitted. The display mode of the detected object Im10 changes according to the detection result of the detection unit 15, so there are at least two states: when there is no detected object Ob1 in the monitoring area A1 ("non-detection") and when there is a detected object Ob1 in the monitoring area A1 ("detection"). Furthermore, in this embodiment, the display mode of the detected object Im10 also changes depending on the distance from the work machine 3 to the detected object Ob1. Therefore, even when there is a detected object Ob1 in the monitoring area A1 ("detection"), the display mode of the detected object Im10 differs further depending on whether the distance from the work machine 3 to the detected object Ob1 is far (detection_distance: far) or close (detection_distance: close). Therefore, as shown in Figure 6, the detected object Im10 has three display modes: "Not detected", "Detection distance: far", and "Detection distance: close". Here, it is assumed that the detection unit 15 determines "Detection distance: close" if the distance from the work machine 3 to the object Ob1 is within a threshold distance (e.g., 2m), and determines "Detection distance: far" if the distance from the work machine 3 to the object Ob1 exceeds the threshold distance.

[0093] In this embodiment, as an example, the display mode of the detected object Im10, which changes according to the detection result of the detection unit 15, includes the display color of the detected object Im10. Therefore, the display color of the detected object Im10 on the display screen Dp1 changes according to the detection result of the detection unit 15. For example, the closer the detected object Ob1 is to the machine 30, the more conspicuous the display color of the detected object Im10 becomes. For example, as the detected object Ob1 approaches the machine 30, the display color of the detected object Im10 changes from yellow to red. Specifically, as shown in Figure 6, the display color of the detected object Im10 is the same as the background color (black or white, etc.) when "not detected", yellow when "detection distance: far", and red when "detection distance: close". Therefore, the operator can easily intuitively grasp that a detected object Ob1 (e.g., a person) is present in the monitoring area A1 by the display color of the detected object Im10 displayed at a fixed position (11th area R11) on the display screen Dp1.

[0094] In this case, multiple detectable objects Ob1 may exist simultaneously in monitoring area A1. In this case, the display mode of the detected object Im10 is determined according to the distance from the machine body 30 of the work machine 3 to the closest detectable object Ob1. For example, if a detectable object Ob1 located within a threshold distance from the work machine 3 and a detectable object Ob1 located beyond the threshold distance from the work machine 3 are detected simultaneously, the display color of the detected object Im10 will be red, based on the detectable object Ob1 located within the threshold distance from the work machine 3. In short, the display mode of the detected object Im10 changes according to the distance from the work machine 3 to the detectable object Ob1. The control method according to this embodiment determines the display mode of the detected object Im10 according to the distance from the work machine 3 to the closest detectable object Ob1 when multiple detectable objects Ob1 exist in monitoring area A1. As a result, the display mode of the detected object Im10 is determined according to the detection result of the most urgent detectable object Ob1, making it easier for the operator to intuitively grasp the detection result.

[0095] By the way, in this embodiment, since the object Ob1 to be detected in the monitoring area A1 is the object to be detected by the detection unit 15, if the object Ob1 to be detected is present in the monitoring area A1, it will naturally be captured in the images Im11, Im12, and Im13 of the monitoring area A1. In other words, as shown in Figure 7, when the object Ob1 to be detected is captured in the images Im11, Im12, and Im13 on the display screen Dp1, the display mode of the detection object Im10 will indicate that the object Ob1 to be detected is present in the monitoring area A1. In Figure 7, only the second region R2 of the display screen Dp1 is shown, and the illustration of regions other than the second region R2 is omitted.

[0096] Thus, when the detection result indicates the presence of the object Ob1 in the monitoring area A1, at least a portion of the object will be captured in the images Im11, Im12, and Im13 on the display screen Dp1. This allows the operator to confirm the object Ob1 on the images Im11, Im12, and Im13. In other words, the operator (user U1) can check the situation of the sides and rear of the work machine 3, which are often blind spots from the operating unit 321, on the display screen Dp1 shown on the display device 2. Therefore, compared to a configuration where only the detected object Im10 is displayed, when the object Ob1 is present in the monitoring area A1, it becomes easier to grasp the situation of the object Ob1 in detail on the display screen Dp1.

[0097] Furthermore, in this embodiment, as shown in Figure 7, the display screen Dp1 includes auxiliary objects I2 and I3 that represent the detection results of the detection unit 15. Auxiliary object I2 is a band-shaped (frame-shaped) image that highlights the captured image Im11, Im12, and Im13 that contains the object to be detected Ob1. Auxiliary object I3 is an image that indicates the direction in which the object to be detected Ob1 is located as seen from the driver unit 321. In the example in Figure 7, it is assumed that the object to be detected Ob1 (here, "person") is located in a small area A11 to the left of the driver unit 321, which is captured by the left camera 341. Therefore, among the captured images Im11, Im12, and Im13, the captured image Im11 is highlighted by auxiliary object I2, and auxiliary object I3 is displayed below the captured image Im11, indicating that the object to be detected Ob1 is located to the left of the driver unit 321.

[0098] The display modes of auxiliary objects I2 and I3 are also preferably changed according to the position of the object to be detected Ob1 in the monitoring area A1, similar to the detection object Im10. For example, the display modes of auxiliary objects I2 and I3, such as display color, size (including size, line thickness, etc.), shape, operating state (including animation, rotation / static, etc.), or display pattern (including blinking pattern, etc.), are changed according to the position of the object to be detected Ob1 in the monitoring area A1. Specifically, the closer the object to be detected Ob1 is to the machine 30, the more conspicuous the display mode of auxiliary objects I2 and I3 will be changed to a display color. For example, as the object to be detected Ob1 approaches the machine 30, the display color of auxiliary objects I2 and I3 will change from yellow to red.

[0099] In this embodiment, the display colors of auxiliary objects I2 and I3 are the same as those of detection object Im10. That is, the display colors of auxiliary objects I2 and I3 are yellow when "detection distance: far" and red when "detection distance: close". However, unlike detection object Im10, auxiliary objects I2 and I3 are displayed only when the object to be detected Ob1 is present in monitoring area A1, and are not displayed when the object to be detected Ob1 is not present in monitoring area A1.

[0100] Thus, the control method according to this embodiment further includes displaying auxiliary objects I2 and I3, separate from the detection object Im10, on the display screen Dp1 when the detection result indicates the presence of the detection target object Ob1 in the monitoring area A1. In other words, when the detection target object Ob1 is present in the monitoring area A1, the display screen Dp1 displays not only the captured images Im11, Im12, Im13 and the detection object Im10 of the monitoring area A1, but also auxiliary objects I2 and I3, separate from the detection object Im10. Therefore, the operator can more easily confirm the presence or absence of the detection target object Ob1 in the monitoring area A1 by looking at the display screen Dp1.

[0101] Furthermore, auxiliary objects I2 and I3 indicate the orientation of the object Ob1 as seen from at least the work machine 3. For example, captured images Im11, Im12, and Im13 include images of multiple sub-areas A11, A12, and A13 contained within the monitoring area A1. Auxiliary object I2 then highlights the image of the sub-area A11, A12, and A13 in which the object Ob1 is located on the display screen Dp1. In other words, auxiliary object I2 highlights the captured image Im11, Im12, and Im13 that contains the object Ob1. This makes it easier for the operator to determine which of the multiple sub-areas A11, A12, and A13 contains the object Ob1 if it is present within the monitoring area A1.

[0102] Furthermore, the display mode of the detection object Im10 includes the display color of the detection object Im10, and the display colors of auxiliary objects I2 and I3 are the same as the display color of the detection object Im10 when the detection result indicates the presence of the detection target object Ob1 in monitoring area A1. In other words, when the detection target object Ob1 is present in monitoring area A1, auxiliary objects I2 and I3 are displayed in the same color as the detection object Im10. Therefore, auxiliary objects I2 and I3 and the detection object Im10 are easier for the operator to see.

[0103] Here, the detection object Im10 and auxiliary objects I2 and I3 are displayed not only when the functions related to the suppression process are enabled, but also when the functions related to the suppression process are disabled. In other words, even if the suppression processing unit 12 is disabled and both the first suppression status information I11 and the second suppression status information I12 are grayed out and displayed as inactive, if the object to be detected Ob1 is present in the monitoring area A1, the detection object Im10 and auxiliary objects I2 and I3 will be displayed.

[0104] [3.3] Overall Processing Next, the overall flow of the control method will be explained with reference to Figure 8. Figure 8 is a flowchart showing an example of the control method process.

[0105] As shown in Figure 8, the image acquisition unit 14 of the control system 1 acquires captured images Im11, Im12, and Im13 from the left camera 341, the right camera 342, and the rear camera 343 (S1). Then, the display processing unit 11 of the control system 1 displays a display screen Dp1, which includes the captured images Im11, Im12, Im13 and the detected object Im10, on the display device 2 (S2).

[0106] Here, the display processing unit 11 of the control system 1 determines whether or not the object Ob1 is present in the monitoring area A1 based on the detection result of the detection unit 15 (S3). If the object Ob1 is present in the monitoring area A1 (S3: Yes), the display processing unit 11 changes the display color (display mode) of the detected object Im10 on the display screen Dp1 to the display color at the time of detection (yellow or red) (S4). Furthermore, the display processing unit 11 displays auxiliary objects I2 and I3 on ​​the display screen Dp1 (S5). On the other hand, if the object to be detected, Ob1, is not present in the monitoring area A1 (S3: No), the display processing unit 11 changes the display color (display mode) of the detected object Im10 on the display screen Dp1 to the display color (background color) when no object is detected (S6). Furthermore, the display processing unit 11 hides the auxiliary objects I2 and I3 on ​​the display screen Dp1 (S7).

[0107] The control system 1 repeatedly executes the processes in steps S1 to S7 described above. As a result, the display screen Dp1 corresponding to the detection result of the detection unit 15 is displayed on the display device 2. However, the flowchart shown in Figure 8 is merely an example, and processes may be added or omitted as appropriate, or the order of processes may be changed as appropriate.

[0108] [4] Modified form The following lists some modifications of Embodiment 1. The modifications described below can be combined and applied as appropriate.

[0109] The control system 1 in this disclosure includes a computer system. The computer system mainly consists of one or more processors and one or more memories as hardware. The functions of the control system 1 in this disclosure are realized by the execution of a program recorded in the memory of the computer system by the processor. The program may be pre-recorded in the memory of the computer system, provided via a telecommunications line, or provided on a non-temporary recording medium such as a memory card, optical disk, or hard disk drive that can be read by the computer system. Furthermore, some or all of the functional parts included in the control system 1 may be composed of electronic circuits.

[0110] Furthermore, it is not essential for control system 1 to have at least some of its functions integrated into a single housing; the components of control system 1 may be distributed across multiple housings. Conversely, functions that are distributed across multiple devices (e.g., control system 1 and display device 2) in Embodiment 1 may be integrated into a single housing. Moreover, at least some of the functions of control system 1 may be implemented by the cloud (cloud computing) or the like.

[0111] Furthermore, the power source for the work machine 3 is not limited to a diesel engine; for example, it may be an engine other than a diesel engine, or a motor (electric motor), or a hybrid power source including an engine and a motor (electric motor).

[0112] Furthermore, the display device 2 is not limited to a dedicated device, but may be a general-purpose terminal such as a laptop computer, tablet terminal, or smartphone. Moreover, the display unit 23 is not limited to a configuration that directly displays the display screen, such as a liquid crystal display or an organic EL display, but may also be configured to display the display screen by projection, such as a projector.

[0113] Furthermore, the input method for information to the operation unit 22 may be other than push-button switches, touch panels, and operation dials. For example, the operation unit 22 may use methods such as a keyboard, a pointing device such as a mouse, voice input, gesture input, or input of operation signals from another terminal.

[0114] Furthermore, the restriction processing performed by the restriction processing unit 122 is not limited to processing that prohibits (disables) the operation of the work machine 3 (such as turning). The restriction processing may be, for example, processing that reduces the speed of the operation of the work machine 3 (such as turning), processing that narrows the operating range (such as turning angle) of the work machine 3, or processing that limits the permissible area of ​​operation of the work machine 3.

[0115] Furthermore, the restraint process does not necessarily have to include multiple specific processes (such as sound output processing and limiting processing) to restrain the operation of the work machine 3. Moreover, even if the restraint process includes multiple specific processes, it is not necessary to be able to switch each specific process on and off individually; it may be possible to switch multiple specific processes on and off at once. Also, it is not necessary for the restraint status information I1 to indicate whether each specific process is enabled or disabled. Furthermore, it is not necessary to display in the restraint status information I1 that the function related to the restraint process is disabled when it is disabled; for example, if the function related to the restraint process is disabled, the restraint status information I1 may be hidden.

[0116] Furthermore, the functions related to deterrence processing by the deterrence processing unit 12 are not essential, and the deterrence processing unit 12 can be omitted as appropriate. In addition, displaying the deterrence status information I1 is not essential, and the deterrence status information I1 may be hidden regardless of whether the functions related to deterrence processing are enabled or disabled.

[0117] Furthermore, the sensors for detecting the object Ob1 in the monitoring area A1 surrounding the work machine 3 are not limited to the left camera 341, the right camera 342, and the rear camera 343, but may include one, two, or four or more cameras (image sensors). In addition, for example, the object Ob1 in the monitoring area A1 may be detected by a camera capable of capturing images in all directions as seen from the work machine 3, such as a 360-degree camera. Also, the sensors for detecting the object Ob1 in the monitoring area A1 may include, in addition to or instead of cameras, sensors such as a human presence sensor, sonar sensor, radar, or LiDAR (Light Detection and Ranging). Here, the sensor for detecting the object Ob1 in the monitoring area A1 may be a 3D sensor that measures the distance to the object Ob1 using the TOF (Time Of Flight) method, which measures the distance to the distance point based on the round-trip time it takes for light or sound to reach the distance point and return.

[0118] Furthermore, the object to be detected Ob1 may include, in addition to or instead of "people," moving objects such as vehicles (including other work machinery), structures such as walls and pillars, plants, animals, steps, ditches, or other obstacles.

[0119] Furthermore, when the detection result indicates the presence of the detected object Ob1 in monitoring area A1, it is not essential that at least a portion of the detected object Ob1 be captured in the captured images Im11, Im12, and Im13 on the display screen Dp1. Also, it is not essential that the detected object Im10 is placed adjacent to the captured images Im11, Im12, and Im13 on the display screen Dp1. For example, the detected object Im10 may be superimposed on the captured images Im11, Im12, and Im13 on the display screen Dp1.

[0120] Furthermore, it is not essential to reflect the positional relationship of the monitoring area A1 relative to the work machine 3 in real space in the positional relationship of the captured images Im11, Im12, and Im13 relative to the detected object Im10 on the display screen Dp1. Additionally, it is not essential to reflect the positional relationship of the monitoring area A1 relative to the rotating unit 32 in real space in the positional relationship of the captured images Im11, Im12, and Im13 relative to the detected object Im10 on the display screen Dp1.

[0121] Furthermore, when the detection result indicates the presence of the object Ob1 in monitoring area A1, it is not mandatory to display auxiliary objects I2 and I3, which are separate from the detection object Im10, on the display screen Dp1. Moreover, it is not mandatory for auxiliary objects I2 and I3 to indicate the direction in which the object Ob1 is located, at least as seen from the work machine 3. Furthermore, it is not mandatory for the display color of auxiliary objects I2 and I3 to be the same as the display color of the detection object Im10 when the detection result indicates the presence of the object Ob1 in monitoring area A1.

[0122] Furthermore, when multiple detection targets Ob1 exist in monitoring area A1, it is not essential to determine the display mode of the detection object Im10 according to the distance from the work machine 3 to the nearest detection target Ob1. Moreover, it is not essential that the display mode of the detection object Im10 changes according to the distance from the work machine 3 to the detection target Ob1.

[0123] (Embodiment 2) As shown in Figures 9 and 10, the work machine 3 according to this embodiment differs from the work machine 3 according to Embodiment 1 in the display content of the second area R2 of the display screen Dp1. Hereinafter, components similar to those in Embodiment 1 will be denoted by common reference numerals and their descriptions will be omitted as appropriate. In Figures 9 and 10, only the second area R2 of the display screen Dp1 is shown, and the illustration of areas other than the second area R2 is omitted.

[0124] In the example shown in Figure 9, the second region R2 displays two images from the captured images Im11, Im12, and Im13 of the monitoring area A1 (see Figure 7): Im12 captured by the right camera 342 and Im13 captured by the rear camera 343. In this example, the captured images Im12 and Im13 are arranged side by side in the second region R2, and an icon I4 is displayed below these captured images. Icon I4 schematically represents the positional relationship of the imaging ranges (small areas A12, A13) of the right camera 342 and the rear camera 343 as seen from the aircraft 30.

[0125] In the example shown in Figure 9, the detection object Im10 is located in the center of the captured images Im11 and Im12. However, the detection object Im10 may be located, for example, above the captured image Im11 or the captured image Im12, in other locations.

[0126] In the example in Figure 10, the overhead image Im100 of the monitoring area A1 is displayed in the second region R2. The overhead image Im100 is a type of captured image generated by transforming and combining the captured images Im11, Im12, and Im13 from the left camera 341, the right camera 342, and the rear camera 343. In such an overhead image Im100, the auxiliary object I2 is a band-shaped (frame-shaped) image that highlights the small area where the detected object Ob1 exists. The overhead image Im100 may also be an image in which the captured images are combined into an animation.

[0127] In the example shown in Figure 10, the detection object Im10 is positioned at the apex of the overhead image Im100. That is, the overhead image Im100 is a fan-shaped image with an arc-shaped outline on the side furthest from the machine 30, and the detection object Im10 is positioned at a location corresponding to the apex of this fan shape. In this disclosure, "arc-shaped" is not limited to arcs that constitute part of a perfect circle, but includes all curved lines that are convex outward. Therefore, even if the overhead image Im100, as shown in Figure 10, consists of curved lines whose outline is convex outward as a whole, it can still be called a fan-shaped image with an arc-shaped outline. In other words, on the display screen Dp1, the captured image (overhead image Im100) is displayed with an arc-shaped outline for at least the monitoring areas A1 to the left, right, and rear of the work machine 3, and the detection object Im10 is positioned at a location corresponding to the apex of the captured image (overhead image Im100). This makes it easier for the operator to intuitively understand the position of the detected object Ob1. However, this is not limited to this example; the detected object Im10 may be placed at a position other than the vertex angle of the overhead image Im100, for example, below the overhead image Im100.

[0128] The dual-screen display (see Figure 9) and the overhead image Im100 (see Figure 10) illustrated in this embodiment may be switched at will by, for example, the operation unit 22 of the display device 2. In this case, it is preferable that the user U1 can select any display format, including the triple-screen display (see Figure 7) described in Embodiment 1.

[0129] The configuration according to Embodiment 2 (including modified versions) can be adopted in appropriate combination with the various configurations (including modified versions) described in Embodiment 1.

[0130] [Notes on the invention] The following is an overview of the invention extracted from the above-described embodiments. Note that each configuration and processing function described below can be selected and combined as desired.

[0131] <Note 1> Acquiring images of the monitoring area around the work machine, Displaying the captured image and the detected object on a display device, The system displays the detected object at the same position on the display screen regardless of the detection result of the detection unit that detects the object to be detected in the monitoring area, while changing the display mode of the detected object according to the detection result. A method for controlling industrial machinery.

[0132] <Note 2> When the detection result indicates the presence of the object to be detected in the monitoring area, at least a portion of the object to be detected is captured in the image on the display screen. Control method for the work machine described in Appendix 1.

[0133] <Note 3> The detection object is positioned adjacent to the captured image on the display screen. A control method for the work machine described in Appendix 1 or 2.

[0134] <Note 4> The positional relationship of the captured image with respect to the detected object on the display screen reflects the positional relationship of the monitoring area with respect to the work machine in real space. A control method for the work machine described in any of the appendices 1 to 3.

[0135] <Note 5> The aforementioned work machine comprises a traveling section and a swivel section that can rotate relative to the traveling section, The positional relationship of the captured image with respect to the detected object on the display screen reflects the positional relationship of the monitoring area with respect to the rotating part in real space. A control method for the work machine described in any of the appendices 1 to 4.

[0136] <Note 6> The system further includes displaying an auxiliary object on the display screen, separate from the detected object, when the detection result indicates the presence of the object to be detected in the monitoring area. A control method for the work machine described in any of the appendices 1 to 5.

[0137] <Note 7> The auxiliary object indicates at least the direction in which the object to be detected is located as seen from the work machine. Control method for the work machine described in Appendix 6.

[0138] <Note 8> The display mode of the detection object includes the display color of the detection object, and the display color of the auxiliary object is the same as the display color of the detection object when the detection result indicates the presence of the object to be detected in the monitoring area. A control method for the work machine described in Appendix 6 or 7.

[0139] <Note 9> The display mode of the detection object changes according to the distance from the work machine to the object to be detected. When multiple objects to be detected exist in the monitoring area, the display mode of the detected object is determined according to the distance from the work machine to the object closest to it. A control method for the work machine described in any of the appendices 1 to 8.

[0140] <Note 10> In the display screen, the captured image is displayed with its outline formed in an arc shape for at least the left, right, and rear of the work machine's monitoring area, and the detection object is positioned at a location corresponding to the vertex angle of the captured image. A control method for the work machine described in any of the appendices 1 to 9.

[0141] <Note 11> The control method for the work machine described in any of the appendices 1 to 10, A control program for a work machine to be executed by one or more processors. [Explanation of Symbols]

[0142] 1. Control system for industrial machinery 2 Display device 3. Working Machines 11 Display Processing Unit 14 Image acquisition unit 15 Detection unit 30 aircraft 31 Running section 32 Swivel section A1 Surveillance Area Dp1 display screen I2, I3 Auxiliary Objects Im10 Detected Object Im11, Im12, Im13 Acquired Images Im100 Overhead view image (captured image) Ob1 Detected object

Claims

1. Detecting multiple objects in the monitoring area around the work machine, Displaying the screen on a display device, The system includes determining the display mode of the detection results displayed on the display screen according to the distance from the work machine to the closest detection target among the plurality of detection targets, A method for controlling industrial machinery.

2. The aforementioned display mode includes the display color. A method for controlling a work machine according to claim 1.

3. If the distance to the nearest object to be detected is within the threshold distance, the detection result is displayed in a first display mode; if the distance exceeds the threshold distance, the detection result is displayed in a second display mode different from the first display mode. A method for controlling a work machine according to claim 1 or 2.

4. A control method for a work machine according to claim 1 or 2, A control program for a work machine to be executed by one or more processors.

5. A detection unit that detects multiple objects in the monitoring area around the work machine, It comprises a display processing unit that displays the display screen on a display device, The display processing unit determines the display mode of the detection results to be displayed on the display screen according to the distance from the work machine to the detection target among the plurality of detection targets. Control system for industrial machinery.

6. A control system for a work machine according to claim 5, The unit comprises a body on which the aforementioned display device is mounted, Agricultural machinery.

Images