Control method for a work machine, control program for a work machine, control system for a work machine, and work machine

Abstract

The present invention provides a control method for a work machine, a control program for a work machine, a control system for a work machine, and a work machine that allow operators to intuitively grasp various information related to the operation of the work machine. [Solution] The control method for the work machine includes acquiring an image of the monitoring area around the work machine, displaying an overhead image Im100 based on the image in a first display area R11, and displaying the entire specific image Im200 in a second display area R12. The first display area R11 is a region of the display screen Dp1 displayed on the display device in which a part of the circumferential direction as seen from the machine's reference position P1, which corresponds to the work machine in a plan view, is cut out as a notched area R13. The second display area R12 is a fan-shaped region of the display screen Dp1 that is located within the notched area R13.

Description

Technical Field

[0001] The present invention relates to a control method for a working machine, a control program for a working machine, a control system for a working machine, and a working machine, which are used for a working machine having a function of detecting a detection target in a surrounding monitoring area.

Background Art

[0002] As a related art, there is known a working machine (excavator) that generates an output image based on a plurality of input images captured by cameras mounted on an upper revolving body (for example, see Patent Document 1). The working machine according to the related art includes cameras (imaging devices) mounted at three locations on the left, right, and rear of the upper revolving body so as to image three directions of the upper revolving body. In this working machine, at least the left, right, and rear areas of the working machine are projected onto a display device (display unit) installed in the cab, regardless of the direction of the lower traveling body, and an image of the working machine viewed from above (bird's-eye view image) is generated as an output image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above related art, simply a bird's-eye view image is displayed on the display device, and there are cases where it is difficult for an operator to intuitively grasp various information related to the operation of the working machine.

[0005] An object of the present invention is to provide a control method for a working machine, a control program for a working machine, a control system for a working machine, and a working machine, which enable an operator to intuitively and easily grasp various information related to the operation of the working machine.

Means for Solving the Problems

[0006] A control method for a work machine according to one aspect of the present invention comprises acquiring an image of a monitoring area surrounding the work machine, displaying an overhead image based on the image in a first display area, and displaying the entirety of a specific image in a second display area. The first display area is a region on the display screen displayed on the display device in which a part of the circumferential direction, as viewed from the machine's reference position corresponding to the work machine in a plan view, is cut out as a notched area. The second display area is a fan-shaped region on the display screen that is arranged within the notched area.

[0007] A control program for a work machine according to one aspect of the present invention is a program that causes one or more processors to execute a control method for the work machine.

[0008] A control system for a work machine according to one aspect of the present invention comprises an image acquisition unit, a first display processing unit, and a second display processing unit. The image acquisition unit acquires an image of the monitoring area surrounding the work machine. The first display processing unit displays an overhead image based on the image in a first display area on a display screen displayed on a display device, in which a part of the circumferential direction as seen from the machine's reference position, which corresponds to the work machine in a plan view, is cut out as a notched area. The second display processing unit displays the entirety of a specific image in a fan-shaped second display area located within the notched area on the display screen.

[0009] A work machine according to one aspect of the present invention comprises a control system for the work machine and a machine body on which the display device is mounted. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide a control method for a work machine, a control program for a work machine, a control system for a work machine, and a work machine that allow operators to intuitively grasp various information related to the operation of the work machine. [Brief explanation of the drawing]

[0011] [Figure 1]Figure 1 is a schematic perspective view showing the overall configuration of the work machine according to Embodiment 1. [Figure 2] Figure 2 is a schematic diagram showing the hydraulic circuit and other components of the work machine according to Embodiment 1. [Figure 3] Figure 3 is a schematic plan view of the work machine according to Embodiment 1, as seen from above, illustrating the monitoring area and other elements set around the work machine. [Figure 4] Figure 4 is a schematic external view of a display device whose display screen is shown by the control system for work machines according to Embodiment 1. [Figure 5] Figure 5 shows an example of a display screen shown by the control system for work machines according to Embodiment 1. [Figure 6] Figure 6 shows an example of the display of the second region in the display screen shown by the control system for work machines according to Embodiment 1. [Figure 7] Figure 7 shows an example of an overhead view displayed by the control system for work machinery according to Embodiment 1. [Figure 8] Figure 8 shows an example of an overhead view displayed by the control system for a work machine according to Embodiment 1. [Figure 9] Figure 9 is a flowchart showing an example of the operation of the control system for a work machine according to Embodiment 1. [Figure 10] Figure 10 shows an example of an overhead view displayed by the control system for work machinery according to Embodiment 2. [Figure 11] Figure 11 shows an example of an overhead view displayed by the control system for work machinery according to Embodiment 2. [Modes for carrying out the invention]

[0012] The embodiments of the present invention will be described below with reference to the attached drawings. The following embodiments are examples that embody the present invention and are not intended to limit the technical scope of the present invention.

[0013] (Embodiment 1) [1] Overall structure As shown in FIG. 1, a working machine 3 according to this embodiment includes a traveling unit 31, a slewing unit 32, and a working unit 33 on a machine body 30. Further, as shown in FIG. 2, the working machine 3 further includes a control system 1 for working machines (hereinafter, also simply referred to as "control system 1"). In addition, the machine body 30 further includes a display device 2 and an operating device or the like.

[0014] The "working machine" referred to in the present disclosure means various working machines. As an example, it is a working vehicle such as a backhoe (including a hydraulic excavator, a mini excavator, etc.), a wheel loader, and a carrier. The working machine 3 includes a working unit 33 configured to be capable of performing one or more operations including at least a lifting operation. The working machine 3 is not limited to a "vehicle", and may be, for example, a working flying object such as a working ship, a drone, or a multicopter. Further, the working machine 3 is not limited to a construction machine (construction equipment), and may be, for example, an agricultural machine (agricultural equipment) such as a rice transplanter, a tractor, or a combine. In this embodiment, unless otherwise specified, the working machine 3 is a backhoe with a lifting function (crane function), and a case where, in addition to the lifting operation, excavation work, leveling work, trench excavation work, loading work, or the like can be performed as work will be described as an example.

[0015] In this embodiment, for convenience of explanation, the vertical direction in a state where the working machine 3 can be used is defined as the up-down direction D1. Further, in a non-slewing state of the slewing unit 32, the front-rear direction D2 and the left-right direction D3 are defined based on the direction seen from a user (operator) boarding the working machine 3 (the operating unit 321). In other words, each direction used in this embodiment is a direction defined based on the machine body 30 of the working machine 3, and the direction in which the machine body 30 moves when the working machine 3 moves forward is "front", and the direction in which the machine body 30 moves when the working machine 3 moves backward is "rear". Similarly, the direction in which the front end of the machine body 30 moves when the working machine 3 slews to the right is "right", and the direction in which the front end of the machine body 30 moves when the working machine 3 slews to the left is "left". However, these directions are not intended to limit the usage direction (direction during use) of the working machine 3.

[0016] The working machine 3 includes an engine as a power source. In the working machine 3, for example, a hydraulic pump 41 (see FIG. 2) is driven by the engine, and hydraulic oil is supplied from the hydraulic pump 41 to hydraulic actuators (including a hydraulic motor 43 and a hydraulic cylinder 44, etc.) of each part of the machine body 30, thereby driving the machine body 30. Further, the working machine 3 is controlled, for example, by an operator (user) boarding on the operation part 321 of the machine body 30 operating an operation lever or the like of the operation device.

[0017] In the present embodiment, since it is assumed that the working machine 3 is a ride-on type backhoe as described above, the working part 33 is driven according to the operation of an operator (user) boarding on the operation part 32, and performs operations such as excavation work. The operation part 321 where the user boards is provided on the swing part 32.

[0018] The traveling part 31 has a traveling function and is configured to be able to travel (including turning) on the ground. The traveling part 31 has, for example, a pair of left and right crawlers 311 and a blade 312, etc. The traveling part 31 further has a traveling hydraulic motor 43 (hydraulic actuator) for driving the crawler 311, etc.

[0019] The swing part 32 is located above the traveling part 31 and is configured to be able to swing around a rotation axis along the vertical direction with respect to the traveling part 3>. The swing part 32 has a swing hydraulic motor (hydraulic actuator), etc. The swing part 32 is equipped with an engine, a hydraulic pump 41, etc. in addition to the operation part 321. Further, a boom bracket 322 to which the working part 33 is attached is provided at the front end of the swing part 32.

[0020] The working part 33 is configured to be able to perform operations including lifting work. The working part 33 is supported by the boom bracket 322 of the swing part 32 and performs operations. The working part 33 has a bucket 331, a boom 332, an arm 333, etc. The working part 33 further has hydraulic actuators (including a hydraulic cylinder 44 and a hydraulic motor, etc.) for driving each part.

[0021] The bucket 331 is a type of attachment (working tool) that is mounted on the machine body 30 of the work machine 3, and consists of any tool selected from several types of attachments according to the nature of the work. For example, the bucket 331 is detachably mounted to the machine body 30 and can be replaced according to the nature of the work. In addition to the bucket 331, various other tools can be used as attachments for the work machine 3, such as breakers, augers, crushers, forks, fork claws, steel frame cutters, asphalt milling machines, brush cutters, rippers, mulchers, tilt rotators, and tampers. The work unit 33 performs its work by driving the bucket 331 with power from the drive unit.

[0022] The boom 332 is rotatably supported by the boom bracket 322 of the slewing section 32. Specifically, the boom 332 is rotatably supported by the boom bracket 322 around a rotation axis along the horizontal direction. The boom 332 has a shape that extends upward from the base end supported by the boom bracket 322. The arm 333 is connected to the tip of the boom 332. The arm 333 is rotatably supported relative to the boom 332 around a rotation axis along the horizontal direction. A bucket 331 is attached to the tip of the arm 333.

[0023] The work unit 33 operates by receiving power from the engine, which serves as the power source. Specifically, the engine drives the hydraulic pump 41, and hydraulic fluid is supplied from the hydraulic pump 41 to the hydraulic actuators (hydraulic cylinders 44, etc.) of the work unit 33, thereby operating each part of the work unit 33 (bucket 331, boom 332, and arm 333).

[0024] In this embodiment, the work unit 33 has a multi-joint structure in which the boom 332 and the arm 333 are configured to rotate independently. That is, by each of the boom 332 and the arm 333 rotating around a rotation axis along the horizontal direction, the multi-joint work unit 33 including the boom 332 and the arm 333 can be extended or folded as a whole.

[0025] The traveling unit 31 and the slewing unit 32, like the working unit 33, operate by receiving power from the engine as a power source. In other words, the slewing unit 32 and the traveling unit 31 operate when hydraulic fluid is supplied from the hydraulic pump 41 to the hydraulic motor 43 of the traveling unit 31 and the hydraulic motor of the slewing unit 32, etc.

[0026] As described above, the engine functions as a power source that supplies power to each part. Here, the engine is mounted on the slewing section 32 together with the hydraulic pump 41, etc. In this embodiment, as an example, the engine is a diesel engine. The engine is driven by fuel (diesel in this case) supplied from the fuel tank.

[0027] Here, the machine body 30 is equipped with various sensors (including cameras) for detecting the object Ob1 (see Figure 3) in the monitoring area A1 (see Figure 3) surrounding the work machine 3, including a camera that takes images of the area around the machine body 30. In this embodiment, as an example, as shown in Figure 3, multiple cameras (three in this case), including a left camera 341, a right camera 342, and a rear camera 343, are mounted on the rotating section 32 of the machine body 30. The left camera 341, the right camera 342, and the rear camera 343 are connected to the control system 1 and output the images they capture to the control system 1. Figure 3 is a plan view of the work machine 3 seen from above, schematically representing the monitoring area A1 set around the work machine 3, the object Ob1, and the machine body 30 of the work machine 3 (including the left camera 341, the right camera 342, and the rear camera 343).

[0028] The left camera 341, the right camera 342, and the rear camera 343 are positioned to the left, right, and rear of the driver's unit 321, respectively, so that they can capture images of the monitoring area A1 that is to the left, right, and rear of the operator seated in the driver's unit 321 of the rotating unit 32. In other words, as shown in Figure 3, the monitoring area A1 includes several (in this case, three) sub-areas A11, A12, and A13, and the left camera 341 captures the sub-area A11 (left area) which is to the left of the operator seated in the driver's unit 321. Similarly, the right camera 342 captures the sub-area A12 (right area) which is to the right of the operator seated in the driver's unit 321, and the rear camera 343 captures the sub-area A13 (rear area) which is behind the operator seated in the driver's unit 321. This makes it possible to cover the sides (left and right) and rear, which are often blind spots for the operator, using the left camera 341, the right camera 342, and the rear camera 343.

[0029] Figure 2 schematically shows the hydraulic and electrical circuits (electrical connection relationships) of the work machine 3 according to this embodiment. In Figure 2, solid lines represent high-pressure (hydraulic oil) oil passages, dotted lines represent low-pressure (pilot oil) oil passages, and dashed arrows indicate the paths of electrical signals.

[0030] As shown in Figure 2, the work machine 3 is equipped with a hydraulic pump 41, a hydraulic motor 43 (not shown in Figure 2), and a hydraulic cylinder 44, as well as a pilot pump 42, a remote control valve 45, a control valve 46, and a directional control valve 47, etc.

[0031] Hydraulic fluid from the engine-driven hydraulic pump 41 is supplied to the hydraulic motor 43 of the travel section 31, the hydraulic motor of the slewing section 32, and the hydraulic cylinder 44 of the work section 33, etc. This drives the hydraulic actuators such as the hydraulic motor 43 and the hydraulic cylinder 44.

[0032] Hydraulic actuators such as the hydraulic motor 43 and hydraulic cylinder 44 are equipped with a pilot-operated directional control valve 47 that can switch the direction and flow rate of the hydraulic fluid from the hydraulic pump 41. The directional control valve 47 is driven by pilot oil supplied from the pilot pump 42, which serves as an input command.

[0033] Here, for example, a remote control valve 45 is provided in the supply path for pilot oil to the directional control valve 47 corresponding to the hydraulic cylinder 44 of the work unit 33. The remote control valve 45 outputs work operation commands for the work unit 33 in response to the operation of the operating lever. The work operation commands instruct the work unit 33 to perform operations such as extending and retracting. In addition, an electromagnetic control valve 46 (solenoid valve) is inserted between the remote control valve 45 and the pilot pump 42. The control valve 46 is connected to the power supply 351 via a cutoff relay 352 and a cutoff switch 353, and operates in accordance with the current supplied from the power supply 351.

[0034] Similarly, a remote control valve is also provided in the supply path for pilot oil to the directional control valve corresponding to the hydraulic motor 43 of the travel unit 31. This remote control valve outputs a travel operation command for the travel unit 31 in response to the operation of the operating lever. The travel operation command instructs the travel unit 31 to travel (forward or backward, etc.). Furthermore, a remote control valve is also provided in the supply path for pilot oil to the directional control valve corresponding to the hydraulic motor of the slewing unit 32. This remote control valve outputs a slewing operation command for the slewing unit 32 in response to the operation of the operating lever. The slewing operation command instructs the slewing unit 32 to slewing (left turn or right turn, etc.). An electromagnetic control valve 46 (solenoid valve) is also inserted between these remote control valves and the pilot pump 42. The control valve 46 is connected to the power supply 351 via a cutoff relay 352 and a cutoff switch 353, and operates in accordance with the current supplied from the power supply 351.

[0035] The control valve 46 opens the flow path of pilot oil from the pilot pump 42 to the remote control valve 45 when energized, i.e., when current is supplied, and closes the flow path of pilot oil when de-energized, i.e., when the supply current is cut off. As a result, when the current supplied to the control valve 46 is cut off, the hydraulic actuator corresponding to the remote control valve 45 becomes unable to drive, and the output of the hydraulic actuator is forcibly stopped regardless of the operation of the operating lever.

[0036] Here, the cutoff relay 352 is connected to the control system 1 and switches on / off in response to a control signal (electrical signal) from the control system 1. The cutoff switch 353 switches on / off in response to the operation of the cutoff lever, for example, it turns on when the cutoff lever is operated downwards. Therefore, when both the cutoff relay 352 and the cutoff switch 353 are on, the control valve 46 is energized, and the flow path of pilot oil from the pilot pump 42 to the remote control valve 45 is opened, so the hydraulic actuator is driven in response to the operation of the operating lever. Conversely, when at least one of the cutoff relay 352 and the cutoff switch 353 is off, the control valve 46 is de-energized, and the flow path of pilot oil is blocked, so the hydraulic actuator cannot be driven.

[0037] For example, if at least one of the cutoff relay 352 and cutoff switch 353, which are connected to a control valve 46 inserted between the remote control valve corresponding to the hydraulic motor of the slewing section 32 and the pilot pump 42, is in the off state, the hydraulic motor of the slewing section 32 becomes inoperable. In this state, the output of the hydraulic actuator (hydraulic motor of the slewing section 32) is forcibly stopped regardless of the operation of the operating lever, thus preventing the slewing operation of the slewing section 32.

[0038] The control system 1 primarily consists of a computer system having one or more processors such as a CPU (Central Processing Unit) and one or more memories such as ROM (Read Only Memory) and RAM (Random Access Memory), and performs various processes (information processing). In this embodiment, the control system 1 is an integrated controller that controls the entire work machine 3, and consists of, for example, an electronic control unit (ECU). However, the control system 1 may be provided separately from the integrated controller. The control system 1 will be explained in detail in the section "[2] Configuration of the Control System".

[0039] The display device 2 is located in the control unit 321 of the machine body 30 and is a user interface for receiving operation input from the user (operator) and outputting various information to the user. The display device 2 accepts various operations from the user by outputting electrical signals corresponding to the user's operations. As a result, the user (operator) can view the display screen Dp1 (see Figure 4) displayed on the display device 2 and operate the display device 2 as needed.

[0040] As shown in Figure 2, the display device 2 comprises a control unit 21, an operation unit 22, and a display unit 23. The display device 2 is configured to communicate with the control system 1 and can exchange data with the control system 1. In this embodiment, as an example, the display device 2 is a dedicated device used in the work machine 3.

[0041] The control unit 21 controls the display device 2 according to data from the control system 1. Specifically, the control unit 21 outputs electrical signals corresponding to user operations received by the operation unit 22, and displays the display screen Dp1 generated by the control system 1 on the display unit 23.

[0042] The operation unit 22 is a user interface for receiving operation input from a user (operator) to the display screen Dp1 displayed on the display unit 23. The operation unit 22 accepts various operations by user U1 (see Figure 4) by outputting electrical signals corresponding to the user U1's operations. In this embodiment, as an example, the operation unit 22 includes a plurality of (in this case, six) mechanical push-button switches 221 to 226, as shown in Figure 4. These plurality of push-button switches 221 to 226 are arranged close to the display area (below in the example of Figure 4) so ​​as to be along the periphery of the display area of ​​the display unit 23. These plurality of push-button switches 221 to 226 are associated with items displayed on the display screen Dp1, which will be described later, and when any of the plurality of push-button switches 221 to 226 are operated, any of the items on the display screen Dp1 are operated (selected).

[0043] Furthermore, the operation unit 22 may include a touch panel and an operation dial. In this case as well, an operation on the operation unit 22 will result in an operation (selection) of one of the items on the display screen Dp1.

[0044] The display unit 23 is a user interface for presenting information to the user U1 (operator), such as a liquid crystal display or an organic EL display that displays various types of information. The display unit 23 presents various types of information to the user through display. In this embodiment, as an example, the display unit 23 is a full-color liquid crystal display with a backlight, and as shown in Figure 4, it has a horizontally elongated "landscape" display area.

[0045] The display device 2 presents various information to the user U1 (operator) operating the work machine 3 via the display screen Dp1. In other words, the user U1 operating the work machine 3 can visually obtain various information related to the work machine 3 by looking at the display screen Dp1 displayed on the display device 2. For example, by displaying information related to the operating status of the work machine 3, such as the cooling water temperature and hydraulic oil temperature, on the display device 2, the user U1 can check the information related to the operating status of the work machine 3 necessary for operating the work machine 3 on the display device 2. In addition, the display device 2 can also display images of the surrounding area of ​​the work machine 3 (images of the monitoring area A1) captured by the left camera 341, the right camera 342, and the rear camera 343 on the display screen Dp1. This allows the user U1 (operator) to check the situation of the side and rear of the work machine 3, which are often blind spots from the driver's unit 321, on the display screen Dp1 displayed on the display device 2 when operating the work machine 3.

[0046] Furthermore, the display device 2 may have a function to output sound (including voice) to the user U1 (operator). In this case, the display device 2 outputs a sound such as a beep or voice in response to a sound control signal from the control system 1.

[0047] In addition to the above-described configuration, the aircraft 30 is further equipped with an operating lever, a cutoff lever, a communication terminal, a fuel tank, and a battery. Furthermore, the aircraft 30 is equipped with sensors to monitor its operating status, such as a coolant temperature sensor, a hydraulic oil temperature sensor, a tachometer for measuring engine speed, and an hour meter for measuring operating time. The aircraft 30 is also equipped with sensors to detect the status of the cutoff lever and starter key switch, etc.

[0048] [2] Control system configuration Next, the configuration of the control system 1 according to this embodiment will be described with reference to Figure 2. The control system 1 controls the display device 2 to display the display screen Dp1 on the display device 2. In this embodiment, the display device 2 is mounted on the machine body 30 of the work machine 3 as described above. The control system 1 is a component of the work machine 3 and together with the machine body 30, etc., constitutes the work machine 3. In other words, the work machine 3 according to this embodiment comprises at least the control system 1 and the machine body 30 (including the traveling section 31, the rotating section 32, and the working section 33) on which the display device 2 is mounted.

[0049] In this disclosure, "screen" as used in display screen Dp1, etc., means the image (video) displayed on the display device 2, and includes illustrations, figures, photographs, text, and videos. That is, the control system 1 can display on the display device 2 a display screen Dp1 that includes illustrations, etc., representing information about the operating status of the work machine 3, such as the cooling water temperature and the hydraulic oil temperature. Here, if the display screen Dp1 includes videos, etc., the display screen Dp1 does not include a constant video, but rather a video that changes moment by moment.

[0050] As shown in Figure 2, the control system 1 comprises a display processing unit 11, an image acquisition unit 12, a detection unit 13, and a control processing unit 14. In this embodiment, as an example, the control system 1 mainly consists of a computer system having one or more processors, so these multiple functional units (display processing unit 11, etc.) are realized by one or more processors executing a control program for the work machine. These multiple functional units included in the control system 1 may be distributed and provided in multiple housings, or they may be provided in a single housing.

[0051] The control system 1 is configured to communicate with devices installed in various parts of the aircraft body 30. Specifically, the control system 1 is connected to at least the display device 2, the cutoff relay 352, the left camera 341, the right camera 342, and the rear camera 343. This allows the control system 1 to control the display device 2, control the cutoff relay 352 to control the control valve 46, and acquire images captured by the left camera 341, the right camera 342, and the rear camera 343. Here, the control system 1 may exchange various types of information (data) directly with each device, or indirectly via a relay or the like.

[0052] The display processing unit 11 performs display processing to display at least the display screen Dp1 on the display device 2. Specifically, the display processing unit 11 generates the display screen Dp1 based on data acquired by the image acquisition unit 12, and controls the display device 2 to display the display screen Dp1 on the display unit 23 of the display device 2. Furthermore, the display processing unit 11 operates in accordance with the operation received by the operation unit 22 of the display device 2. For example, the display processing unit 11 displays the captured images Im11, Im12, Im13 (see Figure 5) captured by the left camera 341, the right camera 342, and the rear camera 343 on the display screen Dp1. In other words, the display processing unit 11 and the image acquisition unit 12 display images of the monitoring area A1 (each sub-area A11, A12, A13) around the work machine 3 on the display device 2.

[0053] Furthermore, the display processing unit 11 has a full-screen mode that displays the second region R2 (see Figure 5) of the display screen Dp1 in full screen (enlarged). In addition, the display processing unit 11 can display on the display screen Dp1 at least in full-screen mode an overhead image Im100 (see Figure 7) generated based on the captured images Im11, Im12, and Im13, and at least a portion of a specific image Im200 (see Figure 7) that is different from the overhead image Im100.

[0054] Here, the display processing unit 11 includes a first display processing unit 111 and a second display processing unit 112. As will be described in detail later, the first display processing unit 111 performs a first display process to display the overhead image Im100, which is generated based on the captured images Im11, Im12, and Im13, in the first display area R11 (see Figure 7) of the display screen Dp1 displayed on the display device 2. The second display processing unit 112 performs a second display process to display at least a part of the specific image Im200 in the second display area R12 (see Figure 7) of the display screen Dp1 displayed on the display device 2.

[0055] Furthermore, the display processing unit 11 displays on the display screen Dp1, in addition to the captured images Im11, Im12, Im13, the overhead image Im100, and the specific image Im200, as well as detection objects representing the detection result of the detection unit 13 regarding the presence or absence of the detection target object Ob1 in the monitoring area A1. The display manner of the detection objects changes according to the detection result of the detection unit 13. Therefore, the display manner of the detection objects on the display screen Dp1 differs depending on whether or not the detection target object Ob1 is present in the monitoring area A1.

[0056] In this disclosure, "objects" such as detected objects include marks, images, figures, photographs, text, or videos displayed on the display screen Dp1, or combinations thereof. Furthermore, "display mode" as used in this disclosure includes the display color, size (including dimensions, line thickness, etc.), shape, operating state (including animation, rotation / stillness, etc.), presence or absence of display, or display pattern (including blinking patterns, etc.) of the object, or combinations thereof. In short, the display processing unit 11 changes the display mode, such as the display color of the detected object, according to the detection result of the detection unit 13.

[0057] The image acquisition unit 12 performs an image acquisition process to acquire captured images of the monitoring area A1 surrounding the work machine 3. In this embodiment, the image acquisition unit 12 periodically or irregularly acquires the outputs of the left camera 341, the right camera 342, and the rear camera 343 from the left camera 341, the right camera 342, and the rear camera 343. In other words, the image acquisition unit 12 acquires image data (captured images) of the monitoring area A1 (each sub-area A11, A12, A13) surrounding the work machine 3. The data acquired by the image acquisition unit 12 is stored in, for example, memory.

[0058] The detection unit 13 performs a detection process to detect the object Ob1 in the monitoring area A1 surrounding the work machine 3. In other words, the detection unit 13 determines the presence or absence of the object Ob1 in the monitoring area A1 and outputs a detection result indicating whether or not the object Ob1 exists in the monitoring area A1. In this embodiment, as an example, the object Ob1 is a "person". In other words, if the work machine 3 moves, or if a "person" moves around the work machine 3, and as a result a "person" enters the monitoring area A1 surrounding the work machine 3, the detection unit 13 detects the "person" as the object Ob1. If there are multiple objects Ob1 in the monitoring area A1, the detection unit 13 may also detect the number of objects Ob1 (people).

[0059] In this embodiment, the detection unit 13 detects the object to be detected Ob1 in the monitoring area A1 based on the output (image data) of the left camera 341, the right camera 342, and the rear camera 343. Specifically, the detection unit 13 extracts feature quantities from the image by performing image processing on the image data acquired by the image acquisition unit 12, and determines whether or not the object to be detected Ob1 (in this embodiment, "person") is captured in the image based on these feature quantities. If the object to be detected Ob1 is captured in the image, the detection unit 13 determines whether the object to be detected Ob1 is captured in the image captured by the left camera 341, the right camera 342, or the rear camera 343. In other words, the detection unit 13 distinguishes whether the object to be detected Ob1 is located in the small area A11 captured by the left camera 341, the small area A12 captured by the right camera 342, or the small area A13 captured by the rear camera 343, and then detects the object to be detected Ob1.

[0060] The deterrent processing unit 14 executes deterrent processing to deter the operation of the work machine 3 based on the detection result of the detection unit 13. In this embodiment, the deterrent processing unit 14 executes deterrent processing when the detection result of the detection unit 13 indicates the presence of a detected object Ob1 (in this case, a person) in the monitoring area A1. In this disclosure, "deterrent processing" means processing that acts in a direction that suppresses the operation of the work machine 3 in some way. As an example, the deterrent processing includes processing that indirectly deter the operation of the work machine 3 by controlling the display device 2 to warn the user U1 (operator) operating the work machine 3 with sound or light (including display). Furthermore, the deterrent processing includes processing that directly deter the operation of the work machine 3 by controlling the travel unit 31, the swivel unit 32, the work unit 33, etc. of the work machine 3.

[0061] Specifically, in this embodiment, as an example, the restraint processing unit 14 turns off the cutoff relay 352 when a detected object Ob1 is present in the monitoring area A1. As a result, the control valve 46 connected to the power supply 351 via the cutoff relay 352 becomes de-energized, and the output of the hydraulic actuator corresponding to the control valve 46 is forcibly stopped. In other words, in this embodiment, the restraint processing includes a restriction processing that restricts the operation of the work machine 3. In this disclosure, "restriction processing" means processing that acts in a direction that restricts the operation of the work machine 3 in some way. As an example, the restriction processing includes processing that prohibits the travel operation of the travel unit 31 (making it impossible to travel), processing that prohibits the rotation operation of the slewing unit 32 (making it impossible to rotate), and processing that prohibits the operation of the work unit 33 (making it impossible to work). As a result, the operation of the work machine 3 can be forcibly restricted without the operation of the user U1 (operator). In other words, contact between the machine body 30 and the detected object Ob1 due to the operation of the work machine 3 can be avoided.

[0062] By the way, the detection unit 13 is not an essential component of the control system 1. For example, the control system 1 may be configured to acquire detection results from an external detection unit, and the restraining processing unit 14 may execute restraining processing based on those detection results.

[0063] [3] Control method for working machinery The following describes an example of a control method (hereinafter simply referred to as "control method") for the work machine 3, which is mainly performed by the control system 1, with reference to Figures 5 to 9.

[0064] The control method according to this embodiment is executed by a control system 1, which mainly consists of a computer system; in other words, it is embodied in a control program for a work machine (hereinafter simply referred to as the "control program"). That is, the control program according to this embodiment is a computer program that causes one or more processors to execute each process related to the control method. Such a control program may be executed in cooperation with, for example, the control system 1 and the display device 2.

[0065] Here, the control system 1 executes the following various processes related to the control method when a specific pre-set start operation is performed to execute the control program. The start operation is, for example, the operation to start the engine of the work machine 3. On the other hand, the control system 1 terminates the following various processes related to the control method when a specific pre-set end operation is performed. The end operation is, for example, the operation to stop the engine of the work machine 3.

[0066] [3.1] Display screen First, the configuration of the display screen Dp1 displayed on the display unit 23 of the display device 2 by the control method according to this embodiment will be described. In the drawings showing the display screen Dp1 displayed on the display unit 23 of the display device 2, such as Figure 5, the dashed lines, leader lines, and reference numerals representing areas are all there for illustrative purposes only and are not actually displayed on the display device 2.

[0067] The display screen Dp1 shown in Figure 5 is the home screen Dp11 that is initially displayed depending on the control method. In other words, the display processing unit 11 of the control system 1 generates a display screen Dp1 as shown in Figure 5, for example, and executes a display process to display it on the display device 2 (display unit 23). The home screen Dp11 is the basic display screen Dp1 that is first displayed on the display device 2 while the work machine 3 is in operation. From the home screen Dp11, it is possible to transition to various display screens Dp1, including menu screens, crane screens, mode screens, and PTO screens, according to operations on the operation unit 22.

[0068] As shown in Figure 5, the display screen Dp1, which serves as the home screen Dp11, includes the first area R1, the second area R2, the third area R3, the fourth area R4, the fifth area R5, the sixth area R6, the seventh area R7, the eighth area R8, the ninth area R9, and the tenth area R10. In the control method according to this embodiment, as an example, the second area R2, which occupies most of the display screen Dp1, displays the captured images Im11, Im12, Im13, etc., of the monitoring area A1 (each sub-area A11, A12, A13).

[0069] Specifically, the display screen Dp1 is divided into four vertical (up and down) sections. The top three sections are then further divided into three horizontal (left and right) sections. This results in a total of 10 sections on the display screen Dp1. The second section from the top consists of three sections from left to right: Section 1 R1, Section 2 R2, and Section 3 R3. The bottom section is Section 4 R4. The third section from the top consists of three sections from left to right: Section 5 R5, Section 6 R6, and Section 7 R7. The top section consists of three sections from left to right: Section 8 R8, Section 9 R9, and Section 10 R10. In terms of vertical size, the second section from the top (Section 1 R1, Section 2 R2, and Section 3 R3) is the largest of the four vertically divided sections. In terms of horizontal size, the middle region (region 2 R2, region 6 R6, and region 9 R9) is the largest among the three regions divided horizontally.

[0070] However, the arrangement and size of these regions are merely examples and can be changed as needed. Furthermore, it is not necessary for each region to be clearly separated by a boundary line. For example, in the example in Figure 5, the second region R2 and the third region R3 are clearly separated by a boundary line, while there is no boundary line between the first region R1 and the second region R2. Of course, the first region R1 and the second region R2 may also be clearly separated by a boundary line.

[0071] The first region R1 is a rectangular area that is elongated vertically. In the first region R1, for example, remaining fuel information G1 regarding the remaining amount of engine fuel (e.g., diesel fuel) is displayed. The display processing unit 11 generates the remaining fuel information G1 on the display screen Dp1 based on the output of the fuel level sensor (sensor signal), etc.

[0072] The second region R2 is a horizontally elongated rectangular area, and as described above, captured images Im11, Im12, Im13, etc. are displayed in the second region R2. Captured image Im11 is an image of a small area A11 to the left of the driver unit 321, captured by the left camera 341, and captured image Im12 is an image of a small area A12 to the right of the driver unit 321, captured by the right camera 342. Captured image Im13 is an image of a small area A13 behind the driver unit 321, captured by the rear camera 343. The display processing unit 11 displays the captured images Im11, Im12, and Im13 acquired by the image acquisition unit 12 in real time.

[0073] Furthermore, an icon Im10, which is modeled after the aircraft 30 as seen from above, is displayed in the center of the second region R2. Icon Im10 schematically represents the positional relationship of the imaging ranges (small areas A11, A12, A13) of the left camera 341, the right camera 342, and the rear camera 343 as seen from the aircraft 30. In this embodiment, the captured images Im11, Im12, and Im13 have a rectangular cutout in the center of the second region R2. Therefore, the icon Im10 displayed in the center of the second region R2 is positioned to fit into the cutout portion of the captured images Im11, Im12, and Im13.

[0074] The third region R3 is a vertically elongated rectangular area. Icons (Im1) corresponding to the operating status of each part of the work machine 3 are displayed in the third region R3. Multiple icons (Im1) can be displayed in the third region R3, and the design (pattern) of each icon (Im1) indicates which state it represents, for example, the battery, seat belt, coolant temperature, hydraulic oil temperature, etc. Here, each icon (Im1) indicates its operating status by its display characteristics, such as its display color or size. The display processing unit 11 determines the status of each part of the work machine 3 using the outputs of various sensors (including a coolant temperature sensor and a hydraulic oil temperature sensor) that detect the operating status of each part of the work machine 3. If an abnormal value is detected in any part, the display processing unit 11 displays a warning by changing the display characteristics, such as the display color, of the icon (Im1) for that part.

[0075] The fourth area R4 is a band-shaped area that extends across the entire width of the display screen Dp1. The fourth area R4 displays various items for operating the display screen Dp1. As an example in Figure 5, the fourth area R4 has six items: "Menu," "Crane," "Mode," "Camera," "PTO," and "Switch," arranged from left to right in that order. These six items are associated with the six push-button switches 221 to 226 on the operation unit 22 located directly below them. For example, the push-button switch 221 is associated with the "Menu" item, and the push-button switch 222 is associated with the "Crane" item. Therefore, for example, when the push-button switch 224 corresponding to the "Camera" item is operated by user U1 (see Figure 4), the "Camera" item is operated (selected).

[0076] Furthermore, in this embodiment, in order to respond to operations such as the operation dial (or cursor keys) of the operation unit 22, one of the items is highlighted in the fourth area R4. In the example in Figure 5, the "Menu" item is highlighted, and the highlighted item is switched by operations such as the operation dial (or cursor keys). User U1 can select the desired item by operating the OK button while the desired item is highlighted. For example, if the OK button is operated while the highlight is moved to the "Camera" item, the "Camera" item will be operated (selected). Also, if the operation unit 22 includes a touch panel, user U1 can select the desired item by touching the desired item on the display screen Dp1.

[0077] Area 5 R5 displays warning icons indicating that abnormal values ​​have been detected by various sensors (including coolant temperature sensor and hydraulic oil temperature sensor). Area 6 R6 displays information about the working unit 33 currently operating in the working machine 3, for example. Area 7 R7 displays information about the operating status of the working machine 3, for example, the engine speed. Area 8 R8 displays the current time, for example. Area 9 R9 displays information indicating the item to which the currently displayed screen Dp1 belongs, for example. Area 10 R10 displays information about the operating time (hour meter) of the working machine 3, for example.

[0078] In this embodiment, the display processing unit 11 switches the display screen Dp1 to be displayed on the display device 2 between multiple screens, including the home screen Dp11 described above and the enlarged screen Dp12 shown in Figure 7. In other words, the display screen Dp1 displayed on the display device 2 can be switched from the home screen Dp11 described above to the enlarged screen Dp12 shown in Figure 7.

[0079] The enlarged screen Dp12 shown in Figure 7 is a screen that displays the second area R2 of the display screen Dp1 in full screen by enlarging the second area R2 compared to the home screen Dp11. In other words, the display processing unit 11 switches from the normal mode of displaying the home screen Dp11 to full-screen mode, causing the display unit 23 of the display device 2 to display the enlarged screen Dp12, which displays the second area R2 in full screen.

[0080] In the example shown in Figure 7, the enlarged screen Dp12 includes only the second region R2, and the entire screen becomes the second region R2. In other words, in the enlarged screen Dp12, the first region R1 and the third to tenth regions R3 to R10 are hidden, and the second region R2 is displayed in an enlarged view.

[0081] Therefore, in the enlarged screen Dp12, the captured images Im11, Im12, and Im13 are displayed enlarged compared to the home screen Dp11, resulting in a display mode that provides excellent visibility of the captured images Im11, Im12, and Im13. On the other hand, unlike the home screen Dp11, the enlarged screen Dp12 does not include operational information related to the operating status of the work machine 3, such as the remaining amount information G1 of the first region R1.

[0082] [3.2] Details of the overhead view Next, we will describe in detail the overhead image Im100 displayed by the control method according to this embodiment.

[0083] As described above, the display processing unit 11 can display the overhead image Im100 on the display screen Dp1 (enlarged screen Dp12) at least in full-screen mode when displaying the enlarged screen Dp12. That is, the display processing unit 11 can switch the image displayed in the second area R2 of the display screen Dp1 between the captured images Im11, Im12, Im13 as shown in Figure 6 and the overhead image Im100 as shown in Figure 7, depending on the viewpoint switching operation. An example of a viewpoint switching operation is operating (selecting) the "camera" item or operating the viewpoint switching switch. Alternatively, the display processing unit 11 may automatically switch between the captured images Im11, Im12, Im13 and the overhead image Im100 depending on the detection result of the detection unit 13, for example.

[0084] The overhead view image Im100 is an image of the machine 30 viewed from above (above), and in this embodiment, regardless of the orientation of the running unit 31, it is an image that includes the left, right, and rear areas as seen from the perspective of a user (operator) riding in the work machine 3 (or its operating unit 321). Such an overhead view image Im100 is a type of captured image generated by transforming and combining the captured images Im11, Im12, and Im13 from the left camera 341, the right camera 342, and the rear camera 343. In this embodiment, as an example, the overhead view image Im100 is generated by the first display processing unit 111 of the display processing unit 11 based on the captured images Im11, Im12, and Im13, but it is not limited to this example and may be generated by a device other than the display processing unit 11. The overhead view image Im100 may also be an image in which the captured images are combined into an animation.

[0085] In the example shown in Figure 7, the display processing unit 11 sets a first display area R11 and a second display area R12 in the second area R2 of the display screen Dp1 (magnified screen Dp12), displays the overhead image Im100 in the first display area R11, and displays the specific image Im200 in the second display area R12. The first display area R11 is a region of the display screen Dp1 displayed on the display device 2 in which a part of the circumferential direction as seen from the machine's reference position P1, which corresponds to the work machine 3 in a plan view, is cut out as a notched area R13. The second display area R12 is a region of the display screen Dp1 displayed on the display device 2 that is located within the notched area R13.

[0086] Specifically, in the center of the second region R2, an icon Im10, which is modeled after the machine 30 as seen from above, is displayed. The center of this icon Im10, which is the center of the second region R2, is the machine's reference position P1, which corresponds to the work machine 3 in a plan view. The first display region R11, where the overhead view image Im100 is displayed, has a shape in which a part of the circumferential direction (circumferential direction) centered on this machine's reference position P1 (the center of the second region R2) is cut out as a notched region R13. In Figure 7, the machine's reference position P1 is indicated by a black circle, but this is merely for explanatory purposes, similar to reference numerals, and is not actually displayed on the display device 2.

[0087] More specifically, the first display area R11 is a fan-shaped region with an arc-shaped periphery on the side farther from the local reference position P1, and the icon Im10 is positioned at a location corresponding to the apex angle of the fan. In this disclosure, "arc-shaped" is not limited to arcs that constitute part of a perfect circle, but includes all curved lines that are convex outward. Therefore, even if the first display area R11 consists of a curved line whose periphery (outline) as a whole is convex outward, as shown in Figure 7, it can still be described as a fan-shaped region with an arc-shaped periphery. In other words, the periphery of the first display area R11 includes an arc R110 and a pair of radially aligned edges R111 and R112 that connect to both ends of the arc R110, and the region enclosed by this arc R110 and the pair of edges R111 and R112 is fan-shaped.

[0088] Then, within the circular (annular) region centered on the local reference position P1, the area other than the first display region R11, which is cut out by the pair of edges R111 and R112, becomes the notched region R13. In the example in Figure 7, the angle between the pair of edges R111 and R112 in the first display region R11 (the central angle of the sector) is approximately 260 degrees. Therefore, the central angle of the notched region R13 is approximately 100 degrees.

[0089] Figure 8 shows an example of the display screen Dp1 (enlarged screen Dp12) that is actually displayed on the display device 2. As shown in Figure 8, the overhead view image Im100 displayed in the first display area R11 is an image obtained by seamlessly combining the captured images Im11, Im12, and Im13 by performing a coordinate transformation so that it is an image of the aircraft 30 viewed from above (from above). In Figure 8, the icon Im10 is not an object that mimics the aircraft 30, but an object that represents the position corresponding to the aircraft 30 with a circle.

[0090] In other words, the control method according to this embodiment includes acquiring captured images Im11, Im12, and Im13 of the monitoring area around the work machine 3, and displaying an overhead view image Im100 based on the captured images Im11, Im12, and Im13 in a first display area R11. The first display area R11 is a region of the display screen Dp1 displayed on the display device 2 in which a part of the circumferential direction as seen from the machine's reference position P1, which corresponds to the work machine 3 in a plan view, is cut out as a notched area R13. The control method further includes displaying at least a part of a specific image Im200 in a second display area R12 located within the notched area R13 of the display screen Dp1.

[0091] In short, the display screen Dp1, which includes the overhead image Im100 and the specific image Im200, is displayed on the display device 2. Moreover, the overhead image Im100 is displayed in a first display area R11, which has a shape in which a part of the circumferential direction as viewed from the machine's reference position P1 is cut out as a notch area R13, and at least a part of the specific image Im200 is displayed in a second display area R12 located within the notch area R13. Therefore, the display device 2 can display the overhead image Im100, which has a partially cut-out shape, along with another image (specific image Im200) using the notch area R13, so that the overhead image Im100 and the specific image Im200 can be displayed together. As a result, the operator can view the overhead image Im100 and the specific image Im200, which are displayed together, without any sense of incongruity, and it becomes easier for the operator to intuitively grasp various information regarding the operation of the work machine 3.

[0092] In this embodiment, as shown in Figures 7 and 8, the second display area R12 is positioned above the first display area R11 on the display screen Dp1. Here, "above" means above as seen from the perspective of the user (operator) viewing the display screen Dp1, and refers to one of the vertical directions (up and down directions) within the display screen Dp1, regardless of the direction in real space. In other words, the first display area R11 has a shape in which the upper side of the display screen Dp1 is cut out as a notch area R13 as seen from the machine's reference position P1, and the second display area R12 is positioned within this notch area R13. With this arrangement, the visibility of the overhead image Im100 displayed in the first display area R11 can be maintained, while the specific image Im200 can also be displayed in a position that is easy for the operator to see.

[0093] Here, the notched area R13 is positioned in front of the work machine 3 in the overhead view image Im100. That is, in this embodiment, the upper part of the display screen Dp1 corresponds to the front of the work machine 3 in the overhead view image Im100, and the second display area R12 is positioned in front of the work machine 3. For the operator, the area in front of the work machine 3 is the easiest to see, so by making the area in the overhead view image Im100 that corresponds to the front of the work machine 3 the notched area R13, it is possible to display the overhead view image Im100 without significantly reducing the visibility around the work machine 3.

[0094] Furthermore, in this embodiment, as shown in Figures 7 and 8, a gap is provided between the first display area R11 and the second display area R12 in the display screen Dp1. In other words, the second display area R12 is set to be spaced apart from the pair of edges R111 and R112 in the first display area R11. As a result, a gap is formed between the overhead image Im100 displayed in the first display area R11 and the specific image Im200 displayed in the second display area R12, making it easier for the operator to distinguish between the overhead image Im100 and the specific image Im200. Consequently, this contributes to improved visibility of both the overhead image Im100 and the specific image Im200.

[0095] Furthermore, in this embodiment, at least a portion of the periphery of the second display area R12 in the display screen Dp1 is arranged along a portion of the periphery of the first display area R11. In the example in Figure 7, the periphery of the second display area R12 is hexagonal, and two sides of this hexagon are arranged along a pair of edges R111 and R112 of the periphery of the first display area R11. In particular, two sides of the periphery (hexagon) of the second display area R12 are parallel to the pair of edges R111 and R112. This allows the specific image Im200 displayed in the second display area R12 to be displayed relatively large, contributing to improved visibility of the specific image Im200.

[0096] Furthermore, in this embodiment, the specific image Im200 is an image relating to the captured images Im11, Im12, and Im13, which constitute a part of the overhead image Im100. In other words, as described above, the overhead image Im100 is generated by combining the captured images Im11, Im12, and Im13, and the second display processing unit 112 displays the image relating to the captured images Im11, Im12, and Im13 as the specific image Im200 in the second display area R12. The image relating to the captured images Im11, Im12, and Im13 may be, for example, at least a part of the captured images Im11, Im12, and Im13, or it may be an image to which some image processing such as coordinate transformation or filtering has been applied to the captured images Im11, Im12, and Im13. This allows the operator to view the images Im11, Im12, and Im13 along with the overhead image Im100, making it easier to check the situation around the aircraft 30.

[0097] Here, the specific image Im200 is an image relating to the captured image Im13 of the rear side of the work machine 3. In other words, the image relating to the captured image Im13, which is an image of the small area A13 behind the driver's unit 321 captured by the rear camera 343, is displayed as the specific image Im200 in the second display area R12. This makes it possible to cover the area behind the machine 30, which is often a blind spot for the operator, with the specific image Im200.

[0098] In this embodiment, the specific image Im200 is an image that is converted from the captured image Im13 in a different manner than the overhead view image Im100. That is, since the overhead view image Im100 is an image of the machine 30 viewed from above (above), the captured image Im13 captured by the rear camera 343 is transformed, such as by coordinate transformation, so that it becomes a plan view. In contrast, the captured image Im13 displayed as the specific image Im200 in the second display area R12 is transformed, such as by coordinate transformation, so that it becomes an image as seen from the perspective of the user (operator) riding in the work machine 3 (operator's unit 321), similar to the example in Figure 6. Therefore, the small area A13 displayed as the specific image Im200 is displayed with different conversion manners for the overhead view image Im100 and the specific image Im200, allowing the operator to check it more carefully.

[0099] Furthermore, the display screen Dp1 includes an object (icon Im10) that represents the positional relationship of the imaging range of the overhead view image Im100 as seen from the work machine 3. That is, as described above, in the center of the second region R2, icon Im10 is displayed, schematically representing the positional relationship of the imaging ranges (small areas A11, A12, A13) of the left camera 341, the right camera 342, and the rear camera 343 as seen from the machine 30. Here, since icon Im10 is placed at the machine's reference position P1 of the overhead view image Im100, the overhead view image Im100 is displayed around icon Im10 so as to correspond to the positional relationship when the machine 30 is viewed from above. In other words, the overhead image Im100 is displayed such that the captured image Im11 of small area A11 to the left of aircraft 30 is positioned to the left of icon Im10, the captured image Im12 of small area A12 to the right of aircraft 30 is positioned to the right of icon Im10, and the captured image Im13 of small area A13 behind aircraft 30 is positioned below icon Im10.

[0100] Thus, in this embodiment, the control method reflects the positional relationship of the overhead image Im100 to the icon Im10 on the display screen Dp1, as well as the positional relationship of the monitoring area A1 to the work machine 3 in real space. Therefore, the operator can easily intuitively understand which part of the monitoring area A1 the overhead image Im100 captures by looking at the display screen Dp1.

[0101] Furthermore, the display of the overhead image Im100 is not limited to the full-screen mode that displays the enlarged screen Dp12, but may also be performed in the normal mode that displays the home screen Dp11, which includes the first area R1 and the third area R3 to the tenth area R10 in addition to the second area R2.

[0102] [3.3] Overall Processing Next, the overall flow of the control method will be explained with reference to Figure 9. Figure 9 is a flowchart showing an example of the control method, specifically the process related to the display of the overhead image Im100.

[0103] As shown in Figure 9, the image acquisition unit 12 of the control system 1 acquires captured images Im11, Im12, and Im13 from the left camera 341, the right camera 342, and the rear camera 343 (S1). Then, the display processing unit 11 of the control system 1 generates an overhead view image Im100 based on the captured images Im11, Im12, and Im13 (S2).

[0104] The first display processing unit 111 of the display processing unit 11 displays the overhead image Im100 in the first display area R11 of the display screen Dp1, which has a shape in which a part of the circumferential direction as seen from the self-reference position P1 corresponding to the work machine 3 in a plan view is cut out as a notched area R13 (S3). Furthermore, the second display processing unit 112 of the display processing unit 11 displays at least a part of the specific image Im200 in the second display area R12 of the display screen Dp1, which is located within the notched area R13 (S4).

[0105] As a result, as shown in Figures 7 and 8, the overhead image Im100 and the specific image Im200 are displayed in the first display area R11 and the second display area R12, respectively, on the display screen Dp1. The control system 1 repeatedly executes the processes in steps S1 to S4 described above. However, the flowchart shown in Figure 9 is merely an example, and processes may be added or omitted as appropriate, or the order of processes may be changed as appropriate.

[0106] [4] Modified form The following lists some modifications of Embodiment 1. The modifications described below can be combined and applied as appropriate.

[0107] The control system 1 in this disclosure includes a computer system. The computer system mainly consists of one or more processors and one or more memories as hardware. The functions of the control system 1 in this disclosure are realized by the execution of a program recorded in the memory of the computer system by the processor. The program may be pre-recorded in the memory of the computer system, provided via a telecommunications line, or provided on a non-temporary recording medium such as a memory card, optical disk, or hard disk drive that can be read by the computer system. Furthermore, some or all of the functional parts included in the control system 1 may be composed of electronic circuits.

[0108] Furthermore, it is not essential for control system 1 to have at least some of its functions integrated into a single housing; the components of control system 1 may be distributed across multiple housings. Conversely, functions that are distributed across multiple devices (e.g., control system 1 and display device 2) in Embodiment 1 may be integrated into a single housing. Moreover, at least some of the functions of control system 1 may be implemented by the cloud (cloud computing) or the like.

[0109] Furthermore, the power source for the work machine 3 is not limited to a diesel engine; for example, it may be an engine other than a diesel engine, or a motor (electric motor), or a hybrid power source including both an engine and a motor (electric motor).

[0110] Furthermore, the display device 2 is not limited to a dedicated device, but may be a general-purpose terminal such as a laptop computer, tablet terminal, or smartphone. Moreover, the display unit 23 is not limited to a configuration that directly displays the display screen, such as a liquid crystal display or an organic EL display, but may also be configured to display the display screen by projection, such as a projector.

[0111] Furthermore, the input method for information to the operation unit 22 may be other than push-button switches, touch panels, and operation dials. For example, the operation unit 22 may use methods such as a keyboard, a pointing device such as a mouse, voice input, gesture input, or input of operation signals from another terminal.

[0112] Furthermore, the restriction processing performed by the restraint processing unit 14 is not limited to processing that prohibits (disables) the operation of the work machine 3 (such as turning). The restriction processing may be, for example, processing that reduces the speed of the operation of the work machine 3 (such as turning), processing that narrows the operating range of the work machine 3 (such as turning angle), or processing that limits the permissible area of ​​operation of the work machine 3.

[0113] Furthermore, the functions related to deterrence processing by the deterrence processing unit 14 are not essential, and the deterrence processing unit 14 can be omitted as appropriate. Moreover, the functions related to detection processing by the detection unit 13 are not essential in the first place, and the detection unit 13 can be omitted as appropriate.

[0114] Furthermore, in addition to or as an alternative to the three-screen display that shows the three images Im11, Im12, and Im13 of the monitoring area A1 in the second area R2 of the display screen Dp1, the display processing unit 11 may also perform a one-screen display or a two-screen display. In the one-screen display, the display processing unit 11 displays one of the images Im11, Im12, and Im13 in the second area R2, and in the two-screen display, it displays two of the images Im11, Im12, and Im13 in the second area R2. As an example of a two-screen display, the display processing unit 11 displays two images in the second area R2: the image Im12 captured by the right camera 342 and the image Im13 captured by the rear camera 343. The images Im12 and Im13 are arranged side by side in the second area R2, for example.

[0115] Furthermore, the sensors for detecting the object Ob1 in the monitoring area A1 surrounding the work machine 3 are not limited to the left camera 341, the right camera 342, and the rear camera 343, but may include one, two, or four or more cameras (image sensors). In addition, for example, the object Ob1 in the monitoring area A1 may be detected by a camera capable of capturing images in all directions as seen from the work machine 3, such as a 360-degree camera. Also, the sensors for detecting the object Ob1 in the monitoring area A1 may include, in addition to or instead of cameras, sensors such as a human presence sensor, sonar sensor, radar, or LiDAR (Light Detection and Ranging). Here, the sensor for detecting the object Ob1 in the monitoring area A1 may be a 3D sensor that measures the distance to the object Ob1 using the TOF (Time Of Flight) method, which measures the distance to the distance point based on the round-trip time it takes for light or sound to reach the distance point and return.

[0116] Furthermore, the object to be detected Ob1 may include, in addition to or instead of "people," moving objects such as vehicles (including other work machinery), structures such as walls and pillars, plants, animals, steps, ditches, or other obstacles.

[0117] Furthermore, it is not mandatory for the second display area R12 to be positioned above the first display area R11 on the display screen Dp1; for example, the second display area R12 may be positioned below the first display area R11 on the display screen Dp1. It is also not mandatory for there to be a gap between the first display area R11 and the second display area R12 on the display screen Dp1; the first display area R11 and the second display area R12 may be positioned without any gap.

[0118] It is not necessary for the specific image Im200 to be an image relating to the captured images Im11, Im12, and Im13, which constitute part of the overhead view image Im100. For example, the specific image Im200 may be information relating to the operating status of the work machine 3, such as the cooling water temperature and the hydraulic oil temperature. Furthermore, the specific image Im200 does not have to be an image relating to the captured image Im13 on the rear side of the work machine 3; for example, it may be an image relating to the captured image Im11 on the left side of the work machine 3. It is also not necessary for the conversion method of the specific image Im200 from the captured images Im11, Im12, and Im13 to be different from that of the overhead view image Im100; it may be the same conversion method as the overhead view image Im100.

[0119] Furthermore, it is not mandatory for the notch region R13 to be positioned in front of the work machine 3 in the overhead view image Im100; for example, the notch region R13 may be positioned behind the work machine 3 in the overhead view image Im100. It is also not mandatory for the display screen Dp1 to include an object (icon Im10) that represents the positional relationship of the imaging range of the overhead view image Im100 as seen from the work machine 3; the icon Im10 may be omitted as appropriate.

[0120] (Embodiment 2) As shown in Figures 10 and 11, the work machine 3 according to this embodiment differs from the work machine 3 according to Embodiment 1 in that the display content of the second area R2 of the display screen Dp1 is different. Hereinafter, components similar to those in Embodiment 1 will be denoted by common reference numerals and their descriptions will be omitted as appropriate.

[0121] In this embodiment, at least a part of the periphery of the second display area R12 is not arranged along a part of the periphery of the first display area R11. That is, in this embodiment, at least a part of the periphery of the second display area R12 is not arranged along a part of the periphery of the first display area R11. For example, in the example of FIG. 10, the periphery of the second display area R12 forms a rectangle, and a pair of adjacent corner portions of the rectangle face a pair of edges R111 and R112 of the periphery of the first display area R11, respectively. Also, in the example of FIG. 11, although the periphery of the second display area R12 forms a hexagon, two sides of the hexagon are inclined with respect to a pair of edges R111 and R112 of the periphery of the first display area R11. Thus, it is not essential that at least a part of the periphery of the second display area R12 is arranged along a part of the periphery of the first display area R11.

[0122] Also, in this embodiment, in the display screen Dp1, the distance from the rear reference position P2 corresponding to the rear of the work machine 3 to the notch area R13 in the circumferential direction centered on the own machine reference position P1 in the first display area R11 is different clockwise and counterclockwise. That is, as shown in FIGS. 10 and 11, when the rear reference position P2 is set at a position corresponding to the rear of the work machine 3 in the bird's-eye view image Im100 (directly below the own machine reference position P1), the clockwise distance L1 from the rear reference position P2 to the edge R111 is different from the counterclockwise distance L2 from the rear reference position P2 to the edge R112. In the example of FIG. 10, the clockwise distance L1 from the rear reference position P2 to the edge R111 is smaller than the counterclockwise distance L2 from the rear reference position P2 to the edge R112 (L1 < L2). In the example of FIG. 11, the clockwise distance L1 from the rear reference position P2 to the edge R111 is larger than the counterclockwise distance L2 from the rear reference position P2 to the edge R112 (L1 > L2).

[0123] Thus, the first display area R11 on which the overhead view image Im100 is displayed is configured asymmetrically on the display screen Dp1. For example, in the work machine 3, the driver unit 321 may be offset to one side of the left-right direction D3 of the machine body 30, and in such a case, the overhead view image Im100 will also be generated asymmetrically. According to the configuration of this embodiment, it is possible to display such asymmetrical overhead view image Im100 based on the actual positional relationship.

[0124] The configuration according to Embodiment 2 can be adopted in appropriate combination with the various configurations (including modified versions) described in Embodiment 1.

[0125] [Notes on the invention] The following is an overview of the invention extracted from the above-described embodiments. Note that each configuration and processing function described below can be selected and combined as desired.

[0126] <Note 1> Acquiring images of the monitoring area around the work machine, The overhead view image based on the aforementioned captured image is displayed in a first display area on the display screen of the display device, in which a part of the circumferential direction as seen from the reference position of the machine, which corresponds to the work machine in a plan view, is cut out as a notched area. The display screen includes displaying at least a portion of a specific image in a second display area located within the notched area. A method for controlling industrial machinery.

[0127] <Note 2> The second display area is positioned above the first display area on the display screen. Control method for the work machine described in Appendix 1.

[0128] <Note 3> In the aforementioned display screen, a gap is provided between the first display area and the second display area. A control method for the work machine described in Appendix 1 or 2.

[0129] <Note 4> In the display screen, at least a portion of the periphery of the second display area is arranged along a portion of the periphery of the first display area. A control method for the work machine described in any of the appendices 1 to 3.

[0130] <Note 5> The aforementioned specific image is an image relating to the captured image which constitutes a part of the overhead view image. A control method for the work machine described in any of the appendices 1 to 4.

[0131] <Note 6> The aforementioned specific image is an image relating to the captured image of the rear side of the work machine. Control method for the work machine described in Appendix 5.

[0132] <Note 7> The aforementioned specific image is an image that is different from the overhead view image in terms of the conversion method from the captured image. A control method for the work machine described in Appendix 5 or 6.

[0133] <Note 8> The aforementioned cutout region is positioned in the overhead view image at a location corresponding to the front of the work machine. A control method for the work machine described in any of the appendices 1 to 7.

[0134] <Note 9> The aforementioned display screen includes an object that represents the positional relationship of the imaging range of the overhead image as seen from the work machine. A control method for the work machine described in any of the appendices 1 to 8.

[0135] <Note 10> In the first display area, the distance from the rear reference position corresponding to the rear of the work machine to the notched area in the circumferential direction centered on the machine's reference position differs between clockwise and counterclockwise directions. A control method for the work machine described in any of the appendices 1 to 9.

[0136] <Note 11> The control method for the work machine described in any of the appendices 1 to 10, A control program for a work machine to be executed by one or more processors. [Explanation of Symbols]

[0137] 1. Control system for industrial machinery 2 Display device 3. Working Machines 12 Image acquisition unit 30 aircraft 111 First Display Processing Unit 112 Second display processing unit A1 Surveillance Area Dp1 display screen im10 Icon (Object) Im11, Im12, Im13 Acquired Images Im100 overhead view Im200 Specific Image L1,L2 distance Ob1 Detected object P1 Player's reference position P2 Rear reference position R11 1st display area R12 2nd display area R13 Notch area

Claims

1. Acquiring images of the monitoring area around the work machine, The overhead view image based on the aforementioned captured image is displayed in a first display area on the display screen of the display device, in which a part of the circumferential direction as seen from the reference position of the machine, which corresponds to the work machine in a plan view, is cut out as a notched area. The display screen includes the ability to display the entirety of a specific image in a fan-shaped second display area located within the notched area. A method for controlling industrial machinery.

2. In the display screen, at least a portion of the periphery of the second display area is arranged along a portion of the periphery of the first display area. A method for controlling a work machine according to claim 1.

3. The aforementioned specific image is an image relating to the captured image which constitutes a part of the overhead view image. A method for controlling a work machine according to claim 1 or 2.

4. The aforementioned specific image is an image relating to the captured image of the rear side of the work machine. A method for controlling a work machine according to claim 3.

5. A control method for a work machine according to claim 1 or 2, A control program for a work machine to be executed by one or more processors.

6. An image acquisition unit that acquires images of the monitoring area around the work machine, A first display processing unit that displays an overhead view image based on the aforementioned captured image in a first display area on a display screen displayed on a display device, the first display area having a shape in which a part of the circumferential direction as seen from the machine's reference position corresponding to the work machine in a plan view is cut out as a notched area, The display screen includes a second display processing unit that displays the entirety of a specific image in a fan-shaped second display area located within the notched area. Control system for industrial machinery.

7. A control system for a work machine according to claim 6, The unit comprises a body on which the aforementioned display device is mounted, A type of machinery used for industrial work.

Images