Method for controlling work machine, control program for work machine, control system for work machine, and work machine

Abstract

Provided are a method for controlling a work machine, a control program for a work machine, a control system for a work machine, and a work machine, whereby an operator can easily and intuitively grasp the operating state of a probe device. A method for controlling a work machine includes: acquiring a captured image (Im100) of a monitored area around the work machine; and causing a display screen (Dp1) including the captured image (Im100) to be displayed on a display device. The method for controlling the work machine further includes, when a detection processing unit for detecting the object to be detected (Ob1) in the monitoring region is active, displaying an active target (X1), which moves in at least one of the circumferential direction and the radial direction of a virtual circle centered on a reference point set for the captured image (Im100), on the captured image (Im100) in the display screen (Dp1) so as to overlap the active target (X1) on the captured image (Im100).

Description

Technical Field

[0001] The present invention relates to a control method for a working machine, a control program for a working machine, a control system for a working machine, and a working machine used for a working machine having a function of detecting a detection target object in a surrounding monitoring area. Background Art

[0002] As a related art, a working machine (excavator) capable of detecting objects existing in the surroundings is known (for example, refer to Patent Document 1). The working machine involved in the related art includes a camera mounted on an upper rotating body, a display device, and an object detection device. The object detection device detects a specified object within a specified detection range set around the excavator. In the working machine, the image displayed by the display device includes a first area for displaying a camera image and a second area for displaying the working status of the object detection function.

[0003] Patent Document 1: International Publication No. 2018 / 008542

[0004] In the above-mentioned related technologies, the display device displays a large amount of information. For the operator, it is possible to ignore the working status of the detection processing unit (object detection function), or it is necessary to look at the display device in order not to ignore the working status of the detection device. Summary of the invention

[0005] An object of the present invention is to provide a control method for a working machine, a control program for a working machine, a control system for a working machine, and a working machine, which allow an operator to easily and intuitively grasp the operating status of a detection processing unit.

[0006] A control method for a working machine involved in one form of the present invention comprises: obtaining a captured image of a monitoring area around the working machine; displaying a display screen including the captured image on a display device; and when a detection processing unit for detecting a detection object in the monitoring area is effective, displaying a valid target moving in at least one of the circumferential and radial directions of an imaginary circle on the captured image in the display screen in an overlapping manner, the imaginary circle being centered on a reference point set for the captured image.

[0007] A control program for a working machine according to one aspect of the present invention is a program for causing one or more processors to execute the control method of the working machine described above.

[0008] A control system for a working machine according to one aspect of the present invention comprises an image acquisition unit and a display processing unit. The image acquisition unit acquires a captured image of a monitoring area around the working machine. The display processing unit displays a display screen including the captured image on a display device. When the detection processing unit for detecting a detection object in the monitoring area is effective, the display processing unit displays a valid target moving in at least one of the circumferential direction and the radial direction of an imaginary circle on the captured image in the display screen, and the imaginary circle is centered on a reference point set for the captured image.

[0009] A working machine according to one aspect of the present invention includes the working machine control system described above and a machine body on which the display device is mounted.

[0010] According to the present invention, it is possible to provide a working machine control method, a working machine control program, a working machine control system, and a working machine that allow an operator to easily and intuitively understand the operating status of a detection processing unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a schematic perspective view showing the overall structure of the working machine according to the first embodiment.

[0012] Figure 2 This is a schematic diagram showing a hydraulic circuit and the like of a working machine according to the first embodiment.

[0013] Figure 3 This is a schematic plan view schematically showing a monitoring area and the like set around the working machine when the working machine according to the first embodiment is viewed from above.

[0014] Figure 4 This is a schematic external view of a display device that displays a display screen by the working machine control system according to the first embodiment.

[0015] Figure 5 This is a diagram showing an example of a display screen displayed by the working machine control system according to the first embodiment.

[0016] Figure 6 This is a diagram showing a display example of the second area in the display screen displayed by the working machine control system according to the first embodiment.

[0017] Figure 7 This is a diagram showing a display example of the second area in the display screen displayed by the working machine control system according to the first embodiment.

[0018] Figure 8This is a diagram showing an example of the operation of a valid object in the second area of ​​the display screen displayed by the working machine control system according to the first embodiment.

[0019] Fig. 9 This is a flowchart showing an operation example of the working machine control system according to the first embodiment.

[0020] Fig.10 This is a diagram showing an example of the operation of a valid object in the second area of ​​the display screen displayed by the working machine control system according to the second embodiment.

[0021] Description of Reference Numerals

[0022] 1...control system for working machine; 2...display device; 3...working machine; 5...detection device; 11...display processing unit; 14...image acquisition unit; 30...machine body; A1...monitoring area; C1...imaginary circle; E101...first end; E102...second end; Dp1...display screen; Im10...machine body target; Im100...captured image; Ob1...detection object; P1...reference point; X1...effective target. DETAILED DESCRIPTION

[0023] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following embodiment is an example of a concrete embodiment of the present invention and is not intended to limit the technical scope of the present invention.

[0024] (Implementation Method 1)

[0025] [1] Overall structure

[0026] like Figure 1 As shown in FIG. 1 , the working machine 3 according to the present embodiment includes a traveling unit 31, a rotating unit 32, and a working unit 33 in a machine body 30. Figure 2 As shown, the working machine 3 further includes a working machine control system 1 (hereinafter also simply referred to as a "control system 1"). In addition, the machine body 30 further includes a display device 2, an operating device, and the like.

[0027] The so-called "working machinery" in the present disclosure refers to various working machines, as an example, working vehicles such as backhoe excavators (including hydraulic excavators, mini excavators, etc.), wheel loaders and carriers. The working machinery 3 has a working unit 33, and the working unit 33 is configured to be able to perform at least one operation including a lifting operation. The working machinery 3 is not limited to a "vehicle", for example, it can also be a working ship, a working aircraft such as a drone or a multi-rotor helicopter, etc. In addition, the working machinery 3 is not limited to engineering machinery (construction machinery), for example, it can also be agricultural machinery (agricultural machinery) such as a rice transplanter, a tractor or a combine harvester. In this embodiment, unless otherwise specified, the following situation is cited as an example for explanation: the working machinery 3 is a backhoe excavator with a lifting function (with a crane function), and in addition to the lifting operation, it can also perform excavation operations, land leveling operations, trenching operations or loading operations as operations.

[0028] In addition, in the present embodiment, for the sake of convenience, the vertical direction when the working machine 3 is in a usable state is defined as the up-down direction D1. And, when the rotating part 32 is not rotating, the front-to-back direction D2 and the left-to-right direction D3 are defined based on the direction observed by the user (operator) riding on the working machine 3 (driving part 321). In other words, the directions used in the present embodiment are all directions defined based on the body 30 of the working machine 3, and the direction in which the body 30 moves when the working machine 3 moves forward is the "front", and the direction in which the body 30 moves when the working machine 3 moves backward is the "rear". Similarly, the direction in which the front end of the body 30 moves when the working machine 3 rotates right is the "right", and the direction in which the front end of the body 30 moves when the working machine 3 rotates left is the "left". However, these directions are not intended to limit the use direction of the working machine 3 (the direction when in use).

[0029] The working machine 3 includes an engine serving as a power source. In the working machine 3, for example, a hydraulic pump 41 (see Figure 2 ), the hydraulic pump 41 supplies hydraulic oil to the hydraulic actuators (including the hydraulic motor 43 and the hydraulic cylinder 44) of each part of the machine body 30, thereby driving the machine body 30. In addition, the working machine 3 is controlled, for example, by a user (operator) riding on the driving part 321 of the machine body 30 operating an operating lever of an operating device.

[0030] In this embodiment, the working machine 3 is assumed to be a riding backhoe as described above, so the working unit 33 is driven according to the operation of the user (operator) riding on the driving unit 321 to perform work such as excavation. The driving unit 321 for the user to ride on is provided on the swing unit 32.

[0031] The travel unit 31 has a travel function and is configured to be able to travel on the ground (including turning). The travel unit 31 includes, for example, a pair of left and right crawler tracks 311 and a bulldozer 312. The travel unit 31 also includes a travel hydraulic motor 43 (hydraulic actuator) for driving the crawler tracks 311.

[0032] The slewing section 32 is located above the traveling section 31 and is configured to be slewing relative to the traveling section 31 about a rotation axis along the vertical direction. The slewing section 32 has a slewing hydraulic motor (hydraulic actuator) and the like. In addition to the driving section 321, the slewing section 32 is also equipped with an engine and a hydraulic pump 41 and the like. In addition, a boom bracket 322 for mounting the working section 33 is provided at the front end of the slewing section 32.

[0033] The working unit 33 is configured to be able to perform operations including lifting operations. The working unit 33 is supported by the boom bracket 322 of the rotating unit 32 to perform operations. The working unit 33 has a bucket 331, a boom 332, and a boom 333. The working unit 33 also has a hydraulic actuator (including a hydraulic cylinder 44 and a hydraulic motor, etc.) for driving each part.

[0034] The bucket 331 is a kind of accessory (working tool) mounted on the body 30 of the working machine 3, and is composed of any tool selected from a variety of accessories according to the content of the work. As an example, the bucket 331 is detachably mounted on the body 30 and replaced according to the content of the work. As accessories for the working machine 3, in addition to the bucket 331, there are various tools such as a crusher, an auger, a crusher, a fork tooth, a fork claw, a steel frame cutter, an asphalt cutter, a lawn mower, a tiller, a mulcher, a tiltrotator (Japanese: チルトローテータ) and a tamping machine. The working unit 33 drives the bucket 331 using power from a driving device to perform the work.

[0035] The boom 332 is supported by the boom bracket 322 of the slewing portion 32 so as to be rotatable. Specifically, the boom 332 is supported by the boom bracket 322 so as to be rotatable around a rotation axis along the horizontal direction. The boom 332 has a shape extending upward from a base end portion supported by the boom bracket 322. The arm 333 is connected to the front end of the boom 332. The arm 333 is supported so as to be rotatable relative to the boom 332 around a rotation axis along the horizontal direction. The bucket 331 is mounted at the front end of the arm 333.

[0036] The working part 33 receives power from the engine as a power source to operate. Specifically, the engine drives the hydraulic pump 41, and the hydraulic pump 41 supplies working oil to the hydraulic actuator (hydraulic cylinder 44, etc.) of the working part 33, so that each part of the working part 33 (bucket 331, boom 332 and arm 333) operates.

[0037] In this embodiment, the working part 33 has a multi-joint structure in which the boom 332 and the arm 333 are independently rotatable. That is, the multi-joint working part 33 including the boom 332 and the arm 333 can be extended or folded as a whole by rotating the boom 332 and the arm 333 around a rotation axis in the horizontal direction.

[0038] The travel unit 31 and the revolving unit 32 also operate by receiving power from the engine as a power source, similarly to the working unit 33. That is, the hydraulic pump 41 supplies hydraulic oil to the hydraulic motor 43 of the travel unit 31 and the hydraulic motor 43 of the revolving unit 32, so that the revolving unit 32 and the travel unit 31 operate.

[0039] As described above, the engine functions as a power source for supplying power to each part. Here, the engine is mounted on the rotary part 32 together with the hydraulic pump 41 and the like. In the present embodiment, as an example, the engine is a diesel engine. The engine is driven by being supplied with fuel (here, light oil) from a fuel tank.

[0040] Here, the machine body 30 is equipped with a camera or the like that captures images of the surroundings of the machine body 30 for detecting a monitoring area A1 (see FIG. Figure 3 ) in the detection object Ob1 (refer to Figure 3 ) of various sensor types (including cameras). In this embodiment, as an example, Figure 3 As shown in FIG. 1 , a plurality of (here, three) cameras (photographing devices) including a left camera 341, a right camera 342, and a rear camera 343 are mounted on the rotating portion 32 of the machine body 30. Figure 3 As shown, the detection device 5 for detecting the detection object Ob1 in the monitoring area A1 is mounted on the rotating part 32 of the machine body 30 .

[0041] The left camera 341 , the right camera 342 , and the rear camera 343 are connected to the control system 1 , and output images captured by each of them to the control system 1 . Figure 3 It is a top view of the working machine 3 viewed from above, schematically showing a monitoring area A1 set around the working machine 3, a detection target object Ob1, and a body 30 of the working machine 3 (including a left camera 341, a right camera 342, and a rear camera 343).

[0042] The left camera 341, the right camera 342, and the rear camera 343 are respectively arranged to face left, right, and rear with respect to the driving unit 321 as a reference so as to be able to photograph the monitoring area A1 which is the left, right, and rear as viewed from the operator of the driving unit 321 of the revolving unit 32. Figure 3As shown, the monitoring area A1 includes a plurality of (here, three) small areas A11, A12, and A13, and the left camera 341 photographs the small area A11 (left area) which is on the left side from the perspective of the operator riding on the driving unit 321. Similarly, the right camera 342 photographs the small area A12 (right area) which is on the right side from the perspective of the operator riding on the driving unit 321, and the rear camera 343 photographs the small area A13 (rear area) which is on the rear side from the perspective of the operator riding on the driving unit 321. Thus, the left camera 341, the right camera 342, and the rear camera 343 can cover the side (left and right) and rear side which are easily blind spots for the operator.

[0043] The detection device 5 is connected to the control system 1, and outputs the detection result of the detection object Ob1 in the monitoring area A1 to the control system 1. Here, as an example, the detection device 5 is set toward the rear so that the monitoring area A1 that can be photographed by the left camera 341, the right camera 342, and the rear camera 343 can be used as an object, and the detection object Ob1 in the monitoring area A1 can be detected. That is, the detection range of the detection device 5 for the detection object Ob1 is the same as the photographable range (monitoring area A1) of the left camera 341, the right camera 342, and the rear camera 343.

[0044] In this embodiment, the detection device 5 has a sensor 51 (see Figure 2 ) and the imaging unit 52 (refer to Figure 2 ). The sensor 51 is a three-dimensional sensor that measures the distance to the detection object Ob1 by using a TOF (Time Of Flight) method that measures the distance to the distance measurement point based on the round-trip time for radio waves, light, or sound to reach the distance measurement point and return. The sensor 51 is, for example, a distance measurement sensor that uses radio waves as a medium to determine the distance to the detection object Ob1, the direction of the detection object Ob1, etc., like a millimeter wave radar. The shooting unit 52 is a camera (including an image sensor and an optical element) that captures an image of the detection area (monitoring area A1) of the sensor 51. In this way, the detection device 5 formed by combining the sensor 51 and the shooting unit 52 can determine the three-dimensional position and attributes (shape, size, color, movement, etc.) of the detection object Ob1 when the detection object Ob1 exists in the monitoring area A1.

[0045] That is, the detection result of the detection device 5 can include the presence or absence of the detection object Ob1 in the monitoring area A1, the position of the detection object Ob1 in the monitoring area A1 when the detection object Ob1 exists in the monitoring area A1, the attributes of the detection object Ob1, etc.

[0046] In summary, the detection device 5 detects the detection object Ob1 in the monitoring area A1 around the working machine 3. The detection device 5 determines whether the detection object Ob1 in the monitoring area A1 exists (is there or not), and outputs a detection result indicating whether the detection object Ob1 exists in the monitoring area A1. In the present embodiment, as an example, the detection object Ob1 is a "person". That is, when the working machine 3 moves or the "person" around the working machine 3 moves and the "person" intrudes into the monitoring area A1 around the working machine 3, the detection device 5 detects the "person" as the detection object Ob1. In the case where there are multiple detection objects Ob1 in the monitoring area A1, the detection device 5 can also detect the number (number of people) of the detection objects Ob1.

[0047] exist Figure 2 , the hydraulic circuit and the electrical circuit (electrical connection relationship) of the working machine 3 according to the present embodiment are schematically shown. Figure 2 In the figure, the solid line represents the high-pressure (for working oil) oil circuit, the dotted line represents the low-pressure (for pilot oil) oil circuit, and the dot-dash arrows represent the paths of electrical signals.

[0048] like Figure 2 As shown, the working machine 3 includes a hydraulic pump 41 and a hydraulic motor 43 ( Figure 2 In addition to the hydraulic cylinder 44 (not shown in the figure) and the hydraulic cylinder 44, it also includes a pilot pump 42, a remote control valve 45, a control valve 46, a direction switching valve (regulating valve) 47, etc.

[0049] The hydraulic oil from the hydraulic pump 41 driven by the engine is supplied to the hydraulic motor 43 of the travel unit 31, the hydraulic motor of the swing unit 32, and the hydraulic cylinder 44 of the working unit 33. Thus, the hydraulic actuators such as the hydraulic motor 43 and the hydraulic cylinder 44 are driven.

[0050] The hydraulic actuators such as the hydraulic motor 43 and the hydraulic cylinder 44 are provided with a pilot-operated direction switching valve 47 capable of switching the direction and flow rate of the hydraulic oil from the hydraulic pump 41. The direction switching valve 47 is driven by pilot oil supplied as an input command from the pilot pump 42.

[0051] Here, for example, a remote control valve 45 is provided in the supply path of the pilot oil of the direction switching valve 47 corresponding to the hydraulic cylinder 44 of the working part 33. The remote control valve 45 outputs the operation instruction of the working part 33 according to the operation of the operating lever. The operation instruction instructs the expansion action and the contraction action of the working part 33. In addition, an electromagnetic control valve 46 (solenoid valve) is inserted between the remote control valve 45 and the pilot pump 42. The control valve 46 is connected to the power supply 351 via the cut-off relay 352 and the cut-off switch 353, and operates according to the supply current from the power supply 351.

[0052] Similarly, a remote control valve is also provided in the supply path of the pilot oil of the direction switching valve corresponding to the hydraulic motor 43 of the travel part 31. The remote control valve outputs the travel operation instruction of the travel part 31 according to the operation of the operating lever. The travel operation instruction indicates the travel action (forward or backward, etc.) of the travel part 31. In addition, a remote control valve is also provided in the supply path of the pilot oil of the direction switching valve corresponding to the hydraulic motor of the rotating part 32. The remote control valve outputs the rotation operation instruction of the rotating part 32 according to the operation of the operating lever. The rotation operation instruction indicates the rotation action (left rotation or right rotation, etc.) of the rotating part 32. In addition, an electromagnetic control valve 46 (solenoid valve) is also inserted between these remote control valves and the pilot pump 42. The control valve 46 is connected to the power supply 351 via the cut-off relay 352 and the cut-off switch 353, and operates according to the supply current from the power supply 351.

[0053] The control valve 46 opens the flow path of the pilot oil from the pilot pump 42 to the remote control valve 45 when it is energized, i.e., supplied with current, and blocks the flow path of the pilot oil when it is not energized, i.e., when the supply current is cut off. Therefore, by cutting off the supply current to the control valve 46, the hydraulic actuator corresponding to the remote control valve 45 becomes unable to be driven, and the output of the hydraulic actuator is forcibly stopped regardless of the operation of the operating lever.

[0054] Here, the cut-off relay 352 is connected to the control system 1, and switches on / off according to the control signal (electrical signal) from the control system 1. The cut-off switch 353 switches on / off according to the operation of the cut-off lever, for example, it is turned on when the cut-off lever is operated to the bottom. Therefore, when the cut-off relay 352 and the cut-off switch 353 are both turned on, the control valve 46 becomes energized, and the flow path of the pilot oil from the pilot pump 42 to the remote control valve 45 is opened, so that the hydraulic actuator is driven according to the operation of the operating lever. In contrast, when at least one of the cut-off relay 352 and the cut-off switch 353 is disconnected, the control valve 46 becomes non-energized, and the flow path of the pilot oil is cut off, so that the hydraulic actuator cannot be driven.

[0055] For example, when at least one of the cut-off relay 352 and the cut-off switch 353 connected to the control valve 46 between the remote control valve corresponding to the hydraulic motor of the rotary unit 32 and the pilot pump 42 is disconnected, the hydraulic motor of the rotary unit 32 cannot be driven. In this state, the output of the hydraulic actuator (the hydraulic motor of the rotary unit 32) is forcibly stopped regardless of the operation of the operating lever, so the rotary motion of the rotary unit 32 is prohibited.

[0056] The control system 1 is mainly composed of a computer system having one or more processors such as a CPU (Central Processing Unit) and one or more memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and performs various processes (information processing). In the present embodiment, the control system 1 is an integrated controller that controls the entire working machine 3, for example, it is composed of an electronic control unit (ECU). However, the control system 1 can also be provided separately from the integrated controller. Details about the control system 1 are described in the column "[2] Structure of the control system".

[0057] The display device 2 is disposed in the driving section 321 of the machine body 30, and is a user interface for receiving operation inputs by the user (operator) and outputting various information to the user. The display device 2 receives various operations performed by the user by, for example, outputting electrical signals corresponding to the user's operations. Thus, the user (operator) can visually confirm the display screen Dp1 (see FIG. 1 ) displayed on the display device 2. Figure 4 ), and in addition, the display device 2 can be operated as needed.

[0058] like Figure 2 As shown, the display device 2 includes a control unit 21, an operation unit 22, and a display unit 23. The display device 2 is configured to be able to communicate with the control system 1 and to transmit and receive data between itself and the control system 1. In this embodiment, the display device 2 is a dedicated device for the working machine 3 as an example.

[0059] The control unit 21 controls the display device 2 according to data from the control system 1 . Specifically, the control unit 21 outputs an electric signal according to a user operation received by the operation unit 22 , and displays the display screen Dp1 generated by the control system 1 on the display unit 23 .

[0060] The operation unit 22 is a user interface for receiving operation inputs from a user (operator) on the display screen Dp1 displayed on the display unit 23. The operation unit 22 outputs, for example, Figure 4 ) to receive various operations performed by the user U1. In this embodiment, as an example, Figure 4 As shown, the operation unit 22 includes a plurality of (six in this case) mechanical push button switches 221 to 226. The plurality of push button switches 221 to 226 are arranged along the periphery of the display area of ​​the display unit 23 close to the display area (in Figure 4The plurality of button switches 221 to 226 are associated with items displayed on the display screen Dp1 described later, and any item on the display screen Dp1 is operated (selected) by operating any one of the plurality of button switches 221 to 226.

[0061] In addition, the operation unit 22 may include a touch panel, an operation dial, etc. In this case as well, any one of the items on the display screen Dp1 is operated (selected) by operating the operation unit 22 .

[0062] The display unit 23 is a user interface such as a liquid crystal display or an organic EL display that displays various information and is used to present information to the user U1 (operator). The display unit 23 presents various information to the user by displaying. In this embodiment, as an example, the display unit 23 is a full-color liquid crystal display with a backlight, such as Figure 4 The display area shown has a "horizontal length" that is longer in the horizontal direction.

[0063] The display device 2 displays various information to the user U1 (operator) operating the working machine 3 through the display screen Dp1. That is, the user U1 operating the working machine 3 can visually obtain various information related to the working machine 3 by observing the display screen Dp1 displayed on the display device 2. As an example, the display device 2 displays information related to the operating state of the working machine 3, such as the cooling water temperature and the working oil temperature, so that the user U1 can confirm the information related to the operating state of the working machine 3 required for operating the working machine 3 through the display device 2. In addition, the display device 2 can also display the peripheral image of the working machine 3 (the image of the monitoring area A1) captured by the left camera 341, the right camera 342, and the rear camera 343 on the display screen Dp1. As a result, when operating the working machine 3, the user U1 (operator) can confirm the conditions of the side and rear of the working machine 3, which are easy to become blind spots when viewed from the driving unit 321, through the display screen Dp1 displayed on the display device 2.

[0064] Furthermore, the working machine 3 includes a sound output unit 36 ​​(see Figure 2 ). The sound output unit 36 ​​includes a buzzer or a speaker, etc., which receives an electrical signal and outputs a sound. The sound output unit 36 ​​is connected to the control system 1, and outputs a sound such as a buzzer or a voice according to a sound control signal from the control system 1. In this embodiment, the sound output unit 36 ​​is provided in the driving unit 321 of the body 30 in the same manner as the display device 2. The sound output unit 36 ​​can also be provided integrally with the display device 2.

[0065] In addition to the above-mentioned structure, the machine body 30 is also equipped with an operating lever, a stop lever, a communication terminal, a fuel tank, a battery, etc. In addition, the machine body 30 is equipped with sensors for monitoring the operating state of the machine body 30, such as a cooling water temperature sensor, a working oil temperature sensor, a tachometer for measuring the engine speed, and a timer for measuring the operating time. In addition, the machine body 30 is also equipped with sensors for detecting the state of the stop lever and the start key switch, etc.

[0066] [2]Control system structure

[0067] Next, refer to Figure 2 The structure of the control system 1 involved in the present embodiment is described. The control system 1 controls the display device 2 to display the display screen Dp1 on the display device 2. In the present embodiment, the display device 2 is mounted on the body 30 of the working machine 3 as described above. The control system 1 is a structural element of the working machine 3, and constitutes the working machine 3 together with the body 30 and the like. In other words, the working machine 3 involved in the present embodiment at least includes: the control system 1; and the body 30 (including the traveling part 31, the rotating part 32, and the working part 33) on which the display device 2 is mounted.

[0068] The "screen" such as the display screen Dp1 in the present disclosure refers to an image (image) displayed by the display device 2, including images, graphics, photos, textual expressions, and dynamic images. That is, the control system 1 can display the display screen Dp1 including images and other information related to the operating state of the working machine 3, such as the cooling water temperature and the working oil temperature, on the display device 2. Here, when the display screen Dp1 includes dynamic images, the display screen Dp1 is not a constant image, but includes an image that changes from moment to moment.

[0069] like Figure 2 As shown, the control system 1 includes a display processing unit 11, a containment processing unit 12, a switching processing unit 13, an image acquisition unit 14, and a detection processing unit 15. In this embodiment, as an example, the control system 1 uses a computer system having one or more processors as a main structure, so that the above-mentioned multiple functional units (display processing unit 11, etc.) are implemented by executing a control program for a working machine through one or more processors. The above-mentioned multiple functional units included in the control system 1 can be dispersedly arranged in multiple housings or in one housing.

[0070] The control system 1 is configured to be able to communicate with the devices provided in each part of the machine body 30. That is, the control system 1 is connected to at least the display device 2, the detection device 5, the sound output unit 36, the cutoff relay 352, the left camera 341, the right camera 342, and the rear camera 343. Thus, the control system 1 can control the display device 2 and the sound output unit 36, control the cutoff relay 352 to control the control valve 46, and obtain the detection result of the detection device 5 and the image captured by the left camera 341, the right camera 342, and the rear camera 343. Here, the control system 1 can directly transmit and receive various information (data) with each device, or can do so indirectly via a repeater or the like.

[0071] The image acquisition unit 14 performs an image acquisition process of acquiring a captured image of the monitoring area A1 around the working machine 3. In the present embodiment, the image acquisition unit 14 acquires the output of the left camera 341, the right camera 342, and the rear camera 343 from the left camera 341, the right camera 342, and the rear camera 343 periodically or irregularly. That is, the image acquisition unit 14 acquires image data (captured images) of the monitoring area A1 (each small area A11, A12, A13) around the working machine 3. The data acquired by the image acquisition unit 14 is stored in, for example, a memory or the like.

[0072] The detection processing unit 15 performs detection processing for detecting the detection object Ob1 in the monitoring area A1. In this embodiment, the detection device 5 detects the detection object Ob1 in the monitoring area A1, so the detection processing unit 15 detects the detection object Ob1 in the monitoring area A1 by acquiring the detection result of the detection device 5 from the detection device 5.

[0073] The containment processing unit 12 performs containment processing to contain the movement of the working machine 3 based on the detection result of the detection device 5. In the present embodiment, when the detection result of the detection device 5 indicates that there is a detection object Ob1 (here, a person) in the monitoring area A1, the containment processing unit 12 performs containment processing. The so-called "containment processing" in the present disclosure refers to processing that acts in a certain direction of suppressing the movement of the working machine 3. As an example, the containment processing includes the following processing: indirectly contain the movement of the working machine 3 by warning the user U1 (operator) operating the working machine 3 through sound or light (including display). In addition, the containment processing includes the following processing: directly contain the movement of the working machine 3 by controlling the travel part 31, the rotation part 32, and the working part 33 of the working machine 3.

[0074] In the present embodiment, the containment processing unit 12 includes a sound output processing unit 121 and a restriction processing unit 122 .

[0075] When there is a detection object Ob1 in the monitoring area A1, the sound output processing unit 121 outputs a report sound from the sound output unit 36 ​​by controlling the sound output unit 36. That is, in the present embodiment, the containment processing includes a sound output processing of outputting a report sound. The report sound may be a simple buzzer sound or a voice such as "Please pay attention". In addition, the report sound may also change according to the detection result of the detection device 5 (the distance from the machine body 30 to the detection object Ob1, etc.). Thus, the user U1 (operator) operating the working machine 3 is warned by means of the report sound, so that the movement of the working machine 3 can be indirectly contained, and thus the operating freedom of the working machine 3 is high. That is, the user U1 operates the working machine 3 while paying attention to the detection object Ob1, so that the operation of the working machine 3 can be continued while avoiding contact with the detection object Ob1.

[0076] When the detection object Ob1 exists in the monitoring area A1, the restriction processing unit 122 controls the cut-off relay 352 to turn off the cut-off relay 352. As a result, the control valve 46 connected to the power supply 351 via the cut-off relay 352 becomes non-energized, and the output of the hydraulic actuator corresponding to the control valve 46 is forcibly stopped. That is, in the present embodiment, the containment processing includes a restriction processing for restricting the action of the working machine 3. The so-called "restriction processing" in the present disclosure refers to a processing that acts in a direction of restricting the action of the working machine 3. As an example, the restriction processing includes a processing of prohibiting the driving action of the driving unit 31 (impossible to perform driving action), prohibiting the turning action of the turning unit 32 (impossible to perform turning action), and prohibiting the action of the working unit 33 (impossible to perform work). As a result, regardless of the operation of the user U1 (operator), the action of the working machine 3 can be forcibly restricted. That is, the contact between the machine body 30 and the detection object Ob1 caused by the action of the working machine 3 can be avoided.

[0077] Here, the restriction processing performed by the restriction processing unit 122 includes at least a processing for restricting the rotation action of the rotation unit 32. Specifically, the restriction processing unit 122 is configured to be able to control the cut-off relay 352 connected to the control valve 46 corresponding to the hydraulic motor of the rotation unit 32, and to disconnect the cut-off relay 352 when the detection object Ob1 exists in the monitoring area A1. As a result, when the detection object Ob1 exists in the monitoring area A1, the hydraulic motor of the rotation unit 32 becomes undriveable, and if the rotation unit 32 is in the rotation action, the rotation unit 32 is urgently stopped, and if the rotation unit 32 is not in the rotation action, the rotation action of the rotation unit 32 is prohibited. That is, in the present embodiment, the working machine 3 includes: a traveling unit 31; and a rotation unit 32 capable of rotating relative to the traveling unit 31. The restriction processing at least restricts the rotation action of the rotation unit 32. Thus, when the detection object Ob1 is present in the monitoring area A1 that is a blind spot for the user U1 (operator), contact between the machine body 30 and the detection object Ob1 due to the rotation of the rotation unit 32 can be avoided.

[0078] The switching processing unit 13 switches the detection processing unit 15 between validity and invalidity. That is, the switching processing unit 13 switches the detection processing unit 15 between validity and invalidity of the function related to the detection processing of the detection object Ob1 in the monitoring area A1. In short, the detection processing unit 15 is not always valid, and can be switched between validity and invalidity. If the detection processing unit 15 is valid, then when the detection object Ob1 exists in the monitoring area A1, the detection object Ob1 is detected by the detection processing unit 15, and thus the containment processing by the containment processing unit 12 is performed. On the other hand, if the detection processing unit 15 is invalid, then even when the detection object Ob1 exists in the monitoring area A1, the detection object Ob1 is not detected by the detection processing unit 15, and thus the containment processing by the containment processing unit 12 is not performed.

[0079] In the present embodiment, the switching processing unit 13 switches the detection processing unit 15 between being effective and being ineffective according to whether or not to operate the detection device 5. That is, the switching processing unit 13 operates the detection device 5 when the detection processing unit 15 is effective, and does not operate the detection device 5 when the detection processing unit 15 is ineffective. Thus, when the detection processing unit 15 is ineffective, the power consumption caused by the detection device 5 can be suppressed.

[0080] However, the present invention is not limited to this structure, and the switching processing unit 13 may also disable the detection processing unit 15 by, for example, disabling the detection processing (acquisition of the detection result from the detection device 5) of the detection processing unit 15 or disabling the acquired detection result. In this case, when the detection processing unit 15 is disabled, although the detection device 5 itself operates, the containment processing by the containment processing unit 12 is not performed even if the detection target object Ob1 exists in the monitoring area A1. In short, the switching processing unit 13 may also disable the detection processing unit 15 by disabling the processing using the detection result of the detection device 5.

[0081] In the present embodiment, as an example, the switching between enabling and disabling of the detection processing unit 15 is performed by the user U1 (operator) operating the display device 2. That is, if the user U1 operates the operation unit 22 of the display device 2 so as to enable the detection processing unit 15, the switching processing unit 13 accepts the operation and enables the detection processing unit 15. On the other hand, if the user U1 operates the operation unit 22 of the display device 2 so as to disable the detection processing unit 15, the switching processing unit 13 accepts the operation and disables the detection processing unit 15.

[0082] In addition, in the present embodiment, the containment processing performed by the containment processing unit 12 includes the sound output processing performed by the sound output processing unit 121 and the restriction processing performed by the restriction processing unit 122. In this way, the containment processing includes a plurality of specific processing (sound output processing and restriction processing, etc.) for containing the action of the working machine 3. Here, the containment processing can also be switched to be effective and ineffective independently for each specific processing. That is, the switching processing unit 13 can also switch the sound output processing unit 121 and the restriction processing unit 122 in the containment processing unit 12 independently to be effective and ineffective. As an example, the sound output processing unit 121 can be enabled and the restriction processing unit 122 can be disabled, or the sound output processing unit 121 can be disabled and the restriction processing unit 122 can be enabled. In this way, only the required specific processing can be enabled according to the situation, and the freedom of the containment processing is improved.

[0083] The display processing unit 11 at least performs a display process for displaying the display screen Dp1 on the display device 2. Specifically, the display processing unit 11 generates the display screen Dp1 based on the data obtained by the image acquisition unit 14, and controls the display device 2 so that the display screen Dp1 is displayed on the display unit 23 of the display device 2. In addition, the display processing unit 11 operates according to the operation received by the operation unit 22 of the display device 2. For example, the display processing unit 11 displays the captured image Im100 (see FIG. 1 ) captured by the left camera 341, the right camera 342, and the rear camera 343. Figure 5That is, the display processing unit 11 causes the display device 2 to display an image of the monitoring area A1 around the working machine 3 .

[0084] Here, in addition to the captured image Im100, the display processing unit 11 can also display on the display screen Dp1 an effective object X1 (see Figure 5 ). When the detection processing unit 15 is effective, the display processing unit 11 displays the effective target X1 in a manner superimposed on the captured image Im100 in the display screen Dp1. When the detection processing unit 15 is ineffective, the display processing unit 11 does not display the effective target X1. The so-called "target" such as the effective target X1 in the present disclosure includes a mark, image, graphic, photo, text expression or dynamic image displayed in the display screen Dp1, or a combination thereof.

[0085] Thus, the user U1 (operator) can understand that the detection processing unit 15 is effective by displaying the effective target X1 on the display screen Dp1. That is, the user U1 can visually confirm whether the detection processing unit 15 is effective or ineffective, and can operate the working machine 3 based on the understanding of whether the detection processing unit 15 is effective or ineffective.

[0086] In addition, in this embodiment, the detection processing unit 15 obtains the detection result of the detection device 5 outside the control system 1 and performs the detection processing of the detection object Ob1 based on the detection result, but the present invention is not limited to this structure. For example, the detection processing unit 15 can also perform the detection processing of the detection object Ob1 in the monitoring area A1 based on the output of the sensor 51 and / or the imaging unit 52 outside the control system 1.

[0087] [3] Control method of operating machinery

[0088] Below, refer to Figures 5 to 9 , an example of a control method of the working machine 3 executed by the control system 1 (hereinafter referred to as “control method”) will be mainly described.

[0089] The control method involved in this embodiment is executed by a control system 1 whose main structure is a computer system, and therefore, in other words, is specifically embodied by a control program for a working machine (hereinafter referred to as a "control program"). That is, the control program involved in this embodiment is a computer program for causing one or more processors to execute various processes involved in the control method. Such a control program can also be executed by the control system 1 and the display device 2 in cooperation, for example.

[0090] Here, when a specific start operation preset for executing the control program is performed, the control system 1 performs the following various processes involved in the control method. The start operation is, for example, the start operation of the engine of the working machine 3. On the other hand, when a specific end operation preset is performed, the control system 1 ends the following various processes involved in the control method. The end operation is, for example, the stop operation of the engine of the working machine 3.

[0091] [3.1] Display screen

[0092] Here, first, the structure of the display screen Dp1 displayed on the display unit 23 of the display device 2 by the control method according to the present embodiment will be described. Figure 5 In the drawings showing the display screen Dp1 displayed on the display unit 23 of the display device 2 , the dot-dash lines, lead lines, and reference numerals indicating regions are only provided for explanation and are not actually displayed on the display device 2 .

[0093] Figure 5 The display screen Dp1 shown is the main screen that is first displayed by the control method. The main screen is the basic display screen Dp1 that is first displayed on the display device 2 during the operation of the working machine 3. The display screen Dp1 can be transferred from the main screen to various display screens Dp1 including a menu screen, a crane screen, a mode screen, and a PTO screen, etc., according to the operation of the operation unit 22.

[0094] like Figure 5 As shown, the display screen Dp1 includes a first area R1, a second area R2, a third area R3, a fourth area R4, a fifth area R5, a sixth area R6, a seventh area R7, an eighth area R8, a ninth area R9, and a tenth area R10. In the control method involved in this embodiment, as an example, the captured image Im100 of the monitoring area A1 and the valid target X1 are displayed in the second area R2 occupying most of the display screen Dp1.

[0095] Specifically, the display screen Dp1 is divided into four areas in the vertical direction (up and down direction). Moreover, the three areas from the top are further divided into three areas in the horizontal direction (left and right direction). Thus, the display screen Dp1 is divided into a total of 10 areas. Moreover, the areas of the second layer from the top are the first area R1, the second area R2, and the third area R3 from the left. The area of ​​the bottom layer is the fourth area R4. Moreover, the areas of the third layer from the top are the fifth area R5, the sixth area R6, and the seventh area R7 from the left, and the areas of the top layer are the eighth area R8, the ninth area R9, and the tenth area R10 from the left. For the vertical size, among the four areas divided in the vertical direction, the area of ​​the second layer from the top (the first area R1, the second area R2, and the third area R3) is the widest. For the horizontal size, among the three areas divided in the horizontal direction, the area in the middle (the second area R2, the sixth area R6, and the ninth area R9) is the widest.

[0096] However, the configuration and size of each area described above are only examples and can be changed appropriately. In addition, each area does not necessarily have to be clearly divided by a boundary line. Figure 5 In the example, the second region R2 and the third region R3 are clearly divided by a boundary line, but there is no boundary line between the first region R1 and the second region R2. Of course, the first region R1 and the second region R2 may also be clearly divided by a boundary line.

[0097] The first region R1 is a vertically long rectangular region. The first region R1 displays, for example, remaining amount information G1 related to the remaining amount of fuel (eg, diesel) of the engine. The display processing unit 11 generates the remaining amount information G1 on the display screen Dp1 based on the output (sensor signal) of the remaining amount sensor.

[0098] The second region R2 is a rectangular region that is long in the horizontal direction. The second region R2 displays the captured image Im100 of the monitoring region A1 and the valid target X1, etc. The second region R2 also displays the aircraft target Im10 and the captured image Im200 of the rear camera 343, etc.

[0099] The captured image Im100 is a bird's-eye view image generated by performing coordinate conversion and synthesizing the captured image of the left camera 341, the captured image of the right camera 342, and the captured image of the rear camera 343. That is, the captured image Im100 is an image of a small area A11 to the left of the driving unit 321 captured by the left camera 341, an image of a small area A12 to the right of the driving unit 321 captured by the right camera 342, and an image of a small area A13 to the rear of the driving unit 321 captured by the rear camera 343, and is displayed in the form of a bird's-eye view of the monitoring area A1 from above the machine body 30. The display processing unit 11 synthesizes the captured images acquired by the image acquisition unit 14 and displays them in real time.

[0100] The valid target X1 is a target that is valid for the detection processing unit 15 and is superimposed on the captured image Im100 displayed on the display screen Dp1. The details of the valid target X1 are described in the column "[3.2] Details".

[0101] The aircraft object Im10 is displayed in the eleventh region R11 set in the center of the second region R2. In the present embodiment, as an example, the aircraft object Im10 is an image (icon) simulating the aircraft 30 viewed from above. The captured image Im200 is an image of the small region A13 behind the driving unit 321 captured by the rear camera 343.

[0102] The third region R3 is a rectangular region that is longer in the vertical direction. In the third region R3, images (icons) Im1 corresponding to the operating status of each part of the working machine 3 are displayed. A plurality of images Im1 can be displayed in the third region R3, and the state of any one of the battery, seat belt, cooling water temperature, working oil temperature, etc., is represented by the appearance design (pattern) of each image Im1. Here, each image Im1 represents the operating status by displaying a display form such as display color or size. The display processing unit 11 uses the outputs of various sensors (including cooling water temperature sensors and working oil temperature sensors) that detect the operating status of each part of the working machine 3 to determine the state of each part of the working machine 3. Moreover, when an abnormal value is detected at any part, the display processing unit 11 performs a warning display by changing the display form such as the display color of the image Im1 of the part.

[0103] The fourth region R4 is a strip-shaped region extending over the entire width of the display screen Dp1. In the fourth region R4, various items for operating the display screen Dp1 are displayed. Figure 5As an example, the six items "Menu", "Crane", "Mode", "Camera", "PTO", and "Switch" are arranged in order from the left in the fourth region R4. These six items correspond to the six button switches 221 to 226 of the operation unit 22 located directly below them. For example, the "Menu" item corresponds to the button switch 221, and the "Crane" item corresponds to the button switch 222. Therefore, for example, if the button switch 224 corresponding to the "Camera" item is pressed by the user U1 (refer to Figure 4 ) operation, the “Camera” item is operated (selected).

[0104] Furthermore, in the present embodiment, any item is highlighted in the fourth region R4 in a manner corresponding to the operation of the operation dial (or cursor key) or the like of the operation unit 22. Figure 5 In the example, the "Menu" item is highlighted, and the highlighted item is switched by operating the dial (or cursor key) or the like. The user U1 can select the desired item by operating the decision button while the desired item is highlighted. Therefore, for example, if the decision button is operated while the highlight is moved to the "Camera" item, the "Camera" item is operated (selected). In addition, when the operation unit 22 includes a touch panel, the user U1 can select the desired item by touching the desired item on the display screen Dp1.

[0105] In the fifth region R5, a warning display image (icon) indicating that an abnormal value is detected by various sensors (including a cooling water temperature sensor and a working oil temperature sensor) is displayed. In the sixth region R6, for example, information related to the operating part 33 of the working machine 3 is displayed. In the seventh region R7, for example, information related to the operating state of the working machine 3, such as the engine speed, is displayed. In the eighth region R8, for example, the current time is displayed. In the ninth region R9, for example, information indicating the item to which the currently displayed display screen Dp1 belongs is displayed. In the tenth region R10, for example, information related to the operating time (timer) of the working machine 3 is displayed.

[0106] [3.2] Details

[0107] Next, the control method according to the present embodiment will be described in detail.

[0108] The control method involved in this embodiment includes: obtaining a captured image Im100 of the monitoring area A1 around the working machine 3 (image acquisition processing); and displaying a display screen Dp1 including the captured image Im100 on the display device 2 (display processing). The control method also includes: in the display processing, when the detection processing unit 15 for detecting the detection object in the monitoring area A1 is effective, the effective target X1 is superimposed on the captured image Im100 on the display screen Dp1. The effective target X1 is located at a reference point P1 (reference point P1) set for the captured image Im100. Figure 6 ) is centered on an imaginary circle C1 (refer to Figure 6 ) moves in at least one of the circumferential and radial directions.

[0109] In short, by displaying the display screen Dp1 including the captured image Im100 on the display device 2, the user U1 can confirm the presence or absence of the detection object Ob1 in the monitoring area A1 through the captured image Im100 in the display screen Dp1. Here, if the detection processing unit 15 is effective, the effective target X1 moving in at least one of the circumferential direction and the radial direction of the imaginary circle C1 centered on the reference point P1 is superimposed on the captured image Im100 in the display screen Dp1. Therefore, the user U1 can understand whether the detection processing unit 15 is effective by dynamically displaying the effective target X1 at a position that naturally enters the field of vision when observing the captured image Im100. Therefore, according to the control method involved in this embodiment, the operator can easily and intuitively understand the working state of the detection device 5.

[0110] Below, refer to Figure 6 to Figure 8 , the display of the second region R2 in the display screen Dp1 is described in more detail. Figure 6 to Figure 8 An example of the display content of the second region R2 in the display screen Dp1 is shown. Figure 6 to Figure 8 In the figure, only the periphery of the second region R2 in the display screen Dp1 is shown, and the rest of the figure is omitted. Figure 6 and Figure 7 The reference point P1 and the virtual circle C1 shown are both virtual points or circles, and are not actually displayed on the display device 2 .

[0111] like Figure 6 As shown in FIG. 1 , the captured image Im100 is a fan-shaped image in which the outer shape of the side farther from the machine body 30 is arc-shaped. The so-called "arc-shaped" in the present disclosure is not limited to the arc that constitutes a part of a perfect circle, but includes all curved lines that bulge outward. Figure 6In this way, even if the captured image Im100 is composed of curved lines that protrude outward as a whole on the outer shape line, it can be called a fan-shaped image that makes the outer shape into an arc shape. Here, the captured image Im100 is a fan-shaped image formed by cutting off the upper side of the display screen Dp1 in the arc, so that the left side, right side and bottom side of the captured image Im100 become images of the monitoring area A1 (the small areas A11, A12, A13) corresponding to the left, right and rear of the body 30, respectively. That is, in the display screen Dp1, the captured image Im100 is displayed in a state where the outer shape of at least the monitoring area A1 on the left, right and rear of the working machine 3 is formed into an arc shape.

[0112] More specifically, when an imaginary circle C1 centered at a reference point P1 located near the center of the second region R2 is set, the captured image Im100 is fan-shaped in a shape lacking a portion of the circumferential direction of the imaginary circle C1, that is, a removal range from the first end E101 to the second end E102. Figure 6 In the example of , the captured image Im100 has a linear first end E101 extending from the reference point P1 to the upper left, and a linear second end E102 extending from the reference point P1 to the upper right in the display screen Dp1.

[0113] In addition, if Figure 6 As shown, the body object Im10 is arranged at the top corner of the captured image Im100. That is, the body object Im10 is arranged at a position on the reference point P1 corresponding to the top corner of the fan-shaped captured image Im100. In other words, the control method involved in this embodiment further includes: displaying the body object Im10 at the reference point P1 in the display screen Dp1.

[0114] In the present embodiment, the upper part of the machine object Im10 corresponds to the front of the machine 30 in the real space, the lower part of the machine object Im10 corresponds to the rear of the machine 30 in the real space, the right side of the machine object Im10 corresponds to the right side of the machine 30 in the real space, and the left side of the machine object Im10 corresponds to the left side of the machine 30 in the real space. Specifically, the lower part of the machine object Im10 is configured to simulate the arc shape of the rear part (counterweight) of the rotating part 32, and the upper part of the machine object Im10 is configured to simulate the shape of the front part (working part 33) of the rotating part 32. In the present embodiment, the working part 33 is arranged at a position offset to the right relative to the center of the left-right direction D3 of the rotating part 32, and the driving part 321 is arranged at a position offset to the left relative to the center of the left-right direction D3 of the rotating part 32. Therefore, in the machine object Im10, the graphic simulating the working part 33 is located in the upper right part, and the graphic simulating the driving part 321 is located in the left part.

[0115] In addition, in the present embodiment, the aircraft object Im10 is displayed based on the direction of the revolving unit 32 on which the driving unit 321 is mounted, rather than the direction of the traveling unit 31. That is, the revolving unit 32 is relatively revolved with respect to the traveling unit 31, so that the direction in the actual space corresponding to the top of the aircraft object Im10 changes. As a result, the top of the aircraft object Im10 always corresponds to the front of the revolving unit 32, which is the front from the operator riding on the driving unit 321.

[0116] Here, the direction of the body 30 represented by the body target Im10 and the direction of the captured image Im100 observed from the body 30 are made consistent. That is, the front of the body 30 simulated by the body target Im10 and the front in the captured image Im100 are both facing the top of the display screen Dp1. As a result, it is easy for the operator to intuitively grasp the position of the detection object Ob1 by observing the captured image Im100. However, the present invention is not limited to this example, and the body target Im10 can also be arranged at a position other than the top corner of the captured image Im100, such as below the captured image Im100.

[0117] In addition, the captured image Im200 of the rear camera 343 is arranged in the exclusion range between the first end E101 and the second end E102 in the circumferential direction of the imaginary circle C1. That is, the captured image Im200 is arranged above the captured image Im100 in the display screen Dp1. In this embodiment, as an example, the captured image Im200 has a pentagonal shape including two sides along the first end E101 and the second end E102. Thus, the captured image Im200 of the small area A13 behind the driving unit 321 captured by the rear camera 343 can be displayed by utilizing the dead zone in the second area R2.

[0118] Furthermore, in the present embodiment, the detection object Ob1 in the monitoring area A1 is taken as the detection object of the detection device 5. Therefore, when the detection object Ob1 exists in the monitoring area A1, the detection object Ob1 is naturally reflected in the captured image Im100 of the monitoring area A1. In this way, when the detection result indicates the existence of the detection object Ob1 in the monitoring area A1, at least a part of the detection object is reflected in the captured image Im100 in the display screen Dp1. Thus, the user U1 can confirm the detection object Ob1 on the captured image Im100. That is, the operator (user U1) can confirm the conditions of the side and rear of the working machine 3, which are likely to become blind spots when viewed from the driving unit 321, through the display screen Dp1 displayed on the display device 2. Therefore, when the detection object Ob1 exists in the monitoring area A1, it is easy to grasp the conditions of the detection object Ob1 in detail through the display screen Dp1.

[0119] In addition, when the detection processing unit 15 is effective, the effective target X1 is displayed in the display screen Dp1 in a form superimposed on the captured image Im100. The effective target X1 moves in at least one of the circumferential direction and the radial direction of the virtual circle C1 centered on the reference point P1. In the present embodiment, the effective target X1 moves continuously in the circumferential direction of the virtual circle C1.

[0120] like Figure 6 and Figure 7 As shown in , the effective target X1 is an image (target) having a length along the radius of the imaginary circle C1 and moving in the circumferential direction of the imaginary circle C1. Figure 6 As shown in the example, the effective target X1 is an elongated target extending from the reference point P1, the center of the imaginary circle C1, toward the arc-shaped outer shape (outer periphery) of the captured image Im100. Such an effective target X1 extends in a clockwise direction ( Figure 7 Thus, the effective target X1 can easily enter the field of vision of the user U1, and it is easier for the user U1 to understand whether the detection processing unit 15 is effective.

[0121] Here, as described above, the captured image Im100 has a shape that lacks a portion of the circumference of the imaginary circle C1, that is, a range from the first end E101 to the second end E102. Figure 8 As shown in FIG. 1 , the effective target X1 skips the removal range and moves from the first end E101 to the second end E102. Figure 8 As shown in the upper layer of , the effective target X1 moves clockwise ( Figure 8 If the first end E101 of the captured image Im100 is reached, then Figure 8 As shown in the lower layer, it moves to the second end E102 of the captured image Im100.

[0122] Specifically, the effective target X1 continuously moves on the captured image Im100 in the circumferential direction of the imaginary circle C1, and discontinuously moves from the first end E101 to the second end E102 by crossing the removal range (the range from the first end E101 to the second end E102) where the captured image Im100 is interrupted. Thus, the effective target X1 is displayed overlapping only with respect to the captured image Im100, and does not overlap with the captured image Im200 of the rear camera 343. Therefore, it is easier for the user U1 to understand whether the detection processing unit 15 is effective.

[0123] In addition, in the present embodiment, the effective target X1 has transparency. That is, even if the effective target X1 is displayed superimposed on the captured image Im100, the portion of the captured image Im100 that overlaps with the effective target X1 is displayed through the effective target X1. Therefore, the user U1 can visually confirm the entire captured image Im100 through the effective target X1, and the visibility of the captured image Im100 is improved compared to a case where a portion of the captured image Im100 is blocked by the effective target X1.

[0124] Furthermore, the captured image Im100 is a bird's-eye view image. That is, the effective target X1 is displayed superimposed on the captured image Im100 in a manner that moves on the captured image Im100 that is a bird's-eye view image of the surveillance area A1 from above. Therefore, for the user U1, the effective target X1 looks like a radar scanning the surveillance area A1. Therefore, the user U1 can easily and intuitively understand that the effective target X1 indicates the validity of the detection processing unit 15.

[0125] In addition, in the control method involved in the present embodiment, it is preferred that the display form of the effective target X1 is changed according to the detection result of the detection device 5. In this case, the display form of the effective target X1 changes according to the detection result of the detection device 5, so there are at least two types: "not detected" when the detection object Ob1 does not exist in the monitoring area A1, and "detected" when the detection object Ob1 exists in the monitoring area A1. In addition, the display form of the effective target X1 is preferably changed according to the distance from the working machine 3 to the detection object Ob1 and / or the orientation of the detection object Ob1 (i.e., the position of the detection object Ob1). Therefore, even when the detection object Ob1 is "detected" in the monitoring area A1, the display form of the effective target X1 is further different for the case where the distance from the working machine 3 to the detection object Ob1 is long and the case where the distance from the working machine 3 to the detection object Ob1 is short.

[0126] The display form of the effective target X1 that changes according to the detection result of the detection device 5 includes, for example, the display color of the effective target X1. In this case, the display color of the effective target X1 in the display screen Dp1 changes according to the detection result of the detection device 5. For example, the closer the detection object Ob1 is to the machine body 30, the display form of the effective target X1 changes to a more eye-catching display color. As an example, if the detection object Ob1 is close to the machine body 30, the display color of the effective target X1 changes from yellow to red.

[0127] In addition, the display form of the effective target X1 that changes according to the detection result of the detection device 5 may also include the moving speed, moving range and / or moving direction of the effective target X1. For example, the display form of the effective target X1 changes in such a way that the closer the detection object Ob1 is to the body 30, the faster the moving speed of the effective target X1.

[0128] In this way, the display form of the effective target X1 changes according to the detection result of the detection device 5, so that the user U1 can easily and intuitively grasp the presence of the detection object Ob1 (such as a person) in the monitoring area A1 by the display form of the effective target X1.

[0129] Here, sometimes there are multiple detection objects Ob1 in the monitoring area A1 at the same time. In this case, it is preferable to determine the display form of the effective target X1 based on the position of the detection object Ob1 closest to the body 30 of the working machine 3 among the multiple detection objects Ob1. For example, when the detection object Ob1 existing at a position within the threshold distance from the working machine 3 and the detection object Ob1 existing at a position farther than the threshold distance from the working machine 3 are detected at the same time, the display form of the effective target X1 is determined based on the detection object Ob1 existing at the threshold distance from the working machine 3. Thus, the display form of the effective target X1 is determined based on the detection result of the detection object Ob1 with the highest urgency, so the operator can easily grasp the detection result intuitively.

[0130] [3.3] Overall processing

[0131] Next, refer to Fig. 9 The overall flow of processing related to the control method will be described. Fig. 9 This is a flowchart showing an example of processing related to the control method.

[0132] like Fig. 9 As shown, the image acquisition unit 14 of the control system 1 acquires captured images from the left camera 341, the right camera 342, and the rear camera 343 (S1). Then, the display processing unit 11 of the control system 1 displays the display screen Dp1 including the captured image Im100 on the display device 2 (S2).

[0133] Here, the display processing unit 11 of the control system 1 determines whether the detection processing unit 15 is valid or invalid through the switching processing unit 13 (S3). When the detection processing unit 15 is valid (S3: Yes), the display processing unit 11 causes the valid target X1 moving in the circumferential direction of the virtual circle C1 to be superimposed on the captured image Im100 on the display screen Dp1 (S4). On the other hand, when the detection processing unit 15 is invalid (S3: No), the display processing unit 11 causes the valid target X1 to not be displayed (S5).

[0134] The control system 1 repeatedly executes the processing of the above steps S1 to S5. Thus, the display screen Dp1 is displayed on the display device 2 at any time. Fig. 9 The flowchart shown is merely an example, and processing may be appropriately added or omitted, and the order of processing may be appropriately changed.

[0135] [4] Modifications

[0136] The following lists modifications of Embodiment 1. The modifications described below can be applied in combination as appropriate.

[0137] The control system 1 in the present disclosure includes a computer system. The computer system has one or more processors and one or more memories as hardware as the main structure. The functions of the control system 1 in the present disclosure are realized by executing the program recorded in the memory of the computer system by the processor. The program can be pre-recorded in the memory of the computer system, or provided through an electrical communication line, or recorded in a non-temporary recording medium such as a memory card, an optical disk, a hard disk drive, etc. that can be read by the computer system. In addition, part or all of the functional parts contained in the control system 1 can also be composed of electronic circuits.

[0138] In addition, it is not necessary for the control system 1 to integrate at least a part of the functions of the control system 1 into one housing, and the components of the control system 1 may also be dispersed in multiple housings. On the contrary, in the first embodiment, the functions dispersed in multiple devices (such as the control system 1 and the display device 2) may also be integrated in one housing. Furthermore, at least a part of the functions of the control system 1 may also be implemented by the cloud (cloud computing) or the like.

[0139] The power source of the working machine 3 is not limited to the diesel engine, and may be, for example, an engine other than the diesel engine, a motor (electric motor), or a hybrid power source including an engine and a motor (electric motor).

[0140] In addition, the display device 2 is not limited to a dedicated device, and may be a general-purpose terminal such as a laptop computer, a tablet terminal, or a smart phone. Furthermore, the display unit 23 is not limited to a form that directly displays a display screen like a liquid crystal display or an organic EL display, and may be a structure that displays a display screen by projection like a projector.

[0141] In addition, the information input form of the operation unit 22 may be in forms other than button switches, touch panels, and operation dials. For example, the operation unit 22 may be in forms such as a keyboard, a pointing device such as a mouse, voice input, gesture input, or input of an operation signal from another terminal.

[0142] In addition, the restriction processing performed by the restriction processing unit 122 is not limited to the processing of prohibiting the operation (rotation operation, etc.) of the working machine 3 (impossible to operate) as long as it is the processing of restricting the operation of the working machine 3. The restriction processing may be, for example, the processing of slowing down the operation (rotation operation, etc.) of the working machine 3, the processing of reducing the operation range (rotation angle, etc.) of the working machine 3, or the processing of restricting the allowed area of ​​the operation of the working machine 3.

[0143] In addition, the containment process does not necessarily include a plurality of specific processes (such as sound output process and restriction process) for containing the operation of the working machine 3. Furthermore, even if the containment process includes a plurality of specific processes, it is not necessarily possible to switch between valid and invalid state for each specific process independently, and it is also possible to switch between valid and invalid state for a plurality of specific processes at once.

[0144] In addition, the function related to the containment processing by the containment processing unit 12 is not essential, and the containment processing unit 12 can be omitted as appropriate.

[0145] In addition, the detection device 5 for detecting the detection object Ob1 in the monitoring area A1 around the working machine 3 can also include, in addition to or instead of the sensor 51 and the shooting unit 52, a human body sensor, a sonar sensor, a radar or a LiDAR (Light Detection and Ranging) sensor.

[0146] In addition, the detection processing unit 15 may detect the detection object Ob1 in the monitoring area A1 based on the output (image data) of the left camera 341, the right camera 342, and the rear camera 343, for example. Specifically, the detection processing unit 15 performs image processing on the image data acquired by the image acquisition unit 14 to extract the feature quantity in the image, and based on the feature quantity, determines whether the detection object Ob1 (in this embodiment, "person") is reflected in the image. Here, when the detection object Ob1 is reflected in the image, the detection processing unit 15 determines which of the images captured by the left camera 341, the right camera 342, and the rear camera 343 has the detection object Ob1 reflected. That is, the detection processing unit 15 distinguishes which of the small area A11 captured by the left camera 341, the small area A12 captured by the right camera 342, and the small area A13 captured by the rear camera 343 has the detection object Ob1, and detects the detection object Ob1.

[0147] In addition, the captured image Im100 displayed on the display screen Dp1 is not limited to the bird's-eye view image generated by performing coordinate conversion and synthesizing the captured images of the left camera 341, the right camera 342, and the rear camera 343. The captured image displayed on the display screen Dp1 may be, for example, an original image (i.e., an unsynthesized image) captured by at least one of the left camera 341, the right camera 342, and the rear camera 343.

[0148] In addition, the camera for capturing the captured image Im100 is not limited to the left camera 341, the right camera 342, and the rear camera 343, and may include one, two, or more than four cameras (image sensors). Furthermore, the captured image may be captured by a camera that can capture an all-round image from the perspective of the working machine 3, such as a panoramic camera (360-degree camera).

[0149] In addition, the detection object Ob1 may include moving objects such as vehicles (including other working machines), structures such as walls and columns, plants, animals, steps, ditches, or other obstacles in addition to or in place of "people".

[0150] In addition, the positional relationship of the captured image Im100 in the display screen Dp1 with respect to the machine object Im10 does not necessarily reflect the positional relationship of the monitoring area A1 in the actual space with respect to the working machine 3. Furthermore, the positional relationship of the captured image Im100 in the display screen Dp1 with respect to the machine object Im10 does not necessarily reflect the positional relationship of the monitoring area A1 in the actual space with respect to the swing unit 32.

[0151] In addition, the display form of the effective target X1 does not necessarily need to change according to the detection result of the detection device 5.

[0152] (Implementation Method 2)

[0153] like Fig.10 As shown, the display content of the second region R2 of the display screen Dp1 of the working machine 3 according to this embodiment is different from that of the working machine 3 according to the first embodiment. Hereinafter, the same reference numerals are given to the same configurations as those in the first embodiment, and the description thereof is omitted as appropriate. Fig.10 In FIG. 1 , only the second region R2 in the display screen Dp1 is shown, and illustration of regions other than the second region R2 is omitted.

[0154] In this embodiment, the effective target X1 is directed from the reference point P1 side toward the virtual circle C1 (refer to Figure 6 ) moves in the radial direction of the imaginary circle C1. Fig.10As shown in the example, the effective target X1 is an arc-shaped (circular ring-shaped) target centered at the reference point P1, which is the center of the imaginary circle C1. Fig.10 The status shown in the upper layer is Fig.10 The state shown in the lower layer of FIG. 1 moves toward the outside of the captured image Im100 ( Fig.10 The valid target X1 moves in a manner of expanding in the direction indicated by the dotted arrow in the figure. Such a valid target X1 moves toward the outer peripheral side (i.e., the direction away from the reference point P1) along the radial direction of the imaginary circle C1 with the reference point P1 as the center. As a result, the valid target X1 easily enters the field of vision of the user U1, and it is easier for the user U1 to understand whether the detection processing unit 15 is effective.

[0155] In this embodiment, the effective object X1 is not displayed in the exclusion range from the first end E101 to the second end E102 of the captured image Im100. Therefore, the effective object X1 has a shape lacking the exclusion range from the first end E101 to the second end E102, similarly to the captured image Im100.

[0156] If the effective target X1 moves to the outermost end of the movable range, it moves to the innermost end of the movable range. That is, if the effective target X1 moves from the reference point P1 side to the outer periphery of the captured image Im100 which is the outermost end of the movable range, it moves to the inner periphery of the captured image Im100 which is the innermost end of the movable range.

[0157] Specifically, the effective target X1 moves continuously on the captured image Im100 in the radial direction of the imaginary circle C1, and if it moves to the outermost end where the captured image Im100 is interrupted, it moves discontinuously from the outer periphery to the inner periphery of the captured image Im100. Thus, the effective target X1 repeatedly moves on the captured image Im100, making it easier for the user U1 to understand whether the detection processing unit 15 is effective.

[0158] The configuration according to the second embodiment can be adopted in combination with the various configurations (including modified examples) described in the first embodiment as appropriate.

[0159] [Notes on the invention]

[0160] The following is a summary of the invention extracted from the above-mentioned embodiment. In addition, each structure and each processing function described in the following notes can be selected and combined arbitrarily.

[0161] <Note 1>

[0162] A control method for a working machine, comprising:

[0163] Acquire a captured image of a monitoring area around the operating machine;

[0164] Displaying a display screen including the captured image on a display device; and

[0165] When the detection processing unit for detecting the detection object in the above-mentioned monitoring area is effective, the effective target moving in at least one of the circumferential direction and the radial direction of the imaginary circle is overlapped and displayed on the above-mentioned captured image in the above-mentioned display screen, and the above-mentioned imaginary circle is centered on the reference point set for the above-mentioned captured image.

[0166] <Note 2>

[0167] A method for controlling a working machine according to Note 1, wherein:

[0168] The method further comprises: displaying the machine target at the reference point in the display screen.

[0169] <Note 3>

[0170] A method for controlling a working machine according to Note 1 or 2, wherein:

[0171] The display form of the effective target is changed according to the detection result of the detection processing unit.

[0172] <Note 4>

[0173] A method for controlling a working machine according to any one of Notes 1 to 3, wherein:

[0174] The effective goal stated above is transparency.

[0175] <Note 5>

[0176] A method for controlling a working machine according to any one of Notes 1 to 4, wherein:

[0177] The captured image is a bird's-eye view image.

[0178] <Note 6>

[0179] A method for controlling a working machine according to any one of Notes 1 to 5, wherein:

[0180] The effective target has a length along the radius of the imaginary circle and moves in the circumferential direction of the imaginary circle.

[0181] <Note 7>

[0182] A method for controlling a working machine according to Note 6, wherein:

[0183] The captured image has a shape that lacks a portion of the circumference of the imaginary circle, that is, a removal range from the first end to the second end.

[0184] The effective target moves from the first end to the second end skipping the removal range.

[0185] <Note 8>

[0186] A method for controlling a working machine according to any one of Notes 1 to 5, wherein:

[0187] The effective target moves in a radial direction of the imaginary circle from the reference point side toward an outer peripheral side of the imaginary circle.

[0188] <Note 9>

[0189] A method for controlling a working machine according to Note 8, wherein:

[0190] If the effective target moves to the outermost end of the movable range, it moves to the innermost end of the movable range.

[0191] <Note 10>

[0192] A control program for a working machine, for causing one or more processors to execute the control method for the working machine described in any one of Appendix 1 to 9.

Claims

1. A method for controlling a working machine, characterized in that: have: Acquire a captured image of a monitoring area around the operating machine; Displaying a display screen including the captured image on a display device; as well as When the detection processing unit for detecting the detection object in the monitoring area is effective, a valid target moving in at least one of the circumferential and radial directions of an imaginary circle is superimposed and displayed on the captured image in the display screen, and the imaginary circle is centered on a reference point set for the captured image.

2. The control method of the working machine according to claim 1, characterized in that: The method further comprises: displaying a machine target at the reference point in the display screen.

3. The control method for a working machine according to claim 1 or 2, characterized in that: The display form of the effective target is changed according to the detection result of the detection processing unit.

4. The control method for a working machine according to claim 1 or 2, characterized in that: The effective target has transparency.

5. The control method for a working machine according to claim 1 or 2, characterized in that: The captured image is a bird's-eye view image.

6. The control method of the working machine according to claim 1 or 2, characterized in that: The effective target has a length along the radius of the imaginary circle and moves in the circumferential direction of the imaginary circle.

7. The control method of the working machine according to claim 6, characterized in that: The captured image has a shape that lacks a portion of the circumference of the imaginary circle, that is, a removal range from the first end to the second end. The valid target moves from the first end to the second end skipping the removal range.

8. The control method for a working machine according to claim 1 or 2, characterized in that: The effective target moves in the radial direction of the imaginary circle from the reference point side toward the outer peripheral side of the imaginary circle.

9. The control method of the working machine according to claim 8, characterized in that: If the effective target moves to the outermost end of the movable range, it moves to the innermost end of the movable range.

10. A control program for a working machine, characterized in that: Used to cause one or more processors to execute the control method of the working machine according to claim 1 or 2.

11. A control system for a working machine, characterized in that: have: an image acquisition unit that acquires a captured image of a monitoring area around the working machine; and A display processing unit displays a display screen including the captured image on a display device, When the detection processing unit for detecting the detection object in the monitoring area is effective, the display processing unit will overlap and display the effective target moving in at least one of the circumferential direction and the radial direction of an imaginary circle on the captured image in the display screen, and the imaginary circle is centered on a reference point set for the captured image.

12. A working machine, characterized in that: have: The control system for a working machine according to claim 11; and A body is provided for carrying the display device.

Images