Work machine

Abstract

A work machine is provided and includes a space-recognizing member configured to detect surroundings of the work machine; and a control device configured to perform a predetermined operation in a case in which a stripe pattern where colored regions of two different colors are represented in an alternating manner is detected from information acquired by the space-recognizing member.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation application of International Application No. PCT / JP2024 / 045796, filed on Dec. 24, 2024, and designated the U.S., which is based upon and claims priority to Japanese Patent Application No. 2023-222341, filed on Dec. 28, 2023, the entire contents of which are incorporated herein by reference.BACKGROUND1. Technical Field

[0002] The present disclosure relates to a work machine.2. Description of Related Art

[0003] A technique of performing various settings for causing a work machine to perform work in a work region has been proposed. For example, a technique of performing settings related to a work region based on a road cone appearing in an image has been proposed.SUMMARY

[0004] A work machine according to one aspect of the present disclosure includes: a space-recognizing member configured to detect surroundings of the work machine; and a control device configured to perform a predetermined operation in a case in which a stripe pattern in which colored regions of two different colors are represented in an alternating manner is detected from information acquired by the space-recognizing member.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] FIG. 1 is a side diagram of an excavator according to one embodiment of the present disclosure;

[0006] FIG. 2 is a top diagram of the excavator according to the one embodiment;

[0007] FIG. 3 is a diagram schematically illustrating an example of a configuration of the excavator according to the one embodiment;

[0008] FIG. 4 is a diagram illustrating a configuration example of a controller and a communication terminal of the excavator according to the one embodiment;

[0009] FIG. 5 is a diagram illustrating an example of types of objects identifiable by a trained model according to the one embodiment;

[0010] FIG. 6 is a diagram illustrating an example of a region in front of an operator’s seat in a cab of the excavator according to the one embodiment;

[0011] FIG. 7 is a diagram illustrating transition of a screen of the communication terminal according to the one embodiment;

[0012] FIG. 8 is a diagram illustrating an arrangement example of stripe-patterned objects, which are provided for performing operational control of the excavator according to the one embodiment;

[0013] FIG. 9 is a diagram illustrating an arrangement example of stripe-patterned objects, which are provided for performing operational control of the excavator according to the one embodiment;

[0014] FIG. 10 is a diagram illustrating a configuration example of a communication terminal and a controller of an excavator according to another embodiment of the present disclosure;

[0015] FIG. 11 is a diagram illustrating an arrangement example of stripe-patterned objects, which are provided for performing operational control of the excavator according to the another embodiment;

[0016] FIG. 12 is a diagram illustrating a configuration example of a communication terminal and a controller of an excavator according to yet another embodiment of the present disclosure; and

[0017] FIG. 13 is a diagram illustrating transition of a screen of the communication terminal according to the yet another embodiment.DETAILED DESCRIPTION

[0018] Since various objects are present in the work region, it is conceivable for the work machine to erroneously detect an object. When an object is erroneously detected, the work machine may malfunction.

[0019] The present disclosure provides, in one aspect, a technique of improving safety by performing a predetermined operation in response to detecting a stripe pattern.

[0020] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The embodiments described below are not intended to limit the invention and are exemplary. All features or combinations thereof described in the embodiments are not necessarily essential to the invention. In the drawings, identical or corresponding components are assigned with identical or corresponding reference signs, and thus descriptions thereof may be omitted.

[0021] The embodiments of the present disclosure will be described below taking an excavator as an example of a work machine; however, the following description does not intend any limitation to the excavator. The present disclosure may be applicable to construction machines, standard machines, application equipment, forestry machines, or transfer machines based on hydraulic excavators.One embodiment

[0022] First, an outline of an excavator 100 according to the present embodiment will be described with reference to FIGS. 1 and 2. FIGS. 1 and 2 are side and top diagrams of the excavator 100 according to the present embodiment.

[0023] As illustrated in FIGS. 1 and 2, the excavator 100 according to the present embodiment includes a lower moving body 1, an upper slewing body 3 slewably mounted on the lower moving body 1 via a slewing mechanism 2, an attachment AT configured to perform various operations, and a cab 10. In the following, the front side of the excavator 100 (the upper slewing body 3) corresponds to a direction in which the attachment extends with respect to the upper slewing body 3 when the excavator 100 is seen, in a plan view (top view), from directly above along the slewing axis of the upper slewing body 3. Also, the left side and the right side of the excavator 100 (the upper slewing body 3) respectively correspond to the left side and the right side when seen from an operator seated on an operator’s seat in the cab 10.

[0024] The lower moving body 1 includes, for example, a pair of right and left crawlers 1C. Specifically, the crawlers 1C include a left crawler 1CL and a right crawler 1CR. The lower moving body 1 moves the excavator 100 by the left crawler 1CL and the right crawler 1CR respectively hydraulically driven by a left traveling hydraulic motor 2ML and a right traveling hydraulic motor 2MR.

[0025] The upper slewing body 3 slews with respect to the lower moving body 1 by the slewing mechanism 2 hydraulically driven by a slewing hydraulic motor 2A. That is, the slewing hydraulic motor 2A is a slewing driver configured to drive the upper slewing body 3, serving as a driven component, and can change the direction of the upper slewing body 3.

[0026] The attachment AT (an example of an attachment) includes a boom 4, an arm 5, and a bucket 6.

[0027] The boom 4 is attached to the center of the front part of the upper slewing body 3 in a tiltable manner. The arm 5 is attached to the distal end of the boom 4 in a vertically swingable manner. The bucket 6 is attached to the distal end of the arm 5 in a vertically swingable manner.

[0028] The bucket 6 is an example of a work tool. The bucket 6 is used for excavation work or the like. Also, in accordance with content of work or the like, any other work tool may be attached to the distal end of the arm 5 instead of the bucket 6. The other work tools may be, for example, other types of buckets, such as a large-scale bucket, a slope bucket, a dredging bucket, and the like. Also, the other work tools may be of types other than buckets, such as an agitator, a breaker, a grapple, and the like.

[0029] The boom 4, the arm 5, and the bucket 6 are respectively hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, which are hydraulic actuators using hydraulic oil discharged from a main pump 14 (described below).

[0030] The excavator 100 may be configured such that some driven components, such as the lower moving body 1, the upper slewing body 3, the boom 4, the arm 5, and the bucket 6 are electrically driven. That is, the excavator 100 may be a hybrid excavator, an electric excavator, or the like, in which some driven components are driven by electric actuators.

[0031] An imaging device S6, which is an example of a space-recognizing member, is configured to image the surroundings of the excavator 100 to acquire image data representing the surroundings of the excavator 100. The imaging device S6 includes a camera S6F configured to image the front of the excavator 100, a camera S6L configured to image the left side of the excavator 100, a camera S6R configured to image the right side of the excavator 100, and a camera S6B configured to image the rear of the excavator 100.

[0032] The camera S6F is attached to the outside of the cab 10, such as the roof of the cab 10, the side surface of the boom 4, or the like. The camera S6F may be attached, for example, to the ceiling of the cab 10, i.e., to the inside of the cab 10. The camera S6L is attached to the left end of the upper surface of the upper slewing body 3. The camera S6R is attached to the right end of the upper surface of the upper slewing body 3. The camera S6B is attached to the rear end of the upper surface of the upper slewing body 3.

[0033] Each of the imaging devices S6 (the cameras S6F, S6B, S6L, and S6R) is, for example, a monocular wide-angle camera having a very wide angle of view. For example, the imaging device S6 includes an imaging element, such as a CCD, a CMOS, or the like, and is configured to output a captured image to a display device 40. Further, image information acquired by the imaging device S6 is imported into a controller 30.

[0034] The imaging device S6 according to the present embodiment, which is an example of the space-recognizing member, may be any device, as long as it can detect a stripe pattern, a feature of an article, or the like, present around a work machine. The imaging device S6 may be a stereo camera, a range image camera, or the like.

[0035] The controller (control device) 30 is a control device configured to control the excavator 100. For example, the controller 30 is mainly formed by a computer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile auxiliary storage device, various input / output interfaces, and the like. The controller 30 reads out a program from the nonvolatile storage device, loads the read-out program into the volatile storage device, and causes the CPU to execute the program to implement various functions. The various functions include, for example, a machine guidance function to guide an operator’s manual operation of the excavator 100. The controller 30 may include a contact avoidance function to cause the excavator 100 to automatically or autonomously operate or stop for avoiding contact between the excavator 100 and an object present in a monitoring range around the excavator 100.

[0036] For example, the controller 30 sets a target number of revolutions based on an operation or the like performed by an operator or the like, and performs drive control to rotate an engine 11 at a constant speed.

[0037] A boom angle sensor S1 is attached to the boom 4, and is configured to detect a tilt angle (hereinafter referred to as “boom angle”) of the boom 4 with respect to the upper slewing body 3, e.g., an angle formed by a straight line, connecting the fulcrums of both ends of the boom 4, with respect to a slewing plane of the upper slewing body 3 in a side view. The boom angle sensor S1 may include, for example, a rotary encoder, an acceleration sensor, a 6-axis sensor, or an IMU (Inertial Measurement Unit). Also, the boom angle sensor S1 may include, for example, a potentiometer using a variable resistor, or a cylinder stroke sensor configured to detect the amount of stroke of a hydraulic cylinder (the boom cylinder 7) corresponding to the boom angle. The same applies to an arm angle sensor S2 and a bucket angle sensor S3 below. A detection signal corresponding to the boom angle detected by the boom angle sensor S1 is imported into the controller 30.

[0038] The arm angle sensor S2 is attached to the arm 5, and is configured to detect a rotation angle (hereinafter referred to as “arm angle”) of the arm 5 with respect to the boom 4, e.g., an angle formed by a straight line, connecting the fulcrums of both ends of the arm 5, with respect to the straight line connecting the fulcrums of both ends of the boom 4 in a side view. A detection signal corresponding to the arm angle detected by the arm angle sensor S2 is imported into the controller 30.

[0039] The bucket angle sensor S3 is attached to the bucket 6, and is configured to detect a rotation angle (hereinafter referred to as “bucket angle”) of the bucket 6 with respect to the arm 5, e.g., an angle formed by a straight line connecting the fulcrum and the distal end (toe) of the bucket 6, with respect to the straight line connecting the fulcrums of both ends of the arm 5 in a side view. A detection signal corresponding to the bucket angle detected by the bucket angle sensor S3 is imported into the controller 30.

[0040] A machine body tilt sensor (an example of a posture detector) S4 is configured to detect a tilt state of the machine body (the upper slewing body 3 or the lower moving body 1) with respect to the horizontal plane. For example, the machine body tilt sensor S4 is attached to the upper slewing body 3, and is configured to detect tilt angles (hereinafter referred to as “front-rear tilt angle” and “left-right tilt angle”) of the excavator 100 (i.e., the upper slewing body 3) around two axes in the front-rear direction and the left-right direction. The machine body tilt sensor S4 may include, for example, a rotary encoder, an acceleration sensor, a 6-axis sensor, or an IMU. Detection signals corresponding to the tilt angles (the front-rear tilt angle and the left-right tilt angle) detected by the machine body tilt sensor S4 are imported into the controller 30.

[0041] A slewing angle sensor S5 is configured to output detection information of a slewing state of the upper slewing body 3. For example, the slewing angle sensor S5 detects a slewing angular velocity and a slewing angle of the upper slewing body 3. The slewing angle sensor S5 may include, for example, a gyro sensor, a resolver, or a rotary encoder. Detection signals corresponding to the slewing angle and the slewing angular velocity of the upper slewing body 3 output by the slewing angle sensor S5 are imported into the controller 30.

[0042] The positioning device PS is configured to measure the position and direction of the upper slewing body 3. The positioning device PS is, for example, a GNSS (Global Navigation Satellite System) compass, and detects the position and direction of the upper slewing body 3. Detection signals corresponding to the position and direction of the upper slewing body 3 are imported into the controller 30. Also, among the functions of the positioning device PS, the function to detect the direction of the upper slewing body 3 may be replaced by an orientation sensor attached to the upper slewing body 3.

[0043] The cab 10 is an operation compartment in which an operator rides. The cab 10 is provided on the front-left side of the upper slewing body 3.

[0044] As will be described below, when the excavator 100 is operated remotely or fully automatically, the cab 10 may be omitted.

[0045] A communication device T1 communicates with an external device through a predetermined network including a mobile communication network having a base station as a terminal, a satellite communication network, the Internet network, or the like. The communication device T1 is, for example, a mobile communication module corresponding to a mobile communication standard, such as LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation), or the like, or a satellite communication module for connecting to a satellite communication network.

[0046] The excavator 100 drives actuators (e.g., hydraulic actuators) in response to operations performed by an operator riding in the cab 10, thereby driving operating components (hereinafter referred to as “driven components”), such as the lower moving body 1, the upper slewing body 3, the boom 4, the arm 5, the bucket 6, and the like.

[0047] The excavator 100 may be configured to be remotely operated from the outside of the excavator 100, instead of or in addition to being operated by the operator in the cab 10. When the excavator 100 is remotely operated, the cab 10 may be unmanned.

[0048] Also, the excavator 100 may automatically operate the actuators regardless of contents of the operations performed by the operator. Thus, the excavator 100 implements the function to automatically operate at least some of the driven components, such as the lower moving body 1, the upper slewing body 3, the boom 4, the arm 5, and the bucket 6, i.e., what is referred to as “automatic operation function” or “machine control function”.

[0049] The automatic operation function may include the function to automatically operate driven components (actuators) other than the operation target driven components (actuators) in response to the operation or remote operation performed by an operator on an operation device 26, i.e., what is referred to as “semi-automatic operation function” or “operation support-type machine control function”. Also, the automatic operation function may include the function to automatically operate at least some of a plurality of driven components (hydraulic actuators) on the premise that there is no operation or remote operation performed by an operator on the operation device 26, i.e., what is referred to as “fully automatic operation function” or “fully automatic-type machine control function”. When the fully automatic operation function is effective in the excavator 100, the cab 10 may be unmanned. Also, the semi-automatic operation function, the fully automatic operation function, or the like may include a mode in which contents of operations of the driven components (actuators) to be automatically operated are automatically determined in accordance with a predetermined rule. Also, the semi-automatic operation function, the fully automatic operation function, or the like may include a mode in which the excavator 100 autonomously performs various determinations, and autonomously determines contents of operations of the driven components (hydraulic actuators) to be automatically operated, in accordance with the determination results (i.e., “automatic operation function”).

[0050] FIG. 3 is a diagram schematically illustrating an example of a configuration of the excavator 100 according to the present embodiment.

[0051] In FIG. 3, a mechanical power system, a hydraulic oil line, a pilot line, and an electric control system are indicated by a double line, a solid line, a broken line, and a dotted line, respectively.

[0052] A drive system of the excavator 100 according to the present embodiment includes the engine 11, a regulator 13, a main pump 14, and a control valve unit 17. Also, a hydraulic drive system of the excavator 100 according to the present embodiment includes hydraulic actuators such as moving hydraulic motors 1L and 1R, the slewing hydraulic motor 2A, the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9 configured to respectively hydraulically drive the lower moving body 1, the upper slewing body 3, the boom 4, the arm 5, and the bucket 6, as described above.

[0053] The engine 11 is a main power source in the hydraulic drive system, and is mounted, for example, at the rear of the upper slewing body 3. Specifically, the engine 11 revolves at a predetermined target number of revolutions under the direct or indirect control performed by the controller 30, which will be described below, thereby driving the main pump 14 and a pilot pump 15. The engine 11 is, for example, a diesel engine using diesel oil as fuel.

[0054] The regulator 13 is configured to control the discharge amount of the main pump 14. For example, the regulator 13 adjusts the angle (tilt angle) of a swash plate of the main pump 14 in accordance with a control command from the controller 30.

[0055] Similar to the engine 11, the main pump 14 is mounted, for example, at the rear of the upper slewing body 3. The main pump 14 is configured to supply hydraulic oil to the control valve unit 17 through a high-pressure hydraulic line. The main pump 14 is driven by the engine 11 as described above. The main pump 14 is, for example, a variable displacement hydraulic pump, and the stroke length of a piston is adjusted by adjusting the tilt angle of the swash plate using the regulator 13 under the control of the controller 30 as described above, thereby controlling a discharge flow rate (discharge pressure).

[0056] The control valve unit 17 is a hydraulic control device configured to control the hydraulic system of the excavator 100. In the present embodiment, the control valve unit 17 includes control valves 171 to 176. The control valve 175 includes a control valve 175 L and a control valve 175R, and the control valve 176 includes a control valve 176L and a control valve 176R. The control valve unit 17 is configured to selectively supply hydraulic oil discharged from the main pump 14 to one or more hydraulic actuators through the control valves 171 to 176. The control valves 171 to 176 control, for example, a flow rate of hydraulic oil flowing from the main pump 14 to hydraulic actuators, and a flow rate of hydraulic oil flowing from the hydraulic actuators to a hydraulic oil tank. The hydraulic actuators include the boom cylinder 7, the arm cylinder 8, the bucket cylinder 9, the moving hydraulic motors 1L and 1R, and the slewing hydraulic motor 2A. More specifically, the control valve 171 corresponds to the left moving hydraulic motor 1L, the control valve 172 corresponds to the right moving hydraulic motor 1R, and the control valve 173 corresponds to the slewing hydraulic motor 2A. Also, the control valve 174 corresponds to the bucket cylinder 9, the control valve 175 corresponds to the boom cylinder 7, and the control valve 176 corresponds to the arm cylinder 8.

[0057] The pilot pump 15 is an example of a pilot pressure generator, and is configured to supply hydraulic oil to hydraulic control devices through a pilot line. In the present embodiment, the pilot pump 15 is a fixed displacement hydraulic pump. However, the pilot pressure generator may be realized by the main pump 14. That is, the main pump 14 may have the function to supply hydraulic oil to various hydraulic control devices through the pilot line, in addition to the function to supply hydraulic oil to the control valve unit 17 through the hydraulic oil line. In this case, the pilot pump 15 may be omitted.

[0058] A discharge pressure sensor 28 is configured to detect the discharge pressure of the main pump 14. In the present embodiment, the discharge pressure sensor 28 outputs the detected value to the controller 30. The discharge pressure sensor 28 includes, for example, discharge pressure sensors 28L and 28R, as described below.

[0059] The operation device 26 is a device used by an operator to operate the actuators. The operation device 26 includes, for example, an operating lever and an operating pedal. The actuators include at least one of hydraulic actuators or electric actuators.

[0060] An operation sensor 29 is configured to detect contents of operations performed by an operator using the operation device 26. In the present embodiment, the operation sensor 29 is configured to detect an operational direction and an operational amount of the operation device 26 corresponding to each of the actuators, and output the detected values to the controller 30. In the present embodiment, the controller 30 controls an opening area of a proportional valve 31 in accordance with the output of the operation sensor 29. Then, the controller 30 supplies hydraulic oil discharged from the pilot pump 15 to a pilot port of the corresponding control valve in the control valve unit 17. The pressure (pilot pressure) of the hydraulic oil supplied to each pilot port is, in principle, a pressure in accordance with the operational direction and the operational amount of the operation device 26 corresponding to each hydraulic actuator. Thus, the operation device 26 is configured to supply the hydraulic oil discharged from the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17.

[0061] The proportional valve 31 functioning as a control valve for machine control is disposed in a conduit connecting the pilot pump 15 and the pilot port of the control valve in the control valve unit 17, and is configured to change the flow path area of the conduit. In the present embodiment, the proportional valve 31 operates in accordance with a control command output from the controller 30. Therefore, the controller 30 can supply the hydraulic oil discharged from the pilot pump 15 to the pilot port of the control valve in the control valve unit 17 via the proportional valve 31, independently of the operation performed by an operator on the operation device 26.

[0062] With this configuration, even if the specific operation device 26 is not operated, the controller 30 can operate the hydraulic actuator corresponding to that specific operation device 26.

[0063] The control system of the excavator 100 according to the present embodiment includes the controller 30, the display device 40, an input device 42, a storage medium 47, and a communication device T1.

[0064] The controller 30 (an example of the control device) is provided, for example, in the cab 10, and is configured to control driving of the excavator 100. The functions of the controller 30 may be implemented by any hardware, software, or combination thereof. For example, the controller 30 is mainly formed by a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile auxiliary storage device, various input / output interfaces, and the like. The controller 30 implements various functions, for example, by executing various programs stored in the ROM or nonvolatile auxiliary storage device on the CPU.

[0065] For example, the controller 30 sets a target number of revolutions based on an operation or the like performed by an operator, and performs drive control to rotate the engine 11 at a constant speed.

[0066] Also, for example, the controller 30 outputs a control command to the regulator 13, if necessary, to change the discharge amount of the main pump 14.

[0067] Also, for example, the controller 30 performs control related to the machine guidance function to guide a manual operation performed by an operator on the excavator 100 using the operation device 26. Further, for example, the controller 30 performs control related to the machine control function to automatically support a manual operation performed by an operator on the excavator 100 using the operation device 26.

[0068] Some of the functions of the controller 30 may be implemented by another controller (control device). That is, the functions of the controller 30 may be implemented in a divided manner by a plurality of controllers. For example, the machine guidance function and the machine control function may be implemented by corresponding dedicated controllers (control devices).

[0069] The display device 40 is provided at a place that can be easily seen by an operator seated in the cab 10, and is configured to display various information images under the control of the controller 30. The display device 40 may be connected to the controller 30 via an in-vehicle communication network, such as a CAN (Controller Area Network) or the like, or may be connected to the controller 30 via a one-to-one dedicated line.

[0070] The input device 42 is provided within reach of the operator seated in the cab 10. The input device 42 is configured to receive various operational inputs from the operator, and output a signal corresponding to the operational input to the controller 30. The input device 42 includes: a touch panel mounted on a display of the display device configured to display various information images; a knob switch provided at the distal end of a lever portion of each of operation levers 26A and 26B; and a button switch, a lever, a toggle, a rotary dial, and the like, which are disposed around the display device 40. A signal corresponding to the content of the operation performed on the input device 42 is imported into the controller 30.

[0071] The storage medium 47 is provided, for example, in the cab 10, and is configured to store various information under the control of the controller 30. The storage medium 47 is a nonvolatile storage device, such as a semiconductor memory or the like. The storage medium 47 may store information output by various devices during the operation of the excavator 100, or may store information acquired via various devices before the start of operation of the excavator 100.Functional Configuration of Controller

[0072] Next, an outline of the machine guidance function and the machine control function of the excavator 100 will be described with reference to FIG. 4. FIG. 4 is a diagram illustrating a configuration example of a communication terminal 500 and the controller 30 of the excavator 100 according to the present embodiment.

[0073] The communication terminal 500 includes an imaging device 501, a CPU 502, and a display device 503. The communication terminal 500 is, for example, a terminal carried by an operator of the excavator 100.

[0074] The imaging device 501 is configured to image the surroundings of the communication terminal 500, and acquire image data representing the surroundings of the communication terminal 500.

[0075] The CPU 502 is configured to control the communication terminal 500. For example, the CPU 502 reads out a program from a storage medium (not shown), loads the read-out program into a volatile storage device (not shown), and causes the CPU 502 to execute the program, thereby implementing various functions.

[0076] The display device 503 is configured to display various information images under the control of the CPU 502.

[0077] As illustrated in FIG. 4, the controller 30 is connected to an in-vehicle imaging device S11 and the storage medium 47.

[0078] The in-vehicle imaging device S11 is, for example, a driving recorder. The in-vehicle imaging device S11 is configured to image the surroundings of the excavator 100, image the status of the interior of the cab 10 of the excavator 100, and record the acquired image data in, for example, the storage medium 47. Also, the in-vehicle imaging device S11 may transmit the acquired image data to the controller 30.

[0079] The storage medium 47 is a nonvolatile and readable / writable storage medium (e.g., an SSD). The storage medium 47 stores a trained model 47A and a setting storage 47B.

[0080] The setting storage 47B is configured to store settings for an operation of the excavator 100 to be performed in accordance with the type of an object. The setting storage 47B stores, for example, the type of an object in association with an operation to be performed in response to detecting the object of this type. The setting storage 47B may also store conditions for executing the operation in association with each other. The setting storage 47B according to the present embodiment may associate one or more of the size of the object detected by the imaging device S6 and the region where the object is detected in the image information, as the conditions for executing the operation.

[0081] A configuration in which the controller 30 controls an operation based on the detected object will be described. The controller 30 includes, as the machine guidance function and the machine control function, an object detector 301, a stripe pattern detector 302, a determinator 303, a storage 304, an operation specifier 305, an operation controller 306, and a display controller 307.

[0082] The object detector 301 is configured to input image information acquired by the imaging device S6 into the trained model 47A to be stored in the storage medium 47 and having undergone machine learning, and receive, from the trained model 47A, the type of an object appearing in the image information and a coordinate region (in the image information) in which the object appears. Thus, the object detector 301 specifies the type of an object present in the monitored region around the excavator 100, and the region where the object appears in the image information.

[0083] Specifically, the object detector 301 loads the trained model 47A from the storage medium 47 into a main storage, such as a RAM or the like, and causes the CPU to execute the loaded trained model 47A, thereby specifying the type of an object around the excavator 100 using the trained model 47A.

[0084] The trained model 47A is mainly formed by a neural network. The neural network of the trained model 47A may use what is referred to as a deep neural network that includes one or more intermediate layers (hidden layers) between an input layer and an output layer. In the neural network, a weighting parameter representing a connection strength to a lower-level layer is defined for each of a plurality of neurons forming each intermediate layer. The neural network is configured in a manner that the neurons of each layer output the sum of values, obtained by multiplying each of the input values from the plurality of neurons of an upper-level layer by the weighting parameter defined for each neuron of the upper-level layer, to the neurons of the lower-level layer through a threshold function.

[0085] As a result of causing the trained model 47A to undergo machine learning, specifically, deep learning, the weighting parameter of the neural network is optimized.

[0086] Training data used for the machine learning includes, for example, image information in which an object appears, information identifying the type of the object, and a coordinate region (occupied region) where the object appears. The trained model 47A trained through machine learning using the above training data outputs, in response to an input of image information, the information identifying the type of the object appearing in the image information, and the coordinate region of the appearing object.

[0087] The neural network of the trained model 47A may be, for example, a convolutional neural network (CNN). The CNN is a neural network to which the existing image processing techniques (convolutional processing and pooling processing) are applied. Specifically, the CNN repeatedly performs convolutional processing and pooling processing in combination for the input images to extract feature data (feature map) smaller in size than the input image information. Then, the pixel value of each pixel of the extracted feature map is input to a neural network formed by a plurality of fully connected layers, and the output layer of the neural network can output, for example, a predicted probability indicating the presence of an object for each type of object.

[0088] In this manner, the trained model 47A may be configured to receive an input of image information, and output the coordinates and size (i.e., the occupied region of the object on the input image) of the object in the image information, and the type of the object.

[0089] That is, the trained model 47A may be configured to detect an object appearing in the image information (determine a region where the object appears in the image information) and determine the type of the object. In this case, an output from the trained model 47A may be in an image information format in which information pertaining to the occupied region of the object appearing in the image information and information pertaining to the type of the object are superimposed.

[0090] For example, when a person appears in the image information, the trained model 47A outputs an indication that the person appears, and the coordinates and size of the person appearing therein (i.e., the occupied region of the object in the input image). Further, the trained model 47A also identifies objects other than the person.

[0091] FIG. 5 is a diagram illustrating an example of types of objects identifiable by the trained model 47A according to the present embodiment. The trained model 47A outputs information identifying the types of objects, for example, when the shapes of the objects illustrated in FIG. 5 are included in the input image information.

[0092] The trained model 47A according to the present embodiment enables identification of a safety jacket, a road cone, a gate, a surface portion, a barricade, and a guard fence illustrated in FIG. 5.

[0093] Furthermore, the trained model 47A enables identification of work machines including excavators, vehicles, guardrails, bars (including cone bars), signs, signboards, batons, poles, ropes, megaphones, speakers, pipe covers, chain stands, flags, curtains, streamers, and the like. Moreover, the trained model 47A may enable recognition of animals, buildings, and the like.

[0094] In the present embodiment, the controller 30 detects objects of the above types. However, the object to be detected has fewer features than those of a person, resulting in difficulty reliably detecting the object. For example, there is a possibility that a triangular object present in a work site is erroneously detected as a road cone.

[0095] In view of this, in the present embodiment, a stripe pattern is applied to an object to be detected by the controller 30. In other words, the controller 30 detects an object to which a stripe pattern is applied among the objects present in the work site. Even if the controller 30 erroneously detects a triangular prism present in the work site as a road cone, no operation based on the erroneous detection is not performed unless the stripe pattern is applied to the triangular prism. This can suppress erroneous operation of the excavator 100 to improve safety.

[0096] As illustrated in FIG. 4, the stripe pattern detector 302 specifies a stripe-patterned region from image information acquired by the imaging device S6.

[0097] The stripe-patterned (striped) region according to the present embodiment is a region where regions of two different colors are repeated in an alternating manner.

[0098] Although the two different colors present in the stripe pattern preferably have a high contrast, any level of contrast is sufficient as long as workers present in the work site are identifiable. The two different colors that are conceivable are, for example, yellow and black, white and red, white and green, yellow-red and white, yellow and white, and red and yellow, which does not intend any limitation to the color combinations. For example, red, yellow, green, blue, or the like, which are JIS safety colors (e.g., JIS Z 9101:2018), may be combined with one of two contrasting colors, i.e., white or black. Use of a combination of these colors can urge the workers to prevent danger, call attention, or the like.

[0099] That is, in the present embodiment, the stripe pattern is often used as a pattern to urge the workers to prevent danger or call attention to a worker in the work site. Therefore, stripe-patterned work equipment, ropes, tape, and the like are easily available in the work site. Therefore, in the present embodiment, the stripe pattern is used as a pattern for controlling the operation of the excavator 100.

[0100] In the present embodiment, it is possible to easily obtain a pattern for controlling the operation of the excavator 100, and to alert workers present around the excavator 100.

[0101] Also, for detecting the stripe-patterned region, the repeated number of patterns may be defined in advance. For example, the stripe pattern detector 302 may detect, as the stripe-patterned region, a region where color bands of two different colors are repeated five times or more.

[0102] Also, preferably, a first area of a region having one of the two different colors and a second area of a region having the other color are nearly equal to each other. However, the first area and the second area are not limited to being nearly equal to each other. For example, the first area may be larger or smaller than the second area. However, each of the first area and the second area is such an area that is detectable from the excavator 100 at least several meters away.

[0103] Also, when a stripe-patterned object is present in a region detectable by the imaging device S6, the stripe pattern detector 302 detects the stripe-patterned region regardless of the direction in which this stripe-patterned object is disposed.

[0104] For example, there is a case in which a stripe-patterned object (e.g. a cone bar or tape) is disposed from the front side of the excavator 100 in a direction that is farther away from the excavator 100. In this case, the stripe pattern appearing in the image information acquired by the imaging device S6 has a larger color area on the front side, and a smaller color area as the stripe-patterned object goes in the direction that is farther away from the excavator 100. Even in such a case, the stripe pattern detector 302 appropriately detects the stripe-patterned region. Also, the stripe pattern detector 302 may detect the direction or the like of the stripe-patterned region based on change in the area for each color.

[0105] As a modified example, the stripe pattern detector 302 may detect only the region where the color bands of two colors included in the stripe-patterned region are arranged obliquely. That is, typical structures and typical objects may be provided with stripe patterns in the vertical or horizontal direction. Therefore, the stripe pattern detector 302 according to this modified example detects only the stripe-patterned region where the color bands are arranged obliquely. An angle of the direction of an oblique line distinguishing the color bands may be any angle, and may be, for example, about 45 degrees according to the layout of safety markings (e.g., JIS Z 9101:2018). This can control the operation of the excavator 100 in accordance with the layout typically used for a sign or the like. In this modified example, operational control is performed in response to detecting an obliquely striped pattern, thereby suppressing erroneous detection and improving reliability.

[0106] When the object detector 301 specifies the type of the object present around the excavator 100 from the image information acquired by the imaging device S6, the determinator 303 determines whether or not the object and the stripe pattern are related to each other based on the positional relationship between the region (on the image information) of the object specified by the object detector 301 and the region (on the image information) of the stripe pattern specified by the stripe pattern detector 302.

[0107] For example, the determinator 303 determines whether or not, in the image information, the region where the stripe pattern is present is included in the region where the object is present. The determinator 303 according to the present embodiment regards that, when the region where the stripe pattern is present is included in the region where the object is present, the stripe pattern is applied to the object. In this case, the determinator 303 determines that the object detected by the object detector 301 is related to the stripe pattern.

[0108] The stripe-patterned object means an object related to the operational control of the excavator 100. That is, even if an object is erroneously detected, the operational control based on the object is not performed unless a stripe pattern is applied to the object. That is, in the present embodiment, the erroneous detection of an object unrelated to the operational control can be suppressed, thereby improving the detection accuracy.

[0109] What is determined by the determinator 303 according to the present embodiment is not limited to cases in which a stripe pattern is directly applied to the object. For example, when the distance between the region where the stripe pattern is present and the region where the object is present is less than a predetermined threshold (e.g., a predetermined number of dots), the determinator 303 may determine that the object is related to the stripe pattern. Furthermore, when the stripe pattern is applied to enclose at least a part of the object or when the stripe pattern is applied to the edge of the object, the determinator 303 determines that the object is related to the stripe pattern. That is, although various modes in which the stripe pattern is applied to the object are conceivable, the controller 30 according to the present embodiment can recognize whether or not the object and the stripe pattern are related to each other regardless of the various modes, by performing the determination described above. In other words, since the workers in the work site can easily provide the object to which the stripe pattern is applied, the operational control of the excavator 100 is easily performed, thereby improving work efficiency.

[0110] For example, stripe-patterned tape may be attached to enclose the surface portion illustrated in FIG. 5. In this case, the determinator 303 may determine, as a stripe-patterned surface portion, a part of the surface portion enclosed by the stripe-patterned tape in the surface portion detected by the object detector 301. Then, the controller 30 may perform character recognition or the like only for the region determined to be the surface portion. The controller 30 may control the operation of the excavator 100 in accordance with the recognized characters or the like. For example, in the present embodiment, the region enclosed by the stripe-patterned tape may be present on the rear surface of a dump truck. In this case, as a result of performing character recognition for the region, when the characters extracted via the character recognition are predetermined characters, the controller 30 may perform operational controls such as loading earth and sand or the like.

[0111] Furthermore, the determinator 303 may determine whether or not, in the image information, the distance between the region where the object is present and the region where the stripe pattern is present is less than a predetermined threshold. When it is determined that the distance between the region where the object is present and the region where the stripe pattern is present is less than a predetermined threshold, it is determined that the object and the stripe pattern are related to each other, i.e., the stripe pattern is applied to the object. The predetermined threshold is defined from embodiment to embodiment, and thus descriptions thereof are omitted.

[0112] The storage 304 stores, in the setting storage 47B, setting information acquired from the in-vehicle imaging device S11. The setting information retains, in association with each other, the type of an object and, when the object of this type is detected, the operational control to be performed on the excavator 100. Furthermore, the setting information may include additional conditions for performing the operational control.

[0113] FIG. 6 is a diagram illustrating an example of a region visible forward from an operator’s seat 50 in the cab 10 of the excavator 100 according to the present embodiment.

[0114] As illustrated in FIG. 6, the bucket 6 can be seen from a front window 110 of the cab 10. The operator’s seat 50 is provided at the center of the cab 10, and the operation levers 26A and 26B are respectively disposed at both sides of the operator’s seat 50. The display device 40 and the input device 42 are disposed right-forward of the operator’s seat 50 (on the right side below the front window). While sitting on the operator’s seat 50, the operator operates the operation lever 26A with his / her left hand and the operation lever 26B with his / her right hand, thereby moving the bucket 6 to a desired position to perform excavation work.

[0115] In the example illustrated in FIG. 6, a seal of a two-dimensional code 1601 is attached below a left side window 112. The two-dimensional code 1601 includes a URL (Uniform Resource Locator) configured to install an application for setting the operation of the excavator 100 (hereinafter referred to as an operation setting application) in a portable terminal.

[0116] Although the present embodiment describes an example in which the seal of the two-dimensional code 1601 is attached below the left side window 112, there is no limitation on the position where the seal of the two-dimensional code 1601 is to be attached. For example, the seal may be attached below a right side window 111.

[0117] The in-vehicle imaging device S11 is provided above the front window 110. The in-vehicle imaging device S11 can image a status of the interior of the cab 10.

[0118] In the present embodiment, when the operator captures an image of the two-dimensional code 1601 using the imaging device 501 of the communication terminal 500, an operation setting application 521 for setting the operation of the excavator 100 is installed in the communication terminal 500. Then, the CPU 502 of the communication terminal 500 executes the operation setting application 521 to implement various functions.

[0119] FIG. 7 is a diagram illustrating transition of a screen of the communication terminal 500 according to the present embodiment. The CPU 502 of the communication terminal 500 executes the operation setting application 521 to display a screen 1701 illustrated in FIG. 7. The screen 1701 illustrates a detection target selection field 1711, a detection region selection field 1712, a display size input field 1713, a control selection field 1714, and an OK button 1715.

[0120] The detection target selection field 1711 is a field configured to select the type of an object to be detected.

[0121] The detection region selection field 1712 is a field configured to select a region where an object of the selected type is to be detected. The detection region selection field 1712 enables selection of “lower region of the image” or the like. As the object approaches the excavator 100, the region where the object appears moves downward in the image information. Therefore, by selecting the region where the object is to be detected (e.g., a lower region), it is possible to set the relative positional relationship between the excavator 100 and the object when performing the operational control. When no region is selected in the detection region selection field 1712, the entire region of the image information is a detection target.

[0122] The display size input field 1713 is configured to set a size at which the object of the selected type is to be detected. The closer the object to the excavator 100, the larger the size at which the object appears. Therefore, by setting the size at which the object is to be detected, it is possible to set the relative positional relationship between the excavator 100 and the object when the operational control is performed. When no size is set in the display size input field 1713, the object is a detection target regardless of the size.

[0123] The control selection field 1714 is a field configured to select an operation to be executed by the excavator 100 when an object of the selected type is detected and when conditions set in the detection target selection field 1711 and the display size input field 1713 are satisfied. When no conditions are set in the detection target selection field 1711 and the display size input field 1713, the control selection field 1714 is a field configured to select an operation to be executed by the excavator 100 when an object of the selected type is detected.

[0124] A selectable operation may be any operation that can be executed by the excavator 100. The selectable operation may be, for example, deceleration and stop of the excavator 100, stop of the attachment AT, notification to the surroundings, or display of a warning screen on the display device 40 or the like. Further, the selectable operation may be a start operation for saving image information (still image or moving image) acquired by the imaging device S6 or the in-vehicle imaging device S11. Moreover, the selectable operation may be a start operation for performing recording using a microphone provided in the excavator 100. In addition, the selectable operation may be a start operation for performing a remote conference or the like with an external device that uses an image or the like acquired by the in-vehicle imaging device S11.

[0125] The OK button 1715 is a button to be pressed when selection and settings are completed in the detection target selection field 1711, the detection region selection field 1712, the display size input field 1713, and the control selection field 1714.

[0126] In response to receiving the selection of the OK button 1715, the CPU 502 of the communication terminal 500 displays a screen 1702. The screen 1702 displays a two-dimensional code 1721 for acquiring setting information including the contents selected or set by the detection object selection field 1711, the detection region selection field 1712, the display size input field 1713, and the control selection field 1714.

[0127] When the in-vehicle imaging device S11 captures an image of the two-dimensional code 1721, the storage 304 of the excavator 100 extracts the setting information included in the two-dimensional code 1721, and stores the extracted setting information in the setting storage 47B.

[0128] This enables the excavator 100 to perform the operations set by the communication terminal 500. Although the present embodiment is described using a case in which the operations are set by the communication terminal 500, the present embodiment does not intend any limitation to the case in which the operations are set by the communication terminal 500. For example, the operations may be set based on information input from the input device 42. The operations to be performed by the excavator 100 may be defined based on the information input from the input device (an example of the operation device) 42 or the communication terminal (an example of the external terminal) 500. That is, in the present embodiment, it is possible to set the control to cause the excavator 100 to operate in accordance with the conditions of the work site or the like. Therefore, since operations corresponding to the conditions are possible, it is possible to improve safety. Further, since flexible operations are possible, it is possible to improve convenience.

[0129] The operation specifier 305 is configured to specify the operations of the excavator 100 based on the determination result of the determinator 303 and the setting information stored in the setting storage 47B. For example, when an object of a predetermined type has been detected in the determination result, the operation specifier 305 specifies the operation in accordance with the type of the object based on the setting information or the like. Also, the operation specifier 305 may specify the operations in accordance with the conditions, such as the region where the object appears, the size of the appearing object, and the like.

[0130] The operation controller 306 is configured to perform various controls in accordance with the operations specified by the operation specifier 305. For example, the operation controller 306 outputs a control command to the proportional valve 31 corresponding to the operation in order to perform the operation specified by the operation specifier 305. Therefore, the excavator 100 according to the present embodiment can execute an operation corresponding to the type of the detected object.

[0131] The display controller 307 is configured to perform control to display information on the display device 40.

[0132] When it is determined by the determinator 303 that a stripe-patterned object is present, the display controller 307 superimposes a frame (information) indicating a region where this object is present on image information acquired by the imaging device S6, and displays the superimposed image information on the display device 40. The present embodiment does not intend any limitation to the example in which a target, on which the frame (information) indicating a region where the object is present is to be superimposed, is superimposed on image information acquired by the imaging device S6. The target may be superimposed on screen information generated based on the image information.

[0133] Although the present embodiment describes the example in which the frame is superimposed on the stripe-patterned object, the display controller 307 may display a frame (information) indicating a region where an object detected by the object detector 301 is present, and a frame (information) indicating a region where a stripe pattern detected by the stripe pattern detector 302 is present. Furthermore, the display controller 307 may display only one of a frame (information) indicating a region where an object is present or a frame (information) indicating a region where a stripe pattern is present.

[0134] The display controller 307 according to the present embodiment is not limited to a mode in which a frame indicating an object, a stripe pattern, or the like is superimposed on image information, and displayed. For example, the display controller 307 may pop-up display a window indicating that a stripe-patterned object is detected. Furthermore, the display controller 307 may pop-up display a window indicating that a stripe pattern is detected.

[0135] In the present embodiment, since a frame (information) indicating the above-described region is displayed, an operator can recognize an object present around the excavator 100. Therefore, the operator can recognize an operation to be executed in response to the object, i.e., can understand what response should be taken, which thereby can improve operability. Furthermore, the operator can recognize an operation in advance, i.e., can suppress an operation not intended by the operator, which can improve safety.

[0136] Next, an example of the operational control of the excavator 100 according to the present embodiment will be described. In the present embodiment, there are various operational controls in response to detecting a stripe-patterned object, in addition to operational controls based on the setting information.

[0137] FIG. 8 is a diagram illustrating an arrangement example of stripe-patterned objects, which are provided for performing operational control of the excavator 100 according to the present embodiment. In the example illustrated in FIG. 8, stripe-patterned road cones 1811 to 1814, stripe-patterned cone bars 1821 and 1822, and stripe-patterned tape 1841 are provided around the excavator 100. The tape 1841 is, for example, attached to the ground.

[0138] In the example illustrated in FIG. 8, the attachment AT and the like of the excavator 100 are set so as not to contact the cone bars 1821 and 1822. That is, when the determinator 303 determines that stripe patterns are applied to the cone bars 1821 and 1822, in response to determining that the attachment AT approaches the cone bars 1821 and 1822 closer than a predetermined distance in accordance with the operation or the like of the operator, the operation controller 306 performs control to stop the attachment AT. Thus, it is possible to perform work such that the excavator 100 and an excavator 1800 do not interfere with each other. The predetermined distance may be, for example, 2 m, or may be a distance defined from embodiment to embodiment. The present embodiment has been described using the case in which the attachment AT approaches the cone bars 1821 and 1822 closer than the predetermined distance. However, the present embodiment does not intend to limit a reference structure for determining the stop control to the attachment AT, and the reference structure may be any structure of the excavator 100.

[0139] As another example, decelerating and stopping control is set by the setting information in response to detecting tape in a lower region of the image information. In this case, when the determinator 303 determines that the stripe-patterned tape 1841 is present in the lower region of the image information, the operation controller 306 performs the decelerating and stopping control. Thus, when approaching the vicinity of the tape 1841, the excavator 100 decelerates and stops. Therefore, the excavator 100 can be prevented from moving out of the work region.

[0140] Further, the controller 30 according to the present embodiment may perform operational control with person detection and stripe-patterned object detection combined. For example, when the object detector 301 further detects a person 1831 and a person 1832, and the determinator 303 determines that a stripe pattern is applied to the cone bar 1822, the determinator 303 may further determine the positional relationship between the stripe-patterned cone bar 1822 and the persons 1831 and 1832.

[0141] When the determinator 303 determines that the person 1831 is closer to the excavator 100 than is the stripe-patterned cone bar 1822, i.e., determines that the person 1831 moves beyond the cone bar 1822, the operation controller 306 may perform control to cause the imaging device S6 or the in-vehicle imaging device S11 to start capturing an image in the moving direction of the person 1831. Also, the operation controller 306 may perform control to output a warning sound in the direction in which the person 1831 is present. The present embodiment has been described using a case in which conditions for starting the control are those in which it is determined that the person is closer to the excavator 100 than is the stripe-patterned object. However, the present embodiment does not intend to limit the start of the control to the case in which the above-described conditions are satisfied. The conditions for starting the control may be any case in which the positional relationship between the object, determined to be related to the stripe pattern, and the person satisfies predetermined conditions. The predetermined conditions may be defined from embodiment to embodiment. In the present embodiment, by starting the above-described control based on the positional relationship between the person and the object, it is possible to perform the control in accordance with the surrounding status including the persons, which can improve safety and convenience.

[0142] Further, the controller 30 may detect movement of the stripe-patterned object to perform operational control based on the detected movement.

[0143] FIG. 9 is a diagram illustrating an arrangement example of stripe-patterned objects, which are provided for performing operational control of the excavator 100 according to the present embodiment. In the example illustrated in FIG. 9, stripe-patterned fences 1901 to 1905 are provided around the excavator 100. The fences 1901 to 1905 are arranged to enclose the work region of the excavator 100.

[0144] In the example illustrated in FIG. 9, the attachment AT and the like of the excavator 100 are set so as not to contact the fences 1901 to 1905. That is, when the determinator 303 determines that stripe patterns are applied to the fences 1901 to 1905, the operation controller 306 performs control to stop the attachment AT or the lower moving body 1 when the attachment AT or the lower moving body 1 approaches the vicinity of the fences 1901 to 1905 in accordance with the operation or the like performed by the operator. Thus, the excavator 100 can be prevented from contacting the fences 1901 to 1905. Then, it is assumed that the fence 1902 is moved for a person 1911 to enter the work site.

[0145] When the determinator 303 determines that the stripe-patterned fence 1902 has been moved, the operation controller 306 may perform control to start one or more of: stop of the operation of the excavator 100; notification to the operator; warning the surroundings; and capturing an image of the surroundings of the fence 1902 using the imaging device S6 or the in-vehicle imaging device S11. Safety is thereby improved.

[0146] The present embodiment does not intend any limitation to the case in which the operator is riding in the cab 10 of the excavator 100. The configuration of the present embodiment may be applied to a case in which the excavator 100 is operated from a remote control compartment.

[0147] In the present embodiment, since the controller 30 performs operational control in accordance with the type of the stripe-patterned object, it is possible to perform an appropriate operation in accordance with various conditions in the work site. Therefore, it is possible to improve work efficiency and safety.Another embodiment

[0148] The above embodiment has been described using the case in which the imaging device S6 is used as the space-recognizing member configured to detect the surroundings of the work machine. However, the above embodiment does not intend to limit the space-recognizing member, configured to detect the surroundings of the work machine, to the imaging device S6. Therefore, another embodiment will be described using a case in which a device other than the imaging device S6 is used as the space-recognizing member configured to detect the surroundings of the work machine.

[0149] FIG. 10 is a diagram illustrating a configuration example of the communication terminal 500 and the controller 30 of the excavator 100 according to the present embodiment. In the present embodiment, the same reference signs are assigned to the components of the above-described embodiment, and thus descriptions thereof are omitted. The excavator 100 according to the present embodiment includes a three-dimensional measurement sensor (an example of the detector) S12 as an example of the space-recognizing member.

[0150] The three-dimensional measurement sensor S12 is configured to recognize an object present in a three-dimensional space around the excavator 100, and to measure (calculate) a positional relationship, such as a distance up to, a direction toward, or the like, a recognized object from the three-dimensional measurement sensor S12 or the excavator 100. The three-dimensional measurement sensor S12 may include, for example, a distance sensor capable of measuring a distance up to an object around the excavator 100 and a direction toward the object around the excavator 100, such as an ultrasonic sensor, a millimeter-wave radar, an infrared sensor, a LIDAR (Light Detecting and Ranging) sensor, or the like. Also, the three-dimensional measurement sensor S12 may be, for example, a sensor capable of recognizing a distance up to an object and a direction toward the object, and may include an imaging device such as a stereo camera, a distance image camera, a depth camera, or the like.

[0151] For example, four three-dimensional measurement sensors S12 are provided. The three-dimensional measurement sensors S12 are respectively provided near the imaging devices S6 illustrated in FIG. 2. Specifically, the three-dimensional measurement sensors S12 include a front recognition sensor attached to the front end of the upper surface of the cab 10, a rear recognition sensor attached to the rear end of the upper surface of the upper slewing body 3, a left recognition sensor attached to the left end of the upper surface of the upper slewing body 3, and a right recognition sensor attached to the right end of the upper surface of the upper slewing body 3. Also, an upper recognition sensor configured to recognize an object present in a space above the upper slewing body 3 may be attached to the excavator 100.

[0152] A controller 30A according to the present embodiment includes a three-dimensional object detector 311 instead of the object detector 301 of the one embodiment.

[0153] Also, the controller 30A includes a determinator 303A and an operation specifier 305A, configured to perform different processes, instead of the determinator 303 and the operation specifier 305 in the above-described embodiment.

[0154] The three-dimensional object detector 311 is configured to acquire, from the three-dimensional measurement sensor S12, detection information including a distance up to an object in a range measurable by the three-dimensional measurement sensor S12 and a direction toward the object in the range measurable by the three-dimensional measurement sensor S12.

[0155] Then, the three-dimensional object detector 311 inputs the detection information acquired from the three-dimensional measurement sensor S12 to a trained model 47C to be stored in the storage medium 47 and having undergone machine learning, and receives, from the trained model 47C, the shape of the object appearing in the detection information, and the direction and distance in which the object of the specified shape is present.

[0156] In this manner, the trained model 47C may be configured to receive an input of the detection information, and to be capable of outputting the shape, direction, and distance of the object present in the range to be detected.

[0157] The training data used for machine learning of the trained model 47C includes, for example, detection information indicating that the object is present in the range to be detected, the shape of the object present in this range, and the direction and distance in which the object of the specified shape is present. The trained model 47C trained via machine learning using the training data is configured to, in response to an input of the detection information, output the shape of the object appearing in the detection information, as well as the distance up to the object and the direction toward the object.

[0158] The shape of the object output from the trained model 47C may be, for example, one or more of a stepped shape present on the road surface, a flat surface portion perpendicular to the ground, and a columnar shape provided perpendicular to the ground, a bar shape, a planar shape, and the like. The present embodiment describes examples of the shape of the object, and the shape of the object to be detected may be any shape.

[0159] When the shape of the object is specified by the three-dimensional object detector 311, the determinator 303A determines whether or not the object and the stripe pattern are related to each other based on the positional relationship between the direction and distance of the object of the specified shape and the direction in which the stripe pattern detected by the stripe pattern detector 302 is present. Conversion from the region where the stripe pattern appears in the image information into the direction in which the stripe pattern is present in the real work site may be performed by a well-known method, for example, based on the direction in which the imaging device S6 is disposed.

[0160] For example, the determinator 303A determines whether or not the direction in which the stripe pattern is present is included in the range in which the object of the specified shape is present. When the direction in which the stripe pattern is present is included in the range where the object is present, the determinator 303A according to the present embodiment regards that the stripe pattern is applied to the object. In this case, the determinator 303A determines that the object of the specified shape and the stripe pattern are related to each other.

[0161] The determinator 303A according to the present embodiment is not limited to the case in which a stripe pattern is directly applied to a striped object. The determinator 303A determines that an object and a stripe pattern are related to each other when the stripe pattern is applied to enclose at least a part of the object or when the stripe pattern is applied to the edge of the object. The controller 30A according to the present embodiment can recognize whether or not the object and the stripe pattern are related to each other in various patterns by performing the determination described above. In other words, since the workers in the work site can easily provide the object related to the stripe pattern, the operational control of the excavator 100 is easily performed, thereby improving work efficiency.

[0162] The storage 304 stores, in the setting storage 47B, the setting information acquired from the in-vehicle imaging device S11. The setting information retains, in association with each other, the shape of an object and, when the object of this shape is detected, the operational control performed on the excavator 100. Furthermore, the setting information may include additional conditions for performing the operational control. The additional conditions may be, for example, a distance up to the object and a direction toward the object.

[0163] Further, in the communication terminal 500, the CPU 502 executes an operation setting application 521A instead of the operation setting application 521 in the embodiment described above. The operation setting application 521A receives a correspondence relationship between the shape of the object, the distance up to the object, and the like, and the operational control performed by the excavator 100 when the object of this shape is detected and the conditions related to this distance and the like are satisfied. Then, the operation setting application 521A performs control to deliver the setting information, including the correspondence relationship, to the controller 30A. This control is similar to that in the embodiment described above, and thus descriptions thereof are omitted.

[0164] The operation specifier 305A is configured to specify the operation of the excavator 100 based on the determination result of the determinator 303A and the setting information stored in the setting storage 47B. For example, when it is determined in the determination result that the object and the stripe pattern are related to each other, the operation specifier 305A specifies the operation in accordance with the shape of this object based on the setting information or the like.

[0165] FIG. 11 is a diagram illustrating an arrangement example of stripe-patterned objects, which are provided for performing operational control of the excavator 100 according to the present embodiment. In FIG. 11, it is assumed that the excavator 100 performs work to disassemble an object 2111 and remove the object 2111.

[0166] A sheet 2101 is provided around the object 2111. Both ends 2102 and 2103 of the sheet 2101 are respectively provided with stripe patterns.

[0167] The three-dimensional object detector 311 detects the sheet 2101 as a plurality of flat surface portions perpendicular to the ground.

[0168] The stripe patterns are applied to regions corresponding to both ends of the flat surface portions. Therefore, the determinator 303A determines that the detected plurality of flat surface portions are related to the stripe patterns.

[0169] In the example illustrated in FIG. 11, the attachment AT and the like of the excavator 100 are set so as not to contact the flat surface portions related to the stripe patterns. That is, when the attachment AT approaches the vicinity of the sheet 2101, which is a flat surface portion, in accordance with the operation or the like performed by the operator, the operation controller 306 performs control to stop the attachment AT.

[0170] That is, in the present embodiment, when the sheet 2101 is provided, the excavator 100 can perform work without influencing the outside of the sheet 2101, which can improve safety.

[0171] In the present embodiment, since the controller 30A performs operational control in accordance with the shape of the stripe-patterned object, it is possible to perform an appropriate operation in accordance with various conditions in the work site, which can improve work efficiency and safety.Modified Example of Another embodiment

[0172] The above embodiments have been described using the case of detecting the shape of the object or the type of the object. However, the above-described embodiments do not intend any limitation to the example in which the three-dimensional measurement sensor S12 detects the shape of the object. For example, the excavator 100 may include a sensor configured to detect a reflective material, and may perform operational control when the sensor detects the reflective material.

[0173] For example, it is assumed that the three-dimensional measurement sensor S12 is a sensor configured to detect a distance up to an object and a direction toward the object using infrared rays. In this modified example, the three-dimensional measurement sensor S12 has a reflective material detection mode configured to detect only the reflective material. When the reflective material detection mode is set, the three-dimensional measurement sensor S12 detects the direction toward the reflective material and the distance up to the reflective material, and outputs, to the controller 30A, the detected direction and distance as reflective material detection information including the direction toward the reflective material and the distance up to the reflective material.

[0174] The object to which the reflective material is applied includes, for example, a worker’s safety jacket, a work machine, a vehicle, a work tool, and the like. In this modified example, when there is an object provided with both a stripe pattern and a reflective material, the operational control corresponding to the reflective material is performed. As for the object provided with both the stripe pattern and the reflective material, the reflective material may be provided on the stripe pattern, or the reflective material may be provided on the object provided with the stripe pattern. That is, in this modified example, the operational control is performed when the reflective material and the stripe pattern are related to each other.

[0175] The three-dimensional object detector 311 detects the distance up to the reflective material and the direction toward the reflective material based on the received reflective material detection information. Then, the determinator 303A determines whether or not the reflective material and the stripe pattern are related to each other based on the direction in which the reflective material is present and the direction in which the stripe pattern is present. Conversion from the region where the stripe pattern appears in the image information into the direction in which the stripe pattern is present in the real work site may be performed by a well-known method, for example, based on the direction in which the imaging device S6 is disposed.

[0176] The operation specifier 305A specifies the operation of the excavator 100 based on the determination result of the determinator 303A and the setting information stored in the setting storage 47B. The setting information according to this modified example includes information related to the operational control to be performed in accordance with the reflective material. Specifically, when it is determined in the determination result that the reflective material and the stripe pattern are related to each other, the operation specifier 305A specifies the operation in accordance with the reflective material based on the setting information or the like. The subsequent process is similar to that in the above-described embodiments, and thus descriptions thereof are omitted.

[0177] In this modified example, in response to detecting the reflective material and the stripe pattern, the operational control of the excavator 100 is performed. This enables the excavator 100 to perform the operational control in accordance with the reflective material provided for calling attention, which can improve safety.Yet another embodiment

[0178] In the present embodiment, a case of specifying the type of an object and the shape of the object will be described.

[0179] FIG. 12 is a diagram illustrating a configuration example of the communication terminal 500 and a controller 30B of the excavator 100 according to the present embodiment. In the present embodiment, the same reference signs are assigned to the components of the above-described embodiments, and thus descriptions thereof are omitted.

[0180] The controller 30B according to the present embodiment includes the object detector 301 according to the one embodiment, and the three-dimensional object detector 311 according to the another embodiment.

[0181] Also, the controller 30B includes a determinator 303B and an operation specifier 305B, instead of the determinator 303A and the operation specifier 305A according to the another embodiment.

[0182] When the type of the object is specified by the object detector 301 and the shape of the object is specified by the three-dimensional object detector 311, the determinator 303B specifies a correspondence relationship between the type of the object and the shape of the object based on the positional relationship between the direction in which the object of the specified type is present and the direction in which the object of the specified shape is present. When the determinator 303B specifies the correspondence relationship, it is possible to recognize a direction in which the object of the specified type is present, and a distance up to the object of the specified type. Conversion from the region where the object appears in the image information into the direction in which the object or the stripe pattern is present in the real work site may be performed by a well-known method, for example, based on the direction in which the imaging device S6 is disposed.

[0183] The determinator 303B determines whether or not the object and the stripe pattern are related to each other based on the positional relationship between the direction in which the object of the specified type and shape is present and the direction in which the stripe pattern detected by the stripe pattern detector 302 is present. This determination is similar to that in the above-described embodiments, and thus descriptions thereof are omitted.

[0184] The storage 304 retains, in the setting storage 47B, the setting information acquired from the in-vehicle imaging device S11. The setting information retains, in association with each other, the type of an object and, when the object of this type is detected, the operational control to be performed on the excavator 100. Furthermore, the setting information may include information pertaining to a position at which the object is present as a condition for starting the operational control.

[0185] In the communication terminal 500, the CPU 502 executes an operation setting application 521B instead of the operation setting application 521A in the above-described embodiments.

[0186] FIG. 13 is a diagram illustrating transition of a screen of the communication terminal 500 according to the present embodiment. The CPU 502 of the communication terminal 500 executes the operation setting application 521B to display a screen 2301 illustrated in FIG. 13. The screen 2301 illustrates height range designation fields 2311A and 2311B, distance range designation fields 2312A and 2312B, directional range designation fields 2313A and 2313B, a detection target selection field 2314, a control selection field 2315, and an OK button 2316.

[0187] The height range designation fields 2311A and 2311B, the distance range designation fields 2312A and 2312B, and the directional range designation fields 2313A and 2313B are fields configured to perform settings relating to a position at which an object is present as a condition for performing operational control. When an object indicated below is present in a range set in these designation fields, operational control is started. These designation fields 2311A, 2311B, 2312A, 2312B, 2313A, and 2313B need not be set unless they are necessary for setting conditions for starting the operational control.

[0188] The detection object selection field 2314 is a field configured to select the type of an object to be detected.

[0189] The control selection field 2315 is a field configured to select the operation to be executed by the excavator 100 when the object of the selected type is detected and when the conditions related to the position of the object are satisfied.

[0190] The OK button 2316 is a button to be pressed when selection and settings are completed in the designation fields 2311A, 2311B, 2312A, 2312B, 2313A, and 2313B, the detection target selection field 2314, and the control selection field 2315.

[0191] In response to receiving the selection of the OK button 2316, the CPU 502 of the communication terminal 500 displays a screen 2302. The screen 2302 displays a two-dimensional code 2321 for acquiring setting information including the contents selected or set by the designation fields 2311A, 2311B, 2312A, 2312B, 2313A, and 2313B, the detection target selection field 2314, and the control selection field 2315.

[0192] When the in-vehicle imaging device S11 captures an image of the two-dimensional code 2321, the storage 304 of the excavator 100 extracts the setting information included in the two-dimensional code 2321 and stores the setting information in the setting storage 47B.

[0193] The operation specifier 305B is configured to specify the operation of the excavator 100 based on the determination result of the determinator 303B and the setting information stored in the setting storage 47B. For example, when it is determined in the determination result that the object and the stripe pattern are related to each other, the operation specifier 305B specifies the operation in accordance with the type and position (one or more of the distance and the direction) of the object based on the setting information or the like. The subsequent process is the similar to that in the above-described embodiments, and thus descriptions thereof are omitted. Thus, for example, it is possible to control the excavator 100 to decelerate, stop, or the like, for example, when a cone bar is present 5 m or less from the excavator 100 and 50 cm or less in height from the excavator 100.

[0194] The present embodiment does not intend any limitation to the case in which the operator is riding in the cab 10 of the excavator 100. The configuration of the present embodiment may be applied to a case in which the excavator 100 is operated from a remote control compartment. Further, the excavator 100 may perform autonomous control. In this case, in consideration of the type, direction, and orientation of the stripe-patterned object, a trajectory of movement of the attachment AT or the lower moving body 1 of the excavator 100 may be generated, and the operational control may be performed in accordance with the trajectory of movement.

[0195] In the present embodiment, since the controller 30B performs operational control in accordance with the type and position (one or more of the distance and the direction) of the stripe-patterned object, it is possible to perform an appropriate operation in accordance with various conditions in the work site. This can improve work efficiency and safety.Modified Example

[0196] The above embodiments have been described using the case of specifying one or more of the type and shape of the stripe-patterned object. However, the above-described embodiments do not intend any limitation to the method of specifying one or more of the type and shape of the stripe-patterned object. As a modified example, the controller may control the excavator 100 to perform a predetermined operation when a stripe pattern where colored regions of two different colors are represented in an alternating manner is detected from the information acquired by the imaging device S6. That is, since the stripe pattern is a pattern for calling attention to the surroundings, the excavator 100 performs operational control defined in accordance with the stripe pattern in response to detecting the stripe pattern, which can improve safety.Effects

[0197] In the above-described embodiments, in response to detecting the stripe pattern, the excavator 100 performs operational control defined in accordance with the detected stripe pattern. The stripe pattern is both a pattern for calling attention or the like to the surroundings, and a pattern easily obtainable using tape or the like. Therefore, the acquired stripe pattern can be easily applied to an object or the like. Also, in many cases, the stripe pattern is applied in advance to work equipment or the like. In addition, the stripe pattern can be detected from image information with high accuracy.

[0198] Conventionally, erroneous detection may occur when detecting an object. Therefore, in the above-described embodiments, after detecting an object, when there is a relationship between the object (including a reflective material) and the stripe pattern, control is performed based on the detected object. By combining the detection of the stripe pattern and the detection of the object, erroneous detection can be suppressed, and the accuracy of the operational control can be improved. Therefore, it is possible to improve safety in the work site.

[0199] The above embodiments and modified examples have been described using the case in which the excavator 100 is used as a work machine. However, the embodiments and modified examples do not intend to limit the work machine to the excavator 100, and the work machine may be a road machine including an asphalt finisher or the like. Furthermore, the work machine may be a forklift, a crawler crane, or an overhead crane, which are movable in accordance with the operation performed by an operator, or may be a fixed power crane.

[0200] Although the embodiments of the work machine according to the present disclosure have been described above, the present disclosure is not limited to the above embodiments. Various alterations, modifications, substitutions, additions, deletions, and combinations are possible within the scope of the recited claims. These also fall within the technical scope of the present disclosure.

Claims

1. A work machine, comprising:a space-recognizing member configured to detect surroundings of the work machine; anda controller including control circuitry configured to perform a predetermined operation in a case in which a stripe pattern where colored regions of two different colors are represented in an alternating manner is detected from information acquired by the space-recognizing member.

2. The work machine according to claim 1, whereinthe space-recognizing member includes an imaging device configured to acquire image information representing the surroundings of the work machine, andin a case in which a type of an object present around the work machine is specified from the image information, the controller is configured to perform the predetermined operation in accordance with the type of the object in a case in which the object and the stripe pattern are determined to be related to each other based on a positional relationship in the image information between the object and the stripe pattern.

3. The work machine according to claim 2, whereinthe type of the object specified by the controller includes one or more of: a road cone, a fence, a guardrail, a pole, a surface portion, a vehicle, and the work machine.

4. The work machine according to claim 2, whereinin a case in which a region where the stripe pattern is present is included in a region where the object is present in the image information, in a case in which a distance between the region where the object is present and the region where the stripe pattern is present in the image information is less than a predetermined threshold, or in a case in which the region where the stripe pattern is present encloses the object, the controller is configured to determine that the object and the stripe pattern are related to each other.

5. The work machine according to claim 2, whereinin a case in which a distance between the work machine and the object determined to be related to the stripe pattern is less than a predetermined distance, the controller is configured to perform the predetermined operation.

6. The work machine according to claim 2, whereinin a case in which a person present around the work machine is detected from the image information, the controller is further configured to perform the predetermined operation in a case in which a positional relationship between the person and the object determined to be related to the stripe pattern satisfies a predetermined condition.

7. The work machine according to claim 2, whereinin a case in which movement of the object determined to be related to the stripe pattern is detected, the controller is configured to perform the predetermined operation.

8. The work machine according to claim 2, whereinthe space-recognizing member further includes a detector capable of acquiring detection information representing a distance up to the object present around the work machine and a direction toward the object present around the work machine,in a case in which the type of the object present around the work machine is specified from the image information, and a shape of the object is specified based on the detection information, the controller is configured to:specify a correspondence relationship between the object of the specified type and the object of the specified shape; anddetermine whether or not the object and the stripe pattern are related to each other based on a positional relationship between a direction in which the object of the specified type and shape is present and a direction in which the detected stripe pattern is present, andin a case in which the controller determines that the object and the stripe pattern are related to each other, the controller is configured to:perform the predetermined operation in accordance with the type of the object and one or more of the distance up to the object and the direction toward the object.

9. The work machine according to claim 1, whereinthe space-recognizing member includes a detector capable of acquiring detection information representing a distance up to the object present around the work machine and a direction toward the object present around the work machine, andin a case in which a shape of the object is specified based on the detection information, the controller is configured to perform the predetermined operation in accordance with the specified shape of the object in a case in which the object and the stripe pattern are determined to be related to each other based on a direction in which the object is present and a direction in which the stripe pattern is detected.

10. The work machine according to claim 9, whereinthe shape of the object specified by the controller includes one or more of: a flat surface portion perpendicular to a ground, a stepped shape provided on a ground, a columnar shape provided perpendicular to a ground, and a bar shape.

11. The work machine according to claim 9, whereinin a case in which the direction in which the stripe pattern is present is included in a range where the object is present, or the direction in which the stripe pattern is present encloses at least a part of the object, the controller is configured to determine that the object and the stripe pattern are related to each other.

12. The work machine according to claim 1, whereinthe space-recognizing member includes a detector capable of acquiring reflective material detection information representing a distance up to a reflective material present around the work machine and a direction toward the reflective material present around the work machine, and whereinin a case in which the reflective material is detected based on the reflective material detection information, the controller is configured to perform the predetermined operation in accordance with the specified reflective material in a case in which the reflective material and the stripe pattern are determined to be related to each other based on a direction in which the reflective material is present and a direction in which the stripe pattern is detected.

13. The work machine according to claim 1, whereinthe stripe pattern is a pattern where colored regions of two colors are obliquely represented in an alternating manner.

14. The work machine according to claim 1, whereinthe predetermined operation is defined based on information input from an operation device or an external terminal.

15. The work machine according to claim 1, whereinthe controller is configured to display, on a display device, information indicating that the stripe pattern is present.

16. The work machine according to claim 15, whereinthe controller is configured to:superimpose, on screen information based on image information acquired by an imaging device included in the space-recognizing member, information indicating a region where the stripe pattern is present; anddisplay the superimposed screen information on the display device.

Images