Work machine

Abstract

A work machine according to one aspect of the present disclosure comprises: a space recognition device for detecting the surroundings of the work machine; and a control device configured to perform a predetermined operation when a stripe pattern in which color areas of two different colors are alternately represented is detected from information acquired by the space recognition device.

Description

Work machinery

[0001] The present invention relates to a work machine.

[0002] Conventionally, various techniques have been proposed for setting various settings for a work area of ​​a work machine in order to perform work in the work area. For example, a technique has been proposed in which settings related to the work area are set based on road cones shown in an image (see Patent Document 1).

[0003] International Publication No. 2020 / 080538

[0004] However, because various objects exist in the work area, it is possible that the work machine may erroneously detect an object. If an object is erroneously detected, there is a possibility that the work machine may malfunction.

[0005] One aspect of the present invention provides a technology for improving safety by performing a predetermined action when a striped pattern is detected.

[0006] A work machine according to one aspect of the present invention comprises a spatial recognition device for detecting the surroundings of the work machine, and a control device configured to perform a predetermined operation when a striped pattern consisting of alternating areas of two different colors is detected from information acquired by the spatial recognition device.

[0007] According to one aspect of the present invention, safety is improved by performing a predetermined action when a striped pattern is detected.

[0008] 1 is a side view of a shovel according to the first embodiment. FIG. 2 is a top view of a shovel according to the first embodiment. FIG. 3 is a diagram schematically illustrating an example of the configuration of a shovel according to the first embodiment. FIG. 4 is a diagram illustrating an example of the configuration of a controller of a shovel and a communication terminal according to the first embodiment. FIG. 5 is a diagram illustrating an example of the type of object that can be identified by a trained model according to the first embodiment. FIG. 6 is a diagram illustrating an example of an area that is visible forward from the driver's seat in the cabin of the shovel according to the first embodiment. FIG. 7 is a diagram illustrating screen transitions of a communication terminal according to the first embodiment. FIG. 8 is a diagram illustrating an example of the arrangement of objects that have been given a striped pattern and that have been prepared for performing operation control of the shovel according to the first embodiment. FIG. 9 is a diagram illustrating an example of the arrangement of objects that have been given a striped pattern and that have been prepared for performing operation control of the shovel according to the first embodiment. FIG. 10 is a diagram illustrating an example of the configuration of a controller of a shovel and a communication terminal according to a second embodiment. FIG. 11 is a diagram illustrating an example of the arrangement of objects that have been given a striped pattern and that have been prepared for performing operation control of the shovel according to the second embodiment. FIG. 11 is a diagram illustrating an example of the configuration of a controller of a shovel and a communication terminal according to a third embodiment. FIG. 12 is a diagram illustrating screen transitions of a communication terminal according to the third embodiment.

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments described below are merely examples and do not limit the invention, and all features and combinations described in the embodiments are not necessarily essential to the invention. In addition, identical or corresponding components in each drawing are designated by identical or corresponding reference numerals, and descriptions thereof may be omitted.

[0010] In the following embodiments of the present invention, an example will be described in which a shovel is used as an example of a work machine, but the present invention is not limited to a shovel. The present invention may be applied to construction machines, standard machines, application machines, forestry machines, or conveyance machines based on hydraulic shovels.

[0011] First Embodiment First, an overview of a shovel 100 according to this embodiment will be described with reference to Figures 1 and 2. Figures 1 and 2 are a top view and a side view, respectively, of the shovel 100 according to this embodiment.

[0012] 1 and 2 , an excavator 100 according to this embodiment includes a lower traveling body 1, an upper rotating body 3 rotatably mounted on the lower traveling body 1 via a rotating mechanism 2, an attachment AT for performing various tasks, and a cabin 10. Hereinafter, the front of the excavator 100 (upper rotating body 3) corresponds to the direction in which the attachment extends from the upper rotating body 3 when the excavator 100 is viewed in a plan view (top view) from directly above along the rotation axis of the upper rotating body 3. Furthermore, the left and right sides of the excavator 100 (upper rotating body 3) correspond to the left and right sides, respectively, as viewed from an operator seated in the operator's seat in the cabin 10.

[0013] The lower traveling structure 1 includes, for example, a pair of left and right crawlers 1C. Specifically, the crawlers 1C include a left crawler 1CL and a right crawler 1CR. The left crawler 1CL and the right crawler 1CR of the lower traveling structure 1 are hydraulically driven by a left traveling hydraulic motor 2ML and a right traveling hydraulic motor 2MR, respectively, to cause the excavator 100 to travel.

[0014] The upper rotating body 3 rotates relative to the lower traveling body 1 as a result of the rotating mechanism 2 being hydraulically driven by the rotating hydraulic motor 2A. In other words, the rotating hydraulic motor 2A is a rotating drive part that drives the upper rotating body 3 as a driven part, and can change the orientation of the upper rotating body 3.

[0015] The attachment AT (an example of an attachment) includes a boom 4 , an arm 5 , and a bucket 6 .

[0016] The boom 4 is attached to the front center of the upper rotating body 3 so as to be able to tilt up and down, and an arm 5 is attached to the tip of the boom 4 so as to be able to rotate up and down, and a bucket 6 is attached to the tip of the arm 5 so as to be able to rotate up and down.

[0017] The bucket 6 is an example of a work tool. The bucket 6 is used, for example, for excavation work. In addition, instead of the bucket 6, another work tool may be attached to the tip of the arm 5 depending on the type of work, etc. The other work tool may be, for example, another type of bucket, such as a large bucket, a slope bucket, or a dredging bucket. The other work tool may also be a type of work tool other than a bucket, such as an agitator, a breaker, or a grapple.

[0018] The boom 4, arm 5, and bucket 6 are hydraulically driven by hydraulic oil discharged from a main pump 14 (described later) through a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, which serve as hydraulic actuators.

[0019] The excavator 100 may be configured such that some of the driven elements, such as the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, and the bucket 6, are electrically driven. That is, the excavator 100 may be a hybrid excavator, an electric excavator, or the like, in which some of the driven elements are driven by electric actuators.

[0020] The imaging device S6 is an example of a spatial recognition device, and captures images of the periphery of the shovel 100 and acquires image data showing the periphery of the shovel 100. The imaging device S6 includes a camera S6F that captures images in front of the shovel 100, a camera S6L that captures images to the left of the shovel 100, a camera S6R that captures images to the right of the shovel 100, and a camera S6B that captures images behind the shovel 100.

[0021] Camera S6F is attached to the outside of the cabin 10, such as on the roof of the cabin 10 or on the side of the boom 4. Camera S6F may also be attached, for example, to the ceiling of the cabin 10, i.e., inside the cabin 10. Camera S6L is attached to the left end of the top surface of the upper rotating body 3, camera S6R is attached to the right end of the top surface of the upper rotating body 3, and camera S6B is attached to the rear end of the top surface of the upper rotating body 3.

[0022] The image capturing device S6 (cameras S6F, S6B, S6L, and S6R) is, for example, a single-lens wide-angle camera having a very wide angle of view. The image capturing device S6 has an image capturing element such as a CCD or CMOS, and outputs captured images to the display device 40. Furthermore, image information captured by the image capturing device S6 is also input to the controller 30.

[0023] The image capturing device S6 according to the present embodiment is an example of a spatial recognition device, and it is sufficient if it can detect features of stripes and objects present around the work machine. The image capturing device S6 may also be a stereo camera, a distance imaging camera, or the like.

[0024] The controller (control device) 30 is a control device for controlling the shovel 100. For example, the controller 30 is mainly configured with a computer including a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), a non-volatile auxiliary storage device, various input / output interfaces, etc. The controller 30 then reads programs from the non-volatile storage device, loads them into a volatile storage device, and has the CPU execute the programs to realize various functions. The various functions include, for example, a machine guidance function that guides the operator in manually operating the shovel 100. The controller 30 may also include a contact avoidance function that automatically or autonomously operates or stops the shovel 100 to avoid contact between the shovel 100 and an object present within a monitoring range around the shovel 100.

[0025] For example, the controller 30 sets a target rotation speed based on an operation by an operator or the like, and performs drive control to rotate the engine 11 at a constant speed.

[0026] The boom angle sensor S1 is attached to the boom 4 and detects the elevation / depression angle of the boom 4 relative to the upper rotating unit 3 (hereinafter referred to as the "boom angle"), for example, the angle formed by a line connecting the fulcrums at both ends of the boom 4 relative to the rotation plane of the upper rotating unit 3 in a side view. The boom angle sensor S1 may include, for example, a rotary encoder, an acceleration sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), etc. The boom angle sensor S1 may also include a potentiometer using a variable resistor, a cylinder stroke sensor that detects the stroke amount of a hydraulic cylinder (boom cylinder 7) corresponding to the boom angle, etc. The same applies to the arm angle sensor S2 and the bucket angle sensor S3 below. A detection signal corresponding to the boom angle detected by the boom angle sensor S1 is input to the controller 30.

[0027] The arm angle sensor S2 is attached to the arm 5 and detects the rotation angle of the arm 5 relative to the boom 4 (hereinafter referred to as the "arm angle"), for example, the angle formed by a line connecting the fulcrums at both ends of the arm 5 with a line connecting the fulcrums at both ends of the boom 4 in a side view. A detection signal corresponding to the arm angle detected by the arm angle sensor S2 is input to the controller 30.

[0028] Bucket angle sensor S3 is attached to bucket 6 and detects the rotation angle of bucket 6 relative to arm 5 (hereinafter referred to as "bucket angle"), for example, the angle, in a side view, formed by a line connecting the fulcrum of bucket 6 and its tip (toe) with respect to a line connecting the fulcrums at both ends of arm 5. A detection signal corresponding to the bucket angle from bucket angle sensor S3 is input to controller 30.

[0029] The machine body tilt sensor S4 (an example of an attitude detection unit) detects the tilt state of the machine body (the upper rotating body 3 or the lower running body 1) with respect to a horizontal plane. The machine body tilt sensor S4 is attached to, for example, the upper rotating body 3, and detects the tilt angles of the excavator 100 (i.e., the upper rotating body 3) about two axes in the fore-aft and lateral directions (hereinafter referred to as the "fore-aft tilt angle" and the "lateral tilt angle"). The machine body tilt sensor S4 may include, for example, a rotary encoder, an acceleration sensor, a six-axis sensor, an IMU, etc. Detection signals corresponding to the tilt angles (fore-aft tilt angle and lateral tilt angle) by the machine body tilt sensor S4 are input to the controller 30.

[0030] The swing angle sensor S5 outputs detection information related to the swing state of the upper swing body 3. The swing angle sensor S5 detects, for example, the swing angular velocity and swing angle of the upper swing body 3. The swing angle sensor S5 may include, for example, a gyro sensor, a resolver, a rotary encoder, etc. The detection signals corresponding to the swing angle and swing angular velocity of the upper swing body 3 detected by the swing angle sensor S5 are input to the controller 30.

[0031] The positioning device PS measures the position and orientation of the upper rotating body 3. The positioning device PS is, for example, a Global Navigation Satellite System (GNSS) compass, and detects the position and orientation of the upper rotating body 3, and a detection signal corresponding to the position and orientation of the upper rotating body 3 is input into the controller 30. Furthermore, among the functions of the positioning device PS, the function of detecting the orientation of the upper rotating body 3 may be substituted by a direction sensor attached to the upper rotating body 3.

[0032] The cabin 10 is a control room where an operator sits, and is mounted on the front left side of the upper rotating body 3.

[0033] As will be described later, the cabin 10 may be omitted when the excavator 100 is remotely operated or operates in a fully automatic manner.

[0034] The communication device T1 communicates with external devices through a predetermined network including a mobile communication network with a base station as an end, a satellite communication network, the Internet, etc. The communication device T1 is, for example, a mobile communication module compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation), or a satellite communication module for connecting to a satellite communication network.

[0035] The excavator 100 operates actuators (e.g., hydraulic actuators) in response to operations by an operator in the cabin 10, and drives operating elements (hereinafter referred to as "driven elements") such as the lower running body 1, upper rotating body 3, boom 4, arm 5, and bucket 6.

[0036] Furthermore, instead of or in addition to being configured to be operable by an operator in the cabin 10, the shovel 100 may be configured to be remotely operable from outside the shovel 100. When the shovel 100 is remotely operated, the inside of the cabin 10 may be unmanned.

[0037] The excavator 100 may also automatically operate the actuators regardless of the operation by the operator. This allows the excavator 100 to realize a function of automatically operating at least some of the driven elements such as the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, and the bucket 6, i.e., a so-called "automatic driving function" or "machine control function."

[0038] The automatic driving function may include a function for automatically operating driven elements (actuators) other than the driven element (actuator) to be operated in response to an operator's operation of the control device 26 or remote operation, i.e., a so-called "semi-automatic driving function" or "operation-assisted machine control function." The automatic driving function may also include a function for automatically operating at least some of the multiple driven elements (hydraulic actuators) without the operator's operation of the control device 26 or remote operation, i.e., a so-called "fully automatic driving function" or "fully automatic machine control function." When the fully automatic driving function is enabled in the excavator 100, the interior of the cabin 10 may be unmanned. The semi-automatic driving function, fully automatic driving function, etc. may also include a mode in which the operation of the driven element (actuator) to be operated automatically is determined according to predetermined rules. In addition, the semi-automatic driving function and the fully automatic driving function may include a mode in which the excavator 100 autonomously makes various decisions and, based on the results of those decisions, autonomously determines the operation content of the driven element (hydraulic actuator) that is the target of automatic driving (so-called "automatic driving function").

[0039] FIG. 3 is a diagram schematically illustrating an example of the configuration of the shovel 100 according to this embodiment.

[0040] In FIG. 3, the mechanical power system, hydraulic oil lines, pilot lines, and electrical control system are indicated by double lines, solid lines, dashed lines, and dotted lines, respectively.

[0041] The drive system of the excavator 100 according to this embodiment includes the engine 11, the regulator 13, the main pump 14, and the control valve unit 17. Furthermore, the hydraulic drive system of the excavator 100 according to this embodiment includes hydraulic actuators such as the traveling hydraulic motors 1L, 1R, the swing hydraulic motor 2A, the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9 that hydraulically drive the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, and the bucket 6, respectively, as described above.

[0042] The engine 11 is the main power source in the hydraulic drive system and is mounted, for example, on the rear of the upper rotating body 3. Specifically, the engine 11 rotates at a constant speed at a preset target speed under direct or indirect control by a controller 30 (described later), and drives the main pump 14 and the pilot pump 15. The engine 11 is, for example, a diesel engine that uses diesel as fuel.

[0043] The regulator 13 controls the discharge amount of the main pump 14. For example, the regulator 13 adjusts the angle (tilt angle) of the swash plate of the main pump 14 in response to a control command from the controller 30.

[0044] The main pump 14 is mounted on the rear of the upper rotating body 3, for example, similar to the engine 11, and supplies hydraulic oil to the control valve unit 17 through a high-pressure hydraulic line. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, under the control of the controller 30, the tilt angle of the swash plate is adjusted by the regulator 13, thereby adjusting the stroke length of the piston and controlling the discharge flow rate (discharge pressure).

[0045] The control valve unit 17 is a hydraulic control device that controls the hydraulic system in the excavator 100. In this embodiment, the control valve unit 17 includes control valves 171 to 176. The control valve 175 includes a control valve 175L and a control valve 175R, and the control valve 176 includes a control valve 176L and a control valve 176R. The control valve unit 17 is configured to selectively supply hydraulic oil discharged by the main pump 14 to one or more hydraulic actuators via the control valves 171 to 176. The control valves 171 to 176 control, for example, the flow rate of hydraulic oil flowing from the main pump 14 to the hydraulic actuators and the flow rate of hydraulic oil flowing from the hydraulic actuators to a hydraulic oil tank. The hydraulic actuators include a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, traveling hydraulic motors 1L and 1R, and a swing hydraulic motor 2A. More specifically, the control valve 171 corresponds to the left traveling hydraulic motor 1L, the control valve 172 corresponds to the right traveling hydraulic motor 1R, and the control valve 173 corresponds to the swing hydraulic motor 2A. Furthermore, the control valve 174 corresponds to the bucket cylinder 9, the control valve 175 corresponds to the boom cylinder 7, and the control valve 176 corresponds to the arm cylinder 8.

[0046] The pilot pump 15 is an example of a pilot pressure generating device, and is configured to supply hydraulic oil to hydraulic control devices via a pilot line. In this embodiment, the pilot pump 15 is a fixed displacement hydraulic pump. However, the pilot pressure generating device may be realized by the main pump 14. That is, the main pump 14 may have a function of supplying hydraulic oil to various hydraulic control devices via a pilot line, in addition to a function of supplying hydraulic oil to the control valve unit 17 via a hydraulic oil line. In this case, the pilot pump 15 may be omitted.

[0047] The discharge pressure sensor 28 is configured to detect the discharge pressure of the main pump 14. In the present embodiment, the discharge pressure sensor 28 outputs the detected value to the controller 30. The discharge pressure sensor 28 includes, for example, discharge pressure sensors 28L and 28R, as described below.

[0048] The operating device 26 is a device used by an operator to operate the actuator. The operating device 26 includes, for example, an operating lever and an operating pedal. The actuator includes at least one of a hydraulic actuator and an electric actuator.

[0049] The operation sensor 29 is configured to detect the operation content of the operator using the operation device 26. In this embodiment, the operation sensor 29 detects the operation direction and operation amount of the operation device 26 corresponding to each actuator and outputs the detected value to the controller 30. In this embodiment, the controller 30 controls the opening area of ​​the proportional valve 31 in accordance with the output of the operation sensor 29. The controller 30 then supplies the hydraulic oil discharged from the pilot pump 15 to the pilot ports of the corresponding control valves in the control valve unit 17. The pressure of the hydraulic oil supplied to each pilot port (pilot pressure) is, in principle, a pressure corresponding to the operation direction and operation amount of the operation device 26 corresponding to each hydraulic actuator. In this way, the operation device 26 is configured to supply the hydraulic oil discharged from the pilot pump 15 to the pilot ports of the corresponding control valves in the control valve unit 17.

[0050] The proportional valve 31, which functions as a control valve for machine control, is disposed in a pipe connecting the pilot pump 15 and the pilot port of the control valve in the control valve unit 17, and is configured so that the flow path area of ​​the pipe can be changed. In this embodiment, the proportional valve 31 operates in response to a control command output by the controller 30. Therefore, the controller 30 can supply the hydraulic oil discharged by the pilot pump 15 to the pilot port of the control valve in the control valve unit 17 via the proportional valve 31, regardless of the operation of the operating device 26 by the operator.

[0051] With this configuration, the controller 30 can operate the hydraulic actuator corresponding to a specific operating device 26 even when no operation is being performed on that specific operating device 26.

[0052] The control system of the shovel 100 according to this embodiment includes a controller 30, a display device 40, an input device 42, a storage medium 47, and a communication device T1.

[0053] The controller 30 (an example of a control device) is provided, for example, in the cabin 10 and controls the driving of the excavator 100. The functions of the controller 30 may be realized by any hardware, software, or a combination thereof. For example, the controller 30 is configured mainly with a microcomputer including a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), a non-volatile auxiliary storage device, various input / output interfaces, etc. The controller 30 realizes various functions by, for example, executing various programs stored in the ROM or non-volatile auxiliary storage device on the CPU.

[0054] For example, the controller 30 sets a target rotation speed based on an operation by an operator or the like, and performs drive control to rotate the engine 11 at a constant speed.

[0055] Furthermore, for example, the controller 30 outputs a control command to the regulator 13 as necessary to change the discharge rate of the main pump 14 .

[0056] Furthermore, for example, the controller 30 performs control related to a machine guidance function that guides (provides guidance for) the manual operation of the shovel 100 by the operator via the operation device 26. Furthermore, the controller 30 performs control related to a machine control function that automatically assists the manual operation of the shovel 100 by the operator via the operation device 26.

[0057] Note that some of the functions of the controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers. For example, the machine guidance function and the machine control function may be realized by a dedicated controller (control device).

[0058] The display device 40 is provided in a location that is easily visible to an operator seated in the cabin 10, and displays various information images under the control of the controller 30. The display device 40 may be connected to the controller 30 via an in-vehicle communication network such as a Controller Area Network (CAN), or may be connected to the controller 30 via a one-to-one dedicated line.

[0059] The input device 42 is provided within reach of an operator seated in the cabin 10, accepts various operational inputs from the operator, and outputs signals corresponding to the operational inputs to the controller 30. The input device 42 includes a touch panel mounted on the display of a display device that displays various information images, knob switches provided at the tips of the lever portions of the operation levers 26A to 26B, button switches, levers, toggles, rotary dials, etc. provided around the display device 40. A signal corresponding to the content of an operation on the input device 42 is taken in by the controller 30.

[0060] The storage medium 47 is provided, for example, inside the cabin 10, and stores various pieces of information under the control of the controller 30. The storage medium 47 is, for example, a non-volatile storage device such as a semiconductor memory. The storage medium 47 may store information output by various devices while the shovel 100 is operating, or may store information obtained via various devices before the operation of the shovel 100 is started.

[0061] [Functional Configuration of Controller] Next, an overview of the machine guidance function and the machine control function of the shovel 100 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the controller 30 and the communication terminal 500 of the shovel 100 according to this embodiment.

[0062] The communication terminal 500 includes an imaging device 501, a CPU 502, and a display device 503. The communication terminal 500 is, for example, a terminal carried by the operator of the shovel 100.

[0063] The imaging device 501 captures an image of the periphery of the communication terminal 500 and acquires image data showing the periphery of the communication terminal 500 .

[0064] The CPU 502 controls the communication terminal 500. For example, the CPU 502 reads a program from a storage medium (not shown), loads it into a volatile storage device (not shown), and executes it to realize various functions.

[0065] The display device 503 displays various information images under the control of the CPU 502 .

[0066] As shown in FIG. 4, the controller 30 is connected to the in-vehicle image capturing device S11 and a storage medium 47.

[0067] The in-vehicle image capturing device S11 is, for example, a driving recorder that captures images of the surroundings of the shovel 100 and also captures images of the inside of the cabin 10 of the shovel 100, and records the captured image data, for example, in the storage medium 47. The in-vehicle image capturing device S11 may also transmit the captured image data to the controller 30.

[0068] The storage medium 47 is a non-volatile readable / writable storage medium (e.g., an SSD), and stores a trained model 47A and a setting storage unit 47B.

[0069] The setting storage unit 47B stores settings for operations of the shovel 100 that are performed according to the type of object. The setting storage unit 47B, for example, stores the type of object and the operation that is performed when that type of object is detected, in association with each other. The setting storage unit 47B may also store conditions for performing an operation in association with each other. The setting storage unit 47B according to the present embodiment may associate one or more of the size of the object detected by the imaging device S6 and the area in which the object is detected in the image information, as the conditions for performing an operation.

[0070] The following describes the configuration for controlling an operation based on a detected object by the controller 30. The controller 30 includes, as machine guidance and machine control functions, an object detection unit 301, a stripe pattern detection unit 302, a determination unit 303, a storage unit 304, an operation identification unit 305, an operation control unit 306, and a display control unit 307.

[0071] The object detection unit 301 inputs image information captured by the imaging device S6 into the trained model 47A that has undergone machine learning and is stored in the storage medium 47, and receives the type of object captured in the image information and the coordinate area (in the image information) in which the object is captured from the trained model 47A. In this way, the object detection unit 301 identifies the type of object present in the monitoring area around the excavator 100 and the area in the image information in which the object is captured.

[0072] Specifically, the object detection unit 301 loads the trained model 47A from the storage medium 47 into a main storage device such as RAM and executes it on the CPU, thereby identifying the type of object around the shovel 100 using the trained model 47A.

[0073] The trained model 47A is mainly composed of a neural network. The neural network of the trained model 47A may be a so-called deep neural network having one or more intermediate layers (hidden layers) between an input layer and an output layer. In the neural network, weighting parameters representing the connection strength with the lower layer are defined for each of the multiple neurons constituting each intermediate layer. The neural network is configured in such a manner that the neurons in each layer output the sum of values ​​obtained by multiplying each of the input values ​​from the multiple neurons in the upper layer by the weighting parameters defined for each neuron in the upper layer to the neurons in the lower layer via a threshold function.

[0074] Machine learning, specifically, deep learning, is performed on the trained model 47A, and as a result, the weighting parameters of the neural network are optimized.

[0075] The training data used for machine learning includes, for example, image information showing an object, information identifying the type of the object, and the coordinate area (occupied area) where the object is shown. By performing machine learning using the training data, when image information is input, the trained model 47A outputs information identifying the type of object shown in the image information and the coordinate area of ​​the object shown.

[0076] The neural network of the trained model 47A may be, for example, a convolutional neural network (CNN). A CNN is a neural network that applies existing image processing techniques (convolution processing and pooling processing). Specifically, a CNN extracts feature data (feature map) smaller in size than the input image information by repeatedly combining convolution processing and pooling processing on an input image. The pixel values ​​of each pixel in the extracted feature map are then input to a neural network composed of multiple fully connected layers, and the output layer of the neural network can output, for example, a predicted probability of the presence of each object type.

[0077] In this way, the trained model 47A may be configured to receive image information and output the coordinates and size of an object in the image information (i.e., the area occupied by the object on the input image) and the type of the object.

[0078] That is, the trained model 47A may be configured to detect an object shown in the image information (determine the area in which the object is shown in the image information) and determine the type of the object. In this case, the trained model 47A may output the image information in the form of image information to which information regarding the area occupied by the object shown in the image information and the type of the object is superimposed.

[0079] For example, when a person is captured in image information, the trained model 47A outputs a message indicating that a person is captured, as well as the coordinates and size of the captured person (i.e., the area occupied by the object on the input image). Furthermore, the trained model 47A can also identify objects other than people.

[0080] 5 is a diagram showing an example of the types of objects that can be identified by the trained model 47A according to this embodiment. For example, when the input image information includes the shape of an object shown in FIG. 5, the trained model 47A outputs information that identifies the type of object.

[0081] The trained model 47A according to this embodiment is capable of identifying safety jackets, road cones, gates, face masks, barricades, and guard fences, as shown in FIG. 5 .

[0082] Furthermore, the trained model 47A can identify, for example, work machines including shovels, vehicles, guardrails, bars (including cones), signs, billboards, batons, poles, ropes, megaphones, speakers, pipe covers, chain stands, flags, curtains, streamers, etc. Furthermore, the trained model 47A may be capable of recognizing animals, buildings, etc.

[0083] In this embodiment, the controller 30 detects the above-mentioned types of objects. However, the objects to be detected have fewer features than people, making reliable detection difficult. For example, a triangular object at a work site may be mistaken for a traffic cone.

[0084] Therefore, in this embodiment, objects to be detected by the controller 30 are given a striped pattern. In other words, the controller 30 detects objects present at the work site that have a striped pattern. Even if the controller 30 mistakenly detects a triangular prism present at the work site as a road cone, the controller 30 will not perform an operation based on the mistaken detection unless the triangular prism has a striped pattern. Therefore, it is possible to suppress malfunction of the excavator 100 and achieve improved safety.

[0085] Returning to FIG. 4, the stripe pattern detection unit 302 identifies an area to which a stripe pattern is applied from the image information captured by the image capture device S6.

[0086] The region to which the stripes are applied according to this embodiment is a region in which regions of two different colors are alternately repeated.

[0087] The two different colors in the striped pattern preferably have high contrast, but they need only be distinguishable by workers at the work site. Possible two different colors include, but are not limited to, yellow and black, white and red, white and green, yellow-red and white, yellow and white, and red and yellow. For example, JIS safety colors may be applied. A combination of one of the JIS safety colors (e.g., JIS Z 9101:2018), such as red, yellow, green, or blue, with one of the two contrasting colors (white and black) may also be used. Using these color combinations can also help prevent danger and alert workers.

[0088] That is, in this embodiment, stripes are often used at work sites as patterns to prevent danger or to call the attention of workers. For this reason, protective equipment, ropes, tapes, etc. with stripes are easily available at work sites. Therefore, in this embodiment, stripes are used as patterns for controlling the operation of the shovel 100.

[0089] In this embodiment, it is possible to easily obtain a pattern for controlling the operation of the shovel 100 and also to alert workers in the vicinity.

[0090] In addition, the number of repetitions may be predetermined in order to detect a region as having a striped pattern. For example, the stripe pattern detection unit 302 may detect a region as having a striped pattern when two different color bands are repeated five or more times.

[0091] Furthermore, it is preferable that the first area of ​​one of the two different color regions and the second area of ​​the other of the two different color regions are approximately equal. However, the first area and the second area are not limited to being approximately equal, and for example, the first area may be larger or smaller than the second area. However, the first area and the second area have areas that can be detected from the shovel 100 at least several meters away.

[0092] Furthermore, if an object with a striped pattern is present in an area detectable by the imaging device S6, the striped pattern detection unit 302 detects the area with the striped pattern regardless of the orientation of the object.

[0093] For example, an object with a striped pattern (e.g., a cone or tape) may be arranged from the front to the back of the shovel 100 relative to the shovel 100. In this case, the striped pattern captured in the image information transferred to the imaging device S6 has a larger area of ​​color in the front and a smaller area of ​​color toward the back. Even in such a case, the striped pattern detection unit 302 appropriately detects the area with the striped pattern. Furthermore, the striped pattern detection unit 302 may detect the orientation of the area with the striped pattern based on the change in area for each color.

[0094] As a modified example, the stripe pattern detection unit 302 may detect a striped area only if two color bands within the striped area are arranged diagonally. In other words, vertical or horizontal stripes may be applied to nature, ordinary structures, and ordinary objects. Therefore, the stripe pattern detection unit 302 in this modified example detects only striped areas in which two color bands are arranged diagonally. The diagonal line separating the color bands may be at any angle, but may be approximately 45 degrees, for example, based on the layout of safety markings (e.g., JIS Z 9101:2018). This allows the operation of the excavator 100 to be controlled according to the layout typically used for signs, etc. In this modified example, by controlling operation when diagonal stripes are detected, false detections can be suppressed and reliability can be improved.

[0095] When the object detection unit 301 identifies the type of object present around the shovel 100 from the image information captured by the imaging device S6, the determination unit 303 determines whether the object and the striped pattern are associated based on the positional relationship between the area (on the image information) of the object identified by the object detection unit 301 and the area (on the image information) of the striped pattern identified by the striped pattern detection unit 302.

[0096] For example, the determination unit 303 determines whether an area of ​​the image information where a striped pattern exists is included in an area where an object exists. If the area where a striped pattern exists is included in an area where an object exists, the determination unit 303 according to this embodiment can determine that the object has a striped pattern. In this case, the determination unit 303 determines that the object detected by the object detection unit 301 is associated with the striped pattern.

[0097] An object with a striped pattern means that it is an object related to the operation control of the shovel 100. In other words, even if an object is erroneously detected, if the object does not have a striped pattern, operation control based on the object will not be performed. In other words, in this embodiment, erroneous detection of objects unrelated to operation control can be suppressed, and detection accuracy can be improved.

[0098] Note that the determination unit 303 according to this embodiment is not limited to cases where stripes are directly applied to an object. For example, the determination unit 303 may determine that an object is associated with a striped pattern when the distance between the area where the striped pattern exists and the area where the object exists is closer than a predetermined threshold (e.g., a predetermined number of dots). Furthermore, the determination unit 303 may determine that an object is associated with a striped pattern even when the stripes surround at least a portion of the object or when the stripes are applied to the edge of the object. In other words, although various patterns of stripes may be applied to objects, the controller 30 according to this embodiment can recognize whether an object is associated with a striped pattern by making the above-described determination, regardless of the various patterns. In other words, it becomes easier for a worker at a work site to prepare an object with a striped pattern, which makes it easier to control the operation of the shovel 100, thereby improving work efficiency.

[0099] For example, striped tape may be attached to surround the surface portion in FIG. 5 . In this case, the determination unit 303 may determine that a portion of the surface portion detected by the object detection unit 301 that is surrounded by the striped tape is a surface portion having a striped pattern. The controller 30 may then perform character recognition or the like only on the area determined to be the surface portion. The controller 30 may control the operation of the excavator 100 in accordance with the recognized characters or the like. For example, in the present embodiment, an area surrounded by striped tape may exist on the rear portion of the dump truck. In this case, the controller 30 may perform character recognition on the area and, if the characters extracted by the character recognition are predetermined characters, perform operation control such as loading earth and sand.

[0100] Furthermore, the determination unit 303 may determine whether the distance between the area where the object exists and the area where the striped pattern exists in the image information is closer than a predetermined threshold. If it is determined that the distance between the area where the object exists and the area where the striped pattern exists is closer than the predetermined threshold, it determines that the object and the striped pattern are associated with each other, in other words, that the striped pattern is applied to the object. The predetermined threshold is determined depending on the embodiment, so a description thereof will be omitted.

[0101] The saving unit 304 saves the setting information acquired from the in-vehicle image capturing device S11 in the setting storage unit 47B. The setting information stores the type of object and the operation control to be performed on the excavator 100 when the object of that type is detected, in association with each other. Furthermore, the setting information may include further conditions for performing the operation control.

[0102] FIG. 6 is a diagram illustrating an example of the area visible ahead from the driver's seat 50 in the cabin 10 of the excavator 100 of this embodiment.

[0103] As shown in Figure 6, bucket 6 can be seen from front window 110 of cabin 10. Cabin 10 has driver's seat 50 in the center, with control levers 26A, 26B located on either side. Display device 40 and input device 42 are located to the right front of driver's seat 50 (below and right of the front window). The operator sits in driver's seat 50 and operates control lever 26A with his left hand and control lever 26B with his right hand to move bucket 6 to a desired position and perform excavation work.

[0104] 6, a sticker with a two-dimensional code 1601 is attached below the left side window 112. The two-dimensional code 1601 includes a uniform resource locator (URL) for installing an application for setting the operation of the excavator 100 (hereinafter referred to as an operation setting application) on the mobile terminal.

[0105] In this embodiment, an example is described in which a sticker with two-dimensional code 1601 is attached under the left side window 112, but the position where the sticker with two-dimensional code 1601 is attached is not limited, and it may be attached under the right side window 111, for example.

[0106] An in-vehicle image capturing device S11 is provided above the front windshield 110. Images of the situation inside the cabin 10 can be captured by the in-vehicle image capturing device S11.

[0107] In this embodiment, when the operator captures an image of the two-dimensional code 1601 with the imaging device 501 of the communication terminal 500, an operation setting application 521 for setting the operation of the shovel 100 is installed in the communication terminal 500. Then, the CPU 502 of the communication terminal 500 executes the operation setting application 521 to realize various functions.

[0108] Fig. 7 is a diagram showing screen transitions of the communication terminal 500 according to this embodiment. The CPU 502 of the communication terminal 500 executes the operation setting application 521 to display a screen 1701 shown in Fig. 7. The screen 1701 includes a detection target selection field 1711, a detection area selection field 1712, a display size input field 1713, a control selection field 1714, and an OK button 1715.

[0109] The detection target selection field 1711 is a field for selecting the type of object to be detected.

[0110] The detection area selection field 1712 is a field for selecting an area in which to detect a selected type of object. The detection area selection field 1712 allows selection of, for example, the "lower area of ​​the image." The closer the object is to the shovel 100, the lower the area in which the object appears in the image information will be. Therefore, by selecting an area in which to detect the object (for example, the lower area), the relative positional relationship between the shovel 100 and the object can be set when performing operation control. Note that if no area is selected in the detection area selection field 1712, the entire area of ​​the image information will be subject to detection.

[0111] The display size input field 1713 is a field for setting the size at which the selected type of object is detected. The closer the object is to the shovel 100, the larger the size at which the object is displayed. Therefore, by setting the size at which the object is detected, the relative positional relationship between the shovel 100 and the object can be set when performing operation control. Note that if no size is set in the display size input field 1713, the object will be detected regardless of its size.

[0112] The control selection field 1714 is a field for selecting an operation to be performed by the shovel 100 when an object of the selected type is detected and when the conditions set in the detection target selection field 1711 and the display size input field 1713 are satisfied. Note that if no conditions are set in the detection target selection field 1711 and the display size input field 1713, the field is used to select an operation to be performed by the shovel 100 when an object of the selected type is detected.

[0113] The selectable operation may be any operation that can be performed by the shovel 100. Examples of the selectable operation include decelerating and stopping the shovel 100, stopping the attachment AT, notifying those in the vicinity, or displaying a warning screen on the display device 40 or the like. Furthermore, the selectable operation may be starting to save image information (still images or video) captured by the imaging device S6 or the in-vehicle imaging device S11. It may also be starting to record audio using a microphone provided in the shovel 100. It may also be starting a remote conference with an external device using images captured by the in-vehicle imaging device S11 or the like.

[0114] An OK button 1715 is a button to be pressed when selection and setting are completed in the detection target selection field 1711, the detection area selection field 1712, the display size input field 1713, and the control selection field 1714.

[0115] When the CPU 502 of the communication terminal 500 receives the selection of the OK button 1715, it displays a screen 1702. The screen 1702 displays a detection target selection field 1711, a detection area selection field 1712, a display size input field 1713, and a two-dimensional code 1721 for acquiring setting information including the content selected or set in the control selection field 1714.

[0116] When the in-vehicle image capturing device S11 captures the two-dimensional code 1721, the storage unit 304 of the excavator 100 extracts the setting information contained in the two-dimensional code 1721 and stores the setting information in the setting memory unit 47B.

[0117] This allows the shovel 100 to perform the operation set by the communication terminal 500. In the present embodiment, a case where an operation is set by the communication terminal 500 will be described, but the present invention is not limited to the case where an operation is set by the communication terminal 500. For example, an operation may be set based on information input by the input device 42. In this way, the operation to be performed by the shovel 100 may be determined based on information input from the input device (an example of an operation device) 42 or the communication terminal (an example of an external terminal) 500. In other words, in the present embodiment, it is possible to set control for the operation of the shovel 100 in accordance with the situation at the work site, etc. Therefore, it is possible to achieve improved safety because operation can be performed in accordance with the situation, and improved convenience because flexible operation can be performed.

[0118] The action identification unit 305 identifies the action of the shovel 100 based on the determination result by the determination unit 303 and the setting information stored in the setting storage unit 47B. For example, when a predetermined type of object is detected in the determination result, the action identification unit 305 identifies the action according to the type of the object based on the setting information, etc. Furthermore, the action identification unit 305 may identify the action based on conditions such as the area in which the object is captured and the size of the object captured.

[0119] The operation control unit 306 performs various controls in accordance with the operation identified by the operation identification unit 305. For example, the operation control unit 306 outputs a control command to the proportional valve 31 corresponding to the operation identified by the operation identification unit 305 so as to perform the operation. Therefore, the shovel 100 according to this embodiment can perform an operation according to the type of object detected.

[0120] The display control unit 307 controls the display of information on the display device 40 .

[0121] When the determination unit 303 determines that an object having a striped pattern is present, the display control unit 307 superimposes a frame (information) indicating the area where the object is present on the image information captured by the imaging device S6 and displays the frame (information) on the display device 40. Note that this embodiment is not limited to the example in which the frame (information) indicating the area where the object is present is superimposed on the image information captured by the imaging device S6, but may be superimposed on screen information generated based on the image information.

[0122] Furthermore, although the present embodiment describes an example in which a frame is superimposed on an object with a striped pattern, the display control unit 307 may separately display a frame (information) indicating an area in which an object detected by the object detection unit 301 exists, and a frame (information) indicating an area in which a striped pattern detected by the striped pattern detection unit 302 exists. Furthermore, the display control unit 307 may display only one of the frame (information) indicating an area in which an object exists, and the frame (information) indicating an area in which a striped pattern exists.

[0123] The display control unit 307 according to the present embodiment is not limited to displaying a frame representing an object or a striped pattern superimposed on image information. For example, the display control unit 307 may display a pop-up window indicating that an object with a striped pattern has been detected. Furthermore, the display control unit 307 may display a pop-up window indicating that a striped pattern has been detected.

[0124] In this embodiment, a frame (information) indicating the above-mentioned area is displayed, allowing the operator to recognize objects present around the shovel 100. Therefore, the operator can recognize the action to be performed in response to the object, and can understand what action to take. This can improve operability. Furthermore, being able to make the operator aware of the action in advance can, in other words, prevent the operator from performing an action unintended by the operator, thereby improving safety.

[0125] Next, an example of operation control of the shovel 100 according to this embodiment will be described. Note that in this embodiment, the operation control when an object with a striped pattern is detected includes various operation controls other than the operation control based on the setting information.

[0126] Fig. 8 is a diagram showing an example of the arrangement of striped objects prepared for controlling the operation of the shovel 100 according to this embodiment. In the example shown in Fig. 8, striped road cones 1811 to 1814, striped cone bars 1821 and 1822, and striped tape 1841 are provided around the shovel 100. The tape 1841 is, for example, attached to the ground.

[0127] In the example shown in FIG. 8 , the attachment AT of the shovel 100 is configured not to come into contact with the cone bars 1821, 1822. In other words, when the determination unit 303 determines that the cone bars 1821, 1822 have stripes, and when the determination unit 303 determines that the attachment AT has come closer to the cone bars 1821, 1822 than a predetermined distance in accordance with the operator's operation, the operation control unit 306 performs control to stop the attachment AT. This allows the shovel 100 and the shovel 1800 to perform work without interfering with each other. Note that the predetermined distance may be, for example, 2 m, or may be any distance determined depending on the implementation. In this embodiment, the case where the attachment AT comes closer to the cone bars 1821, 1822 than a predetermined distance has been described. However, in this embodiment, the configuration used as the basis for determining the stop control is not limited to the attachment AT, and may be any configuration of the shovel 100.

[0128] As another example, the setting information may be configured to control the shovel 100 to decelerate and stop when tape is detected in the area below the image information. In this case, when the determination unit 303 determines that tape 1841 with a striped pattern is present in the area below the image information, the operation control unit 306 controls the shovel 100 to decelerate and stop. As a result, the shovel 100 decelerates and stops when it approaches the vicinity of tape 1841. Therefore, the shovel 100 can be prevented from moving outside the working area.

[0129] Furthermore, the controller 30 according to this embodiment may perform operation control by combining human detection and striped object detection. For example, if the object detection unit 301 further detects a person 1831 and a person 1832 and the determination unit 303 determines that the cone bar 1822 has a striped pattern, the determination unit 303 may further determine the positional relationship between the striped cone bar 1822 and the person 1831 and the person 1832.

[0130] If the determination unit 303 determines that the person 1831 is closer to the shovel 100 than the striped cone bar 1822, in other words, if the determination unit 303 determines that the person 1831 has entered in front of the cone bar 1822, the operation control unit 306 may control the image capture device S6 or the in-vehicle image capture device S11 to start capturing an image in the direction from which the person 1831 has entered. The operation control unit 306 may also control the output of a warning sound in the direction from which the person 1831 is present. In the present embodiment, the condition for starting the control is a case in which it is determined that the person is closer to the shovel 100 than the striped object. However, in this embodiment, the start of the control is not limited to a case in which the above-described condition is satisfied, but may be a case in which the positional relationship between the person and the object determined to be associated with the striped pattern satisfies a predetermined condition. The predetermined condition may be determined according to the implementation. In the present embodiment, the above-described control is started based on the positional relationship between the person and the object, enabling control according to the surrounding situation, including the person, thereby achieving improved safety and convenience.

[0131] Furthermore, the controller 30 may detect the movement of the object to which the striped pattern is applied, and perform operation control based on the detected movement.

[0132] 9 is a diagram showing an example of the arrangement of striped objects prepared for controlling the operation of the shovel 100 according to this embodiment. In the example shown in FIG. 9, fences 1901 to 1905 with striped patterns are provided around the shovel 100. The fences 1901 to 1905 are arranged so as to surround the working area of ​​the shovel 100.

[0133] In the example shown in Figure 9, the attachment AT of the shovel 100 is set so as not to come into contact with the fences 1901 to 1905. In other words, if the determination unit 303 determines that the fences 1901 to 1905 are striped, when the attachment AT or the lower running structure 1 approaches the vicinity of the fences 1901 to 1905 in accordance with the operation of the operator, the operation control unit 306 performs control to stop the attachment AT or the lower running structure 1. This makes it possible to prevent the shovel 100 from coming into contact with the fences 1901 to 1905. It is then assumed that a person 1911 moves the fence 1902 in order to enter the work site.

[0134] When the determination unit 303 determines that the striped fence 1902 has moved, the operation control unit 306 may perform control to stop the operation of the shovel 100, notify the operator, warn those in the vicinity, and start capturing images of the area around the fence 1902 using the image capture device S6 or the in-vehicle image capture device S11. This can improve safety.

[0135] This embodiment is not limited to the case where an operator is on board the cabin 10 of the shovel 100, but may also be applied to the case where the shovel 100 is operated from a remote control room.

[0136] In this embodiment, the controller 30 controls the operation according to the type of object to which the striped pattern is applied, so that it is possible to cause the robot to perform an appropriate operation according to various conditions at the work site, thereby improving work efficiency and safety.

[0137] Second Embodiment In the above-described embodiment, a case has been described in which the image capture device S6 is used as the spatial recognition device for detecting the periphery of the work machine. However, the above-described embodiment does not limit the spatial recognition device for detecting the periphery of the work machine to the image capture device S6. Therefore, in the second embodiment, a case will be described in which a spatial recognition device other than the image capture device S6 is used as the spatial recognition device for detecting the periphery of the work machine.

[0138] 10 is a diagram showing an example of the configuration of the controller 30 and communication terminal 500 of the shovel 100 according to this embodiment. In this embodiment, the same reference numerals are assigned to the configurations of the above-described embodiments, and description thereof will be omitted. The shovel 100 according to this embodiment is provided with a three-dimensional measurement sensor (an example of a detection device) S12 as an example of a spatial recognition device.

[0139] The three-dimensional measuring sensor S12 is configured to recognize objects present in three-dimensional space around the shovel 100, and measure (calculate) positional relationships such as the distance and direction from the three-dimensional measuring sensor S12 or the shovel 100 to the recognized object. The three-dimensional measuring sensor S12 may include a distance sensor capable of measuring the distance and direction to objects around the shovel 100, such as an ultrasonic sensor, millimeter-wave radar, an infrared sensor, or a LIDAR (Light Detecting and Ranging). The three-dimensional measuring sensor S12 may also be any sensor capable of recognizing the distance and direction to an object, and may include an imaging device such as a stereo camera, a distance imaging camera, or a depth camera.

[0140] For example, four three-dimensional measurement sensors S12 are provided. The three-dimensional measurement sensors S12 are located near the imaging device S6 in Fig. 2. Specifically, the three-dimensional measurement sensors S12 include a forward recognition sensor attached to the front end of the upper surface of the cabin 10, a rearward recognition sensor attached to the rear end of the upper surface of the upper rotating body 3, a leftward recognition sensor attached to the left end of the upper surface of the upper rotating body 3, and a rightward recognition sensor attached to the right end of the upper surface of the upper rotating body 3. In addition, an upward recognition sensor that recognizes objects present in the space above the upper rotating body 3 may be attached to the excavator 100.

[0141] A controller 30A according to this embodiment is provided with a three-dimensional object detection unit 311 instead of the object detection unit 301 of the first embodiment.

[0142] Furthermore, the controller 30A is provided with a determination unit 303A and a motion identification unit 305A that perform different processing, instead of the determination unit 303 and the motion identification unit 305 of the above-described embodiment.

[0143] The three-dimensional object detection unit 311 acquires detection information from the three-dimensional measurement sensor S12, including the distance and direction to an object within the range that can be measured by the three-dimensional measurement sensor S12.

[0144] The three-dimensional object detection unit 311 then inputs the detection information acquired from the three-dimensional measurement sensor S12 into a trained model 47C that has undergone machine learning and is stored in the storage medium 47, and receives from the trained model 47C the shape of the object depicted in the detection information, as well as the direction and distance of the object whose shape has been identified.

[0145] In this way, the trained model 47C may be configured to receive detection information and output the shape, direction, and distance of objects present within the detection range.

[0146] The training data used for machine learning of the trained model 47C includes, for example, detection information indicating that an object exists within a detection range, the shape of the object existing within the range, and the direction and distance of the object whose shape has been identified. By performing machine learning using the training data, when detection information is input, the trained model 47C outputs the shape of the object shown in the detection information, and the distance and direction to the object.

[0147] The shape of an object output from the trained model 47C may be, for example, one or more of a step shape present on the road surface, a flat surface perpendicular to the ground, a pillar shape perpendicular to the ground, a rod shape, a surface shape, etc. Note that this embodiment shows only an example of the shape of an object, and the shape of an object to be detected may be any shape.

[0148] When the shape of an object is identified by the three-dimensional object detection unit 311, the determination unit 303A determines whether the object is associated with the striped pattern based on the positional relationship between the direction and distance of the object whose shape has been identified and the direction in which the striped pattern exists detected by the striped pattern detection unit 302. Conversion from the area in which the striped pattern appears in the image information to the direction in which the striped pattern exists at the actual work site can be performed using a well-known method based on the orientation in which the imaging device S6 is installed, etc.

[0149] For example, the determination unit 303A determines whether the direction in which the striped pattern exists is included in the range in which the object whose shape is identified exists. The determination unit 303A according to this embodiment determines that the object has a striped pattern if the direction in which the striped pattern exists is included in the range in which the object exists. In this case, the determination unit 303A determines that the object whose shape is identified and the striped pattern are associated with each other.

[0150] Note that the determination unit 303A according to the present embodiment is not limited to cases where stripes are directly applied to a striped object, but also determines that an object is associated with a striped pattern when stripes are applied to at least a portion of the object or when stripes are applied to the edge of the object. By making the above-described determination, the controller 30A according to the present embodiment can recognize whether an object is associated with a striped pattern regardless of various patterns. In other words, since it becomes easier for a worker at a work site to prepare an object associated with a striped pattern, it becomes easier to control the operation of the shovel 100, thereby improving work efficiency.

[0151] The saving unit 304 saves the setting information acquired from the in-vehicle image capturing device S11 in the setting storage unit 47B. The setting information stores the shape of an object and the operation control to be performed on the excavator 100 when an object of that shape is detected, in association with each other. Furthermore, the setting information may include further conditions for performing the operation control. For example, the distance and direction to the object may be set as the further condition.

[0152] Furthermore, in the communication terminal 500, the CPU 502 executes an operation setting application 521A instead of the operation setting application 521 of the above-described embodiment. The operation setting application 521A receives a correspondence between the shape of an object, the distance to the object, etc., and the operation control that the shovel 100 will perform when an object of that shape is detected and conditions related to the distance, etc. are satisfied. The operation setting application 521A then performs control to transfer setting information including that correspondence to the controller 30A. This control is similar to that of the above-described embodiment, and therefore a description thereof will be omitted.

[0153] The action identification unit 305A identifies an action of the shovel 100 based on the determination result by the determination unit 303A and the setting information stored in the setting storage unit 47B. For example, when it is determined in the determination result that there is an association between an object and a striped pattern, the action identification unit 305A identifies an action according to the shape of the object based on the setting information, etc.

[0154] Fig. 11 is a diagram showing an example of the arrangement of objects with striped patterns prepared for controlling the operation of the shovel 100 according to this embodiment. In Fig. 11, it is assumed that the shovel 100 is performing an operation to demolish and remove an object 2111.

[0155] A sheet 2101 is provided around the object 2111. Both end portions 2102 and 2103 of the sheet 2101 are provided with a striped pattern.

[0156] The three-dimensional object detection unit 311 detects the sheet 2101 as a plurality of planar portions perpendicular to the ground.

[0157] A striped pattern is applied to the areas corresponding to both ends of the flat surface portion, and therefore the determining unit 303A determines that the detected flat surface portions are associated with a striped pattern.

[0158] 11, the attachment AT of the shovel 100 is set not to come into contact with the flat surface associated with the striped pattern. In other words, when the attachment AT approaches the vicinity of the sheet 2101, which is a flat surface, in accordance with the operation of the operator, the operation control unit 306 performs control to stop the attachment AT.

[0159] That is, in this embodiment, when the seat 2101 is provided, the shovel 100 can perform work without affecting anything outside the seat 2101. Therefore, improved safety can be achieved.

[0160] In this embodiment, the controller 30A controls the operation according to the shape of the object to which the striped pattern is applied, so that it is possible to cause the robot to perform an appropriate operation according to various conditions at the work site, thereby improving work efficiency and safety.

[0161] (Modification of the Second Embodiment) In the above-described embodiment, the case where the shape or type of an object is detected has been described. However, the above-described embodiment is not limited to the example where the three-dimensional measuring sensor S12 detects the shape of an object. For example, the shovel 100 may be provided with a sensor that detects reflective material, and operation control may be performed when the sensor detects reflective material.

[0162] For example, the three-dimensional measuring sensor S12 is a sensor that detects the distance and direction to an object using infrared rays. In this modification, the three-dimensional measuring sensor S12 has a reflective material detection mode that detects only reflective materials. When the reflective material detection mode is set, the three-dimensional measuring sensor S12 detects the direction and distance to the reflective material and outputs the detected information, including the direction and distance to the reflective material, to the controller 30A.

[0163] Examples of objects to which reflective material is applied include a worker's safety jacket, work machinery, a vehicle, and protective equipment. In this modified example, when an object to which stripes and reflective material are applied exists, operation control corresponding to the reflective material is performed. An object to which stripes and reflective material are applied may be an object to which reflective material is applied in the stripes, or an object to which stripes are applied may have reflective material applied. In other words, in this modified example, operation control is performed when a reflective material and a striped pattern are associated.

[0164] The three-dimensional object detection unit 311 detects the distance and direction to the reflective material based on the received reflective material detection information. The determination unit 303A then determines whether there is a correlation between the reflective material and the striped pattern based on the direction in which the reflective material is located and the direction in which the striped pattern is located. The area in which the striped pattern is captured in the image information can be converted to the direction in which the striped pattern is located at the actual work site using a well-known method based on the orientation in which the imaging device S6 is installed, etc.

[0165] The operation identification unit 305A identifies an operation of the shovel 100 based on the determination result by the determination unit 303A and the setting information stored in the setting storage unit 47B. The setting information according to this modification includes information related to operation control to be executed in accordance with the reflective material. Specifically, when it is determined in the determination result that there is an association between the reflective material and the striped pattern, the operation identification unit 305A identifies an operation in accordance with the reflective material based on the setting information, etc. The subsequent processing is the same as in the embodiment described above, and therefore a description thereof will be omitted.

[0166] In this modification, when a reflective material and a striped pattern are detected, the operation of the shovel 100 is controlled. This enables the shovel 100 to control its operation in accordance with the reflective material that has been applied to draw attention, thereby improving safety.

[0167] Third Embodiment In this embodiment, a case where the type and shape of an object are identified will be described.

[0168] 12 is a diagram showing an example of the configuration of the controller 30B and communication terminal 500 of the shovel 100 according to this embodiment. In this embodiment, the same reference numerals are assigned to the configurations of the above-described embodiment, and description thereof will be omitted.

[0169] The controller 30B according to this embodiment is provided with the object detection unit 301 of the first embodiment and the three-dimensional object detection unit 311 of the second embodiment.

[0170] Furthermore, the controller 30B is provided with a determination unit 303B and a motion identification unit 305B instead of the determination unit 303A and the motion identification unit 305A of the second embodiment.

[0171] When the object type is identified by the object detection unit 301 and the shape of the object is identified by the three-dimensional object detection unit 311, the determination unit 303B determines the correspondence between the type of object and the shape of the object based on the positional relationship between the direction in which the object whose type is identified exists and the direction in which the object whose shape is identified exists. By determining the correspondence, the determination unit 303B can recognize the direction and distance in which the object whose type is identified exists. Note that conversion from the area in which the object is captured in the image information to the direction in which the object or striped pattern is located in the actual work site can be performed using a well-known method based on the orientation in which the imaging device S6 is installed, etc.

[0172] The determination unit 303B then determines whether or not the object and the striped pattern are associated with each other based on the positional relationship between the direction in which the object, whose type and shape have been identified, exists and the direction in which the striped pattern detected by the striped pattern detection unit 302 exists. This determination is the same as in the above-described embodiment, and therefore a description thereof will be omitted.

[0173] The saving unit 304 saves the setting information acquired from the in-vehicle image capturing device S11 in the setting storage unit 47B. The setting information stores the type of object and the operation control to be performed on the excavator 100 when the object of that type is detected, in association with each other. Furthermore, the setting information may include information regarding the location of the object as a condition for starting the operation control.

[0174] In the communication terminal 500, the CPU 502 executes an operation setting application 521B instead of the operation setting application 521A of the above-described embodiment.

[0175] Fig. 13 is a diagram showing screen transitions of communication terminal 500 according to this embodiment. CPU 502 of communication terminal 500 executes operation setting application 521B to display screen 2301 shown in Fig. 13. Screen 2301 shows height range specification fields 2311A and 2311B, distance range specification fields 2312A and 2312B, direction range specification fields 2313A and 2313B, detection target selection field 2314, control selection field 2315, and OK button 1715.

[0176] The height range specification fields 2311A and 2311B, distance range specification fields 2312A and 2312B, and direction range specification fields 2313A and 2313B are fields for setting the position of an object as a condition for performing motion control. Motion control is started when the following objects are present within the ranges set in these specification fields. Note that these specification fields 2311A, 2311B, 2312A, 2312B, 2313A, and 2313B do not need to be set if they are not necessary for setting the conditions for starting motion control.

[0177] The detection target selection field 2314 is a field for selecting the type of object to be detected.

[0178] The control selection field 2315 is a field for selecting an operation to be performed by the excavator 100 when an object of the selected type is detected and when a condition regarding the position of the object is satisfied.

[0179] An OK button 2316 is a button to be pressed when selection and setting in the specification fields 2311A, 2311B, 2312A, 2312B, 2313A, 2313B, detection target selection field 2314, and control selection field 2315 are completed.

[0180] When the CPU 502 of the communication terminal 500 receives the selection of the OK button 2316, it displays a screen 2302. The screen 2302 displays specification fields 2311A, 2311B, 2312A, 2312B, 2313A, and 2313B, a detection target selection field 2314, and a two-dimensional code 2321 for acquiring setting information including the content selected or set in the control selection field 2315.

[0181] When the in-vehicle image capturing device S11 captures the two-dimensional code 2321, the storage unit 304 of the excavator 100 extracts the setting information contained in the two-dimensional code 2321 and stores the setting information in the setting memory unit 47B.

[0182] The motion identification unit 305B identifies the motion of the shovel 100 based on the determination result by the determination unit 303B and the setting information stored in the setting storage unit 47B. For example, when the determination result indicates that an object is associated with a striped pattern, the motion identification unit 305B identifies a motion according to the type and position (one or more of distance and direction) of the object based on the setting information, etc. The subsequent processing is the same as in the above-described embodiment, and therefore a description thereof will be omitted. This makes it possible to perform control such as decelerating and stopping the shovel 100 when a cone bar is present within 5 m of the shovel 100 and within 50 cm of the shovel 100.

[0183] This embodiment is not limited to the case where an operator is on board the cabin 10 of the shovel 100, but may also be applied to the case where the shovel 100 is operated from a remote control room. Furthermore, the shovel 100 may perform autonomous control. In this case, a movement trajectory of the attachment AT or the lower traveling body 1 of the shovel 100 may be generated taking into consideration the type, direction, and orientation of the object to which the striped pattern is applied, and operation control may be performed according to the movement trajectory.

[0184] In this embodiment, the controller 30B controls the operation according to the type and position (one or more of distance and direction) of the object to which the striped pattern is applied, so that it is possible to cause the robot to perform an appropriate operation according to various conditions at the work site, thereby realizing improved work efficiency and improved safety.

[0185] (Variations) In the above-described embodiment, a case has been described in which one or more of the type of object to which a striped pattern has been applied and the shape of the object are identified. However, the above-described embodiment is not limited to a method of identifying one or more of the type of object to which a striped pattern has been applied and the shape of the object. As a variation, the controller may control the shovel 100 to perform a predetermined operation when a striped pattern in which two different color regions are alternately displayed is detected from information acquired by the imaging device S6. In other words, because stripes are patterns for calling attention to the surrounding area, when the shovel 100 detects the striped pattern, it is possible to improve safety by performing an operation control determined in accordance with the striped pattern.

[0186] <Operation> In the above-described embodiment, when the shovel 100 detects the striped pattern, it performs operation control determined according to the striped pattern. The striped pattern is a pattern for calling attention to the surroundings, and is also a pattern that is easy to obtain using tape or the like. Therefore, it is easy to impart the obtained striped pattern to an object or the like. Furthermore, striped patterns are often imparted in advance to tools and the like. Furthermore, the detection of striped patterns from image information can be performed with high accuracy.

[0187] On the other hand, conventional object detection can sometimes result in false detection. Therefore, in the above-described embodiment, when an object is detected, if there is a correlation between the object (including a reflective material) and the striped pattern, control is performed based on the object. By combining the detection of the striped pattern and the detection of the object, false detection can be suppressed and the accuracy of operation control can be improved. Therefore, safety at the work site can be improved.

[0188] In the above-described embodiment and modified examples, the case where the shovel 100 is used as the work machine has been described. However, the embodiment and modified examples do not limit the work machine to the shovel 100, and the work machine may be road machinery including an asphalt finisher or the like. Furthermore, the work machine may be a forklift, crawler crane, or overhead crane that can be moved according to the operation of an operator, or may be a fixed power crane.

[0189] While the embodiments of the work machine according to the present invention have been described above, the present invention is not limited to the above-described embodiments. Various changes, modifications, substitutions, additions, deletions, and combinations are possible within the scope of the claims. Naturally, these also fall within the technical scope of the present invention.

[0190] This application claims priority based on Japanese Patent Application No. 2023-222341, filed on December 28, 2023, the entire contents of which are incorporated herein by reference.

[0191] DESCRIPTION OF SYMBOLS 100 Excavator 1 Lower traveling body 2 Swing mechanism 3 Upper rotating body 4 Boom 5 Arm 6 Bucket 7 Boom cylinder 8 Arm cylinder 9 Bucket cylinder 17 Control valve unit S1 Boom angle sensor S2 Arm angle sensor S3 Bucket angle sensor S5 Swing angle sensor S6 Imaging device S11 In-vehicle imaging device S12 Three-dimensional measurement sensor PS Positioning device T1 Communication device 30, 30A, 30B Controller 301 Object detection unit 302 Stripe pattern detection unit 303, 303A, 303B Determination unit 304 Storage unit 305, 305A, 305B Action identification unit 306 Action control unit 307 Display control unit 311 Three-dimensional object detection unit 47 Storage medium 47A, 47C Learned model 47B Setting storage unit 500 Communication terminal 501 Imaging device 502 CPU 521, 521A, 521B Operation setting application 503 Display device

Claims

1. A working machine comprising: a space recognition device for detecting the surroundings of the working machine; and a control device configured to perform a predetermined operation when a striped pattern in which different two-color color regions are alternately represented is detected from the information acquired by the space recognition device.

2. The working machine according to claim 1, wherein the space recognition device includes an imaging device capable of acquiring image information representing the surroundings of the working machine, and the control device is configured to perform a predetermined operation according to the type of the object when it is determined that there is a relationship between the object and the striped pattern based on the positional relationship in the image information between the object and the striped pattern when the type of the object existing around the working machine is specified from the image information.

3. The working machine according to claim 2, wherein the types of the objects that can be specified by the control device are any one or more of a load cone, a fence, a guard rail, a pole, a face, a vehicle, and a working machine.

4. The working machine according to claim 2, wherein the control device determines that there is a relationship between the object and the striped pattern when the region where the striped pattern exists in the image information is included in the region where the object exists, when the distance between the region where the object exists and the region where the striped pattern exists in the image information is closer than a predetermined threshold value, or when the region where the striped pattern exists surrounds the object.

5. The working machine according to claim 2, wherein the control device is configured to perform the predetermined operation when the working machine is closer than a predetermined distance to the object determined to be related to the striped pattern.

6. The working machine according to claim 2, wherein the control device is further configured to perform the predetermined operation when the positional relationship between the object determined to be related to the striped pattern and a person detected from the image information around the working machine satisfies a predetermined condition.

7. The working machine according to claim 2, wherein the control device is configured to perform the predetermined operation when the movement of the object determined to be related to the striped pattern is detected.

8. The spatial recognition device further includes a detection device capable of acquiring detection information representing the distance and direction to the object existing around the work machine. When the control device identifies the type of the object existing around the work machine from the image information and identifies the shape of the object based on the detection information, it identifies the correspondence relationship between the object whose type is identified and the object whose shape is identified, and determines whether the object is related to the stripe pattern based on the positional relationship between the direction in which the object whose type and shape are identified exists and the direction in which the detected stripe pattern exists. When it is determined that the object is related to the stripe pattern, it is configured to perform the predetermined operation according to the type of the object and any one or more of the distance and direction of the object. The work machine according to claim 2.

9. The spatial recognition device includes a detection device capable of acquiring detection information representing the distance and direction to an object existing around the work machine. When the control device identifies the shape of the object based on the detection information, if it is determined that the object is related to the stripe pattern based on the direction in which the object exists and the direction in which the stripe pattern is detected, it is configured to perform a predetermined operation according to the identified shape of the object. The work machine according to claim 1.

10. The shape of the object that can be identified by the control device is any one or more of a planar portion perpendicular to the ground, a stepped shape provided on the ground, a columnar shape provided perpendicular to the ground, and a rod shape. The work machine according to claim 9.

11. The control device determines that the object is related to the stripe pattern when the direction in which the stripe pattern exists is included in the range where the object exists, or when the direction in which the stripe pattern exists surrounds at least a part of the object. The work machine according to claim 9.

12. The spatial recognition device includes a detection device capable of acquiring reflection material detection information representing the distance and direction to a reflection material existing around the work machine. When the control device detects the reflection material based on the reflection material detection information, if it is determined that there is a relationship between the reflection material and the stripe pattern based on the direction in which the reflection material exists and the direction in which the stripe pattern is detected, the control device is configured to perform a predetermined operation according to the specified reflection material. The work machine according to claim 1.

13. The stripe pattern is characterized in that two-color color regions are alternately represented in an oblique direction. The work machine according to claim 1.

14. The predetermined operation is determined based on information input from an operating device or an external terminal. The work machine according to claim 1.

15. The control device is configured to display information indicating the presence of the stripe pattern on a display device. The work machine according to claim 1.

16. The control device is configured to superimpose information indicating a region where the stripe pattern exists on screen information based on image information captured by an imaging device included in the spatial recognition device and display the result on the display device. The work machine according to claim 15.

Images