Excavator

Abstract

To provide a technique capable of improving determination accuracy in various environmental conditions when performing determination on surrounding objects of a shovel based on environmental information surrounding the shovel.SOLUTION: A shovel 100 in accordance with an embodiment of the present invention includes: an imaging device 70 acquiring captured images as environment information surrounding the shovel 100; and a determination part 344 performing determination on surrounding objects of the shovel 100 based on captured images acquired with the imaging device 70 using a learned model LM which has been through mechanical learning. The learned model ML is updated to an additionally learned model which is subjected to additional learning with a management device 200 based on teacher information generated from the surrounding captured images acquired by at least one of the shovel 100 or other shovels, and the determination part 344 performs determination based on the captured images acquired with the imaging device 70 by using the updated leaned model when the learned model is updated.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a shovel. [Background technology]

[0002] For example, a technique is known for making judgments about objects around the shovel (e.g., determining the presence or absence of objects, their type, etc.) based on environmental information that represents the situation around the shovel (e.g., captured images of the area around the shovel and reflected wave data of detection waves transmitted around the shovel).

[0003] For example, in Patent Document 1, image processing techniques such as optical flow and pattern matching are used to detect surrounding objects based on captured images of the periphery of the shovel (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6290497 Summary of the Invention [Problem to be solved by the invention]

[0005] However, while excavators can be used in a variety of environments, in Patent Document 1 and other documents, judgments are made based on pre-defined judgment criteria, so there is a possibility that appropriate judgment accuracy cannot be ensured depending on the environmental conditions.

[0006] In view of the above problem, a technology is provided that can improve the accuracy of determination under various environmental conditions when determining objects around a shovel based on environmental information around the shovel. [Means for solving the problem]

[0007] In one embodiment of the present disclosure, an environmental information acquisition unit that acquires environmental information around the excavator; a determination unit that uses a trained model that has undergone machine learning to make a determination regarding an object around the shovel based on the environmental information acquired by the environmental information acquisition unit, The trained model is The environmental information acquisition unit Obtained environmental information Environmental information that caused the trained model to make an erroneous judgment when the judgment was made, selected from Based on the teacher information generated from the model, additional learning is performed by an external device that can communicate with the excavator, and the model is updated to an additionally trained model. When the trained model is updated, the determination unit performs the determination using the updated trained model based on environmental information acquired by the environmental information acquisition unit. Shovels are provided. [Effects of the Invention]

[0008] According to the above-described embodiment, it is possible to improve the accuracy of determination under various environmental conditions when determining objects around the shovel based on environmental information around the shovel. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the configuration of an excavator support system. [Figure 2] FIG. [Figure 3] FIG. 2 is a functional block diagram showing an example of a functional configuration of the excavator support system. [Figure 4A] FIG. 10 is a conceptual diagram illustrating an example of a determination process performed by a determination unit. [Figure 4B] FIG. 10 is a conceptual diagram illustrating an example of a determination process performed by a determination unit. [Figure 5] 10A and 10B are diagrams illustrating specific examples of results of a determination process performed by a determination unit. [Figure 6] FIG. 2 is a sequence diagram illustrating an example of an operation of the excavator support system. [Figure 7] FIG. 10 is a functional block diagram showing another example of the functional configuration of the excavator support system. [Figure 8]FIG. 10 is a functional block diagram showing yet another example of the functional configuration of the excavator support system. [Figure 9] FIG. 10 is a conceptual diagram illustrating another example of the determination process by the determination unit. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment will be described with reference to the drawings.

[0011] [Outline of the excavator support system] First, with reference to FIG. 1, the excavator support system SYS according to this embodiment will be described.

[0012] FIG. 1 is a schematic diagram showing an example of the configuration of the excavator support system SYS.

[0013] The shovel support system SYS includes a plurality of shovels 100 and a management device 200, and supports determinations regarding objects around the shovel 100, which are executed by each shovel 100. Determinations regarding objects around the shovel 100 include, for example, determination of the presence or absence of objects around the shovel 100 (i.e., determination regarding detection of objects around the shovel 100) and determination of the type of object around the shovel 100 (i.e., determination regarding the classification of objects detected around the shovel 100). The following description will be given on the assumption that each of the plurality of shovels 100 has the same configuration with respect to the shovel support system SYS.

[0014] <Outline of the excavator> The excavator 100 includes a lower running body 1, an upper rotating body 3 mounted on the lower running body 1 so as to be freely rotatable via a rotating mechanism 2, a boom 4, an arm 5, and a bucket 6 that constitute attachments, and a cabin 10.

[0015] The lower traveling body 1 includes a pair of left and right crawlers 1C, specifically a left crawler 1CL and a right crawler 1CR. The left crawler 1CL and the right crawler 1CR are hydraulically driven by traveling hydraulic motors 2M (2ML, 2MR), respectively, to allow the excavator 100 to travel.

[0016] The upper rotating body 3 is driven by a swing hydraulic motor 2A to swing relative to the lower traveling body 1. The upper rotating body 3 may also be electrically driven by an electric motor instead of being hydraulically driven by the swing hydraulic motor 2A. Hereinafter, for convenience, the side of the upper rotating body 3 to which the attachment AT is attached will be referred to as the front, and the side to which the counterweight is attached will be referred to as the rear.

[0017] A boom 4 is pivotally attached to the front center of the upper rotating body 3 so as to be able to tilt up and down, an arm 5 is pivotally attached to the tip of the boom 4 so as to be able to rotate up and down, and a bucket 6 is pivotally attached to the tip of the arm 5 so as to be able to rotate up and down. The boom 4, arm 5, and bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, which serve as hydraulic actuators, respectively.

[0018] The cabin 10 is a cab in which an operator sits, and is mounted on the front left side of the upper rotating body 3.

[0019] The shovel 100 is also equipped with a communication device 90. The shovel 100 is communicably connected to the management device 200 via a predetermined communication network (hereinafter simply referred to as a "communication network") that may include, for example, a mobile phone network terminated at a base station, a satellite communication network using communication satellites in the sky, the Internet network, etc. This allows the shovel 100 to acquire various types of information from the management device 200 and transmit various types of information to the management device 200. Details will be described later.

[0020] The excavator 100 operates operating elements (driven elements) such as the lower traveling body 1, upper rotating body 3, boom 4, arm 5, and bucket 6 in response to an operation of the operating device 26 by an operator seated in the cabin 10.

[0021] Furthermore, instead of or in addition to being operated by an operator in the cabin 10, the shovel 100 may be remotely operated by an operator of a predetermined external device (e.g., the management device 200). In this case, the shovel 100 transmits image information (captured images) output by, for example, an imaging device 70 (described later) to the management device 200. This allows the operator to remotely operate the shovel 100 while checking the image information displayed on a display device (e.g., a display device 230 (described later)) provided in the management device 200. The shovel 100 may then operate driven elements such as the lower traveling structure 1, the upper rotating structure 3, the boom 4, the arm 5, and the bucket 6 in response to a remote operation signal received from the management device 200 indicating the content of the remote operation. The following description will be given on the assumption that the operator's operation includes at least one of the operator's operation of the operating device 26 and the operator's remote operation of the management device 200.

[0022] <Overview of the management device> The management device 200 (an example of an external device) is installed at a geographically separate location outside the shovels 100 and manages a plurality of shovels 100. The management device 200 is installed, for example, in a management center or the like located outside the work site where the shovels 100 work, and is a server device (terminal device) mainly composed of one or more server computers or the like. In this case, the server device may be an in-house server operated by the business operator that operates the shovel support system SYS or an associated business operator related to that business operator, or it may be a so-called cloud server. The management device 200 may also be a fixed or portable computer terminal located in a management office or the like within the work site of the shovel 100.

[0023] The management device 200 is communicably connected to each of the multiple shovels 100 via a communication network. This allows the management device 200 to transmit various types of information to the shovels 100 and receive various types of information from the shovels 100. Details will be described later.

[0024] Furthermore, the management device 200 may be configured to be able to remotely control the shovel 100. Specifically, the management device 200 may display image information from the imaging device 70 transmitted from the shovel 100 on a display device (e.g., display device 230), and the remote operator may remotely control the shovel 100 while checking this image information. In this case, the remote operator may use an operation device for remote operation provided on the management device 200 (e.g., a general-purpose operation device such as a touch panel, touchpad, or joystick, or a dedicated operation device simulating the operation device 26). The management device 200 transmits a remote operation signal including the content of the remote operation to the shovel 100 via a communication network. As a result, the shovel 100 can operate in response to the remote operation signal from the management device 200, for example, under the control of the controller 30 described below, and the management device 200 can support the remote operation of the shovel 100.

[0025] [Example of excavator support system configuration] The specific configuration of the shovel support system SYS (the shovel 100, the management device 200) will be described with reference to FIGS. 2 to 5 in addition to FIG.

[0026] Fig. 2 is a top view of the shovel 100. Fig. 3 is a configuration diagram showing an example of the configuration of the shovel support system SYS according to this embodiment. Fig. 4 (Figs. 4A and 4B) is a conceptual diagram showing an example of the determination processing by the determination unit 344, which will be described later. Fig. 5 is a diagram showing a specific example of the result of the determination processing by the determination unit 344.

[0027] As described above, each of the multiple shovels 100 has the same configuration with respect to the shovel support system SYS, and therefore, in FIG. 3, only the detailed configuration of one shovel 100 is shown.

[0028] <Excavator configuration> As described above, the excavator 100 includes hydraulic actuators such as the traveling hydraulic motor 2M (2ML, 2MR), the swing hydraulic motor 2A, the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9 as components related to the hydraulic system. The excavator 100 also includes an engine 11 and an operating device 26 as components related to the hydraulic system. The excavator 100 also includes a controller 30, a recording device 32, a determination device 34, a display device 40, an imaging device 70, an orientation detection device 85, a communication device 90, a boom angle sensor S1, an arm angle sensor S2, a bucket angle sensor S3, a machine body inclination sensor S4, and a swing state sensor S5 as components related to the control system.

[0029] The engine 11 is a drive source for the excavator 100 (its hydraulic system), and is mounted, for example, on the rear of the upper rotating body 3. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 operates to maintain a predetermined rotation speed (set rotation speed), for example, under the control of the controller 30 or the like. The rotating shaft of the engine 11 is connected to the rotating shafts of a main pump that supplies hydraulic oil to the hydraulic actuators and a pilot pump that supplies hydraulic oil to hydraulic devices of the operating system, such as the operating device 26, and the power of the engine 11 is transmitted to the main pump and the pilot pump.

[0030] The operating device 26 is positioned within reach of an operator seated in the driver's seat inside the cabin 10, and the operator inputs operations to operate the various operating elements (lower running body 1, upper rotating body 3, boom 4, arm 5, bucket 6, etc.), in other words, the hydraulic actuators that drive the various operating elements.

[0031] The operating device 26 is, for example, a hydraulic pilot type. In this case, the operating device 26 receives hydraulic oil from a pilot pump to generate a predetermined pilot pressure (pilot pressure corresponding to the operation content). The operating device 26 then applies the pilot pressure to a pilot port of a corresponding control valve in a control valve that drives and controls the hydraulic actuator. As a result, the operation content of the operating device 26 (e.g., operation direction and operation amount) is reflected in the operation of the control valve, and the hydraulic actuator realizes the operation of various operating elements (driven elements) in accordance with the operation content of the operating device 26.

[0032] Furthermore, the operating device 26 may be an electrical type that outputs, for example, an electrical signal (hereinafter, "operation signal") corresponding to the operation content. In this case, the electrical signal output from the operating device 26 may be input, for example, to the controller 30, which may then output a control command corresponding to the operation signal, i.e., a control command corresponding to the operation content of the operating device 26, to a predetermined hydraulic control valve (hereinafter, "operation control valve"). The operation control valve may then use hydraulic oil supplied from a pilot pump or a main pump to output a pilot pressure corresponding to the control command from the controller 30, and apply the pilot pressure to a pilot port of a corresponding control valve in a control valve. In this way, the operation content of the operating device 26 is reflected in the operation of the control valve, and the hydraulic actuator realizes the operation of various operating elements according to the operation content of the operating device 26.

[0033] Furthermore, when the shovel 100 is remotely operated, for example, the controller 30 may use the above-mentioned operation control valve to realize the remote operation of the shovel 100. Specifically, the controller 30 may output a control command corresponding to the content of the remote operation specified in a remote operation signal received by the communication device 90 to the operation control valve. Then, the operation control valve may output a pilot pressure corresponding to the control command from the controller 30 using hydraulic oil supplied from a pilot pump or a main pump, and cause the pilot pressure to act on a pilot port of the corresponding control valve in the control valve. In this way, the content of the remote operation is reflected in the operation of the control valve, and the hydraulic actuator realizes the operation of various operating elements (driven elements) in accordance with the content of the remote operation.

[0034] The controller 30 is mounted, for example, inside the cabin 10, and controls the driving of the excavator 100. The functions of the controller 30 may be realized by any hardware or any combination of hardware and software. For example, the controller 30 is configured mainly with a computer including a CPU (Central Processing Unit), a memory device (main storage device) such as RAM (Random Access Memory), a non-volatile auxiliary storage device such as ROM (Read Only Memory), and various input / output interface devices.

[0035] For example, the controller 30 receives output signals from various sensors such as the orientation detection device 85, the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the machine body inclination sensor S4, and the swing state sensor S5, and grasps various states of the shovel 100 (for example, the direction and posture of the upper swing body 3).The controller 30 then performs various controls of the shovel 100 according to the various grasped states.

[0036] Furthermore, for example, when the determination device 34 detects a monitored object (e.g., a person, truck, other construction machinery, utility pole, suspended load, pylon, building, etc.) within a predetermined monitoring area around the shovel 100 (e.g., a work area within 5 meters of the shovel 100), the controller 30 performs control (hereinafter referred to as "contact avoidance control") to avoid contact between the shovel 100 and the monitored object. The controller 30 includes, for example, a notification unit 302 and an operation control unit 304 as functional units related to the contact avoidance control, which are realized by executing on the CPU one or more programs installed in an auxiliary storage device or the like.

[0037] The recording device 32 records images captured by the imaging device 70 at a predetermined timing. The recording device 32 may be realized by any hardware or any combination of hardware and software. For example, the recording device 32 may be configured mainly with a computer similar to the controller 30. The recording device 32 includes, for example, a recording control unit 322 as a functional unit realized by executing one or more programs installed in an auxiliary storage device or the like on a CPU. The recording device 32 also includes, for example, a storage unit 324 as a storage area defined in an internal memory of the auxiliary storage device or the like.

[0038] The determination device 34 determines objects around the shovel 100 (e.g., object detection determination, object classification determination, etc.) based on images captured by the imaging device 70. The determination device 34 may be realized by any hardware or any combination of hardware and software. For example, the determination device 34 may be configured mainly with a computer having a configuration similar to that of the controller 30, i.e., a CPU, a memory device, an auxiliary storage device, an interface device, etc., as well as an image processing device that performs high-speed calculations through parallel processing in conjunction with processing by the CPU. The control device 210 of the management device 200, described below, may have a similar configuration. The image processing device may include a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc. The determination device 34 includes a display control unit 342 and a determination unit 344 as functional units realized by executing one or more programs installed in an auxiliary storage device or the like on the CPU. The determination device 34 also includes a storage unit 346 as a storage area defined in an internal memory such as an auxiliary storage device.

[0039] It should be noted that the controller 30, the recording device 32, and the determination device 34 may be partly or entirely integrated into one.

[0040] The display device 40 is provided in a location that is easily visible to an operator seated in the driver's seat inside the cabin 10, and displays various information images. The display device 40 is, for example, a liquid crystal display or an organic EL (Electroluminescence) display. For example, the display device 40 displays an image showing the surroundings of the shovel 100 based on an image captured by the imaging device 70 under the control of the determination device 34 (display control unit 342). Specifically, the display device 40 may display the image captured by the imaging device 70. The display device 40 may also display a converted image generated by the determination device 34 (display control unit 342) in which a predetermined conversion process (for example, viewpoint conversion process) has been performed on the image captured by the imaging device 70. The converted image may be, for example, a viewpoint conversion image that combines an overhead image viewed from directly above the shovel 100 and a horizontal image viewed horizontally from the shovel 100. Furthermore, the viewpoint conversion image may be a composite image obtained by converting the images captured by the front camera 70F, rear camera 70B, left camera 70L, and right camera 70R (described later) into a viewpoint conversion image using an overhead image and a horizontal image, and then combining them.

[0041] The imaging device 70 (an example of an environmental information acquisition unit) captures images of the surroundings of the shovel 100 and outputs the captured images (an example of environmental information). The imaging device 70 includes a front camera 70F, a rear camera 70B, a left camera 70L, and a right camera 70R. The images captured by the imaging devices 70 (the front camera 70F, the rear camera 70B, the left camera 70L, and the right camera 70R) are taken into the determination device 34.

[0042] The front camera 70F is attached to, for example, the front end of the upper surface of the cabin 10, and captures an image of the situation in front of the upper rotating body 3.

[0043] The rear camera 70B is attached to, for example, the rear end of the upper surface of the upper rotating body 3, and captures an image of the situation behind the upper rotating body 3.

[0044] The left camera 70L is attached to, for example, the left end of the top surface of the upper rotating body 3, and captures an image of the situation to the left of the upper rotating body 3.

[0045] The right camera 70R is attached to, for example, the right end of the top surface of the upper rotating body 3, and captures an image of the situation to the right of the upper rotating body 3.

[0046] The orientation detection device 85 is configured to detect information regarding the relative relationship between the orientation of the upper rotating body 3 and the orientation of the lower rotating body 1 (hereinafter, "orientation-related information"). For example, the orientation detection device 85 may be configured by a combination of a geomagnetic sensor attached to the lower rotating body 1 and a geomagnetic sensor attached to the upper rotating body 3. Alternatively, the orientation detection device 85 may be configured by a combination of a GNSS (Global Navigation Satellite System) receiver attached to the lower rotating body 1 and a GNSS receiver attached to the upper rotating body 3. When the upper rotating body 3 is configured to be driven by an electric motor, the orientation detection device 85 may be configured by a resolver attached to the electric motor. Alternatively, the orientation detection device 85 may be disposed in, for example, a center joint provided in association with the rotation mechanism 2 that realizes relative rotation between the lower rotating body 1 and the upper rotating body 3. The detection information by the orientation detection device 85 is input to the controller 30.

[0047] The communication device 90 is any device that connects to a communication network and communicates with external devices such as the management device 200. The communication device 90 may be, for example, a mobile communication module that complies with a predetermined mobile communication standard such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation).

[0048] The boom angle sensor S1 is attached to the boom 4 and detects the elevation / depression angle (hereinafter referred to as the "boom angle") θ1 of the boom 4 relative to the upper rotating body 3. The boom angle θ1 is, for example, the angle of ascent from the state in which the boom 4 is lowered to its lowest position. In this case, the boom angle θ1 is greatest when the boom 4 is raised to its highest position. The boom angle sensor S1 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), etc., and the same may apply to the arm angle sensor S2, bucket angle sensor S3, and machine body tilt sensor S4 below. The boom angle sensor S1 may also be a stroke sensor attached to the boom cylinder 7, and the same may apply to the arm angle sensor S2 and bucket angle sensor S3 below. A detection signal corresponding to the boom angle θ1 detected by the boom angle sensor S1 is input to the controller 30.

[0049] The arm angle sensor S2 is attached to the arm 5 and detects the rotation angle θ2 of the arm 5 relative to the boom 4 (hereinafter referred to as the "arm angle"). The arm angle θ2 is, for example, the opening angle of the arm 5 from its most closed state. In this case, the arm angle θ2 is maximum when the arm 5 is most open. A detection signal corresponding to the arm angle θ2 detected by the arm angle sensor S2 is input to the controller 30.

[0050] The bucket angle sensor S3 is attached to the bucket 6 and detects the rotation angle θ3 of the bucket 6 relative to the arm 5 (hereinafter referred to as the "bucket angle"). The bucket angle θ3 is the opening angle of the bucket 6 from its fully closed state. In this case, the bucket angle θ3 is maximum when the bucket 6 is fully opened. A detection signal corresponding to the bucket angle θ3 detected by the bucket angle sensor S3 is input to the controller 30.

[0051] The machine body tilt sensor S4 detects the tilt state of the machine body (e.g., the upper rotating body 3) with respect to a predetermined plane (e.g., a horizontal plane). The machine body tilt sensor S4 is attached, for example, to the upper rotating body 3, and detects the tilt angles of the excavator 100 (i.e., the upper rotating body 3) about two axes in the fore-aft and lateral directions (hereinafter referred to as the "fore-aft tilt angle" and the "lateral tilt angle"). The detection signals corresponding to the tilt angles (fore-aft tilt angle and lateral tilt angle) by the machine body tilt sensor S4 are input to the controller 30.

[0052] The turning state sensor S5 is attached to the upper rotating body 3 and outputs detection information related to the turning state of the upper rotating body 3. The turning state sensor S5 detects, for example, the turning angular acceleration, turning angular velocity, turning angle, etc. of the upper rotating body 3. The turning state sensor S5 may include, for example, a gyro sensor, a resolver, a rotary encoder, etc.

[0053] If the vehicle tilt sensor S4 includes a gyro sensor, a six-axis sensor, an IMU, or the like that can detect angular velocities around three axes, the rotation state (e.g., rotation angular acceleration) of the upper rotating body 3 may be detected based on the detection signal of the vehicle tilt sensor S4. In this case, the rotation state sensor S5 may be omitted.

[0054] When the determination device 34 (determination unit 344) detects a monitored object within the monitoring area around the shovel 100, the notification unit 302 notifies the operator or the like of that fact. As a result, when a monitored object intrudes into a relatively close area around the shovel 100, the operator or the like can recognize the intrusion even if the object is located in a blind spot as seen from the cabin 10, and can ensure safety by, for example, ceasing operation of the operating device 26.

[0055] For example, the alarm unit 302 outputs a control signal to a sound output device (e.g., a speaker, buzzer, etc.) mounted inside the cabin 10 to notify an operator, etc. that a monitored object has been detected within a monitoring area close to the shovel 100.

[0056] Furthermore, for example, as will be described later, a notification may be given via the display device 40 to indicate that the determination device 34 has detected an object to be monitored within the monitoring area around the shovel 100.

[0057] Furthermore, for example, when the shovel 100 is remotely operated, the notification unit 302 may transmit to the management device 200 a signal (hereinafter referred to as an "alert signal") indicating that a monitoring target has been detected within a monitoring area around the shovel 100. In this way, the management device 200 can control a display device (for example, the display device 230) or a sound output device provided in the management device 200 in response to the alert signal received from the shovel 100, and notify the remote operator of that fact.

[0058] The operation control unit 304 (an example of a control unit) restricts the operation of the shovel 100 when the determination device 34 (determination unit 344) detects a monitored object within a monitoring area around the shovel 100. As a result, when a monitored object enters a monitoring area close to the shovel 100, the operation of the shovel 100 is restricted, thereby reducing the possibility of contact between the shovel 100 and the monitored object. In this case, restricting the operation of the shovel 100 may include slowing down the operation of various operation elements (driven elements) of the shovel 100 as an output in response to the operation content (operation amount) of the operator or the like on the operation device 26. Furthermore, restricting the operation of the shovel 100 may include stopping the operation of the operation elements (driven elements) of the shovel 100 regardless of the operation content of the operation device 26. Furthermore, the operation elements (driven elements) of the shovel 100 that are subject to the restriction of the operation of the shovel 100 may be all operation elements that can be operated by the operation device 26, or may be some operation elements necessary to avoid contact between the shovel 100 and the monitored object.

[0059] For example, the operation control unit 304 may output a control signal to a pressure-reducing valve provided in a pilot line on the secondary side of the hydraulic pilot-type operating device 26 to reduce the pilot pressure corresponding to the operation of the operating device 26 by an operator or the like. Alternatively, the operation control unit 304 may control a solenoid valve (operation control valve) by outputting a control signal limited to an operation amount smaller than the operation amount (operation amount) corresponding to the operation signal input from the electric operating device 26, thereby reducing the pilot pressure acting on the control valve from the solenoid valve. Alternatively, the operation control unit 304 may output a control signal limited to an operation amount smaller than the remote operation amount (operation amount) specified by the remote operation signal to the operation control valve, thereby reducing the pilot pressure acting on the control valve from the operation control valve. This reduces the pilot pressure acting on the control valve that controls the hydraulic oil supplied to the hydraulic actuator, corresponding to the operation of the operating device 26 or the remote operation, thereby restricting the operation of various operating elements (driven elements).

[0060] The recording control unit 322 (an example of a recording unit) records images captured by the imaging device 70 (front camera 70F, rear camera 70B, left camera 70L, and right camera 70R) in the storage unit 324 at predetermined times (hereinafter, "recording timings"). This allows images captured by the imaging device 70 to be recorded in the storage unit 324 at predetermined times, even though the storage unit 324 has a limited capacity. Furthermore, as described below, this reduces the transmission capacity when the captured images in the storage unit 324 are transmitted to the management device 200, thereby reducing communication costs. Specifically, for example, when the recording timings arrive, the recording control unit 322 acquires the captured image corresponding to the recording timing from among the captured images stored in a ring buffer defined in RAM or the like, including past images, and records the image in the storage unit 324.

[0061] The recording timing may be, for example, a periodic timing that is specified in advance. Furthermore, the recording timing may be when the shovel 100 is in a state that is likely to cause an erroneous determination when the determination device 34 (determination unit 344) determines an object around the shovel 100 based on the image captured by the imaging device 70. Specifically, the recording timing may be when the shovel 100 is traveling or swinging. Furthermore, the recording timing may be when the determination unit 344 determines that an object has been detected in the monitoring area around the shovel 100. Furthermore, the recording timing may be started when the controller is turned on, when the gate lock lever is released, or when the operation lever is turned on. The same applies to the shovel support system SYS (shovel 100) in FIGS. 7 and 8 described below.

[0062] 3, the determination result of the determination unit 344 is input to the recording device 32 (recording control unit 322), but if the recording timing is specified independently of the determination result of the determination unit 344, the determination result of the determination unit 344 does not need to be input to the recording device 32. The same applies to the case of FIG. 8, which will be described later.

[0063] As described above, captured images IM1 are recorded in the memory unit 324 under the control of the recording control unit 322 during the period from the completion of the initial processing after startup of the shovel 100 until the shutdown of the shovel 100. One or more captured images IM1 recorded in the memory unit 324 are transmitted to the management device 200 via the communication device 90 (an example of an environmental information transmission unit) at a predetermined timing (hereinafter referred to as "image transmission timing").

[0064] The image transmission timing may be, for example, when a stop operation of the shovel 100 is performed (for example, a key switch is turned OFF). The transmission timing may also be when the free space in the storage unit 324 falls below a predetermined threshold. This is because the total size of the captured images IM1 recorded in the storage unit 324 may become relatively large between the start-up and shutdown of the shovel 100. The image transmission timing may also be, for example, after the completion of the initial processing after the start-up of the shovel 100. In this case, the storage unit 324 may be a storage area defined in a non-volatile internal memory, and the captured images IM1 recorded between the previous start-up and shutdown of the shovel 100 may be transmitted to the management device 200. The same applies to the shovel support system SYS (shovel 100) in FIGS. 7 and 8 described below.

[0065] Note that the captured image IM1 may be transmitted to the management device 200 via the communication device 90 each time it is recorded in the storage unit 324.

[0066] As described above, the display control unit 342 causes the display device 40 to display an image showing the surroundings of the shovel 100 (hereinafter referred to as an "excavator surroundings image").

[0067] For example, the display control unit 342 causes the display device 40 to display an image captured by the imaging device 70 as an image around the excavator. Specifically, the display control unit 342 may cause the display device 40 to display images captured by some of the cameras selected from the front camera 70F, the rear camera 70B, the left camera 70L, and the right camera 70R. In this case, the display control unit 342 may switch the camera corresponding to the captured image to be displayed on the display device 40 in response to a predetermined operation by an operator or the like. Furthermore, the display control unit 342 may cause the display device 40 to display all of the images captured by the front camera 70F, the rear camera 70B, the left camera 70L, and the right camera 70R.

[0068] Furthermore, for example, the display control unit 342 generates a converted image by performing a predetermined conversion process on the image captured by the imaging device 70 as the shovel surroundings image, and causes the generated converted image to be displayed on the display device 40. The converted image may be, for example, a viewpoint converted image that combines an overhead image seen from directly above the shovel 100 with a horizontal image seen horizontally in the distance from the shovel 100. The viewpoint converted image may also be a composite image (hereinafter referred to as a "viewpoint converted composite image") that is generated by converting the images captured by the front camera 70F, the rear camera 70B, the left camera 70L, and the right camera 70R into viewpoint converted images that combine the overhead image and the horizontal image, and then combining the images using a predetermined method.

[0069] Furthermore, when the determination unit 344 detects an object to be monitored within a predetermined monitoring area around the shovel 100, the display control unit 342 superimposes and displays an image that highlights the area on the shovel surroundings image that corresponds to the detected object (hereinafter, "detected object area"). This allows the operator or the like to easily confirm the detected object on the shovel surroundings image. Specific display modes will be described later (see FIG. 5).

[0070] When the shovel 100 is remotely operated, a function similar to the display control unit 342 may be provided in the management device 200. This allows the remote operator to check the image of the shovel's surroundings and check detected objects on the image of the shovel's surroundings through a display device (for example, display device 230) provided in the management device 200.

[0071] The determination unit 344 uses the trained model LM that has undergone machine learning and is stored in the memory unit 346 to make a determination regarding objects around the shovel 100 based on the image captured by the imaging device 70. Specifically, the determination unit 344 loads the trained model LM from the memory unit 346 into a main storage device such as a RAM (path 344A) and causes the CPU to execute it, thereby making a determination regarding objects around the shovel 100 based on the image captured by the imaging device 70.

[0072] For example, the determination unit 344 detects the object to be monitored while determining whether or not the object to be monitored is present within the monitoring area around the shovel 100, as described above.

[0073] Furthermore, for example, the determination unit 344 determines (specifies) the type of the detected monitored object, that is, classifies the detected monitored object into a predefined classification list of monitored objects (hereinafter referred to as the "monitored object list"). As described above, the monitored object list may include people, trucks, other construction machinery, utility poles, suspended loads, pylons, buildings, etc.

[0074] Furthermore, for example, the determination unit 344 determines the state of a monitored object detected within the monitoring area around the excavator 100. Specifically, if the detected monitored object is a person, the determination unit 344 may determine which of predefined state classifications (hereinafter referred to as "state classifications") the detected person falls into, such as "sitting," "standing," and "lying down." Furthermore, if the detected monitored object is a truck, the determination unit 344 may determine whether the left and right side seats of the bed of the detected truck are open or closed. More specifically, the determination unit 344 may determine which of the state classifications the truck falls into, such as "left and right side seats closed," "only the left side seat open," "only the right side seat open," and "left and right side seats open."

[0075] Furthermore, for example, the determination unit 344 determines the state of each part of the object to be monitored that is detected within the monitoring area around the shovel 100. Specifically, if the detected object to be monitored is a person, the determination unit 344 may determine the state of each part of the person (for example, the left and right arms, the left and right palms, the left and right fingers, etc.). This allows the determination device 34 to recognize the person's movements, such as gestures, for example.

[0076] For example, as shown in FIGS. 4A and 4B, the trained model LM is configured around a neural network 401.

[0077] In this example, neural network 401 is a so-called deep neural network that has one or more intermediate layers (hidden layers) between an input layer and an output layer. In neural network 401, a weighting parameter representing the connection strength with a lower layer is defined for each of the multiple neurons that make up each intermediate layer. The neural network 401 is configured in such a manner that the neurons in each layer output the sum of values ​​obtained by multiplying each of the input values ​​from the multiple neurons in the upper layer by the weighting parameter defined for each neuron in the upper layer to the neurons in the lower layer via a threshold function.

[0078] The management device 200 (learning unit 2103) performs machine learning, specifically deep learning, on the neural network 401, as described below, to optimize the weighting parameters. As a result, as shown in FIG. 4A , for example, the neural network 401 receives an image captured by the imaging device 70 as an input signal x and outputs, as an output signal y, the probability (predicted probability) of the presence of each object type corresponding to a predefined monitoring target list (in this example, "person," "truck," etc.). The neural network 401 is, for example, a convolutional neural network (CNN). The CNN is a neural network that applies existing image processing techniques (convolution processing and pooling processing). Specifically, the CNN extracts feature data (feature map) smaller in size than the captured image by repeatedly combining convolution processing and pooling processing on the image captured by the imaging device 70. The pixel values ​​of each pixel in the extracted feature map are then input into a neural network consisting of multiple fully connected layers, and the output layer of the neural network can output, for example, a predicted probability of the presence of each object type.

[0079] The neural network 401 may be configured to receive an image captured by the imaging device 70 as an input signal x and output the position and size of an object in the captured image (i.e., the area occupied by the object in the captured image) and the type of the object as an output signal y. In other words, the neural network 401 may be configured to detect an object in the captured image (determine the area occupied by the object in the captured image) and determine the classification of the object. In this case, the output signal y may be configured in the form of image data in which information about the area occupied by the object and its classification is superimposed on the captured image as the input signal x. This allows the determination unit 344 to identify the relative position (distance and direction) of the object from the shovel 100 based on the position and size of the area occupied by the object in the image captured by the imaging device 70, which are output from the trained model LM (neural network 401). This is because the imaging devices 70 (front camera 70F, rear camera 70B, left camera 70L, and right camera 70R) are fixed to the upper rotating body 3, and the imaging range (angle of view) is specified (fixed) in advance. Then, if the position of an object detected by the trained model LM is within the monitored area and is classified as an object in the monitored object list, the determination unit 344 can determine that a monitored object has been detected within the monitored area.

[0080] For example, the neural network 401 may be configured to include neural networks corresponding to a process of extracting an occupied area (window) in which an object exists in a captured image and a process of identifying the type of object in the extracted area. In other words, the neural network 401 may be configured to perform object detection and object classification in a stepwise manner. Furthermore, for example, the neural network 401 may be configured to include neural networks corresponding to a process of defining object classification and object occupation areas (bounding boxes) for each grid cell in which the entire captured image is divided into a predetermined number of partial areas, and a process of combining object occupation areas for each type based on the object classification for each grid cell to determine a final object occupation area. In other words, the neural network 401 may be configured to perform object detection and object classification in parallel.

[0081] The determination unit 344 calculates a predicted probability for each type of object in the captured image, for example, at each predetermined control cycle. When calculating the predicted probability, if the current determination result matches the previous determination result, the determination unit 344 may further increase the current predicted probability. For example, if the previous determination predicted the object in a predetermined area of ​​the captured image to be a "person" (y1), and the object is again determined to be a "person" (y1), the current predicted probability of the object being determined to be a "person" (y1) may be further increased. As a result, for example, if the determination results regarding the classification of objects in the same image area consistently match, the predicted probability is calculated to be relatively high. Therefore, the determination unit 344 can suppress erroneous determinations.

[0082] The determination unit 344 may also determine an object in a captured image by taking into account the movement, such as traveling or turning, of the shovel 100. This is because even if an object around the shovel 100 is stationary, the position of the object in the captured image may move due to the traveling or turning of the shovel 100, making it impossible to recognize the object as the same object. For example, due to the traveling or turning of the shovel 100, an image area determined as a "person" (y1) in the current process may differ from an image area determined as a "person" (y1) in the previous process. In this case, if an image area determined as a "person" (y1) in the current process is within a predetermined range from an image area determined as a "person" (y1) in the previous process, the determination unit 344 may consider the image area to be the same object and perform continuous matching determination (i.e., determination of a state in which the same object is continuously detected). When performing continuous matching determination, the determination unit 344 may add the image area used in the current determination to the image area used in the previous determination and also include an image area within a predetermined range from this image area. This allows the determination unit 344 to continuously perform coincidence determination for the same object around the shovel 100 even if the shovel 100 travels or turns.

[0083] 4B, the neural network 401 may be configured to receive an image captured by the imaging device 70 as an input signal x and output the state of each part of the person detected in the captured image as an output signal y. In this example, the neural network 401 outputs output signals y1(t) to y4(t) corresponding to the state of the right arm, the state of the left arm, the state of the right palm, and the state of the left palm in time series. The output signals y1(t) to y4(t) represent the output signals y1 to y4 at time t. This allows the determination device 34 to recognize the gesture movements of the worker captured in the image captured by the imaging device 70 based on the changes in the output signals y1 to y4 obtained from multiple captured images between times t1 and tn, i.e., the changes in the state of the right arm, the state of the left arm, the state of the right palm, and the state of the left palm. In this way, the probability of each action content of the object entered in the classification table is calculated based on the time-series changes in each part of the object. The action content with the highest probability is then recognized as the action content of the detected object. Specifically, in this example, the determination device 34 can recognize a gesture by the worker requesting an emergency stop.

[0084] For example, at time t1, the neural network 401 outputs output signals y1(t1) to y4(t1) corresponding to a raised right arm, an open right palm, a raised left arm, and an open left palm. Thereafter, at time t2, the neural network 401 outputs output signals y1(t2) to y4(t2) corresponding to a lowered right arm, an open right palm, a lowered left arm, and an open left palm. The states of the output signals y1 to y4 at times t1 and t2 are repeated until time tn, and the determination device 34 may recognize a gesture of the worker appearing in the image captured by the imaging device 70 requesting an emergency stop, based on the output signals y1 to y4 of the neural network 401 between times t1 and tn (i.e., the determination result of the determination unit 344). At this time, a probability is calculated for each of the operator's actions (gestures), such as "raise attachment," "lower attachment," "horizontal movement (swing)," "horizontal movement (travel)," "crawler spin turn," "stop," "sudden stop," and "release." Then, "sudden stop," which has been calculated to have the highest probability, is recognized as the gesture requested by the operator. As a result, the determination device 34 outputs a signal requesting a sudden stop to the controller 30, and the controller 30 (operation control unit 304) can stop the operation of the actuator that drives the driven element in response to the signal. In this way, the controller 30 can control the actuator based on the action of the object around the excavator 100.

[0085] The determination result by the determination unit 344 is displayed on the display device 40 via the display control unit 342, for example.

[0086] 5, a main screen 41V is displayed on the display device 40, and a camera image display area 41m on the main screen 41V displays an image captured by the imaging device 70. This captured image corresponds to the input signal x in FIG. 4A. In addition to the camera image display area 41m, the main screen 41V also includes a date and time display area 41a, a driving mode display area 41b, an attachment display area 41c, an average fuel consumption display area 41d, an engine control status display area 41e, an engine operating time display area 41f, a coolant temperature display area 41g, a remaining fuel amount display area 41h, a rotation speed mode display area 41i, a urea water remaining amount display area 41j, and a hydraulic oil temperature display area 41k.

[0087] The date and time display area 41a is an area on the main screen 41V that displays the current date and time.

[0088] The traveling mode display area 41b is an area on the main screen 41V that displays a graphic representing the current traveling mode of the excavator 100.

[0089] The attachment display area 41c is an area on the main screen 41V that displays a graphic that simulates the type of attachment currently attached to the excavator 100.

[0090] The average fuel consumption display area 41d is an area on the main screen 41V that displays the current average fuel consumption of the excavator 100. The average fuel consumption is, for example, the amount of fuel consumed in a predetermined period of time.

[0091] The engine control state display area 41e is an area on the main screen 41V where a graphic representing the control state of the engine 11 is displayed.

[0092] The engine operation time display area 41f is an area on the main screen 41V that displays the total operation time of the engine 11 from a predetermined timing.

[0093] The coolant temperature display area 41g is an area on the main screen 41V that displays the current temperature state of the coolant for the engine 11.

[0094] The remaining fuel amount display area 41h is an area on the main screen 41V that displays the remaining amount of fuel stored in the fuel tank of the excavator 100.

[0095] The rotation speed mode display area 41i is an area for displaying a mode (rotation speed mode) relating to the current set rotation speed of the engine 11.

[0096] The urea water remaining amount display area 41j is an area on the main screen 41V that displays the remaining amount of urea water stored in the urea water tank.

[0097] In this example, the camera image display area 41m displays an image captured by the rear camera 70B of the imaging device 70, and the image shows a worker PS working behind the shovel 100 and a truck TK parked behind the shovel 100.

[0098] As described above, the determination unit 344 inputs image data of the image captured by the rear camera 70B into the trained model LM (neural network 401), thereby acquiring the object's occupation area in the captured image and the type of object in that occupation area, which are output from the trained model LM. Therefore, in this example, a box icon 501 surrounding the occupation area of ​​an object (worker PS) classified as a "person" output from the trained model LM, and a text information icon 502 indicating that the detected (classified) object is a person, are superimposed on the captured image. Furthermore, a box icon 503 surrounding the occupation area of ​​an object (truck TK) classified as a "truck" output from the trained model LM, and a text information icon 504 indicating that the detected (classified) object is a truck, are superimposed on the captured image. This allows an operator or the like to easily recognize the detected object and the type of the detected object. Furthermore, the camera image display area 41m of the display device 40 may also display the above-mentioned predicted probabilities (specifically, the predicted probability of the presence of a "person" or the predicted probability of the presence of a "truck") used in the determination by the determination unit 344. If it is determined that a person is present within a predetermined range around the shovel 100 before the operator operates the control device 26, the controller 30 (operation control unit 304) may disable the actuator or limit its operation to a very slow speed even if the operator operates the control lever. Specifically, in the case of a hydraulic pilot-type control device 26, if it is determined that a person is present within a predetermined range around the shovel 100, the controller 30 can disable the actuator by locking the gate lock valve. In the case of an electric control device 26, the actuator can be disabled by disabling the signal from the controller 30 to the control valve. The same applies when another type of control device 26 is used, as long as it uses an operation control valve that outputs a pilot pressure corresponding to a control command from the controller 30 and applies the pilot pressure to a pilot port of a corresponding control valve in the control valve.When it is desired to operate the actuator at a slow speed, the control signal from the controller 30 to the operation control valve can be limited to a content corresponding to a relatively small pilot pressure, thereby causing the actuator to operate at a slow speed. In this way, when it is determined that a detected object is present within a predetermined range around the excavator 100, the actuator is not driven even if the operation device 26 is operated, or is driven at an operation speed (slow speed) lower than the operation speed corresponding to the operation input to the operation device 26. Furthermore, when it is determined that a person is present within a predetermined range around the excavator 100 while the operator is operating the operation device 26, the operation of the actuator may be stopped or slowed down regardless of the operator's operation. Specifically, when the operation device 26 is of a hydraulic pilot type, when it is determined that a person is present within a predetermined range around the excavator 100, the controller 30 stops the actuator by locking the gate lock valve. Furthermore, when an operating control valve is used that outputs a pilot pressure corresponding to a control command from the controller 30 and applies the pilot pressure to a pilot port of a corresponding control valve within the control valve, the controller 30 can disable the actuator or slow down its operation by disabling the signal to the operating control valve or outputting a deceleration command to the operating control valve. Furthermore, if the detected object is a truck, control related to stopping or slowing down the actuator does not need to be performed. For example, the actuator may be controlled to avoid the detected truck. In this way, the type of detected object is recognized, and the actuator is controlled based on that recognition.

[0099] The image captured by the imaging device 70 may be displayed over the entire display area of ​​the display device 40. The display device 40 may also display a converted image (for example, the above-mentioned synthetic viewpoint converted image) based on the image captured by the imaging device 70, in which case a box icon or a text information icon may be superimposed on a portion of the converted image that corresponds to the area occupied by the object. When the shovel 100 is remotely operated, content similar to that shown in FIG. 5 may be displayed on a display device of the management device 200 (for example, the display device 230).

[0100] A trained model LM is stored in the memory unit 346. When an updated version of the trained model is received from the management device 200 via the communication device 90, that is, when an updated trained model (hereinafter referred to as an "additionally trained model") is received from the management device 200 as described below, the trained model LM in the memory unit 346 is updated to the received additionally trained model. This allows the determination unit 344 to use the additionally trained model that has been additionally trained by the management device 200, thereby improving the accuracy of determination regarding objects around the shovel 100 in accordance with updates to the trained model.

[0101] <Configuration of management device> The management device 200 includes a control device 210 , a communication device 220 , a display device 230 , and an input device 240 .

[0102] The control device 210 controls various operations of the management device 200. The control device 210 includes, for example, a determination unit 2101, a teacher data generation unit 2102, and a learning unit 2103 as functional units realized by executing one or more programs stored in a ROM or a non-volatile auxiliary storage device on a CPU. The control device 210 also includes, for example, memory units 2104 and 2105 as memory areas defined in a non-volatile internal memory of the auxiliary storage device or the like.

[0103] The communication device 220 is any device that connects to a communication network and communicates with the outside, such as a plurality of excavators 100 .

[0104] The display device 230 is, for example, a liquid crystal display or an organic EL display, and displays various information images under the control of the control device 210.

[0105] The input device 240 accepts operational input from a user. The input device 240 includes, for example, a touch panel mounted on a liquid crystal display or an organic EL display. The input device 240 may also include a touch pad, a keyboard, a mouse, a trackball, etc. Information regarding the operational state of the input device 240 is taken into the control device 210.

[0106] The determination unit 2101 uses the trained model LM stored in the memory unit 2105 and subjected to machine learning by the learning unit 2103 to make a determination regarding objects around the shovel 100 based on captured images IM1 received from multiple shovels 100, i.e., captured images IM1 (path 2101A) read out from the memory unit 2104. Specifically, the determination unit 2101 loads the trained model LM from the memory unit 346 into a main storage device such as RAM (path 2101B) and causes the CPU to execute it, thereby making a determination regarding objects around the shovel 100 based on the captured images IM1 read out from the memory unit 2104. More specifically, the determination unit 2101 sequentially inputs the multiple captured images IM1 stored in the memory unit 2104 into the trained model LM, and makes a determination regarding objects around the shovel 100. A determination result 2101D of the determination unit 2101 is input to the teacher data generation unit 2102. At this time, the judgment results 2101D may be input to the teacher data generation unit 2102 sequentially for each captured image IM1, or may be compiled, for example, in a list, and then input to the teacher data generation unit 2102.

[0107] The teacher data generation unit 2102 (an example of a teacher information generation unit) generates teacher data (an example of teacher information) for the learning unit 2103 to perform machine learning on the learning model, based on a plurality of captured images IM1 received from a plurality of excavators 100. The teacher data represents a combination of an arbitrary captured image IM1 and a correct answer that should be output by the learning model when the captured image IM1 is used as input to the learning model. Furthermore, the learning model is the subject of machine learning, and naturally has the same configuration as the trained model LM, for example, is configured mainly around the above-mentioned neural network 401.

[0108] For example, the teacher data generation unit 2102 reads out captured images IM1 received from multiple shovels 100 from the storage unit 2104 (path 2102A) and displays them on the display device 40, while also displaying a GUI (Graphical User Interface) (hereinafter, "teacher data creation GUI") that allows a manager, worker, etc. of the management device 200 to create teacher data. Then, the manager, worker, etc. uses the input device 240 to operate the teacher data creation GUI and specify the correct answer corresponding to each captured image IM1, thereby creating teacher data in a format that conforms to the algorithm of the learning model. In other words, the teacher data generation unit 2102 can generate multiple teacher data (teacher data sets) in response to operations (tasks) by the manager, worker, etc. that target multiple captured images IM1.

[0109] Furthermore, the teacher data generation unit 2102 generates teacher data for the learning unit 2103 to perform additional learning on the learned model LM, based on the multiple captured images IM1 received from the multiple shovels 100.

[0110] For example, the teacher data generation unit 2102 reads out a plurality of captured images IM1 from the storage unit 2104 (path 2102A), and displays each captured image IM1 and the determination result (output result) 2101D of the determination unit 2101 corresponding to the captured image IM1 side by side on the display device 230. This allows the administrator or operator of the management device 200 to select a combination corresponding to an erroneous determination from the combinations of captured images IM1 and corresponding determination results displayed on the display device 230 via the input device 240. The administrator or operator can then use the input device 240 to operate the teacher data creation GUI and create teacher data for additional learning that represents a combination of a captured image IM1 corresponding to an erroneous determination, i.e., a captured image IM1 that caused the trained model LM to make an erroneous determination, and a correct answer that the trained model LM should output when the captured image IM1 is input. In other words, the teacher data generation unit 2102 can generate multiple teacher data (teacher datasets) for additional learning in accordance with operations (tasks) by an administrator, worker, etc. targeting a captured image IM1 selected from multiple captured images IM1 and corresponding to an incorrect judgment in the trained model LM.

[0111] That is, the teacher data generation unit 2102 generates teacher data for generating the initial trained model LM from the multiple captured images IM1 received from the multiple shovels 100. Then, at each predetermined timing (hereinafter, "additional learning timing"), the teacher data generation unit 2102 generates teacher data for additional learning from captured images IM1 that result in an erroneous determination by the trained model LM, which are selected from the captured images IM1 received from the multiple shovels 100 after the most recent trained model LM is implemented in the multiple shovels 100.

[0112] Note that some of the captured images IM1 received from each of the multiple shovels 100 may be used as the base of a validation data set for the trained model LM. In other words, the captured images IM1 received from each of the multiple shovels 100 may be divided into captured images IM1 for generating training data and captured images IM1 for generating a validation data set.

[0113] The timing of additional learning may be a periodically specified timing, for example, when one month has passed since the previous machine learning (additional learning). The timing of additional learning may also be, for example, when the number of captured images IM1 exceeds a predetermined threshold, that is, when the number of captured images IM1 required for additional learning by the learning unit 2103 has been collected.

[0114] The learning unit 2103 generates a trained model LM by having the learning model perform machine learning based on the training data 2102B (training data set) generated by the training data generation unit 2102. Then, the generated training model LM is subjected to accuracy verification using a verification data set prepared in advance, and is then stored in the storage unit 2105 (path 2103B).

[0115] Furthermore, the learning unit 2103 generates an additional trained model by performing additional learning on the trained model LM (path 2103A) read from the storage unit 2105 based on the training data (trainer dataset) generated by the training data generation unit 2102. Then, the accuracy of the additional trained model is verified using a verification dataset prepared in advance, and the trained model LM in the storage unit 2105 is updated with the additional trained model whose accuracy has been verified (path 2103B).

[0116] For example, as described above, when the learning model is configured mainly with the neural network 401, the learning unit 2103 applies a known algorithm such as backpropagation to optimize weighting parameters so as to reduce the error between the output of the learning model and the teacher data, thereby generating a learned model LM. The same applies to the generation of an additional learned model.

[0117] The initial trained model LM generated from the learning model may be generated by an external device different from the management device 200. In this case, the teacher data generation unit 2102 may be configured to generate only teacher data for additional learning, and the learning unit 2103 may be configured to generate only the additional trained model.

[0118] The storage unit 2104 stores (preserves) the captured images IM1 received from each of the plurality of shovels 100 via the communication device 220.

[0119] The captured image IM1 that has been used for generating the training data by the training data generating unit 2102 may be stored in a storage device separate from the storage unit 2104.

[0120] The trained model LM is stored (saved) in the memory unit 2105. The trained model LM updated with the additionally trained model generated by the learning unit 2103 is transmitted to each of the multiple shovels 100 via the communication device 220 (an example of a model transmission unit) at a predetermined timing (hereinafter, "model transmission timing"). This allows the same updated trained model LM, that is, the additionally trained model, to be shared among the multiple shovels 100.

[0121] The model transmission timing may be when the trained model LM in the storage unit 2105 is updated, that is, immediately after the trained model LM is updated, or when a predetermined time has elapsed after the update, etc. Furthermore, the model transmission timing may be, for example, when a confirmation reply to a notification of the trained model LM update, which is sent to the multiple excavators 100 via the communication device 220, is received by the communication device 220 after the trained model LM is updated.

[0122] [Specific operation of the excavator support system] Next, with reference to FIG. 6, a specific operation of the excavator support system SYS will be described.

[0123] FIG. 6 is a sequence diagram showing an example of the operation of the excavator support system SYS.

[0124] In step S10, the communication devices 90 of the multiple shovels 100 transmit captured images IM1 to the management device 200 at their respective image transmission timings. As a result, the management device 200 receives the captured images IM1 from each of the multiple shovels 100 via the communication devices 220 and cumulatively stores them in the memory unit 2104.

[0125] In step S12, the determination unit 2101 of the management device 200 inputs the plurality of captured images IM1 received from the plurality of excavators 100 and stored in the storage unit 2104 into the trained model LM, and performs a determination process.

[0126] In step S14, an administrator or worker of the management device 200 verifies the judgment results based on the trained model LM and, via the input device 240, specifies (selects) from among the multiple captured images IM those captured images IM that have been incorrectly judged by the trained model LM.

[0127] In step S16, the teacher data generation unit 2102 of the management device 200 generates a teacher data set for additional learning in response to an operation of the teacher data creation GUI via the input device 240 by an administrator, worker, or the like.

[0128] In step S18, the learning unit 2103 of the management device 200 performs additional learning of the trained model LM using the teacher dataset for additional learning, generates an additional trained model, and updates the trained model LM in the memory unit 2104 with the additional trained model.

[0129] In step S20, the communication device 220 of the management device 200 transmits the updated trained model LM to each of the multiple excavators 100.

[0130] The timing at which the updated trained model LM is transmitted to the shovel 100 (model transmission timing) may be different for each of the multiple shovels 100, as described above.

[0131] In step S22, each of the multiple excavators 100 updates the trained model LM in the memory unit 346 to the updated trained model received from the management device 200.

[0132] [Another example of an excavator support system configuration] Next, another example of the configuration of the excavator support system SYS according to this embodiment will be described with reference to FIG.

[0133] 7 is a functional block diagram showing another example of the functional configuration of the excavator support system SYS according to this embodiment. In the following, the following description will focus on the parts of this example that are different from the example described above (FIG. 3).

[0134] <Excavator configuration> The shovel 100 includes, as a configuration related to the control system, a recording device 32, a determination device 34, etc., similar to the example described above, and the recording device 32 includes a recording control unit 322 and a memory unit 324, similar to the example described above.

[0135] As in the above example, the recording control unit 322 records the captured images of the imaging device 70 (front camera 70F, rear camera 70B, left camera 70L, and right camera 70R) in the storage unit 324 at a predetermined recording timing. At this time, the recording control unit 322 also records the judgment result of the judgment unit 344 based on the captured image to be recorded, that is, the learned model LM. In this example, the recording control unit 322 records in the storage unit 324 a captured image IM2 (hereinafter, "captured image with judgment information") that includes information on the judgment result (judgment information) as tag information or the like.

[0136] The captured image and the determination information may be recorded as separate files. In this case, the captured image and the determination information may be associated with each other in the storage unit 324.

[0137] As described above, captured images IM2 with determination information are recorded in the storage unit 324 under the control of the recording control unit 322 during the period from the completion of the initial processing after startup of the shovel 100 until the shovel 100 is stopped. One or more captured images IM2 with determination information recorded in the storage unit 324 are transmitted to the management device 200 via the communication device 90 at a predetermined image transmission timing.

[0138] <Configuration of management device> Similar to the example described above, the management device 200 includes a control device 210, a communication device 220, a display device 230, and an input device 240, and the control device 210 includes a teacher data generation unit 2102, a learning unit 2103, and storage units 2104 and 2105. That is, in this example, unlike the example of FIG. 3, the determination unit 2101 is omitted.

[0139] The teacher data generation unit 2102 generates teacher data for the learning unit 2103 to perform additional learning on the learned model LM based on multiple captured images IM2 with judgment information received from multiple shovels 100 via the communication device 220 and stored in the memory unit 2104.

[0140] For example, the teacher data generation unit 2102 displays the captured image and the judgment information included in the captured image IM2 with judgment information side by side. This allows the administrator or operator of the management device 200 to select a combination corresponding to an erroneous judgment from among the combinations of captured images and corresponding judgment information (judgment results) displayed on the display device 230 via the input device 240. The administrator or worker can then use the input device 240 to operate the teacher data creation GUI to create teacher data for additional learning that represents a combination of captured images corresponding to the erroneous judgment combination and the correct answer that the trained model LM should output when the captured images are input. In other words, the teacher data generation unit 2102 can generate multiple teacher data sets (teacher datasets) for additional learning in response to operations (tasks) performed by the administrator or worker on captured images selected from multiple captured images and corresponding to an erroneous judgment in the trained model LM. Furthermore, in this example, the management device 200 does not need to input multiple captured images received from multiple excavators 100 into the trained model LM and perform processing to obtain a judgment result, thereby improving processing efficiency for additional learning.

[0141] [Another example of the configuration of an excavator support system] Next, with reference to FIG. 8, still another example of the configuration of the excavator support system SYS according to this embodiment will be described.

[0142] 8 is a functional block diagram showing yet another example of the functional configuration of the excavator support system SYS according to this embodiment. In the following, the following description will focus on the parts of this example that are different from the example described above (FIG. 3).

[0143] <Configuration of management device> The management device 200 includes a control device 210, a communication device 220, a display device 230, an input device 240, and a computer graphics generation device 250 (hereinafter referred to as a "CG (Computer Graphics) image generation device").

[0144] The CG image generation device 250 generates computer graphics (hereinafter, "CG image") IM3 representing the surroundings of the shovel 100 at the work site in response to operations by an operator or the like of the management device 200. For example, the CG image generation device 250 is configured primarily with a computer including, for example, a CPU, a memory device such as RAM, an auxiliary storage device such as ROM, and various input / output interface devices, and is pre-installed with application software that enables the operator or the like to create the CG image IM3. The operator or the like then creates the CG image IM3 on the display screen of the CG image generation device 250 via a predetermined input device. This allows the CG image generation device 250 to generate a CG image IM3 representing the surroundings of the shovel 100 at the work site in response to work (operations) by the operator or the like of the management device 200. The CG image generation device 250 can also generate a CG image IM3 representing a work environment under weather conditions and sunshine conditions that differ from those corresponding to the actual captured image (for example, captured image IM1) based on the captured image of the surroundings of the shovel 100. The CG image IM3 generated by the CG image generating device 250 is input to the control device 210.

[0145] The CG image IM3 may be generated (created) outside the management device 200.

[0146] The control device 210 includes a determination unit 2101, a teacher data generation unit 2102, a learning unit 2103, and storage units 2104 and 2105, as in the above example.

[0147] The determination unit 2101 uses a trained model LM stored in the memory unit 2105 and on which machine learning has been performed by the learning unit 2103 to make a determination regarding objects around the shovel 100, based on a plurality of captured images IM1 (path 2101A) and CG images IM3 (path 2101C) read from the memory unit 2104. Specifically, the determination unit 2101 loads the trained model LM from the memory unit 346 into a main storage device such as RAM (path 2101B) and causes the CPU to execute it, thereby making a determination regarding objects around the shovel 100, based on the captured images IM1 and CG images IM3 read from the memory unit 2104. More specifically, the determination unit 2101 sequentially inputs the plurality of captured images IM1 and CG images IM3 stored in the memory unit 2104 into the trained model LM, and makes a determination regarding objects around the shovel 100. A determination result 2101D of the determination unit 2101 is input to the teacher data generation unit 2102. At this time, the judgment results 2101D may be input to the teacher data generation unit 2102 sequentially for each of the multiple captured images IM1 and CG images IM3, or may be compiled, for example, in a list, and then input to the teacher data generation unit 2102.

[0148] The teacher data generation unit 2102 generates teacher data for the learning unit 2103 to machine-train the learning model based on multiple captured images IM1 received from multiple shovels 100 and CG image IM3 generated by the CG image generation device 250 (stored in the memory unit 2104).

[0149] For example, the teacher data generation unit 2102 reads out from the storage unit 2104 captured images IM1 received from multiple excavators 100 and CG images IM3 generated by the CG image generation device 250 (paths 2102A and 2102C), displays them on the display device 40, and also displays a teacher data creation GUI. Then, a manager, worker, or the like uses the input device 240 to operate the teacher data creation GUI and specify the correct answer corresponding to each captured image IM1 or CG image IM3, thereby creating teacher data in a format consistent with the learning model algorithm. In other words, the teacher data generation unit 2102 can generate multiple teacher data (teacher datasets) in response to operations (tasks) by a manager, worker, or the like targeting multiple captured images IM1 and CG images IM3.

[0150] In addition, the teacher data generation unit 2102 generates teacher data for the learning unit 2103 to perform additional learning on the learned model LM based on multiple captured images IM1 received from multiple shovels 100 and a CG image IM3 generated by the CG image generation device 250 (stored in the memory unit 2104).

[0151] The teacher data generation unit 2102 reads out a plurality of captured images IM1 and CG images IM3 from the storage unit 2104 (paths 2102A and 2102C), and displays each captured image IM1 or CG image IM3 and the determination result (output result) of the determination unit 2101 (trained model LM) corresponding to the captured image IM1 or CG image IM3 side by side on the display device 230. This allows the administrator or operator of the management device 200 to select a combination corresponding to an incorrect determination from the combinations of the captured image IM1 or CG image IM3 and the corresponding determination result of the trained model LM displayed on the display device 230 via the input device 240. The administrator or operator can then use the input device 240 to operate the teacher data creation GUI and create teacher data for additional learning that represents a combination of the captured image IM1 or CG image IM3 corresponding to the incorrect determination combination and the correct answer that the trained model LM should output when the captured image IM1 or CG image IM3 is input. In other words, the teacher data generation unit 2102 can generate multiple teacher data sets (teacher datasets) for additional learning in response to operations (tasks) by a manager, worker, or the like targeting at least one of the captured images IM1 and CG images IM3 selected from the multiple captured images IM1 and IM3 corresponding to an erroneous determination by the trained model LM. This allows teacher data to be generated using the captured images IM1 collected from multiple excavators 100 as well as the CG image IM3, thereby enhancing the teacher data. In particular, the CG image IM3 allows for the flexible creation of various virtual work site situations, i.e., various environmental conditions. Therefore, by generating a teacher dataset using the CG image IM3, the trained model LM can achieve relatively high determination accuracy adapted to various work sites at an earlier stage.

[0152] Note that, because the CG image IM3 generated by the CG image generation device 250 is artificially created, the presence or absence and positions of monitoring targets such as people, trucks, pylons, and utility poles in the CG image IM3 are known. In other words, the correct answer that the trained model LM should output when the CG image IM3 is input is known. Therefore, the CG image generation device 250 can output, together with the CG image IM3, data related to the correct answer that the trained model LM should output when the CG image IM3 is input (hereinafter, "correct answer data") to the control device 210. Therefore, the control device 210 (teacher data generation unit 2102) can automatically extract erroneous determinations in the determination process by the trained model LM (determination unit 2101) using the CG image IM3 as input, based on the correct answer data input from the CG image generation device 250, and automatically generate multiple pieces of teacher data (teacher datasets) for additional learning that represent combinations of the CG image IM3 corresponding to the extracted erroneous determination and the correct answer that the trained model LM should output when the CG image IM3 is input. The learning unit 2103 can then perform additional learning of the trained model LM, such as the above-mentioned backpropagation, based on the training data automatically generated by the training data generation unit 2102. In other words, the control device 210 can also automatically generate an additional trained model based on the CG image IM3 generated by the CG image generation device 250 and the correct answer data.

[0153] [Effect] Next, the operation of the excavator support system SYS according to the present embodiment will be described.

[0154] In this embodiment, the imaging device 70 acquires captured images of the surroundings of the shovel 100 on which it is mounted (hereinafter, "the shovel itself"). Furthermore, the determination unit 344 uses a trained model LM that has undergone machine learning to make a determination regarding objects around the shovel itself, based on the captured images IM1 acquired by the imaging device 70. The trained model LM is then updated to an additionally trained model that has undergone additional learning based on acquired environmental information, specifically, training data generated from images captured by the imaging device 70 acquired by the shovel itself and other shovels 100 (hereinafter, "other shovels"). When the trained model LM is updated, the determination unit 344 makes the above determination based on the captured images acquired by the imaging device 70, using the updated trained model LM.

[0155] Specifically, the teacher data generation unit 2102 of the management device 200 generates teacher data based on the captured image IM1 acquired by the shovel 100 and received from the shovel 100. Furthermore, the learning unit 2103 of the management device 200 performs additional learning on the same trained model (i.e., the trained model LM) as the trained model used by the shovel 100 to determine surrounding objects, based on the teacher data generated by the teacher data generation unit 2102, to generate an additional trained model. Then, the communication device 220 of the management device 200 transmits the additional trained model that has been additionally trained by the learning unit 2103 to the shovel 100, and updates the trained model LM of the shovel 100 with the additional trained model.

[0156] As a result, the trained model LM of the shovel 100 is updated with an additional trained model that has undergone additional training using training data generated from images captured by the imaging device 70 acquired under various environments by the shovel itself and other shovels. Therefore, the shovel 100 can improve the accuracy of judgment under various environmental conditions when making judgments about objects around the shovel 100 based on environmental information around the shovel 100 (images captured by the imaging device 70).

[0157] Note that the shovel 100 (an example of a second shovel) that acquires captured images of the surroundings that serve as the source of training data and the shovel 100 (an example of a first shovel) that uses the trained model LM to make judgments about surrounding objects based on the captured images may be separate. In this case, the imaging device 70 (an example of a second environmental information acquisition unit) of the former shovel 100 acquires captured images of the surroundings of the shovel 100 for generating training data, and the imaging device 70 (an example of a first environmental information acquisition unit) of the latter shovel 100 acquires captured images of the surroundings of the shovel 100 for making judgments about surrounding objects. Furthermore, for example, the management device 200 may be configured to collect captured images of the surroundings from one or more shovels (an example of a second shovel) different from the shovel 100 that does not have a function for making judgments about objects, generate and update the trained model LM, and transmit the trained model LM to the shovel 100 (an example of a first shovel). In other words, the trained model LM used by the shovel 100 (determination unit 344) to make judgments about objects around the shovel 100 may be updated to an additional trained model that has undergone additional learning based on training data generated from environmental information (specifically, captured images of the surroundings) acquired from at least one of the shovel itself and another shovel different from the shovel itself.

[0158] Furthermore, in this embodiment, the multiple shovels 100, including the shovel itself, each have the same trained model LM and use the trained model LM to make judgments about surrounding objects. That is, there are multiple shovels 100 (an example of a first shovel) that use the trained model LM to make judgments about surrounding objects based on surrounding environmental information (images captured by the imaging device 70). The trained model LM possessed by each of the multiple shovels 100 may be updated to the same additional trained model generated by the management device 200.

[0159] This makes it possible to achieve relatively high determination accuracy uniformly across multiple shovels 100 under a variety of environmental conditions.

[0160] Furthermore, in this embodiment, the trained model LM may be updated to an additionally trained model that has undergone additional learning based on training data generated from captured images of the surroundings acquired by multiple shovels 100. In other words, there are multiple shovels 100 (an example of a second shovel) that acquire surrounding environmental information (images captured by the imaging device 70) for generating training data.

[0161] This allows for the use of captured images from multiple shovels 100, making it possible to prepare training data that is compatible with a wider variety of environments. Therefore, the shovel 100 can further improve the accuracy of determination under a variety of environmental conditions when making determinations about objects around the shovel 100 based on environmental information around the shovel 100 (images captured by the imaging device 70).

[0162] The trained model LM may be updated to an additionally trained model that has undergone additional learning based on training data generated from captured images of the surroundings acquired by a single shovel 100. In other words, there may be only one shovel 100 (an example of a second shovel) that acquires surrounding environmental information (images captured by the imaging device 70) for generating training data.

[0163] Furthermore, in this embodiment, the recording control unit 322 records, at a predetermined recording timing, captured images obtained by the imaging device 70 as environmental information. Then, the trained model LM of the shovel 100 is updated in the management device 200 to an additionally trained model that has been additionally trained based on training data generated from the captured images recorded by the recording control unit 322.

[0164] This makes it possible to selectively record captured images of appropriate quality for use as training data (for example, images captured under operating conditions that are likely to result in erroneous determination and that should be applied to additional learning, or images captured when there is a specific possibility that an erroneous determination has occurred, etc.). This makes it possible to suppress overlearning of the trained model LM, and further improve the accuracy of determination regarding objects around the excavator 100.

[0165] In addition, in this embodiment, the predetermined recording timing may be when the excavator itself is turning or traveling.

[0166] This makes it possible to specifically record captured images during the turning operation and traveling operation of the excavator 100, which are operating conditions that are prone to erroneous determination and that should be applied to additional learning.

[0167] Furthermore, in this embodiment, the determination unit 344 makes a determination regarding the detection of an object around the own excavator. The predetermined recording timing may be when the determination unit 344 determines that an object around the own excavator has been detected.

[0168] This makes it possible to record the captured image at the time when there is a possibility of an erroneous determination regarding the detection of an object around the shovel 100, that is, an erroneous detection.

[0169] [Transformation / Improvement] Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the claims.

[0170] For example, in the above-described embodiment, the environmental information around the shovel 100 serving as input information for the trained model LM is an image captured by the imaging device 70, but this is not limited to this example. Specifically, the input information for the trained model LM may be output information (range image data, reflected wave spectrum data, etc.) from any spatial recognition sensor (an example of an environmental information acquisition unit) such as a LIDAR (Light Detecting and Ranging), millimeter-wave radar, or range image sensor mounted on the shovel 100. In this case, the management device 200 performs machine learning of the trained model and additional training of the trained model LM based on training data generated from the output information of the spatial recognition sensors received from multiple shovels 100.

[0171] Furthermore, in the above-described embodiment and modified example, one trained model LM is generated, and a determination regarding objects around the shovel 100 is made using the single trained model LM, but this is not limited to this example. Specifically, trained models may be generated for a plurality of predefined environmental conditions around the shovel 100, a trained model that matches the environmental conditions around the shovel 100 may be selected, and a determination regarding objects around the shovel 100 may be made using the selected trained model. In this case, the environmental conditions around the shovel 100 may include, for example, weather conditions such as rain, sunshine, and cloudy, and the type of background reflected in the captured image (for example, geographical conditions such as a residential area with houses reflected or a mountainous area with forests reflected, and conditions such as the presence or absence of pavement). As a result, the trained model undergoes additional learning in accordance with each environmental condition, allowing the shovel 100 to improve the determination accuracy for each environmental condition.

[0172] Furthermore, in the above-described embodiments and variant examples, a trained model LM is generated using a machine learning technique centered on a neural network, but other machine learning techniques may be applied instead of or in addition to a neural network.

[0173] For example, Fig. 9 is a conceptual diagram showing another example of the determination process using the learned model LM. Specifically, Fig. 9 is a conceptual diagram showing a determination method for an object when a support vector machine (SVM) is applied as a machine learning technique.

[0174] As shown in Figure 9, two-dimensional vector information (hereinafter referred to as "feature vector") of a predetermined feature extracted from environmental information around the shovel 100 (for example, an image captured by the imaging device 70) acquired by the shovel 100 is plotted on a two-dimensional plane as training data.

[0175] For example, it is assumed that the feature vector group 902 (plots indicated by white circles in the figure) and the feature vector group 903 (plots indicated by black circles in the figure) correspond to environmental information when an object is not present and when an object is present, respectively. Then, using the SVM technique, a separation line 901 can be defined between the feature vector group 902 as training data and the feature vector group 903 plotted by black circles. Therefore, by using the separation line 901, a feature vector is calculated from the environmental information acquired by the shovel 100, and the presence or absence of an object can be determined, i.e., object detection can be performed, depending on whether the calculated feature vector is on the feature vector group 902 side or the feature vector group 903 side from the separation line 901. For example, because the feature vector 904 is on the feature vector group 903 side from the separation line 901, the presence of an object can be recognized from the environmental information acquired by the shovel 100, and the object can be detected. In a similar manner, it is also possible to classify detected objects (as to whether they are of a particular type or not).

[0176] Specifically, the feature vectors used are, for example, image features such as HOG (Histogram of Oriented Gradients) features and LBP (Local Binary Pattern) features of the image captured by the imaging device 70. Furthermore, since feature vectors are usually specified in a multidimensional space exceeding three dimensions, when classifying the multidimensional space corresponding to the feature vector into the presence or absence of an object or into a specific type of object (for example, a "person") and other types of objects, a separating hyperplane is defined by the SVM technique.

[0177] In this way, by applying the SVM technique, the management device 200 can generate feature vectors as training data from multiple pieces of environmental information collected from multiple shovels 100, and can define a separating hyperplane corresponding to the trained model LM based on the training data. After defining the separating hyperplane, the management device 200 can further generate feature vectors as additional training data from the environmental information collected from the multiple shovels 100, and update the separating hyperplane based on the previous training data and the additional training data. As a result, similar to the above-described embodiment, the shovel 100 can improve the accuracy of determination under various environmental conditions when determining objects around the shovel 100 based on environmental information around the shovel 100.

[0178] In addition to neural networks and SVMs, other known machine learning methods, such as decision tree methods such as random forests, nearest neighbor methods, and naive Bayes classifiers, can also be applied.

[0179] Furthermore, in the above-described embodiment and modified examples, the shovel 100 makes a determination regarding a surrounding object using the trained model LM, but another construction machine may also make a determination regarding a surrounding object using the trained model LM. That is, the shovel assistance system SYS according to the above-described embodiment may be configured to include, instead of or in addition to the shovel 100, other construction machines such as road machines such as bulldozers, wheel loaders, and asphalt finishers, and forestry machines equipped with harvesters, etc.

[0180] Finally, this application claims priority based on Japanese Patent Application No. 2018-205906, filed on October 31, 2018, the entire contents of which are incorporated herein by reference. [Explanation of symbols]

[0181] 30 Controllers 32 Recording Device 34 Judgment device 40 Display device 70 imaging device (environmental information acquisition unit, first environmental information acquisition unit, second environmental information acquisition unit) 90 Communication equipment (environmental information transmission unit) 100 Shovel (1st Shovel, 2nd Shovel) 200 Management device (external device) 210 Control device 220 Communication equipment (model transmitter) 230 Display device 240 Input Device 250 Computer Graphics Generation Device 302 Information Department 304 Operation control unit (control unit) 322 Recording control unit (recording unit) 324 Storage section 342 Display control unit 344 Judgment section 346 Storage section 2101 Judgment section 2102 Teacher data generation unit (teacher information generation unit) 2103 Learning Department 2104,2105 Storage section IM1 captured image IM2 Captured image with judgment information IM3 Computer Graphics LM pre-trained model SYS Excavator Support System

Claims

1. an environmental information acquisition unit that acquires environmental information around the excavator; a determination unit that uses a trained model that has undergone machine learning to make a determination regarding an object around the shovel based on the environmental information acquired by the environmental information acquisition unit, The trained model is updated to an additional trained model that has been additionally trained by an external device capable of communicating with the shovel based on teacher information generated from environmental information that caused the trained model to make an erroneous judgment when the judgment was made, selected from the environmental information acquired by the environmental information acquisition unit, When the trained model is updated, the determination unit performs the determination using the updated trained model based on environmental information acquired by the environmental information acquisition unit. Shovel.

2. transmitting the environmental information acquired by the environmental information acquisition unit and information relating to the determination result of the determination unit based on the environmental information to the external device; The trained model is updated to an additionally trained model that has been additionally trained in the external device based on teacher information generated from environmental information that caused the trained model to make an erroneous judgment when the judgment was made, selected from the environmental information acquired by the environmental information acquisition unit based on information related to the judgment result. The shovel according to claim 1.

3. The trained model is updated to an additionally trained model in which the additional learning is performed based on the teacher information generated from environmental information acquired from at least one of the own excavator and another excavator different from the own excavator. The shovel according to claim 1 or 2.

4. The trained model is updated to the additionally trained model that has undergone the additional learning based on the teacher information generated from environmental information of an artificially generated work site. The shovel according to any one of claims 1 to 3.

5. a recording unit that records environmental information acquired by the environmental information acquisition unit during a swing operation or a traveling operation of the shovel itself, The trained model is updated to the additionally trained model in which the additional learning is performed based on the teacher information generated from the environmental information recorded by the recording unit. A shovel according to any one of claims 1 to 4.

6. A recording unit is provided, the determination unit performs the determination regarding detection of an object around the shovel itself, the recording unit records the environmental information acquired by the environmental information acquisition unit when the determination unit determines that an object has been detected around the local shovel, The trained model is updated to the additionally trained model in which the additional learning is performed based on the teacher information generated from the environmental information recorded by the recording unit. A shovel according to any one of claims 1 to 4.

7. An actuator; a control unit that controls the actuator based on the determination, A shovel according to any one of claims 1 to 6.

8. An operating device is provided, the actuator is driven based on an operation on the operation device, When it is determined based on the determination that the object is present within a predetermined range around the shovel, the control unit does not drive the actuator even if the operation device is operated. The shovel according to claim 7.

Images