Systems including operating machines and operating machines

Abstract

An image clearly showing a working machine can be easily obtained. A posture data generating unit (163) estimates the posture of a working device relative to a main body of the working machine in a captured image showing the working machine. An action state image generating unit (165) creates a three-dimensional model representing the three-dimensional shape of the working machine based on the posture of the working device. A specific viewpoint image generating unit (166) creates image data including a two-dimensional image obtained by observing the three-dimensional model from a viewpoint position representing the position of the viewpoint where the three-dimensional model is virtually observed.

Description

Technical Field

[0001] The present disclosure relates to a system including a work machine, and a work machine. Background Art

[0002] The wheel loader has a bucket that can rotate relative to the boom at the front end of the boom that can move in the up-down direction. During the excavation work of the wheel loader, the bucket is pressed into the sand pile by moving the vehicle forward and the boom is raised. Thus, the sand is dug onto the bucket.

[0003] In the past, a wheel loader has been proposed that includes a sensor for detecting an inclination angle of a boom relative to a horizontal direction, and a sensor for detecting an inclination angle of a bucket relative to the boom (for example, refer to Japanese Patent Application Publication No. 2018-135649 (Patent Document 1)).

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2018-135649 Summary of the invention

[0007] Problems to be solved by the invention

[0008] In the case of a wheel loader, a sensor for detecting the posture of the working device is installed, and the work classification can be completed according to the posture of the working device detected by the sensor. On the other hand, in a small machine or other cheap machine, a sensor is not installed, and the posture of the working device cannot be detected by the sensor.

[0009] Since a wheel loader performs work while moving relative to various objects, there are many external disturbances, making it difficult to perform work classification using image processing with high accuracy.

[0010] The present disclosure provides a system including a work machine and a work machine capable of easily acquiring an image clearly showing the work machine for work classification.

[0011] Means for solving problems

[0012] According to one embodiment of the present disclosure, a system including a work machine and a computer is provided. The work machine has a work machine body and a working device mounted on the work machine body. The computer infers the posture of the working device relative to the work machine body in a captured image showing the work machine, creates a three-dimensional model representing the three-dimensional shape of the work machine based on the posture of the working device, and creates image data including a two-dimensional image obtained by observing the three-dimensional model from a viewpoint position representing the position of the viewpoint where the three-dimensional model is hypothetically observed.

[0013] According to one embodiment of the present disclosure, a working machine is provided, the working machine comprising: a working machine body; a working device mounted on the working machine body; and a computer mounted on the working machine body. The computer infers the posture of the working device relative to the working machine body in a captured image showing the working machine, creates a three-dimensional model representing the three-dimensional shape of the working machine based on the posture of the working device, and creates image data including a two-dimensional image obtained by observing the three-dimensional model from a viewpoint position representing the position of the viewpoint where the three-dimensional model is virtually observed.

[0014] Effects of the Invention

[0015] According to the present disclosure, an image clearly showing a working machine can be easily acquired, and work can be classified with high accuracy using the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a side view showing a wheel loader as an example of a working machine.

[0017] Figure 2 It is a schematic block diagram showing the structure of the entire system including the wheel loader.

[0018] Figure 3 This is a schematic diagram for explaining the operation of the wheel loader during the excavation and loading operation.

[0019] Figure 4 This is a table showing a method of determining the motion during the excavation and loading work of the wheel loader.

[0020] Figure 5 It is a schematic diagram showing the configuration of a computer included in a system including a working machine.

[0021] Figure 6 1 is a block diagram showing the system configuration of a wheel loader before shipment.

[0022] Figure 7 Detailed description of the invention is a flowchart showing a method for producing a learned task classification inference model.

[0023] Figure 8 This is a first schematic diagram showing a process for learning a work classification inference model.

[0024] Fig. 9 This is a second schematic diagram showing the process for learning the work classification inference model.

[0025] Fig.10 1 is a block diagram showing a system configuration of a factory-shipped wheel loader.

[0026] Fig.11This is a flowchart showing the processing executed by the computer to estimate the job classification after shipping.

[0027] Fig.12 This is a block diagram showing the configuration of a system for estimating the posture of a working machine using a learned posture estimation model.

[0028] Fig.13 This is a schematic diagram showing a process of estimating the posture of the working device from a captured image using a posture estimation model that has been learned so as to obtain the posture of the working device from the captured image.

[0029] Fig.14 This is a block diagram showing a system configuration for learning a posture estimation model.

[0030] Fig.15 Detailed description of the invention is a flowchart showing a method for producing a learned posture estimation model.

[0031] Fig.16 This is a schematic diagram showing the process for learning the posture estimation model.

[0032] Fig.17 This is a schematic diagram showing a modification example related to the learning of the work classification inference model. DETAILED DESCRIPTION

[0033] Hereinafter, the implementation mode will be described based on the accompanying drawings. In the following description, the same reference numerals are used to mark the same components. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0034] <Overall Structure>

[0035] In the embodiment, a wheel loader 1 will be described as an example of a working machine. Figure 1 It is a side view of a wheel loader 1 which is an example of a working machine according to the embodiment.

[0036] like Figure 1 As shown, the wheel loader 1 includes a body frame 2, a working device 3, a traveling device 4, and a cab 5. The body frame 2, the cab 5, etc. constitute a body (working machine body) of the wheel loader 1. The working device 3 and the traveling device 4 are mounted on the body of the wheel loader 1.

[0037] The travel device 4 is a device for traveling the vehicle body of the wheel loader 1, and includes travel wheels 4a, 4b. The wheel loader 1 can travel automatically by driving the travel wheels 4a, 4b to rotate, and can perform desired work using the work implement 3.

[0038] The vehicle body frame 2 includes a front frame 2a and a rear frame 2b. The front frame 2a and the rear frame 2b are installed so as to be able to swing to the left and right directions relative to each other. A pair of steering cylinders 11 are installed so as to straddle the front frame 2a and the rear frame 2b. The steering cylinders 11 are hydraulic cylinders. The steering cylinders 11 are extended and retracted by the working oil from the steering pump (not shown), thereby changing the traveling direction of the wheel loader 1 to the left and right.

[0039] In this specification, the direction in which the wheel loader 1 travels straight is referred to as the front-rear direction of the wheel loader 1. In the front-rear direction of the wheel loader 1, the side on which the working device 3 is arranged relative to the vehicle body frame 2 is referred to as the front direction, and the side opposite to the front direction is referred to as the rear direction. The left-right direction of the wheel loader 1 refers to the direction perpendicular to the front-rear direction when viewed from above. The right side and the left side of the left-right direction when viewed from the front direction are the right direction and the left direction, respectively. The up-down direction of the wheel loader 1 refers to the direction perpendicular to the plane defined by the front-rear direction and the left-right direction. In the up-down direction, the side where the ground is located is the lower side, and the side where the sky is located is the upper side.

[0040] The front frame 2a is provided with a working device 3 and a pair of running wheels (front wheels) 4a. The working device 3 is arranged in front of the vehicle body. The working device 3 is pumped by a working device pump 25 (see Figure 2 ) is driven by the working oil of the engine 21. The working device pump 25 is a hydraulic pump driven by the engine 21 and operates the working device 3 by the discharged working oil. The working device 3 includes a boom 14 and a bucket 6 as a working tool. The bucket 6 is arranged at the front end of the working device 3. The bucket 6 is an example of an attachment that is detachably mounted on the front end of the boom 14. Depending on the type of work, the attachment can be replaced by a grab hook, a hook, or a plow claw, etc.

[0041] The base end of the boom 14 is rotatably attached to the front frame 2a via a boom pin 9. The bucket 6 is rotatably attached to the boom 14 via a bucket pin 17 located at the front end of the boom 14.

[0042] The front frame 2a and the boom 14 are connected by a pair of boom cylinders 16. The boom cylinders 16 are hydraulic cylinders. The base end of the boom cylinder 16 is mounted on the front frame 2a. The front end of the boom cylinder 16 is mounted on the boom 14. The boom cylinder 16 is connected to the front frame 2a by a pair of boom cylinders 16 ... Figure 2 ) is extended and retracted under the action of the working oil of the boom 14, thereby raising and lowering the boom 14. The boom cylinder 16 drives the boom 14 to rotate up and down around the boom pin 9.

[0043] The working device 3 further includes a bell crank 18, a bucket cylinder 19, and a connecting rod 15. The bell crank 18 is rotatably supported on the boom 14 by a supporting pin 18a located substantially in the center of the boom 14. The bucket cylinder 19 connects the bell crank 18 to the front frame 2a. The connecting rod 15 is connected to a connecting pin 18c provided at the front end of the bell crank 18. The connecting rod 15 connects the bell crank 18 to the bucket 6.

[0044] The bucket cylinder 19 is a hydraulic cylinder and a work tool cylinder. The base end of the bucket cylinder 19 is mounted on the front frame 2a. The front end of the bucket cylinder 19 is mounted on a connecting pin 18b provided at the base end of the bell crank 18. The bucket cylinder 19 is driven by a working device pump 25 (see Figure 2 ) is extended and retracted by the working oil of the bucket 6, thereby the bucket 6 rotates up and down. The bucket cylinder 19 drives the bucket 6 to rotate around the bucket pin 17 as the center.

[0045] The rear frame 2b is provided with a cab 5 and a pair of travel wheels (rear wheels) 4b. The cab 5 is arranged behind the boom 14. The cab 5 is mounted on the vehicle body frame 2. A seat for an operator to sit on, and an operating device described later are arranged in the cab 5.

[0046] A position detection sensor 64 is disposed on the ceiling of the cab 5. The position detection sensor 64 includes a GNSS antenna and a global coordinate calculator. The GNSS antenna is an antenna for a global navigation satellite system (RTK-GNSS (Real Time Kinematic-Global Navigation Satellite Systems)).

[0047] A camera 65 is also mounted on the roof of the cab 5. The camera 65 of the embodiment is a single-lens reflex camera. The camera 65 is arranged at the front end of the roof of the cab 5. The camera 65 takes a picture of the front of the cab 5. The camera 65 takes a picture of the working device 3. The image captured by the camera 65 includes at least a part of the working device 3.

[0048] An IMU (Inertial Measurement Unit) 66 is disposed in the cab 5. The IMU 66 detects the inclination of the vehicle body frame 2. The IMU 66 detects the inclination angle of the vehicle body frame 2 with respect to the front-rear direction and the left-right direction.

[0049] Figure 2 2 is a schematic block diagram showing the configuration of the entire system including the wheel loader 1 according to the embodiment. Figure 2 The overall system of the embodiment includes a wheel loader 1 and a second processing device 70 configured to be able to communicate with the wheel loader 1 through wireless communication or wired communication.

[0050] The wheel loader 1 includes an engine 20 , a power take-off unit 22 , a power transmission mechanism 23 , a cylinder driving unit 24 , a first angle detector 29 , a second angle detector 48 , a rotation mechanism 60 , and a first processing device 30 (controller).

[0051] The engine 20 is, for example, a diesel engine. The output of the engine 20 is controlled by adjusting the amount of fuel injected into the cylinder of the engine 20. The engine 20 is provided with a temperature sensor 31. The temperature sensor 31 outputs a detection signal indicating the temperature to the first processing device 30.

[0052] The power take-off unit 22 is a device that distributes the output of the engine 20 to the power transmission mechanism 23 and the cylinder drive unit 24. The power transmission mechanism 23 is a mechanism that transmits the driving force from the engine 20 to the front wheel 4a and the rear wheel 4b, for example, a transmission. The power transmission mechanism 23 changes the speed of the rotation of the input shaft 21 and outputs it to the output shaft 23a. A vehicle speed detection unit 27 for detecting the vehicle speed of the wheel loader 1 is installed on the output shaft 23a of the power transmission mechanism 23. The wheel loader 1 includes the vehicle speed detection unit 27.

[0053] The vehicle speed detection unit 27 is, for example, a vehicle speed sensor. The vehicle speed detection unit 27 detects the rotation speed of the output shaft 23a and thereby detects the rotation speed of the output shaft 23a. Figure 1 ) is used to detect the moving speed of the wheel loader 1. The vehicle speed detection unit 27 functions as a rotation sensor for detecting the rotation speed of the output shaft 23a. The vehicle speed detection unit 27 functions as a movement detector for detecting the movement based on the travel device 4. The vehicle speed detection unit 27 outputs a detection signal indicating the vehicle speed of the wheel loader 1 to the first processing device 30.

[0054] The cylinder driving unit 24 includes a work implement pump 25 and a control valve 26. The output of the engine 20 is transmitted to the work implement pump 25 via the power take-off unit 22. The hydraulic oil discharged from the work implement pump 25 is supplied to the boom cylinder 16 and the bucket cylinder 19 via the control valve 26.

[0055] The boom cylinder 16 is provided with first hydraulic pressure detectors 28a and 28b for detecting the hydraulic pressure in the oil chamber of the boom cylinder 16. The wheel loader 1 includes the first hydraulic pressure detectors 28a and 28b. The first hydraulic pressure detectors 28a and 28b include, for example, a pressure sensor 28a for detecting the head side pressure and a pressure sensor 28b for detecting the bottom side pressure.

[0056] The pressure sensor 28a is installed on the cover side of the boom cylinder 16. The pressure sensor 28a can detect the pressure of the working oil in the oil chamber on the cylinder cover side of the boom cylinder 16 (the cover side pressure). The pressure sensor 28a outputs a detection signal indicating the cover side pressure of the boom cylinder 16 to the first processing device 30. The pressure sensor 28b is installed on the bottom side of the boom cylinder 16. The pressure sensor 28b can detect the pressure of the working oil in the oil chamber on the cylinder bottom side of the boom cylinder 16 (the bottom side pressure). The pressure sensor 28b outputs a detection signal indicating the bottom side pressure of the boom cylinder 16 to the first processing device 30.

[0057] The first angle detector 29 is, for example, a potentiometer attached to the boom pin 9. The first angle detector 29 detects a boom angle indicating a raised angle (tilt angle) of the boom 14. The first angle detector 29 outputs a detection signal indicating the boom angle to the first processing device 30.

[0058] Specifically, if Figure 1 As shown, the boom reference line A is a straight line passing through the center of the boom pin 9 and the center of the bucket pin 17. The boom angle θ1 is an angle formed by a horizontal line H extending forward from the center of the boom pin 9 and the boom reference line A. The case where the boom reference line A is horizontal is defined as the boom angle θ1 = 0°. When the boom reference line A is located above the horizontal line H, the boom angle θ1 is set to positive. When the boom reference line A is located below the horizontal line H, the boom angle θ1 is set to negative.

[0059] The second angle detector 48 is, for example, a potentiometer attached to the support pin 18a. The second angle detector 48 detects a bucket angle indicating an inclination angle of the bucket 6 relative to the boom 14. The second angle detector 48 outputs a detection signal indicating the bucket angle to the first processing device 30.

[0060] Specifically, if Figure 1 As shown, bucket reference line B is a straight line passing through the center of bucket pin 17 and blade tip 6a of bucket 6. Bucket angle θ2 is an angle formed by boom reference line A and bucket reference line B. The case where blade tip 6a of bucket 6 is horizontal on the ground when bucket 6 is grounded is defined as bucket angle θ2 = 0°. When bucket 6 is moved in the excavation direction (upward), bucket angle θ2 is set to positive. When bucket 6 is moved in the unloading direction (downward), bucket angle θ2 is set to negative.

[0061] The second angle detector 48 can also detect the bucket angle θ2 by detecting the angle (bell crank angle) of the bell crank 18 relative to the boom 14. The bell crank angle is the angle formed by a straight line passing through the center of the support pin 18a and the center of the connecting pin 18b and the boom reference line A. The second angle detector 48 can also be a potentiometer or a proximity switch mounted on the bucket pin 17. Alternatively, the second angle detector 48 can also be a stroke sensor arranged on the bucket cylinder 19.

[0062] The rotating mechanism 60 connects the front frame 2a and the rear frame 2b so as to be rotatable. The rotation of the front frame 2a relative to the rear frame 2b is performed by extending and retracting the articulated cylinder connected between the front frame 2a and the rear frame 2b. In addition, by bending (articulating) the front frame 2a relative to the rear frame 2b, the turning radius of the wheel loader during turning can be further reduced, and trenching and slope pulling operations based on offset driving can be performed. An articulation angle sensor 61 is provided in the rotating mechanism 60. The articulation angle sensor 61 detects the articulation angle. The articulation angle sensor 61 outputs a detection signal indicating the articulation angle to the first processing device 30.

[0063] The position detection sensor 64 outputs a detection signal indicating the position of the wheel loader 1 to the first processing device 30. The imaging device 65 outputs an image captured by the imaging device 65 to the first processing device 30. The IMU 66 outputs a detection signal indicating the tilt angle of the wheel loader 1 to the first processing device 30.

[0064] like Figure 2 As shown, the wheel loader 1 includes operating devices operated by an operator in a cab 5 . The operating devices include a forward / reverse switching device 49 , an accelerator operating device 51 , a boom operating device 52 , a speed change operating device 53 , a bucket operating device 54 , and a brake operating device 58 .

[0065] The forward / reverse switching device 49 includes a forward / reverse switching operating member 49a and a forward / reverse switching detection sensor 49b. The forward / reverse switching operating member 49a is operated by the operator to indicate the switching of the vehicle between forward and reverse. The forward / reverse switching operating member 49a can be switched to each position of forward (F), neutral (N) and reverse (R). The forward / reverse switching detection sensor 49b detects the position of the forward / reverse switching operating member 49a. The forward / reverse switching detection sensor 49b outputs a detection signal (forward, neutral, reverse) of the forward / reverse instruction represented by the position of the forward / reverse switching operating member 49a to the first processing device 30. The forward / reverse switching device 49 includes an FNR switching lever capable of switching between forward (F), neutral (N) and reverse (R).

[0066] The throttle operation device 51 includes an throttle operation member 51a and an throttle operation detection unit 51b. The throttle operation member 51a is operated by an operator to set a target rotation speed of the engine 20. The throttle operation detection unit 51b detects the operation amount (throttle operation amount) of the throttle operation member 51a. The throttle operation detection unit 51b outputs a detection signal indicating the throttle operation amount to the first processing device 30.

[0067] The brake operating device 58 includes a brake operating member 58a and a brake operating detection unit 58b. The brake operating member 58a is operated by an operator to operate the deceleration force of the wheel loader 1. The brake operating detection unit 58b detects the operation amount (brake operating amount) of the brake operating member 58a. The brake operating detection unit 58b outputs a detection signal indicating the brake operating amount to the first processing device 30. As the brake operating amount, the pressure of the brake oil may also be used.

[0068] The boom operating device 52 includes a boom operating member 52a and a boom operation detection unit 52b. The boom operating member 52a is operated by an operator to cause the boom 14 to perform an ascending action or a descending action. The boom operation detection unit 52b detects the position of the boom operating member 52a. The boom operation detection unit 52b outputs a detection signal of an ascending command or a descending command of the boom 14 indicated by the position of the boom operating member 52a to the first processing device 30.

[0069] The speed shift operating device 53 includes a speed shift operating member 53a and a speed shift operation detection unit 53b. The speed shift operating member 53a is operated by an operator to control the speed shift from the input shaft 21 to the output shaft 23a of the power transmission mechanism 23. The speed shift operation detection unit 53b detects the position of the speed shift operating member 53a. The speed shift operation detection unit 53b outputs a speed shift detection instruction represented by the position of the speed shift operating member 53a to the first processing device 30.

[0070] The bucket operating device 54 includes a bucket operating member 54a and a bucket operation detection unit 54b. The bucket operating member 54a is operated by an operator to make the bucket 6 perform an excavation action or a discharge action. The bucket operation detection unit 54b detects the position of the bucket operating member 54a. The bucket operation detection unit 54b outputs a detection signal of an operation instruction in a backward tilt direction or a discharge direction of the bucket 6 indicated by the position of the bucket operating member 54a to the first processing device 30.

[0071] The hinge operation device 55 includes a hinge operation member 55a and a hinge operation detection unit 55b. The hinge operation member 55a is operated by an operator to bend (hinged) the front frame 2a relative to the rear frame 2b by means of the rotation mechanism 60. The hinge operation detection unit 55b detects the position of the hinge operation member 55a. The hinge operation detection unit 55b outputs a detection signal of a bending instruction relative to the left direction or a bending instruction relative to the right direction indicated by the position of the hinge operation member 55a to the first processing device 30.

[0072] The first processing device 30 is composed of a microcomputer including a storage device such as a RAM (Random Access Memory) and a ROM (Read Only Memory) and a computing device such as a CPU (Central Processing Unit). The first processing device 30 can be realized as a part of the function of a controller for controlling the operation of the engine 20, the working device 3 (the boom cylinder 16, the bucket cylinder 19, etc.), the power transmission mechanism 23, etc. of the wheel loader 1. The signal of the forward and reverse command detected by the forward and reverse switching device 49, the signal of the vehicle speed of the wheel loader 1 detected by the vehicle speed detection unit 27, the signal of the boom angle detected by the first angle detector 29, the signal of the cover side pressure of the boom cylinder 16 detected by the pressure sensor 28a, and the signal of the bottom side pressure of the boom cylinder 16 detected by the pressure sensor 28b are mainly input to the first processing device 30.

[0073] The wheel loader 1 further includes a display unit 40 and an output unit 45. The display unit 40 is a monitor that is disposed in the cab 5 and is visually recognized by the operator.

[0074] The output unit 45 outputs the working machine motion information including the motion information of the wheel loader 1 to a server (second processing device 70) provided outside the wheel loader 1. The output unit 45 may output the working machine motion information including the motion information of the wheel loader 1 at predetermined intervals, or may output the working machine motion information of a plurality of periods at a time. The output unit 45 may also have a communication function such as wireless communication, for example, and communicate with the second processing device 70. Alternatively, the output unit 45 may also be, for example, an interface of a portable storage device (memory card, etc.) accessible to the second processing device 70. The second processing device 70 has a display unit that functions as a monitor, and is capable of displaying motion images based on the working machine motion information output from the output unit 45. The second processing device 70 is provided at a position different from the wheel loader 1, and as an example, the motion image of the wheel loader 1 during operation can be recognized at a remote location using the display unit.

[0075] <Operation of Wheel Loader 1 and Classification of Work>

[0076] The wheel loader 1 of the present embodiment performs an excavation operation of excavating an excavation object 100 such as soil into the bucket 6 and a loading operation of loading the load (excavation object 100 ) in the bucket 6 into a transport machine such as a dump truck 110 .

[0077] Figure 3 The diagram is a schematic diagram for explaining the operation of the wheel loader 1 in the excavation and loading operation according to the embodiment. The wheel loader 1 repeatedly performs the following plurality of operations in sequence to excavate an excavation object 100 and load the excavation object 100 into a transport machine such as a dump truck 110.

[0078] like Figure 3 As shown in (A), the wheel loader 1 moves forward toward the excavation object 100. In this no-load forward movement, the operator operates the boom cylinder 16 and the bucket cylinder 19 to set the working device 3 to an excavation posture in which the front end of the boom 14 is located at a low position and the bucket 6 is facing horizontally, so that the wheel loader 1 moves forward toward the excavation object 100.

[0079] The operator advances the wheel loader 1 until Figure 3 As shown in (B), the blade tip 6a of the bucket 6 sinks into the excavation object 100. In this excavation (entry) step, the blade tip 6a of the bucket 6 sinks into the excavation object 100.

[0080] like Figure 3 As shown in (C), the operator then operates the boom cylinder 16 to raise the bucket 6, and operates the bucket cylinder 19 to tilt the bucket 6 backward. Through this excavation (digging) action, as shown by the curved arrow in the figure, the bucket 6 rises along the bucket trajectory L, and the excavation object 100 is excavated into the bucket 6. Thus, the excavation work of digging the excavation object 100 is performed.

[0081] Depending on the type of excavation object 100, the excavation operation may be completed by tilting the bucket 6 backward only once. Alternatively, during the excavation operation, the bucket 6 may be repeatedly tilted backward, neutralized, and tilted backward again.

[0082] In such Figure 3 After the bucket 6 has dug the excavation object 100 as shown in (D), the operator moves the wheel loader 1 backward in the load-carrying backward movement. The operator can move the boom up while moving backward, or can also move the boom up while moving backward. Figure 3 As shown in (E), the boom is raised while moving forward.

[0083] like Figure 3 As shown in (E), the operator moves the wheel loader 1 forward to approach the dump truck 110 while maintaining the bucket 6 raised or while raising the bucket 6. The bucket 6 is positioned substantially directly above the loading platform of the dump truck 110 by this forward movement of loading.

[0084] like Figure 3 As shown in (F), the operator unloads the bucket 6 at a predetermined position to load the cargo (excavation object) in the bucket 6 into the loading platform of the dump truck 110. This action is the so-called earth discharge action. After that, the operator lowers the boom 14 while moving the wheel loader 1 backward to return the bucket 6 to the excavation posture. This action is a backward and boom lowering action. The above are typical actions that form a cycle of excavation and loading operations.

[0085] Figure 4 is a table showing a method of determining the motion in the excavation and loading operation of the wheel loader 1. Figure 4 In the table shown, the top row of "Action" shows Figure 3 (A)~ Figure 3 The names of the actions shown in (F).

[0086] The rows of "forward / reverse switching lever", "working equipment operation" and "working equipment cylinder pressure" show the parameters for determining whether the current operation of the wheel loader 1 is Figure 3 (A)~ Figure 3 Which of the actions shown in (F) is performed by the first processing device 30 ( Figure 2 ) are used. Here, in this specification, the determination of which action the current action of the wheel loader 1 in the excavation and loading operation is is called the operation classification. The operation classification shows the content of the action of the wheel loader 1 in the excavation and loading operation.

[0087] More specifically, in the row of “forward / reverse switching lever”, the determination condition regarding the forward / reverse switching lever is indicated by a circle.

[0088] In the row of “Working Equipment Operation”, the operator's operation on the working equipment 3 is indicated by a circle. More specifically, the row of “Boom” shows determination conditions related to the operation on the boom 14 , and the row of “Bucket” shows determination conditions related to the operation on the bucket 6 .

[0089] The row of "Working Equipment Cylinder Pressure" shows the current hydraulic pressure of the hydraulic cylinder of the working equipment 3, for example, the judgment condition of the hydraulic pressure of the cylinder bottom chamber of the boom cylinder 16. Here, regarding the hydraulic pressure, four reference values ​​A, B, C, and P are pre-set, and multiple pressure ranges (range less than the reference value P, range from reference value A to C, range from reference value B to P, range less than the reference value C) are defined by the reference values ​​A, B, C, and P, and these pressure ranges are set as the above judgment conditions. The four reference values ​​A, B, C, and P are A>B>C>P.

[0090] The first processing device 30 can determine which action the wheel loader 1 is currently performing by using a combination of determination conditions of the “forward / reverse switching lever”, “boom”, “bucket”, and “work implement cylinder pressure” corresponding to each action.

[0091] The following instructions are carried out Figure 4 The specific operation of the first processing device 30 under the control shown in FIG. Figure 4 The combinations of the determination conditions of the "forward / reverse switching lever", "boom", "bucket" and "working equipment cylinder pressure" corresponding to each work process shown are stored in advance in the storage unit 30j ( Figure 2 ). The first processing device 30 grasps the currently selected forward / reverse switching lever (F, N, R) based on the signal from the forward / reverse switching device 49. The first processing device 30 grasps the type of current operation on the boom 14 (lowering, neutral, or raising) based on the signal from the boom operation detection unit 52b. The first processing device 30 grasps the type of current operation on the bucket 6 (unloading, neutral, or backward tilting) based on the signal from the bucket operation detection unit 54b. Furthermore, the first processing device 30 grasps the type of current operation on the bucket 6 (unloading, neutral, or backward tilting) based on the signal from the bucket operation detection unit 54b. Figure 2 The signal of the pressure sensor 28 b shown grasps the current hydraulic pressure in the cylinder bottom chamber of the boom cylinder 16 .

[0092] The first processing device 30 compares the current combination of the forward / reverse switching lever, boom operation type, bucket operation type, and lift cylinder hydraulic pressure (i.e., the current working state) with the combination of the determination conditions of "forward / reverse switching lever", "boom", "bucket", and "working equipment cylinder pressure" corresponding to each working process stored in advance. As a result of the comparison process, the first processing device 30 determines which action corresponds to the combination of determination conditions that best matches the current working state.

[0093] Here, corresponding to Figure 4 The combination of determination conditions for each operation of the excavation and loading work shown will be described below as an example.

[0094] In the no-load forward motion, the forward / reverse switching lever is F, the boom operation and the bucket operation are both in neutral, and the working device cylinder pressure is less than the reference value P. In the excavation (reaching) motion, the forward / reverse switching lever is F, the boom operation and the bucket operation are both in neutral, and the working device cylinder pressure is in the range of reference values ​​A to C. In the excavation (digging) motion, the forward / reverse switching lever is F or R, the boom operation is raised or in neutral, the bucket operation is tilted backward, and the working device cylinder pressure is in the range of reference values ​​A to C. Regarding the bucket operation, a judgment condition of alternating backward tilt and neutral may be further added. This is because, depending on the state of the excavated object, there is a case where the bucket 6 is repeatedly tilted backward, neutralized, and tilted backward again.

[0095] In the cargo-carrying reverse action, the forward-reverse switching lever is R, the boom is in neutral or raised, the bucket is in neutral, and the working device cylinder pressure is in the range of reference values ​​B to P. In the cargo-carrying forward action, the forward-reverse switching lever is F, the boom is in raised or neutral, the bucket is in neutral, and the working device cylinder pressure is in the range of reference values ​​B to P. In the soil discharge action, the forward-reverse switching lever is F, the boom is in raised or neutral, the bucket is in unloading, and the working device cylinder pressure is in the range of reference values ​​B to P.

[0096] In the reverse and boom lowering operation, the forward / reverse switching lever is in the R position, the boom is lowered, the bucket is tilted backward, and the working equipment cylinder pressure is less than the reference value P.

[0097] exist Figure 4 1 also shows a simple travel action in which the wheel loader 1 is simply traveling. In the simple travel action, the operator moves the wheel loader 1 forward with the boom 14 in a low position. There are cases where the bucket 6 is loaded with cargo and the vehicle is transporting cargo, and there are cases where the bucket 6 is not loaded with cargo and the vehicle is traveling. In the simple travel action, the forward / reverse switching lever is at F, the boom operation and the bucket operation are both in neutral, and the working device cylinder pressure is less than the reference value C.

[0098] The information on the operation of the wheel loader 1 determined by the first processing device 30 is output to the second processing device 70 via the output unit 45 as a part of the working machine operation information.

[0099] <Detailed Configuration of Computer 102A>

[0100] Figure 5 1 is a schematic diagram showing the structure of a computer 102A included in a system including a working machine. The system of the embodiment is for obtaining an image clearly showing a working machine and using the image to compare with a reference Figure 4 The system performs work classification in the excavation and loading work performed by the wheel loader 1 regardless of the table described above. Figure 5 The computer 102A shown is composed of Figure 2 The computer 102A may be a computer designed specifically for the system of the embodiment, or may be a general-purpose PC (Personal Computer).

[0101] The computer 102A includes a processor 103, a storage device 104, a communication interface 105, and an I / O interface 106. The processor 103 is, for example, a CPU.

[0102] The storage device 104 includes a medium that stores information such as stored programs and data in a manner that the processor 103 can read. The storage device 104 includes a system memory such as RAM (Random Access Memory) or ROM (Read Only Memory), and an auxiliary storage device. The auxiliary storage device can be, for example, a magnetic recording medium such as a hard disk, an optical recording medium such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a semiconductor memory such as a flash memory. The storage device 104 can be built into the computer 102A. The storage device 104 can also include an external recording medium 109 connected to the computer 102A in a removable manner. The external recording medium 109 can be a CD-ROM.

[0103] The communication interface 105 is, for example, a wired LAN (Local Area Network) module or a wireless LAN module, and is an interface for communicating via a communication network. The I / O interface 106 is, for example, a USB (Universal Serial Bus) port, and is an interface for connecting to an external device.

[0104] The computer 102A is connected to an input device 107 and an output device 108 via an I / O interface 106. The input device 107 is a device for a user to input data to the computer 102A. The input device 107 includes, for example, a pointing device such as a mouse or a trackball. The input device 107 may include a device for text input such as a keyboard. The output device 108 includes, for example, a display (display 40, Figure 2 ).

[0105] Figure 6 1 is a block diagram showing the system configuration of the wheel loader 1 before shipment. Figure 5 The processor 103 and the storage device 104 shown constitute Figure 4 The processor 103 includes an operation data generating unit 161 .

[0106] The operation data generating unit 161 receives an input of a detection signal when the pressure of the working oil in the oil chamber of the boom cylinder 16 is detected from the first hydraulic pressure detectors 28a and 28b. The operation data generating unit 161 receives an input of a detection signal when the accelerator operation amount is detected from the accelerator operation detecting unit 51b. The operation data generating unit 161 receives an input of a detection signal when the vehicle speed detecting unit 27 detects the vehicle speed of the wheel loader 1. Alternatively, the vehicle speed detecting unit 27 may output a detection signal when the rotation speed of the output shaft 23a is detected to the operation data generating unit 161, and the operation data generating unit 161 calculates the vehicle speed of the wheel loader 1 based on the detection signal.

[0107] The processor 103 includes a timer 162. The operation data generating unit 161 reads the current time from the timer 162, and calculates the elapsed time from the start of the excavation work when the wheel loader 1 is performing the excavation work.

[0108] The start of the excavation operation, i.e., the transition of the movement of the wheel loader 1 from the no-load forward movement to the excavation (reaching) movement, is determined by detecting the increase in the hydraulic pressure in the oil chamber of the boom cylinder 16 when the blade tip 6a of the bucket 6 reaches the excavation object 100 and the load of the excavation object 100 begins to act on the bucket 6, and confirming whether the working device 3 is in a posture to start the excavation operation based on the boom angle θ1 and the bucket angle θ2. The operation start time point can also be determined based on the load borne by the boom cylinder 16 during the operation. The start of the operation can also be determined based on the shooting data obtained by the camera 65 shooting the surroundings of the wheel loader 1.

[0109] The hydraulic pressure of the boom cylinder 16, the accelerator operation amount, the vehicle speed, and the elapsed time from the start of the excavation work are included in the operation data related to the motion of the wheel loader 1. The operation data includes data related to the travel of the wheel loader 1, such as the accelerator operation amount and the vehicle speed.

[0110] The processor 103 has a posture data generating unit 163. The posture data generating unit 163 receives an input of a detection signal when the boom angle θ1 is detected from the first angle detector 29. The posture data generating unit 163 receives an input of a detection signal when the bucket angle θ2 is detected from the second angle detector 48. The boom angle θ1 and the bucket angle θ2 constitute posture data representing the posture of the working device 3 relative to the working machine body (vehicle body).

[0111] The processor 103 has a work classification determination unit 164. The work classification determination unit 164 receives an input of a detection signal of a forward / reverse instruction from the forward / reverse switching detection sensor 49b. The work classification determination unit 164 receives an input of a detection signal of a lifting instruction or a lowering instruction of the boom 14 from the boom operation detection unit 52b. The work classification determination unit 164 receives an input of a detection signal of an operation instruction of the bucket 6 in a backward tilting direction or a unloading direction from the bucket operation detection unit 54b. The work classification determination unit 164 also receives an input of a detection signal when the hydraulic pressure of the cylinder bottom chamber of the boom cylinder 16 is detected from the first hydraulic pressure detector 28b.

[0112] The work classification determination unit 164 refers to the input detection signals. Figure 4 The table shown in FIG. 1 is used to determine which operation the current operation of the wheel loader 1 is (ie, to classify the operation).

[0113] The processor 103 includes an action state image generating unit 165. The action state image generating unit 165 generates action state image data based on the posture data generated by the posture data generating unit 163. The action state image data includes three-dimensional model shape data showing the three-dimensional shape of the wheel loader 1 in operation. The three-dimensional model shape data includes data of the working device 3, the vehicle body, the traveling wheels 4a, 4b, etc. constituting the wheel loader 1. The action state image data also includes operation data generated by the operation data generating unit 161.

[0114] The processor 103 has a specific viewpoint image generation unit 166. The specific viewpoint image generation unit 166 generates a two-dimensional image obtained by observing the three-dimensional model from a specific viewpoint position based on the three-dimensional model generated by the action state image generation unit 165 and the viewpoint position representing the position of the viewpoint where the three-dimensional model is virtually observed. The viewpoint position can be set to an arbitrary position. By adjusting the viewpoint position, it is possible to generate and display a two-dimensional image obtained by observing the three-dimensional model from an arbitrary direction. The specific viewpoint image generation unit 166 creates image data including the two-dimensional image. The image data also includes the operation data generated by the operation data generation unit 161.

[0115] The processor 103 includes a work classification estimation unit 167. The storage device 104 stores a work classification estimation model 180.

[0116] The work classification inference model 180 is an artificial intelligence model for inferring which action the current action of the wheel loader 1 in a series of excavation and loading operations is. The work classification inference model 180 is configured to infer which action the current action of the wheel loader 1 is based on the image data generated by the specific viewpoint image generation unit 166. The computer 102A infers the action of the wheel loader 1 in the excavation and loading operation by using the artificial intelligence work classification inference model 180. The work classification inference unit 167 uses the work classification inference model 180 to output the inferred work classification when the work classification is inferred based on the two-dimensional image obtained by observing the three-dimensional model of the wheel loader 1 from a specific viewpoint position.

[0117] More specifically, the work classification inference unit 167 reads the work classification inference model 180 from the storage device 104. The work classification inference unit 167 inputs the image data including the two-dimensional image generated by the specific viewpoint image generation unit 166 and the operation data generated by the operation data generation unit 161 into the work classification inference model 180, thereby outputting the inference result of the work classification. By inputting the operation data into the work classification inference model 180 in addition to the image obtained by converting the three-dimensional model of the wheel loader 1 into two dimensions, the accuracy of the inference of the work classification is improved.

[0118] The work classification inference model 180 includes a neural network. The work classification inference model 180 includes, for example, a deep neural network such as a convolutional neural network (CNN).

[0119] The model in the embodiment may be installed in hardware, software that can be executed on hardware, firmware, or a combination thereof. The model may include a program, an algorithm, and data executed by the processor 103. The functions of the model may be executed by a single module, or may be distributed and executed in multiple modules. The model may also be distributed and configured in multiple computers.

[0120] The processor 103 includes a determination unit 168. The determination unit 168 compares the inference result of the job classification inferred by the job classification inference unit 167 with the result of the job classification performed by the job classification determination unit 164. The determination unit 168 determines whether the inferred job classification output by the job classification inference unit 167 is consistent with the result of the job classification performed by the job classification determination unit 164.

[0121] The processor 103 includes an adjustment unit 169. The adjustment unit 169 updates the work classification estimation model 180 based on the comparison result between the estimated work classification determined by the determination unit 168 and the work classification. In this way, the work classification estimation model 180 is learned. The work classification estimation model 180 is learned by the factory before the wheel loader 1 is shipped.

[0122] <Method for Producing the Learning-Completed Task Classification Inference Model 180>

[0123] Figure 7 1 is a flowchart showing a method for producing the learned work classification inference model 180 . Figure 8 as well as Fig. 9 1 is a schematic diagram showing a process for learning the work classification inference model 180. Figure 6 Some of the contents described above are repeated, but the following references Figures 7 to 9 The process for learning the work classification estimation model 180 for estimating the work classification will be described.

[0124] like Figure 7 As shown, first, in step S101, operation data 201 is generated. In step S102, gesture data 203 is generated.

[0125] More specifically, the computer 102A calculates the time elapsed from the start of the excavation operation at a certain time during the excavation operation by the operation data generation unit 161. The operation data generation unit 161 generates the operation data 201 ( Figure 8). More specifically, the computer 102A generates posture data 203 by the posture data generation unit 163 detecting the boom angle θ1 and the bucket angle θ2 at that time based on the detection results of the first angle detector 29 and the second angle detector 48 ( Figure 8 ).

[0126] Next, in step S103, the operation of the wheel loader 1 is determined. More specifically, the computer 102A determines the operation of the wheel loader 1 based on the detection results of various sensors including the forward / reverse switching detection sensor 49b, the boom operation detection unit 52b, the bucket operation detection unit 54b, and the first hydraulic pressure detector 28b, and with reference to the detection results of the various sensors including the forward / reverse switching detection sensor 49b, the boom operation detection unit 52b, the bucket operation detection unit 54b, and the first hydraulic pressure detector 28b. Figure 4 The table shown in FIG. 1 is used to identify the current operation of the wheel loader 1, that is, to classify the operation, and to generate the operation classification result 204 ( Figure 8 ).

[0127] Next, in step S104 , an operation state image is generated. More specifically, the computer 102A and the operation state image generation unit 165 generate a three-dimensional model representing the three-dimensional shape of the wheel loader 1 based on the posture data 203 . Figure 8 The operation state image 205 shown includes the wheel loader 1 in operation. The operation state image 205 displays the working device 3, the vehicle body, the travel wheels 4a, 4b, etc. constituting the wheel loader 1. The operation state image 205 also displays the operation data generated by the operation data generation unit 161. The operation state image 205 displays the elapsed time, the boom cylinder pressure, the accelerator operation amount, and the vehicle speed.

[0128] Next, in step S105, a specific viewpoint image is generated. More specifically, the computer 102A generates a two-dimensional image of the three-dimensional model observed from a specific viewpoint position based on the action state image 205 generated in step S104 and the viewpoint position representing the position of the viewpoint of the three-dimensional model contained in the action state image 205. The two-dimensional image can also be referred to as a virtual captured image obtained by capturing the three-dimensional model from the viewpoint position through a virtual camera. The specific viewpoint image 206 ( Figure 8 , 9 ) includes the two-dimensional image. The specific viewpoint image 206 also displays operation data, specifically, elapsed time, boom cylinder pressure, accelerator operation amount, and vehicle speed.

[0129] Figure 8 The specific viewpoint image 206 shown includes the wheel loader 1 when viewed from the left side. The viewpoint position in this case is a position on the left side of the wheel loader 1. Fig. 9The specific viewpoint image 206 shown includes the wheel loader 1 when viewed from the right side. The viewpoint position in this case is the right side position of the wheel loader 1. The viewpoint position can be set to any position. By changing the viewpoint position, a plurality of specific viewpoint images 206 are generated from one action state image 205. A plurality of specific viewpoint images 206 can be generated by photographing one action state image 205 from any viewpoint position.

[0130] Figure 8 The illustrated learning data 188A includes a specific viewpoint image 206 and a task classification result 204 . Fig. 9 The learning data 188B shown also includes a specific viewpoint image 206 and a work classification result 204. The work classification result 204 adds annotations to the specific viewpoint image 206. The work classification result 204 adds annotations to the action state image 205 and the viewpoint position, which are metadata for creating the specific viewpoint image 206. The work classification result 204 adds annotations to the operation data and posture data 203, which are metadata for creating the action state image 205.

[0131] In the learning data 188A and 188B, the work classification results 204 are the same because they are generated for the wheel loader 1 at the same time, but the specific viewpoint images 206 are different. By changing the viewpoint position, a plurality of learning data 188A and 188B are generated from one action state image 205. Thus, the amount of learning data used to learn the work classification inference model 180 increases.

[0132] The processing of steps S101 to S105 does not necessarily need to be performed in this order. The processing of step S101 or S102 and the processing of S103 may be performed simultaneously, or the processing of step S103 may be performed after the processing of steps S101, S102, S104 and S105 are performed in sequence.

[0133] Next, in step S106, the work classification is estimated. More specifically, the computer 102A reads the work classification estimation unit 167 from the storage device 104. The work classification estimation model 180 includes Figure 8 , 9 The neural network shown in FIG. 1 includes an input layer 181, an intermediate layer (hidden layer) 182, and an output layer 183. The intermediate layer 182 is multi-layered. The input layer 181, the intermediate layer 182, and the output layer 183 have one or more units (neurons). The number of units in the input layer 181, the intermediate layer 182, and the output layer 183 can be appropriately set.

[0134] The units of adjacent layers are combined with each other, and a weight is set for each combination. A bias is set for each unit. A threshold is set for each unit. The output value of each unit is determined by whether the value obtained by adding the bias to the sum of the product of the input value to each unit and the weight exceeds the threshold.

[0135] The work classification inference model 180 is learned as follows: an inferred work classification inferred from a two-dimensional image obtained by observing a three-dimensional model representing the three-dimensional shape of the wheel loader 1 in operation from a specific viewpoint position and a specific viewpoint image 206 including operation data related to the movement of the wheel loader 1 is output. The parameters of the work classification inference model 180 adjusted by learning are stored in the storage device 104. The parameters of the work classification inference model 180 include, for example, the number of layers of the neural network, the number of units in each layer, the connection relationship between units, the weight of the connection between each unit, the bias associated with each unit, and the threshold value of each unit.

[0136] The work classification inference unit 167 inputs the specific viewpoint image 206 including the two-dimensional image and the operation data generated by the specific viewpoint image generation unit 166 to the input layer 181. An output value indicating the inferred work classification is output from the output layer 183. For example, the computer 102A uses the specific viewpoint image 206 as an input to the input layer 181 and performs forward propagation calculation processing of the neural network of the work classification inference model 180. As a result, the computer 102A obtains the inferred work classification in which the work classification is inferred as the output value output from the output layer 183 of the neural network.

[0137] Next, in step S107, the inferred job classification is determined. More specifically, the computer 102A determines whether the inferred job classification is consistent with the job classification result 204 by comparing the inferred job classification output from the output layer 183 of the job classification inference model 180 with the job classification result 204 included in the learning data 188A and 188B.

[0138] The computer 102A uses the specific viewpoint image 206 as input data and the task classification result 204 at that time as teacher data to learn the task classification inference model 180. The computer 102A calculates the weight of the combination between each unit, the bias of each unit, and the error of each threshold of each unit through back propagation based on the result of the judgment obtained by comparing the inferred task classification with the task classification result 204.

[0139] Next, in step S108, the parameters of the job classification inference model 180 are adjusted. More specifically, the adjustment unit 169 of the computer 102A adjusts the parameters of the job classification inference model 180, such as the weight of the combination between each unit, the bias of each unit, and the threshold of each unit, based on the result of the judgment obtained by the judgment unit 168 comparing the inferred job classification with the job classification result 204. In this way, the job classification inference model 180 is updated. And, if the same specific viewpoint image 206 is input to the input layer 181, the probability of outputting the inferred job classification consistent with the teacher data, i.e., the job classification result 204, is increased. The parameters of the updated job classification inference model 180 are stored in the storage device 104.

[0140] When the next time the work classification is estimated, the specific viewpoint image 206 is input to the updated work classification estimation model 180, and the output of the estimated work classification is obtained. The computer 102A repeatedly performs the processing of steps S101 to S108 until the estimated work classification output by the work classification estimation model 180 is consistent with the work classification result 204 at the time when the operation data and posture data 203 based on the specific viewpoint image 206 are obtained. In this way, the parameters of the work classification estimation model 180 are optimized, and the learning of the work classification estimation model 180 is performed.

[0141] When the work classification inference model 180 has accumulated enough learning to obtain an inferred work classification with sufficient accuracy, the computer 102A ends the learning of the work classification inference model 180. In this way, the learned work classification inference model 180 is created. Then, the process ends (end).

[0142] It should be noted that the initial values ​​of various parameters of the work classification inference model 180 may also be assigned by a template. Alternatively, the initial values ​​of the parameters may also be assigned manually by human input. When relearning the work classification inference model 180, the computer 102A may also prepare the initial values ​​of the parameters based on the values ​​stored in the storage device 104 as the parameters of the work classification inference model 180 to be relearned.

[0143] <Estimation of Job Classification Using the Learned Job Classification Estimation Model 180>

[0144] Fig.10 1 is a block diagram showing the system configuration of a factory-shipped wheel loader 1. Figure 6 The computer 102A shown in the figure includes a computer 102B. The computer 102B includes a processor 103 and a storage device 104.

[0145] The processor 103 has Figure 6The same operation data generation unit 161, timer 162, action state image generation unit 165, specific viewpoint image generation unit 166 and work classification estimation unit 167 are also provided. The processor 103 further includes an image processing unit 171. The processor 103 does not include Figure 6 The work classification determination unit 164, the determination unit 168 and the adjustment unit 169 are shown. The storage device 104 has a work classification estimation model 180 that has been learned.

[0146] Fig.11 1 is a flowchart showing the processing performed by the computer 102B to estimate the job classification after shipment. Fig.10 , 11 Next, a description will be given of a process of estimating the work classification in the excavation work of the wheel loader 1 after shipment.

[0147] First, in step S201, operation data 201 is generated. More specifically, the computer 102B calculates the time elapsed from the start of the excavation operation at a certain time during the excavation operation by the operation data generation unit 161. The operation data generation unit 161 generates the operation data at that time based on the detection results of various sensors including the first hydraulic pressure detectors 28a and 28b, the accelerator operation detection unit 51b, and the vehicle speed detection unit 27.

[0148] Next, in step S202, a captured image is acquired. More specifically, the computer 102B acquires a captured image captured by the imaging device (camera) 65 from the imaging device 65 by the image processing unit 171. The wheel loader 1 is displayed in the captured image. Typically, at least a portion of the working device 3 is displayed in the captured image.

[0149] Next, in step S203 , posture data is generated. More specifically, the computer 102B outputs posture data, specifically, the boom angle θ1 and the bucket angle θ2 , based on the image captured by the imaging device 65 , through the posture data generating unit 163 .

[0150] The posture data generating unit 163 may generate the posture data by obtaining the positions of the feature points set in the working device 3 within the captured image.

[0151] Alternatively, the posture data generating unit 163 may also generate posture data by using a posture inference model learned through artificial intelligence. Fig.12 This is a block diagram showing the configuration of a system for estimating the posture of the working device 3 using the learned posture estimation model 280 . Fig.13 This is a schematic diagram showing a process of estimating the posture of the working device 3 from the captured image 271 using the posture estimation model 280 that has been learned so as to obtain the posture of the working device 3 from the captured image 271 .

[0152] The posture estimation model 280 is an artificial intelligence model for obtaining the posture of the working device 3 relative to the vehicle body. The posture estimation model 280 is stored in the storage device 104. The posture estimation model 280 includes Fig.13 The posture inference model 280 is, for example, a deep neural network such as a convolutional neural network (CNN).

[0153] The neural network includes an input layer 281, an intermediate layer (hidden layer) 282, and an output layer 283. The intermediate layer 282 is multi-layered. The input layer 281, the intermediate layer 282, and the output layer 283 have one or more units (neurons). The number of units in the input layer 281, the intermediate layer 282, and the output layer 283 can be set appropriately.

[0154] The units of adjacent layers are combined with each other, and a weight is set for each combination. A bias is set for each unit. A threshold is set for each unit. The output value of each unit is determined by whether the value obtained by adding the bias to the sum of the product of the input value to each unit and the weight exceeds the threshold.

[0155] The posture estimation model 280 that has been learned is learned in a manner that determines the posture of the working device 3 based on the captured image 271. The parameters of the posture estimation model 280 obtained by learning are stored in the storage device 104. The parameters of the posture estimation model 280 include, for example, the number of layers of the neural network, the number of units in each layer, the connection relationship between the units, the weight of the connection between the units, the bias associated with each unit, and the threshold of each unit.

[0156] The posture data generation unit 163 reads the posture estimation model 280 and the optimal values ​​of the learned parameters from the storage device 104, thereby obtaining the learned posture estimation model 280. The posture data generation unit 163 calculates the captured image 271 captured by the imaging device 65 and obtained by the image processing unit 171 as input data to the posture estimation model 280. The posture data generation unit 163 inputs the captured image 271 to each unit included in the input layer 281 of the learned posture estimation model 280. The output layer 283 of the learned posture estimation model 280 outputs the estimated posture 277 in which the posture of the working device 3 relative to the working machine body is estimated. The estimated posture 277 specifically includes the boom angle θ1 and the bucket angle θ2.

[0157] In this way, the posture data generation unit 163 can generate the estimated posture 277 in which the posture of the work machine 3 is estimated from the captured image 271 using the learned posture estimation model 280 .

[0158] The learning of the posture estimation model 280 is described below. Fig.142 is a block diagram showing a system configuration for learning the posture estimation model 280 . Fig.15 2 is a flowchart showing a method for producing the posture estimation model 280 after learning. Fig.16 It is a schematic diagram showing the process for learning the posture estimation model 280.

[0159] Fig.14 The system shown in FIG. 1 includes a camera 65 and an image processing unit 171. Fig.15 In step S301 shown, the image processing unit 171 receives input of a captured image captured by the imaging device 65 from the imaging device (camera) 65. The image processing unit 171 performs image processing on the input captured image.

[0160] Fig.14 The system shown also has a first angle detector 29, a second angle detector 48, and an angle conversion unit 262. Fig.15 In step S302 shown, the angle conversion unit 262 receives an input of a detection signal when the boom angle θ1 is detected from the first angle detector 29. The angle conversion unit 262 receives an input of a detection signal when the bucket angle θ2 is detected from the second angle detector 48. The boom angle θ1 and the bucket angle θ2 constitute posture data representing the posture of the working device 3 relative to the working machine body (vehicle body).

[0161] The angle conversion unit 262 obtains the boom angle θ1 and bucket angle θ2 detected at the time when the image is captured by the image capture device 65 in association with the captured image. The image captured by the image capture device 65 including the work device 3 and the boom angle θ1 and bucket angle θ2 at the time when the image is captured constitute learning data for learning the posture estimation model. Fig.16 The learning data set 270 shown includes a plurality of learning data 270A, 270B, 270C, 270D, ... obtained by annotating a captured image with posture data (boom angle θ1 and bucket angle θ2) of the posture of the work machine 3 detected at the time when the captured image was captured.

[0162] The learning dataset 270 may also include Fig.16 The learning data set 270 may include a plurality of learning data including images of the working device 3 in the same posture taken in different environments such as daytime, backlighting, and night.

[0163] The posture data generating unit 163 includes a work machine posture estimating unit 265 , an error detecting unit 266 , and a posture estimation model updating unit 267 .

[0164] exist Fig.15 In step S303 shown, the work device posture estimation unit 265 uses the posture estimation model 280 to obtain an estimated posture of the work device 3 estimated from the captured image. More specifically, the work device posture estimation unit 265 reads the posture estimation model 280 from the storage device 104 and inputs the captured image to the posture estimation model 280. The work device posture estimation unit 265 uses the captured image as an input to the input layer 281 and performs forward propagation calculation processing of the neural network of the posture estimation model 280. As a result, the work device posture estimation unit 265 obtains the estimation results of the boom angle θ1 and the bucket angle θ2 as output values ​​output from the output layer 283 of the neural network.

[0165] exist Fig.15 In step S304 shown, the error detection unit 266 compares the estimation results of the boom angle θ1 and the bucket angle θ2 estimated by the work machine posture estimation unit 265 with the detection results of the boom angle θ1 and the bucket angle θ2 obtained by the angle conversion unit 262. The error detection unit 266 calculates the error of the estimation results of the boom angle θ1 and the bucket angle θ2 with respect to the true value.

[0166] The posture data generation unit 163 uses the captured image as input data and the posture data of the posture of the working device 3 detected at the time when the captured image is captured as teacher data to learn the posture estimation model 280. The error detection unit 266 calculates the weight of the combination between each unit, the bias of each unit, and the error of each threshold of each unit through back propagation based on the error of the calculated output value.

[0167] exist Fig.15 In step S305 shown, the posture estimation model updating unit 267 updates the posture estimation model 280 based on the errors of the boom angle θ1 and the bucket angle θ2 calculated by the error detection unit 266. The posture estimation model updating unit 267 updates the parameters of the posture estimation model 280 such as the weight of the combination between each unit, the bias of each unit, and the threshold of each unit. In addition, if the same captured image is input to the input layer 281, an output value closer to the true value can be output from the output layer 283. The updated parameters of the posture estimation model 280 are stored in the storage device 104.

[0168] When estimating the posture of the working device 3 next time, the captured image is input to the updated posture estimation model 280, and the estimation result of the posture of the working device 3 is output. The posture data generation unit 163 repeatedly performs the processing from step S301 to step S305 until the estimation result of the posture of the working device 3 output by the posture estimation model 280 is consistent with the posture data of the posture of the detected working device 3. In this way, the parameters of the posture estimation model 280 are optimized, and the posture estimation model 280 is learned.

[0169] When the posture estimation model 280 has accumulated enough learning and outputted an estimation result with sufficiently high accuracy, the posture data generation unit 163 ends the learning of the posture estimation model 280. In this way, the posture estimation model 280 that has completed learning is created. Next, the processing ends ( Fig.15 in the “end” section).

[0170] return Fig.10 , 11 Next, in step S204, an action state image is generated. More specifically, the computer 102B generates a three-dimensional model representing the three-dimensional shape of the wheel loader 1 based on the posture data generated in step S203. The action state image generation unit 165 generates an action state image including the three-dimensional model and the operation data generated in step S201.

[0171] Next, in step S205, a specific viewpoint image is generated. More specifically, the computer 102B generates a two-dimensional image of the three-dimensional model observed from a specific viewpoint position based on the action state image generated in step S204 and the viewpoint position indicating the position of the viewpoint at which the three-dimensional model is virtually observed. The specific viewpoint image generation unit 166 generates a specific viewpoint image including the two-dimensional image and the operation data.

[0172] Next, in step S206 , the work classification is estimated. More specifically, the computer 102B causes the work classification estimation unit 167 to read the work classification estimation model 180 and the learned optimal values ​​of the parameters from the storage device 104 , thereby acquiring the learned work classification estimation model 180 .

[0173] The work classification inference unit 167 uses the specific viewpoint image generated in step S205 as input data to the work classification inference model 180. The work classification inference unit 167 inputs the specific viewpoint image to each unit included in the input layer 181 of the learned work classification inference model 180. The output layer 183 of the learned work classification inference model 180 outputs the inferred work classification 177 ( Fig.10 ).

[0174] Finally, in step S207, the computer 102B generates management data including the job classification. The computer 102B records the management data in the storage device 104. Then, the processing ends ( Fig.11 in the “end” section).

[0175] As described above, in the system of the embodiment, Fig.10 , 11 As shown, the computer 102B estimates the posture of the working device 3 relative to the working machine body in the captured image of the wheel loader 1. The computer 102B creates a three-dimensional model representing the three-dimensional shape of the wheel loader 1 based on the posture of the working device 3. The computer 102B creates image data including a two-dimensional image obtained by observing the three-dimensional model from a viewpoint position representing the position of the viewpoint where the three-dimensional model is virtually observed.

[0176] In this way, the wheel loader 1 is clearly displayed in the two-dimensional image obtained by observing the three-dimensional model from a specific direction, and it is easy to obtain a two-dimensional image that clearly displays the wheel loader 1. By setting the viewpoint position such as the front side position of the wheel loader 1 where the entire working device 3 can be observed to create a two-dimensional image, and using the two-dimensional image to perform work classification, it is possible to reduce external interference caused by the influence of obstacles and the like during the movement of the wheel loader 1 during work. Therefore, it is possible to perform work classification with high accuracy.

[0177] like Figure 1 As shown, the camera 65 is mounted on the cab 5, and the captured image used for estimating the posture of the working device 3 is captured by the camera 65. By capturing the working device 3 using the camera 65, the computer 102B classifies the work based on the two-dimensional image created based on the captured image, thereby realizing the wheel loader 1 that can classify the work based on only the structure mounted on the vehicle body.

[0178] like Figure 8 , 9 As shown, the above-mentioned image data includes operation data related to the operation of the wheel loader 1. By performing work classification using the image data including the operation data, the accuracy of work classification can be further improved.

[0179] like Figure 8 , 9 As shown, the operation data includes data related to the travel of the wheel loader 1. Since the operation data unique to the wheel loader 1 that travels and performs work is included in the image for classifying work, the work unique to the wheel loader 1 can be classified with high accuracy.

[0180] like Fig.13 As shown, the posture inference model 280 that has been learned is used to obtain an inferred posture 277 that infers the posture of the working device 3 based on the captured image 271. By using the posture inference model 280 of artificial intelligence that is suitable for inferring the posture of the working device 3, the posture of the working device 3 can be easily and accurately inferred. The captured image 271 can also be obtained by capturing the image 271 by the camera 65. In this case, the posture of the working device 3 can be inferred only based on the structure mounted on the vehicle body. In this way, the posture of the working device 3 can be inferred more closely to real time, and the inference result can be displayed on the display unit 40 ( Figure 2 ), and can quickly notify the operator of the inferred results.

[0181] <Variation Example Related to Learning of Work Classification Inference Model 180>

[0182] Fig.17 1 is a schematic diagram showing a modification example related to the learning of the work classification inference model 180. Figures 6 to 9 In the description of FIG. 1 , an example in which the work classification estimation model 180 is trained before shipping of the wheel loader 1 is described. The learning data for training the work classification estimation model 180 may be collected from a plurality of wheel loaders 1 .

[0183] Fig.12 The first wheel loader 1 (wheel loader 1A), the second wheel loader 1 (wheel loader 1B), and the third wheel loader 1 (wheel loader 1C) shown are of the same model. The wheel loaders 1A, 1B, and 1C are factory-fresh and are at a work site.

[0184] The computer 102A obtains the operation data 201 and the posture data 203 from each wheel loader 1A, 1B, 1C. The computer 102A generates an action state image 205 based on the posture data 203, and further generates a specific viewpoint image 206. The computer 102A also obtains the work classification result 204 in association with the posture data 203. The computer 102A uses these learning data to train the work classification inference model 180 so that the inferred work classification can be obtained by inferring the work classification based on the specific viewpoint image 206 and the operation data 201.

[0185] The computer 102A can also communicate with the computer 102A through the communication interface 105 ( Figure 5 ) obtains the operation data 201, posture data 203, and work classification result 204 from each wheel loader 1A, 1B, 1C. Alternatively, the computer 102A may obtain the operation data 201, posture data 203, and work classification result 204 from each wheel loader 1A, 1B, 1C via the external recording medium 109.

[0186] The computer 102A may be located at the same work site as the wheel loaders 1A, 1B, 1C. Alternatively, the computer 102A may be located at a remote location away from the work site, such as a management center. The wheel loaders 1A, 1B, 1C may be located at the same work site or at different work sites.

[0187] The learned work classification estimation model 180 is provided to each wheel loader 1A, 1B, 1C via the communication interface 105 or the external recording medium 109. In this way, each wheel loader 1A, 1B, 1C includes the learned work classification estimation model 180.

[0188] When each wheel loader 1A, 1B, 1C has already stored the work classification inference model 180, the stored work classification inference model 180 is rewritten. The work classification inference model 180 may be rewritten periodically by periodically performing the above-mentioned collection of learning data and learning of the work classification inference model 180. The latest updated values ​​of the parameters of the work classification inference model 180 are stored in the storage device 104 each time.

[0189] The learned work classification inference model 180 is also provided to the fourth wheel loader 1 (wheel loader 1D). The work classification inference model 180 is provided to both the wheel loaders 1A, 1B, 1C that provide learning data and the wheel loader 1D that does not provide learning data. The wheel loader 1D may be located at the same work site as any one of the wheel loaders 1A, 1B, 1C, or may be located at a different work site from the wheel loaders 1A, 1B, 1C. The wheel loader 1D may also be at a stage before shipment.

[0190] The wheel loader 1D may be a different model from the wheel loaders 1A, 1B, and 1C. For example, the wheel loaders 1A, 1B, and 1C may be medium-sized or larger models, and the wheel loader 1D may be a small model. In the case of a wheel loader, regardless of the size of the vehicle body, the ratio of the working device to the main body of the working machine does not change significantly. It may be that posture data is obtained in a medium-sized or larger model equipped with a sensor, and the work classification inference model 180 that has been learned in association with the work classification for the posture of the working device is applied to a small model without a sensor, namely the wheel loader 1D. In this way, the work classification can be inferred with high accuracy even in a small model.

[0191] In the above embodiment, the work classification inference model 180 includes a neural network, but the present invention is not limited thereto, and the work classification inference model 180 may be a model that can accurately infer the work classification from the image data using machine learning, such as a support vector machine or a decision tree.

[0192] In the embodiment, an example in which the first processing device 30 estimates the work classification has been described, but the present invention is not limited thereto, and the second processing device 70 may estimate the work classification.

[0193] It should be understood that the embodiments disclosed this time are illustrative in all aspects and are not restrictive. The scope of the present invention is indicated by the claims rather than the above description, and includes all modifications within the meaning and scope equivalent to the claims.

[0194] Description of reference numerals:

[0195] 1, 1A, 1B, 1C, 1D...wheel loader; 2...body frame; 2a...front frame; 2b...rear frame; 3...working device; 4...traveling device; 5...operating cab; 6...bucket; 6a...shovel tip; 9...boom pin; 14...boom; 16...boom cylinder; 17...bucket pin; 19...bucket cylinder; 27...vehicle speed detection unit; 28a, 28b...first hydraulic detector; 29...first angle detector; 30...first processing device; 30j...storage unit; 45...output unit; 48...second angle detector; 49...forward / reverse switching device; 51...accelerator operating device; 52...boom operating device; 53...speed change operating device; 54...bucket operating device; 55...articulation operating device; 58...brake operating device; 65...photographing device; 70...second processing device; 100...excavation object; 102A, 102B...computer; 103...processor; 104...storage device; 110...dump truck; 161...operation data generation unit; 162...timer; 163...posture data generation unit; 164...job classification determination unit; 165...action state image generation unit; 166...specific viewpoint image generation unit; 167...job classification inference unit; 168...judgment unit; 169...adjustment unit; 171...image processing unit; 177...inferred job classification; 180...job classification inference model; 181, 281...input layer; 182, 282...intermediate layer; 183, 283...output layer; 188A, 188B, 270A, 270B, 270C, 270D...learning data; 201...operation data; 203...posture data; 204...job classification result; 205...action state image; 206...specific viewpoint image; 262...angle conversion unit; 265...working device posture inference unit; 266...error detection unit; 267...posture inference model updating unit; 270...learning data set; 271...captured image; 277...inferred posture; 280...posture inference model.

Claims

1. A system comprising a working machine, wherein: The system including the working machine comprises: A working machine having a working machine body and a working device mounted on the working machine body; A camera mounted on the working machine body to capture an image including at least a portion of the working device; as well as computer, The computer obtains the captured image including at least a portion of the working device, generates an inferred posture of the working device relative to the working machine body based on the captured image, creates a three-dimensional model representing the three-dimensional shape of the working machine based on the inferred posture, sets a viewpoint position representing the position of the viewpoint from which the three-dimensional model is hypothetically observed, and creates image data including a two-dimensional image obtained by observing the three-dimensional model from the viewpoint position.

2. The system comprising a work machine according to claim 1, wherein: The image data includes operation data related to the operation of the working machine.

3. The system comprising a work machine according to claim 2, wherein: The operation data includes data related to travel of the work machine.

4. A system comprising a working machine according to any one of claims 1 to 3, wherein: The computer has a learned posture estimation model for obtaining the posture of the working device. The computer is programmed to obtain an estimated posture of the working device estimated from the captured image using the learned posture estimation model.

5. The system comprising a work machine according to claim 4, wherein: The learned posture estimation model performs a learning process using a learning data set so that, when the captured image is input, the estimated posture is output based on the captured image.

6. The system comprising a work machine according to claim 4, wherein: The learned posture estimation model is generated by a learning process using a learning data set including a plurality of learning data in which the posture of the working device is labeled with respect to the captured image.

7. A working machine comprising: The main body of the operating machine; A working device, which is installed on the working machine body; a photographing device mounted on the working machine body and configured to photograph an image including at least a portion of the working device; and A computer mounted on the working machine body, in, The computer obtains the captured image including at least a portion of the working device, generates an inferred posture of the working device relative to the working machine body based on the captured image, creates a three-dimensional model representing the three-dimensional shape of the working machine based on the inferred posture, sets a viewpoint position representing the position of the viewpoint from which the three-dimensional model is hypothetically observed, and creates image data including a two-dimensional image obtained by observing the three-dimensional model from the viewpoint position.

Images