System including work machine, control method, and work machine

Abstract

Provided is a system capable of notifying an operator in consideration of the state of the operator. A system including a work machine is provided with: an electroencephalograph that detects brain waves of an operator who operates the work machine; a notification unit that notifies an operator; and a controller. The controller receives a signal indicating the detection result of the brain wave from the electroencephalograph, calculates the amplitude of the brain wave, and determines the manner of notification by the notification unit on the basis of the magnitude of the amplitude.

Description

Technical Field

[0001] The present disclosure relates to a system including a working machine, a control method, and a working machine. Background Art

[0002] Japanese Patent Publication No. 2001-067569 (Patent Document 1) discloses a danger warning system that detects when a person enters a dangerous area and issues a warning. The portable unit can be worn on a person and has a sensor group such as a brain wave sensor, a blood pressure sensor, and a sweat sensor. In the case where a person approaches a dangerous area without knowing it, a normal warning is issued. In the case where a person is identified as approaching a dangerous area, no warning is issued. After the normal warning is issued, the detection signal from the sensor group is comprehensively judged. If it is determined that the person is annoyed by the warning, the output level of the warning is reduced to control the person's annoyance.

[0003] Prior Art Literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2001-067569 Summary of the invention

[0006] Problems to be solved by the invention

[0007] In a conventional alarm system for notifying an operator of a working machine of an alarm, the same alarm is always issued regardless of the difficulty of the work or changes in the operator's state.

[0008] In the present disclosure, a system including a working machine, a control method, and a working machine are proposed, which are capable of notifying an operator in consideration of the state of the operator.

[0009] Means for solving problems

[0010] According to one aspect of the present disclosure, a working machine or a system including a working machine is proposed, which comprises: an electroencephalograph that detects brain waves of an operator operating the working machine; a notification unit that notifies the operator; and a controller that receives a signal representing the detection result of the brain waves from the electroencephalograph, calculates the amplitude of the brain waves, and determines the notification method of the notification unit based on the magnitude of the amplitude.

[0011] According to one aspect of the present disclosure, a control method for a notification unit for notifying an operator operating a working machine is proposed. The control method includes the following processing. The first processing is to detect the operator's brain waves. The second processing is to calculate the amplitude of the brain waves. The third processing is to determine the method of notification by the notification unit based on the amplitude of the brain waves.

[0012] Effects of the Invention

[0013] The system including a working machine, a control method, and a working machine according to the present disclosure can notify an operator in consideration of the state of the operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a side view schematically showing the structure of a hydraulic excavator.

[0015] Figure 2 It is a block diagram showing the system configuration of an embodiment.

[0016] Figure 3 This is a side view of the head schematically showing the electrode arrangement of an electroencephalograph.

[0017] Figure 4 It is a schematic diagram representing the theory of human attention resources.

[0018] Figure 5 This is a schematic diagram showing the relationship between the difficulty of a task and the allocation of attention resources.

[0019] Figure 6 This is a graph showing brain waves when stimulation is applied during a task with a low level of difficulty.

[0020] Figure 7 This is a graph showing brain waves when stimulation is applied during a task of medium difficulty.

[0021] Figure 8 This is a graph showing brain waves when stimulation is applied during a high-difficulty task.

[0022] Fig. 9 This is a graph showing the relationship between the amplitude of brain waves and the difficulty of tasks.

[0023] Fig.10 This is a flowchart showing the flow of a process for notifying an operator.

[0024] Fig.11 This is a diagram showing a method of notifying an operator of their proficiency and the difficulty of a task.

[0025] Fig.12 It is a diagram showing the amount of attention resources when mind wandering occurs.

[0026] Fig.13 This is a graph showing the amplitude of brain waves when mind wandering occurs.

[0027] Fig.14 This is a schematic diagram showing an example of issuing an alarm based on the temporal change of the electroencephalogram amplitude.

[0028] Fig.15This is a schematic diagram showing another example of issuing an alarm based on the temporal change of the electroencephalogram amplitude. DETAILED DESCRIPTION

[0029] Hereinafter, the implementation mode will be described based on the accompanying drawings. In the following description, the same reference numerals are used for the same components and constituent elements. Their names and functions are also the same. Therefore, their detailed description will not be repeated. It is also predetermined from the beginning to extract any structure from the implementation mode and to combine them arbitrarily.

[0030] <Overall structure of hydraulic excavator 1>

[0031] In the embodiment, a hydraulic excavator 1 will be described as an example of a working machine. Figure 1 It is a side view schematically showing the structure of the hydraulic excavator 1 .

[0032] like Figure 1 As shown, the hydraulic excavator 1 includes a working device 2 and a vehicle body 3. The vehicle body 3 includes a traveling body 31, a swing ring 32, a swing body 33, and a swing motor 35.

[0033] The traveling body 31 includes a pair of left and right crawler devices 311 and a traveling motor 312. The pair of left and right crawler devices 311 each include a crawler belt. When the traveling motor 312 is driven, the pair of left and right crawler belts rotate, and the hydraulic excavator 1 travels by itself.

[0034] The rotating body 33 is provided on the traveling body 31 via a rotating ring 32. The rotating ring 32 is connected to a rotating motor 35. The rotating ring 32 rotates by the rotation drive of the rotating motor 35. The rotating body 33 rotates relative to the traveling body 31 as the rotating ring 32 rotates.

[0035] The revolving body 33 includes a frame 331 on which the working device 2 is mounted, a cab 332, and a vehicle body controller 40 (see Figure 2 The operator's cab 332 is disposed on, for example, the front left side (vehicle front side) of the revolving body 33 .

[0036] The working machine 2 is supported by the frame 331 in front of the swing body 33 and on the right side of the cab 332. The working machine 2 includes a boom 21, an arm 22, a bucket 23, a boom cylinder 211, an arm cylinder 221, a bucket cylinder 231, and the like.

[0037] The boom 21 is attached to the revolving body 33. The base end portion of the boom 21 is rotatably connected to the revolving body 33 via a boom base pin (not shown).

[0038] The arm 22 is attached to the front end of the boom 21. The base end of the arm 22 is rotatably connected to the front end of the boom 21 via a boom front end pin 242.

[0039] The bucket 23 is attached to the front end of the arm 22. The bucket 23 is rotatably connected to the front end of the arm 22 via an arm front end pin 243. The bucket 23 is an example of an attachment that can be attached to the front end of the work machine 2.

[0040] The boom 21 can be driven by a boom cylinder 211. The boom cylinder 211 is driven by hydraulic oil supplied from a hydraulic source. By this driving, the boom 21 can be rotated in the vertical direction relative to the swing body 33 around a boom base pin (not shown).

[0041] The arm 22 can be driven by an arm cylinder 221. The arm cylinder 221 is driven by hydraulic oil supplied from a hydraulic source. By this driving, the arm 22 can rotate in the vertical direction relative to the boom 21 around the boom tip pin 242.

[0042] The bucket 23 can be driven by a bucket cylinder 231. The bucket cylinder 231 is driven by hydraulic oil supplied from a hydraulic source. By this driving, the bucket 23 can be rotated in the vertical direction relative to the arm 22 around the arm tip pin 243. In this way, the working device 2 can be driven.

[0043] <System Structure>

[0044] Figure 2 is a block diagram showing the system structure of an implementation method. Figure 2 As shown, the hydraulic excavator 1 mainly includes a vehicle body controller 40 and a hydraulic circuit from a hydraulic pump 51 to a hydraulic actuator 80 . Figure 2 The solid line in represents the hydraulic circuit. Figure 2 The dotted line in the figure represents the electrical circuit. Figure 2 1 , only a part of the electric circuit constituting the hydraulic excavator 1 according to the embodiment is shown.

[0045] The vehicle body controller 40 is a controller for controlling the operation of the hydraulic excavator 1 , and is composed of a CPU (Central Processing Unit), a nonvolatile memory, a timer, and the like.

[0046] The body controller 40 is connected to a monitor 36. The monitor 36 is, for example, disposed in the cab 332. The body controller 40 displays body information of the hydraulic excavator 1 on the monitor 36. The body information of the hydraulic excavator 1 includes, for example, a peripheral image of the hydraulic excavator 1, a remaining fuel level, a temperature of cooling water, a temperature of hydraulic oil, an operating condition of an air conditioner, and the like. The monitor 36 may include, for example, a machine monitor.

[0047] The hydraulic pump 51 supplies hydraulic oil used for the operation of the working device 2, the rotation of the revolving body 33, and the travel operation of the traveling body 31. The hydraulic pump 51 is connected to the drive shaft of the engine. The rotational driving force of the engine is transmitted to the hydraulic pump 51, thereby driving the hydraulic pump 51 and discharging the hydraulic oil. The hydraulic pump 51 is, for example, a variable displacement hydraulic pump having a swash plate and changing the discharge capacity by changing the tilting angle of the swash plate.

[0048] The tank 53 is a tank for storing oil used by the hydraulic pump 51 . The oil stored in the tank 53 is sucked out from the tank 53 by driving the hydraulic pump 51 , and is supplied to the main valve 70 through the hydraulic oil supply passage 55 .

[0049] The main valve 70 is a spool valve type valve that switches the direction of the working oil flow by moving a rod-shaped spool valve. The spool valve moves in the axial direction to adjust the supply amount of working oil to various hydraulic actuators 80. The working oil is oil supplied to the hydraulic actuator 80 to operate the hydraulic actuator 80.

[0050] The hydraulic actuator 80 includes Figure 1 The hydraulic actuator 80 includes a boom cylinder 211, an arm cylinder 221, and a bucket cylinder 231. Figure 1 The travel motor 312 and the swing motor 35 are shown. The hydraulic oil is supplied to the hydraulic actuator 80 via the main valve 70. By controlling the supply and discharge of the hydraulic oil to the hydraulic actuator 80, the operation of the work machine 2, the swing of the swing body 33, and the travel operation of the travel body 31 are controlled.

[0051] The hydraulic oil discharged from the main valve 70 returns to the tank 53 via the hydraulic oil discharge passage 56 .

[0052] A portion of the oil delivered from the hydraulic pump 51 is branched from the working oil supply oil passage 55 and flows into the pilot oil passage 57. A portion of the oil delivered from the hydraulic pump 51 is decompressed by the self-pressure reducing valve, and the decompressed oil is used as pilot oil. The pilot oil is oil supplied to the main valve 70 in order to operate the spool valve of the main valve 70. The main valve 70 has a pair of pressure receiving chambers 72. By supplying pilot oil having a predetermined hydraulic pressure (pilot pressure) to each pressure receiving chamber 72, the spool valve moves according to the pilot pressure, and the main valve 70 is controlled.

[0053] The pilot oil passage 57 is provided with an operating device 61. The operating device 61 is disposed in the cab 332. The operating device 61 is operated by an operator to operate the hydraulic excavator 1. The pilot pressure applied to the main valve 70 is controlled by operating the operating device 61. The operating device 61 has an operating lever. Pilot oil having a pressure corresponding to the amount of operation of the operating lever is output from the operating device 61 and supplied to each pressure receiving chamber 72 of the main valve 70.

[0054] The pilot oil passage 57 is provided with a solenoid valve 63. The solenoid valve 63 is connected to the pilot oil passage 57 upstream of the operating device 61 in the flow direction of the pilot oil. The solenoid valve 63 adjusts the pilot oil supplied to the operating device 61 based on a control signal from the vehicle body controller 40.

[0055] A branch pipe is connected to the downstream of the operating device 61 in the flow direction of the pilot oil in the pilot oil passage 57. A pressure gauge 65 is provided on the branch pipe. The pressure gauge 65 detects the pressure of the pilot oil after passing through the operating device 61. The detection signal of the hydraulic pressure detected by the pressure gauge 65 is input to the vehicle body controller 40. The detection signal of the hydraulic pressure detected by the pressure gauge 65 is an example of an operation signal corresponding to the operation of the operating device 61.

[0056] The pressure gauge 65 may be a pressure sensor that outputs an electrical signal proportional to the pilot pressure. The vehicle body controller 40 may also be configured to generate an electric signal when the pilot pressure detected by the pressure sensor is equal to or greater than a predetermined threshold value, such as 5 kg / cm 2 In the above case, it is determined that the operating device 61 is operated.

[0057] The pressure gauge 65 may also be a pressure switch that switches on and off when the pilot pressure reaches a set pressure. The threshold of the pressure switch may also be set to, for example, 5 kg / cm 2 In this case, the vehicle body controller 40 may detect that pressure is generated in the pilot oil passage 57 when the pressure switch is switched from OFF to ON, and determine that the operating device 61 is operated.

[0058] The vehicle body controller 40 can determine whether the hydraulic excavator 1 is performing work based on the detection result of the pressure gauge 65. When the pressure gauge 65 detects a change in the pressure of the pilot oil supplied to the main valve 70 that supplies the working oil to any one of the hydraulic actuators 80 of the boom cylinder 211, the arm cylinder 221, and the bucket cylinder 231, the vehicle body controller 40 can also determine that the hydraulic excavator 1 is performing work using the working device 2.

[0059] The vehicle body controller 40 may also determine that the hydraulic excavator 1 is traveling when the pressure gauge 65 detects a change in the pressure of the pilot oil supplied by the main valve 70 that supplies the operating oil to the travel motor 312. The vehicle body controller 40 may also determine that the hydraulic excavator 1 is rotating when the pressure gauge 65 detects a change in the pressure of the pilot oil supplied by the main valve 70 that supplies the operating oil to the rotation motor 35.

[0060] The operating device 61 may also be an electrical operating device. The electrical operating device outputs a signal indicating the operating direction and the operating amount of the operating device to the vehicle body controller 40. The signal output by the electrical operating device is an example of an operating signal corresponding to the operation of the operating device 61. The vehicle body controller 40 may also determine that the hydraulic excavator 1 is performing work when receiving a signal from the electrical operating device.

[0061] The electroencephalograph 41 is connected to the vehicle body controller 40. The electroencephalograph 41 detects the brain waves of the operator who operates the hydraulic excavator 1. The operator riding in the cab 332 is obliged to wear a helmet, and the electroencephalograph 41 may be installed inside the helmet. The operator riding in the cab 332 may also wear a head cap provided with the electroencephalograph 41. The vehicle body controller 40 receives a signal indicating the detection result of the brain waves from the electroencephalograph 41.

[0062] Figure 3 This is a side view of the head schematically showing the electrode configuration of the electroencephalograph 41. The electrodes are configured according to the international 10-20 method. The electrode configuration (Cz) in the center is determined by the midpoint between the root of the nose and the occipital tubercle and the midpoint between the front points of the left and right auricles. The electrode configuration (Pz) in the center of the head is determined by dividing the line passing through the median line and connecting Cz and the occipital tubercle into a 2:3 position.

[0063] In this embodiment, the focus is on P300, which has a peak 300ms after the stimulation is applied, among the brain waves detected by the electroencephalograph 41. P300 is known as an example of a phenomenon-related potential. P300 is known as a biological signal associated with the amount of attention of a person. It is known that P300 has a high amplitude on the midline of the brain, and has a maximum amplitude particularly at Pz.

[0064] P300 is used as an indicator for knowing the state of an operator operating a work machine. Using P300, it is possible to estimate how much attention the operator operating the work machine pays to the operation. For example, it is possible to estimate using the dual-topic method. In the dual-topic method, a sub-topic different from the main topic is assigned during the execution of the main topic, and the amount of attention paid to the sub-topic can be estimated based on the amplitude of P300 when the sub-topic is assigned. The main topic is, for example, the operation of the hydraulic excavator 1. As a sub-topic, for example, tactile stimulation of an intensity that does not hinder the operation of the hydraulic excavator 1 is given to the fingertips of the operator. The tactile stimulation may also be, for example, electrical stimulation, vibration, etc.

[0065] return Figure 2The vehicle body controller 40 that receives the detection result of the brain wave from the electroencephalograph 41 filters the raw data of the brain wave and extracts P300 in real time. The vehicle body controller 40 monitors the state of P300, specifically, the amplitude of P300. The vehicle body controller 40 analyzes the state of the operator in real time based on the state of P300, and uses the analysis result to monitor the timing of notifying the operator.

[0066] An alarm 42 is connected to the vehicle body controller 40. The alarm 42 issues an alarm for notifying the operator. It is also possible to perform auditory notifications such as alarm sounds, voices, stereos, etc. The volume of the voice, the length of the voice, etc. can also be changed. It is also possible to perform tactile notifications such as vibrations. The monitor 36 connected to the vehicle body controller 40 can also perform visual notifications for notifying the operator. The monitor 36 can include, for example, a head-up display. The monitor 36 and the alarm 42 correspond to an example of a notification unit that notifies the operator.

[0067] The vehicle body controller 40 is connected to a communication unit 44. The communication unit 44 communicates with a base station 100 outside the hydraulic excavator 1. The base station 100 is provided with a remote controller 140, a remote monitor 136, a remote alarm 142, and a remote communication unit 144. The communication unit 44 of the hydraulic excavator 1 communicates with the remote communication unit 144 of the base station 100 via a network. The network includes at least one of the Internet, a local area network (LAN), a mobile phone communication network, and a satellite communication network.

[0068] For example, the vehicle body controller 40 generates information displayed on the remote monitor 136 of the base station 100 and a signal indicating an alarm issued from the remote alarm 142 of the base station 100. The communication unit 44 transmits the signal generated by the vehicle body controller 40 to the remote communication unit 144. The remote controller 140 receives the signal generated by the vehicle body controller 40 via the remote communication unit 144. The remote controller 140 causes the remote monitor 136 to display information and causes the remote alarm 142 to issue an alarm. The remote alarm 142 notifies a person located in the base station 100, for example, a manager of an operator operating the hydraulic excavator 1. The manager who receives the notification can grasp the state of the operator in real time.

[0069] <Attention Resource Theory>

[0070] Figure 4 This is a diagram showing the theory of human attention resources. The theory of human attention resources is a hypothesis that the total amount of attention that can be used per unit time is fixed for each person. Figure 4As shown, when the operator operates the working machine, a part of the attention resources in a certain total amount is allocated to the operation of the working machine. The amount of attention resources in the certain total amount that is not allocated to the operation of the working machine but can be allocated to other parts is the surplus amount of attention. The state of the operator is displayed based on the surplus amount of attention.

[0071] Figure 5 This is a diagram showing the relationship between the difficulty of a task and the allocation of attention resources. Figure 5 The figure shows the amount of attention resources allocated to the operation of the working machine and the surplus amount of attention when there is no load, that is, when the working machine is not working, when the working machine is performing low-difficulty work, when the working machine is performing medium-difficulty work, and when the working machine is performing high-difficulty work.

[0072] As described above, the amount of attention resources that can be used by the operator is fixed. Regardless of the difficulty of the operation, the total amount of attention resources is fixed. If the difficulty of the operation is low, the amount of attention resources allocated to the operation of the operating machine is small. Accordingly, the surplus amount of attention becomes larger. On the contrary, if the difficulty of the operation is high, the amount of attention resources allocated to the operation of the operating machine becomes large. Accordingly, the surplus amount of attention becomes smaller.

[0073] Figure 6 This is a graph showing the output pattern of the operator's brain waves with respect to the passage of time when electrical stimulation is applied to the operator while the working machine is performing a work of low difficulty. Figure 7 This is a graph showing the output pattern of the operator's brain waves with respect to the passage of time when electrical stimulation is applied to the operator while the operator is performing work of varying degrees of difficulty in a working machine. Figure 8 This is a graph showing the output pattern of the operator's brain waves with respect to the passage of time when electrical stimulation is applied to the operator when the working machine performs a high-difficulty operation. Figures 6 to 8 The horizontal axis represents the passage of time. Figures 6 to 8 The vertical axis represents the brain wave amplitude.

[0074] exist Figures 6 to 8 In the experiment, P300 is detected. At time 0, electrical stimulation is applied to the operator. P300 takes a minimum value m before time 250ms and a maximum value M after time 300ms. The difference between the maximum value M and the minimum value m is the amplitude of the brain wave (P300), which becomes an indicator of the degree to which the operator's attention is directed to the applied electrical stimulation. The more attention is directed to the electrical stimulation, the greater the amplitude of P300.

[0075] Fig. 9 This is a graph showing the relationship between the amplitude of brain waves and the difficulty of tasks. Fig. 9 The vertical axis represents the brain wave amplitude. Fig. 9The value represents the average value of the P300 amplitude calculated for a plurality of operators. Fig. 9 Error bars shown represent standard errors.

[0076] like Figures 6 to 9 As shown in FIG. 1 , the amplitude of P300 is larger in low-difficulty operations, smaller in medium-difficulty operations compared to low-difficulty operations, and further smaller in high-difficulty operations. The smaller the difficulty of operating the work machine, the larger the amplitude of P300. This is believed to be because, Figure 5 As shown, when the difficulty level of operating the working machine is low, the margin of attention is large, and when the difficulty level of operating the working machine is high, the margin of attention is small.

[0077] like Figure 6 , 9 As shown in , when the difficulty of the task is low and the operator has time to spare, the amplitude of P300 becomes larger because the brain can fully respond to the electrical stimulation. Figure 8 , 9 As shown in FIG. 1 , when the difficulty of the operation is high, the operator's attention is directed to the difficult operation, and the operator's attention resources are largely allocated to the operation of the operation machine, so it is difficult for brain waves to be emitted in response to electrical stimulation. Since the brain cannot respond to electrical stimulation, the amplitude of P300 becomes smaller.

[0078] Therefore, during the operation of the working machine, by acquiring the amplitude of P300 of the operator operating the working machine, the state of the operator (margin of attention) can be estimated.

[0079] It should be noted that the difficulty of the operation varies for each operator. Even if an inexperienced operator finds the operation difficult, an experienced operator may find it easy because it is a familiar operation. The margin of attention when performing a certain operation may be different between an inexperienced operator and an experienced operator. The margin of attention when performing a certain operation may be smaller for an inexperienced operator and larger for an experienced operator. During the operation, the margin of attention of each operator can be estimated by obtaining the amplitude of P300 of each operator.

[0080] The amount of attention that can be paid sometimes varies depending on the operator's physical and mental health. When there is an instability in the health state, attention resources are allocated to the instability in the health state, so the amount of attention resources that can be allocated to other areas is reduced, and the amount of attention that can be paid becomes smaller. If there is no instability in the health state, the amount of attention that can be paid becomes larger.

[0081] <Notification control for operators>

[0082] Next, control for notifying the operator based on the brain waves of the operator operating the working machine will be described. Fig.10This is a flowchart showing the flow of a process for notifying an operator.

[0083] The difficulty level of a certain task (whether it is a difficult task (high difficulty level) or an easy task (low difficulty level)) is different for each operator. The margin of attention in a predetermined task is different for each operator. The margin of attention also varies depending on the health status of the operator. Therefore, in order to make the system conform to the characteristics of the individual, initial settings are performed in step S1.

[0084] For example, the operator riding in the cab 332 can also perform the N-back task using the monitor 36 disposed in the cab 332 before starting the work. The N-back task is a cognitive task that can set the difficulty in stages. As an N-back task, a task can also be implemented in which text is displayed on the monitor 36 at a certain time interval to confirm whether the currently displayed text is the same as the text displayed N times ago. During the implementation of the N-back task, electrical stimulation of an intensity that does not interfere with the task is applied to the operator's fingers. The greater the surplus of attention resources, the more attention resources are allocated to the electrical stimulation, so the brain response (P300) to the electrical stimulation becomes larger. The surplus of attention resources can be estimated by P300. The electroencephalograph 41 detects the brain waves of the operator during the implementation of the N-back task. The body controller 40 extracts P300 from the brain waves of the operator and calculates the amplitude of P300. The body controller 40 determines the threshold for the alarm sound based on the amplitude of P300.

[0085] The operator may also perform the N-back task before getting on the cab 332. For example, the operator may also perform the N-back task in advance using an application on a smartphone. The operator sends the detection result of P300 in the N-back task performed in advance or the amplitude of P300 calculated based on the detection result of P300 to the vehicle body controller 40. The vehicle body controller 40 determines the threshold for issuing the alarm sound based on the amplitude of P300.

[0086] Alternatively, the N-back task may be replaced by a simple operation of a working machine and the operator's P300 may be measured during the operation. The operator in the cab 332 performs certain operations of several working machines, and the electroencephalograph 41 detects the operator's P300 during the operation. The vehicle body controller 40 calculates the amplitude of P300. The vehicle body controller 40 determines the threshold for sounding the alarm based on the amplitude of P300.

[0087] After the operator gets on the cab 332 and before starting work, the operator wears an electroencephalograph 41 for detecting brain waves and an external stimulation device for applying tactile stimulation. The external stimulation device applies tactile stimulation such as electrical stimulation and vibration to the operator.

[0088] After the initial setting of the system, in step S2, the vehicle body controller 40 determines whether the working machine is performing work. Figure 2 As described above, the vehicle body controller 40 receives an input of an operation signal corresponding to the operation of the operation device 61 from the pressure gauge 65 or the operation device 61, thereby being able to determine that the hydraulic excavator 1 is performing work. The work referred to here includes not only work using the work implement 2, but also the rotation of the revolving body 33 and the travel of the hydraulic excavator 1 given to the travel body 31.

[0089] If it is determined that the working machine is not in operation (No in step S2), the determination of step S2 is repeated. If it is determined that the working machine is in operation (Yes in step S2), the process proceeds to step S3, where the electroencephalograph 41 detects the brain waves of the operator when tactile stimulation is applied to the operator operating the working machine. The vehicle body controller 40 receives a signal indicating the detection result of the brain waves from the electroencephalograph 41.

[0090] In step S4, the body controller 40 filters the detection result of the electroencephalogram input from the electroencephalograph 41 to extract P300. The body controller 40 calculates the amplitude of P300.

[0091] In step S5, the vehicle body controller 40 determines whether it is a situation that needs to be notified to the operator. For example, if a person is allowed to approach the working machine, or the working machine is in a situation where it may come into contact with an obstacle while continuing to work, it is determined that the operator of the working machine needs to be notified. The surrounding conditions of the working machine can be obtained by a sensor mounted on the working machine or arranged outside the working machine. The sensor can also be a camera that takes a picture of the subject. The camera can be arranged at a specified location on the work site or mounted on a drone.

[0092] If it is determined that notification is necessary (YES in step S5 ), the process proceeds to step S6 , and the vehicle body controller 40 determines the form of notification to be given to the operator by the notification unit (monitor 36 , alarm 42 ).

[0093] Fig.11 FIG. 1 is a diagram showing a method of notifying an operator of his / her proficiency and the difficulty of a task. Fig.11 In the figure, it is indicated what kind of notification is appropriate for an experienced operator or a novice operator when performing a low-difficulty or high-difficulty operation. The operator's proficiency is determined by the initial setting of step S1. The difficulty of the operation is determined based on the amplitude of P300 calculated in step S4. When the amplitude of P300 is large, the operation is determined to be of low difficulty, and when the amplitude of P300 is small, the operation is determined to be of high difficulty.

[0094] When a skilled operator performs a low-difficulty operation, the operator has a large margin of attention, so even if a complex notification is made, the operator can be expected to recognize and understand the content of the notification. Therefore, instead of simply generating a single warning sound, the operator can be notified of a voice message of a condition or instruction. Alternatively, if the skilled operator has a large margin of attention, the operator can be expected to allocate attention resources to understand the surrounding conditions of the operating machine. Therefore, by not making a notification, the operator can be prevented from being annoyed by the notification.

[0095] When a skilled operator performs a highly difficult task, he or she has little time to pay attention, so if a complex notification is given, the contents of the notification may not be understood and the alarm may be meaningless. Therefore, by providing a notification based on a warning sound, the operator's attention can be reliably called.

[0096] When a new operator with little experience is performing a task, if a complex notification is given, the operator may not understand the content of the notification and it may be meaningless as an alarm. Therefore, a notification based on a warning sound is given regardless of the difficulty of the task. When performing a task with high difficulty, the operator's attention can be reliably called by increasing the volume of the warning sound.

[0097] By determining the difficulty of the operation felt by the operator based on the amplitude of P300 and determining the notification method of the notification unit based on the difficulty of the operation, the operator can be notified appropriately according to the operator's proficiency. Since the operator's state can be considered when notifying the operator, the operator who receives the notification can more reliably recognize, for example, that an object that hinders the operation of the operating machine exists around the operating machine.

[0098] return Fig.10 In step S7, the vehicle body controller 40 outputs a signal instructing the notification method determined in step S6 to the alarm device 42. The alarm device 42, which receives the signal, sounds an alarm to notify the operator.

[0099] In step S8, the vehicle body controller 40 determines whether the operation is completed. The vehicle body controller 40 can determine that the operation is not completed when the operation signal corresponding to the operation of the operating device 61 continues to be input from the pressure gauge 65 or the operating device 61. The vehicle body controller 40 can determine that the operation is completed if there is no input of the operation signal corresponding to the operation of the operating device 61 from the pressure gauge 65 or the operating device 61.

[0100] If it is determined in step S5 that notification is not required (No in step S5) and if it is determined in step S8 that the operation is not completed (No in step S8), return to step S3 and repeat the detection of the operator's brain waves and the calculation of the amplitude of P300.

[0101] If it is determined in step S8 that the operation is completed (Yes in step S8), the processing is terminated ( Fig.10 of "end").

[0102] <Notification control when distracted>

[0103] During the operation, the operator's attention may be diverted from the operation of the work machine, and sometimes the phenomenon of distraction (distraction) may occur, which is not related to the operation. As mentioned above, when the operator performs a low-difficulty operation, the amount of attention resources allocated to the operation of the work machine is small. In a high-difficulty operation, most of the attention resources are allocated to the operation of the work machine, so distraction is not easy to occur. Distraction is easy to occur when performing a low-difficulty operation.

[0104] Fig.12 This is a diagram showing the amount of attention resources when distraction occurs. Fig.12 The horizontal bars shown represent Figure 4 The same amount of attention. Fig.12 The upper part of the graph shows the allocation of attention resources when there is no distraction in a low-difficulty task. Since it is a low-difficulty task, the amount of attention resources allocated to the operation of the work machine is relatively small, and the margin of attention is large. Therefore, if there is no distraction, the amplitude of P300 is larger.

[0105] exist Fig.12 The lower part of the figure shows the allocation of attention resources when distraction occurs in low-difficulty operations. The amount of attention resources allocated to the operation of the work machine is the same as when distraction does not occur. Since part of the attention resources is allocated to distraction, the surplus amount of attention becomes smaller than when distraction does not occur.

[0106] Fig.13 This is a graph showing the amplitude of brain waves when mind wandering occurs. Fig. 9 Likewise, Fig.13 The vertical axis represents the brain wave amplitude. In the brain wave amplitude of the low-difficulty task, the dotted line represents Fig. 9 The amplitude of P300 when distraction does not occur is also shown in FIG. The solid line shows the amplitude of P300 when distraction occurs. Since attention resources are allocated to distraction, the surplus of attention decreases, and as a result, the amplitude of P300 decreases when distraction occurs.

[0107] The vehicle body controller 40 can determine that distraction has occurred based on the temporal change in the amplitude of P300. The electroencephalograph 41 detects the brain waves of the operator when tactile stimulation is applied to the operator operating the working machine. Figures 6 to 8As described above, when a tactile stimulus is applied, the amplitude of P300 relative to the tactile stimulus can be calculated. If multiple tactile stimuli are applied at random intervals, the amplitude of each P300 relative to each tactile stimulus can be calculated. By arranging the amplitudes of multiple P300s obtained when multiple tactile stimuli are applied in a time series, the temporal change in the amplitude of P300 can be grasped. The vehicle body controller 40 can decide whether to notify the operator based on the temporal change in the amplitude of P300. The decision on whether to notify the operator corresponds to an example of the decision on the notification method.

[0108] Fig.14 is a schematic diagram showing an example of an alarm being issued based on temporal changes in the amplitude of electroencephalograms. Fig.14 As shown, a threshold is set for the amplitude of P300. This threshold is set to a different value for each operator. This threshold can be determined in the initial setting of step S1 described above.

[0109] exist Fig.14 The figure shows the amplitude of P300 obtained by the reaction to the first to Nth tactile stimulation. Within a certain period of time, that is, from the application of the first tactile stimulation to the application of the Nth tactile stimulation, the amplitude of P300 does not exceed the threshold, and the state of the small amplitude of P300 continues for a certain period of time.

[0110] In such a case, the vehicle body controller 40 notifies the operator through the notification unit. If the amplitude of P300 remains small despite the difficulty of the work being performed being low, it can be determined that distraction has occurred. In this case, by notifying the operator, distraction can be eliminated and the operation of the work machine can be concentrated.

[0111] In addition, if Figure 8 , 9 As shown in FIG. 1 , when the operator performs a high-difficulty operation, the amplitude of P300 also decreases. When the amplitude of P300 is small, it is possible that the high-difficulty operation is continued. By notifying the operator, it is possible to provide an opportunity for the operator to realize whether he is unconsciously fatigued by performing a high-difficulty operation for a certain period of time.

[0112] Fig.15 This is a schematic diagram showing another example of issuing an alarm based on the temporal change of the electroencephalogram amplitude. Fig.15Indicates that the amplitude of the P300 obtained by the response to the (K+1)th tactile stimulation is sharply reduced compared to the amplitude of the P300 obtained by the response to the Kth tactile stimulation. A threshold value is set for the increase or decrease value of the amplitude of the P300. Compared to the amplitude of the P300 detected when the tactile stimulation is applied for the Kth time, the amplitude of the P300 detected when the tactile stimulation is applied for the (K+1)th time is reduced by more than the threshold value.

[0113] In such a case, the vehicle body controller 40 notifies the operator through the notification unit. The reason why the amplitude of P300 has decreased by more than a predetermined amount since the last detection of P300 is that the amount of attention resources has been allocated to the distraction due to the distraction, so it can be determined that it is caused by the reduction of the margin of attention. In this case, by notifying the operator, distraction can be eliminated and the operation of the working machine can be concentrated.

[0114] By notifying the operator based on the temporal change of the amplitude of the operator's P300 during the operation of the working machine, the operator's distraction can be eliminated in advance. By increasing the operator's attention margin, for example, the operator can pay more attention to the surrounding conditions of the working machine. For example, the operator can recognize the existence of objects that hinder the operation of the working machine around the working machine earlier and more reliably.

[0115] In the above-described embodiment, an example has been described in which the vehicle body controller 40 that controls the operation of the hydraulic excavator 1 determines the method of notification to the operator. The controller that determines the method of notification to the operator does not necessarily have to be mounted on the hydraulic excavator 1 .

[0116] Alternatively, a system may be configured such that a controller mounted on the hydraulic excavator 1 sends the detection result of the brain wave obtained by the electroencephalograph 41 to an external controller, and the external controller that receives the detection result of the brain wave calculates the amplitude of the brain wave and controls the method of determining the notification based on the amplitude of the brain wave. Alternatively, a system may be configured such that a controller mounted on the hydraulic excavator 1 calculates the amplitude of the brain wave based on the detection result of the brain wave and sends the calculated amplitude of the brain wave to an external controller, and the external controller that receives the amplitude of the brain wave controls the method of determining the notification based on the amplitude of the brain wave.

[0117] The external controller may be arranged at the working site of the hydraulic excavator 1 or at a location far away from the working site of the hydraulic excavator 1. For example, the external controller may be arranged at Figure 2 The external controller may also be Figure 2 A remote controller 140 is shown.

[0118] In the embodiment, an example is described in which the hydraulic excavator 1 has a cab 332 and an operator is a manned vehicle riding in the cab 332. The hydraulic excavator 1 may also be an unmanned vehicle. The hydraulic excavator 1 may not have a cab 332 for an operator to ride and operate. The hydraulic excavator 1 may not have a control function for an operator riding. The hydraulic excavator 1 may also be a working machine dedicated to remote control. The hydraulic excavator 1 may be controlled by a wireless signal from a remote control device.

[0119] When the hydraulic excavator 1 is remotely controlled, the electroencephalograph detects the brain waves of the operator who operates the remote control device, and a system is constructed in which the notification unit notifies the operator who operates the remote control device. The electroencephalograph that detects the brain waves of the operator does not need to be mounted on the hydraulic excavator 1. The notification unit that notifies the operator does not need to be mounted on the hydraulic excavator 1. When the hydraulic excavator 1 is remotely controlled, it is preferred to set a controller at a location where the remote control device outside the hydraulic excavator 1 is set, and the controller calculates the amplitude of the brain waves from the signal indicating the detection result of the brain waves by the electroencephalograph, and determines the method of notification by the notification unit based on the amplitude.

[0120] In the embodiment, the hydraulic excavator 1 is described as an example of a working machine, but the concept of the present disclosure is not limited to the hydraulic excavator 1 and can be applied to other types of working machines such as a bulldozer, a wheel loader, and a dump truck.

[0121] The embodiments are described above, but it should be understood that the embodiments disclosed this time are illustrative in all aspects and not restrictive. The scope of the present invention is shown by the technical solution rather than the above description, and is intended to include the meaning equivalent to the technical solution and all changes within the scope.

[0122] Description of reference numerals:

[0123] 1 hydraulic excavator; 2 working device; 3 vehicle body; 31 traveling body; 33 swinging body; 35 swing motor; 36 monitor; 40 vehicle body controller; 41 electroencephalograph; 42 alarm; 44 communication unit; 61 operating device; 65 pressure gauge; 80 hydraulic actuator; 100 base station; 136 remote monitor; 140 remote controller; 142 remote alarm; 144 remote communication unit; 312 traveling motor; M maximum value; m minimum value.

Claims

1. A system comprising a working machine, wherein: The system has: an electroencephalograph for detecting brain waves of an operator operating the working machine; a notifying unit configured to notify the operator; and Controller, The controller receives a signal indicating a detection result of the electroencephalogram from the electroencephalograph, calculates an amplitude of the electroencephalogram, and determines a form of notification to be performed by the notification unit based on the amplitude.

2. The system according to claim 1, wherein: The controller determines the notification method based on the temporal change in the amplitude.

3. The system according to claim 2, wherein: A threshold is set for the amplitude. The controller determines the notification method based on the fact that the amplitude does not exceed the threshold value within a certain period of time.

4. The system according to claim 2, wherein: The controller determines the form of the notification based on the fact that the amplitude has decreased by more than a predetermined amount since the last time the electroencephalogram was detected.

5. The system according to any one of claims 1 to 4, wherein: The working machine includes a communication unit that communicates with an external base station.

6. The system according to any one of claims 1 to 4, wherein: The controller determines whether the working machine is performing work.

7. A control method for a notification unit for notifying an operator operating a working machine, wherein: The control method comprises: detecting brain waves of the operator; calculating the amplitude of the brain wave; and The form of notification by the notification unit is determined based on the magnitude of the amplitude.

8. A working machine, wherein: The working machine comprises: an electroencephalograph for detecting brain waves of an operator operating the working machine; a notifying unit configured to notify the operator; and Controller, The controller receives a signal indicating a detection result of the electroencephalogram from the electroencephalograph, calculates an amplitude of the electroencephalogram, and determines a form of notification to be performed by the notification unit based on the amplitude.

Images