Work machine safety device

Abstract

To take safety measures to prevent accidents in a work machine based on a safe driving behavior of an occupant of the work machine.SOLUTION: A work machine safety device 100 disclosed herein includes: an imaging device 200 that images an occupant of a work machine 10; a processor 303 that executes predetermined processing based on captured image data representing the captured image of the occupant; and an alarm issuing device 400 that outputs a predetermined warning signal and / or a predetermined alert. The processor 303 executes: extraction processing that extracts a driving behavior of the work machine by the occupant from the captured image data; determination processing that determines whether or not the extracted driving behavior belongs to a predetermined safe driving behavior pattern; and alert issuing control processing that causes the alert issuing device 400 to output a warning signal and / or an alert when the determination processing determines that the driving behavior does not belong to the safe driving behavior pattern.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a safety device for a working machine.

Background Art

[0002] At work sites where working machines such as excavators and forklifts are used, safety measures are taken to prevent accidents of the working machines. For example, working machines equipped with mirrors and cameras have been put into practical use so that an operator can visually recognize obstacles existing around the vehicle body of the working machine. And a system that displays an image captured by a camera on a display in the cab of the working machine is known.

[0003] On the other hand, in recent years, in vehicles such as automobiles, it has been known to acquire data during driving of the vehicle by a driver and operation behaviors during driving of the vehicle by the driver, and to evaluate and diagnose driving skills. Also, there is a technique for evaluating a driver's safety confirmation behavior based on gaze measurement data.

[0004] For example, Patent Document 1 discloses a driving technique discrimination device for determining whether or not a driver's visual safety confirmation has become a mere formality. In this driving technique discrimination device, a driving action history is generated from driving operation information and behavior information stored in a history storage unit, and a driving action that is dangerous based on the distance from surrounding vehicles and for which the gaze movement with respect to the driving action was correct can be extracted from the driving action history.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In order to prevent accidents of construction machinery such as excavators and forklifts, it is necessary for the operator of the construction machinery to perform appropriate driving operations. However, at the site where such construction machinery is used, for example, as a result of overemphasizing work efficiency, a situation may occur in which the safety confirmation behavior of the operator with respect to the driving operation of the construction machinery becomes formalized.

[0007] And it is known to use a camera or the like to determine the safety confirmation of the driver. For example, according to the technique described in Patent Document 1, a face image of the driver is acquired by a camera or the like, a driving operation history is generated, and it is determined whether or not the direction of the driver's line of sight is in an appropriate direction at an appropriate timing. However, in this technique, since the determination is made retrospectively based on the driving operation history, safety measures for accident prevention cannot be taken in real time. In addition, in construction machinery such as excavators and forklifts, it may be difficult to determine whether or not safety confirmation is being performed only by the line of sight of the operator.

[0008] An object of the present disclosure is to take safety measures for preventing accidents of construction machinery based on the safe driving behavior of the operator of the construction machinery.

Means for Solving the Problems

[0009] The safety device for a construction machine of the present disclosure is installed on the construction machine, and includes a photographing device that photographs an operator of the construction machine, a processor that acquires photographing image data representing an image of the operator photographed by the photographing device, and executes predetermined processing based on the photographing image data, and a reporting device that outputs a predetermined alarm signal or / and a predetermined report. The processor executes an extraction process of extracting the driving behavior of the construction machine by the operator from the photographing image data, a determination process of determining whether or not the extracted driving behavior belongs to a predetermined safe driving behavior pattern, and a reporting control process of causing the reporting device to output the alarm signal or / and the report when it is determined by the determination process that the driving behavior does not belong to the safe driving behavior pattern.

[0010] For such a safety device of a working machine, by simply arranging a photographing device, a processor, and a warning device on an existing working machine, safety measures for preventing accidents of the working machine can be easily achieved. And, by executing warning control processing based on the result of the above determination processing, it is possible to warn the crew about dangerous actions in real time. In the case of post - discrimination based on the driving operation history as in the prior art, since the notification is made after a lapse of time from the dangerous action, there is little motivation for action correction, and the dangerous action is likely to recur. On the other hand, according to the present disclosure, by warning the crew about dangerous actions in real time, the suppression of the crew's actions can work strongly and the recurrence can also be reduced.

[0011] And, in the safety device of the above - mentioned working machine, the safe driving action pattern is the finger - pointing call before the operation of the working machine by the crew. The processor extracts the finger - pointing action of the crew as the driving action, and determines whether the extracted finger - pointing action of the crew belongs to the execution pattern of the finger - pointing call. When it is determined that the finger - pointing action of the crew does not belong to the execution pattern of the finger - pointing call, the warning device may output the warning signal and / or the warning. Further, the working machine is provided with a speed sensor that acquires the traveling speed and traveling direction of the working machine. The safe driving action pattern is that the line of sight of the crew during the operation of the working machine is directed toward the traveling direction of the working machine. The processor extracts the face orientation of the crew as the driving action, and determines whether the extracted face orientation of the crew belongs to the traveling direction of the working machine acquired by the speed sensor. When it is determined that the face orientation of the crew does not belong to the traveling direction of the working machine, during the operation of the working machine by the crew, the warning device may output the warning signal and / or the warning.

[0012] Also, in the safety device of the work machine of the present disclosure, the work machine is a forklift, the photographing device is configured to be able to photograph the surroundings of the forklift including the load moved by the forklift, the safe driving behavior pattern is the confirmation by the operator of the state of insertion of the claws of the forklift into the load, the processor extracts the direction of the operator's face as the driving behavior, obtains the insertion rate of the claws of the forklift into the load based on the image data of the claws of the forklift photographed by the photographing device, and based on the insertion rate and the direction of the operator's face, determines whether the operator has confirmed the state of insertion of the claws of the forklift into the load. When it is determined that the operator has not confirmed the insertion state, during the operation of the work machine by the operator, the warning signal or / and the notification may be output by the notification device.

[0013] And, in this case, in the determination process, the processor may determine that the operator has not confirmed the state of insertion of the claws of the forklift into the load when the face of the operator is directed in a direction different from the load even though the insertion rate is less than a predetermined threshold. Further, the processor detects the vertical edge of the work machine by performing a predetermined filter process on the image data photographed by the photographing device, identifies the part where the detected vertical edges are continuously connected as the claws of the work machine and calculates its length, and obtains the insertion rate based on the calculated length of the claws of the work machine and the predetermined total length of the claws of the work machine. Alternatively, the processor is a neural network model having an input layer that receives an input of predetermined image data, an intermediate layer that extracts a feature amount representing the category of the insertion state from the input image data, and an output layer that outputs an identification result based on the feature amount, and the insertion rate may be obtained by inputting the image data of the claws of the forklift photographed by the photographing device into a pre-trained model constructed by performing learning using data including an image representing the positional relationship between the claws of the forklift and the load.

[0014] Also, in the above extraction process, the processor has an input layer that receives an input of predetermined image data, an intermediate layer that extracts a feature amount representing the skeletal information of a person from the input image data, and an output layer that outputs an identification result based on the feature amount. The neural network model is a pre-trained model constructed by performing learning using data including an image representing a person. By inputting the captured image data into the pre-trained model, the driving behavior may be extracted. In this way, by using the pre-trained model to identify the part positions of the skeleton of the operator of the work machine in the captured image data and the driving behavior of the operator based on this, the driving behavior can be accurately extracted from the captured image data. Or, the processor has an input layer that receives an input of predetermined image data, an intermediate layer that extracts a feature amount representing the orientation of a person's face and / or the orientation of a person's arm from the input image data, and an output layer that outputs an identification result based on the feature amount. The neural network model is a pre-trained model constructed by performing learning using data including an image representing a person. By inputting the captured image data into the pre-trained model, the driving behavior may be extracted.

Effects of the Invention

[0015] According to the present disclosure, safety measures for preventing accidents of work machines can be taken based on the safe driving behavior of the operators of work machines.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Mode for Carrying Out the Invention

[0017] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The configurations of the following embodiments are examples, and the present disclosure is not limited to the configurations of the embodiments.

[0018] <First Embodiment> The outline of the safety device of the working machine in the first embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing the schematic configuration of the safety device of the working machine in the present embodiment. The safety device 100 of the working machine according to the present embodiment is installed in the working machine 10 and is a device for taking safety measures for preventing accidents of the working machine 10. Here, the working machine 10 in the present embodiment is a forklift. And the safety device 100 includes a photographing device 200, an image processing device 300, and a reporting device 400.

[0019] The imaging device 200 is a device installed on the work machine 10 for imaging the operator of the work machine 10, and has a function of receiving input of images such as still images and moving images. Specifically, it is realized by a camera using an image sensor such as Charged-Coupled Devices (CCD), Metal-oxide-semiconductor (MOS), or Complementary Metal-Oxide-Semiconductor (CMOS). And the imaging device 200 in this embodiment is composed of a hemispherical camera. In this embodiment, as shown in Fig. 1(a), it is installed on the upper surface of the driver's cab of the forklift which is the work machine 10, and the optical axis of the hemispherical camera is directed downward. In this way, by installing a hemispherical camera on the forklift, it is possible to image the operator of the forklift, the vehicle body of the forklift, and its surroundings over a wide range without lack of vision with as few cameras as possible.

[0020] The image processing device 300 acquires imaging image data representing an image of the operator of the work machine 10 captured by the imaging device 200, and executes predetermined processing based on the imaging image data.

[0021] Here, the image processing device 300 may be any electronic device as long as it has processing capabilities for arithmetic processing and processing such as data acquisition, generation, and update. For example, it is a computer. That is, the image processing device 300 can be configured as a computer having a processor such as a CPU or GPU, a main storage device such as a RAM or ROM, and an auxiliary storage device such as an EPROM, a hard disk drive, or a removable medium. The removable medium may be, for example, a USB memory or a disk recording medium such as a CD or DVD. The auxiliary storage device stores an operating system (OS), various programs, various tables, and the like.

[0022] The alarm device 400 is a device that outputs a predetermined alarm signal and / or a predetermined alarm. Here, the predetermined alarm includes a warning light, a warning sound, and the like. In this embodiment, as shown in FIG. 1(a), as the alarm device 400, for example, a Patlite (registered trademark) may be arranged on the upper surface of the driver's cab of the forklift. Then, the safety device 100 can warn the operator of the working machine 10 by, for example, the warning light emitted by the Patlite (registered trademark). Note that the alarm device 400 may include, for example, a Patlite (registered trademark) and a speaker. In this case, the safety device 100 can warn the operator of the working machine 10 by the warning light and the warning sound.

[0023] And in this embodiment, in addition to the warning by the above-mentioned alarm device 400, the working machine 10 may be decelerated or stopped. In this case, when the image processing device 300 determines that the driving behavior of the operator of the working machine 10 does not belong to the safe driving behavior pattern by the determination process described later, the operation of the working machine 10 can be stopped using a well-known technique.

[0024] Further, the safety device 100 may further include a display device 500. Here, the display device 500 is a device configured to be able to display the captured image data captured by the imaging device 200 and / or predetermined data obtained by processing based on the captured image data. Such a display device 500 is provided, for example, in the driver's cab of the forklift. Thereby, the operator of the forklift can visually recognize the dangerous behavior he / she has performed via the display device 500.

[0025] As described above, the safety device 100 of the working machine of the present disclosure can easily implement safety measures for preventing accidents of the working machine 10 by simply arranging the imaging device 200, the image processing device 300, and the alarm device 400 on the existing working machine 10.

[0026] Next, based on FIG. 1(b), the components of the image processing apparatus 300 will be described in detail. FIG. 1(b) is a diagram showing in more detail the components of the image processing apparatus 300 included in the safety apparatus 100 in the first embodiment.

[0027] The image processing apparatus 300 has a storage unit 302 and a control unit 303 as functional units. It loads a program stored in an auxiliary storage device into the working area of the main storage device and executes it. By controlling each functional unit through the execution of the program, each function that meets a predetermined purpose in each functional unit can be realized. However, some or all of the functions may be realized by a hardware circuit such as an ASIC or an FPGA.

[0028] The storage unit 302 is configured to include a main storage device and an auxiliary storage device. The main storage device is a memory in which a program executed by the control unit 303 and data used by the control program are expanded. The auxiliary storage device is a device in which a program executed by the control unit 303 and data used by the control program are stored.

[0029] Further, the storage unit 302 may store a pre-trained model described later. This pre-trained model can be used for the extraction process described later. Here, when the function of the image processing apparatus 300 is realized by a hardware circuit such as an FPGA, the pre-trained model may be stored in a memory built into the FPGA. Also, the storage unit 302 stores a predetermined safe driving behavior pattern described later.

[0030] The control unit 303 is a functional unit that controls the processes performed by the image processing apparatus 300. The control unit 303 can be realized by an arithmetic processing unit such as a CPU. The control unit 303 further includes four functional units: an acquisition unit 3031, an extraction processing unit 3032, a determination processing unit 3033, and a reporting control processing unit 3034. Each functional unit may be realized by executing a stored program by the CPU. Note that the control unit 303 functions as a processor according to the present disclosure by executing the processes of the acquisition unit 3031, the extraction processing unit 3032, the determination processing unit 3033, and the reporting control processing unit 3034. Then, when this processor is mounted on an integrated circuit constituting the image processing apparatus 300, the processor that executes the processes of the above functional units is installed in the working machine 10.

[0031] Here, the processing flow performed by the control unit 303 will be described with reference to FIG. 2. FIG. 2 is a flowchart showing the processing flow performed by the control unit 303 for preventing an accident of the working machine 10 in the safety device 100 according to the present embodiment. In the present embodiment, when the power of the safety device 100 is turned on, the execution of this flow is started, and it is repeatedly executed at a predetermined operation cycle during the operation of the safety device 100.

[0032] In this flow, first, in S101, the acquisition unit 3031 acquires captured image data representing an image of an operator of the working machine 10 captured by the imaging device 200. Here, the imaging device 200 and the image processing apparatus 300 each have a communication unit, and by connecting these communication units via a network, the acquisition unit 3031 can acquire the above-described captured image data. Note that the network may be wireless, wired, or a combination of wireless and wired.

[0033] Next, in S102, the extraction processing unit 3032 executes an extraction process of extracting the driving action of the working machine 10 by the operator from the captured image data. The details of this extraction process will be described later.

[0034] Next, in S103, the determination processing unit 3033 determines whether or not the above driving behavior belongs to a predetermined safe driving behavior pattern. If a negative determination is made in S103, the control unit 303 proceeds to the process of S104. If an affirmative determination is made in S103, the execution of this flow is terminated.

[0035] Here, in the determination process in S103, by comparing the predetermined safe driving behavior pattern stored in the storage unit 302 in advance with the image data including the driving behavior of the working machine 10 by the occupant extracted by the extraction process, it can be determined whether or not the driving behavior of the working machine 10 by the occupant belongs to the predetermined safe driving behavior pattern. Note that the above safe driving behavior pattern is the finger-pointing designation before the operation of the working machine 10 by the occupant and that the line of sight of the occupant during the operation of the working machine 10 is directed toward the traveling direction of the working machine 10, and the details thereof will be described later.

[0036] If a negative determination is made in S103, next, in S104, the reporting control processing unit 3034 executes the reporting control process. In the present embodiment, the reporting control processing unit 3034 causes a warning light to be output from the reporting device 400 (for example, Patlite (registered trademark)). Note that when the reporting device 400 is configured by, for example, a Patlite (registered trademark) and a speaker, the reporting control processing unit 3034 may cause a warning light to be output from the Patlite (registered trademark) and a warning sound to be output from the speaker. Further, in the present embodiment, in addition to the above, a process of operating the operation control device of the working machine 10 may be executed so that the operation of the working machine 10 is stopped. After the process of S104, the execution of this flow is terminated.

[0037] According to the processes described above, safety measures for preventing accidents of the working machine 10 can be taken based on the safe driving behavior of the occupant of the working machine 10.

[0038] (Extraction process) Next, the details of the extraction process executed by the extraction processing unit 3032 will be described. In the extraction process of the present embodiment, by inputting the photographed image data into the pre-trained model, the driving behavior of the work machine 10 by the operator is extracted. The pre-trained model is constructed by performing learning using data including images representing people.

[0039] Here, FIG. 3 is a diagram for explaining the identification result obtained from the input to the pre-trained model in the present embodiment and the neural network constituting the pre-trained model. In the present embodiment, a neural network model generated by deep learning is used as the pre-trained model. The pre-trained model 30 in the present embodiment includes an input layer 31 that receives an input of predetermined image data, an intermediate layer (hidden layer) 32 that extracts a feature amount representing the skeletal information of a person from the image data input to the input layer 31, and an output layer 33 that outputs an identification result based on the feature amount. In the example of FIG. 3, the pre-trained model 30 has one intermediate layer 32, the output of the input layer 31 is input to the intermediate layer 32, and the output of the intermediate layer 32 is input to the output layer 33. However, the number of intermediate layers 32 is not limited to one layer, and the pre-trained model 30 may have two or more intermediate layers 32.

[0040] Also, according to FIG. 3, each of the layers 31 to 33 includes one or more neurons. For example, the number of neurons in the input layer 31 can be set according to the input image data. Also, the number of neurons in the output layer 33 can be set according to the driving behavior that is the identification result.

[0041] Then, the neurons of adjacent layers are appropriately connected, and a weight (connection weight) is set for each connection based on the result of machine learning. In the example of FIG. 3, each neuron is connected to all the neurons of the adjacent layer, but the connection of neurons is not limited to such an example and can be set as appropriate.

[0042] Such a pre-trained model 30 is constructed by performing supervised learning using, for example, training data that is a pair of image data including an image representing a person and an image label representing the part positions of the person's skeleton. Specifically, a pair of a feature amount and a label is given to a neural network, and the weights of the connections between neurons are tuned so that the output of the neural network is the same as the label. In this way, a pre-trained model for learning the features of the training data and estimating the result from the input is inductively obtained.

[0043] Note that, as the image data used for learning to construct the pre-trained model, an image representing a person in a non-operating state or an image representing a person driving a working machine or the like may be used.

[0044] Also, the pre-trained model 30 may be constructed by performing unsupervised learning. For unsupervised learning, for example, domain adaptation, which is a type of transfer learning, can be used. According to this, a pre-trained model can be obtained without preparing a large amount of labeled training data.

[0045] Then, by inputting the photographed image data into such a pre-trained model 30, the part positions of the skeleton of the operator of the working machine 10 in the photographed image data are identified, and based on this, the driving behavior of the working machine 10 by the operator is extracted.

[0046] Here, FIG. 4 is a diagram illustrating the part positions of the skeleton of the operator of the working machine 10 identified by the pre-trained model 30 in the present embodiment. In FIG. 4, for the image of the operator included in the photographed image data, the arms, torso, and face, which are parts of the skeleton of the operator, are represented by dots and line segments.

[0047] Then, the extraction processing unit 3032 can extract the direction of the face of the operator of the working machine 10 (which direction among left, right, front, back, and diagonal is being faced) and the pointing action (which direction among left, right, front, back, and diagonal is being pointed by the direction of the arm) as the driving actions of the working machine 10 by the operator. In the example shown in FIG. 4, it is extracted that the face of the operator of the working machine 10 is facing forward.

[0048] Here, in the present embodiment, as described above, the functions of the image processing apparatus 300 can be realized by a hardware circuit such as an FPGA. In this case, the pre-trained model can be stored in the memory built into the FPGA. According to this, the extraction processing using the pre-trained model can be executed as quickly as possible, and thus, safety measures for preventing accidents of the working machine 10 can be taken more quickly.

[0049] As described above, according to the extraction processing of the present embodiment, by using the pre-trained model 30 to identify the part positions of the skeleton of the operator of the working machine 10 in the captured image data and the driving actions of the operator based on this, the driving actions can be accurately extracted from the captured image data.

[0050] In the above description of the extraction processing, an example in which the driving actions of the working machine 10 by the operator are extracted based on the part positions of the skeleton of the operator of the working machine 10 has been described. However, in the extraction processing in the present embodiment, the driving actions of the working machine 10 by the operator may be extracted by inputting the captured image data into the pre-trained model 30 and identifying the direction of the face and the direction of the arm of the operator of the working machine 10 in the captured image data. In this case, the pre-trained model 30 includes an input layer 31 that receives an input of predetermined image data, an intermediate layer (hidden layer) 32 that extracts feature amounts representing the direction of the face of a person and / or the direction of the arm of a person from the image data input to the input layer 31, and an output layer 33 that outputs an identification result based on the feature amounts. And such a pre-trained model 30 can be constructed, for example, by performing supervised learning using teacher data that is a set of image data including an image representing a person and a label of an image representing the direction of the face of the person and / or the direction of the arm of the person.

[0051] (Determination Process) Next, the details of the determination process executed by the determination processing unit 3033 will be described. As described above, the determination processing unit 3033 determines whether the extracted driving behavior belongs to a predetermined safe driving behavior pattern.

[0052] Here, the safe driving behavior pattern in the present embodiment is the pointing and calling before the operation of the work machine 10 by the occupant. And the above-mentioned pointing and calling is the pointing confirmation for the operation of the work machine 10, and is implemented when the work machine 10 starts, stops, or turns.

[0053] In this case, the above-mentioned extraction processing unit 3032 extracts the pointing action of the occupant as the driving behavior. Then, the determination processing unit 3033 determines whether the extracted pointing action of the occupant belongs to the implementation pattern of the pointing and calling. Note that the implementation pattern of the above-mentioned pointing and calling is predetermined and stored in the storage unit 302, and for example, the pointing action forward when the work machine 10 starts, the pointing action backward when the work machine 10 reverses, the pointing action to the stop line when the work machine 10 stops, and the pointing action to the left and right when the work machine 10 turns.

[0054] And the determination processing unit 3033 can determine, for example, whether the pointing action of the occupant is a forward pointing action when the work machine 10 starts. Note that the work machine 10 is equipped with vehicle sensors such as a speed sensor and an acceleration sensor, and the determination processing unit 3033 can recognize the start, stop, turn, etc. of the work machine 10 based on the detection values of the vehicle sensors.

[0055] Also, at this time, the determination processing unit 3033 may recognize information regarding the start, stop, turning, etc. of the working machine 10 based on the image data representing the surroundings of the working machine 10 captured by the imaging device 200. In this case, the imaging device 200 may be disposed at a position where it can capture the surroundings of the working machine 10 together with the operator of the working machine 10. And the image data representing the surroundings of the working machine 10 that can be captured by the imaging device 200 includes a stop line related to the start and stop of the working machine 10 and its position, a no-entry mark and its characters, and the determination processing unit 3033 can recognize these by image recognition processing. Then, the determination processing unit 3033, for example, recognizes a stop line based on the image data representing the surroundings of the working machine 10 captured by the imaging device 200, and at the timing when the working machine 10 reaches the stop line, can determine whether the pointing action of the operator is a forward-pointing action.

[0056] Also, the safe driving behavior pattern in the present embodiment may be that the line of sight of the operator while driving the working machine 10 is directed toward the traveling direction of the working machine 10.

[0057] In this case, the above extraction processing unit 3032 extracts the orientation of the operator's face as a driving action. And the determination processing unit 3033 determines whether the extracted orientation of the operator's face belongs to the traveling direction of the working machine 10. Note that the traveling direction of the working machine 10 can be recognized by a speed sensor that acquires the traveling speed and traveling direction of the working machine 10.

[0058] Then, based on the result of the above determination processing, when the warning control processing unit 3034 executes warning control processing, a warning can be given about the dangerous behavior of the operator in real time. In the case of post-discrimination based on the driving operation history as in the prior art, since the notification comes after a lapse of time from the dangerous behavior, there is little motivation for behavior correction and the dangerous behavior is likely to recur. In contrast, according to the present disclosure, by giving a warning about the dangerous behavior of the operator in real time, the suppression of the operator's behavior can work strongly and the recurrence can be reduced.

[0059] As described above, according to the present embodiment, safety measures for preventing accidents of the working machine can be taken based on the safe driving behavior of the operator of the working machine.

[0060] <Second Embodiment> The second embodiment will be described with reference to FIGS. 5 and 6. In the safety device 100 of the working machine according to the present embodiment, the imaging device 200 is configured to be able to image the surroundings of the forklift including the load moved by the forklift which is the working machine 10. And, when it is determined that the operator has not confirmed the state of insertion of the claws of the forklift into the load, during the operation of the forklift by the operator, the alarm signal or / and the alarm is output by the alarm device 400 in the safety device 100 of the working machine according to the present embodiment.

[0061] Here, FIG. 5 is a flowchart showing the processing flow performed by the control unit 303 for preventing an accident of the working machine 10 in the safety device 100 according to the present embodiment. In the present embodiment, when the power of the safety device 100 is turned on, the execution of this flow starts and is repeatedly executed at a predetermined calculation cycle during the operation of the safety device 100.

[0062] In this flow, first, in S201, the acquisition unit 3031 acquires imaging image data representing an image of the operator of the working machine 10, the load moved by the forklift which is the working machine 10, and the claws of the forklift, imaged by the imaging device 200.

[0063] Next, in S202, the extraction processing unit 3032 executes an extraction process of extracting the driving behavior of the working machine 10 by the operator from the imaging image data. In the present embodiment, the direction of the operator's face is extracted as the driving behavior.

[0064] Next, in S203, the control unit 303 acquires the insertion rate. Here, the insertion rate is the insertion rate of the claws of the forklift into the load and can be calculated based on the image data of the claws of the forklift imaged by the imaging device 200.

[0065] Specifically, the control unit 303 extracts the vertical component by performing Sobel filter processing on the above-described captured image data in the vertical and horizontal directions. Further, by subtracting the horizontal filter output from this, the diagonal component extracted by the vertical filter is removed. Thereby, the vertical edge of the forklift, which is the work machine 10, can be detected.

[0066] Then, the control unit 303 identifies the portion where the above-described vertical edges are continuously connected as the claws of the forklift, and calculates the length thereof based on the image pixels. Note that the length of the claws calculated here is the length of the claws of the portion of the forklift claws that are not inserted into the load. And the control unit 303 calculates the above-described insertion rate based on the length of the claws calculated in this way and the total length of the claws of the forklift that is predetermined and stored in the storage unit 302. For example, the insertion rate can be calculated by dividing the insertion length of the claws into the load, which is obtained by subtracting the length of the claws calculated as described above from the total length of the claws of the forklift, by the total length of the claws of the forklift.

[0067] Alternatively, the control unit 303 may also acquire the insertion rate using a pre-trained model. This pre-trained model is constructed by performing learning using data including an image representing the positional relationship between the claws of the forklift and the load, and has an input layer, an intermediate layer (hidden layer), and an output layer, similar to the pre-trained model 30 described in the explanation of FIG. 3 above.

[0068] Here, in the pre-trained model in the present embodiment, the intermediate layer extracts a feature amount representing the category of the insertion state of the claws of the forklift into the load from the input image data.

[0069] FIG. 6 is a diagram for explaining the category of the insertion state of the claws of the forklift into the load. Here, the above-described category is not a continuous insertion rate, but a category divided every certain ratio, such as when the insertion rate of the claws of the forklift into the load is 1 / 2 or 1 / 4. In FIG. 6, the case where the insertion state belongs to the 1 / 2 category is illustrated.

[0070] And such a pre-trained model is constructed by performing supervised learning using, for example, teacher data that is a pair of image data representing the positional relationship between the fork of the forklift and the cargo, and an image label representing the category of the insertion state of the fork of the forklift into the cargo. Specifically, a pair of a feature amount and a label is given to a neural network, and the weights of the connections between neurons are tuned so that the output of the neural network is the same as the label. In this way, a pre-trained model for learning the features of the teacher data and estimating the result from the input is inductively obtained.

[0071] Then, by inputting the image data of the fork of the forklift photographed by the photographing device 200 into such a pre-trained model, an insertion rate is obtained.

[0072] Then, returning to FIG. 5, next, in S204, the determination processing unit 3033 determines whether or not the above driving action belongs to a predetermined safe driving action pattern. And when a negative determination is made in S204, the control unit 303 proceeds to the process of S205, and when an affirmative determination is made in S204, the execution of this flow is terminated.

[0073] Here, the safe driving action pattern in the present embodiment is the confirmation by the operator of the insertion state of the fork of the forklift into the cargo. In the determination process in S204, based on the insertion rate obtained in the process of S203 and the direction of the face of the operator extracted in the process of S202, it is determined whether or not the operator has confirmed the insertion state of the fork of the forklift into the cargo. Specifically, the determination processing unit 3033 can determine that the operator has not confirmed the insertion state of the fork of the forklift into the cargo when the face of the operator is directed in a direction different from the cargo even though the insertion rate is less than a predetermined threshold value. Note that the above threshold value is, for example, 75% (3 / 4 with respect to the total length of the fork of the forklift).

[0074] And, when a negative determination is made in S204, next, in S205, the transmission control processing unit 3034 executes transmission control processing. Then, after the processing of S205, the execution of this flow ends.

[0075] And, as described above, according to this embodiment, safety measures for preventing accidents of the working machine can be taken based on the safe driving actions of the operator of the working machine.

[0076] <Other Modification Examples> The above embodiment is merely an example, and the present disclosure can be appropriately modified and implemented without departing from the gist thereof. For example, the processes and means described in the present disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0077] Also, the processes described as being performed by one device may be shared and executed by a plurality of devices. For example, the extraction processing unit 3032 may be formed in an integrated circuit separate from the image processing device 300. At this time, the separate integrated circuit is preferably configured to cooperate with the image processing device 300. Also, the processes described as being performed by different devices may be executed by one device. In the integrated circuit, how each function is realized by a hardware configuration can be flexibly changed.

[0078] The present disclosure can also be realized by supplying a computer program that implements the functions described in the above embodiments to a computer and causing one or more processors included in the computer to read and execute the program. Such a computer program may be provided to the computer by a non-transitory computer-readable storage medium connectable to the system bus of the computer. The non-transitory computer-readable storage medium includes, for example, any type of disk such as a magnetic disk (e.g., a floppy (registered trademark) disk, a hard disk drive (HDD), etc.), an optical disk (e.g., a CD-ROM, a DVD disk, a Blu-ray disk, etc.), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, and any type of medium suitable for storing electronic instructions.

Explanation of Signs

[0079] 10 ··· Machine tool 100 ··· Safety device 200 ··· Imaging device 300 ··· Image processing device 303 ··· Control unit 400 ··· Transmitting device

Claims

1. An imaging device installed on a work machine and configured to image an operator of the work machine; a processor configured to obtain imaging image data representing an image of the operator imaged by the imaging device and execute a predetermined process based on the imaging image data; a notification device configured to output a predetermined alarm signal and / or a predetermined notification; wherein the processor executes an extraction process of extracting a driving action of the work machine by the operator from the imaging image data, a determination process of determining whether or not the extracted driving action belongs to a predetermined safe driving action pattern, and a notification control process of causing the notification device to output the alarm signal and / or the notification when it is determined by the determination process that the driving action does not belong to the safe driving action pattern, a safety device for a work machine.

2. wherein the safe driving action pattern is a pointing call before operation of the work machine by the operator, and the processor extracts a pointing action of the operator as the driving action, determines whether or not the extracted pointing action of the operator belongs to an execution pattern of the pointing call, and causes the notification device to output the alarm signal and / or the notification when it is determined that the pointing action of the operator does not belong to the execution pattern of the pointing call, the safety device for a work machine according to claim 1.

3. wherein the work machine includes a speed sensor configured to obtain a traveling speed and a traveling direction of the work machine, the safe driving action pattern is that the line of sight of the operator during operation of the work machine faces the traveling direction of the work machine, and the processor extracts the orientation of the face of the operator as the driving action, determines whether or not the extracted orientation of the face of the operator belongs to the traveling direction of the work machine obtained by the speed sensor, and causes the notification device to output the alarm signal and / or the notification during operation of the work machine by the operator when it is determined that the orientation of the face of the operator does not belong to the traveling direction of the work machine, the safety device for a work machine according to claim 1.

4. wherein the work machine is a forklift, the imaging device is configured to be able to image the surroundings of the forklift including a load moved by the forklift, the safe driving action pattern is confirmation of the insertion state of the claws of the forklift into the load by the operator, and the processor extract the orientation of the driver's face as the driving action, acquire the insertion rate of the forklift's claws into the load based on the image data of the forklift's claws captured by the imaging device, determine whether the driver is checking the insertion state of the forklift's claws into the load based on the insertion rate and the orientation of the driver's face, when it is determined that the driver has not checked the insertion state, during the operation of the work machine by the driver, cause the warning device to output the warning signal and / or the notification, The safety device for a work machine according to claim 1.

5. The processor, in the determination process, when the orientation of the driver's face is directed in a direction different from the load even though the insertion rate is less than a predetermined threshold, determine that the driver has not checked the insertion state of the forklift's claws into the load, The safety device for a work machine according to claim 4.

6. The processor, detect the vertical edge of the work machine by performing a predetermined filter process on the image data captured by the imaging device, identify the part where the detected vertical edges are continuously connected as the claws of the work machine and calculate its length, and acquire the insertion rate based on the calculated length of the claws of the work machine and the predetermined total length of the claws of the work machine, The safety device for a work machine according to claim 5.

7. The processor, a neural network model having an input layer that receives an input of predetermined image data, an intermediate layer that extracts a feature amount representing the category of the insertion state from the input image data, and an output layer that outputs an identification result based on the feature amount, and the insertion rate is acquired by inputting the image data of the forklift's claws captured by the imaging device into a pre-trained model constructed by performing learning using data including an image representing the positional relationship between the forklift's claws and the load, The safety device for a work machine according to claim 5.

8. in the extraction process, the processor, A neural network model having an input layer for receiving an input of predetermined image data, an intermediate layer for extracting a feature amount representing the skeletal information of a person from the input image data, and an output layer for outputting an identification result based on the feature amount, wherein the driving behavior is extracted by inputting the captured image data into a pre-trained model constructed by performing learning using data including an image representing a person. The safety device for a working machine according to claim 1.

9. In the extraction process, the processor A neural network model having an input layer for receiving an input of predetermined image data, an intermediate layer for extracting a feature amount representing the orientation of a person's face and / or the orientation of a person's arm from the input image data, and an output layer for outputting an identification result based on the feature amount, wherein the driving behavior is extracted by inputting the captured image data into a pre-trained model constructed by performing learning using data including an image representing a person. The safety device for a working machine according to claim 1.

Images