Work site monitoring system

Abstract

To provide a work site monitoring system capable of realizing safety improvement.SOLUTION: A work site monitoring system in accordance with an embodiment of the disclosure comprises a control device configured to determine whether or not there occurs an abnormality that may affect a work site on a basis of information from an imaging device or a sound collection device provided at the work site or the vicinity of the work site, and to output information indicating the occurrence of the abnormality when it is determined that the abnormality occurs.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a work site monitoring system. [Background technology]

[0002] Conventionally, there have been proposed techniques for recording the surrounding conditions in preparation for occurrence of an abnormality at a work site. For example, there has been proposed a technique for recording image information around the shovel and information about sounds around the shovel as information about the surrounding conditions of the shovel (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 025123 Summary of the Invention [Problem to be solved by the invention]

[0004] However, since work sites are often subject to land development and the like, there is a possibility that they may be affected by natural disasters. For this reason, there is a demand for technology that can appropriately detect abnormalities at or near work sites.

[0005] One aspect of the present invention provides a technology that detects abnormalities that may affect a work site based on information from a sound collection device or an imaging device, and enables notification to workers, etc. [Means for solving the problem]

[0006] A work site monitoring system according to one embodiment of the present invention includes a control device configured to determine whether or not an abnormality has occurred that may affect the work site based on information from an imaging device or a sound collecting device installed at or near the work site, and if it is determined that an abnormality has occurred, to output information indicating that an abnormality has occurred. Effect of the Invention

[0007] According to one aspect of the present invention, improved safety is achieved. [Brief description of the drawings]

[0008] [Figure 1] 1 is a schematic diagram showing an example of a work site anomaly monitoring system according to a first embodiment; [Diagram 2] FIG. 2 is a diagram illustrating a configuration of a management device according to the first embodiment. [Diagram 3] 3 is a diagram illustrating an example of a table structure of a sound feature information storage unit according to the first embodiment. FIG. [Figure 4] FIG. 13 is a diagram illustrating a concept of how a trained model according to the first embodiment is installed in a management device. [Diagram 5] FIG. 2 is a diagram illustrating an example of the situation around the work site at the work end time and the situation around the work site at the work start time according to the first embodiment. [Figure 6] 3A to 3C are diagrams showing an example of a sound signal acquired from the sound collection device according to the first embodiment. [Figure 7] 13 is a diagram illustrating an example of a situation around a work site at a first time during work according to a second embodiment, and a situation around the work site at a second time during work. FIG. [Figure 8] FIG. 11 is a schematic diagram showing an example of a work site anomaly monitoring system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is an example, not a limitation of the invention, and all features and combinations described in the embodiment are not necessarily essential to the invention. In addition, the same or corresponding components in each drawing are denoted by the same or corresponding reference numerals, and the description may be omitted.

[0010] (First embodiment) First, an overview of a work site abnormality monitoring system SYS according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing an example of a work site abnormality monitoring system SYS according to the first embodiment.

[0011] <Equipment that constitutes the excavator movement control system> As shown in FIG. 1, the work site abnormality monitoring system SYS according to the first embodiment includes a management device 500, an alarm device 700, a mobile communication terminal 800, sound collection devices 611-617, and image capture devices 711-718.

[0012] The work site abnormality monitoring system SYS is used to detect abnormalities at the work site WS and in the vicinity of the work site WS.

[0013] The work site WS is an area where work is performed by work machines, workers, etc. For example, at the work site WS, various works are performed by various work machines such as the excavator 100 and workers from the work start time to the work finish time.

[0014] The work site WS may be, for example, a work site for cutting earth, which is a type of construction work. Any type of work may be performed at the work site WS, and may include, for example, excavation work, slope shaping, and the like.

[0015] In conventional cutting works, there is a tendency for abnormalities such as slope collapse to occur. Inspections have been carried out visually at work sites to predict slope collapses, etc. However, visual inspections can lead to human errors such as overlooking abnormalities or improper communication.

[0016] On the other hand, a technique for installing various sensors has been proposed to measure the behavior of slope collapse and the like. For example, in order to prevent rock falls and the like at work sites, a technique for installing various sensors (e.g., crack displacement meters, extensometers, inclinometers, etc.) on target stones and a technique for installing rock fall detection devices on protective nets and the like have been proposed. Furthermore, a technique for measuring the distance to a slope with an optical range finder has also been proposed. Furthermore, a technique for measuring the water level underground has also been proposed to detect landslides and the like. Thus, various techniques have been proposed to detect abnormalities such as slope collapse at or around work sites, but the various sensors are expensive and the installation burden is large.

[0017] Therefore, in this embodiment, an example will be described in which the management device 500 uses sound collection devices 611-617 and image capture devices 711-718 to detect abnormalities that may affect the work site WS at or around the work site WS.

[0018] The sound collecting devices 611 to 617 are, for example, microphones, and are configured to collect sounds generated in the surroundings.

[0019] The sound collecting devices 611-612 are provided to monitor the work site WS, and the sound collecting devices 613-617 are provided to monitor the periphery of the work site WS (in other words, outside the work site). In this embodiment, an example in which the sound collecting devices 611-617 are provided at the work site WS and the periphery of the work site WS will be described. However, the present invention is not limited to the method of providing the sound collecting devices at the work site WS and the periphery of the work site WS, and the sound collecting devices may be provided at either the work site WS or the periphery of the work site WS.

[0020] The imaging devices 711 to 718 are, for example, infrared cameras with a very wide angle of view. Note that the imaging devices 711 to 718 are not limited to infrared cameras, and may be monocular cameras, stereo cameras, or the like.

[0021] The imaging devices 711-712 are provided to monitor the work site WS, and the imaging devices 713-718 are provided to monitor the periphery of the work site WS (in other words, outside the work site). In this embodiment, an example in which the imaging devices 711-718 are provided at the work site WS and the periphery of the work site WS will be described. However, the present invention is not limited to the method of providing the imaging devices at the work site WS and the periphery of the work site WS, and the imaging devices may be provided at either the work site WS or the periphery of the work site WS.

[0022] The management device 500 is configured to determine whether or not an abnormality that may affect the work site WS has occurred based on information transmitted from the sound collection devices 611-617 and the image capture devices 711-718 installed at or near the work site WS, and when it is determined that an abnormality has occurred, output information indicating that an abnormality has occurred to the notification device 700 or the mobile communication terminal 800. The management device 500 outputs the information to the notification device 700 or the mobile communication terminal 800, so that a person related to the work site WS can recognize that an abnormality has occurred at the work site WS or near the work site WS. In this embodiment, an example is described in which the sound collection devices 611-617 and the image capture devices 711-718 are provided, but only one of the sound collection devices and the image capture devices may be provided.

[0023] The notification device 700 is a device that issues a notification to an operator, a manager, or the like when an abnormality is detected by the management device 500, based on a signal received from the management device 500, to make the operator or the manager aware of the occurrence of the abnormality. Any method of notification may be used, and for example, a sound may be output using a speaker or the like, or the occurrence of the abnormality may be displayed on a display device.

[0024] The notification method is not limited to the method of outputting audio or video near the work site WS, and notification may be sent to a terminal carried by a worker, manager, etc. The mobile communication terminal 800 is a terminal carried by a worker, etc., capable of wireless communication. The management device 500 may transmit information indicating that an abnormality has occurred to the mobile communication terminal 800. When the mobile communication terminal 800 receives the information, it outputs by audio or on a screen that an abnormality has occurred.

[0025] In this embodiment, the management device 500 monitors whether or not an abnormality occurs between the work end time and the work start time. Next, the configuration of the management device 500 will be described.

[0026] 2 is a diagram illustrating an example of the configuration of a management device 500 according to this embodiment. As shown in FIG. 2, the management device 500 includes a control device 501 and a storage medium 502.

[0027] The storage medium 502 is a non-volatile storage medium and includes a sound feature information storage unit 521, a trained model LM, and a position information storage unit 522.

[0028] The control device 501 includes, for example, a CPU (Central Processing Unit) (not shown) and performs various controls by executing a program (not shown) stored in a storage medium 502. The control device 501 according to this embodiment includes an acquisition unit 511, a sound determination unit 512, a control command generation unit 513, a difference extraction unit 514, an image determination unit 515, a map generation unit 516, and an output unit 517.

[0029] The sound feature information storage unit 521 stores feature information of an audio signal in association with an abnormality that may have occurred when the audio signal indicated by the feature information is detected.

[0030] FIG. 3 is a diagram illustrating a table structure of the sound feature information storage unit 521 according to the present embodiment. As shown in FIG. 3, the sound feature information storage unit 521 stores sound patterns, types, and abnormalities in association with each other. The sound patterns indicate sound patterns that can be detected by the sound collection devices 611 to 617. The sound patterns indicate, for example, patterns of changes in volume (sound pressure level) with time transition. Note that in this embodiment, the feature information of the audio signal used for abnormality detection is not limited to the sound pattern, but may be the frequency characteristics of the sound. The type indicates the type of the surrounding situation when the sound indicated by the sound pattern is detected. The abnormality indicates whether the situation is abnormal or not. Note that in this embodiment, a case where a correspondence relationship is maintained will be described, but the present invention is not limited to this method. For example, whether an abnormality occurs may be determined using a trained model that has been machine-learned based on an audio signal when an abnormality occurs.

[0031] By using the sound feature information storage unit 521 as shown in FIG. 3, it is possible to estimate that the generated sound is a root breaking sound or a landslide precursor based on the acquired audio signal. In this case, it can be considered as an abnormality (a precursor to a disaster). Similarly, by using the sound feature information storage unit 521, it is possible to estimate that the generated sound is a wind sound or a rain sound based on the acquired audio signal. In this case, it can be considered as no abnormality. In this way, natural sounds that have no effect on the work site WS, such as wind sounds or rain sounds, are associated so as to be determined as no abnormality.

[0032] Furthermore, the sound feature information storage unit 521 may associate the sound pattern of the engine sound of the shovel 100 with the type "engine sound" and the abnormality "no abnormality". Furthermore, the sound pattern of the work sound by the worker may be associated with the type "work sound" and the abnormality "no abnormality". In this way, when a sound related to work by the worker or the shovel 100 is detected, it can be deemed that no abnormality exists. By storing the above-mentioned correspondence relationship in the sound feature information storage unit 521, natural sounds that have no effect on the work site WS or sounds related to work by the worker or work machine are prevented from being determined to be abnormal, so that work interruptions due to erroneous detection can be prevented.

[0033] The trained model LM outputs the type of situation occurring in the difference based on the difference between a plurality of pieces of image information captured by the imaging devices 711 to 718, and also outputs a determination result as to whether or not an abnormality has occurred. The type of situation that is output includes the type of abnormality that may affect the work site WS.

[0034] FIG. 4 is a diagram illustrating a concept of how the trained model LM according to this embodiment is installed in the management device 500.

[0035] 4, the information processing device 1401 performs machine learning. In this embodiment, the information processing device 1401 is provided separately from the management device 500.

[0036] A dataset 1451 used for machine learning includes image information showing the difference between multiple images, and is tagged with an attribute showing the type of situation represented by the image information showing the difference and an attribute showing whether the situation is abnormal or not. In other words, the dataset 1451 is a group of teacher data for generating a trained model LM that outputs a "type of situation" and "presence or absence of abnormality" when "image information showing the difference" is input.

[0037] Among the "types of situation," types of abnormalities that may affect the work site WS include, for example, falling rocks, cracks or shifts in slopes, slope collapse, fallen trees, leaning trees, spring water, dead trees, etc. When any of these situations are detected, an attribute is assigned so that it is determined that "anomaly exists."

[0038] Among the "types of situations," types of situations that are not abnormal include misalignment of the cone position, swaying of tree branches or leaves, movement of the shovel, movement of the shovel, movement of the worker, movement of the worker, changes in the shape of the ground due to cutting, filling, and excavation, etc. When any of these situations are detected, an attribute is added so that it is determined to be "normal."

[0039] The learning unit 1411 performs machine learning using the data set, for example, learning by backpropagation using a neural network, to generate a trained model LM. The trained model LM generated by the learning unit 1411 is stored in the storage medium 502 of the management device 500.

[0040] The trained model LM is mainly composed of a neural network 401. The neural network 401 of the trained model LM may be a so-called deep neural network having one or more intermediate layers (hidden layers) between an input layer and an output layer. In the neural network, a weighting parameter representing the connection strength with a lower layer is defined for each of a plurality of neurons constituting each intermediate layer. The neural network is configured in such a manner that a neuron in each layer outputs the sum of values ​​obtained by multiplying each of the input values ​​from a plurality of neurons in the upper layer by a weighting parameter defined for each neuron in the upper layer to a neuron in the lower layer through a threshold function.

[0041] In this way, the trained model LM is subjected to machine learning by the learning unit 1411, specifically, deep learning, and as a result, the weighting parameters of the neural network are optimized.

[0042] The trained model LM performs machine learning using the above dataset to obtain input signals (x1, x2, ..., x L1 ), an output signal (y1, y2, ..., y LN ) can be output.

[0043] The position information storage unit 522 stores information on the positions at which the sound collection devices 611-617 and the imaging devices 711-718 are installed. The position information storage unit 522 also stores the imaging direction of each of the imaging devices 711-718.

[0044] The acquiring unit 511 acquires audio signals from each of the sound collecting devices 611 to 617. The acquiring unit 511 also acquires image information from each of the imaging devices 711 to 718.

[0045] For example, the acquisition unit 511 acquires a sound signal from each of the sound collection devices 611 to 617 during the period from the work end time to the work start time.

[0046] Furthermore, the acquisition unit 511 acquires image information captured by each of the imaging devices 711-718 at the work end time, and image information captured by each of the imaging devices 711-718 at the work start time.

[0047] The sound determination unit 512 determines whether the sound signals acquired from each of the sound collection devices 611-617 during the period from the work end time to the work start time match the sound patterns, etc. stored in the sound feature information storage unit 521 by a predetermined standard or more.

[0048] When the sound determination unit 512 determines that the sound pattern or the like matches a predetermined standard or more, it determines the type of the surrounding situation of each of the sound collection devices 611 to 617 and the presence or absence of an abnormality based on the type and abnormality associated with the sound pattern or the like. In this embodiment, an example of determining whether or not the sound pattern matches a predetermined standard or more will be described as a method of determining whether or not an abnormality has occurred. However, this embodiment does not limit the determination method to this method, and it is sufficient to determine whether or not information about the sound (for example, frequency characteristics) meets a predetermined standard. The management device 500 according to this embodiment can make the manager or worker know what kind of situation is occurring around the work site WS by determining the type of the surrounding situation. Therefore, the management device 500 can make the manager or worker quickly take action (for example, evacuation, etc.) corresponding to the situation occurring around the work site WS. Therefore, it is possible to improve safety. In this way, in this embodiment, it is possible to detect an abnormality based on a sound signal, so that it is possible to improve safety.

[0049] When the sound determination unit 512 determines that an abnormality has occurred, the control command generation unit 513 may generate a control command for starting imaging for the imaging devices 711-718 capable of imaging the periphery of the sound collection devices 611-617 determined to have the abnormality. The imaging devices 711-718 according to this embodiment are, for example, infrared cameras, so that they can monitor the surroundings even at night. When the imaging devices 711-718 start imaging by receiving a control command, they may transmit image information to the management device 500. Then, the management device 500 may store the image information transmitted from the imaging devices 711-718, or may use the image information to detect an abnormality. The method of detecting an abnormality may be a method described later, and a description thereof will be omitted.

[0050] In this manner, in this embodiment, it is possible to combine detection of an abnormality by the sound collection devices 611-617 and detection of an abnormality by the imaging devices 711-718. For example, when the management device 500 performs the above-described control and determines that an abnormality has occurred in the sound collection devices 611-617, it is possible to start imaging by the imaging devices 711-718 and perform abnormality detection. This makes it easier to grasp the surrounding situation when a sound signal suspected to be abnormal is detected.

[0051] The difference extraction unit 514 determines whether there is an area where there is a difference between the image information captured by each of the imaging devices 711-718 at the work end time and the image information captured by each of the imaging devices 711-718 at the work start time.

[0052] Then, if the difference extraction unit 514 determines that there is an area where a difference occurs between the image information captured by each of the imaging devices 711-718 at the work end time and the image information captured by each of the imaging devices 711-718 at the work start time, it extracts the area where a difference occurs from the image information captured at the work start time as difference image information.

[0053] In this embodiment, when an abnormality is detected by sound determination unit 512, if imaging by imaging devices 711-718 is started after the work end time by the above-mentioned control, difference extraction unit 514 may determine whether or not there is an area in which a difference exists by using image information captured by imaging devices 711-718 after imaging has started, instead of image information captured by each of imaging devices 711-718 at the work start time.

[0054] The image determination unit 515 causes the control device 501 (an example of a computer) to function with the trained model LM. Then, the image determination unit 515 uses the trained model LM to which differential image information indicating the difference between a plurality of pieces of image information is input, to output the type of situation representing the region and the determination result of whether or not an abnormality has occurred. In this embodiment, by using the trained model LM that has been appropriately machine-learned, it is possible to improve the accuracy of detecting an abnormality according to the surrounding situation.

[0055] In this embodiment, a method of determining whether or not there is an abnormality that affects a work site based on differences in multiple image information using a trained model LM will be described. However, this embodiment is not limited to a method using a trained model LM, and any method of determining whether or not there is an abnormality based on differences in multiple image information may be used. In this manner, in this embodiment, it is possible to detect an abnormality that has occurred at or near the work site based on differences in image information, thereby improving safety.

[0056] When either the sound determination unit 512 or the image determination unit 515 determines that an abnormality has occurred, the map generation unit 516 refers to the position information storage unit 522 and generates map information indicating the position where the abnormality has occurred based on the position of the sound collection device 611-617 or the imaging device 711-718 that detected the abnormality.

[0057] For example, the map generating unit 516 refers to the position information storage unit 522 and generates map information indicating the position corresponding to the sound collecting device 611-617 that detected the sound signal determined to be abnormal as the position where the abnormality occurs. When multiple sound collecting devices 611-617 detect the sound signal determined to be abnormal, the map generating unit 516 may estimate the position where the abnormality occurs based on the volume (sound pressure level) of the sound signal detected by the multiple sound collecting devices 611-617 and the positions of the multiple sound collecting devices 611-617, and generate map information indicating the estimated position.

[0058] Furthermore, when there is an area where an abnormality occurs in the image information between the work end time and the work start time, the map generating unit 516 identifies the position information of the imaging devices 711-718 that captured the image information by referring to the position information storage unit 522. Then, the map generating unit 516 estimates the position where the abnormality occurred based on the position information of the imaging devices 711-718, the coordinates of the area where the abnormality occurs in the image information, and the imaging direction of the imaging devices 711-718, and generates map information indicating the position.

[0059] When sound determination unit 512 or image determination unit 515 determines that an abnormality has occurred, output unit 517 outputs map information indicating the occurrence of the abnormality (whether there is an abnormality), the type of abnormality, and the location where the abnormality has occurred to one or more of alarm device 700 and mobile communication terminal 800. In this embodiment, outputting the map information makes it easy for a worker or manager to grasp the location where the abnormality has occurred, making it easier to decide on future actions (e.g., evacuation or repair work). This makes it possible to improve safety and work efficiency at the same time.

[0060] The output method may be changed based on preset information. For example, if an abnormality occurs between the work end time and the work start time, and if it is determined that the abnormality is urgent, the output unit 517 may output to the mobile communication terminal 800 a message indicating that an abnormality with urgent has occurred. In addition, if an abnormality without urgent has occurred, the output unit 517 may not output to the mobile communication terminal 800 a message indicating that an abnormality has occurred. If an abnormality without urgent has occurred, the management device 500 may output from the notification device 700 at the work start time a message indicating that an abnormality has occurred, the type of the abnormality, and map information indicating the location where the abnormality has occurred.

[0061] Furthermore, the management device 500 may be able to set the output destination of information relating to anomalies depending on the type of anomaly.

[0062] In addition, when an abnormality is determined by sound determination unit 512, output unit 517 may output the control command generated by control command generation unit 513 to imaging devices 711-718 capable of capturing images of the area around the position where the abnormality is determined to have occurred.

[0063] This allows, for example, when the sound determination unit 512 determines that a sound pattern corresponding to a "landslide precursor" has been detected, the imaging devices 711-718 to start imaging the area around the position where the sound indicating the "landslide precursor" was generated.

[0064] <Examples of anomaly detection> Next, a specific example of an abnormality occurring between the work end time and the work start time will be described. That is, the management device 500 judges whether an abnormality has occurred based on image information captured at the work end time and image information captured at the work start time at the work site. Note that, although this embodiment will describe an example of capturing images at the work end time and the work start time, it is not limited to a method of capturing images exactly at the work end time and the work start time, and it is sufficient to capture images near the work end time and the work start time.

[0065] FIG. 5 is a diagram illustrating an example of the situation around the work site WS at the work end time and the situation around the work site WS at the work start time according to this embodiment.

[0066] 5 shows a bird's-eye view of the surroundings of the work site WS at the work end time of 18:00. As shown in the situation 1501, sound collecting devices 611-612 and imaging devices 711-712 are provided inside the work site WS.

[0067] As shown in the situation 1501, a slope SL1 exists near the work site WS. The slope SL1 is a slope that becomes higher from the vicinity of the work site WS. In this embodiment, the slope SL1 is monitored from the downstream side (near the work site WS) by the sound collecting devices 613-614 and the imaging devices 713-714.

[0068] Also, the sound collector 615 monitors whether or not abnormal noise is generated from the middle of the slope SL1.

[0069] Furthermore, the slope SL1 is monitored from the upstream side by sound collectors 616 to 617 and imaging devices 715 and 717.

[0070] In the example shown in situation 1501, the direction of the arrow indicates an even higher altitude, although this is not shown.

[0071] Therefore, in order to monitor for abnormalities (such as falling rocks or slope collapses) in high altitude areas, monitoring is performed using sound collecting devices 616 to 617 and imaging devices 716 and 718.

[0072] The above-mentioned imaging devices 711 to 718 capture images at least at the work end time of 18:00. On the other hand, the sound collecting devices 611 to 617 start capturing sounds occurring in the surroundings from the work end time of 18:00.

[0073] The sound collection devices 611-617 output audio signals indicating the acquired sounds to the management device 500. As a result, the sound determination unit 512 of the management device 500 refers to the sound patterns and the like stored in the sound feature information storage unit 521 during the period from the work end time 18:00 to the work start time 9:00, and determines whether or not an abnormality has occurred in the acquired audio signals.

[0074] Fig. 6 is a diagram showing an example of audio signals acquired from sound collection devices 611 to 617 according to this embodiment. In the example shown in Fig. 6, the vertical axis indicates volume and the horizontal axis indicates time. For example, it is assumed that a change in sound as shown in Fig. 6 occurs in the audio signals acquired by sound collection devices 614, 615, and 617.

[0075] The sound determination unit 512 determines that the change in volume in the area indicated by the reference symbol 1601 approximately matches the sound pattern stored in the sound feature information storage unit 521. As a result, the sound determination unit 512 determines that the type that matches the sound pattern is "precursor to landslide" and the abnormality is "abnormal."

[0076] Based on the position information of the sound collecting devices 614, 615, and 617, the map generating unit 516 infers that the area AN1 in FIG. 5 is the location where the abnormality has occurred, and generates map information indicating that location.

[0077] The output unit 517 may output, together with the generated map information, via the notification device 700, information that an abnormality such as a "landslide precursor" has occurred. For example, when there is a possibility that a person is present near the area AN1, the notification device 700 may output a warning to evacuate from the area AN1. The notification device 700 may also notify the mobile communication terminal 800 of the manager of the work site WS that a "landslide precursor" has occurred. By performing the above-mentioned processing, when a landslide precursor or the like occurs, measures such as evacuation can be taken before the landslide occurs.

[0078] 5 shows the surroundings of the work site WS from a bird's-eye view at the work start time of 9:00. The arrangement of the sound collecting devices 611-617 and the imaging devices 711-718 is the same as in the situation 1501.

[0079] The difference extraction unit 514 extracts difference image information indicating an area where a difference occurs between the image information captured at the work end time of 18:00 and the image information captured at the work start time of 9:00. In the example shown in Fig. 5, the difference extraction unit 514 extracts first difference image information indicating an area DR1 where the tree TR1 is captured, second difference image information indicating an area DF2 where the tree TR2 is captured, third difference image information indicating an area DF3 where the tree TR3' is captured, and fourth difference image information indicating an area DF4 where a part of the slope SL1' is captured.

[0080] The image judgment unit 515 inputs each of the first difference image information and the second difference image information into the trained model LM, and receives an output result from the trained model LM indicating that the type of situation is "swaying tree branches or leaves" and that "no abnormality" is obtained.

[0081] The image determination unit 515 inputs the third difference image information to the trained model LM, and receives, as an output result from the trained model LM, the type of the situation being "fallen tree" and "abnormality present."

[0082] The image determination unit 515 inputs the fourth difference image information to the trained model LM, and receives from the trained model LM the output result that the type of situation is "slope collapse" and "abnormality present."

[0083] When there is an area where an abnormality occurs in the image information between the work end time and the work start time, the map generating unit 516 refers to the position information storage unit 522 and identifies the position information of the imaging device 714 that captured the area where the abnormality occurred (area DF3, area DF4).Then, the map generating unit 516 infers the area where the abnormality occurred (area DF3, area DF4) based on the position information of the imaging device 714, the coordinates of the area where the abnormality occurs (area DF3, area DF4) in the captured image information, and the imaging direction of the imaging device 714, and generates map information showing the area (area DF3, area DF4).

[0084] The output unit 517 outputs information indicating that a "slope collapse" and a "fallen tree" have occurred, and map information indicating the areas where these abnormalities have occurred (areas DF3 and DF4), to one or more of the notification device 700 and the mobile communication terminal 800. Therefore, workers working at the work site WS can recognize that a "slope collapse" and a "fallen tree" have occurred in the vicinity of the work site WS, and can carry out work to address the abnormalities.

[0085] In addition, in the work site abnormality monitoring system SYS according to the present embodiment, an example has been described in which, when an abnormality is detected by a sound signal, imaging is started by an imaging device whose imaging range includes the area where the abnormality is detected, but when an abnormality is detected by a sound signal, imaging by the imaging device may not be started for the range where the abnormality is detected. In other words, this embodiment is not limited to a method of combining abnormality detection by a sound signal and abnormality detection by a captured image. In this embodiment, an example is described in which the sound collection devices 611-617 and the imaging devices 711-718 are provided, but the work site abnormality monitoring system may be provided with only a sound collection device and perform only abnormality detection based on the sound signal detected by the sound collection device, or may be provided with only an imaging device and perform only abnormality detection based on the difference in image information captured by the imaging device.

[0086] In this embodiment, an example has been described in which the management device 500 judges whether an abnormality has occurred based on whether there is a difference between image information captured at the end time of work and image information captured at the start time of work at the work site. By making this judgment, it is possible to detect an abnormality that has occurred between the end of work and the start of work. Therefore, workers and managers can recognize an abnormality that has occurred at the work site WS and around the work site WS before starting work. Therefore, work can be started taking into consideration an abnormality that has occurred while work was not being performed. Therefore, it is possible to improve safety and work efficiency.

[0087] (Modification of the first embodiment) In the above embodiment, the abnormality occurring between the work end time and the work start time is a slope collapse. However, the above embodiment shows an example of an abnormality that can be detected between the work end time and the work start time, and other abnormalities may be detected. For example, the intrusion of one or more of a suspicious person and an animal into the work site WS may be detected as an abnormality occurring between the work end time and the work start time.

[0088] In this modification, the imaging devices 711-712 may capture images during the period from the work end time to the work start time. Also, when the management device 500 determines that an abnormality has occurred in the audio signals acquired from the sound collection devices 611 and 612, it may instruct the imaging devices 711-712 to start capturing images.

[0089] The trained model LM is assumed to be a model for which machine learning has been performed by the information processing device 1401 using a data set for identifying one or more of suspicious persons and animals.

[0090] The data set includes attributes such as human intrusion, animal intrusion, etc. as "type of situation" together with the differential image information. The data set is assigned attributes so that when these situations are detected, it is determined that "anomaly exists." In addition, attributes may be assigned so that workers are not included in the detection target of "human intrusion."

[0091] As a result, the image determination unit 515 inputs the difference image information based on the multiple image information captured from the work end time to the trained model LM, and outputs from the trained model LM a determination result of whether or not a situation of "intrusion of a person" or "intrusion of an animal" has occurred based on the difference between the multiple image information between the work end time and the work start time, as well as a determination result of whether or not an abnormality has occurred due to the situation. The subsequent processing is the same as in the above-mentioned embodiment, and a description thereof will be omitted. In this embodiment, the intrusion of a suspicious person or an animal can be detected, so that safety can be improved. Note that this modified example is not limited to an example in which the intrusion of a suspicious person or an animal is detected between the work end time and the work start time, and may be performed during work.

[0092] Second Embodiment In the above-described embodiment, an example of detecting an abnormality that occurs between the work end time and the work start time has been described. However, the above-described embodiment is not limited to a method of detecting an abnormality that occurs between the work end time and the work start time. Therefore, in the second embodiment, an example of determining whether or not an abnormality occurs during work will be described.

[0093] <Examples of anomaly detection> Next, a specific example of a case where an abnormality occurs during work will be described. That is, the management device 500 determines whether or not an abnormality has occurred based on a plurality of pieces of image information captured at different times while work is being performed at the work site WS.

[0094] FIG. 7 is a diagram illustrating the situation around the work site WS at a first time (e.g., 10:00) during work according to this embodiment, and the situation around the work site WS at a second time (e.g., 10:03) during work.

[0095] 7 shows the surroundings of the work site WS from a bird's-eye view at a first time (for example, 10:00) during work. The arrangement of the sound collecting devices 611-617 and the imaging devices 711-718 shown in the situation 1701 is the same as in FIG. 5, so the explanation will be omitted.

[0096] A situation 1701 around the work site WS in Fig. 7 differs from a situation 1501 around the work site WS in Fig. 5 in that a shovel 100 is performing work at the work site WS. The shovel 100 is performing an operation such as excavation at the work site WS.

[0097] 7 shows the surroundings of the work site WS from a bird's-eye view at a second time (eg, 10:03) during work. The arrangement of the sound collecting devices 611-617 and the imaging devices 711-718 is the same as in the situation 1701.

[0098] The difference extraction unit 514 extracts difference image information indicating an area where a difference occurs between the image information captured at the first time 10:00 and the image information captured at the second time 10:03. In the example shown in Fig. 7, the difference extraction unit 514 extracts first difference image information indicating an area DR1 in which the tree TR1 is captured, second difference image information indicating an area DF2 in which the tree TR2 is captured, fifth difference image information indicating an area DF5 in which the tree TR3'' is captured, sixth difference image information indicating an area DF6 in which the spring water SP1 is captured, seventh difference image information indicating an area DF7 in which the shovel 100 is captured, and eighth difference image information indicating an area DF in which the excavated ground surface WS1 is captured.

[0099] The first difference image information and the second difference image information are the same as those in the above-described embodiment, and therefore a description thereof will be omitted.

[0100] The image determination unit 515 inputs the fifth difference image information to the trained model LM, and receives, from the trained model LM, the output result indicating that the type of situation is "tree leaning" and "abnormality present."

[0101] The image determination unit 515 inputs the sixth difference image information to the trained model LM, and receives the situation type as "spring water" and "abnormality present" as output results from the trained model LM.

[0102] The image judgment unit 515 inputs the seventh difference image information to the trained model LM, and receives from the trained model LM an output result indicating that the type of situation is "shovel operation" and "no abnormality."

[0103] The image judgment unit 515 inputs the eighth difference image information to the trained model LM, and receives from the trained model LM an output result indicating that the type of situation is "change in ground shape due to excavation" and that "no abnormality" is received.

[0104] In this way, the trained model LM according to this embodiment outputs a determination result that no abnormality has occurred when the difference between multiple pieces of image information is one or more of the movement of a work machine or a worker, and a change in the construction target due to work by the work machine or the worker at the work site. In this embodiment, by determining that changes due to work by a work machine or a worker are not abnormal, it is possible to suppress erroneous detection of an abnormality. Therefore, it is possible to suppress interruptions of work due to erroneous detection during work, and to improve work efficiency.

[0105] When there is an area where an abnormality occurs in the image information between the first time and the second time, the map generating unit 516 identifies the position information of the imaging device 714 that captured the area where the abnormality occurred (area DF5, area DF6) by referring to the position information storage unit 522. Then, the map generating unit 516 infers the area where the abnormality occurred (area DF5, area DF6) based on the position information of the imaging device 714, the coordinates of the area where the abnormality occurs (area DF5, area DF6) in the captured image information, and the imaging direction of the imaging device 714, and generates map information showing the area (area DF5, area DF6).

[0106] The output unit 517 outputs information indicating that "tree leaning" and "spring water" have occurred and map information indicating the areas where these abnormalities have occurred (areas DF5 and DF6) to one or more of the notification device 700 and the mobile communication terminal 800. Therefore, workers working at the work site WS or in the vicinity of the work site WS can recognize that "tree leaning" and "spring water" have occurred in the vicinity of the work site WS, and can take measures such as evacuation or work in response to the abnormalities.

[0107] In this embodiment, an example has been described in which the management device 500 judges whether an abnormality has occurred based on whether there is a difference between a plurality of pieces of image information captured at different times while work is being performed at the work site. By making this judgment, an abnormality that has occurred during work can be detected. Therefore, workers and managers can recognize abnormalities that have occurred at the work site WS and around the work site WS while working. Therefore, work can be started while taking into consideration actions to respond to abnormalities that have occurred during work (e.g., evacuation, repair work, etc.). Therefore, safety and work efficiency can be improved.

[0108] (Third embodiment) In the above-mentioned embodiment and modified examples, an example has been described in which an abnormality is detected based on audio signals acquired from the sound collection devices 611-617 and image information captured by the imaging devices 711-718. However, the above-mentioned embodiment and modified examples are not limited to detection of an abnormality using the sound collection devices 611-617 and the imaging devices 711-718. For example, since the shovel 100 is equipped with a sound collection device and an imaging device, the sound collection device and the imaging device mounted on the shovel 100 may be used. Therefore, in the third embodiment, an example in which a sound collection device and an imaging device are used will be described.

[0109] First, an overview of a work site abnormality monitoring system SYS1 according to the third embodiment will be described with reference to Fig. 8. Fig. 8 is a schematic diagram showing an example of a work site abnormality monitoring system SYS1 according to the third embodiment.

[0110] As shown in FIG. 8, a work site abnormality monitoring system SYS1 according to the third embodiment includes a management device 500, an alarm device 700, a mobile communication terminal 800, sound collection devices 611-612, image capture devices 711-712, and shovels 100A-100D.

[0111] Each of the shovels 100A to 100D is equipped with a sound collecting device A1, an imaging device S1, and a positioning device (not shown).

[0112] The shovels 100A to 100D are placed at positions where monitoring is desired after work is completed. For example, the shovels 100A and 100D are placed in the work site WS, and the shovels 100B and 100C are placed around the work site WS. Note that, in this embodiment, an example in which the shovels 100 to 100D are placed at the work site WS and around the work site WS will be described, but the shovels 100 to 100D may be placed at either the work site WS or the work site WS.

[0113] Then, the shovels 100A to 100D transmit to the management device 500 image information captured by the imaging device S1 and position information measured by the positioning device after the work end time.

[0114] Furthermore, the shovels 100A to 100D transmit, to the management device 500, image information captured by the imaging device S1 and position information measured by the positioning device before the work start time.

[0115] Thereby, the management device 500 can use the image information transmitted from the shovels 100A to 100D to detect an abnormality based on the difference between the image information captured at different times, similarly to the first embodiment.

[0116] Moreover, the excavators 100A to 100D transmit to the management device 500 the sound signal acquired by the sound collection device A1 and the position information measured by the positioning device from after the work end time until before the work start time.

[0117] This allows the management device 500 to use the audio signals transmitted from the shovels 100A to 100D to detect an abnormality based on the audio signals in the same manner as in the first embodiment.

[0118] In this embodiment, the shovels 100A to 100D are used for anomaly detection as an example of a work machine used for anomaly detection. In this embodiment, the work machine used for anomaly detection is not limited to the shovels 100A to 100D. Any work machine provided with an imaging device or a sound collection device may be used, for example, a crane. In this embodiment, an example in which anomalies are detected by combining the shovels 100A to 100D, the sound collection devices 611 to 612, and the imaging devices 711 to 712 has been described. However, this embodiment is not limited to this combination, and anomalies may be detected by using only the shovels 100A to 100D, for example.

[0119] In this embodiment, by disposing the excavators 100A to 100D at locations to be monitored, it is possible to detect abnormalities at the work site WS and in the vicinity of the work site WS, thereby reducing the workload involved in monitoring.

[0120] <effect> In the above-described embodiment and modified example, an abnormality at or near the work site is detected using one or more of an imaging device and a sound collecting device. Compared to visual inspection, this method can suppress human errors such as ignoring an abnormality or improper communication, thereby improving safety. In addition, it can detect an abnormality without using an expensive sensor, thereby reducing costs.

[0121] Although the embodiment of the work site monitoring system according to the present invention has been described above, the present invention is not limited to the above embodiment. Various changes, modifications, substitutions, additions, deletions, and combinations are possible within the scope of the claims. Naturally, these also fall within the technical scope of the present invention. [Explanation of symbols]

[0122] SYS, SYS1 Abnormality monitoring system 100 Shovel 700 Notification device 800 Mobile communication terminals 611~617 Sound collection device 711~718 Imaging device 500 Management device 501 Control device 502 Storage medium 511 Acquisition Department 512 Sound judgment section 513 Control command generation unit 514 Difference extraction part 515 Image Judgment Unit 516 Map Generation Unit

Claims

1. A control device configured to determine whether or not an abnormality that may affect the work site has occurred based on information from an imaging device or a sound collecting device installed at or near the work site, and to output information indicating that an abnormality has occurred if it is determined that an abnormality has occurred. Equipped with a work site monitoring system.

2. The imaging device is provided at a work site or in the vicinity of the work site, The control device is configured to acquire a plurality of pieces of image information captured at different times from the imaging device, determine whether or not an abnormality that may affect a work site has occurred based on a difference between the plurality of pieces of image information, and output the information when it is determined that an abnormality has occurred. The work site monitoring system of claim 1 .

3. The plurality of pieces of image information include image information captured near a work end time and image information captured near a work start time. The work site monitoring system according to claim 2 .

4. The plurality of pieces of image information include image information captured at different times while work is being performed at the work site. The work site monitoring system according to claim 2 .

5. The control device includes a trained model that causes a computer to output, when image information is input, a determination result of whether or not an abnormality has occurred, as well as a type of abnormality that may affect the work site; The control device is configured to output a determination result of whether or not an abnormality has occurred and a type of the abnormality by using the trained model to which image information indicating a difference between the plurality of image information has been input. The work site monitoring system according to claim 2 .

6. The trained model is configured to output a determination result that no abnormality has occurred when the difference in the plurality of image information is one or more of the movement of a work machine or a worker, and a change in a construction target due to the work of the work machine or the worker at the work site.

6. A work site monitoring system according to claim 5.

7. The trained model is configured to output a determination result that an abnormality has occurred when an intrusion of at least one of a person and an animal into the work site is detected based on a difference in the plurality of pieces of image information.

6. A work site monitoring system according to claim 5.

8. The sound collecting device is provided at a work site or in the vicinity of the work site, The control device is configured to determine that an abnormality that may affect a work site has occurred when the sound information collected by the sound collecting device satisfies a predetermined criterion, and to output the information. The work site monitoring system of claim 1 .

9. The control device is configured to determine a type of an abnormality that may affect the work site based on the characteristics of the sound information, and output the determined type of abnormality. The work site monitoring system of claim 8 .

10. The control device is configured not to determine that natural sounds that have no effect on the work site and sounds generated by a work machine or a worker at the work site are abnormal.

10. The work site monitoring system of claim 9.

11. When the control device determines that an abnormality that may affect the work site has occurred, the control device starts capturing images using the imaging device. The work site monitoring system of claim 8.

12. When the control device detects an abnormality in the work site, the control device outputs map information indicating the position where the abnormality occurred based on the position where the imaging device is installed. The work site monitoring system of claim 1 .

13. The imaging device or the sound collecting device is mounted on a work machine arranged at or near the work site. The work site monitoring system of claim 1 .

Images