Work management device and work management method

Abstract

To enable more accurate work management by accurately estimating work being performed using a trained model built through machine learning.SOLUTION: A work management device (3) is provided, comprising: a first output unit (312) configured to output a detection class estimated to be appearing in work at the time of acquisition of a captured image by inputting the captured image to an indicator estimation model (321), and a work estimation unit (314) configured to estimate work being performed at the time of acquisition of the captured image on the basis of an output result of the first output unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a work management device and the like. [Background technology]

[0002] The plumbing work management system of Patent Document 1 estimates the current work status of plumbing work from characteristic images detected from captured images that include the work area in the plumbing work, and determines the content of the notification to those involved in the plumbing work based on the estimated current work status. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-107341 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 does not specifically disclose a trained model constructed using machine learning. One aspect of the present invention aims to realize a task management device or the like that can accurately manage a task by accurately estimating a task being performed or a task to be performed using a trained model constructed by machine learning. [Means for solving the problem]

[0005] A work management device according to one aspect of the present invention includes an image acquisition unit that acquires a photographed image including a work target area in plumbing work, a first output unit that outputs at least one indicator that is estimated to have appeared in work at a certain point in time by inputting the photographed image into an indicator estimation model constructed using machine learning to estimate at least one indicator that is estimated to have appeared in work at the time the photographed image is acquired, and a first output unit that outputs at least one indicator that is estimated to have appeared in work at a certain point in time by inputting the photographed image into an indicator estimation model constructed using machine learning to estimate at least one indicator that is estimated to have appeared in work at a certain point in time among a plurality of indicators that indicate each of a plurality of things involved in work in the plumbing work, or a combination of these. The work management device identifies a work at a certain time within the specified work time that is the time when the photographed image is acquired based on data prepared in advance that associates each work that may be performed in the plumbing work with a combination of the indicators and combinations of indicators that appear at each of the times when the photographed image is acquired within a specified work time output over time by the first output unit, thereby determining a relationship between at least one work performed in the specified work time and a time period during which the work was performed. and a second output unit that outputs work schedule data indicating the relationship between at least one work piece estimated to be performed in the next specified work time and the time period in which the work piece will be performed by inputting the performance data into a work estimation model constructed using machine learning to estimate the work to be performed in the next specified work time from the performance of the work in the specified work time, wherein the indicator estimation model is constructed by performing machine learning using as training data item images indicating items used in the plumbing work and the indicators, and the work estimation model is constructed by performing machine learning using as training data past work schedule data indicating the relationship between at least one work piece indicated as being scheduled to be performed in a construction plan for past plumbing work and the time period in which the work piece will be performed, past performance data indicating the relationship between at least one work piece performed in the past plumbing work and the time period in which the work piece was performed, and factor data that influenced the work in the past plumbing work.

[0006] A work management method according to one aspect of the present invention includes an image acquisition step of acquiring a photographed image including a work target area in plumbing work; a first output step of inputting the photographed image into an indicator estimation model constructed to estimate at least one indicator appearing in work at a certain point in time out of a plurality of indicators indicating each of a plurality of things involved in the work in the plumbing work, or a combination thereof, using machine learning, thereby outputting at least one indicator estimated to have appeared in the work at the time the photographed image was acquired; and a second output step of outputting at least one indicator estimated to have appeared in the work at the time the photographed image was acquired based on data prepared in advance that associates each work that may be performed in the plumbing work with a combination of the indicators and the combinations of indicators that appear at each of the times the photographed image was acquired within a predetermined work time output over time in the first output step, thereby identifying at least one work performed in the predetermined work time and a time period during which the work was performed. and a second output step of outputting work schedule data indicating the relationship between at least one piece of work estimated to be performed in the next specified work time and the time period in which the work will be performed by inputting the performance data into a work estimation model constructed using machine learning to estimate the work to be performed in the next specified work time from the performance of the work in the specified work time. The indicator estimation model is constructed by performing machine learning using as training data item images indicating items used in the plumbing work and the indicators, and the work estimation model is constructed by performing machine learning using as training data past work schedule data indicating the relationship between at least one piece of work indicated as being scheduled to be performed in a construction plan for past plumbing work and the time period in which the work will be performed, past performance data indicating the relationship between at least one piece of work performed in the past plumbing work and the time period in which the work will be performed, and factor data that influenced the work in the past plumbing work. Effect of the Invention

[0007] According to one aspect of the present invention, a trained model constructed by machine learning is used to accurately estimate work that is being performed or is scheduled to be performed, thereby enabling the work to be managed with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of a schematic configuration of a waterworks construction management system. [Diagram 2] 11 is a diagram showing an example of an output result of a first output unit. FIG. [Diagram 3] 11A and 11B are diagrams illustrating an example of an operation for which an operation state determination unit determines an operation state. [Figure 4] 13 is a diagram showing an example of a display of a result of determination of a work state by a work state determination unit. FIG. [Diagram 5] FIG. 13 is a diagram showing an example of display of performance data created by a data creating unit. [Figure 6] 13 is a diagram showing a display example of work schedule data output by a second output unit. FIG. [Figure 7] 11 is a diagram showing an example of a relationship between a determination result by an operation state determination unit and a notification content by a display control unit. FIG. [Figure 8] 4 is a flowchart illustrating an example of a process in the work management device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [Configuration of the waterworks construction management system] 1 is a block diagram showing an example of a schematic configuration of a plumbing work management system 1 according to an embodiment of the present invention. The plumbing work management system 1 manages various works for plumbing work such as installing pipes (e.g., progress of the plumbing work). The plumbing work management system 1 includes a camera 2, a work management device 3, and a display device 4.

[0010] The camera 2 photographs various tasks carried out at one or more plumbing work sites. The camera 2 may be capable of photographing the work area where the various tasks are performed. Examples of the camera 2 include a camera attached to the helmet or work clothes (e.g., the arm part of the work clothes) of a worker (a person involved in the plumbing work) to photograph the work at hand, or a camera mounted on various work tools (e.g., a backhoe). Examples of the camera 2 include a camera installed near the work area. The camera 2 includes a photographing unit 21 and a communication unit 22.

[0011] The photographing unit 21 photographs at least the work target area in the plumbing work. The photographing unit 21 may photograph still images or may photograph moving images. Photographing may be performed multiple times within the same plumbing work. The work target area (and therefore the camera 2) may be fixed or may be movable with each photographing session as a unit. In other words, the photographing unit 21 creates a photographed image including the work target area each time photographing is performed.

[0012] The communication unit 22 is communicatively connected to the work management device 3, and transfers data of the images captured by the imaging unit 21 to the work management device 3. The communication between the communication unit 22 and the work management device 3 is wireless, but may be wired.

[0013] The work management device 3 manages various works being carried out as plumbing works or various works to be carried out. When managing works at multiple plumbing works sites, the work management device 3 manages the works for each site. Details of the work management device 3 will be described later.

[0014] The display device 4 displays various information related to the plumbing work to the manager who manages the plumbing work (a person involved in the plumbing work). The display device 4 may display, for example, the work status determination result output from the work management device 3, performance data showing the performance of various works in a specified work time, or work schedule data showing the schedule of various works to be performed in the next specified work time.

[0015] The display device 4 is a monitor or a personal computer on which a manager manages various tasks. However, the display device 4 may also be an electronic device (e.g., a tablet or a smartphone) carried by a worker. In this case, the display device 4 can display various information related to the plumbing work to the worker. The plumbing work management system 1 may also include an audio output device (e.g., a speaker) that outputs the various information by voice.

[0016] In this embodiment, the above-mentioned predetermined work time indicates the total work time (e.g., 8:00 to 16:30) specified for a certain day, and the next predetermined work time indicates the total work time (e.g., 8:00 to 16:30) specified for the next day of a certain day. However, the predetermined work time may indicate, for example, the work time from the start of work to the time when the photographed image is acquired, in which case the next predetermined work time indicates the work time after the time when the photographed image is acquired.

[0017] [Configuration of the work management device] The work management device 3 includes a control unit 31 and a memory unit 32. The control unit 31 comprehensively controls each unit of the work management device 3. The memory unit 32 stores various programs and various data used by the control unit 31. The memory unit 32 stores, for example, an index estimation model 321 and an work estimation model 322 described below. The memory unit 32 may be realized by a storage device different from the work management device 3 and communicably connected to the work management device 3.

[0018] The control unit 31 includes an image acquisition unit 311 , a first output unit 312 , a position identification unit 313 , an activity estimation unit 314 , an activity state determination unit 315 , a data creation unit 316 , a second output unit 317 , and a display control unit 318 .

[0019] The image acquisition unit 311 acquires a captured image transferred from the camera 2, the captured image including a work area for plumbing work.

[0020] The first output unit 312 inputs the captured image acquired by the image acquisition unit 311 to the index estimation model 321 stored in the storage unit 32. The index estimation model 321 is a trained model constructed using machine learning to estimate at least one index appearing in work at a certain point in time among multiple indexes for evaluating the work status in plumbing work. In this embodiment, the above index is referred to as a detection class that needs to be detected from the captured image to evaluate the work status.

[0021] Examples of detection classes include single objects such as various pipes (pipes) used in plumbing work, workers, ordinary people other than workers, various tools used in plumbing work, and ditches in which pipes are placed. Detection classes may be combinations of multiple such single objects, and for example, a combination of a ditch and a worker in the ditch may be set as one detection class.

[0022] Examples of various pipes include straight pipes and irregular pipes. Examples of various work implements (e.g., work machines (work vehicles) and tools) include asphalt cutters, backhoes, dump trucks, rammers, cones (or poles), sheet piles, ratchets, sleepers, and ladders. In addition, the work implements set as the detection class may be a part of the work implements, and for example, a bucket and a sling belt that are part of a backhoe may be set as the detection class.

[0023] As the detection class, an object that allows the work state to be determined may be set. For example, as the detection class, an object that characterizes proper work (standard work) and an object that characterizes dangerous work may be set. An example of an object that characterizes standard work is the working posture of a worker performing standard work. An example of an object that characterizes dangerous work is a groove where no safety cone is installed.

[0024] The index estimation model 321 is a trained model constructed to extract and output the above-mentioned detection classes contained in a captured image as a result of inputting the captured image. The index estimation model 321 may be, for example, a neural network (e.g., a Convolutional Neural Network (CNN)) including at least an input layer, an intermediate layer, and an output layer.

[0025] In the index estimation model 321, the input layer is a layer to which a captured image is input. The intermediate layer is a layer in which parameters are learned based on an area image including a work area, an item image showing an item used in plumbing work, and a detection class. The output layer is a layer that outputs at least one detection class.

[0026] For example, data in which a detection class is associated as correct answer data with each of a plurality of area images and each of a plurality of article images is prepared as training data. The index estimation model 321 is constructed by sequentially inputting training data and learning parameters (weights and biases) so as to minimize a loss function. The index estimation model 321 is constructed, for example, by a model generation device (not shown), but is not limited to this and may be constructed, for example, by the control unit 31. In this case, the control unit 31 also has a function as a learning device that constructs the index estimation model 321 by machine learning using the training data.

[0027] In addition, the other trained models described later may also be constructed by sequentially inputting training data and learning parameters so as to minimize the loss function, and may be constructed by a model generation device or by the control unit 31.

[0028] The area image may be a photographed image taken by the camera 2, a difference image, or a composite image. The difference image is, for example, an image obtained by extracting the difference between two photographed images taken over time. The composite image is, for example, an image in which various background images (virtual background images) prepared in advance are embedded in the background area of ​​the photographed work target area. The item image may be an image of the above-mentioned various pipes and various work tools. As the item image, a plurality of images of one pipe and one work tool with their sizes changed may be prepared. In this way, by preparing processed images other than the photographed images, it becomes possible to prepare various training data from a small number of photographed images. In addition, the area image and the item image may include an image suitable for estimating (inferring) the detection class.

[0029] The index estimation model 321 constructed in this way causes the computer to function such that, when a photographed image is input to the input layer, at least one detection class estimated to have appeared in the work at the time of acquiring the photographed image is output from the output layer through calculations by the intermediate layer. That is, the first output unit 312 outputs at least one detection class estimated to have appeared in the work at the time of acquiring the photographed image by inputting the photographed image to the index estimation model 321. Note that the time of acquiring the photographed image is the time of acquiring the photographed image by the image acquiring unit 311, but may be read as the time of acquiring the image of the work target area by the camera 2.

[0030] FIG. 2 is a diagram showing an example of the output result of the first output unit 312. Reference numeral 201 in FIG. 2 indicates the output result from the start of work to time T1 on a certain day, and reference numeral 202 indicates the output result from the start of work to the end of work on a certain day. In reference numerals 201 and 202, the vertical axis indicates the number of each detection class, and the horizontal axis indicates the work time. For example, a number is assigned to each detection class described above, such as "0" for the worker, "1" for the straight pipe, "2" for the irregular pipe, "3" for the backhoe, and "4" for the sling belt. In the example of FIG. 2, "0" to "7" are shown as the numbers of the detection classes, but the detection classes are numbered as many as the number of detection classes set in advance.

[0031] For example, consider a case where a worker, an irregular pipe, a backhoe, and a sling belt are included in a photographed image acquired at time T1. In this case, the first output unit 312 outputs a combination of "0" and "2" to "4" corresponding to these detection classes at time T1, as shown by reference numeral 201 in FIG. 2. Since the first output unit 312 inputs the photographed images to the index estimation model 321 over time from the start of work, as shown by reference numeral 201, at time T1, it can output a combination of detection classes at each time point from the start of work (8:00) to time T1. As a result, as shown by reference numeral 202, at the end of work for one day, the first output unit 312 can output a combination of detection classes at each time point from the start of work to the end of work for one day.

[0032] As a result of the above calculation, the index estimation model 321 outputs a numerical value of each detection class (e.g., a numerical value greater than or equal to 0 and less than or equal to 1). The larger the numerical value of a detection class, the higher the probability that an object represented by that detection class is included in the captured image. Therefore, the index estimation model 321 estimates a detection class that indicates a numerical value equal to or greater than a predetermined threshold as the detection class included in the captured image. The index estimation model 321 may output, as an output result, the numerical value, for example, together with the detection class, as the accuracy (reliability) of the estimation of the detection class.

[0033] The position identification unit 313 identifies, in the captured image, the position of the detection class output by the first output unit 312. The position identification unit 313 can identify the position of the detection class in the captured image by inputting the captured image to a position estimation model constructed to estimate the position of the detection class in the captured image using training data in which article images are associated with detection classes, for example. The position estimation model may be stored in the storage unit 32, and may be, for example, a trained model realized by R-CNN (Regions with CNN).

[0034] The operation estimation unit 314 estimates the operation being performed at the time of acquiring the captured image based on the output result of the first output unit 312. Specifically, the operation estimation unit 314 estimates the operation being performed at the time of acquiring the most recently acquired captured image by the image acquisition unit 311 by analyzing the combination of detection classes that appear at each of the times of acquiring the captured image output over time by the first output unit 312. That is, the operation estimation unit 314 estimates the operation currently being performed from the combination of detection classes estimated based on the most recently acquired captured image by the image acquisition unit 311 and the combination of detection classes estimated based on the captured image at least one point in time immediately prior thereto (the past captured image).

[0035] Each task that may be performed in plumbing work is associated with a combination of detection classes and stored in the memory unit 32. The task estimation unit 314 identifies the task corresponding to the combination based on the output result (combination of detection classes) of the first output unit 312 at each time point as the task being performed at the time point when the photographed image was acquired.

[0036] For example, in the work process of hanging and installing a straight pipe, there is a task of removing a foreign object from a receiving groove. For example, a combination of a worker, a straight pipe, a receiving groove, and an object (foreign object) present in the receiving groove is associated with this task as a detection class. The task estimation unit 314 determines whether the above task is being performed by determining the presence or absence of a worker, a straight pipe, a receiving groove, and an object located in the receiving groove at each successive point in time in the captured image based on the output result of the first output unit 312.

[0037] The activity estimation unit 314 may estimate an activity using an performed activity estimation model constructed to estimate an activity being performed at the time of capturing a captured image, using training data in which each activity is associated with a combination of a detection class. In this case, the activity estimation unit 314 can estimate the activity by inputting the output result of the first output unit 312 to the performed activity estimation model. The performed activity estimation model may be stored in the storage unit 32, and may be, for example, a trained model realized by CNN.

[0038] Furthermore, when the position specifying unit 313 specifies the position of the detection class in the photographed image, the work estimation unit 314 may estimate the work being performed in different areas in the photographed image based on the position of the detection class. For example, the work estimation unit 314 may determine whether the photographed image includes a scene of multiple plumbing works being performed based on the type of the estimated detection class and the position of the specified detection class. When the work estimation unit 314 determines that plumbing works are being performed in different areas in the photographed image, it estimates the work being performed in each area.

[0039] The work state determination unit 315 determines whether the work state of the work estimated by the work estimation unit 314 is appropriate by analyzing the combination of detection classes that appear at each time point of acquisition of the captured images output over time by the first output unit 312. That is, the work state determination unit 315 determines whether the work state of the work estimated to be currently being performed is appropriate from the combination of detection classes estimated based on the captured images most recently acquired by the image acquisition unit 311 and the combination of detection classes estimated based on the captured images at least one time point immediately before that. This determination result indicates the performance of each work.

[0040] Specifically, the work status determination unit 315 determines whether the work status is appropriate by determining whether the work procedures are appropriate, whether safety is guaranteed, and whether there is a delay in the work.

[0041] Whether or not a work procedure is appropriate may be determined based on, for example, the combination of detection classes at each successive point in time in the captured image and the position or change in position over time of a certain detection class in the captured image at each point in time.

[0042] Fig. 3 is a diagram showing an example of the work for which the work state determination unit 315 determines the work state. Fig. 3 shows an example of the work performed in the work process from pipe hanging and installation to joint connection check, divided into the case of a straight pipe and the case of a special-shaped pipe.

[0043] As shown in FIG. 3, when a foreign object is being removed from the receiving groove, the work state determination unit 315 determines whether the work procedure is appropriate by, for example, determining the presence or absence of an object in the receiving groove at each successive time point in the captured image (positional change of the object). The work state determination unit 315 may also perform similar processing when a foreign object adhering to a region within about 30 cm from the end face of the outer surface of the insertion port and to the inner surface of the receiving port is being removed. That is, the work state determination unit 315 may determine whether the work procedure is appropriate by, for example, determining the presence or absence of an object in these positions at each successive time point (positional change of the object). In other words, the work state determination unit 315 may determine that the area of ​​the object occupying the receiving groove or the like is decreasing in the captured image with each passing time, and when it is determined that the area is equal to or smaller than a predetermined area, the work state determination unit 315 may determine that the foreign object has been removed. In this case, the work state determination unit 315 may determine that the work procedure for removing the foreign object from the receiving groove or the like is appropriate.

[0044] Furthermore, the work state determination unit 315 can determine whether the work procedure is appropriate based on the captured image in this way, and can also determine whether the work quality is ensured. For example, if the captured image shows that there is almost no object in the receiving groove, the work state determination unit 315 can determine that the work quality of the work of removing foreign objects from the receiving groove and the like is good.

[0045] Similarly, for the other tasks shown in FIG. 3, the task status determination unit 315 can determine the suitability of the task procedure and task quality by analyzing the combination of detection classes and the positions of the detection classes in the captured image.

[0046] In addition, whether or not safety is guaranteed may be determined based on, for example, the combination of detection classes at each successive point in time in the captured image and the position or change in position over time of a certain detection class in the captured image at each point in time.

[0047] Here, examples of accidents that should be prevented during each work task include the following: (Accident 1) An ordinary person falling into a ditch. (Disaster 2) A work vehicle (e.g., a dump truck or a backhoe) comes into contact with a worker or a member of the public. (Workers and members of the public are collectively referred to as "persons.") (Accident 3) While a pipe was being lifted, it fell and came into contact with a person. (Disaster 4) The contents of a bucket fall and come into contact with a person. (Disaster 5) Ditch collapse. (Disaster 6) A pipe placed at the scene came into contact with a person. (Disaster 7) Tools or other objects fell into a ditch and came into contact with a person. (Accident 8) A worker falls from a ladder.

[0048] The work state determination unit 315 may determine whether or not the above disasters (Disaster 1) to (Disaster 8) are likely to occur based on the combination of detection classes and the position or time-dependent change in position of a certain detection class, as described above. The following shows a processing example when it is determined that the above disasters (Disaster 1) to (Disaster 8) are likely to occur.

[0049] When a ditch, a safety cone, and a civilian are extracted as the output result (detection class) of the indicator estimation model 321, the work state determination unit 315 may determine whether or not the above-mentioned disaster (Disaster 1) has occurred. For example, consider a case where the work state determination unit 315 detects a civilian for a predetermined time within a closed area formed by a curve connecting the safety cones (or a curve surrounding the periphery of the ditch) in the captured image. In this case, the work state determination unit 315 determines that the above-mentioned disaster (Disaster 1) may occur and that safety is not guaranteed (i.e., a dangerous state). The work state determination unit 315 may detect the above-mentioned curve and a civilian from the captured image after performing a bird's-eye view transformation on the captured image.

[0050] When a worker or a helmet is extracted as a detection class instead of a general person, the occurrence of the above-mentioned disaster (disaster 1) may be determined. In this case, when a helmet is detected for a predetermined time in the closed space, the work state determination unit 315 may presume that a worker is present in the closed space and may determine that a dangerous state exists. When a groove is extracted as a detection class but a safety cone is not extracted, the work state determination unit 315 may also determine that a dangerous state exists in which the above-mentioned disaster (disaster 1) may occur.

[0051] When a work vehicle and a person are extracted as detection classes, the work state determination unit 315 may determine whether or not the above-mentioned disaster (disaster 2) has occurred. For example, when the distance between the work vehicle or a part thereof (e.g., bucket) and a person in the captured image is equal to or less than a predetermined value, the work state determination unit 315 may determine that the above-mentioned disaster (disaster 2) is in a dangerous state where the above-mentioned disaster may occur. The captured image may be acquired from a camera 2 installed on the work vehicle, or a camera 2 installed at a location where waterworks work is being carried out. In addition, the work state determination unit 315 may detect the work vehicle, the person, and the above-mentioned distance from the captured image after performing a bird's-eye view transformation on the captured image. In addition, the work state determination unit 315 may detect the direction of the person's face in the captured image, and when the work vehicle is not positioned in that direction, determine that the above-mentioned disaster (disaster 2) is in a dangerous state where the above-mentioned disaster may occur.

[0052] When the pipe and the sling belt are extracted as the detection class, the work state determination unit 315 may determine whether or not the above-mentioned disaster (disaster 3) has occurred. For example, the work state determination unit 315 may detect the inclination of the pipe with respect to the ground in the captured image, and when the inclination is a predetermined angle (e.g., 30°) or more, determine that the above-mentioned disaster (disaster 3) is in a dangerous state where the above-mentioned disaster (disaster 3) may occur. In addition, the work state determination unit 315 may calculate the hoisting angle between the two sling belts in the captured image, and when the hoisting angle is a predetermined angle (e.g., 60°) or more, determine that the above-mentioned disaster (disaster 3) is in a dangerous state where the above-mentioned disaster (disaster 3) may occur. When the hoisting angle is a predetermined angle or more, there is a risk that the pipe, which is the suspended load, will slip sideways. In addition, the work state determination unit 315 may determine that the above-mentioned disaster (disaster 3) is in a dangerous state where the above-mentioned disaster (disaster 3) may occur even when only one sling belt is detected in the captured image.

[0053] When a bucket and a person are extracted as the detection class, the work state determination unit 315 may determine whether or not the above-mentioned disaster (Disaster 4) has occurred. For example, when a person is located under the bucket in the captured image, or when a person is located within a predetermined range around the bucket, the work state determination unit 315 may determine that the above-mentioned disaster (Disaster 4) is likely to occur, which is a dangerous state. The work state determination unit 315 may also detect the relative positions of the bucket and the person from the captured image after performing a bird's-eye view transformation on the captured image. The work state determination unit 315 may also determine that the above-mentioned disaster (Disaster 4) is likely to occur, when the inclination of the opening surface of the bucket with respect to the ground in the captured image is equal to or greater than a predetermined angle.

[0054] When a ditch and an arrow pile are extracted as the detection class, the work state determination unit 315 may determine whether or not the above-mentioned disaster (Disaster 5) has occurred. For example, when the shape of the ditch is a predetermined shape (a shape that may collapse) in the captured image, the work state determination unit 315 may determine that the above-mentioned disaster (Disaster 5) is in a dangerous state where the above-mentioned disaster (Disaster 5) may occur. Furthermore, when the angle of the arrow pile driven into the ditch with respect to the ground in the captured image is equal to or less than a predetermined angle (when the degree of tilting of the arrow pile is large), the work state determination unit 315 may determine that the above-mentioned disaster (Disaster 5) is in a dangerous state where the above-mentioned disaster (Disaster 5) may occur. Furthermore, when a rod for preventing the arrow pile from tilting cannot be detected in the captured image, the work state determination unit 315 may determine that the above-mentioned disaster (Disaster 5) is in a dangerous state where the above-mentioned disaster (Disaster 5) may occur. Furthermore, when a ditch is extracted as the detection class but an arrow pile is not extracted, the work state determination unit 315 may also determine that the above-mentioned disaster (Disaster 5) is in a dangerous state where the above-mentioned disaster (Disaster 5) may occur.

[0055] When a pipe, a sleeper, and a ratchet are extracted as detection classes, the work state determination unit 315 may determine whether or not a disaster (disaster 6) described above has occurred. The sleeper and the ratchet are intended to prevent the pipe from rolling. For example, when the position of a pipe does not change for a predetermined time in the captured image, the work state determination unit 315 may determine that the pipe is waiting to be joined. When the work state determination unit 315 determines that the positional relationship between the pipe waiting to be joined, the sleeper, and the ratchet is not a predetermined positional relationship in the captured image, the work state determination unit 315 may determine that a dangerous state exists in which a disaster (disaster 6) described above may occur.

[0056] Also, consider a case where a pipe is extracted as a detection class, and a pipe is located outside the ditch (on the ground) for a predetermined time in the captured image, but a sleeper or a ratchet is not extracted as a detection class. In this case, the work state determination unit 315 cannot detect a sleeper or a ratchet required for the pipe placed on the ground in the captured image, and may determine that the above-mentioned (disaster 6) is in a dangerous state where the disaster may occur.

[0057] When the outer periphery (line) of the ditch and a tool are extracted as the detection class, the work state determination unit 315 may determine whether or not a disaster (Disaster 7) has occurred. For example, when a tool is located on the outer periphery of the ditch in the captured image, the work state determination unit 315 may determine that a dangerous state exists in which a disaster (Disaster 7) may occur. In addition, the work state determination unit 315 may detect the outer periphery of the ditch and a tool from the captured image after performing bird's-eye view transformation on the captured image.

[0058] When a ladder, a ditch side wall, and a worker are extracted as detection classes, the work state determination unit 315 may determine whether or not the above-mentioned disaster (disaster 8) has occurred. For example, the work state determination unit 315 may calculate the angle between the ladder and the ditch side wall on which the ladder is installed in the captured image, and determine that a dangerous state in which the above-mentioned disaster (disaster 8) may occur exists when the angle is equal to or smaller than a predetermined angle. Furthermore, the work state determination unit 315 may determine that a dangerous state in which the above-mentioned disaster (disaster 8) may occur exists when a worker is performing work other than ascending and descending the ladder in the captured image.

[0059] When detecting an object from a captured image, the working state determination unit 315 may create three-dimensional point cloud data of the captured image and detect the object from the three-dimensional point cloud data, or may detect the object by edge detection processing. Also, the working state determination unit 315 may measure the actual distance between objects using a distance measurement sensor (e.g., a TOF (Time of Flight) sensor).

[0060] The work state determination unit 315 may determine the suitability of a work procedure using a procedure estimation model constructed to estimate the suitability of the work procedure using training data in which a combination of detection classes and a position or a position change over time of the detection classes are associated with the suitability of the work procedure. The work state determination unit 315 may determine the safety using a safety estimation model constructed to estimate the safety using training data in which a combination of detection classes and a position or a position change over time of the detection classes are associated with the safety of the work. The work state determination unit 315 may determine the suitability of the work procedure using a quality estimation model constructed to estimate the suitability of the work quality using training data in which a combination of detection classes and a position or a position change over time of the detection classes are associated with the suitability of the work quality. The procedure estimation model, the safety estimation model, and the quality estimation model may be stored in the storage unit 32, and may be, for example, a learned model realized by CNN.

[0061] Furthermore, whether or not a delay has occurred in an operation may be determined by comparing the operation time of the operation estimated by the operation estimation unit 314 with operation process data indicating the relationship between the operation to be performed on that day and the time period during which the operation is performed. The operation status determination unit 315 may determine that a delay has occurred in an operation when it determines that a predetermined time or more has elapsed since the operation time scheduled for the operation. The predetermined time may be set for each operation based on the performance of the operation, etc. Furthermore, the predetermined time may be set in stages, so that the degree of delay may be determined in stages. The operation process data may be stored in the memory unit 32.

[0062] The display control unit 318 controls the display device 4 to display various information on the display device 4. The display control unit 318 may, for example, display the above-mentioned work status determination result in real time at the time the determination result is obtained. By displaying the work status determination result, the display control unit 318 can function as a warning unit that warns the manager or worker when the work status determination unit 315 determines that the work status is not good.

[0063] Fig. 4 is a diagram showing an example of the display of the determination result of the work state by the work state determination unit 315. Reference numeral 401 in Fig. 4 is a diagram showing an example of the display of whether the work procedure is appropriate, whether safety is guaranteed, and whether there is a delay in the work. In the example of reference numeral 401, when the work state is appropriate, it is marked as "normal," and when it is not, it is marked as "abnormal." Reference numeral 402 in Fig. 4 is a diagram showing an example of the display of the result of determining for each site whether there is a delay in the work.

[0064] The example of reference numeral 401 indicates that the work procedure currently being performed in a work process of joining pipes at a certain site is appropriate and no delays have occurred in the work, but safety is not guaranteed. In other words, this example warns that safety is not guaranteed. The example of reference numeral 402 indicates whether or not there is a delay in the work, step by step, for each site. The example of reference numeral 402 indicates that there is no delay in the work at sites A and C, there is a slight delay at site B, and there is a clear delay at site D. In other words, this example warns that there is a delay in the work at least at site D.

[0065] Furthermore, when the working state determination unit 315 determines that the safety is not ensured in order to prevent the above-mentioned (Disaster 1) to (Disaster 8), the display control unit 318 may cause the display device 4 to display an image for calling the attention of the manager or worker. For example, the display control unit 318 may perform the following attention. Note that this attention may be realized by outputting a voice or a warning sound from a sound output device. In the case of (Disaster 1): Warning that safety cones have not been installed. (Disaster 2) In the case of disaster 2: Alert that there are work vehicles nearby. In the case of (Disaster 3): Warning that the pipe is tilted significantly. Or, warning that there is only one sling belt. In the case of (Disaster 4): Alerts that a bucket is nearby, or that the bucket is tilting too far. In the case of (Disaster 5): Warning that the sheet piles have not been driven into the ditch (urging people to drive the sheet piles into the ditch). Or, warning that the sheet piles driven into the ditch are tilted (not driven almost vertically). In the case of (Disaster 6): Warning that sleepers and ratchet stops have not been installed. In the case of (disaster 7): Warning that tools may fall into the ditch. In the case of (Accident 8): Warning that the ladder is unstable. Warning that the worker is performing work other than ascending and descending the ladder.

[0066] The work state determination unit 315 does not have to determine all of whether the work procedure is appropriate, whether the work quality is guaranteed, whether the safety is guaranteed, and whether a delay occurs in the work. The work state determination unit 315 may determine at least one of these work states. The display control unit 318 may display the determination result of the work state determined by the work state determination unit 315.

[0067] The data creation unit 316 analyzes the combinations of detection classes that appear at the respective acquisition times of captured images within all work hours of a certain day, which are output over time by the first output unit 312. The data creation unit 316 creates performance data indicating the relationship between at least one task performed within all work hours of a certain day and the time period in which the task was performed, by analyzing the combinations of detection classes. This performance data indicates the work performance for the day (the work performance during a specified work time), and is a combination of the performance of each task performed on that day.

[0068] In this embodiment, the activity estimation unit 314 analyzes the combination of detection classes to estimate the activities at the time when the photographed images were acquired. Therefore, the data creation unit 316 can create performance data using the estimation results of one day's activities estimated by the activity estimation unit 314.

[0069] The data creation unit 316 may also create procedure suitability data and risk occurrence data by analyzing the combination of the above-mentioned detection classes. Procedure suitability data is data indicating the suitability of the work procedures for each work during all work hours in one day. Risk occurrence data is data indicating the occurrence of a dangerous work during all work hours in one day.

[0070] In this embodiment, the work state determination unit 315 analyzes the combination of detection classes to determine whether the work state of each work at the time when the captured image was acquired is appropriate. Therefore, the data creation unit 316 can create procedure appropriateness data using the determination result of the work state determination unit 315 as to the appropriateness of the work procedure for each work of one day. In addition, the data creation unit 316 can create risk occurrence presence / absence data using the determination result of the work state determination unit 315 as to the safety of each work of one day.

[0071] Furthermore, the data creation unit 316 may analyze the combination of detection classes to create performance data up to the time of capturing the captured image, data on procedure suitability, and data on the presence or absence of a risk. In this case, the data creation unit 316 can create these data in real time.

[0072] In this embodiment, the data creation unit 316 may use the work estimation result by the work estimation unit 314 to create performance data indicating the relationship between the work estimated to have been performed up to the time of acquisition of the photographed image and the time period in which the work was performed. Furthermore, the data creation unit 316 may use the judgment result of the suitability of the work state for each work by the work state judgment unit 315 to create procedure suitability data indicating the suitability of the work procedure for each work up to the time of acquisition of the photographed image. Furthermore, the data creation unit 316 may use, for example, the judgment result of the suitability of the work state to create danger occurrence data indicating suitability indicating the presence or absence of dangerous work up to the time of acquisition of the photographed image.

[0073] As described above, the work status determination unit 315 can also determine whether or not the work quality is ensured based on the captured images. Therefore, the data creation unit 316 may create quality data indicating whether the work quality of each work for one day or up to the time when the captured images were obtained by analyzing the combination of detection classes, similar to the procedure suitability data and risk occurrence data.

[0074] The display control unit 318 may, for example, cause the display device 4 to display the performance data created by the data creation unit 316 in real time. Fig. 5 is a diagram showing an example of display of performance data. In Fig. 5, the performance time of a work process including at least one work (e.g., asphalt cutting) is shown as the performance time of the work. The same applies to Fig. 6.

[0075] In the example of Figure 5, work process data indicating the relationship between the work to be performed on that day and the time period during which the work will be performed ("Work Schedule" in Figure 5) is shown as an implementation schedule, along with performance data ("Work Results" in Figure 5) created by the data creation unit 316 up to the time the captured image was acquired (time T1).

[0076] Furthermore, the display control unit 318 may cause the display device 4 to display information based on the procedure suitability data and the risk occurrence data in real time, as shown in Fig. 5. In the example of Fig. 5, it is shown that a dangerous task occurred in time period Ta based on the risk occurrence data, and that a task with an incorrect procedure occurred in time period Tb based on the procedure suitability data. Furthermore, a comparison with the work process data shows that the task was delayed in time periods Tc and Td. Note that, when a task with poor quality is identified in the quality pass / fail data, the display control unit 318 may show that the task occurred in the performance data of Fig. 5.

[0077] In this way, the display control unit 318 displays the performance data and the like on the display device 4, so that the manager can grasp the progress and working status of the work in real time. Furthermore, if the working status is determined to be unsafe, the manager can also give instructions or guidance to the worker while visually checking the captured images acquired by the image acquisition unit 311. Furthermore, if it is determined that the safety is not guaranteed, the manager can also visually check the captured images showing dangerous work.

[0078] When a day's worth of work is completed, the display control unit 318 can display the performance data for that day on the display device 4 as a performance process chart as shown in Fig. 5. The display control unit 318 can display delayed work and its time period by displaying the performance data for that day together with the work process data. The display control unit 318 can also display, together with the performance data for that day, time periods when dangerous work occurred, time periods when work was performed with incorrect procedures, and time periods when work of poor quality occurred.

[0079] Furthermore, the display control unit 318 may display procedure suitability data, risk occurrence data, and quality pass / fail data separately from the performance data. In this case, the display control unit 318 may display procedure suitability data in a table format such as a work procedure chart, risk occurrence data in a dangerous work chart, and quality pass / fail data in a quality control chart. In this case, the display control unit 318 can notify the manager or worker of work that has been performed with incorrect procedures, work that has involved dangerous work, and work that is of poor quality.

[0080] The data creating unit 316 does not have to create all of the procedure suitability data, risk occurrence data, and quality pass / fail data. The data creating unit 316 may create at least one of the procedure suitability data, risk occurrence data, and quality pass / fail data. The display control unit 318 only needs to display the data created by the data creating unit 316.

[0081] The second output unit 317 inputs the one-day performance data created by the data creation unit 316 to the task estimation model 322 stored in the storage unit 32. The task estimation model 322 is a trained model constructed using machine learning to estimate tasks to be performed during all task hours of the next day based on the performance of tasks during all task hours of a certain day.

[0082] The task estimation model 322 is a trained model constructed so as to output task schedule data indicating the relationship between at least one task estimated to be performed during all task hours of the next day and the time period during which the task is performed, as a result of inputting one day's worth of performance data. The task estimation model 322 may be, for example, a neural network (e.g., CNN) including at least an input layer, an intermediate layer, and an output layer.

[0083] In the work estimation model 322, the input layer is a layer to which one day's worth of performance data is input. The intermediate layer may be a layer in which parameters are learned based on past work schedule data, past performance data, and factor data. The past work schedule data is data indicating the relationship between at least one work indicated as scheduled to be performed in a past waterworks construction plan and the time period during which the work is performed. The past performance data is data indicating the relationship between at least one work performed in past waterworks construction and the time period during which the work is performed. The factor data is data that influenced the work in past waterworks construction. In addition, the output layer causes the computer to function so that, when one day's worth of performance data is input to the input layer, the work schedule data is output from the output layer after being calculated by the intermediate layer. That is, the second output unit 317 outputs the work schedule data by inputting the captured image to the work estimation model 322.

[0084] For example, a plurality of pieces of the above-mentioned past task schedule data, past performance data, and factor data are prepared as training data. Examples of information contained in the factor data include the following information. (1) Information indicating day / night. (2) Weather information (e.g., sunny, cloudy, rainy, or foggy, etc.). (3) Information on the ditch (e.g., ditch width, ditch depth, soil cover, offset value (distance from the sidewalk to the location where the pipe is laid), or information indicating whether or not there is a spring). (4) Information on the soil (e.g., information indicating the hardness of the soil, whether excavated soil will be used for backfilling, or the thickness of the asphalt pavement, etc.). (5) Road information (e.g., information indicating the number of lanes, the presence or absence of sidewalks, the slope angle, whether or not there are road closures, or the width of the work zone, etc.). (6) Pipe information (e.g., pipe type, nominal diameter, number of straight / irregular pipes, pipe alignment, presence or absence of buried objects, location of buried objects, or information indicating the need to remove buried objects, etc.). (7) Information on the work machinery (e.g., information indicating the model, horsepower, number of units, or bucket model number, etc.) (8) Information on the work content (e.g., information indicating the track record of digging by hand or using a backhoe, or the track record of work other than excavation, etc.). (9) Tool information (e.g., information indicating the model number of a lever hoist, etc.). (10) Information about workers (e.g., number of workers, age, gender, or years of work experience, etc.). (11) Information on the surrounding environment (information showing the location of the material storage area relative to the work site, etc.).

[0085] The past performance data may include, for example, information indicating the past performance values ​​of the work hours in plumbing work carried out by a specific contractor who is carrying out the work to be estimated. The past performance data may also include information indicating the past performance values ​​of the work hours in plumbing work carried out nationwide by contractors other than the specific contractor. The factor data may also include the results of analysis of incorrect work procedures, dangerous work, and the causes of dangerous work (information based on past accident cases) that occurred in these plumbing works. When sling belts are the subject of analysis, the analysis results may include information such as the number of sling belts, the hanging angle, and the attachment position relative to the pipe.

[0086] The past work schedule data may be, for example, work schedule data including work procedures that were indicated as work to be carried out in past construction plans for plumbing work by the specific contractor.

[0087] The intermediate layer of the work estimation model 322 may be a layer in which parameters are learned based on work change related data and past progress data as training data. Work change related data is data that indicates the relationship between the result of changing the work process order when the work process order is changed in past plumbing work and the change in work time before and after the change in the work process order. In other words, the work change related data is data that indicates how the work time has changed (for example, how much it has been reduced) as a result of reviewing the work process order. Past progress data is data that indicates the progress rate (progress rate) of work in a specified work time with respect to the entire work of past plumbing work, associated with past performance data. The progress rate can also be called an achievement rate.

[0088] In this case, when one day's worth of performance data is input to the input layer, the middle layer of the work estimation model 322 executes the following estimation process as part of the above calculation. That is, the middle layer of the work estimation model 322 estimates the progress rate of the work indicated by the input performance data relative to the entire waterworks work, based on the progress rate of the work indicated by the past progress data. In terms of estimating this progress rate, the progress rate of the work associated with the past performance data can be defined as correct answer data.

[0089] The intermediate layer estimates the work process sequence for the entire work time of the next day based on the estimated progress rate and the work change related data. For example, when the estimated progress rate is less than the planned progress rate, the intermediate layer can execute a calculation to create work schedule data including a work process sequence that can improve the progress rate, taking into account the relationship between changes in the work process sequence and changes in work time in past waterworks works. Note that the estimated progress rate being less than the planned progress rate means that the work is behind schedule.

[0090] In this way, the task estimation model 322 constructed using the past progress data and task change related data as training data can output task schedule data in which the tasks for the next day have been revised according to the progress of the tasks on a given day.

[0091] Note that, of the above-mentioned past performance data, only data with a progress rate equal to or greater than a predetermined threshold may be used as training data. In this case, past performance data with a low progress rate is not used as training data, so the task estimation model 322 can output task schedule data in which a decrease in the progress rate is suppressed. Therefore, in this case, the task estimation model 322 can output more reliable task schedule data.

[0092] Furthermore, the second output unit 317 may input procedure suitability data to the work estimation model 322, thereby identifying in the work schedule data an operation that is estimated to have a possibility of an error in the work procedure during all work hours of the next day. Furthermore, the second output unit 317 may input risk occurrence data to the work estimation model 322, thereby identifying in the work schedule data an operation that is estimated to have a possibility of a dangerous operation during all work hours of the next day. Furthermore, the second output unit 317 may input quality good / bad data to the work estimation model 322, thereby identifying in the work schedule data an operation that is estimated to have poor quality during all work hours of the next day.

[0093] For example, when the work scheduled data contains an identical task to one identified as a task in which an error has occurred in the procedure in the procedure suitability data, the work estimation model 322 identifies that task in the work scheduled data. Similarly, when the work scheduled data contains an identical task to one identified as a dangerous task in the risk occurrence data, the work estimation model 322 identifies that task in the work scheduled data. Furthermore, when the work scheduled data contains an identical task to one identified as having poor quality in the quality pass / fail data, the work estimation model 322 identifies that task in the work scheduled data.

[0094] The display control unit 318 may, for example, cause the display device 4 to display the work schedule data output by the second output unit 317. FIG. 6 is a diagram showing an example of the display of work schedule data. In the example of FIG. 6, the work schedule data output by the second output unit 317 and estimated to be optimal as the work schedule for the next day is shown as an implementation schedule for the next day. The example of FIG. 6 shows that a dangerous work may occur in time period Tp based on the risk occurrence data, and that an error may occur in the work procedure in time period Tq based on the procedure suitability data. In addition, the example of FIG. 6 shows that the quality may be poor in time period Tr based on the quality pass / fail data.

[0095] The display control unit 318 may display, separately from the work schedule data, the work that the second output unit 317 estimates may be dangerous, the work that the second output unit 317 estimates may have an error in the work procedure, and the work that the second output unit 317 estimates may have poor quality. In this case, the display control unit 318 can notify the manager or worker of these works.

[0096] Then, as described above, by displaying the work schedule data, which is the output result of the second output unit 317, on the display device 4, the display device 4 can function as a suggestion device that suggests revising the work schedule for the next day.

[0097] <Example of using the judgment results based on the work status judgment results> FIG. 7 is a diagram showing an example of the relationship between the determination result by the work status determination unit 315 and the notification content by the display control unit 318. As described above, the work status determination unit 315 outputs the determination results for the work procedure, work quality, safety, and work time (presence or absence of delay) of each work. As shown in FIG. 7, the notification content can be determined based on a combination of the determination results for the work procedure, work quality, safety, and work time of each work. In addition, in the "work time" in FIG. 7, "short" indicates that the time required for the work is shorter than the time indicated in the work process data, and "long" indicates that the time required for the work is longer than the time indicated in the work process data. Also, "standard" indicates that the time required for the work is approximately equal to the time indicated in the work process data.

[0098] For example, as shown in Fig. 7, consider a case where the work status determination unit 315 determines that the work procedure is appropriate, the work quality is good, and the work is safe, but the work time is shorter than planned. In this case, it is possible to propose shortening the work time from the next day onwards to be shorter than the planned work time, and to distribute the workers' manpower to other sites. Also, if it is determined that the work time is longer than planned, it is possible to propose lengthening the work time from the next day onwards to be longer than the current work time, and to replenish the workers' manpower.

[0099] 7, for work that the work status determination unit 315 determines to have an error in the work procedure and work that the work quality is poor, the unit can instruct the user to review the work. For work that the work status determination unit 315 determines to be dangerous, the unit can warn the user that the work is dangerous and suggest ways to improve the work. Even when the work status determination unit 315 makes these determinations, if the unit determines that the work time is shorter or longer than planned, the unit may make the above-mentioned suggestions regarding the work time.

[0100] The display control unit 318 may cause the display device 4 to display the notification content according to the determination result of the work state determination unit 315. In this case, too, the display device 4 may function as a suggestion device that proposes a review of the work schedule for the next day. For example, the display control unit 318 may cause the display device 4 to display an improvement method associated with each dangerous work as an improvement method. Furthermore, the second output unit 317 outputs work schedule data and the like as a result of inputting performance data and the like into the work estimation model 322. Therefore, the display control unit 318 may execute a proposal regarding the length of work time by displaying the work schedule data that reflects the length of work time for that day, output by the second output unit 317. Furthermore, the display control unit 318 may execute a warning of dangerous work by displaying the work schedule data that identifies the dangerous work, output by the second output unit 317. Note that the display control unit 318 may cause the display device 4 to display the notification content in real time at the time when the determination result by the work state determination unit 315 is obtained.

[0101] <Example of output from the second output section> For example, the second output unit 317 may create a dangerous work list with captured images based on the results of the analysis of the dangerous work records and the causes of the dangerous work, and the captured images when the dangerous work occurred. This dangerous work list can be used to assist in the creation of a safety management document. The second output unit 317 may also create work procedure data (work procedure documents) based on the results of the analysis of the past performance data, the records of incorrect work procedures, the records of dangerous work, and the causes of the dangerous work. The second output unit 317 may create the work procedure data in a format that allows reference to past work procedures, for example.

[0102] Furthermore, the task estimation model 322 may be constructed so as to estimate a task to be performed in the next specified task time from the task results in a specified task time. For example, the task estimation model 322 may output not only task schedule data for the next day, but also task schedule data for the day after that. Furthermore, the task estimation model 322 may receive input of task results data up to the time of acquisition of the photographed image, and output task schedule data thereafter.

[0103] Furthermore, it is sufficient that at least one of the procedure suitability data, risk occurrence data, and quality data is input to the task estimation model 322, rather than all of these data. That is, the second output unit 317 only needs to output at least one of tasks estimated to be potentially dangerous, tasks estimated to be potentially erroneous in the task procedure, and tasks estimated to be potentially of poor quality. The display control unit 318 only needs to display the tasks output by the second output unit 317.

[0104] [Processing in the work management device] Fig. 8 is a flowchart showing an example of processing (work management method) in the work management device 3. As shown in Fig. 8, the image acquisition unit 311 acquires a captured image from the camera 2 (S1: image acquisition step). The first output unit 312 inputs the captured image acquired by the image acquisition unit 311 to the index estimation model 321 (S2). The first output unit 312 outputs the estimated detection class as an output result of the index estimation model 321 (S3: output step, first output step).

[0105] The position identification unit 313 estimates the position of the detection class output by the first output unit 312 in the captured image (S4). The work estimation unit 314 analyzes the combination of detection classes output over time by the first output unit 312 to estimate the work being performed at the time the captured image was acquired (i.e., the work currently being performed) at the position of the detection class identified by the position identification unit 313 (S5: work estimation step). When the work estimation unit 314 determines that the captured image includes scenes of multiple plumbing works being performed based on the positions of the detection classes, it estimates the work being performed for each plumbing work.

[0106] The work state determination unit 315 determines the work state of the work estimated by the work estimation unit 314 by analyzing the combination of detection classes output over time by the first output unit 312 (S6). The display control unit 318 causes the display device 4 to display the work state determination result by the work state determination unit 315 (S7).

[0107] Furthermore, the data creation unit 316 creates performance data (performance process chart) up to the time point when the most recent captured image was acquired by analyzing the combination of detection classes output over time by the first output unit 312 (S8). The display control unit 318 displays the performance data created by the data creation unit 316 on the display device 4 in real time (S9). Note that in S8, the data creation unit 316 may create procedure suitability data (e.g., work procedure chart), risk occurrence data (e.g., risk work chart), and quality pass / fail data (e.g., quality control chart) up to the time point when the most recent captured image was acquired. In this case, the display control unit 318 may display the procedure suitability data, risk occurrence data, and quality pass / fail data on the display device 4.

[0108] The control unit 31 determines whether or not the work for one day has been completed (S10). The control unit 31 may determine that the work for one day has been completed, for example, when the transmission of captured images from the camera 2 has ceased for a predetermined period of time or when a user input has been received.

[0109] When the control unit 31 determines that the work for one day has been completed (YES in S10), the data creation unit 316 creates one day's performance data (e.g., an implementation schedule) (S11: data creation step). The second output unit 317 inputs the one day's performance data created by the data creation unit 316 to the work estimation model 322 (S12). The second output unit 317 outputs the next day's work schedule data, which estimates the work to be performed on the next day and the time period during which the work will be performed, as the output result of the work estimation model 322 (S13: second output step). The display control unit 318 causes the display device 4 to display the next day's work schedule data output by the second output unit 317 (S14). On the other hand, when the control unit 31 determines that the work for one day has not been completed (NO in S10), the process returns to S1.

[0110] In S11, the data creation unit 316 may create one day's worth of procedure suitability data (e.g., a work procedure chart), risk occurrence data (e.g., a dangerous work chart), and quality pass / fail data (e.g., a quality control chart). In this case, the second output unit 317 may input these data to the work estimation model 322 to specify in the work schedule data the work that is estimated to have a possible error in the work procedure, the work that is estimated to have a possible dangerous work, and the work that is estimated to have poor quality. Then, the display control unit 318 may display the work schedule data in which these works are specified on the display device 4. In addition to the work schedule data, the display control unit 318 may display the above-mentioned works estimated by the work estimation model 322 on the display device 4.

[0111] Furthermore, the display control unit 318 may display on the display device 4 the one-day performance data, procedure suitability data, risk occurrence data, and quality pass / fail data created in S11.

[0112] [Modification of the work management device] In this embodiment, the work management device 3 has been described as outputting the work status determination result and outputting the work schedule data. However, the work management device 3 may have only the first function of outputting the work status determination result, or may have only the second function of outputting the work schedule data.

[0113] The work management device 3 realizing the first function may include an image acquisition unit 311, a first output unit 312 that outputs the detection class estimated by the index estimation model 321, and a work estimation unit 314 that estimates the work currently being performed based on the detection class.

[0114] Furthermore, the work management device 3 realizing the second function may include the following configuration in addition to the image acquisition unit 311 and the first output unit 312. The work management device 3 may include a data creation unit 316 that creates performance data by analyzing the detection class output by the first output unit 312, and a second output unit 317 that outputs work schedule data by inputting the performance data to the work estimation model 322.

[0115] [Major Effects] (1) The work management device 3 having the above-mentioned first function can automatically estimate the work currently being performed simply by acquiring a photographed image. Therefore, a manager or worker can easily manage the work currently being performed.

[0116] Plumbing work involves many uncertainties, such as sudden changes to planned work on the day of construction. However, the work management device 3 allows managers or workers to manage the work currently being carried out, and therefore to be aware of any work that has been suddenly changed.

[0117] Furthermore, the work management device 3 only needs to be able to acquire captured images from the camera 2, and does not necessarily need to be installed at the site of the waterworks work or realized by electronic equipment carried by the worker. The same applies to the display device 4 that displays the output results of the work management device 3. Therefore, a manager such as a site supervisor who manages the waterworks work can remotely grasp the work currently being performed that has been estimated by the work management device 3, even if he or she is not at the site.

[0118] The work management device 3 also includes a work status determination unit 315 that determines whether the work status of the estimated work is appropriate by analyzing the detection class. Therefore, the manager or worker can understand whether the current work status is appropriate. There are a wide variety of management items (e.g., appropriateness of work procedures, quality of work, safety, and whether or not there is a delay in work time) that the manager must manage in daily waterworks work. The work management device 3 determines whether the work status indicated by these management items is appropriate, so that the manager or worker can easily manage multiple management items.

[0119] Furthermore, in plumbing work, checking the progress of daily work and checking the work content (e.g., checking safety) requires time and experience. The work management device 3 can estimate the appropriateness of the work being performed and the work status simply by acquiring photographed images, so that managers or workers can easily check the progress of daily work and check the work content without relying on experience.

[0120] Furthermore, the work management device 3 can estimate the work being performed and the appropriateness of the work status for each of multiple waterworks construction sites by acquiring photographed images from each of the sites. In the future, it is expected that there will be an increase in cases where one manager will be managing multiple sites simultaneously due to labor shortages, but the work management device 3 allows managers or workers to easily and simultaneously manage the work being performed at multiple sites and the appropriateness of the work status.

[0121] (2) The work management device 3 having the second function can automatically create the work schedule data for the next day just by acquiring the captured images. Therefore, the work management device 3 can easily increase the possibility that the work for the next day will proceed as planned.

[0122] In addition, in waterworks construction, reviewing the implementation schedule according to the work situation requires time and experience. The work management device 3 automatically outputs the work schedule data for the next day just by acquiring the photographed image, so the manager can easily improve the work efficiency without relying on experience.

[0123] [Software implementation example] The control block (particularly the control unit 31) of the work management device 3 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software.

[0124] In the latter case, the work management device 3 includes a computer that executes instructions of a program, which is software that realizes each function. The computer includes, for example, one or more processors, and a computer-readable recording medium that stores the program. The object of the present invention is achieved by the processor reading the program from the recording medium and executing it in the computer. The processor may be, for example, a CPU (Central Processing Unit). The recording medium may be a "non-transient tangible medium", such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like. The device may further include a RAM (Random Access Memory) that expands the program. The program may be supplied to the computer via any transmission medium (such as a communication network or a broadcast wave) that can transmit the program. Note that one aspect of the present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0125] [Other Descriptions of the Disclosure] A work management device according to one embodiment of the present invention includes an image acquisition unit that acquires a captured image including a work area in plumbing work, a first output unit that outputs at least one indicator that is estimated to have appeared in the work at the time the captured image is acquired by inputting the captured image into an indicator estimation model constructed using machine learning to estimate at least one indicator that is appearing in the work at a certain point in time out of multiple indicators for evaluating the work status of the plumbing work, and a work estimation unit that estimates the work being performed at the time the captured image is acquired based on the output result of the first output unit.

[0126] A work management method according to one embodiment of the present invention includes an image acquisition process for acquiring a photographed image including a work area in plumbing work, an output process for outputting at least one indicator estimated to have appeared in the work at the time the photographed image was acquired by inputting the photographed image into an indicator estimation model constructed using machine learning to estimate at least one indicator that appears in the work at a certain point in time out of multiple indicators for evaluating the work status of the plumbing work, and a work estimation process for estimating the work being performed at the time the photographed image was acquired based on the output result of the output process.

[0127] According to the above configuration, by inputting a photographed image into the index estimation model, a combination of indices at the time of capturing the photographed image can be estimated. Therefore, based on the estimated combination of indices, the work being performed at the time of capturing the photographed image can be estimated. Therefore, the person involved in the plumbing work can manage the work being performed.

[0128] In a work management device according to one embodiment of the present invention, the work estimation unit may estimate the work being performed at the time when the most recent captured image was acquired by the image acquisition unit by analyzing a combination of indicators that appear at each of the capture times output by the first output unit over time.

[0129] According to the above configuration, the task currently being performed can be estimated by analyzing a combination of indices estimated from the current and previous captured images.

[0130] A work management device according to one embodiment of the present invention may include a work state determination unit that determines the appropriateness of the work state of the work estimated by the work estimation unit by analyzing a combination of indicators that appear at each of the points in time when the captured images were acquired, which are output over time by the first output unit.

[0131] According to the above configuration, the suitability of the current work status can be determined by analyzing a combination of indices estimated from the current and previous captured images. Therefore, those involved in the plumbing work can understand the suitability of the current work status.

[0132] In a work management device according to one embodiment of the present invention, the work status determination unit may determine as the appropriateness of the work status at least one of whether or not the work procedure is appropriate, whether or not safety is guaranteed, and whether or not a delay has occurred in the work.

[0133] According to the above configuration, it is possible to determine whether the work procedures are appropriate, whether safety is ensured, and / or whether the work is delayed, as the current work status. Therefore, those involved in the plumbing work can grasp the appropriateness of the work procedures, safety, and / or the progress of the work.

[0134] The work management device according to one aspect of the present invention may further include a warning unit that warns persons involved in the plumbing work when the work status determination unit determines that the work status is not good.

[0135] According to the above configuration, it is possible to warn people involved in the plumbing work that the current work conditions are undesirable.

[0136] A work management device according to one embodiment of the present invention includes a position identification unit that identifies a position of an indicator output by the first output unit in the captured image, and when the plumbing work is being carried out in different areas within the captured image, the work estimation unit may estimate the work being carried out in each of the different areas based on the position of the indicator identified by the position identification unit.

[0137] According to the above configuration, when a captured image includes a scene in which a plurality of tasks are being performed, each of the plurality of tasks can be estimated from the captured image.

[0138] A work management device according to one embodiment of the present invention may include a data creation unit that creates performance data indicating a relationship between at least one work performed during a specified work time and a time period during which the work was performed, by analyzing a combination of indicators that appear at each of the points in time when the captured images were acquired within the specified work time, which are output over time by the first output unit.

[0139] According to the above configuration, the user can grasp the performance (progress) of the work performed during a predetermined work time (eg, work time set per day (eg, 8:00 to 16:30)).

[0140] In a work management device according to one embodiment of the present invention, the data creation unit may create at least one of procedure adequacy data indicating the adequacy of work procedures during a specified work time, and hazard occurrence data indicating whether or not dangerous work has occurred during the specified work time, by analyzing combinations of indicators that appear at each of the points in time when the captured images were acquired within a specified work time, which are output over time by the first output unit.

[0141] According to the above configuration, people involved in plumbing work can understand whether the work procedures for work performed during a specified work period were correct and / or whether any dangerous work occurred.

[0142] A work management device according to one embodiment of the present invention includes an image acquisition unit that acquires a captured image including a work area in plumbing work; a first output unit that outputs at least one indicator that is estimated to have appeared in the work at the time the captured image is acquired by inputting the captured image into an indicator estimation model constructed using machine learning to estimate at least one indicator that appears in the work at a certain point in time among multiple indicators for evaluating the work status in the plumbing work; a data creation unit that creates performance data indicating the relationship between at least one task performed in a specified work time and the time period during which the task is performed by analyzing combinations of indicators that appear at each of the times when the captured image is acquired within a specified work time, which are output over time by the first output unit; and a second output unit that outputs work schedule data indicating the relationship between at least one task estimated to be performed in the next specified work time and the time period during which the task is performed by inputting the performance data into a work estimation model constructed using machine learning to estimate work to be performed in the next specified work time from the performance of the work in the specified work time.

[0143] A work management method according to one embodiment of the present invention includes an image acquisition step of acquiring a photographed image including a work area in plumbing work; a first output step of inputting the photographed image into an index estimation model constructed to estimate at least one index appearing in the work at a certain point in time among a plurality of indexes for evaluating the work status in the plumbing work using machine learning, thereby outputting at least one index estimated to have appeared in the work at the time the photographed image was acquired; a data creation step of creating performance data indicating a relationship between at least one work performed in a specified work time and a time period during which the work was performed by analyzing a combination of indices that appear at each of the times when the photographed image was acquired within a specified work time, which are output over time in the first output step; and a second output step of outputting work schedule data indicating a relationship between at least one work estimated to be performed in the next specified work time and a time period during which the work was performed, by inputting the performance data into a work estimation model constructed to estimate work to be performed in the next specified work time from the performance of the work in the specified work time using machine learning.

[0144] According to the above configuration, it is possible to create work schedule data for the next specified work time (e.g., the work time specified for the next day (e.g., 8:00-16:30)) taking into account the work results for the current specified work time. This increases the likelihood that work will proceed as planned in the next specified work time. In addition, by outputting the work schedule data, it is possible to suggest a review of the work schedule for the next specified work time.

[0145] In a work management device according to one embodiment of the present invention, the data creation unit creates at least one of procedure suitability data indicating the suitability of work procedures during a specified work time and hazard occurrence data indicating whether or not dangerous work has occurred during the specified work time by analyzing combinations of indicators that appear at each of the points in time at which the captured images were acquired within a specified work time, which are output by the first output unit over time, and the second output unit (1) identifies in the work schedule data a work that is estimated to have a possibility of an error in the work procedures during the next specified work time by inputting the procedure suitability data into the work estimation model, or (2) identifies in the work schedule data a work that is estimated to have a possibility of an error in the work procedures during the next specified work time by inputting the hazard occurrence data into the work estimation model, or (3) identifies both the work that is estimated to have a possibility of an error in the work procedures in (1) and the work that is estimated to have a possibility of dangerous work in (2) in the work schedule data.

[0146] According to the above configuration, work that may cause an error in the work procedure at the next specified work time and / or dangerous work that may occur at the next specified work time can be reflected in the work schedule data. Therefore, by checking the work schedule data, people involved in plumbing work can know in advance work that may cause an error in the work procedure and / or dangerous work that may occur.

[0147] An index estimation model according to one embodiment of the present invention comprises an input layer to which a captured image including a work area in plumbing work is input, an output layer to output at least one of a plurality of indexes for evaluating the work status in the plumbing work, and an intermediate layer in which parameters are learned based on an image including the work area and an image showing items used in the plumbing work, and the indexes, and causes a computer to function such that when the captured image is input to the input layer, after calculations by the intermediate layer, the output layer outputs at least one indicator that is estimated to have appeared in the work at the time the captured image was obtained.

[0148] According to the above configuration, by inputting a photographed image to the index estimation model, a combination of indices at the time of capturing the photographed image can be estimated. Therefore, it is possible to estimate the work being performed at the time of capturing the photographed image based on the estimated combination of indices.

[0149] A work estimation model according to one embodiment of the present invention comprises an input layer to which actual data indicating the relationship between at least one work performed during a specified work time and the time zone when the work was performed is input, an output layer to output work schedule data indicating the relationship between at least one work in the next specified work time and the time zone when the work was performed, and an intermediate layer in which parameters are learned based on past work schedule data indicating the relationship between at least one work shown as scheduled to be performed in a construction plan for past plumbing work and the time zone when the work was performed, past actual data indicating the relationship between at least one work performed in the past plumbing work and the time zone when the work was performed, and factor data that influenced the work in the past plumbing work, and causes a computer to function so that when the actual data is input to the input layer, the work schedule data is output from the output layer after calculation by the intermediate layer.

[0150] According to the above configuration, the work schedule data for the next specified work time can be created by taking into account the work results for the current specified work time. Therefore, it is possible to increase the possibility that the work will proceed as scheduled in the next specified work time. In addition, by outputting the work schedule data, it is possible to suggest a review of the work schedule for the next specified work time.

[0151] In a work estimation model according to one embodiment of the present invention, the intermediate layer further learns the parameters based on work change-related data indicating the relationship between the result of changing the work process order when the work process order is changed in the past plumbing work and the change in work time before and after the change in the work process order, and past progress data indicating the progress of work at the specified work time with respect to the entire work of the past plumbing work, which is associated with the past actual data, and when the actual data is input to the input layer, the intermediate layer may, as part of the calculation, estimate the progress of the work indicated by the actual data with respect to the entire plumbing work, and estimate the work process order for the next specified work time based on the estimated progress and the work change-related data.

[0152] According to the above configuration, it is possible to output work schedule data in which the work for the next specified work time has been revised in accordance with the progress of the work during the current specified work time.

[0153] [Additional Notes] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0154] 3 Work management device 311 Image Acquisition Unit 312 First output section 313 Location identification part 314 Work Estimation Department 315 Working condition determination unit 316 Data Creation Department 318 Display control section (warning section) 317 Second output section 321 Index Estimation Model 322 Work Estimation Model

Claims

1. an image acquisition unit for acquiring a photographed image including a work area in plumbing work; a first output unit that outputs at least one index that is estimated to be appearing in the work at the time the photographed image is acquired by inputting the photographed image into an index estimation model that is constructed using machine learning to estimate at least one index that is appearing in the work at a certain point in time out of a plurality of indexes that indicate each of a plurality of things involved in the work in the waterworks work, or a combination of these; a data creation unit that creates performance data indicating a relationship between at least one task performed during the specified work period and a time period during which the task was performed by identifying a task performed at a certain time within the specified work period that is the time when the photographed image was captured, based on data that has been prepared in advance and that associates each task that may be performed during the waterworks work with a combination of the indicators, and the combinations of indicators that appear at each of the time points when the photographed image was captured within the specified work period, which are output over time by the first output unit; a second output unit that outputs work schedule data indicating a relationship between at least one work estimated to be performed in the next specified work time and a time zone for performing the work by inputting the performance data into a work estimation model that is constructed using machine learning to estimate a work to be performed in a next specified work time from the performance data of the work in the specified work time, The index estimation model is constructed by performing machine learning using an article image indicating an article used in the waterworks and the index as training data, The work estimation model is constructed by performing machine learning using as training data past work schedule data showing the relationship between at least one work scheduled to be performed in the construction plan of past plumbing work and the time period when the work was performed, past performance data showing the relationship between at least one work performed in the past plumbing work and the time period when the work was performed, and factor data that influenced the work in the past plumbing work, the work management device.

2. The work management device of claim 1, further comprising a work estimation unit that estimates the work being performed at the time the captured image is acquired based on data that corresponds each work that may be performed in the pre-prepared plumbing work with a combination of the indicators and the output result of the first output unit.

3. a work state determination unit that determines whether or not a delay is occurring in the work as a suitability of the work state of the work estimated by the work estimation unit by analyzing a combination of indicators that appear at each of the acquisition points of the photographed images output over time by the first output unit, 3. The work management device according to claim 2, wherein the work status determination unit determines whether or not a delay has occurred in the work by comparing the work time of the work estimated by the work estimation unit with work process data indicating a relationship between the work to be performed at a specified work time and a time zone for performing the work.

4. The work management device according to claim 3 , further comprising a warning unit that warns people involved in the plumbing work when the work status determination unit determines that the work status is not good.

5. An image acquisition step of acquiring a photographed image including a work area in plumbing work; a first output step of inputting the photographed image into an indicator estimation model constructed using machine learning to estimate at least one indicator appearing in the work at a certain point in time out of a plurality of indicators indicating each of a plurality of things involved in the work in the waterworks work, or a combination thereof, and outputting at least one indicator estimated to appear in the work at the time the photographed image was acquired; a data creation process for creating performance data showing a relationship between at least one task performed during the specified work period and a time period during which the task was performed by identifying a task performed at a certain time within the specified work period that is the time when the photographed image was captured, based on data prepared in advance that associates each task that may be performed during the waterworks work with a combination of the indicators, and the combinations of indicators that appear at each of the time points when the photographed image was captured within the specified work period, which are output over time in the first output process; a second output step of inputting the performance data into a task estimation model constructed using machine learning to estimate a task to be performed in a next specified task time from the performance data of the task in the specified task time, thereby outputting task schedule data indicating a relationship between at least one task estimated to be performed in the next specified task time and a time zone for performing the task, The index estimation model is constructed by performing machine learning using an article image indicating an article used in the waterworks and the index as training data, A work management method in which the work estimation model is constructed by performing machine learning using as training data past work schedule data showing the relationship between at least one work scheduled to be performed in a construction plan for past plumbing work and the time period when the work was performed, past performance data showing the relationship between at least one work performed in the past plumbing work and the time period when the work was performed, and factor data that influenced the work in the past plumbing work.

Images