Work management device, work management method, and work execution estimation model

Abstract

To estimate work under execution with high accuracy.SOLUTION: A work management device (3) comprises: a data generation unit (336) that generates time-series data of coordinates; a linear data extraction unit (337) that extracts linear data from the time-series data of coordinates; and a work estimation unit (339) that estimates work that is executed at the time of acquisition of a photographic image based on the linear data and objects included in the photographic image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a work management device and the like. [Background technology]

[0002] The plumbing work management system of Patent Document 1 estimates the current work status of plumbing work from characteristic images detected from captured images that include the work area in the plumbing work, and determines the content of the notification to those involved in the plumbing work based on the estimated current work status. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-107341 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 does not specifically disclose how to estimate the work being performed by taking into account the movement of a feature image in a captured image over time. One aspect of the present invention aims to realize a work management device or the like that can accurately estimate the work being performed by taking into account the movement of an object included in a captured image. [Means for solving the problem]

[0005] In order to solve the above problems, a work management device according to one embodiment of the present invention comprises an image acquisition unit that acquires photographed images over time that include work areas in plumbing work, an image extraction unit that extracts an image of at least one object involved in the plumbing work from each of the photographed images acquired over time, a data generation unit that generates time series data of the coordinates by extracting, over time, coordinates within an area including each image of an object that is estimated to be the same in each of the photographed images, a linear data extraction unit that extracts linear data, which is a portion whose value changes continuously and regularly, from the time series data of the coordinates, and a work estimation unit that estimates the work being performed at the time the photographed images were acquired based on the linear data and images of the objects included in the photographed images.

[0006] In order to solve the above problems, a work management method according to one embodiment of the present invention includes an image acquisition process for acquiring photographed images over time that include a work area in plumbing work; an image extraction process for extracting an image of at least one object involved in the plumbing work from each of the photographed images acquired over time; a data generation process for generating time series data of coordinates by extracting, over time, coordinates within an area that includes each image of an object that is estimated to be the same in each of the photographed images; a linear data extraction process for extracting linear data, which is a portion whose value changes continuously and regularly, from the time series data of coordinates; and a work estimation process for estimating the work being performed at the time the photographed images were acquired based on the linear data and the object from which the time series data of coordinates from which the linear data was extracted was generated.

[0007] A portion of the coordinate time series data whose values ​​change continuously and regularly can be said to indicate that the object from which the coordinate time series data was generated is moving during the time period in which that portion is extracted.

[0008] Therefore, according to the above configuration, the task management device can estimate the task in progress based on the object included in the captured image by also using linear data, taking into account the movement of the object during the time period. Therefore, the task management device can improve the estimation accuracy of the task in progress.

[0009] In one embodiment of the work management device of the present invention, a frame setting unit is provided that sets a frame around the image of the object or around a portion of the image of the object according to the size of the image of the object, and the data generation unit may extract coordinates of a predetermined position in the frame as coordinates within the area.

[0010] According to the above configuration, the work management device can easily extract coordinates within the area of ​​the extracted image of the object.

[0011] In the work management device according to one aspect of the present invention, the data generation unit may normalize the coordinates using a size of an image of the object from which the coordinates are extracted.

[0012] The size of the image of the object depends on the position in the captured image. According to the above configuration, the work management device normalizes the coordinates according to the size of the image of the object from which the coordinates are extracted, thereby making it possible to extract the coordinates while reducing the possibility of the coordinates being affected by perspective in the captured image.

[0013] In a work management device according to one aspect of the present invention, the data generation unit may remove outliers from the time series data of the coordinates, and the linear data extraction unit may extract the linear data from the time series data of the coordinates from which the outliers have been removed.

[0014] According to the above configuration, the work management device can extract linear data more accurately.

[0015] A work management device according to one embodiment of the present invention includes an index output unit that outputs at least one index that is estimated to have appeared in the work at the time the photographed image is acquired by inputting the photographed image into an index estimation model constructed using machine learning to estimate at least one index that is appearing in the work at a certain point in time out of a plurality of indexes including the object for evaluating the work status of the plumbing work, and the work estimation unit may estimate the work being performed at the time the photographed image is acquired based on the linear data and the output result of the index output unit.

[0016] According to the above configuration, the task management device can estimate a combination of indices (objects) at the time of capturing the captured image by inputting the captured image to the index estimation model. Therefore, the task being performed can be estimated based on the estimated combination of indices.

[0017] In one embodiment of the work management device of the present invention, when the linear data extraction unit extracts the linear data, the indicator output as the object from which the time series data of the coordinates from which the linear data was extracted was generated may be weighted more heavily for a time period in which the linear data was extracted than for a time period other than the time period in which the linear data was extracted, assuming that movement of the object is occurring during the time period in which the linear data was extracted.

[0018] According to the above configuration, the work management device can weight a time period in which a movement occurs in an index (object) output from the index estimation model more than a time period other than the time period in question.

[0019] A work management device according to one embodiment of the present invention is equipped with a camera that photographs the work area, and the data generation unit may correct the coordinates based on a predetermined reference position of the camera and the position of the camera when the photographed image was acquired.

[0020] According to the above configuration, the work management device can generate time-series coordinate data with reduced effect of camera movement on coordinate extraction, i.e., under conditions substantially the same as when photographing with a fixed camera.

[0021] In a work management device according to one embodiment of the present invention, the work estimation unit may estimate the work being performed at the time the photographed image is acquired by inputting (i) at least one of a plurality of indicators including the object for evaluating the work status of the plumbing work, and (ii) the linear data, into a performed work estimation model constructed using machine learning to estimate the work being performed at the time the photographed image is acquired.

[0022] According to the above configuration, the work management device can estimate the work being performed by inputting the index (object) and the linear data to the performed work estimation model. Therefore, the work management device can further improve the estimation accuracy of the work.

[0023] A work management device according to one aspect of the present invention may include an output unit that can be communicatively connected to a notification unit and outputs at least one of the estimation result of the work estimation unit and the time series data of the coordinates from the notification unit.

[0024] According to the aforementioned configuration, the work management device can present at least one of the estimation result of the work estimation unit and the time-series data of the coordinates to the user.

[0025] In order to solve the above problems, one embodiment of the present invention provides a work execution estimation model that includes an input layer to which (i) at least one of a plurality of indices for evaluating the work status in plumbing work, and (ii) linear data that is a portion whose value changes continuously and regularly in time series data of coordinates included in an image of an object that is one of the indices included in the photographed images, which is extracted as a result of acquiring photographed images over time that include the work area in the plumbing work, is input; an output layer that outputs the work in the plumbing work; and an intermediate layer in which parameters are learned based on the plurality of indices, a plurality of the linear data, and a plurality of the work, and that causes a computer to function such that when the index and the linear data are input to the input layer, the output layer outputs the work being performed at the time the photographed image was acquired, after calculations by the intermediate layer.

[0026] According to the above configuration, the work management device is able to estimate the work being performed by inputting the index (object) and linear data to the performed work estimation model. Effect of the Invention

[0027] According to one aspect of the present invention, the task being performed can be accurately estimated by taking into account the movement of an object included in a captured image. [Brief description of the drawings]

[0028] [Figure 1] 1 is a block diagram showing an example of a schematic configuration of a waterworks construction management system. [Diagram 2] FIG. 1 is a diagram showing an example of a backhoe equipped with a camera. [Diagram 3] FIG. 11 is a diagram illustrating an example of an output result of an index output unit. [Figure 4] FIG. 13 is a diagram for explaining a frame that is set in a captured image. [Diagram 5] FIG. 11 is a diagram showing an example of time-series data of coordinates generated by a data generating unit. [Figure 6] FIG. 11 is a diagram showing an example of an output result of a weighting processing unit. [Figure 7] 11 is a diagram for explaining correction of coordinates when a camera moving parallel to an object captures the object. FIG. [Figure 8] 11 is a diagram for explaining correction of coordinates when a camera approaching an object captures the object; FIG. [Figure 9] 11A and 11B are diagrams for explaining correction of coordinates when an object is photographed while the camera is rotating about the vertical direction as the rotation axis. [Figure 10] 10 is a flowchart illustrating an example of a process in the work management device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] [Configuration of the waterworks construction management system] 1 is a block diagram showing an example of a schematic configuration of a plumbing work management system 1 according to an embodiment of the present invention. The plumbing work management system 1 manages various works for plumbing work such as installing pipes (e.g., progress of the plumbing work). The plumbing work management system 1 includes a camera 2, a work management device 3, and a display device 4.

[0030] The camera 2 photographs various tasks carried out at one or more plumbing work sites. The camera 2 may be capable of photographing the work area where the various tasks are performed. Examples of the camera 2 include a camera attached to the helmet or work clothes (e.g., the arm part of the work clothes) of a worker (a person involved in the plumbing work) to photograph the work at hand, or a camera mounted on various work tools (e.g., a backhoe). Examples of the camera 2 include a camera installed near the work area.

[0031] FIG. 2 is a diagram showing an example of a backhoe BH equipped with a camera 2. The backhoe BH is equipped with a first camera 2A and a second camera 2B as the camera 2. The first camera 2A is attached to the upper surface of the cabin CA of the backhoe BH. The shooting direction of the first camera 2A may be in front of and behind the driver's seat of the cabin CA. The second camera 2B is attached to the arm AR so that the shooting direction faces the bucket BU side. This allows the camera 2 to shoot the surroundings of the backhoe BH, the bucket BU, and the surroundings of the bucket BU. However, the backhoe BH may be configured to mount only the first camera 2A or the second camera 2B. Also, the shooting direction of the first camera 2A may be only in front of and behind the driver's seat. Also, the first camera 2A may be able to shoot the environment around the backhoe BH by rotating around the vertical direction as the rotation axis.

[0032] The photographing unit 21 photographs at least the work target area in the plumbing work. The photographing unit 21 may photograph still images or may photograph moving images. Photographing may be performed multiple times within the same plumbing work. The work target area (and therefore the camera 2) may be fixed or may be movable with each photographing session as a unit. In other words, the photographing unit 21 creates a photographed image including the work target area each time photographing is performed.

[0033] The position sensor 23 is a sensor that measures the position (absolute coordinates) and orientation of the camera 2 on the ground. The position sensor 23 measures the position of the camera 2 by receiving, for example, a GPS (Global Positioning System) signal. The position sensor 23 also includes, for example, a geomagnetic sensor as a sensor that measures the orientation. The work management device 3 can identify the shooting direction of the camera 2 based on the orientation measured by the position sensor 23.

[0034] The first communication unit 22 is communicatively connected to the work management device 3, and transfers data of the image captured by the imaging unit 21 and data indicating the position and orientation of the camera 2 measured by the position sensor 23 to the work management device 3. The communication between the first communication unit 22 and the second communication unit 31 of the work management device 3 is wireless, but may be wired.

[0035] It should be noted that, when the camera 2 is a stationary camera, the camera 2 does not necessarily need to include the position sensor 23. That is, in this case, the camera 2 does not transmit data indicating the position and orientation of the camera 2 measured by the position sensor 23.

[0036] The work management device 3 manages various works being carried out as plumbing works. When managing works at a plurality of plumbing works sites, the work management device 3 manages the works for each site. Details of the work management device 3 will be described later.

[0037] The display device 4 displays various information related to the plumbing work to a manager who manages the plumbing work (a person involved in the plumbing work). The display device 4 displays, for example, various information output from the work management device 3. The display device 4 displays, for example, the results of work estimation by the work management device 3.

[0038] The display device 4 is a monitor or a personal computer on which a manager manages various tasks. However, the display device 4 may also be an electronic device (e.g., a tablet or a smartphone) carried by a worker. In this case, the display device 4 can display various information related to the plumbing work to the worker. The plumbing work management system 1 may also be equipped with a sound output device (e.g., a speaker) that outputs the various information by voice. In other words, the plumbing work management system 1 may be equipped with a notification unit, such as the display device 4 and the sound output device, that can be communicatively connected to the work management device 3 and that notifies the worker of various data transmitted from the work management device 3.

[0039] [Configuration of the work management device] The work management device 3 includes a second communication unit 31, a control unit 33, and a memory unit 35. The second communication unit 31 receives captured image data and data indicating the position and orientation of the camera 2 from the camera 2. The control unit 33 comprehensively controls each unit of the work management device 3. The memory unit 35 stores various programs and various data used by the control unit 33. The memory unit 35 stores, for example, an index estimation model 351 and a performed work estimation model 352, which will be described later. The memory unit 35 may be realized by a storage device different from the work management device 3, but connected to the work management device 3 so as to be able to communicate with it.

[0040] The control unit 33 includes an image acquisition unit 331, an index output unit 332, a position identification unit 333, an image extraction unit 334, a frame setting unit 335, a data generation unit 336, a linear data extraction unit 337, a weighting processing unit 338, an activity estimation unit 339, and a display control unit 340.

[0041] The image acquisition unit 331 acquires, over time, captured images transferred from the camera 2, the captured images including the area to be worked on in the plumbing work.

[0042] The index output unit 332 inputs the captured image acquired by the image acquisition unit 331 to the index estimation model 351 stored in the storage unit 35. The index estimation model 351 is a trained model constructed using machine learning to estimate at least one index appearing in work at a certain point in time among multiple indexes for evaluating the work status in plumbing work. In this embodiment, the above index is referred to as a detection class that needs to be detected from the captured image to evaluate the work status.

[0043] Examples of detection classes include single objects such as various pipes (pipes) used in plumbing work, workers, ordinary people other than workers, various tools used in plumbing work, and ditches in which pipes are placed. Detection classes may be combinations of multiple such single objects, and for example, a combination of a ditch and a worker in the ditch may be set as one detection class.

[0044] Examples of various pipes include straight pipes and irregular pipes. Examples of various work implements (e.g., work machines (work vehicles) and tools) include asphalt cutters, backhoes, dump trucks, rammers, cones (or poles), sheet piles, ratchets, sleepers, and ladders. The work implements set as detection classes may be parts of the work equipment, and for example, a bucket and a sling belt that are parts of a backhoe may be set as detection classes.

[0045] As the detection class, an object that allows the work state to be determined may be set. For example, as the detection class, an object that characterizes proper work (standard work) and an object that characterizes dangerous work may be set. An example of an object that characterizes standard work is the working posture of a worker performing standard work. An example of an object that characterizes dangerous work is a groove where no safety cone is installed.

[0046] The index estimation model 351 is a trained model constructed to extract and output the above-mentioned detection classes contained in a captured image as a result of inputting the captured image. The index estimation model 351 may be, for example, a neural network (e.g., a Convolutional Neural Network (CNN)) including at least an input layer, an intermediate layer, and an output layer.

[0047] In the index estimation model 351, the input layer is a layer to which a captured image is input. The intermediate layer is a layer in which parameters are learned based on an area image including a work area, an item image showing an item used in plumbing work, and a detection class. The output layer is a layer that outputs at least one detection class.

[0048] For example, data in which a detection class is associated as correct answer data with each of a plurality of area images and each of a plurality of article images is prepared as training data. The index estimation model 351 is constructed by sequentially inputting training data and learning parameters (weights and biases) so as to minimize a loss function. The index estimation model 351 is constructed, for example, by a model generation device (not shown), but is not limited to this and may be constructed, for example, by the control unit 33. In this case, the control unit 33 also has a function as a learning device that constructs the index estimation model 351 by machine learning using the training data.

[0049] In addition, other trained models described later may also be constructed by sequentially inputting training data and learning parameters so as to minimize the loss function, and may be constructed by a model generation device or by the control unit 33.

[0050] The area image may be a photographed image taken by the camera 2, a difference image, or a composite image. The difference image is, for example, an image obtained by extracting the difference between two photographed images taken over time. The composite image is, for example, an image in which various background images (virtual background images) prepared in advance are embedded in the background area of ​​the photographed work target area. The item image may be an image of the above-mentioned various pipes and various work tools. As the item image, a plurality of images of one pipe and one work tool with their sizes changed may be prepared. In this way, by preparing processed images other than the photographed images, it becomes possible to prepare various training data from a small number of photographed images. In addition, the area image and the item image may include an image suitable for estimating (inferring) the detection class.

[0051] The index estimation model 351 constructed in this way causes the computer to function such that, when a captured image is input to the input layer, at least one detection class estimated to have appeared in the work at the time the captured image is acquired is output from the output layer through calculations in the intermediate layer. That is, the index output unit 332 outputs at least one detection class estimated to have appeared in the work at the time the captured image is acquired by inputting the captured image to the index estimation model 351. Note that the time of acquisition of the captured image is the time of acquisition of the captured image by the image acquisition unit 331, but may be read as the time of image capture of the work target area by the camera 2.

[0052] FIG. 3 is a diagram showing an example of the output result of the index output unit 332. Reference numeral 301 in FIG. 3 indicates the output result from the start of work to time T1 on a certain day, and reference numeral 302 indicates the output result from the start of work to the end of work on a certain day. In reference numerals 301 and 302, the vertical axis indicates the number of each detection class, and the horizontal axis indicates the work time. For example, a number is assigned to each detection class described above, such as "0" for the worker, "1" for the straight pipe, "2" for the irregular pipe, "3" for the backhoe (e.g., the main body including the cabin and / or crawler), and "4" for the bucket. In the example of FIG. 3, "0" to "7" are shown as the numbers of the detection classes, but the detection classes are numbered as many as the number of detection classes set in advance.

[0053] For example, consider a case where a worker, an irregular pipe, a backhoe, and a bucket are included in a photographed image acquired at time T1. In this case, as shown by reference numeral 301 in FIG. 3, the index output unit 332 outputs a combination of "0" and "2" to "4" corresponding to these detection classes at time T1. Since the index output unit 332 inputs the photographed images to the index estimation model 351 over time from the start of work, as shown by reference numeral 301, at time T1, the combination of detection classes at each time point from the start of work (8:00) to time T1 can be output. As a result, as shown by reference numeral 302, at the end of work for one day, the index output unit 332 can output a combination of detection classes at each time point from the start of work to the end of work for one day.

[0054] The index estimation model 351 outputs a numerical value (e.g., a numerical value greater than or equal to 0 and less than or equal to 1) for each detection class as a result of the above calculation. The larger the numerical value of a detection class, the higher the probability that an object represented by that detection class is included in the captured image. Therefore, the index estimation model 351 estimates a detection class that indicates a numerical value equal to or greater than a predetermined threshold as a detection class included in the captured image. The index estimation model 351 may output, as an output result, the numerical value, for example, together with the detection class as the accuracy (reliability) of the estimation of the detection class. The predetermined threshold may be set to a degree that allows the work to be appropriately estimated from the combination of detection classes. The predetermined threshold is set to, for example, 0.2.

[0055] The position identification unit 333 identifies the position of the detection class output by the index output unit 332 in the captured image. The position identification unit 333 can identify the position of the detection class in the captured image by inputting the captured image to a position estimation model constructed to estimate the position of the detection class in the captured image using training data in which article images are associated with detection classes, for example. The position estimation model may be stored in the storage unit 35, and may be a trained model realized by, for example, R-CNN (Regions with CNN). The image extraction unit 334 extracts images of objects involved in the plumbing work from each of the captured images acquired over time by the image acquisition unit 331. The image extraction unit 334 extracts images of all objects included in the captured images from the captured images.

[0056] The object is an object that may move during plumbing work. Examples of the object include various pipes used in plumbing work, various work tools used in plumbing work, workers, and ordinary people other than workers. That is, the object is included in a detection class. Therefore, in this embodiment, when the detection class output by the index output unit 332 includes a detection class in which movement may occur during plumbing work, that is, when an image of an object corresponding to the detection class is included in the photographed image, the image extraction unit 334 extracts an image of the object from the photographed image. For example, when a bucket is captured in the photographed image, the index output unit 332 outputs a number corresponding to the bucket as a detection class. In addition, the position identification unit 333 identifies the position of the bucket in the photographed image. Therefore, the image extraction unit 334 can extract an image of the bucket from the photographed image.

[0057] The frame setting unit 335 sets a frame around the image of the object extracted by the image extraction unit 334 according to the size of the image of the object. The frame setting unit 335 may set a frame around a portion of the image of the object. For example, when an image of a bucket is extracted as the object, the frame setting unit 335 may set a frame around the entire bucket, or may set a frame around a portion of the bucket (e.g., the opening of the bucket).

[0058] A known technique may be used as the method for setting the frame by the frame setting unit 335. Furthermore, the frame setting unit 335 may set a frame of a size corresponding to the accuracy of the estimation of the detection class for the image of the object. For example, the frame setting unit 335 may set a smaller frame size as the accuracy is higher. In other words, the frame setting unit 335 may set a frame size so that the higher the accuracy is, the more the frame fits the object. A known technique may also be used as the method for setting the frame according to the accuracy.

[0059] Reference numeral 401 in FIG. 4 shows a part of a captured image after the frame FR has been set. This captured image is an example of an image captured during mechanical digging of a ditch (e.g., forming a ditch DI in a road using a backhoe). In reference numeral 401, for a bucket BU as an object, a detection class of "4" and a probability of estimating the bucket BU of "0.64" are displayed as output results from the index output unit 332. Also, a rectangular frame FR that has been set by the frame setting unit 335 to surround the periphery of the bucket BU are displayed. Note that the shape of the frame FR is not limited to a rectangular shape and may be another shape (e.g., an elliptical shape).

[0060] The data generation unit 336 generates time series data of coordinates by extracting coordinates within an area containing the image of an object that is estimated to be the same in each of the captured images acquired by the image acquisition unit 331 over time.

[0061] In this embodiment, the index output unit 332 estimates the detection class for each captured image. The data generation unit 336 extracts the above coordinates for each detection class corresponding to the target object among the detection classes estimated to be the same in each captured image. Then, the data generation unit 336 generates time series data of the coordinates for each detection class. For example, when an image of a backhoe (main body) and an image of a bucket are extracted from each captured image, the data generation unit 336 extracts the coordinates in the area including the image of the backhoe and the coordinates in the area including the image of the bucket. Then, the data generation unit 336 generates the time series data of the coordinates corresponding to the backhoe and the time series data of the coordinates corresponding to the bucket.

[0062] The data generating unit 336 extracts, as the above-mentioned coordinates, the coordinates of a position set in advance in the frame set by the frame setting unit 335. In this embodiment, the data generating unit 336 extracts the center coordinates of the frame set by the frame setting unit 335 as the coordinates of the preset position.

[0063] Reference numeral 402 in FIG. 4 is a diagram for explaining calculation of center coordinates CO0. As shown by reference numeral 402, when setting a frame FR, the frame setting unit 335 specifies coordinates CO1 and CO2 of two opposing corners among the four corners of the frame FR. The data generation unit 336 calculates center coordinates CO0 from the coordinates CO1 and CO2 of the two corners. Note that the data generation unit 336 may extract any coordinates (e.g., coordinates CO1 or CO2) within the frame FR as the coordinates of the previously set position, without being limited to the center coordinates CO0.

[0064] Reference numeral 501 in FIG. 5 is a diagram showing an example of time series data of coordinates extracted from an image of a certain object generated by the data generating unit 336. In reference numeral 501, the vertical axis indicates the coordinate value in the captured image, and the horizontal axis indicates the work time. The time series data of coordinates shown in reference numeral 501 is time series data of coordinates extracted from the start of work to the end of work on a certain day. Reference numeral 501 shows time series data of coordinates extracted for 30,000 seconds from the start of work (for example, between 8:00 and 16:20). Reference numeral 501 also shows a graph in which the center coordinates CO0 of the frame FR are plotted for an image of a bucket included in the captured image. In reference numeral 501, the X coordinate of the center coordinates CO0 is shown with an "x" mark, and the Y coordinate of the center coordinates CO0 is shown with an "o" mark. In addition, when work is performed from the start of work to time T1, time series data of coordinates extracted from the image of the object from the start of work to time T1 is generated for each object, similar to the output result of the index output unit 332 shown by reference numeral 301 in Figure 3.

[0065] The data generating unit 336 normalizes the extracted coordinates using the size of the image of the object from which the coordinates are extracted. In this embodiment, the data generating unit 336 normalizes the extracted coordinates using the size of the frame from which the coordinates are extracted. Specifically, the data generating unit 336 divides the X coordinate of the extracted center coordinates CO0 by the side length Lx of the frame FR shown in FIG. 4, and divides the Y coordinate of the center coordinates CO0 by the side length Ly of the frame FR shown in FIG. 4. Lx indicates the length of the side extending in the X-axis direction of the captured image in the frame FR, or the length of the side whose extending direction is closer to the X-axis direction than the Y-axis direction in the frame FR. Ly indicates the length of the side extending in the Y-axis direction of the captured image in the frame FR, or the length of the side whose extending direction is closer to the Y-axis direction than the X-axis direction in the frame FR. Reference numeral 502 in FIG. 5 is a diagram showing an example of the result of the data generating unit 336 normalizing the time series data of the coordinates indicated by reference numeral 501. In reference numeral 502, the vertical axis indicates the coordinate value in the captured image, and the horizontal axis indicates the operation time.

[0066] The data generating unit 336 removes outliers from the time series data of the extracted coordinates. For example, for each coordinate, the data generating unit 336 may detect it as an outlier when there are no other coordinates within a range of a circle with the coordinate as the center and a preset distance as the radius. In addition, the data generating unit 336 may detect outliers from the time series data of the coordinates using a known outlier detection method such as a statistics-based outlier detection method. The data generating unit 336 executes outlier detection processing for each of the X coordinate and the Y coordinate.

[0067] Reference numeral 502 in Fig. 5 indicates the result of execution of outlier detection processing on the X coordinate. In reference numeral 502, the outlier is indicated by reference numeral 503. The data generating unit 336 removes the outlier detected in this manner from the time series data of the coordinate. The outlier indicated by reference numeral 502 is merely an example, and indicates a portion of the outliers detected in the outlier detection processing.

[0068] It is not necessary for the data generating unit 336 to perform the normalization process of the coordinates and the removal process of the outliers. However, by performing the normalization process of the coordinates by the data generating unit 336, the coordinates can be extracted in a state where the possibility of being affected by the perspective of the captured image is reduced. In this case, the movement of the object can be more accurately reflected in the linear data extracted by the linear data extracting unit 337. For example, the smaller the movement of the object, the smaller the inclination described later can be made, and the larger the movement of the object, the larger the inclination described later can be made. In addition, by performing the outlier removal process by the data generating unit 336, the linear data extracting unit 337 can extract linear data in a state where erroneously detected coordinates or coordinates with low reliability are removed from the time series data of the coordinates.

[0069] The linear data extraction unit 337 extracts linear data, which is a part whose value changes continuously and regularly, from the time series data of the coordinates generated by the data generation unit 336. In this embodiment, the linear data extraction unit 337 extracts linear data from the time series data of the coordinates from which the data generation unit 336 normalizes and removes outliers in each of the X coordinate and Y coordinate. The data generation unit 336 generates the time series data of the coordinates from the start of the work to the time when the most recent captured image is acquired. For example, when the most recent captured image is acquired at time T1, the data generation unit 336 generates the time series data of the coordinates extracted from all the captured images acquired from the start of the work to time T1. Therefore, in this case, the linear data extraction unit 337 extracts linear data from the time series data of the coordinates extracted from all the captured images acquired from the start of the work to time T1. The linear data extraction unit 337 may extract the linear data using a known linear extraction method such as the least squares method used in simple regression analysis.

[0070] Here, in the time series data of coordinates, a portion where adjacent coordinates change over time is referred to as a continuous change in value. However, even if adjacent coordinates do not change, if a plurality of coordinates extracted within a certain time (several captured images) include coordinates having different values, the coordinates extracted within the certain time may be referred to as a continuous change in value. Note that, when the linear data extraction unit 337 determines that there is a difference of a specified value or more between two adjacent coordinates, it may specify that the change in value between the two coordinates is a discontinuous change in value. The specified value may be set so that significant linear data can be extracted as linear data. Significant linear data is data that reflects the movement of an actual object, and the usefulness of linear data may be verified by an experiment or the like. In addition, a portion where the value changes continuously and is correlated with a straight line or a curved line is referred to as a regular change in value. That is, the linear data may be a straight line or an N-th order curve (N≧2).

[0071] Specifically, the linear data can be said to be a straight line having a slope other than 0, or a curve including a tangent having a slope other than 0. When such a straight line or curve exists in the coordinate time series data, it can be evaluated that the object is moving. In other words, it can be evaluated that the object is moving during the time period when the linear data can be extracted from the coordinate time series data, and that the object is not moving during the time period other than the time period. The task estimation unit 339 described later can estimate the currently performed task by incorporating this linear data as one of the elements of task estimation, taking into account the motion of the object.

[0072] Reference numeral 502 in FIG. 5 indicates an example of the result of linear data extraction from the time series data of the X coordinate. The linear data indicated by reference numeral 502 is merely an example, and indicates a part of the linear data extracted in the linear data extraction process. In reference numeral 502, the linear data is indicated by reference numeral 504. In this embodiment, the linear data extraction unit 337 extracts straight line data as linear data, but as described above, an N-th order curve (N≧2) may be extracted. Note that a straight line or curve to be fitted may be selected for each task, taking into consideration the correlation between the content of the task and the movement of the object involved in the task. The straight line or curve selected for each task may be set in advance through experiments or the like.

[0073] As described above, it can be evaluated that the object is moving during the time period when the linear data is extracted from the coordinate time series data. In the example of reference numeral 502, the control unit 33 can determine that the bucket is moving during the time period when the linear data is extracted.

[0074] The weighting processing unit 338 weights the time period in which the linear data extraction unit 337 extracted linear data more than other time periods in the detection class output by the index output unit 332. This weighting indicates that a movement has occurred in the object that is the source of the time series data of the coordinates in which the linear data was extracted. For example, if the index output unit 332 adds points to the detection class when it outputs the detection class, the weighting processing unit 338 may further add points to the detection class in the time period in which the linear data was extracted. That is, the control unit 33 may give points (e.g., 1 point) when it outputs the detection class, and may further give points (e.g., 1 point) to the time period in which the linear data was extracted. Furthermore, the weighting processing unit 338 may add points to the accuracy of the detection class for the time period in which the linear data was extracted.

[0075] FIG. 6 is a diagram showing an example of the output result of the weighting processing unit 338. Specifically, FIG. 6 is a diagram showing the output result of the index output unit 332 shown in FIG. 3 reflected with the weighting result by the weighting processing unit 338. Reference numeral 601 in FIG. 6 shows the output result shown in reference numeral 301 in FIG. 3 reflected with the weighting result, and reference numeral 602 shows the output result shown in reference numeral 302 in FIG. 3 reflected with the weighting result. In the diagrams shown in reference numerals 601 and 602, the detection class output by the index output unit 332 is marked with a "●" mark, but the time period weighted by the weighting processing unit 338 is marked with a "●" mark larger than the "●" mark marked with the other time periods. The size of this "●" mark may reflect the above score. The weighting processing unit 338 may output, as a result of the weighting, a mark indicating that the time period in which linear data was extracted is in a different state from the time periods other than the time period.

[0076] The operation estimation unit 339 estimates the operation being performed at the time of acquiring the captured image, based on the linear data extracted by the linear data extraction unit 337 and the object included in the captured image. In this embodiment, the operation estimation unit 339 estimates the operation being performed at the time of acquiring the captured image, based on the linear data extracted by the linear data extraction unit 337 and the output result of the index output unit 332.

[0077] The operation estimation unit 339 estimates the operation being performed at the time of acquiring the captured image based on the output result of the index output unit 332. Specifically, the operation estimation unit 339 estimates the operation being performed at the time of acquiring the most recently acquired captured image by the image acquisition unit 331 by analyzing the combination of detection classes that appear at each of the times of acquiring the captured image output over time by the index output unit 332. That is, the operation estimation unit 339 estimates the operation currently being performed from the combination of detection classes estimated based on the most recently acquired captured image by the image acquisition unit 331 and the combination of detection classes estimated based on the captured image at least one point in time immediately prior thereto (the past captured image).

[0078] Furthermore, when linear data is extracted from the time series data of the coordinates of an object corresponding to the estimated detection class, the operation estimation unit 339 determines that the object exists and that the object is moving during the time period during which the linear data is extracted. That is, the operation estimation unit 339 determines that the object is moving during the weighted time period. The operation estimation unit 339 estimates the currently performed operation, including the presence or absence of the object's movement, together with the output result of the indicator output unit 332 at each time point.

[0079] The memory unit 35 may store data associating the content of each task that may be performed in plumbing work with a combination of detection classes and the presence or absence of movement of an object corresponding to the detection class. In this case, the task estimation unit 339 can refer to the memory unit 35 to estimate the task currently being performed from the output result of the index output unit 332 and the linear data extraction result by the linear data extraction unit 337 (the output result of the weighting processing unit 338).

[0080] For example, the task of mechanically digging a trench is associated with a combination of a trench, a backhoe (main body), a bucket, and a dump truck as a detection class. In addition, the above task is associated with continuous movement of the bucket. The task estimation unit 339 determines whether the above task is being performed by determining the following two items in the continuously acquired photographed images based on the output result of the index output unit 332. Presence or absence of detection classes showing trenches, backhoes (main body), buckets, and dump trucks. · Presence or absence of linear data showing bucket movement.

[0081] Also, each task may be associated with a specific movement of an object specific to the task. For example, the slope at each time point of linear data extracted from each of the time series data of the X coordinate and the Y coordinate may be associated with a specific movement of an object specific to the task. For example, the task of mechanically excavating a trench may be associated with a bucket movement such as a linear movement along the trench to be excavated and a movement between the trench and the dump truck. In this case, the task estimation unit 339 can determine whether the task is being performed by taking into account the movement of the bucket when the trench is being formed or the movement when the soil generated during the trench excavation is being loaded onto the dump truck.

[0082] In this embodiment, the work estimation unit 339 estimates the work being performed at the time of acquiring the photographed image by inputting at least one of the multiple detection classes and the linear data to the performed work estimation model 352. Specifically, the work estimation unit 339 estimates the work currently being performed by inputting the detection class output by the index output unit 332 and the linear data extracted by the linear data extraction unit 337 to the performed work estimation model 352. In this case, the work estimation unit 339 can estimate the work currently being performed without referring to data that associates each work that may be performed in plumbing work, a combination of detection classes, and the presence or absence of movement of an object corresponding to the detection class. In other words, in this case, it is not necessary to store the data in the memory unit 35.

[0083] The performed task estimation model 352 is a trained model constructed using machine learning to estimate the task being performed at the time of capturing the captured image. The indicator estimation model 351 may be, for example, a neural network (e.g., a Convolutional Neural Network (CNN)) including at least an input layer, an intermediate layer, and an output layer.

[0084] In the performed work estimation model 352, the input layer is a layer to which at least one of the detection classes and linear data are input. The intermediate layer is a layer in which parameters are learned based on a plurality of detection classes, a plurality of linear data, and a plurality of work contents. The output layer is a layer that outputs the work in the plumbing work.

[0085] For example, data in which the contents of work are associated as correct answer data with each of a plurality of detection classes or a combination of detection classes is prepared as training data. Training data is prepared by associating the contents of work at the time of acquiring the captured image with the detection classes output by the index output unit 332 based on the captured images of various work target areas. Also, for example, data in which the contents of work are associated as correct answer data with each of a plurality of linear data is prepared as training data. Training data is prepared by associating the contents of work at the time of acquiring the captured image with the linear data extracted by the linear data extraction unit 337 based on the captured images of various work target areas. The performed work estimation model 352 is constructed by sequentially inputting training data and learning parameters (weights and biases) to minimize the loss function.

[0086] The performed task estimation model 352 constructed in this way causes the computer to function such that, when the detection class and linear data are input to the input layer, the content of the task at the time of capturing the captured image is output from the output layer via calculations in the intermediate layer. The detection class input to the input layer is the result estimated by the indicator output unit 332 when the captured image is captured, and the linear data input to the input layer is the result extracted by the linear data extraction unit 337 when the captured image is captured. Therefore, the task estimation unit 339 can output the content of the task currently being performed by inputting the detection class and the linear data to the performed task estimation model 352.

[0087] The work estimation unit 339 may estimate the work currently being performed using the output result (weighting result) of the weighting processing unit 338. For example, the performed work estimation model 352 may be constructed using the output result of the weighting processing unit 338. In this case, the work estimation unit 339 may estimate the work currently being performed by inputting the output result to the performed work estimation model 352. Note that weighting is performed for the time period in which linear data was extracted in the estimated detection class. Therefore, the concept of using linear data in constructing the performed work estimation model 352 and in estimating work using the performed work estimation model 352 may be understood to include the use of weighting results.

[0088] As described above, the activity estimation unit 339 performs activity estimation using linear data. Therefore, the activity estimation unit 339 can estimate the activity currently being performed (the activity being performed) by taking into account the movement of the object. Therefore, the activity estimation unit 339 can accurately estimate the activity being performed.

[0089] Furthermore, since the work estimation unit 339 can identify the positional relationship of each object (detection class), it can also estimate how an object is moving relative to other objects by analyzing the magnitude of the inclination of the linear data. The work estimation unit 339 can also estimate, for example, whether a backhoe is moving along a ditch. Therefore, the work estimation unit 339 can estimate the work being performed with higher accuracy. When the performed work estimation model 352 is used, it is also estimated how an object is moving relative to other objects by inputting the linear data and the detection class.

[0090] Furthermore, since the work management device 3 can accurately estimate the work being carried out, it is possible to appropriately manage waterworks works. Therefore, the work management device 3 and the waterworks works management system 1 equipped with the work management device 3 can function as a catalyst for promoting inclusive and sustainable urbanization. Therefore, the work management device 3 and the waterworks works management system 1 can contribute to the achievement of the Sustainable Development Goals (SDGs).

[0091] Furthermore, when the position specifying unit 333 specifies the position of the detection class in the photographed image, the work estimation unit 339 may estimate the work being performed in different areas in the photographed image based on the position of the detection class. For example, the work estimation unit 339 may determine whether the photographed image includes a scene of multiple plumbing works being performed based on the type of the estimated detection class and the position of the specified detection class. When the work estimation unit 339 determines that plumbing works are being performed in different areas in the photographed image, it estimates the work being performed in each area.

[0092] The display control unit 340 controls the display device 4 to display various information on the display device 4. In other words, the display control unit 340 functions as an output unit that outputs various information from the display device 4. For example, the display control unit 340 may output the estimation result (i.e., the content of the work) of the work estimation unit 339 from the display device 4. The display control unit 340 may also output the time series data of the coordinates generated by the data generation unit 336 from the display device 4. In this case, the display control unit 340 may output, for example, the time series data of the normalized coordinates from the display device 4. The display control unit 340 may also output, for example, the time series data of the coordinates including the linear data extracted by the linear data extraction unit 337 from the display device 4. In this case, the display control unit 340 may output, for example, a graph shown by reference numeral 502 in FIG. 5. The display control unit 340 may output, for example, a graph from which the time series data of the coordinates has been removed (a graph showing only the linear data) from the display device 4. Additionally, the display control unit 340 may output the result of weighting by the weighting processing unit 338 from the display device 4. The display control unit 340 may output, for example, the graph shown by reference numeral 601 or 602 in FIG.

[0093] Furthermore, when outputting the work contents from the display device 4, the display control unit 340 may change the display mode between when weighting is performed (when linear data is extracted) and when weighting is not performed (when linear data is not extracted). For example, the display control unit 340 may change the color, shape, size, or the like of the character string indicating the work contents. Due to such a difference in the display mode, the manager and the worker (user) can recognize that a movement is occurring in any of the objects included in the captured image.

[0094] In this embodiment, the index output unit 332 estimates the detection class using the index estimation model 351, but the detection class may be estimated without using the index estimation model 351. In this case, for example, data in which images of objects involved in waterworks are associated with the names of the objects are stored in advance in the storage unit 35. The index output unit 332 extracts images of the objects from the captured image using a known image technique. Then, the index output unit 332 refers to the data to identify the name of the extracted object, i.e., the detection class.

[0095] Furthermore, in this embodiment, the work estimation unit 339 estimates the work using the performed work estimation model 352, but the work may be estimated without using the performed work estimation model 352. In this case, for example, as described above, the work estimation unit 339 stores data in advance in the storage unit 35 that associates the content of each work that may be performed in plumbing work with a combination of detection classes and the presence or absence of movement of an object corresponding to the detection class. Thereby, when the indicator output unit 332 estimates a detection class and the linear data extraction unit 337 extracts linear data, the work estimation unit 339 can estimate the work being performed by referring to the data.

[0096] <Coordinate correction associated with camera movement> As described above, the camera 2 may be mounted on a movable object (backhoe BH shown in FIG. 2). In this case, the positional relationship between the camera 2 and the object changes due to the movement of the camera 2. Therefore, the coordinate values ​​extracted by the data generation unit 336 change due to the movement of the camera 2. When the camera 2 is stationary (fixed camera), the origin position set in each captured image is a fixed position with respect to the work area to be photographed. On the other hand, when the camera 2 is movable, the origin position set in each captured image is not a fixed position with respect to the work area to be photographed. Therefore, if the coordinates are extracted without considering the movement of the camera 2, there is a risk that the presence or absence of actual movement of the object cannot be evaluated from the coordinates. Therefore, when the camera 2 is movable, the data generation unit 336 may correct the coordinates extracted from the captured image by considering the movement of the camera 2.

[0097] For example, the data generating unit 336 may correct the coordinates acquired from the captured image based on a preset reference position of the camera 2 and the position of the camera 2 when the captured image was acquired. The correction of the coordinates based on the reference position of the camera 2 and the position of the camera 2 when the captured image was acquired will be described with reference to Figs. 7 and 8. Fig. 7 is a diagram for explaining the correction of the coordinates when the camera 2 moving parallel to the object captures the object. Fig. 8 is a diagram for explaining the correction of the coordinates when the camera 2 approaching the object captures the object. In Figs. 7 and 8, the symbol AOV indicates the angle of view of the camera 2. The same applies to Fig. 9.

[0098] As a premise for the explanation, the data generating unit 336 acquires data indicating the position of the camera 2 (absolute coordinates of the camera 2; GPS position information) and data indicating the orientation from the camera 2. In addition, the standard size of the frame set for the object when the object is away from the camera 2 by a unit distance (e.g., the length of each of the two adjacent sides, or the area of ​​the frame) is set in advance for each object. In addition, the conversion value between pixels and millimeters (mm) is also specified in advance.

[0099] (Coordinate correction when the camera movement direction is the X-axis direction) Reference numeral 701 in Fig. 7 is a diagram showing the camera 2 moving from position PO1 to position PO2. Reference numeral 702 is a diagram showing an example of a captured image when photographing at position PO1. Reference numeral 703 is a diagram showing an example of a captured image when photographing at position PO2. In this example, position PO1 corresponds to the reference position of camera 2, and position PO2 corresponds to the position of camera 2 when the photographed image was acquired.

[0100] First, the data generating unit 336 specifies the absolute coordinates of the camera 2 at the position PO1. The data generating unit 336 also specifies the distance D11 from the camera 2 to the object OB at the position PO1. As indicated by the reference numeral 702, the image acquiring unit 331 acquires a captured image at the position PO1. For the captured image, the frame setting unit 335 sets a frame FR1 for the image of the object OB, and specifies the coordinates CO01, CO11, and CO21. Also, the lengths Lx and Ly of the sides of the frame FR1 are specified. As described above, since the standard size of the frame set for the object OB is set in advance, the data generating unit 336 calculates the distance D11 (mm) from the ratio of the specified lengths Lx and Ly to the standard size of the frame, and the unit distance between the object OB and the camera 2.

[0101] Next, as shown by reference numeral 701, it is assumed that the camera 2 moves a distance D12 (mm) parallel to the object OB (in the X-axis direction). At this time, as shown by reference numeral 703, the image acquisition unit 331 acquires a photographed image at a position PO2. For the photographed image, the frame setting unit 335 sets a frame FR2 for the image of the object OB, and specifies coordinates CO02, CO12, and CO22. For convenience of explanation, the photographed image shown by reference numeral 703 shows the image of the object OB photographed at a position PO1 where the object OB is not actually included, and the frame FR1 set for the image of the object OB (broken line portion). Furthermore, the data generation unit 336 specifies the absolute coordinates of the camera 2 at the position PO2. The data generation unit 336 calculates the distance D12 (mm) from the absolute coordinates of the camera 2 at the positions PO1 and PO2.

[0102] Moreover, the data generating unit 336 calculates a movement distance D13 (pixels) from the frame FR1 to the frame FR2, which corresponds to the amount of movement of the camera 2, from the distance D11 (mm) and the distance D12 (mm). Here, the total length in the horizontal direction in the captured image at the position PO1 (referred to as the first image plane range) is L1 (mm). The horizontal direction corresponds to the X-axis direction. The first image plane range L1 (mm) can be calculated by a linear function p×D11 (mm). p is a constant, and is a value determined by the optical system (lens or image sensor size, etc.) of the camera 2 used. Also, the moving distance of the object OB in the captured image (referred to as the image plane moving distance) is D112 (mm). The image plane moving distance D112 (mm) can be calculated by a linear function 1×D12 (mm). And the moving distance D13 (pixels) can be calculated by (image plane moving distance D112 (mm) / first image plane range L1 (mm))×(total number of pixels (pixels) in the horizontal direction in the captured image). The data generating unit 336 calculates the first image plane range L1 (mm) and also calculates the image plane movement distance D112 (mm). The data generating unit 336 then calculates the movement distance D13 (pixels) from the first image plane range L1 (mm) and the image plane movement distance D112 (mm). Note that instead of the first image plane range L1 (mm), the total lateral length of the captured image at the position PO2 may be calculated.

[0103] Then, the data generating unit 336 converts each of the coordinates CO02, CO12, and CO22 into coordinates in the captured image when it is assumed that the image was captured at the position PO1 by affine transformation using the movement distance D13 (pixels) in the X-axis direction. Specifically, the data generating unit 336 substitutes the movement distance D13 into Tx in the following formula 1, and substitutes 1 into Ty in the following formula 1. The data generating unit 336 also substitutes the x coordinate and y coordinate of each of the coordinates CO02, CO12, and CO22 into x' and y' in the following formula 1. This allows the data generating unit 336 to acquire coordinates that have been moved by the movement distance D13 (pixels) corresponding to the amount of movement of the camera 2 in the X-axis direction. In other words, the data generating unit 336 can acquire coordinates that have been offset by the amount of movement of the camera 2 in the X-axis direction.

number

[0104] (Coordinate correction when the camera movement direction is the Y axis direction) Reference numeral 801 in Fig. 8 is a diagram showing the camera 2 moving from position PO1 to position PO3. Reference numeral 802 is a diagram showing an example of a captured image when photographing at position PO1. Reference numeral 803 is a diagram showing an example of a captured image when photographing at position PO3. In this example, position PO1 corresponds to the reference position of camera 2, and position PO3 corresponds to the position of camera 2 when the photographed image was acquired.

[0105] First, the data generation unit 336 specifies the absolute coordinates of the camera 2 at the position PO1. The data generation unit 336 also specifies the distance D11 (mm) from the camera 2 to the object OB at the position PO1.

[0106] Next, as shown by reference numeral 801, it is assumed that the camera 2 moves a distance D22 (mm) (in the Y-axis direction) so as to approach the object OB. At this time, as shown by reference numeral 803, the image acquisition unit 331 acquires a photographed image at a position PO2. For the photographed image, the frame setting unit 335 sets a frame FR3 for the image of the object OB, and specifies coordinates CO03, CO13, and CO23. For convenience of explanation, the photographed image shown by reference numeral 803 shows the image of the object OB photographed at a position PO1 where the object OB is not actually included, and the frame FR1 set for the image of the object OB (broken line portion). Furthermore, the data generation unit 336 specifies the absolute coordinates of the camera 2 at the position PO3. The data generation unit 336 calculates the distance D22 (mm) from the absolute coordinates of the camera 2 at the positions PO1 and PO3. In addition, the data generation unit 336 may calculate the distance D11-D22 (mm) or the distance D22 (mm) from the ratio of the lengths Lx and Ly specified in the frame FR3 to the standard size of the frame, and the unit distance between the object OB and the camera 2.

[0107] Here, the total horizontal length of the captured image at position PO1 (referred to as the first image plane range) is L1 (mm). The first image plane range L1 (mm) can be calculated using a linear function p × D11 (mm). The total horizontal length of the captured image at position PO3 (referred to as the second image plane range) is L2 (mm). The second image plane range L2 (mm) can be calculated using a linear function p × (D11-D22) (mm).

[0108] The data generating unit 336 calculates the first image plane range L1 (mm) and the second image plane range L2 (mm). Then, the data generating unit 336 converts each of the coordinates CO03, CO13, and CO23 into coordinates in a captured image when it is assumed that the image is captured at the position PO1 by affine transformation using the ratio between the first image plane range L1 (mm) and the second image plane range L2 (mm). Specifically, the data generating unit 336 substitutes the ratio (first image plane range L1 (mm) / second image plane range L2 (mm)) into Sx and Sy in the following equation 2. In addition, the data generating unit 336 substitutes the x coordinate and the y coordinate of each of the coordinates CO03, CO13, and CO23 into x' and y' in the following equation 2. This allows the data generating unit 336 to obtain coordinates that offset the amount of movement of the camera 2 in the Y-axis direction.

number

[0109] (Coordinate correction when camera 2 rotates) The data generating unit 336 may also correct the coordinates acquired from the captured image based on the angle between a preset reference shooting direction of the camera 2 and the shooting direction of the camera 2 when the captured image was acquired. The above-mentioned correction of the coordinates based on the angle will be explained using Fig. 9. Fig. 9 is a diagram for explaining the correction of the coordinates when the camera 2 shoots an image of an object while rotating around the vertical direction as the rotation axis.

[0110] Reference numeral 901 in FIG. 9 is a diagram showing the state in which the camera 2 rotates from position PO1 to position PO4. Reference numeral 902 is a diagram showing an example of a captured image when photographing at position PO1. Reference numeral 903 is a diagram showing an example of a captured image when photographing at position PO4. In this example, the photographing direction at position PO1 corresponds to the reference photographing direction of the camera 2, and the photographing direction at position PO4 corresponds to the photographing direction of the camera 2 when the photographed image was acquired. Reference numeral 904 is a diagram showing an example of the relationship between distance D32 and distance D33.

[0111] First, the data generation unit 336 specifies the absolute coordinates of the camera 2 at the position PO1. The data generation unit 336 also specifies the distance D11 (mm) from the camera 2 to the object OB at the position PO1.

[0112] Next, as shown by reference numeral 901, it is assumed that the camera 2 has rotated by an angle AN (deg) from the state of position PO1. At this time, as shown by reference numeral 903, the image acquisition unit 331 acquires a captured image at position PO4. For the captured image, the frame setting unit 335 sets a frame FR4 for the image of the object OB, and specifies coordinates CO04, CO14, and CO24. For convenience of explanation, in the captured image shown by reference numeral 903, the image of the object OB captured at position PO1 where the object OB is not actually included, and the frame FR1 set for the image of the object OB are shown (broken line portion).

[0113] Here, in the frame FR1, the distance in the horizontal direction (corresponding to the X-axis direction) between the coordinate CO01 and the coordinate CO11 or CO21 is defined as D33. In addition, in the frame FR4, the distance in the horizontal direction (corresponding to the X-axis direction) between the coordinate CO04 and the coordinate CO14 or CO24 is defined as D32. At this time, as shown by the reference numeral 904, the distance D32 can be calculated as the distance D33×CosAN. That is, the reduction ratio from the frame FR1 to the frame FR4 accompanying the rotation of the camera 2 by the angle AN (deg) can be expressed as CosAN. Therefore, the data generation unit 336 can convert the size of the frame FR4 into the size of the frame FR1 by affine transformation using CosAN. Specifically, the data generation unit 336 substitutes CosAN for Sx in the above equation 2 and 1 for Sy in the above equation 2. Furthermore, the data generation unit 336 substitutes the x coordinate and y coordinate of the coordinates CO14 and CO24, respectively, into x' and y' in the above equation 2. In this way, the data generation unit 336 can obtain the coordinates after the size of the frame FR4 is converted into the size of the frame FR1. Furthermore, the distance between the coordinate CO01 (center coordinate) of frame FR1 and the coordinate CO04 (center coordinate) of frame FR4 is defined as D31. The position of frame FR4 can be converted to the position of frame FR1 by moving the position of frame FR4 by the distance D31. The amount of translation from frame FR1 to frame FR4 accompanying the rotation of camera 2 by angle AN (deg) can be calculated by (angle AN (deg) / angle of view AN0) x (total number of pixels (pixels) in the horizontal direction in the captured image). The data generation unit 336 can transform the position of the frame FR4 into the position of the frame FR1 by affine transformation using the above-mentioned translation amount. Specifically, the data generation unit 336 calculates the above-mentioned translation amount. Then, the data generation unit 336 substitutes the above-mentioned translation amount for Tx in the above-mentioned equation 1, and 1 for Ty in the above-mentioned equation 1. Furthermore, the data generation unit 336 substitutes the x coordinate and the y coordinate of the coordinates CO14 and CO24, respectively, into x' and y' in the above-mentioned equation 1. In this way, the data generation unit 336 can obtain the coordinates after the position of the frame FR4 is transformed into the position of the frame FR1.

[0114] In this way, by performing the above two affine transformations, the data generating unit 336 can convert each of the coordinates CO04, CO14, and CO24 into coordinates in the captured image when it is assumed that the image was captured at the position PO1. That is, the data generating unit 336 can obtain coordinates that have been offset by the amount of rotation of the camera 2.

[0115] As described above, the data generating unit 336 converts the coordinates for each of the movement of the camera 2 in the X-axis direction, the movement of the camera 2 in the Y-axis direction, and the rotation of the camera 2. In this way, the data generating unit 336 can convert the coordinates extracted from all the acquired captured images into the coordinate values ​​extracted in the captured images when they are assumed to have been captured at the reference position and in the reference capturing direction.

[0116] [Processing in the work management device] Fig. 10 is a flowchart showing an example of processing (work management method) in the work management device 3. As shown in Fig. 10, the image acquisition unit 331 acquires a captured image from the camera 2 (S1: image acquisition step). The index output unit 332 inputs the captured image acquired by the image acquisition unit 331 to the index estimation model 351. The index output unit 332 outputs the estimated detection class as an output result of the index estimation model 351. In other words, the index output unit 332 estimates the detection class included in the captured image (S2).

[0117] The position identifying unit 333 identifies the position of the detection class output by the index output unit 332 in the captured image (S3). When there are multiple positions of the detection class identified by the position identifying unit 333, the control unit 33 determines whether or not the captured image includes multiple scenes of plumbing work being carried out, based on the positional relationship of the multiple detection classes. In other words, the control unit 33 determines whether or not plumbing work is being carried out in different areas in the captured image. When the control unit 33 determines that plumbing work is being carried out in different areas in the captured image, it executes the following process for each area.

[0118] The image extraction unit 334 extracts images of objects involved in the waterworks from the captured images acquired by the image acquisition unit 331 (S4: image extraction step). In this embodiment, an image of an object corresponding to the detection class output by the index output unit 332 is extracted.

[0119] Frame setting section 335 sets a frame for the image of the object extracted by image extraction section 334 according to the size of the image of the object (S5). Data generation section 336 calculates the center coordinates of the frame as the coordinates of a position set in advance in the frame set by frame setting section 335 (S6). That is, in S5 and S6, data generation section 336 extracts coordinates within an area including the image of the object extracted by image extraction section 334.

[0120] Camera 2 captures images of the work target area over time. Therefore, image acquisition section 331 acquires captured images over time from camera 2. Therefore, control section 33 sequentially executes processes S2 to S6 on the captured images acquired by image acquisition section 331.

[0121] After the process of S6, the control unit 33 determines whether or not the processes of S2 to S6 have been executed for a prescribed number of captured images (S7). If the control unit 33 determines that the number of captured images for which the processes of S2 to S6 have been executed is less than the prescribed number (NO in S7), the process returns to S1. On the other hand, if the control unit 33 determines that the processes of S2 to S6 have been executed for a prescribed number of captured images (YES in S7), the data generation unit 336 generates time-series data of the extracted coordinates (S8: data generation step). In this embodiment, the data generation unit 336 generates time-series data of the center coordinates of the frame.

[0122] If S7 is YES, coordinates are extracted over time for the object whose image has been extracted by the image extraction unit 334. Therefore, the data generation unit 336 can generate time series data of the extracted coordinates for the object. Furthermore, if there are multiple images of the object corresponding to the detection class in the captured image, the image extraction unit 334 extracts an image for each object. Therefore, the data generation unit 336 can generate time series data of the coordinates for each object. In this embodiment, the data generation unit 336 generates time series data of the X coordinate and time series data of the Y coordinate of the extracted center coordinates.

[0123] The above-mentioned specified number only needs to be specified to the extent that the data generating unit 336 can generate time-series data of coordinates and that the linear data extracting unit 337 can extract significant linear data in the subsequent process of S11. The above-mentioned specified number is, for example, the number of captured images captured by the image acquiring unit 331 in five minutes. For example, when the image acquiring unit 331 acquires one captured image per second, the number of captured images is 18,000 (=5 (minutes) x 60 x 60).

[0124] The data generating unit 336 normalizes the coordinates of the generated time series data of the coordinates (S9). The process of S9 may be executed between the processes of S6 to S8. The data generating unit 336 removes outliers from the time series data of the normalized coordinates (S10). The linear data extracting unit 337 extracts linear data from the time series data of the coordinates from which the data generating unit 336 has removed the outliers (S11: linear data extraction step). The weighting processing unit 338 weights the time period from which the linear data extracting unit 337 extracted the linear data more than other time periods (S12). The control unit 33 executes the processes of S9 to S12 for the time series data of the coordinates of each object. The control unit 33 also executes the processes of S9 to S12 for each of the time series data of the X coordinate and the time series data of the Y coordinate.

[0125] Thereafter, the work estimation unit 339 estimates the work being performed at the time of acquiring the photographed image as the work in progress, based on the linear data extracted by the linear data extraction unit 337 and the object included in the photographed image (S13: work estimation step). In this embodiment, the work estimation unit 339 estimates the work in progress by inputting the detection class corresponding to the object output by the index output unit 332 and the linear data extracted by the linear data extraction unit 337 to the performed work estimation model 352. Note that the work estimation unit 339 may estimate the work in progress based on the result of weighting by the weighting processing unit 338 and the object included in the photographed image.

[0126] The display control unit 340 displays the estimation result of the work estimation unit 339 on the display device 4 (S14). The display control unit 340 may also display time-series data (which may include linear data) of the coordinates of each object, weighting results for each object, and the like on the display device 4. When it is determined that plumbing work is being carried out in different areas within the captured image, the display control unit 340 may display the above various data for each area on the display device 4.

[0127] [Software implementation example] The functions of the work management device 3 (hereinafter referred to as the "device") can be realized by a program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (particularly each part included in the control unit 33).

[0128] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0129] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0130] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0131] [Additional Notes] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0132] 3 Work management device 2 Camera 4 Display device (notification unit) 331 Image Acquisition Unit 332 Index output section 334 Image Extraction Unit 336 Data Generation Department 337 Linear Data Extraction Unit 338 Weighting Processing Unit 339 Work Estimation Department 335 Frame Setting Section 340 Display control unit (output unit) 351 Index Estimation Model 352 Work execution estimation model

Claims

1. an image acquisition unit that acquires photographed images including a work area in a plumbing work over time; an image extraction unit that extracts an image of at least one object involved in the waterworks construction from each of the photographed images acquired over time; a data generating unit that generates time-series data of coordinates by extracting coordinates within an area including an image of an object that is estimated to be the same in each of the captured images over time; a linear data extraction unit that extracts linear data, which is a portion whose value changes continuously and regularly, from the time series data of the coordinates; and a work estimation unit that estimates a work being performed at the time the captured image is acquired based on the linear data and an image of an object included in the captured image.

2. a frame setting unit that sets a frame around the image of the object or around a part of the image of the object according to a size of the image of the object; The work management device according to claim 1 , wherein the data generating unit extracts coordinates of a position that is preset in the frame as the coordinates within the area.

3. The work management device according to claim 1 , wherein the data generating unit normalizes the coordinates using a size of an image of the object from which the coordinates are extracted.

4. The data generation unit removes outliers from the time series data of the coordinates; The work management device according to claim 1 , wherein the linear data extraction unit extracts the linear data from the time series data of the coordinates from which the outliers have been removed.

5. an index output unit that outputs at least one index that is estimated to be appearing in an operation at a certain time point, by inputting the photographed image into an index estimation model that is constructed to estimate at least one index that is appearing in an operation at a certain time point, among a plurality of indices that indicate each of the plurality of objects or a combination thereof, using machine learning; The work management device according to claim 1 , wherein the work estimation unit estimates the work being performed at the time the photographed image is acquired based on the linear data and the output result of the index output unit.

6. The work management device of claim 5, further comprising a weighting processing unit that, when the linear data extraction unit extracts the linear data, weights the time period in which the linear data was extracted more heavily than time periods other than the time period in which the linear data was extracted, in the indicator output as the object from which the time series data of the coordinates from which the linear data was extracted was generated, assuming that movement is occurring in the object during the time period in which the linear data was extracted.

7. A camera is provided to capture an image of the work area, The work management device according to claim 1 , wherein the data generation unit corrects the coordinates based on a preset reference position of the camera and a position of the camera when the captured image was acquired.

8. 8. The work management device according to claim 1, wherein the work estimation unit estimates the work being performed at the time the photographed image is acquired by inputting (i) at least one of a plurality of indicators indicating each of the plurality of objects or a combination thereof, and (ii) the linear data into a performed work estimation model constructed using machine learning to estimate the work being performed at the time the photographed image is acquired.

9. A communication unit can be connected to the notification unit, The work management device according to claim 1 , further comprising an output unit configured to output at least one of an estimation result of the work estimation unit and time series data of the coordinates from the notification unit.

10. an image acquisition step of acquiring photographed images including a work area in a waterworks construction project over time; An image extraction step of extracting an image of at least one object involved in the waterworks construction from each of the photographed images acquired over time; a data generation step of extracting coordinates within an area including an image of an object that is estimated to be the same in each of the captured images over time to generate time-series data of the coordinates; a linear data extraction step of extracting linear data, which is a portion whose value changes continuously and regularly, from the time series data of the coordinates; and a work estimation step of estimating the work being performed at the time the captured image was acquired based on the linear data and the object from which the time series data of the coordinates from which the linear data was extracted was generated.

11. (i) an input layer to which at least one of a plurality of indices indicating each of a plurality of objects involved in waterworks or a combination thereof, and (ii) linear data, which is a portion whose value changes continuously and regularly in time series data of coordinates included in an image of an object that is one of the indices included in a photographed image, extracted as a result of acquiring photographed images including a work area in the waterworks over time, are input; an output layer that outputs work in the waterworks; a hidden layer having parameters learned based on the plurality of indicators, the plurality of linear data, and the plurality of tasks; a performed task estimation model that causes a computer to function such that, when the indicators and the linear data are input to the input layer, the task being performed at the time the captured image was acquired is output from the output layer through calculations by the intermediate layer.

Images