Image-based productivity tracking system

Abstract

A work machine is disclosed that includes a sensing device, a user interface, and a control unit. The control unit may be configured to generate a productivity layer based on productivity data and generate an image layer based on image data. The image data may include images corresponding to construction conditions associated with the work site and information related to geospatial references associated with the images. The control unit may be configured to generate a composite image of the work site based on the map layer, the image layer, and the productivity layer and cause the composite image to be displayed via the user interface. The composite image may position the image layer relative to the map layer and the productivity layer relative to the image layer based on the geospatial references and geographic coordinates corresponding to the geospatial references.

Description

Technical Field

[0001] The present disclosure generally relates to surveying a work site, and for example, to an image-based productivity tracking system for a work site.

Background Art

[0002] A work site may include one or more work machines that are each individually operated to perform work and are collectively managed to complete a construction project according to a site plan. The construction project may be related to construction, paving, demolition, mining, landscaping, forestry, pipelines, and / or other industrial applications. The site plan may include specifications for the construction project to be performed at the work site, which are designed based on an initial or previous survey of the work site. In some cases, a tracking tool may be used to track the progress of the construction project against the site plan. The tracking tool may be updated based on changes in survey data (e.g., related to structures and / or geographical changes within the work site), productivity data (e.g., related to the location and / or operating status of work machines within the work site), and / or other information that can be used to track progress. In some cases, the tracking tool may provide a digital map and / or model of the work site that can be used to guide operators of the work machines. For example, the tracking tool may be made accessible to remote and / or local operators via a mobile device, a control station, a display within a work machine, and / or another user interface.

[0003] In some cases, the tracking tool may provide a map view with an image of the job site that shows the location of each of the machine tools, spare parts, and / or other assets or attributes associated with the job site. The location of each asset and / or attribute within the job site can be updated based on productivity data obtained and / or derived from information provided by the machine tools. However, the image used to depict the job site may be out of date (e.g., based on out-of-date satellite imagery), which can be inaccurate and / or misleading for an operator relying on the tracking tool. In some cases, the operator may use a drone with an imaging device to survey the job site. For example, the drone can be used to capture an updated image of the job site that the operator can use to evaluate visual changes to the job site. However, the information provided via the drone cannot be used in conjunction with the information provided by the tracking tool (e.g., productivity data from the machine tools). Thus, the operator may need to separately analyze the information provided via the tracking tool and the information provided via the drone to evaluate the state of the job site. Analyzing a single construction project using separate systems can result in errors, delays, and inefficient use of machine tools, computing resources, and / or network resources.

[0004] One attempt to generate aerial images encoded with metadata is disclosed in U.S. Patent No. 9,684,673 to Beckett et al. (the “’673 Patent”), filed December 4, 2013. The ’673 Patent discloses a system and method for processing observational data, as well as images and metadata obtained from an Earth observation platform including satellites or aircraft. The ’673 Patent discloses the acquisition of images and metadata associated with the images, the encoding of the images and metadata for generating encoded tiles, the storage of the encoded tiles in a database of encoded tiles, the receipt of requests for map tiles, the acquisition of the encoded tiles associated with the requests, the merging of data from the encoded tiles into the map tiles, and the output of the map tiles. However, the ’673 Patent does not disclose the acquisition of productivity data associated with one or more work machines at a worksite, the generation of a composite image incorporating such productivity data into an image layer and a map layer, or the determination of the progress of construction at the worksite based on such a composite image.

[0005] The image-based productivity tracking system of the present disclosure solves one or more of the above-described problems and / or other problems in the art. SUMMARY OF THE INVENTION

[0006] According to some embodiments, the method comprises: receiving, by a device, map data associated with a work site, the map data including information related to geographical attributes associated with the work site and geographical coordinates associated with the geographical attributes; receiving, by the device, productivity data associated with the work site, the productivity data including information related to the state of construction associated with the work site; receiving, by the device, image data associated with the work site, the image data including an image corresponding to the state of construction and information related to a geospatial reference associated with the image; generating, by the device, a map layer based on the map data; generating, by the device, an image layer based on the image data; generating, by the device, a productivity layer based on the productivity data; generating, by the device, a composite image of the work site, the image layer, and the productivity layer based on the map layer, wherein the composite image positions the image layer relative to the map layer and positions the productivity layer relative to the image layer based on a geospatial reference and geographical coordinates corresponding to the geospatial reference; and causing, by the device, an action to be executed based on the composite image.

[0007] According to some embodiments, the apparatus includes one or more memories communicatively coupled to one or more processors configured to receive productivity data associated with a work site, the productivity data including information related to the status of construction associated with the work site, receive image data associated with the work site, the image data including an image corresponding to the status of construction and information related to a geospatial reference associated with the image, generate an image layer based on the image data and a productivity layer based on the productivity data, and generate a composite image of the work site based on a map layer, the image layer, and the productivity layer, the map layer including information related to geographical attributes associated with the work site and geographical coordinates associated with the geographical attributes, the composite image being generated by positioning the image layer relative to the map layer and the productivity layer relative to the image layer based on the geospatial reference and geographical coordinates corresponding to the geospatial reference, and causing an action to be executed based on the composite image.

[0008] According to some embodiments, the work machine includes a sensing device configured to provide productivity data of the work machine, the productivity data including information related to the status of construction associated with the work site, a user interface configured to display information about the status of construction to an operator of the work machine, and a control unit communicatively coupled to the sensing device and the user interface, the control unit being configured to generate a productivity layer based on the productivity data, generate an image layer based on image data including an image corresponding to the status of construction and information related to a geospatial reference associated with the image, generate a composite image of the work site based on a map layer, the image layer, and the productivity layer, the map layer including information related to geographical attributes associated with the work site and geographical coordinates associated with the geographical attributes, the composite image being generated by positioning the image layer relative to the map layer and the productivity layer relative to the image layer based on the geospatial reference and geographical coordinates corresponding to the geospatial reference, and causing the composite image to be displayed via the user interface.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2-1

Figure 2-2

Figure 3

Best Mode for Carrying Out the Invention

[0010] FIG. 1 is a diagram of an exemplary image-based productivity tracking system 100 described herein. As shown in FIG. 1, the productivity tracking system 100 may include an imaging device 102, a work machine 104, a management platform 106, a control station 108, a network storage device 110, and / or another device configured to facilitate construction work performed at a job site. The productivity tracking system 100 may be used to track the progress of construction work against a site plan. The work machine 104 may include a dozer, a tractor, a loader, a truck, a motor grader, a shovel, a paving machine, and / or another work machine 104 configured to perform work associated with construction. The construction may be related to construction, paving, demolition, mining, landscaping, forestry, pipelines, and / or other industrial applications. The site plan may be designed based on an initial or previous survey of the job site and may include specifications of the construction work to be performed (e.g., a target work path, a target cutting depth, a target grade, a target mat thickness, a target volume of materials to be removed, placed, and / or transported, and / or other specifications of the construction work that may be provided at various granularities).

[0011] In some embodiments, the productivity tracking system 100 can provide and / or support a productivity tracking service that receives information related to construction at the work site, determines the status of the construction, and provides the progress of the construction to an operator associated with the work site. In some examples, the productivity tracking system 100 can provide information related to the progress of the construction in the form of a digital model of the work site (e.g., a two-dimensional digital model and / or a three-dimensional digital model). The digital model can be generated using a combination of image data (e.g., provided by the imaging device 102) updated intermittently, periodically, and / or continuously in real time, and productivity data (e.g., provided by the work machine 104, the management platform 106, the control station 108, and / or the network storage device 110). The productivity tracking service and / or the information provided by the productivity tracking service can be made accessible to a local operator via the user interface of the work machine 104 and / or a subscriber of the productivity tracking service (e.g., a remote operator, a site supervisor, a supervisor, and / or the like) via the control station 108 (e.g., a portable device, a computer, and / or the like).

[0012] In some embodiments, the productivity tracking system 100 may include a plurality of imaging devices 102, a plurality of work machines 104, and / or a plurality of control stations 108 that interact with a management platform 106 and / or a network storage device 110. In some examples, the productivity tracking system 100 may include a plurality of management platforms 106 and / or a plurality of network storage devices 110 that interact with one or more imaging devices 102, one or more work machines 104, and / or one or more control stations 108. The productivity tracking system 100 may be used with manually operated work machines 104 and / or autonomously or semi-autonomously operated work machines 104. For example, the productivity tracking system 100 may be used to guide, navigate, and / or control autonomous or semi-autonomous work machines 104 based on, for example, position data of the work machine 104, coordinate data associated with a work site associated with the work machine 104, coordinate data associated with a site plan, and / or the like. In some embodiments, the work machine 104 may receive guidance, navigation, and / or control information from a local operator, from a remote operator via the control station 108, and / or from another device of the productivity tracking system 100.

[0013] As further shown in FIG. 1, the imaging device 102 includes a digital camera and / or the like configured to capture an image (e.g., a digital image and / or digital video) of a work site. As shown in the embodiment of FIG. 1, the imaging device 102 can be disposed on an unmanned aerial vehicle (e.g., a drone and / or the like) that can traverse the work site in one or more directions while generating image data associated with the work site (e.g., data related to the digital image and / or digital video captured by the imaging device 102). The imaging device 102 and / or the unmanned aerial vehicle may be remotely controlled by an operator and / or may be operated autonomously or semi-autonomously (e.g., based on commands and / or instructions provided via the work machine 104, the management platform 106, the control station 108, and / or the network storage device 110). In some embodiments, the imaging device 102 may be disposed on an unmanned ground vehicle, an unmanned watercraft, and / or another type of unmanned vehicle. In some embodiments, the imaging device 102 may correspond to a handheld digital camera, a digital camera of a portable device (e.g., a smartphone, a tablet, a laptop, and / or the like), and / or another type of digital camera having sufficient pixel density and / or resolution for capturing an image of the work site.

[0014] In some embodiments, imaging device 102 may be configured to determine a geospatial reference associated with the captured image. For example, imaging device 102 and / or the unmanned aerial vehicle may include a Global Positioning System (GPS) device, a Global Navigation Satellite System (GNSS) device, and / or another position sensing device configured to determine the position of imaging device 102 relative to the work site (e.g., geographic coordinates and / or the like). Imaging device 102 may determine the geospatial reference of the captured image based on the position of imaging device 102 when the image is captured. Imaging device 102 may generate image data based on the captured image and may embed or otherwise associate the geospatial reference with the image data. Imaging device 102 may use a geospatially tagged image file format (GeoTIFF), a LASer (LAS) file format, and / or another suitable file format that enables work machine 104, management platform 106, control station 108, and / or network storage device 110 to retrieve the geospatial reference from the image data and associate the position and the image relative to the work site based on the geospatial reference to generate the image data.

[0015] As further shown in FIG. 1, the work machine 104 includes a frame 112, a traction element 114, an operator cab 116, an engine 118, one or more sensing devices 120, and a control unit 122. The traction element 114 may be movably coupled to the frame 112 and may include wheels, tracks, and / or the like driven by the engine 118 to propel the work machine 104. The operator cab 116 may be coupled to the frame 112 and configured to support the operator of the work machine 104 and one or more components of the control unit 122. The engine 118 may include a diesel engine, a gasoline engine, a natural gas engine, a hybrid engine, an electric motor, and / or another power source configured to drive the traction element 114 to propel the work machine 104. In some embodiments, the work machine 104 may include an implement 124 (e.g., a blade, a ripper, a winch, a bucket, a shear, a hammer, and / or the like) movably coupled to the frame 112 and configured to perform work associated with the work machine 104. In some embodiments, the work machine 104 may include an on-board imaging device 126 (e.g., a digital camera and / or something similar to the imaging device 102) fixed to the work machine 104 and configured to capture an image associated with the work site, determine a geospatial reference associated with the image, generate image data based on the image, and embed the geospatial reference in the image data or otherwise associate the geospatial reference with the image data.

[0016] The sensing device 120 includes sensors, switches, encoders, and / or a combination of other devices configured to provide productivity data (e.g., information related to the productivity of the work machine 104). For example, the sensing device 120 can be configured to provide information related to the engine speed, track or wheel speed, travel speed, trip length, trip duration, runtime, fuel consumption rate, engine temperature, coolant temperature, and / or other operating states of the work machine 104. In some embodiments, the sensing device 120 can include a GPS device, a GNSS device, and / or another position sensing device configured to provide the position of the work machine 104 relative to the job site. When the sensing device 120 includes a position sensing device, the position of the work machine 104 determined by the sensing device 120 can be used as a geospatial reference that is embedded in or otherwise associated with the image data generated by the on-board imaging device 126. When the work machine 104 includes the implement 124, the sensing device 120 can include sensors, switches, encoders, and / or another device configured to provide information related to the operating state of the implement 124 (e.g., cutting depth, cutting length, cut volume, volume of material loaded or unloaded, and / or other operating states that can be used to determine the productivity of the work machine 104).

[0017] The control unit 122 includes a processor 128, a memory 130, a user interface 132, and a communication device 134. The processor 128 is implemented as programmable hardware, firmware, and / or a combination of hardware and software to execute functions associated with the work machine 104 and / or the productivity tracking system 100. The memory 130 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device that stores information and / or instructions executed by the processor 128. The user interface 132 includes an input device and an output device. The input device may enable an operator of the work machine 104 to specify instructions, commands, and / or other operating parameters for operating the work machine 104. The output device may enable an operator of the work machine 104 to access an image of the work site, access a site plan, monitor productivity data related to the construction associated with the work machine 104, monitor the status and / or progress of the construction associated with the work machine 104, track the position of the work machine 104 relative to the work site and / or relative to another work machine 104, access a history record associated with the work machine 104, and so on.

[0018] The communication device 134 includes a wireless local area network (WLAN) component (e.g., a Wi-Fi component), a radio frequency (RF) communication component (e.g., a Bluetooth component), and / or the like. The communication device 134 can enable communication between the work machine 104 and the imaging device 102, the management platform 106, the control station 108, and / or the network storage device 110. For example, the communication device 134 can enable the processor 128 to receive map data associated with the work site, productivity data related to the construction associated with the work machine 104, and / or image data associated with the work site from the imaging device 102, the management platform 106, the control station 108, and / or the network storage device 110. In some embodiments, the communication device 134 can enable the processor 128 to transmit productivity data associated with the work machine 104 (e.g., determined via the sensing device 120) and / or image data associated with the work site (e.g., generated by the on-board imaging device 126) to the management platform, the control station 108, and / or the network storage device 110. In some embodiments, the control unit 122 may communicate directly and / or indirectly with the imaging device 102, the control station 108, and / or the network storage device 110 via the management platform 106.

[0019] As further shown in FIG. 1, the management platform 106 includes a processor 136, a memory 138, and a communication device 140. The processor 136 is implemented as programmable hardware, firmware, and / or a combination of hardware and software to execute functions associated with the work machine 104 and / or the productivity tracking system 100. The memory 138 includes RAM, ROM, and / or another type of dynamic or static storage device that stores information and / or instructions executed by the processor 136. The communication device 140 includes a WLAN component (e.g., a Wi-Fi component), an RF communication component (e.g., a Bluetooth component), a positioning component (e.g., a GPS component, a GNSS component), and / or the like. The communication device 140 can enable the processor 136 to exchange map data associated with the work site, productivity data related to the construction associated with the work machine 104, and / or image data associated with the work site having the imaging device 102, the work machine 104, the management platform 106, the control station 108, and / or the network storage device 110. In some embodiments, the communication device 140 can enable the exchange of map data, productivity data, and / or image data among the imaging device 102, the work machine 104, the management platform 106, the control station 108, and / or the network storage device 110.

[0020] As further shown in FIG. 1, the control station 108 includes a processor 142, a memory 144, a user interface 146, and a communication device 148. The processor 142 is implemented as programmable hardware, firmware, and / or a combination of hardware and software to perform functions associated with the work machine 104 and / or the productivity tracking system 100. The memory 144 includes RAM, ROM, and / or another type of dynamic or static storage device that stores information and / or instructions executed by the processor 142. The user interface 146 includes an input device and an output device. The input device may enable a remote operator of the work machine 104 to specify instructions, commands, and / or other operating parameters for operating the work machine 104. The output device may enable a remote operator of the work machine 104 to access an image of the job site, access a site plan, monitor productivity data related to the construction associated with the work machine 104, monitor the status and / or progress of the construction associated with the work machine 104, track the position of the work machine 104 relative to the job site and / or another work machine 104, access a history record associated with the work machine 104, and / or the like.

[0021] The communication device 148 includes a wireless local area network (WLAN) component (e.g., a Wi-Fi component), a radio frequency (RF) communication component (e.g., a Bluetooth component), and / or the like. The communication device 148 can enable communication between the control station 108 and the imaging device 102, the work machine 104, the management platform 106, and / or the network storage device 110. For example, the communication device 148 can enable the processor 142 to exchange map data associated with the work site, productivity data related to the construction associated with the work machine 104, and / or image data associated with the work site having the imaging device 102, the work machine 104, the management platform 106, and / or the network storage device 110. In some embodiments, the communication device 148 can enable the exchange of map data, productivity data, and / or image data between the imaging device 102, the work machine 104, the management platform 106, and / or the network storage device 110. In some embodiments, the control station 108 may communicate directly and / or indirectly with the work machine 104 and / or the network storage device 110 via the management platform 106. Additionally, or alternatively, the control station 108 can function as a user interface of the management platform 106.

[0022] As further shown in FIG. 1, the network storage device 110 includes one or more devices capable of storing, processing, and / or routing information. The network storage device 110 can include, for example, a server device, a device storing a data structure, a device within a cloud computing environment or a data center, and / or the like. In some embodiments, the network storage device 110 can include a communication interface that enables the network storage device 110 to receive information from and / or transmit information to another device associated with the work machine 104, the management platform 106, the control station 108, and / or the productivity tracking system 100. In some embodiments, the network storage device 110 can store information related to the work machine 104 and / or the productivity tracking system 100. For example, the network storage device 110 may be used to store map data associated with a work site, productivity data related to a construction project associated with the work machine 104, and / or image data associated with the work site, and the imaging device 102, the work machine 104, the management platform 106, and / or the control station 108 may be enabled to access the map data, the productivity data, and / or the image data.

[0023] As described above, FIG. 1 is provided as one example. Other examples may differ from those described in relation to FIG. 1.

[0024] Figures 2A - 2D are diagrams of an exemplary embodiment 200 of the productivity tracking system 100 described herein. As shown in FIG. 2A, the productivity tracking system 100 (e.g., via the control unit 122, the management platform 106, and / or the control station 108) can be configured to generate a map layer 202 based on map data associated with the work site 204. The map data can include information related to geographical attributes associated with the work site 204 and geographical coordinates associated with the geographical attributes. For example, the map data can include geographical coordinates corresponding to roads following the work site 204, boundaries of the work site 204, landmarks associated with the work site 204, fixed structures associated with the work site 204, reference points associated with the work site 204, and / or other geographical references that assist in identifying and / or defining the location of the work site 204. In some embodiments, the map layer 202 can be generated using a grid of map tiles 206 obtained based on the map data. The map tiles 206 can provide visual representations of different geographical areas associated with the work site 204, arranged according to the corresponding geographical coordinates.

[0025] In some embodiments, the productivity tracking system 100 may be configured to generate the map layer 202 using map tiles 206 in an arrangement different from that shown. For example, the map layer 202 may be formed using more smaller map tiles 206 than shown, or fewer larger map tiles 206 than shown. In some examples, the productivity tracking system 100 may adjust the dimensions of the individual map tiles 206 to change the zoom level of the map layer 202 and / or the resolution of the map layer 202. Additionally, or alternatively, the productivity tracking system 100 may adjust the pixel resolution of the individual map tiles 206 to change the resolution of the map layer 202. In some embodiments, the map layer 202 may be formed using map tiles 206 arranged in a format different from the grid arrangement. The map data and / or corresponding map tiles 206 may be obtained from the network storage device 110 and / or another resource accessible to the productivity tracking system 100. In some embodiments, the productivity tracking system 100 may be configured to adjust the view (e.g., zoom level, pan direction, and / or the like) of the work site 204 depicted via the map layer 202 according to input received from an operator (e.g., via the user interface 132 of the work machine 104 and / or the user interface 146 of the control station 108).

[0026] As shown in FIG. 2B, the productivity tracking system 100 can be configured to generate an image layer 208 based on image data associated with the work site 204. The image data can include an image corresponding to the state of the construction work performed by the work machine 104 at the work site 204 and information related to the geospatial reference associated with the image. For example, the image can correspond to the state of the work site 204 captured within a threshold time (e.g., in the range of seconds, minutes, hours, days, weeks, and / or the like), to a degree of threshold resolution of the work site 204, and / or to another attribute of the work site 204. The image data may be received from an imaging device 102 configured to capture an image, determine a geospatial reference associated with the image, generate image data based on the image, and embed the geospatial reference in the image data. In some embodiments, the image data may be converted into a grid of image tiles 210 corresponding to the image. Each image tile 210 can depict a cross-sectional image of a different geographic area associated with the work site 204 arranged according to its respective geospatial reference. The grid of image tiles 210 may be sized and spatially arranged to correspond to the grid of map tiles 206. Although shown as an aerial view of the work site 204, the image layer 208 may provide different views of the work site 204 (e.g., a surface level view of the work site 204 using image data provided via the on-board imaging device 126 of the work machine 104).

[0027] In some embodiments, the productivity tracking system 100 may be configured to generate the image layer 208 using an arrangement of image tiles 210 that is different from that shown. The productivity tracking system 100 may adjust the dimensions of the individual image tiles 210 to change the zoom level of the image layer 208 and / or the resolution of the image layer 208. Additionally, or alternatively, the productivity tracking system 100 may adjust the pixel resolution of the individual image tiles 210 to change the resolution of the image layer 208. In some examples, the image layer 208 may be formed using image tiles 210 arranged in a format different from a grid arrangement. The image layer 208 may be updated in real time intermittently, periodically, and / or continuously. For example, the productivity tracking system 100 (e.g., via the control unit 122, the management platform 106, and / or the control station 108) may transmit command signals to the imaging device 102 and / or the on-board imaging device 126 to capture updated images associated with the work site 204 and generate updated image data based on the updated images. In some embodiments, the productivity tracking system 100 may be configured to adjust the view of the work site 204 depicted via the image layer 208 according to an input received from an operator.

[0028] As shown in FIG. 2C, the productivity tracking system 100 can be configured to generate a productivity layer 212 based on productivity data associated with the work site 204. The productivity data can include information related to the state of the construction work associated with the work site 204. The productivity data can be obtained via information provided through the work machine 104 (e.g., the sensing device 120), the management platform 106, the control station 108, and / or the network storage device 110. In some embodiments, the productivity data may be manually input by an operator and / or derived based on the performance of the work machine 104, an inspection of the work site 204, and / or an evaluation of information related to the same type. For example, the productivity data may include information regarding the position of the work machine 104, the work of the work machine 104, the operating parameters of the work machine 104, the operating state of the work machine 104, the measured values associated with the work site 204, the position of the target work path associated with the work site 204, the position of the stockpiles associated with the work site 204, the measured values of the stockpiles, the progress of the construction work with respect to the associated site plan, and / or other information that can be used to track the progress of the construction work. In some embodiments, the productivity data may include a productivity index (e.g., an evaluation, a score, and / or another metric for quantifying the productivity data).

[0029] In some embodiments, the productivity tracking system 100 can be configured to generate a visual representation (e.g., symbols, annotations, and / or the like) associated with productivity data and generate a productivity layer 212 that includes the visual representation. As shown in the embodiment of FIG. 2C, the productivity layer 212 can include a first symbol 214 representing the position and / or heading of the work machine 104 relative to the work site 204, a second symbol 216 representing the position of the stockpiles within the work site 204, and / or the like. In some embodiments, the productivity layer 212 can include information related to the work machine 104 (e.g., position, current work, operating status, productivity index of the work machine 104, and / or the like), information related to the stockpiles (e.g., remaining quantity of the stockpiles, quantity of the stockpiles that have been moved, and / or the like), and / or annotations 218 providing information related to the overall progress of the construction (e.g., general productivity index, percentage of the construction completed relative to the site plan, percentage of the remaining construction relative to the site plan, number of days elapsed since the start of the construction, number of remaining days until completion, and / or the like).

[0030] In some implementations, the productivity tracking system 100 may be configured to update the productivity layer 212 intermittently, periodically, and / or continuously in real time. The productivity tracking system 100 may obtain updated productivity data (e.g., via the control unit 122, the management platform 106, and / or the control station 108) and may be configured to adjust the visual representation of the productivity layer 212 according to the updated productivity data. For example, the position and / or orientation of the first symbol 214 may be adjusted in response to a change in the position and / or heading of the work machine 104, and the content of the annotation 218 may be changed in response to a change in the state or progress of the construction. In some embodiments, the productivity tracking system 100 may update the productivity layer 212 and the image layer 208 according to a synchronization interval such that the productivity data coincides with the image data. In some embodiments, the productivity tracking system 100 may update the image layer 208 based on a change detected in the productivity layer 212 and / or may update the productivity layer 212 based on a change detected in the image layer 208. Additionally, or alternatively, the productivity tracking system 100 may be configured to adjust the diagrams and / or content provided via the productivity layer 212 according to operator input.

[0031] As shown in FIG. 2D, the productivity tracking system 100 can be configured to generate a composite image 220 of the work site 204 based on the map layer 202, the image layer 208, and / or the productivity layer 212. For example, the productivity tracking system 100 may merge, overlay, super-impose, and / or otherwise position the image layer 208 relative to the map layer 202 such that the geospatial reference associated with the image layer 208 corresponds to the geographic coordinates associated with the map layer 202. If map tiles 206 and image tiles 210 are used, the grid of image tiles 210 may be sized and spatially arranged to correspond to the grid of map tiles 206. In some embodiments, the productivity tracking system 100 may merge, overlay, super-impose, and / or otherwise position the productivity layer 212 relative to the image layer 208 and / or the map layer 202. If the productivity layer 212 includes a visual representation related to location information (e.g., the first symbol 214 indicates the location of the work machine 104, the second symbol 216 indicates the location of the stockpile, etc.), the visual representation may be positioned at the corresponding geospatial reference of the image layer 208 and / or the corresponding geographic coordinates of the map layer 202 such that the composite image 220 depicts a combination of information related to the map data, the image data, and / or the productivity data associated with the work site 204.

[0032] In some embodiments, the productivity tracking system 100 may be configured to update the composite image 220 intermittently, periodically, and / or continuously in real time. For example, the productivity tracking system 100 may modify the image layer 208 based on updated image data, modify the productivity layer 212 based on updated productivity data, and generate an updated composite image 220 with the modified image layer 208 and / or the modified productivity layer 212. Additionally, or alternatively, the productivity tracking system 100 may generate a new image layer 208 based on a new productivity layer 212 based on updated image data and / or updated productivity data, and generate a new composite image 220 based on the new image layer 208 and / or the new productivity layer 212. In some examples, the productivity tracking system 100 may enable an operator to interact with the composite image 220 to adjust the diagrams and / or content provided via the composite image 220. For example, the operator may be able to adjust the zoom level, adjust the pan direction, modify the visual display, display different layer combinations, and the like.

[0033] In some embodiments, the productivity tracking system 100 may be configured to cause an action to be performed based on the composite image 220. For example, the productivity tracking system 100 may generate a record of the successive composite images 220 of the work site 204 and the respective timestamps when the successive composite images 220 were generated. In some examples, the productivity tracking system 100 may generate a catalog of successive composite images 220 indexed via the respective timestamps and store the catalog in a network storage device 110 and / or another data structure accessible to the operator. For example, the operator may use the catalog to view the state of the construction at the work site 204 prior to construction. Additionally or alternatively, the operator may use the catalog to compare the successive composite images 220 and observe the progress of the construction at the work site 204 based on the change in state between the successive composite images 220. In some examples, the productivity tracking system 100 may use a change detection model and / or another computer vision model to compare the successive composite images 220, identify visual differences between the successive composite images 220, and determine the status and / or progress of the construction associated with the work site 204 based on the visual differences.

[0034] In some embodiments, the productivity tracking system 100 may be configured to transmit a command signal to the imaging device 102 to capture an updated image associated with the work site 204 and generate updated image data based on the updated image. When the imaging device 102 is an unmanned aerial vehicle, the productivity tracking system 100 may deploy the imaging device 102 to a desired advantageous location and / or traverse a desired path with respect to the work site 204 and transmit a command signal to cause the imaging device 102 to capture an updated image of the work site 204. The productivity tracking system 100 may automatically deploy the imaging device 102 intermittently and / or periodically. Additionally, or alternatively, the productivity tracking system 100 may instruct the imaging device 102 to hover at a fixed position and continuously capture updated images of the work site 204 for a threshold duration. In some examples, the productivity tracking system 100 may be configured to receive survey data related to a site plan associated with the work site 204. The productivity tracking system 100 may generate a survey layer based on the survey data and generate a composite image 220 of the work site 204 based on the survey layer. For example, the survey layer may be positioned relative to the map layer 202, the image layer 208, and / or the productivity layer 212 in a manner that enables an operator to visually observe the progress of the construction at the work site 204 with respect to the site plan.

[0035] As described above, FIGS. 2A-2D are provided as examples. Other examples may differ from those described in relation to FIGS. 2A-2D.

[0036] FIG. 3 is a flowchart of an exemplary process 300 for tracking construction productivity at a work site. One or more of the process blocks in FIG. 3 may be performed by a management platform (e.g., the management platform 106 of the productivity tracking system 100) and / or by a component or group of components separate from or including the management platform (e.g., the imaging device 102 of the work machine 104, the control unit 122, the control station 108 of the productivity tracking system 100, the network storage device 110, and / or another device).

[0037] As shown in FIG. 3, process 300 may include receiving map data associated with a work site, the map data including information related to geographical attributes associated with the work site and geographical coordinates associated with the geographical attributes (block 302). For example, a management platform (e.g., using processor 136, memory 138, communication device 140, and / or the like) may receive map data associated with a work site as described above. The map data may include information related to geographical attributes associated with the work site and geographical coordinates associated with the geographical attributes.

[0038] As further shown in FIG. 3, process 300 may include receiving productivity data associated with a work site, the productivity data including information related to the state of construction associated with the work site (block 304). For example, a management platform (e.g., using processor 136, memory 138, communication device 140, and / or the like) may receive productivity data associated with a work site as described above. The productivity data may include information related to the state of construction associated with the work site.

[0039] As further shown in FIG. 3, process 300 may include receiving image data associated with a work site, the image data including an image corresponding to the state of construction and information related to a geospatial reference associated with the image (block 306). For example, a management platform (e.g., using processor 136, memory 138, communication device 140, and / or the like) may receive image data associated with a work site as described above. The image data may include an image corresponding to the state of construction, and information related to a geospatial reference associated with the image.

[0040] As further shown in FIG. 3, process 300 may include generating a map layer based on map data (block 308). For example, a management platform (e.g., using processor 136, memory 138, communication device 140, and / or the like) may generate a map layer based on the map data as described above.

[0041] As further shown in FIG. 3, process 300 may include generating an image layer based on image data (block 310). For example, a management platform (e.g., using processor 136, memory 138, communication device 140, and / or the like) may generate an image layer based on the image data as described above.

[0042] As further shown in FIG. 3, process 300 may include generating a productivity layer based on productivity data (block 312). For example, a management platform (e.g., using processor 136, memory 138, communication device 140, and / or the like) may generate a productivity layer based on the productivity data as described above.

[0043] As further shown in FIG. 3, process 300 may include generating a composite image of the work site based on the map layer, the image layer, and the productivity layer, and the composite layer positions the image layer relative to the map layer and positions the productivity layer relative to the image layer based on a geospatial reference and the geographic coordinates corresponding to the geospatial reference (block 314). For example, a management platform (e.g., using processor 136, memory 138, communication device 140, and / or the like) may generate a composite image of the work site based on the map layer, the image layer, and the productivity layer as described above. The composite image may position the image layer relative to the map layer and position the productivity layer relative to the image layer based on a geospatial reference and the geographic coordinates corresponding to the geospatial reference.

[0044] As further shown in FIG. 3, process 300 may include causing an action to be performed based on the composite image (block 316). For example, a management platform (e.g., using processor 136, memory 138, communication device 140, and / or the like) may cause an action to be performed based on the composite image as described above.

[0045] Process 300 may include variations and / or additional embodiments to those described in connection with FIG. 3, such as any single embodiment described elsewhere herein, or any combination of embodiments. Although FIG. 3 shows exemplary blocks of process 300, in some embodiments, process 300 may include additional blocks, fewer blocks, different blocks, or blocks in a different arrangement than those depicted in FIG. 3. Additionally, or alternatively, two or more of the blocks of process 300 may be executed in parallel.

Industrial Applicability

[0046] A job site may include one or more work machines that are individually operated to perform work and are collectively managed to complete a project according to a site plan. The project may be related to construction, paving, demolition, mining, landscaping, forestry, pipelines, and / or other industrial applications. The site plan may include specifications for the project to be performed at the job site, designed based on an initial or previous survey of the job site. In some cases, a tracking tool may be used to track the progress of the project against the site plan. The tracking tool may be updated intermittently, periodically, and / or continuously in real time based on updated survey data (e.g., related to structural and / or geographical changes within the job site), productivity data (e.g., related to the location and / or operating status of work machines within the job site), and / or other information that can be used to track progress. In some cases, the tracking tool may provide a two-dimensional digital model and / or a three-dimensional digital model of the job site that can be used to guide an operator of a work machine. For example, the tracking tool may be made accessible to remote and / or local operators via a mobile device, a control station, a display within a work machine, and / or another user interface.

[0047] In some cases, the tracking tool may provide a map view with an image of the work site indicating the respective locations of the machine tools, spare parts, and / or other assets or attributes associated with the work site. The respective locations of the assets and / or attributes within the work site can be updated based on productivity data obtained and / or derived from the information provided by the machine tools. However, the image used to depict the work site may be outdated (e.g., based on outdated satellite imagery), which can be inaccurate and / or misleading for an operator relying on the tracking tool. In some cases, the operator may use a drone with an imaging device to survey the work site. For example, the drone can be used to capture an updated image of the work site that the operator can use to visually evaluate the state of the work site. However, the drone does not provide information (e.g., productivity data from the machine tools) that can be used to track the progress of the work within the work site. Thus, the operator may need to separately analyze the information provided via the tracking tool and the information provided via the drone to evaluate the state of the work site. Using separate systems to analyze a single project can result in errors, delays, inefficient use of machine tools, and inefficient use of computing and / or network resources.

[0048] The image-based productivity tracking system described herein provides a single resource for tracking the progress of a construction project using both updated image data and updated productivity data. For example, an image-based productivity tracking system can generate an image layer using updated image data received from an imaging device, generate a productivity layer using updated productivity data received from a work machine, and generate a composite image of the job site based on the image layer and the productivity layer. The composite image can position the image layer relative to the map layer based on geospatial references associated with the image data and geographic coordinates associated with the map layer. The productivity layer can provide visual representations and / or annotations that index assets and / or attributes within the image layer. In some embodiments, the image-based productivity tracking system can enable manual and / or machine-based visual comparison between successive composite images to determine the progress of the construction project. In some embodiments, the image-based productivity tracking system can control the imaging device to acquire updated images intermittently, periodically, and / or continuously.

[0049] Therefore, the image-based productivity tracking system described herein can enable tracking of the progress of construction work at both image-based and database job sites. By overcoming the need to refer to multiple different systems to obtain image data and productivity data related to the progress of construction, the image-based productivity tracking system saves the computational resources and / or network resources that might otherwise be required to obtain such image data and productivity data. Additionally, since both the image data and the productivity data are processed by a single resource, the image-based productivity tracking system reduces the opportunities for ambiguity and error that might otherwise occur when corroborating analyses from separate systems. Thereby, the operator can track the progress of construction more accurately and in less time with the image-based productivity tracking system. With the image-based productivity tracking system, the operator can quickly identify and correct errors or deviations from the site plan. Further, the image-based productivity tracking system enables the construction to be completed more efficiently, thereby saving energy (e.g., fuel, gas, electricity, and / or the like) and reducing unnecessary wear on work machinery.

Claims

1. An apparatus (122, 106, 108) comprising: one or more memories (130, 138, 144); and one or more processors (128, 136, 142) communicatively coupled to the one or more memories (130, 138, 144), the one or more processors (128, 136, 142) configured to: receive productivity data associated with a work site (204), the productivity data including information related to a state of a construction project associated with the work site (204); receive image data associated with the work site (204), the image data including an image corresponding to the state of the construction project and information related to a geospatial reference associated with the image; generate an image layer (208) based on the image data; generate a productivity layer (212) based on the productivity data; generate a composite image (220) of the work site (204) based on a map layer (202), the image layer (208), and the productivity layer (212), the map layer (202) including information related to geographical attributes associated with the work site (204) and geographical coordinates associated with the geographical attributes, the composite image (220) positioning the image layer (208) relative to the map layer (202) and positioning the productivity layer (212) relative to the image layer (208) based on the geospatial reference and geographical coordinates corresponding to the geospatial reference; cause an action to be performed based on the composite image (220); generate an updated composite image (220) based on updated image data and updated productivity data; use a computer vision model to compare the composite image (220) to the updated composite image (220); identify a visual difference between the composite image (220) and the updated composite image (220); and determine a state of the construction project associated with the work site (204) based on the visual difference.

2. The one or more processors (128, 136, 142) are configured to receive the image data from an imaging device (102) of an unmanned vehicle when receiving the image data. ​ ​ ​ ​ ​ The apparatus (122, 106, 108) according to claim 1, wherein the imaging device (102) is configured to capture the image, determine the geospatial reference associated with the image, generate the image data based on the image, and embed the geospatial reference in the image data.

3. When generating the image layer (208), the one or more processors (128, 136, 142) convert the image data into a grid of one or more image tiles (210) corresponding to the image, wherein the one or more image tiles (210) each have a respective geospatial reference and are arranged according to the respective geospatial reference, convert the one or more image tiles (210) to correspond to one or more map tiles (206) of the map layer (202), and generate the image layer (208) based on the grid of the one or more image tiles (210). The apparatus (122, 106, 108) according to any one of claims 1 to 2 is configured to perform the above.

4. When generating the composite image (220), the one or more processors (128, 136, 142) overlay the image layer (208) on the map layer (202) at geographical coordinates corresponding to the geospatial reference, overlay the productivity layer (212) on the image layer (208), wherein the productivity layer (212) includes location information associated with the productivity data, and the productivity layer (212) is positioned relative to the image layer (208) based on the location information. The apparatus (122, 106, 108) according to any one of claims 1 to 3 is configured to perform the above overlay.

5. When causing the action to be executed, the one or more processors (128, 136, 142) transmit a command signal to the imaging device (102) to capture an updated image associated with the work site (204) and generate updated image data based on the updated image, receive the updated image data, and generate an updated composite image (220) of the work site (204) based on the updated image data. The apparatus (122, 106, 108) according to any one of claims 1 to 4 is configured to perform the above.

6. A work machine (104), A sensing device (120) configured to provide productivity data of the work machine (104), wherein the productivity data includes information related to the state of a construction project associated with a work site (204), the sensing device; A user interface (132, 146) configured to display information related to the state of the construction project to an operator of the work machine (104); A control unit (122) that communicates with the sensing device (120) and the user interface (132, 146), wherein the control unit (122) is Generating a productivity layer (212) based on the productivity data; Generating an image layer (208) based on image data, wherein The image data includes an image corresponding to the state of the construction project and information related to a geospatial reference associated with the image, generating; Generating a composite image (220) of the work site (204) based on a map layer (202), the image layer (208), and the productivity layer (212), wherein The map layer (202) includes information related to geographical attributes associated with the work site (204) and geographical coordinates associated with the geographical attributes; The composite image (220) positions the image layer (208) relative to the map layer (202) and positions the productivity layer (212) relative to the image layer (208) based on the geospatial reference and geographical coordinates corresponding to the geospatial reference, generating; Causing the composite image (220) to be displayed via the user interface (132, 146), a control unit configured to perform; and is provided with, The control unit (122) is Generating an updated composite image (220) based on updated image data and updated productivity data; Using a computer vision model to compare the composite image (220) with the updated composite image (220); Identifying a visual difference between the composite image (220) and the updated composite image (220); A work machine (104) further configured to perform determining the state of the construction project associated with the work site (204) based on the visual difference.

7. When generating the productivity layer (212), the control unit (122) is configured to generate a visual display associated with the productivity data, wherein the visual display includes information related to one or more of the position of the work machine (104), the position of the supplies associated with the work site (204), or the position of the target work path associated with the work site (204), The work machine (104) according to claim 6, wherein the productivity layer (212) is generated based on the visual display.

8. When generating the composite image (220), the control unit (122) superposes the image layer (208) on the map layer (202) at geographical coordinates corresponding to the geospatial reference, and superposes the productivity layer (212) on the image layer (208), wherein the productivity layer (212) includes position information associated with the productivity data, The work machine (104) according to any one of claims 6 to 7, wherein the productivity layer (212) is configured to perform superposition in which the productivity layer (212) is positioned with respect to the image layer (208) based on the position information.

9. The work machine (104) according to any one of claims 6 to 8, further comprising an imaging device (126) configured to capture the image, determine the geospatial reference associated with the image, generate the image data based on the image, and embed the geospatial reference in the image data in a geospatial-tagged image file format (GeoTIFF).

Images