System for verifying a work site

Abstract

A system for validating a work site (214) includes determining a scope of a work site where a job is to be performed. A graphical representation of the work site (214) can be presented to a user via a graphical user interface (200) on a user device (202). The graphical user interface (200) can also include an indication of a current location (216) of the user or user device (202). Sensor data, including data indicative of a location of the user or user device (202), can be used to determine whether the work site (214) is valid. A work site plan (122) can include information regarding additional conditions for validating the work site (214), and additional sensor data can be used to further validate the work site (214). Further aspects can include controlling a machine (104) at the work site (214) when the work site (214) is validated.

Description

Technical Field

[0001] This disclosure generally relates to controlling machines at work sites such as construction, industrial and / or mining sites, and more specifically to systems that utilize sensor and data processing technologies to identify events and / or intervene in such events through machine control. Background Technology

[0002] Many industrial work sites involve machines, equipment, and / or personnel performing various functions. For example, a mine might include excavators, haul trucks, and operators who mine ore or other resources and dispose of waste. The operation of such machines and personnel must be coordinated to maintain efficiency at the site. In some cases, non-line-of-sight (NLOS) methods, such as remote methods, can be used to assign tasks to the work site. Furthermore, such tasks can be performed by remotely controlled machines and / or autonomous or semi-autonomous machines. Remote task assignment and remote or autonomous task execution can increase safety outcomes by reducing human contact with heavy machinery and certain operations. However, if the work site where the task will be performed is unsuitable, NLOS assignment of tasks and remote or autonomous control of machines can have undesirable consequences. For example, attempting to perform a task in an unsuitable work site may result in excessive machine wear or damage, task failure, or other inefficiencies. Therefore, it may be useful to verify the work site before attempting to perform a task there.

[0003] An exemplary system for determining the extent or perimeter of a work site for autonomous or semi-autonomous task execution is described in U.S. Patent No. 6,112,143 (hereinafter referred to as '143 Reference'). Specifically, '143 Reference describes a system for activating a learning mode on a machine. When in learning mode, '143 Reference describes positioning the machine at multiple locations on the perimeter of the work site and determining the perimeter based on location information associated with these locations. As explained in '143 Reference', the perimeter may be displayed to a user for confirmation or otherwise verification of the work site's extent. Once the work site boundaries are established, the learning mode can be deactivated, and the machine may perform tasks autonomously or semi-autonomously within the perimeter. However, '143 Reference does not disclose details relating to determining that the work site is additionally ready for job execution. As a non-limiting example, '143 Reference does not consider the presence of obstacles or other conditions that may affect the machine's ability to perform tasks. Therefore, the techniques described in '143 Reference' may not prevent accidents such as equipment failure due to an unsuitable work site.

[0004] Exemplary embodiments of this disclosure relate to overcoming the aforementioned deficiencies. Summary of the Invention

[0005] In one aspect of this disclosure, a computer-implemented method includes: receiving a work site plan at a computing device, the work site plan including at least one task to be performed by a machine at the work site, a set of work site condition parameters, and the boundaries of the work site; displaying a user interface on a display of the computing device, the user interface including a map including a representation of the boundaries of the work site and a representation of the location of the computing device. The method may further include receiving location information including one or more updated locations of the computing device; generating a verification signal based on the location information indicating that the work site has been verified. The method may further include transmitting the verification signal from the computing device.

[0006] In another aspect of this disclosure, the system includes a computing device; one or more sensors; one or more processors; and a memory storing processor-executable instructions. When executed by the one or more processors, the processor-executable instructions configure the system to perform actions including: receiving a work site plan, the work site plan including: the boundaries of a work site where a machine will perform a task; at least one work site condition parameter associated with the work site; and information about the task to be performed by at least one machine at the work site. The actions may further include receiving sensor data associated with the work site from the one or more sensors, the sensor data including at least one of positioning data or image data; and generating a first signal indicating that the work site has been visually inspected, at least in part based on the sensor data. The actions may further include receiving condition parameter data indicating that the at least one work site condition parameter is satisfied. The actions may further include generating a graphical user interface for display on the computing device, the graphical user interface including a graphical representation of the work site and user interface elements; and generating a verification signal indicating that the work site has been verified, based on the first signal, the condition parameter data, and user input indicating user interaction with the user interface elements. The actions may further include transmitting the verification signal.

[0007] In another aspect of this disclosure, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, perform actions including: receiving a work site plan, the work site plan including the boundaries of a work site where a machine will perform a task, at least one work site condition parameter associated with the work site, and information about the task to be performed by at least one machine at the work site. The actions may further include receiving sensor data associated with the work site from one or more sensors, the sensor data including at least one of positioning data or image data, and generating a first signal indicating that the work site has been visually inspected, at least in part, based on the sensor data. The actions may further include generating a graphical user interface for display on a computing device, the graphical user interface including a graphical representation of the work site and user interface elements, and receiving condition parameter data indicating that the at least one work site condition parameter is satisfied. The actions may further include generating a verification signal indicating that the work site has been verified based on the first signal, the condition parameter data, and user input indicating user interaction with the user interface elements, and transmitting the verification signal. Attached Figure Description

[0008] Figure 1 This is a perspective view of the environment in which the operation is performed, according to exemplary embodiments of the present disclosure.

[0009] Figure 2 This is a schematic diagram of a graphical representation of the environment and a graphical user interface for identifying a specific work site, according to exemplary embodiments of the present disclosure.

[0010] Figure 3 This is a schematic diagram of another graphical user interface according to another exemplary embodiment of the present disclosure.

[0011] Figure 4 This is a schematic diagram of another graphical user interface according to another exemplary embodiment of the present disclosure.

[0012] Figure 5 This is a schematic diagram of another graphical user interface according to another exemplary embodiment of the present disclosure.

[0013] Figure 6 This is a schematic diagram of another graphical user interface according to another exemplary embodiment of the present disclosure.

[0014] Figure 7 This is an exemplary computing environment for verifying a work site, according to exemplary embodiments of the present disclosure.

[0015] Figure 8 This is a flowchart illustrating an exemplary method for verifying a work site according to exemplary embodiments of the present disclosure.

[0016] Figure 9 This is a flowchart illustrating an exemplary method associated with a verification work site according to exemplary embodiments of the present disclosure. Detailed Implementation

[0017] This disclosure generally relates to the sensing, coordination, and / or control of machines and personnel to increase safety and reduce incidents such as accidents or injuries in geographical areas of a work site. In implementations, the systems and techniques described herein can be used in conjunction with methods that allow for the autonomous, semi-autonomous, and / or remotely operated execution of machine-based tasks to verify the work site. The same reference numerals will be used wherever possible in all the accompanying drawings to refer to the same or similar features.

[0018] refer to Figure 1 An exemplary environment 100 may be a mining location on which multiple operations are being performed. For example, environment 100 may include an open-pit mine 102 from which ore and / or other materials can be extracted. Additional or alternative operations (not shown) performed on site may include, but are not limited to, tunneling, blasting and / or other mining or landscaping operations. While such operations are particularly well-suited to mining, aspects of this disclosure are applicable to geographic areas and environments other than mines. For example, but not limited to, the aspects described herein may be applicable to many geographic areas where coordinated activities are performed. In some cases, environment 100 may include aspects of paving sites, industrial sites, factory grounds, building construction sites, road construction sites, quarries, buildings, cities, etc. Furthermore, the characteristics of environment 100 may change over time, for example, as environment 100 evolves from mining, excavation, leveling and / or other machine-based activities and / or due to erosion or other natural changes.

[0019] There may be many machines in environment 100. For example, Figure 1An excavator 104a in an open-pit mine 102 and trucks 104b, 104c, and 104d performing various hauling and dumping operations throughout the work site 100 are shown. Excavators 104a and trucks 104b-104d are merely examples. Other machines, including but not limited to earthmoving machinery such as wheel loaders, dump trucks, backhoes, bulldozers, or material handling equipment, such as tankers for transporting water or fuel, road vehicles, and working machines such as pavers or compactors, may additionally or alternatively be present in environment 100. Furthermore, other locations or types of environments may require different machines, and the techniques described herein can be applied to these different locations / environments. As used herein, the term "machine" can refer to any type of machine performing operations associated with a particular industry (such as mining, construction, agriculture, transportation, oil and gas, manufacturing) or any other industry. Throughout this disclosure, one or more machines, including but not limited to excavators 104a and / or trucks 104b, 104c, and 104d, may be referred to as "machine 104".

[0020] In some embodiments, the tasks, jobs, or operations performed in environment 100 may be substantially or fully autonomous. However, in the illustrated embodiments, a group or personnel 106a, 106b may also be present. For example, personnel 106a may be a worker or operator in an open-pit mine 102, and personnel 106b may be a supervisor or foreman overseeing operations in environment 100. Although only two personnel 106-1, 106-2 are shown, additional personnel may be present in environment 100. Furthermore, although example tasks or jobs may be attributed to personnel 106 for illustrative purposes herein, these are merely examples. In the embodiments described herein, personnel may perform manual labor or operations on machine 104, such as supervising the operation of machine 104, visually inspecting the work site in environment 100, such as identifying obstacles and / or verifying work plans, as further detailed herein, and / or performing other tasks.

[0021] One or more of the machines 104 may be configured with one or more communication components 108 and / or one or more sensing components 110 or otherwise access one or more communication components and / or one or more sensing components. Additionally, one or more personnel may have one or more accompanying communication components 112 and / or one or more sensing components 114. For example, communication components 108, 112 may include transmitters / receivers, including antennas, to facilitate wireless communication, for example, via radio frequency (RF) signals, via wireless networks, such as local area networks (LANs) or wide area networks (WANs), or any other communication protocols. Communication component 108 may be configured to communicate with, for example, a central hub located in a fixed position, which is configured to receive all or, if any, of the communications and route them as appropriate. In other embodiments, communication components 108 may be configured to communicate directly with each other, for example, via a distributed network. Hybrid networks and / or different communication networks and protocols may be used alternatively; this disclosure does not rely on a single arrangement, technology, or protocol. Communication components 108, 112 may generally include any device with which a human user, such as personnel 106, machine operators, etc., can interact. For example, communication components 108 and 112 may include input devices such as touchscreens or keyboards, speakers and / or microphones to facilitate verbal communication, and / or other communication technologies. As a non-limiting example, communication component 108 may include a touchscreen in one of the machines 104, one or more radios accessible in machine 104, a speaker system in one of the machines 104, one or more microphones disposed in one of the machines 104, etc. Furthermore, communication component 112 may include a mobile phone, tablet device, radio, headset, or other electronic device associated with person 106 that facilitates communication between person 106 and remote systems, remote devices, or remotely located persons.

[0022] Sensing component 110 can be any type and any number of sensor patterns fixed to or otherwise associated with machine 104, and sensing component 114 can include any number of sensor patterns associated with personnel 106 or communication components 112 accompanying personnel 106. As a non-limiting example, sensing component 110 and / or sensing component 114 may include positioning sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., imaging sensors, range sensors, RGB, UV, IR, intensity, depth, etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), LiDAR sensors, RADAR sensors, ultrasonic transducers, and / or sonar sensors. Sensing component 110 can be configured to sense the external or internal (e.g., in the cab) conditions of the machine 104 associated with it. As a non-limiting example, sensing component 110 may include a camera positioned on the exterior of the machine, for example, to record video and / or images of a portion of the external environment 100 surrounding one of the machines 104; and / or a camera configured to record the internal environment of one of the machines 104, for example, to record video and / or images of an operator of one of the machines 104. In addition to different types of sensors, sensing component 110 may also include multiple sensors of the same type. For example, multiple microphones, cameras, or LiDAR sensors may be positioned at different locations on one of the machines 104, for example, to provide information about different aspects of the external environment 100 surrounding one of the machines 104, and in some cases, up to a 360-degree view surrounding one of the machines 104.

[0023] In some instances, sensing component 114 may be integrated into one or more electronic devices associated with person 106, including but not limited to communication component 112, devices worn by person 106 such as head-mounted devices, wrist-mounted devices, etc., or devices carried by person such as smartphones, radios, tablets, hooks, etc. In at least some instances of this disclosure, person 106 may carry portable electronic devices, such as mobile phones, tablets, etc., or otherwise associate them with them. The electronic devices may include interactive displays, such as touchscreens or other input devices, such as those as part of communication component 112, which person can interact with, and may include one or more sensors, such as positioning sensors, imaging sensors, etc., as part of sensing component 114, to capture information about portions of environment 100.

[0024] In addition to the sensing element 110 associated with machine 104 and the sensing element 114 associated with person 106, additional sensors may be placed in environment 100. For example, also Figure 1 Additional sensors 116a and 116b are shown (collectively, and when referring to additional and / or alternative sensors not associated with one of the machines 104 and / or the person 106, "additional sensor 116" or "sensor 116"). For example, sensor 116 may include one or more sensor patterns, such as motion sensors, cameras, position sensors, microphones, LiDAR sensors, radar sensors, etc., to monitor conditions in environment 100. Figure 1 In some exemplary embodiments, additional sensor 116a may be associated with workstation 118, for example, to sense conditions at or near the workstation, and additional sensor 116b may be positioned close to open-pit mine 102, for example, to sense conditions at or near mine 102. In some instances, sensor 116 may be placed at various locations throughout the work site 100 to provide additional information about aspects of the work site 100. In some embodiments, sensor 116 may be configured to sense conditions at locations that are likely to be important and / or more likely to be performed. Additional sensor 116 may be associated with other elements in the environment 100, including potentially hazardous elements. As a non-limiting example, additional sensor 116 may be associated with high-voltage sources, transformers, pressurized vessels, fuel storage facilities, radiation sources, hazardous material sites, chemical storage facilities, etc.

[0025] As described above, the work site 100 may also include a workstation 118, which may be a room, building, vehicle, etc., where one or more personnel 108 may reside. For example, the workstation 118 may contain one or more offices where administrators, foremen, and / or various personnel may be stationed. In some embodiments, the workstation 118 may act as a hub, for example, acting as a guide or otherwise controlling the location of various machines and personnel. For this purpose, the workstation 118 may include one or more computing systems 120 configured to implement the processes described herein. For example, the computing system 120 may be configured to receive information from one or more of sensing components 110, 114, and / or sensors 116. The computing system 120 may also be configured to receive information from machine 104 and / or personnel 106 and / or transmit information to machines and / or personnel, for example, via communication components 108, 112. Although Figure 1 A computing system 120 associated with workstation 118 is shown, but aspects of the computing system 120 may execute outside of workstation 118, and in some cases, outside of an environment. As a non-limiting example, the computing system 120 may be a remote computing system, a cloud-based computing system, or other computing system.

[0026] As described, several jobs and / or tasks can be performed in environment 100. In some instances, it may be necessary to control machines to perform those tasks remotely, for example, with an operator far from the cab of one of machines 104, and / or for the machines to perform those tasks with a degree of autonomy. However, if environment 100 or a portion thereof is unsuitable for performing tasks, simply instructing a remote operator to control one of machines 104 to perform a task and / or enabling one of machines 104 to perform a task autonomously may be unsafe or otherwise ineffective. As further described herein, in embodiments of this disclosure, computing system 120 may receive data from sensing components 110, 114 and / or sensors 116 to, for example, verify areas of environment 100 based on determining that certain conditions are met for a portion of environment 100, and authorize additional actions in those areas. In a specific example further described herein, computing system 120 may be configured to automatically verify work site 128 as a prerequisite for performing autonomous, semi-autonomous, and / or remotely controlled work at work site 128.

[0027] More in detail, Figure 1 The computing system 120 is schematically illustrated and may include a work site planning component 122, a work site mapping component 124, and a work site verification component 126. In the example described herein, the work site planning component 122 may be a plan for performing one or more tasks or operations within environment 100. Figure 1As shown, the work site plan 122 may include information including work site identification, task information, work site physical conditions, work site environmental conditions, equipment identification, and / or equipment conditions. For example, a work site identification may be a location or area within environment 100 where the work or task will be performed. In an example detailed below, for example, a work site identification may identify the perimeter of work site 128. Task information may include identification of the task to be performed. For example, task information may identify materials to be excavated, hauled, moved, etc., the amount of materials to be excavated, hauled, moved, etc., and / or other aspects of the task. Task information may also include information from the location where the material will be removed (e.g., within the work site) to the location where the material will be moved (e.g., within or away from the work site). Work site physical conditions may include information about the physical parameters of the work site. Such parameters may include ground and / or material conditions required to complete the task, identification of obstacles (including physical obstacles, people, etc.) that would prevent the performance of the task, and / or identification of the physical conditions necessary to perform the task. Work site environmental conditions may include weather-related conditions, which may include conditions that allow for the safe performance of the task and / or conditions that would prohibit its performance. Weather-related conditions are not limited to this; they may include information about temperature, humidity, wind, precipitation, etc. Equipment identification may be the identification of one or more machines that may be required to perform the task. Equipment conditions may include information about the relative health condition of the machine (e.g., fuel, oil, maintenance) or a part of the machine (e.g., tools or implements associated with the machine or subsystem (e.g., the machine's braking system, control system)). The foregoing conditions and attributes of work site plan 122 are merely examples; more, fewer, and / or different conditions and / or information may affect the performance of work in environment 100.

[0028] In one instance, work site plan 122 may be uploaded to or otherwise accessible to computing system 120. In other instances, computing system 120 may facilitate user interaction, for example, through one or more user interfaces, to define parameters associated with work site plan 122, thereby facilitating the generation of work site plan 122. As a non-limiting example, work site plan 122 may include information about one or more tasks to be performed in the environment and one or more conditions necessary for the performance of such tasks. For example, work site plan 122 may be generated by a foreman, site manager, other personnel 106, one or more remote individuals, by a computer process that identifies the need to complete a job in environment 100, or otherwise. As used herein, "conditions" may refer to attributes, states, or facts concerning machines, personnel, and / or the overall work site.

[0029] like Figure 1As shown, the computing system 120 may further include a work site mapping component 124. The work site mapping component 124 may include the ability to determine areas or sub-areas of environment 100, such as as defined or required by the work site plan 122, and to generate or cause the generation of a visual depiction of the environment and / or those areas. For example, such a visual depiction may be displayed on a display device, for example, via a graphical user interface. For example, the work site mapping component 124 may store or access a map of the environment. In at least some instances, the map may include a two-dimensional and / or three-dimensional representation of environment 100. The work site mapping component 124 may determine location information from the work site plan 122, for example, identifying the location information of work site 128 within environment 100. In the examples described herein, a work site may be an area of ​​environment 100 where one or more specific tasks will be performed. The work site mapping component 124 may also include the ability to designate a portion of environment 100 as work site 128 from the work site plan 122, for example. For example, the site mapping component 124 can receive or determine coordinates representing an area within environment 100, such as latitude / longitude, local coordinates, etc., and identify the area in the map data of the environment. As further detailed herein, the site mapping component 124 can also generate a map for display via a graphical user interface. In addition to a graphical representation including all or part of environment 100, the map can also present indications of areas identified by the site plan 122.

[0030] The computing system 120 may also include a work site verification component 126. As further described herein, the work site verification component 126 may include functionality to determine that the work site is ready to perform one or more tasks (e.g., to verify the work site). In the examples described herein, the work site verification component 126 may receive location information from a device and, based on the location information, confirm that a visual inspection of the work site has been performed. For example, by determining that the device is located at the entire perimeter of the work site, at one or more predetermined locations associated with the work site, or at one or more other locations, the work site verification component 126 may determine that a person associated with the device has adequately inspected the work site. In an example, the work site verification component 126 may receive location information from a location sensor, such as a GPS sensor, located on the device, for example, as one of the sensing components 110, 114. Furthermore, the work site verification component 126 may also, or alternatively, receive verification signals from a device associated with a person approaching the work site. For example, a person may interact with the device, for example, via a touchscreen or similar input device, to confirm that the work site has been visually inspected, and such confirmation may cause a signal indicating verification to be sent to and received by the work site verification component 126. In some instances, visual inspection can be crucial for ensuring a work site is ready for the tasks required by the work site plan. For example, visual inspection can ensure the presence or absence of one or more conditions. As used herein, "condition" can refer to the attributes, state, or facts concerning the presence of machines and / or personnel at the work site. Furthermore, conditions can indicate attributes of the work site, such as environmental conditions (e.g., temperature, wind, precipitation), physical conditions (e.g., surface slope, surface composition), and / or objects at the site (e.g., objects that would prevent the performance of a task and / or might be necessary for the performance of a task). In this context, conditions can also refer to necessary conditions that need to be met before a task can be performed at the work site and can constitute an indication of the presence or absence of any object at the work site, the attributes and state of objects present at the work site, or the attributes of personnel present at the work site, etc.

[0031] Reference Figure 1Non-limiting example implementations of this disclosure are described. As described above, environment 100 may include an open-pit mine 102, as well as additional features and areas. For example, one or more tasks at environment 100 may include, for example, removing material from open-pit mine 102 using excavator 104a, and, for example, moving the removed material to a remote location using haul trucks 104b, 104c, 104d. An example of work site plan 122 may include multiple objectives or tasks to be performed at environment 100, such as achieving objectives, such as removing a certain amount of material from open-pit mine 102. In this example, work site plan 122 may include instructions to begin earthmoving operations in a new section of open-pit mine 102. For example, work site plan 122 may identify work site 128 as an area of ​​environment 100 where this new excavation task will be performed. Work site 128 is... Figure 1 The dashed ellipse in the diagram separates the two. In some instances, work site 128 may be identified in work site plan 122 as the next or additional part of mine 102 to be excavated and / or to remove material. Work site plan 122 may also identify excavator 104a as the appropriate machine to perform the removal task at work site 128. For the purposes of this example, it is also assumed that excavator 104a is configured to operate autonomously, for example, without operator interaction. While excavator 104a may be autonomous in this example, in other instances, excavator 104a may be semi-autonomous, for example, performing some functions without operator interaction, or may be remotely operated, for example, by an operator not in the cab associated with excavator 104a and / or located far from the open-pit mine 102.

[0032] In some implementations, the boundaries of work site 128 can be identified even when it is not visible. For example, work site plan 122 may have been developed at the start of a lengthy mining operation, and extending operations into work site 128 may be the next step in the work site plan. In this instance, work site 128 can be assigned even when it is not visible. In some implementations, based on this NLOS assignment, excavator 104a can be instructed to begin excavation or extraction operations at work site 128. However, since the assignment may be performed without visibility and excavator 104a may be unmanned, an intervention event may have occurred, or conditions may exist (or not exist) at work site 128 that hinder the performance of the task.

[0033] Therefore, the techniques described herein can be used to verify the work site 128 before excavator 104a begins the tasks required by work site plan 122. As a non-limiting example, work site 128 may or may not include certain conditions or attributes that may be necessary. In this example, work site plan 122 may require (i) that work site 128 is free of large obstacles, such as large rocks, machinery, trees, or other obstacles that may interfere with the movement and / or operation of excavator 104a; (ii) that the ground in work site 128 is free of large holes, steep slopes, and certain soil conditions such as being too loose or too muddy; (iii) that the slope of the area where excavator 104a will be operated is acceptable for the use of the excavator; and (iv) that work site 128 is free of operators, maintenance personnel, and any other machinery. Of course, these conditions are listed only as examples, as work site plan 122 may include fewer, additional, or alternative conditions that may hinder the performance of work site tasks.

[0034] In this example, the work site mapping component 124 may determine the work site 128, for example, from the work site plan 122, defining the boundaries of the work site 128 within the environment 100. For instance, the work site plan 122 may identify only the work to be completed at the eastern end of the open-pit mine 102, and such work will involve removing a predetermined amount of material from the open-pit mine 102. Based on this information, the work site mapping component 124 may determine the extent and / or size of the work site 128. Alternatively, the work site plan 122 may specify the amount or size by which the open-pit mine 102 will be expanded. The work site mapping component 124 may include functionality to then determine the extent of the work site 128 based on this information. Without limitation, the work site 128 may be sized based on parameters defined by the work site plan 122, using a trial-and-error method, or other means, and the work site mapping component 124 may determine the boundaries of the work site 128. Again, in this example, the boundaries of the work site 128 may be predefined, for example, by personnel such as a foreman, site manager, etc. In at least some instances, the work site mapping component 124 may include functionality to generate a map of the environment 100 for display to personnel and to allow such personnel to input, for example, by interacting with a touchscreen, defining the boundaries of the work site 128. The work site mapping component 124 may also include functionality to determine points along the boundaries of the work site 128, as further described herein. For example, the work site mapping component 124 may identify predetermined locations along the boundaries of the work site 128 and / or within the work site 128 as points or locations from which personnel may perform visual inspections of the work site 128.

[0035] In this example, once the boundaries and limits of work site 128 are established or mapped by work site mapping component 124, work site verification component 126 can receive information from one or more sensors or sources to verify that excavator 104a can perform the tasks listed by work site plan 122. For example, work site verification component 126 can receive information about the conditions of the environment 100 at or near work site 128 from communication components 108, 112 and / or from sensing components 110, 114, 116. As a non-limiting example, personnel 106a, as the person closest to work site 128, can be responsible for performing a visual inspection of work site 128, for example, to ensure that the conditions are those required to perform the task. In this example, personnel 106a may have a portable device including one or both of communication component 112 and / or sensing component 114. For example, sensing component 114 may include a GPS or other location sensor that transmits location information to computing system 120, for example, via communication component 112. For example, upon receiving location information, the work site verification component 126 can determine, for instance, by comparing the location information with one or more locations specified by the work site mapping component 124 to determine that a person has traversed the work site 128. Such locations may include, for example, locations along the perimeter of the work site 128 and / or locations within the work site 128.

[0036] In this example, the location information generated by the sensing component 114 associated with person 106a can be used to determine the physical location of person 106a such that person 106a is in a position to visually inspect work site 128. Work site verification component 126 may also include additional functionality to further confirm the verification of work site 128. For example, work site verification component 126 may present a user interface to person 106a, for example, via a display of a device including communication component 112 and / or sensing component 114. For example, the user interface may be configured to receive input from person 106a confirming the verification of work site 128. For example, a checklist or similar list of necessary conditions, such as conditions (i)–(iv) above, may be provided to person 106a, and person 106a may confirm the presence / absence of such conditions.

[0037] Therefore, the computing system 120 can facilitate the verification of the work site before performing a task or operation. In some instances, the computing system 120 can provide improved and / or safer results by ensuring that the work site 128 is suitable for performing a new excavation task before authorizing the commencement of a task. Of course, the foregoing examples are for illustrative purposes only. Further examples, details, and modifications are also provided herein.

[0038] Figure 2This is a schematic diagram of an example graphical user interface 200. More specifically, Figure 2 A user device 202 with a display 204 is shown. The user device 202 is shown as a handheld device, such as a tablet computer, but in other embodiments, the user device can be any number of electronic devices, such as a desktop computer, laptop computer, smartphone, etc., including a display and facilitating user interaction with the display 204. The display 204 is configured to display or otherwise render a graphical user interface 200. In the example shown, the graphical user interface 200 includes a map 206, which typically includes features of the environment 100. For example, map 206 may show terrain features such as an open-pit mine 102. Map 206 may also include graphical representations, including but not limited to (e.g., machine representation 208 of machine 104) and / or (e.g., building representation 210 of workstation 118). Although Figure 2 Not shown, but map 206 may graphically depict any additional or other features of work site 102. For example, map 206 may include graphical depictions of: personnel at work site (e.g., personnel 106); sensing devices located at work site (e.g., a representation of sensing device 116); other terrain features, such as roads, changes in altitude, bodies of water, etc.; and any other structures or features in or near environment 100.

[0039] For example Figure 2 As shown, the graphical user interface 200 may include a graphical work site depiction 212. The graphical work site depiction 212 may be a depiction of the work site 214 to be verified. For example, the work site 214 may be... Figure 1 The work site 128 is depicted and further discussed above. In the example graphical user interface 200, the graphical work site depiction 212 is a dashed representation of the perimeter of the work site 214, but in alternative embodiments, other representations may be used for the graphical work site depiction 212. As further detailed herein, the graphical work site depiction 212 provides visual indications to personnel associated with the user device 202 of the work site to be verified. The extent of the graphical work site depiction 212 may be defined by the work site plan 120 or determined, for example, by the work site mapping component 122 at least in part based on the work site plan 120. Furthermore, although the graphical work site depiction 212 is shown as a quadrilateral, other shapes may be used, including but not limited to those used for display. Figure 1The work site 128 is depicted as an elliptical shape. Similarly, in this example, instead of a continuous perimeter representation, the graphical work site depiction 212 can be visualized as a number of points, markers, or locations. In some instances, point locations may be generally aligned along the perimeter of the work site 214, but other instances may include points outside the perimeter. As a non-limiting example, one or more points within the perimeter of the work site 214 may be presented, either outside the perimeter or as an alternative to it. For example, larger work sites may require more than traversing the perimeter to complete a visual inspection. Figure 1 In the examples used, the point can be located at the base of the wall of the open-pit mine 102, or at other locations within the mine 102. Other exemplary embodiments will be understood by those skilled in the art who will benefit from this disclosure.

[0040] As also shown in the figure, the graphical user interface 200 may include a graphical current location depiction 216. In the example shown, the graphical current location depiction 216 is indicated by two concentric circles and the text "You are here". In other embodiments, the graphical current location depiction 216 may be presented on the graphical user interface 200 in other ways, including using graphics, text, animation, etc. As will be understood, the graphical current location depiction 216 may indicate the location of the device 202 on a map 206. The map 206 may be... Figure 1 The representation of environment 100 shown in this example may be associated with, for example, person 106a, user device 202. In other examples, the graphical current location depiction 216 may show the location of person, for example, person 106a, based, for example, on location data obtained from sensors worn by, carried by, or otherwise associated with person 106a.

[0041] The graphical user interface 200 can also provide the user with instructions and context for verifying the work site 214. In this example, the graphical user interface 200 may include: an instruction for the overall task to be performed, such as the text "Verify work site #1"; a map 206 of the environment 100, wherein a graphical depiction 212 of the work site explicitly identifies the work site 214; and a graphical depiction 216 of the user's current location relative to the work site 214. The graphical user interface 200 may also include features that allow the user to interact with the graphical user interface 200. For example, and as... Figure 2As shown, the graphical user interface 200 may include user interface controls 218, 220, and 222. In embodiments of this disclosure, user interface controls 218, 220, and 222 may be buttons or other designated features on the display 204 of a device 202 with which a user can interact. For example, a user may interact with user interface controls 218, 220, and 222 by: touching the display 204 near the user interface control to be selected; selecting one of the user interface controls 218, 220, and 222, or others, using a stylus or other selection device associated with the user device 202. Figure 2 In one example, the user can select user interface control 218 to obtain additional information about the requirements for verifying the work site. For instance, selecting user interface control 218 can cause user device 202 to render an updated graphical user interface with additional information and enable extra functionality related to work site verification. Figure 3 An example of an updated graphical user interface that can be presented in response to the selection of user interface control 218 is shown, which will be described in more detail below.

[0042] Users may choose or otherwise interact with the second user interface control 220 to input comments. For example, selecting the second user interface control 220 may cause a dialog box or an updated graphical user interface to be displayed so that the user can input comments about the work site 214. For example, selecting the user interface control 220 may present the user with an interface that allows text input, such as a keyboard or other similar input mechanism. In other instances, comments may be provided via audio input, via speech-to-text technology, or other methods. In these instances, comments entered by the user may be transmitted to a remote computing system, such as those described above. Figure 1 The described computing system 120. As a non-limiting example, the annotations may include records of attributes or conditions observed at or related to the work site 214.

[0043] Users can interact with third-user interface control 222 to indicate that work site 214 is unsuitable for performing one or more tasks as instructed by the work site plan. As described herein, techniques can be used to determine whether a work site, such as work site 214, is suitable for performing one or more machine-based tasks via autonomous, semi-autonomous, or remote control. Third-user interface control 222 allows users to easily indicate that work site 214 is unsuitable for the desired task. Although not shown, when third-user interface control 222 is selected, graphical user interface 200 may be updated, where dialog boxes, etc., require user confirmation of an "invalid" selection. Additionally or alternatively, selection of third-user interface control 222 may prompt the user to enter comments, including indications of why work site 214 is invalid and / or cannot be verified.

[0044] For example Figure 2 As shown, the graphical user interface 200 may include one or more status indicators associated with the verification of the work site 214. Figure 2 One such option shown is Inspection Status Indicator Option 224, which, when selected, displays a numerical representation of the percentage of worksite inspection completed, along with its graph. Inspection Status Graph 224 is for illustrative purposes only, and additional or different graphs may be used to indicate factors associated with the inspection status. In this example, verification of worksite 214 may require personnel to traverse the perimeter of the worksite or otherwise travel to certain predefined coordinates of worksite 128 to facilitate a complete visual inspection of worksite 128. As further detailed herein, Inspection Status Graph 224 can indicate the progress associated with such travel relative to worksite 214. See details. Figure 2 If the user (or user device 202) indicates that the work site 214 is far away, the inspection status graphic 224 indicates that no work site has been inspected, for example, because the user (or user device) has not yet moved to one or more locations where a visual inspection can be performed.

[0045] As described above, users can access additional information about work site 214 and / or requirements for verifying work site 214, for example, by selecting user interface elements 218, 220, 222. Figure 3 An updated graphical user interface 300 is shown, which can be displayed on the monitor 204 in response to a user selecting a first user interface element 218. In other instances, the graphical user interface 300 can be accessed in other ways, for example, by selecting one of the text “Verify Work Site #1”, a representation of work site 214, a graphical depiction of the work site 212, and / or some other or additional user interface elements. In the illustrated example, and as described above, work site 214 can correspond to the combination of the above. Figure 1 The work site 128 is described, and the purpose of the verification is to enable the excavator 104a to perform one or more tasks in the open-pit mine 102 without the presence of a site or machine operator.

[0046] like Figure 3As shown, the graphical user interface 300 is adapted to an extended instruction section 302. The instruction section 302 provides additional information regarding the verification. For example, the instruction section 302 may provide the user with a list of conditions, objectives, or tasks that must be checked, completed, or otherwise performed to verify the work site against the proposed tasks. In the example shown, the instructions require checking the perimeter, checking the ground composition, identifying workers at the site, and identifying objects at the work site. These listed tasks may generally correspond to or be at least partially based on conditions (i)–(iv) from the examples discussed above, but the tasks described in the instruction section 302 are merely examples. More, fewer, and / or different tasks or instructions may be provided and may be at least partially indicated by the work site plan 122. Figure 3 In this context, the instruction section 320 is an extension of the first user interface control 218 and the second user interface control 220, and the check status graphic 224 can be removed to accommodate the extension. In other instances, more, fewer, or different graphics can be removed from the graphical user interface 200 to accommodate the presentation of the instruction section 302.

[0047] The instructions provided in the extended instruction section 302 can provide a user, such as personnel 106a, with visual indications of each condition required for the task identified by the work site plan 122 to be performed. In some embodiments, the instructions may include tasks to be performed manually by personnel 106a, for example, and / or automatically via sensors using computer-based automation techniques. For example, the last entry in the extended instruction section 302 is “Identify Objects”. To accomplish this task, personnel 106a may be responsible for determining whether there are any objects within the work site 214 that would prevent the excavator from performing the required task. If such objects are present, for example, visually identified by personnel 106a during a visual inspection of the site, personnel 106a may interact with a third user interface control 222, for example, to disable the work site. In other embodiments, personnel 106a may need to capture images of the work site, and such images may be processed, for example, using feature recognition techniques to determine and identify objects and personnel present at the work site. In this embodiment, the user may be notified to cancel the desired task due to the presence of objects and personnel at the work site.

[0048] According to the technology of this disclosure, computer-implemented techniques can be used to confirm that a person has actually inspected the work site 214. For example, the first item listed in the extended instruction section 302 is “Check the perimeter.” Such an instruction can instruct the person 106a or another user associated with the device 202 that he / she must traverse the perimeter of the work site, which is indicated by the work site representation 212. Because the graphical user interfaces 200, 300 display the person’s current location via the current location indicator 216 and the perimeter of the work site 214 via the work site depiction 212, the graphical user interfaces 200, 300 provide the user with a context of the current location, destination location, and directions for navigation from the source location to the destination location. Similarly, this implementation can present… Figure 3 A map of work site 214 (not shown) can help inspectors navigate within work site 214. Figure 4 An example graphical user interface 400 shows an update indicating that personnel have moved, for example, to perform a visual inspection of work site 214. In this example, the current location indicator 216 has been updated to show that device 202 is now located near the lower left corner of the quadrilateral defining work site 214. Also shown, a portion of the perimeter of work site 214 is now indicated by a solid line. In this example, personnel 106a, along with device 202, may have moved from... Figure 2 and 3 The user travels to a first position 402, located at the upper left corner of the adjacent work site 214, using the current location indicator 216 as the reference. From the first position 402, the user may have traversed along line 404 to a second position 406, which is closer to the upper right portion of the work site 214. From the second position 406, the user may have traversed along line 408 to a third position 410, and from the third position 410, the user may have traveled along line 412 to the current position indicated by the current location indicator 216'.

[0049] In addition to including lines 404, 408, and 412, for example as indicators of the location traveled by a person to inspect work site 214, the graphical user interface 400 may also include updates to the inspection status graph 224. For example, and as shown, the inspection status graph 224 may be updated to include, numerically, visually, or otherwise, the amount of work site 214 traversed by the person. In this example, the numerical depiction and the graphical representation below it show that approximately two-thirds, or 66%, of work site 214 has been inspected. For example, the verification percentage may directly correspond to the amount of perimeter that the person has traversed. In this example, sensors on the electronic device 202 may be used to determine the path taken by the person. As a non-limiting example, the electronic device 202 may include one or more position sensors, such as GPS sensors, and the position information may be used to track the movement of the device 202 and thus the movement of the person. In the example shown, the person's path is limited to indicating the location along the perimeter of work site 214. In this example, the device 202 may not completely track the work site perimeter indication 212. In other words, a person may not be traveling along the exact contour depicted by the perimeter indicator 212 with device 202, but the positioning information generated by the positioning sensors on electronic device 202 can indicate that device 202 is within a certain threshold distance from the perimeter, and thus confirm that the person is at or near the perimeter. In some instances, the positioning data generated by electronic device 202 can be generated at a fixed interval, and routes can be inferred between the positions generated in continuous readouts. In other instances, the site mapping component 124 can determine several discrete points along the perimeter of the site 214, and the positioning data can be compared to those locations. For example, points can be associated with each of locations 402, 406, and 410, and additional points can be determined between these locations. As a non-limiting example, a line 404 can be generated in response to confirmation that the user device is at three predetermined points between locations 402 and 406. When using discrete points, as previously described, the status graph 224 can be checked based on the number of points the user device 202 is already at. In some instances, the location of user device 202 within a certain radius of a point is sufficient to confirm that device 202 is located at the corresponding point.

[0050] Figure 5Another graphical user interface 500 is shown. Compared to graphical user interface 400, graphical user interface 500 indicates, as represented by the current location indicator 216”, that the person associated with device 202 has now moved to a location adjacent to first location 402. For example, electronic device 202 may have moved substantially along line 502 from location 504 to first location 402. In this example, since electronic device 202 has now traversed the entire perimeter of work site 214, inspection status graphics 224 has been updated to show that 100% of the work site has been inspected. Additionally, graphical user interface 500 may be updated to include shaded area 506, which indicates that the entire work site 214 has been traversed. In some embodiments, confirming that the person is in a series of locations associated with work site 214, such as the perimeter around work site 214, may be sufficient to confirm that work site 214 has been verified for performing the task associated with work site plan 122. However, as further described herein, additional steps may be taken, such as verifying one or more conditions associated with work site 214, to complete the verification.

[0051] Also in Figure 5 In the example graphical user interface 500 shown, a user interface control 508 may be presented. In the illustrated example, when the user interface control 508 is selected, a verification process for the work site 114 can begin. In this example, the user can interact with the user control 508, for example, by selecting the user control to perform additional tasks associated with the verification work site. In at least some instances, the selection of the verification user control 508 can be displayed on the user device 202 as shown in the image. Figure 6 An exemplary graphical user interface 600 is shown.

[0052] like Figure 6As shown, the graphical user interface 600 includes several features of the graphical user interface 500, but also includes a confirmation parameter user interface 602. Specifically, in this example, the confirmation parameter user interface 602 includes a list 604 of parameters, conditions, or tasks required to verify work site 214. Furthermore, individual conditions or tasks in list 604 may have associated optional user controls 606. In this example, the optional user control 606 can switch between a first state indicating that the condition is met or the task is completed and a second state indicating that the condition is not met or the task is not yet completed. In this example, the items in list 604 generally correspond to instructions presented in the graphical user interface 300. Specifically, and as detailed above, multiple conditions may need to be present (or absent) to verify the work site for performing tasks, for example, from work site plan 122. These conditions may be presented to the user via the graphical user interface 300, and the graphical user interface 600 may be configured to receive confirmation that these conditions have been met, for example, via the optional user control 606. In the example shown, when the user completes navigation around work site 214, such as when completing an inspection as described above, the optional user control 606 associated with the “Perimeter Traversal” entry in list 604 can be automatically “selected” (or otherwise indicated as complete). Also in this example, the site is confirmed to be free of workers and obstacles. For example, “No workers on site” and “No objects on site” could have already been manually confirmed by the user, for example, while traversing the perimeter of work site 214. In this instance, the user associated with user device 202 may have already visually confirmed that no workers and objects are present at the site. List 604 may also indicate to the user that they must still confirm that the ground composition is sufficient to perform the task.

[0053] The graphical user interface 600 also includes a "Confirm Verification" user control 608, which is grayed out in the illustrated example. For example, the user can select the Confirm Verification user control 608 to complete the verification process only when each item in list 604 indicates completion, for example via optional user control 606. Visually, the Confirm Verification user control 608 may simply be grayed out until all tasks in list 604 indicate completion. In some cases, selecting the Confirm Verification user control 608 can cause user device 202 to generate a verification signal and, for example, transmit the verification signal to computing system 120 to confirm the verification of the work site. This signal can indicate to computing system 120 that work site 214 is ready to perform the tasks indicated by work site plan 122.

[0054] Based on examples described in conjunction with graphical user interfaces 200, 300, 400, 500, and 600, this disclosure describes a system that can verify a work site 214 before performing one or more tasks at a work site 214, for example, according to a work site plan 122. Verification of the site may include, for example, generating a map 206 of the work site 214 in a larger environment 100 using a work site mapping component 124, and displaying the map 206 on a user device 202. Sensor data, such as location data, can then be used to determine that the user device 202 has been transmitted around the work site 214, indicating that a person has visually inspected the work site 214. In addition to inferring that a person has inspected the work site 214, other sensor data can also be used to confirm the inspection. As a non-limiting example, data generated by one or more sensors 110, 114, and 118 can be received by a computing system 120, allowing the work site verification component 126 to determine aspects to be verified. In some cases, personnel 106a may use a camera associated with user device 202 to capture image data, such as images and / or video, of the work site 214. The work site verification component 126 may include functionality to determine whether conditions at the site are met based on this image data. For example, the work site verification component 126 may include image processing capabilities, such as feature recognition, to identify people or objects at the work site. Similarly, sensors near the work site, such as additional sensor 116b, may be used to provide image data, environmental data, or other data regarding conditions at or near the work site 214.

[0055] Figure 7 This is a block diagram illustrating an exemplary system 700 for work site verification according to an example described herein. In at least one example, system 700 may include one or more computing devices 702, which in some embodiments may be or may include computing system 120. Computing device 702 may include one or more processors 704 and memory 706 communicatively coupled to processor 704. In the example shown, memory 706 of computing device 702 stores one or more maps 708, one or more work site plans 710 (which may be or may include work site plan 122), work site mapping system 712 (which may be or may include work site mapping component 124), work site verification system 714 (which may be or may include work site verification component 126), and graphical user interface (GUI) generation system 716. Although these systems and components are shown and will be described separately below, the functionality of the various systems may be attributed differently from that discussed. Furthermore, fewer or more systems and components may be used to perform the various functions described herein. Additionally, although for illustrative purposes... Figure 7The map is depicted as residing in memory 706, but it is envisioned that map 708, work site plan 710, work site mapping system 712, work site verification system 714 and / or GUI generation system 716 may additionally or alternatively be accessible by computing device 702 (e.g., stored on memory remote from computing device 702, or otherwise accessible by memory remote from computing device 702).

[0056] In some instances, map 708 may include a map of the environment (e.g., environment 100) in which the operation is to be performed. The map may be any number of data structures modeled in two or three dimensions, capable of providing information about the environment, such as, but not limited to, topology (e.g., intersections), streets, mountains, roads, terrain, and the overall environment. Map 708 may also include data structures capable of providing information about buildings, including but not limited to floor plans, blueprints, layouts, equipment models and equipment locations, and / or other building-centric information. As described above, while map 708 may be stored in memory 706 of computing device 702, in other embodiments, map 708 may be accessed by computing device 702, for example, via network 722.

[0057] In at least one instance, work site plan 710 (which may be the same as or similar to work site plan 122) may include information about tasks, jobs, or functions to be performed. For example, work site plan 710 may include information about the type of job or task to be performed, the location of the job or task, and optionally one or more conditions for performing the job or task. In at least some instances, work site plan 710 may include jobs or tasks to be performed by remotely operated, semi-autonomous, or fully autonomous machines. In these instances, information about one or more conditions for performing the job or task may be based on requirements for performing remote, semi-autonomous, or fully autonomous tasks. Other instances of work site plan 710 are provided herein.

[0058] In some cases, the work site mapping system 712 (which may be or may include functionality associated with the work site mapping component 124) may include the ability to determine the coordinates of a work site to be verified. In one example, the work site mapping system 712 may receive information about the extent of a work site (e.g., work site 214) where a task specified by the work site plan 710 is to be performed. For example, the work site plan 710 may include the coordinates of work site 214, and the work site mapping system 712 may identify the coordinates relative to information from map 708. In other examples, the work site mapping system 712 may use other methods to determine the extent or coordinates of work site 214. For example, the work site mapping system 712 may include the ability to determine the perimeter or area within which the task will be performed, for example, based on the type of work or task, the extent of environment 100, the type of machine available to perform the task, or other information. As a non-limiting example, the work site mapping system 712 may map an area based on the volume of earthwork to be moved, the material to be extracted, etc. As further described herein, the site mapping system 712 can generate a map 206 that is displayed on a monitor 204 of the user device 202.

[0059] In some instances, the job site verification system 714 (which may be or may include functionality associated with the job site verification component 126) may include functionality to determine the suitability of the job site for performing a task. In the above examples, the job site verification system 714 may receive sensor data from one or more sources and determine and verify the job site based on the sensor data. As described above... Figure 3-5 Specifically, the work site verification system 714 can receive location information associated with the personnel responsible for performing visual inspections of the work site. Using this location information, the work site verification system 714 can determine that the work site has been visually inspected. For example, the work site verification system 714 can determine that the personnel have completely traversed the areas associated with the work site, such as the work site perimeter, one or more locations within the work site, etc.

[0060] In addition to using location data to determine that a work site has been visually inspected, the work site verification system 714 can also determine that additional conditions are met. For example, the work site verification system 714 may include the ability to perform image analysis on images of the work site, such as to determine the presence of objects, people, or other conditions at the work site. Also in an example, the work site verification system may receive weather-related information, for example, from sensors near the work site to determine weather-related conditions. Also in an example, the work site verification system 714 may receive information associated with user input at a device associated with the work site.

[0061] In some instances, the graphical user interface (GUI) generation system 716 may include generating one or more interactive interfaces (e.g., GUIs 200, 300, 400, 500, 600) for functionality to be presented on a display. In some instances, the GUI generation system may receive information from a map 708, a work site plan 710, a work site mapping system 712, a work site verification system 714, and / or additional data 718 to generate the GUI. As a non-limiting example, the GUI generation system 716 may use map 708 and data generated by the work site mapping system 712 to generate a map 206 showing the work site 214 in environment 100 and its current location relative to the user device displaying map 206. Furthermore, the GUI generation system 716 may receive information about work site conditions that must be met to perform certain tasks. For example, such information may be displayed as instructions, as in GUI 300, and / or as checklists or similar lists, as in GUI 600. Similarly, in this example, the graphical user interface generation system can receive information about the location of objects in the environment, for example, to configure the GUI to include graphical representations of such objects. Also as described above, the GUI generated by the GUI generation system 716 can provide interactive elements, such as user interface elements that allow users to interact with the GUI. Figure 6 In the example GUI 600, list 604 can be determined based on work site plan 710, and user interface control 606 can be generated to allow the user to confirm that items on list 604 have been completed. GUI generation system 716 can also access templates, logic, APIs, plugins, and / or other software, firmware, or data necessary for rendering the GUI.

[0062] The computing device 702 may also include a communication connection 720, which enables communication between the computing device 702 and other local or remote devices. For example, the communication connection 720 may facilitate communication with other computing devices, such as computing device 724, machine 104, communication devices 108, 112, sensing devices 110, 114, 116, and / or one or more networks, such as network 722. For example, the communication connection 720 may communicate via frequencies defined by the IEEE 802.11 standard, such as... Wi-Fi-based communication can be achieved using short-range wireless frequencies, other radio transmissions, or any suitable wired or wireless communication protocol that enables the corresponding computing device to connect to other computing devices.

[0063] In some implementations, computing device 702 can transmit information such as sensor data to computing device 724 via network 722. Computing device 724 can receive sensor data directly from computing device 702 and / or from sensing devices 110, 114, 116, and can perform some functions belonging to computing device 702. In at least one instance, computing device 724 may include processor 726 and memory 728 communicatively coupled to processor 726. In the illustrated example, memory 728 of computing device 724 may store field verification component 714. Field verification component 714 may correspond to the field verification system 712 described above.

[0064] The processor 704 of computing device 702 and the processor 726 of computing device 724 can be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, processors 704 and 726 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other means or part of means that processes electronic data to transform said electronic data into other electronic data that can be stored in registers and / or memory. In some instances, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors, provided they are configured to implement coded instructions.

[0065] Memory 706 and memory 728 are examples of non-transitory computer-readable media. Memory 706, 728 may store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods described herein and the functions belonging to various systems. In various embodiments, the memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programming, and physical components; those shown in the accompanying drawings are merely examples relevant to the discussion herein.

[0066] For example Figure 7As shown, computing device 702 can also communicate with machine 104, communication devices 108, 112, sensing elements 110, 114, and / or sensor 116. Although computing device 702 is illustrated as communicating with such machines and devices via network 722, in other embodiments, computing device 702 may communicate directly with machines and / or devices. As a non-limiting example, in some embodiments, some aspects and / or functions belonging to computing device 702 may be performed on communication device 108, such as on user device 202. Similarly, machine 104, communication devices 108, 112, sensing elements 110, 114, and / or sensor 116 may communicate directly with computing device 724. Figure 7 As also shown, sensing devices (e.g., sensing components 110, 114, and / or sensor 116) may include one or more sensor systems 732. In at least one instance, sensor system 732 may include positioning sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, UV, IR, intensity, depth sensors, etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), LIDAR sensors, RADAR sensors, ultrasonic transducers, sonar sensors, etc. Sensor system 732 may include multiple instances of each of these or other types of sensors. For example, each machine 104 may have multiple cameras positioned at various locations surrounding the exterior and / or interior of the machine. Sensor system 732 may provide input to computing device 702 and / or computing device 724, for example, via communication system 734. Additionally and / or alternatively, the sensor system 732 may transmit sensor data to the computing device 702 and / or the computing device 724 at a specific frequency via the communication system 734 and / or the network 722 at near real-time after a predetermined period of time.

[0067] It should be noted that, although Figure 7 While illustrated as a distributed system, in alternative instances, components of computing device 702 may be associated with computing device 724, and / or components of computing device 724 may be associated with computing device 702. Furthermore, although various systems and components are shown as discrete systems, these illustrations are merely examples, and more or fewer discrete systems may perform the various functions described herein.

[0068] Figure 8 and Figure 9Flowcharts depicting exemplary processes 800, 900 of this disclosure for verifying a work site are shown. Exemplary methods 800, 900 are illustrated as a set of steps in a logic flowchart, representing operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the steps represent computer-executable instructions stored in memory. For example, when processor 704 executes such instructions, such instructions can cause controller processor 704, various components of computing device 702, computing device 724, machine 104, and / or communication devices 108, 112 to perform the operations. Such computer-executable instructions may include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular abstract data type. The order in which operations are described is not intended to be construed as limiting, and any number of described steps can be combined in any order and / or in parallel to implement the process. For discussion purposes and unless otherwise stated, reference is made to environment 100, GUIs 200, 300, 400, 500, 600, computing system 700, and / or Figure 1-7 The other items shown describe processes 800 and 900.

[0069] As mentioned, process 800 is in Figure 8 The diagram illustrates, and generally describes, a method for verifying a work site. Process 800 includes receiving a work site plan at 802. For example, computing device 702 may receive data indicating a task or job to be performed in environment 100, for example via one or more signals. For example, the work site plan may be work site plans 122, 710, and may include boundary information about the area to be performed, such as the extent of the work site, one or more conditions required to verify the work site for the job, and / or any other information about the task or job.

[0070] Process 800 may also include identifying the work site associated with the work site plan at 804. For example, computing device 702 may determine the scope of the work site, such as work site 214, from the work site plan received at 802, for example, by using a work site mapping system 712. In some instances, the scope, such as size, dimensions, location, etc., may be included in the work site plan, and the work site mapping system 712 may determine the location of work site 214 on one or more maps 708. In other instances, computing device 702 may determine the location of the work site based on the work site plan in other ways. As a non-limiting example, supplementary data 718 may include information about the correspondence between work sites, such as size, shape, required conditions, and tasks to be performed. Thus, computing device 702 may determine details about the work site based on the work site plan.

[0071] Process 800 may also include generating a user interface at 806 that visualizes the work site and verifies relevant information. For example, the techniques described herein can generate a graphical user interface 200 that includes a map 206 showing the work site 214 and the environment. The graphical user interface 200 also includes displaying the current location of the device using the GUI, for example, oriented to personnel associated with the device within the environment and relative to the work site 214. As detailed above, the graphical user interface 200 may also include user interface elements 214, 216, 218 through which users (e.g., administrators, etc.) can obtain additional information about the verification task to be performed after viewing the map 206 on the user device 200. Again, in this example, the GUI may include an inspection status indicator indicating a percentage, quantity, or other metric associated with inspecting or verifying the work site.

[0072] Process 800 may also include receiving positioning information from a position sensor at 808. For example, the techniques described herein may include receiving information about the positioning of a user device 202 on which a graphical user interface generated at 806 is displayed. In other instances, the positioning information may be associated with different sensing devices and personnel responsible for inspecting and / or verifying the work site. As a non-limiting example, the position sensor may be associated with a wearable device, such as a watch, armband, etc. worn by a person, a vehicle that can be driven by a person, an electronic device associated with a person such as a mobile phone, or some other sensing device.

[0073] Process 800 may also include determining at 810 whether the location information confirms an inspection of the work site. For example, work site verification component 714 may determine whether personnel are already in a position to visually inspect the entire work site. In at least some instances, work site verification component 714 may compare the location information with one or more locations associated with the work site. Such locations may include the perimeter of the work site, one or more points on or near the perimeter, one or more points within the work site, or other locations that provide personnel with an opportunity and / or advantageous position to inspect the work site.

[0074] If the location information at step 810 is insufficient to confirm an inspection of the work site, process 800 returns to step 806. For example, if personnel have not yet traversed the perimeter of the work site or have not yet appeared at one or more predetermined locations associated with the work site, an updated graphical user interface can be generated, such as displaying the personnel's updated location based on recently received location information, and / or updating the verification status. An example of an updated GUI can be found in... Figure 4 It is displayed in the middle.

[0075] Alternatively, if the location information at 810 confirms the inspection of the work site, process 800 may include updating the user interface at 812 to indicate completion of the visual inspection. Figure 5 In the example GUI 500 shown, work site 214 can be highlighted, shaded, or otherwise displayed differently, and / or a check status indicator can show that 100% of the work site has been updated. In some instances, other indicators can be displayed on the graphical user interface to indicate that the work site has been checked.

[0076] In some implementations, process 800 may also include receiving confirmation at 814 that condition parameters are met. For example, and as detailed herein, conditions determined by visual inspection in addition to and / or via visual inspection may be necessary for performing certain tasks at the work site. As a non-limiting example, some tasks may only be performed under certain weather conditions, in the absence of people or objects in the work area, when the ground at the work site meets certain parameters including slope, soil composition, etc., and / or when certain other criteria are met. In example graphical user interface 600, a user may confirm multiple parameters by interacting with element 606 associated with list 604. In other instances, instead of or in addition to user confirmation of conditions or parameters, sensor data, such as sensor data from one of the sensing systems 732 associated with one or more of sensing devices 110, 114, 116, may generate data about the work site, and such data may be used to confirm additional parameters necessary for verifying the work site. In at least some instances, additional information may be user-selected confirmation of verification, for example, a user's affirmative action that each condition required for verification is met.

[0077] Process 800 may also include transmitting a work site verification signal at 816. For example, after confirming that the work site has been inspected and all required conditions are met, the work site verification system 714 may transmit a verification signal to the computing system 120. The computing system 120 may then initiate the task at the work site, for example, by transmitting instructions to a remote user to begin a task based on a remotely controlled machine and / or by authorizing an autonomous or semi-autonomous machine to perform the task.

[0078] Process 900 in Figure 9 The diagram illustrates and generally describes methods for verifying, for example, that condition parameters are met before verifying the work site. In some instances, process 900 may be associated with operation 814 discussed above, but operation 814 may include more, fewer, or different functions than those described in process 900, and process 900 may be performed in addition to being associated with process 800.

[0079] Process 900 includes receiving a work site plan at 902. The functions associated with 902 may be substantially the same as those associated with 802 discussed above. In an example, processes 800 and 900 may be executed in parallel, and operations 802 and 902 may be the same. For example, computing device 702 may receive data indicating a task or job to be performed in environment 100, for example, via one or more signals. For example, the work site plan may be work site plans 122, 710, and may include boundary information about the area to be performed, such as the extent of the work site, one or more conditions required for the work site to be verified for the job, and / or any other information about the task or job. Process 900 may also include identifying one or more condition parameters in the work site plan at 904. As detailed herein, the work site plan may list several condition parameters that must be met at the work site before commencing a job or task (e.g., a job or task to be performed autonomously, semi-autonomously, or remotely). Examples of condition parameters are further detailed herein and may include, but are not limited to, physical conditions, environmental conditions, equipment information, equipment conditions, etc.

[0080] Process 900 may also include receiving information associated with individual condition parameters at 906. For example, the site verification component 714 may receive sensor data from one or more sensing systems 732 to determine information related to the condition parameters. As a non-limiting example, the site verification component 714 may receive environmental data, such as weather data, from additional sensors 116 located near the site. In other instances, the site verification component 714 may receive image data from one or more sensor modes, including but not limited to receiving image data from the user device 202. In other instances, 906 may include receiving signals associated with user interaction, such as one of GUIs 300, 400, 500, 600 indicating that she has, for example, confirmed information associated with a visual inspection. As a non-limiting example, the user may use element 606 to confirm that a condition is met.

[0081] Process 900 may also include determining the condition parameters at 908. For example, based on sensor data received at 906, the site verification component 714 may determine the condition parameters. For example, the site verification component 714 may determine that the temperature and humidity at the work site are within acceptable ranges. In other instances, the site verification component 714 may confirm that the machine can perform the task by receiving sensor data from one of the machines 104. In some instances, operation 908 may confirm the condition parameters by receiving information from a user responsible for inspecting (e.g., by traversing) the work site. As mentioned above, such a user may interact with a GUI, such as GUI 500, to confirm that she has determined the condition parameters to be met, for example, through visual inspection.

[0082] Procedure 900 may also include determining at 910 whether all conditional parameters are met. If all conditional parameters are not met, procedure 900 may return to operation 906. In some cases, procedure 900 may include highlighting or additionally alerting the user to those parameters that have not yet been met. Also in this instance, procedure 900 may include requesting, for example, additional information for the parameters that have not yet been met.

[0083] Alternatively, if it is determined at 910 that all condition parameters are satisfied, then process 900 may include generating a condition parameter satisfaction signal at 912 and transmitting the condition parameter satisfaction signal at 914. In some instances, a condition parameter satisfaction signal may be required to generate and transmit a verification signal, as in operation 816 discussed above. In other words, the verification signal may require all condition parameters to be satisfied before being transmitted, thus authorizing the execution of the tasks or operations listed in the work site plan received at 902.

[0084] Industrial applicability

[0085] This disclosure provides systems and methods for verifying a work site 128, for example, for performing jobs or tasks by one or more machines 104 at the work site 128. Such systems and methods can be used to more efficiently and safely coordinate the activities of machines 104 during operation at the work site 128, for example, to allow non-line-of-sight or remote control and / or autonomous or semi-autonomous machine control of machines 104. For example, such systems and methods can enable a computing system 120 to determine that the properties of the work site 128 correspond to the prerequisites for performing the desired task and maintaining accident-free operation at the work site. Therefore, the computing system 120 can confirm that the desired operation can be performed before performing such operation, and thus maximize efficiency at the work site 128. Additionally, such systems and methods can be used to more accurately manage the operation of machines 104 at the work site 128, thereby reducing operating costs.

[0086] As mentioned above Figure 1-8As indicated, an example process for verifying a work site may include receiving a work site plan 122 and determining information regarding the location and extent of the work site 128 based on the work site plan 122. In some instances, the work site plan 122 may also include information regarding additional conditions that must be verified at the work site 128. Based on the location and extent of the work site 128, the work site mapping component 124 can identify the work site 128 in map data. The map data may be displayed to a user, for example, via a graphical user interface on a portable electronic device 202. Furthermore, location data associated with the user, such as from a location sensor on the portable electronic device 202, may be used to display the map data and the user's current location relative to a representation 214 of the work site 128.

[0087] The process may also include, for example, receiving additional location data from a positioning sensor on the portable electronic device 202 to determine whether a user associated with device 202 is already at the location of the visual inspection work site 128. For example, the work site verification component 126 may compare the location data with one or more locations associated with the work site 128. In at least some instances, these locations may be one or more locations along a perimeter, such as a corner of the perimeter, one or more locations within the perimeter, such as the center of the work site, and / or other locations. As a non-limiting example, the techniques described herein may enable the computing system 120 to control one of the machines 104 to begin performing a task when the work site is verified, and to prevent the machine from operating in the absence of such verification. In other embodiments, the computing system 120 may transmit messages, etc., to a remote operator to begin a task or operation at the now-verified work site 128.

[0088] Although various aspects of this disclosure have been specifically shown and described with reference to the foregoing embodiments, those skilled in the art will understand that various additional embodiments can be contemplated through modifications to the disclosed machines, systems, and methods without departing from the spirit and scope of the disclosure. Such embodiments should be understood to fall within the scope of this disclosure as defined by the claims and any equivalents.

Claims

1. A method for verifying a work site, comprising: A work site plan (122) is received at a computing device (202), the work site plan including at least one task to be performed by a machine at a work site (128) and the boundary of the work site (128); Receive sensor data generated by at least one sensor associated with the second computing device from a second computing device located adjacent to the work site. A user interface (200) is displayed on the display (204) of the computing device (202), the user interface including a map (206) including a representation (212) of the boundaries of the work site (128) and a representation (216) of the location of the computing device (202); Receive positioning information including one or more updated positioning (216') of the computing device (202); Based on the positioning information and the sensor data, a verification signal indicating that the work site (128) has been verified is generated; as well as The verification signal is transmitted from the computing device (202). In response to the receipt of the verification signal, instructions are transmitted to the remote user to begin a task based on the remotely controlled machine and / or instructions are transmitted to the machine to begin at least one task at the work site. The machine is autonomous or semi-autonomous, and the at least one task is performed autonomously or semi-autonomously by the machine.

2. The method for verifying a work site according to claim 1, further comprising: The work site has been verified based at least in part on one or more updated locations (216') corresponding to one or more predetermined locations (402, 406, 408) associated with the work site (128).

3. The method for verifying a work site according to claim 2, wherein one or more predetermined locations associated with the work site include at least one of a location adjacent to the boundary of the work site or a location within the boundary of the work site.

4. The method for verifying a work site according to claim 2, further comprising: An updated user interface is generated, at least in part, based on one or more updated locations corresponding to one or more predetermined locations associated with the work site, the updated user interface including at least one user interface element; as well as The computing device receives user input instructing the user to interact with at least one user interface element. The generation of the verification signal is also based on the user input.

5. The method for verifying a work site according to claim 1, wherein the work site plan further includes a set of work site condition parameters, and the method for verifying a work site further includes: Verification of the at least one work site condition parameter is received through user interaction with at least one of the user interface or the computing device. The generation of the verification signal is also based on the verification of the at least one working site condition parameter.

6. The method for verifying a work site according to claim 5, wherein the set of work site condition parameters includes at least one of the following: ground composition, slope associated with the work site, weather conditions associated with the work site, identification of one or more objects at the work site, or identification of one or more persons at the work site.

7. The method for verifying a work site according to claim 1, wherein the sensor data includes at least one of image data, location data, weather data, or altitude data.

8. The method for verifying a work site according to claim 7, wherein the sensor data is generated by sensors associated with a machine located adjacent to the work site.

9. A system for verifying a work site, comprising: Computing device (202); One or more sensors (110, 114, 116). One or more processors (704, 726); and The memory (706, 728) stores processor-executable instructions, which, when executed by the one or more processors (704, 726), configure the system to perform actions including: Receive a work site plan (122), the work site plan (122) including: the boundary (212) of the work site (214) where the machine (104) will perform a task; at least one work site condition parameter (604) associated with the work site (214); and information about the task to be performed by at least one machine (104) at the work site (214); Sensor data associated with the work site (214) is received from one or more sensors (110, 114, 116), the sensor data including at least one of positioning data or image data. Based at least in part on the sensor data, a first signal indicating that the work site (214) has been visually inspected is generated; Receive condition parameter data indicating that at least one work site condition parameter (604) is satisfied; A graphical user interface (200) is generated for display on the computing device (202), the graphical user interface including a graphical representation of the work site (206) and user interface elements (506). Based on the first signal, the condition parameter data, and user input indicating user interaction with the user interface element (506), a verification signal indicating that the work site (214) has been verified is generated; and Transmit the verification signal, In response to the receipt of the verification signal, instructions are transmitted to the remote user to begin a task based on the remotely controlled machine and / or instructions are transmitted to the machine to begin at least one task at the work site. The machine is autonomous or semi-autonomous, and the at least one task is performed autonomously or semi-autonomously by the machine.

10. The system for verifying a work site according to claim 9, wherein the interaction between the user and the user interface element corresponds to at least one of user verification of the at least one work site condition parameter or user verification of the visual inspection.

11. The system for verifying a work site according to claim 9, wherein the sensor data includes the positioning data, and the action further includes: The location data is determined to correspond to multiple locations, the multiple locations including at least one of one or more first locations adjacent to the boundary or one or more second locations inside the boundary.

12. The system for verifying a work site according to claim 11, wherein the one or more sensors include a positioning sensor associated with the computing device.

13. The system for verifying a work site according to claim 9, wherein the one or more sensors include image sensors configured to generate the image data as one or more images of the work site, the action further comprising: Analyze the one or more images; as well as The one or more images are determined to indicate a valid work site.

14. The system for verifying a work site according to claim 13, further comprising: Verification of the at least one work site condition parameter is received through user interaction with at least one of the user interface or the computing device. The generation of the verification signal is also based on the verification of the at least one working site condition parameter.

Images