Augmented reality assistance for movable platforms

Abstract

A described example relates to systems and methods to assist placing a vehicle to perform one or more tasks with a tool carried by the vehicle. The systems and methods can provide graphical guidance to facilitate placing a vehicle at a location to enable a tool, which is carried by the vehicle, to perform a task at a target location. The systems and methods can provide one or more outputs that implement augmented reality.

Description

AUGMENTED REALITY ASSISTANCE FOR MOVABLE PLATFORMSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority to U.S. Provisional App. No. 63 / 682633, filed August 13, 2024, which is incorporated herein by reference in its entirety.FIELD

[0002] This description relates to systems and methods to assist operators of movable platforms, such as by providing an augmented reality experience.BACKGROUND

[0003] An immediate need was identified using various systems, in which a vehicle-mounted tool, such as a robot, performs operations on surfaces (typically, painting of road pavement). For example, an operator drives the vehicle to a work site, then selects an operation to be performed (e.g., painting part of a crosswalk). Additionally, the operator chooses where to perform the operation (e.g., repainting an existing, worn crosswalk, registering the new paint with the prior marking). In the system, the location of the operation (e.g., position and orientation of the crosswalk) can be determined either by pre-specified coordinates (e.g. desired GPS latitude / longitude and heading of the crosswalk) or by interactive placement of an overlay on live video (e.g., moving a virtual template of a crosswalk via a touchscreen to align with a view of an existing, partially-worn crosswalk). Having determined what operation is to be performed and where the operation is to be performed, the system is configured for performing the work.

[0004] However, it can be difficult for the system (e.g., robotically controlled tool) to reach the specified work coordinates, depending on where the vehicle has been placed. In some circumstances, a suitable location to enable the tool to perform a given task can be counter-intuitive. This can lead to frustrating trial-and-error in repositioning the vehicle to enable performing the specified task.SUMMARY

[0005] This description relates to systems and methods to assist operators of movable platforms, such as by providing an augmented reality experience.

[0006] One example provides a system that includes non-transitory memory that stores data and instructions and a processor. The memory can store task data that defines a given task to be performedby a tool at a target location, in which the tool is carried by a moveable platform. The data can also include pose data representative of a current location and orientation of the moveable platform and virtual parking space data represents at least one desired pose of the moveable platform from which the tool is capable to perform the given task at the target location. The processor is configured to access the memory and execute the instructions to cause the processor provide a guidance output based on the pose data and the virtual parking space data, whereby the guidance output facilitates placement of the moveable platform at the desired pose for performing the given task.

[0007] In another example, a method can include storing, in the non-transitory memory, pose data representative of a current location and orientation of a moveable platform, in which a tool is carried by the moveable platform. The method can also include storing, in the non-transitory memory, virtual parking space data representative of at least one desired pose of a moveable platform from which the tool is capable to perform a given task at a target location. The method can also include providing, by a processor, a guidance output based on the pose data and the virtual parking space data, whereby the guidance output facilitates placement of the moveable platform at the desired pose for performing the given task.

[0008] In yet another example, a method includes providing substantially real-time guidance on a display based on a current pose of a moveable platform carrying an automated or semi-automated tool and a predetermined task to be performed by the tool at a predetermined target location, in which the guidance provides instructions on the display for placing the moveable platform at a corresponding pose to enable the tool to perform the predetermined task at the predetermined target location.

[0009] In another example, one or more non-transitory computer-readable media includes instructions which, when executed by a processor, are configured to cause the processor to perform any of the methods.BRIEF DESCRIPTION OF DRAWINGS

[0010] FIG. 1 is a block diagram showing an example of a system.

[0011] FIGS. 2a and 2b depict example graphical outputs showing a first task and a virtual parking spot for performing the task.

[0012] FIGS. 3a, 3b, 3c, 3d, 3e, 3f, 3g, and 3h depict example display outputs as a vehicle moves towards the virtual parking spot for performing a second task.

[0013] FIGS. 4a, 4b, 4c, and 4d depict example display outputs as a vehicle moves forward towards a virtual parking spot for performing a third task.

[0014] FIGS. 5a, 5b, and 5c depict example display outputs as a vehicle moves towards a virtual parking spot for performing a fourth task.

[0015] FIG. 6 depicts another display output showing an example graphical visualization of a site coordination event associated with performing a fifth task.

[0016] FIG. 7 depicts another display output showing an example graphical visualization of another site coordination event associated with performing a sixth task.DETAILED DESCRIPTION

[0017] This disclosure describes systems and methods to assist placing a moveable platform (e.g., a vehicle, cart, or the like) for performing one or more tasks with a tool (e.g., robot) carried by the moveable platform through augmenting reality.

[0018] The following describes one example implementation, although it should be recognized that there can be many other applications and variations based on this disclosure. For example, while the disclosure describes systems and methods to help placing a movable platform for applying markings by an automated or semi- automated tool at a prescribed location (e.g., on or in a surface), the approach described herein is applicable to placing other types of moveable platforms to enable performing other functions by a tool (e.g., robot having a corresponding joint space) that is carried by the moveable platform. As one example, surgical equipment can be mounted to a movable platform, supplemented with augmented reality for the surgeon, who may be guided in how to locate a pose (position and orientation) for the movable platform, such that a specific trajectory can be achieved within reach of the system. The movable platform does not need to be self-propelled. It might be on wheels, and manually pushed into position by an individual or another device. In some examples described herein, the location of one or more supporting platforms (e.g., safety apparatuses) can be coordinated with the position of the movable platform for performing one or more tasks.

[0019] The systems and methods have various technical advantages over existing approaches. For example, any existing approaches tend to rely on coarse GNSS or odometry alone, which suffer from drift and multipath errors, or on fiducial markers and preinstalled infrastructure making them inflexible and time-consuming to deploy. Pure visual or map-based overlays often lack the spatial accuracy required for a robot arm's reachability and leave operators guessing at complex multi degree-of-freedom alignment. Moreover, most systems do not continuously fuse multiple localization sources in real time or assess whether the tool's joint-space trajectories can actually reach the worksite before operation, resulting in trial- and-error maneuvering, misplaced overlays, and under-utilized automation.

[0020] FIG. 1 depicts an example of a system 100 showing an operating environment that can be used to implement systems and methods described herein. The system 100 includes a computer 102 having non-transitory memory 104 that stores data 106 and instractions 108. The memory 104 may also store an operating system that controls or allocates resources of the system. The memory 104 represents a non-transitory machine-readable memory (or other medium), such as RAM, a solid state drive, a hard disk drive or a combination thereof. A processor 110 can access the memory 104 and execute the instructions 108 to process the data 106 and signals and / or perform computational operations based on the instructions 108 stored in the memory 104. The instructions 108 cause the processor core 110 to execute operations. The processor 110 can be a variety of processors including multiple single and multicore processors, co-processors, and other multiple single and multicore processor and co-processor architectures.

[0021] In some examples, the computer 102 can be carried by (e.g., integrated into, mounted to, or otherwise placed in) a movable platform that carries a tool 112, which is being positioned for performing one or more tasks. In other examples, the computer 102 and / or portions of the system 100 can be separate from the movable platform, such as a portable computing apparatus (e.g., a tablet computer, a cellular telephone, a notebook computer, etc.) and / or implemented in a distributed computing environment. Also, or alternatively, the computer 102 (e.g., memory 104 and / or processor 110) can be implemented remotely from the movable platform, such as a web service and / or a cloud computing architecture.

[0022] For sake of consistency and ease of explanation, in the following examples the movable platform is sometimes referred to as a vehicle, such as a car or track, which carries the tool 112. However, it is to be understood that the movable platform that carries the tool can be implemented according to other transportation modalities, such as a cart, trailer, wagon, and the like, which can include a motor to provide for self-propelled motion and / or be manually moved by external forces (e.g., be positioned manually). The tool 112 that is carried by the movable platform can vary over a wide range of technologies, which can depend on the type of movable platform, the use environment, and / or tasks to be performed. Also, while a single tool is described in many examples herein, in other examples, the movable platform can carry any number and / or type of tools according to application requirements. The tool can also be configurable (automatically and / or manually) with parts to enable the given task to be performed at a respective target location.

[0023] The data 106 stored in the memory 104 can include task data 114, platform pose data 116, virtual parking spot data 118, and image data 120. The task data 114 can define attributes of a given task to be performed by the tool at a respective target location, such as described herein. The task data 114can also include task instructions to control the tool for performing the given task. The task data can include attributes and / or task instructions for any number of tasks to be performed at respective target locations. For example, the target location can be defined as a point, an area (a two-dimensional space), or a volume (e.g., a three-dimensional space). The task data 114 can be generated in response to a user input instruction, which can be entered at a user input device 122 (e.g., via a touchscreen, a joystick, mouse, etc.) through a graphical user interface (GUI) 124 that defines or updates one or more attributes of the given task. As an example, the attributes of the task data 114 include a pose (e.g., spatial position and an orientation) for performing the task at the target location. The task data 114 can also define a size (e.g., scope or range) of the task to be performed by the tool 112 at the target location. The task data 114 can be programmed in advance of performing the task. Also, or alternatively, the task data can be programmed in response to a user input instruction that is entered prior to or during performance of a given task at a respective target location in response to a user input instruction, which can be entered through the GUI 124. Thus, the task data 114 can be static or be dynamic, such as being updated during performance of the task (e.g., responsive to a correction applied by an operator).

[0024] The system 100 can further include a localization system 126, which can be carried by (e.g., integrated into, mounted to, or otherwise placed on or in) the movable platform that also carries the tool 112. The localization can be part of the computer 102 or coupled to the computer through a physical or wireless link. The localization system 126 can provide the platform pose data 116. The platform pose data 116 can be representative of a current location and orientation of the movable platform that carries the tool. The platform pose data 116 can include three (or more) degrees of freedom for the platform (e.g., a vehicle or other platform), such as two-dimensional spatial coordinates (e.g., x and y coordinates) and an orientation of the movable platform relative to a surface on which the platform moves. For example, the localization system 126 can include one or more of wheel odometry, laser / ultrasonic odometry, global position system (GPS), global navigation satellite system (GNSS), inertial navigation system (INS), and visual odometry (VO). Thus, the platform pose data can be continuous (e.g., in real time) track the pose of the movable platform (e.g., vehicle) to which the tool is attached.

[0025] The system 100 can also include one or more cameras 128 and / or one or more other sensors 130. The camera 128 can be fixed relative to the movable platform. The camera 128 can be configured to provide image data 120, such as including a real-time image. For example, the camera 128 can be a ground-facing camera attached to the movable platform adjacent the tool 112 and configured with a field of view that includes a zone of reachability for the tool, which defines a region where the tool can perform a given task (e.g., within the range or scope of tool’s joint space). In some examples, such as where thetask includes a treatment (e.g., a marking) to be applied to a surface, the GUI 124 can be overlaid on the image, to provide an interactive visualization of the surface treatment that is to be applied based on the acquired image data 120 and the task data 114. An operator can adjust (e.g., correct) the size, position, and / or orientation of the surface treatment (e.g., marking) through the GUI 124 responsive to user input entered at the user input device 122 to implement such adjustment(s), which further can cause the processor 110 to update the task data 114 based on the adjusted position and / or orientation of the surface treatment.

[0026] Examples of the other sensors 130 include LIDAR, radar, ground penetrating radar, sonar, ultrasonic sensors, wheel encoders, accelerometers, odometry sensors, wheel angle sensors, color camera as well as other sensing modalities that can detect such features that may be detectable along the path of travel. The system 100 can also include a sensor interface (not shown) that can be coupled to the other sensors 130 to receive and perform initial sensor processing (e.g., filtering, analog-to-digital conversion, and the like) to provide corresponding sensor data to the computer 102. As described herein, the computer 102 includes instructions 108 that cause the processor 110 to process sensor data, including from the localization system 126, camera 128, as well as other sensors 130 to perform the functions described herein.

[0027] The virtual parking space data 118 represents at least one desired pose of the mobile platform from which the tool 112 is capable of performing the given task at the target location, such as based on the task data 114. The processor 110 can provide output data that includes guidance output (e.g., a graphical visualization, which can be interactive) based on the task data 114, the platform pose data 116, and the virtual parking space data 118. As described herein, the guidance output facilitates placement of the vehicle at the desired pose for performing the given task. In some examples, the movable platform is a vehicle that can be driven by an operator located in the vehicle or by an operator located remotely from the vehicle based on the guidance that is provided to such operator. In other examples, the movable platform can be moved by application of an external force (e.g., not driven but manually positioned by human and / or other motive forces).

[0028] The system 100 can also include one or more display devices 132 and a tool controller 134, which is configured to control the tool 112. The display device 132 can include one or more screens, heads up display, AR goggles or the like. The computer 102 can control the tool controller 134 and / or tool 112 to perform one or more tasks based on the task data 114 and the platform pose data 118. For example, the processor 110 can execute instructions to control the tool controller 134 and / or tool toperform a given task at a target location as described in U.S. Patent No. 11,550,333, which is incorporated herein by reference in its entirety.

[0029] As mentioned, the task data 114 can include task instructions to control the tool 1 12 for performing the given task. The tool 112 has a zone of reachability relative to the movable platform (e.g., a vehicle or other platform). The tool 112 can be a robotically controlled tool, in which an arm thereof has a joint space (e.g., including a number of arm segments and joints) and the task instructions are programmed to control motion of the robot through a joint space trajectory that has been computed to perform the given task. The instructions 108 further can be configured to compute a joint space trajectory for executing one or more sub-process plans to control the corresponding joint space of the robotic arm to implement the respective surface treatment functions (e.g., surface preparation, application and / or curing functions) at the target location. The joint space trajectory can be stored in the memory 104 as part of the task data 114, for example, which can be communicated to the tool controller to control the tool 112 for performing the given task or series of tasks at the target location(s). Some examples of tasks that can be performed by the tool 112 can include one or more of painting, grinding, digging, drilling, cleaning, cutting, sealing, cutting and / or milling. Other tasks can also be performed depending on the type and configuration of the tool 112, such as described herein.

[0030] By way of further example, a robotically controlled tool (e.g., including a tool controller and one or more tools), which can implement the systems and methods herein, is described in the aboveincorporated U.S. Patent No. 11,550,333. Another example of a robotically controlled tool (e.g., the tool 112) and associated movable platform, which can be configured to implement the system 100 and methods described herein, is the ELECTRA™ system commercially available from RoadPrintz of Cleveland, Ohio. For example, the ELECTRA™ system includes an industrial grade IP67 rated robot arm, having a spray head, mounted on a vehicle chassis as well as includes computing and control electronics.

[0031] As a further example, the instructions 108 include a guidance generator 136 programmed to generate the guidance output, which can be presented on the display device 132. The guidance generator 136 can provide the guidance output based on execution of one or more spatial transform function 138 and a joint space evaluator function 140. The guidance output can include one or more graphical elements that can be overlaid with respect to a representation (e.g., an image based on the image data 120) of the real world, such as including the vehicle, the zone of reachability of the tool, and / or an image acquired by a camera 128 carried by the movable platform (e.g., vehicle). For example, a first graphical element (also referred to herein as a blue box or indicator) is representative ofthe vehicle pose based on the platform pose data 116. A second graphical element (also referred to herein as a green box or indicator) can be representative of the desired pose for the vehicle so that the tool can perform the given task based on the virtual parking space data 118. In examples where the given task includes applying a marking on a surface, the guidance output can further include a third graphical element representative of the marking to be applied on the surface.

[0032] The guidance generator 136 can provide the guidance output to the display device 132, such that the display can include any one or more of the first graphical output, the second graphical output, and the third graphical output. In some examples the camera 128 is fixed relative to the vehicle and configured to provide an image (e.g., a real-time image) that includes at least a portion of the (or the entire) zone of reachability for the tool 112, and at least one of the first graphical element, the second graphical element, and the third graphical element are overlaid (superimposed) on the image. In this way, the guidance on the display device can include a real-time representation visualizing, graphically, where the vehicle is relative to the virtual parking space as well as the target location for the given task. The guidance further can include text and / or graphical (e.g., color-coded) features describing the direction and / or distance to move the vehicle to the virtual parking spot.

[0033] In some examples, the processor 110 can execute instructions, namely, the joint space evaluator 140, programmed to evaluate one or more candidate joint space trajectories for the tool 112 based on the current vehicle pose (defined by the platform pose data 116) to determine if the given task is reachable by the tool as to be successfully completed. The processor 110 further can execute tool controls 142 to enable the tool controller 134 and / or the tool 112 to perform the given task responsive to the joint space evaluator 140 determining that the given task is reachable based on the evaluation. The tool controls 142 further can disable the tool 1 12 from performing the given task responsive to the joint space evaluator 140 determining that the given task is not reachable based on the evaluation.

[0034] In some examples, the tool controls 142 can control the GUI’s ability to activate the tool 112 and / or tool controller 134 to perform the given task based on the joint space evaluation implemented by the joint space evaluator 140. Also, or alternatively, the tool controls 142 can selectively enable or disable the ability of the GUI 124 to activate the tool 112 and / or tool controller 134 based on the platform pose data 116 and the virtual parking space data 118 indicating whether or not the current task is currently within or outside the zone of reachability of the tool. As described herein, the virtual parking space can be determined based on the task data 114 and the vehicle pose data 116. In some examples, the tool controls 142 can include an override mechanism (e.g., responsive to a user input) to enable an operator to activate the tool to perform the given task regardless of whether thegiven task can be successfully completed at the desired target location for a current pose of the movable platform.

[0035] For example, the GUI 124 includes a GUI element (e.g., a radio button) that can be activated in response to a user input entered at the user input device 122 to command the tool 112 (through the tool controller 134) to perform a given task. The GUI element can be enabled (turned on) so a user can activate the button, responsive to a user input, to cause task instructions to be executed based on the task data and thereby cause the tool to perform the given task at the target location. Alternatively, when the given task cannot be successfully completed at the target location, the tool 112 and / or tool controller 134 can be disabled. When the tool 112 and / or tool controller 134 is disabled, the GUI element of the GUI 124 can be disabled (e.g., by graying out or otherwise rendering the GUI element non-responsive or inactive) to thereby prevent the tool 112 and / or tool controller 134 from being activated in response to a user input. This way the system 100 can implement a safety check to avoid inadvertently applying a marking (or performing another task) if the entire task cannot be completed successfully at the intended target location as defined by the task data 114. As a further example, additionally, or alternatively, the system 100 can include a physical button or switch that is operative to activate the tool 112 and / or tool controller 134 and the physical button or switch can be selectively enabled and disabled as described herein.

[0036] In some examples, the system 100 can include plan data that defines a set of tasks (e.g., an ordered sequence of discrete tasks) that are to be performed by the tool at respective target locations based on task data for each respective task in the set of tasks. The plan further can be encoded as a sequence of virtual parking spots that includes target parking spot data for each respective task in the plan data. For example, the virtual parking spot data includes one or more predefined vehicle poses associated with each task in the set of tasks from which the tool can successfully complete each respective task in the set of tasks. The instructions executable by the processor 110 can include a user interface element programmed to select a next task in the set of tasks responsive to a user input. For example, the guidance generator 136 can provide output guidance for the selected next task based on the task data 114 and the virtual parking space data 118 for the next task, and based on the platform pose data 116. As described herein, the task data 114 can be adjusted, dynamically, by the operator, which can cause corresponding changes to the virtual parking space data 118. Any adjustments to the task data 114, the virtual parking space data 118, and / or the platform pose data 116 further can cause the processor to execute the guidance generator 136, spatial transforms 138 and / or the joint space evaluator 140 based on the current data 106 so that resulting guidance is updated accordingly.

[0037] The instructions 108 can be further programmed to lock the target location for a respective task with respect to the current pose of the vehicle (or other movable platform) in response to a corresponding user input instruction entered through the GUI 124 via the user input device 122. For example, in response to a user input instruction locking the target location, a graphical representation of the task to be performed at the defined target location (e.g., a marking to be applied on a surface) is fixed with respect to the defined target location. The defined target location has been registered (e.g., by applying one or more corresponding transforms) in an image acquired by the camera 128 carried by the vehicle. Additionally, the target pose of each subsequent task in a sequence of tasks (e.g., defined by plan data) further can be locked with respect to the locked predefined target location. Prior to or after locking a target location, a user can adjust (e.g., correct) the target pose and / or size for the respective task (or a subsequent task) in response to a user input adjustment instruction entered via the GUI 124 (e.g., using a mouse or touchscreen). For example, as described herein, the user input adjustment instruction can move a graphical representation of the marking (or a representation of another task to be performed) relative to the image that includes the target location on the GUI 124. Thus, a user can ensure that each task is performed more accurately at an intended location in case of errors being introduced by the vehicle pose data, the original target location and / or other sources of error. Each subsequent task can be registered spatially with respect to the adjusted location and a corresponding joint space trajectory can be computed based on the adjusted location and vehicle pose, such as described herein.

[0038] As a further example, in a pre-processing phase for performing a given task, the spatial transform function 138 establishes a spatial transformtaskTpark between the pose of a reference frame of the vehicle, defining the virtual parking spot, with respect to a reference frame of the task. The reference frame of the vehicle for the given task can be stored in the virtual parking spot data 118 and the reference frame of the given task can be stored in the task data 114. The spatial transform function 138 can establish a respective spatial transformtaskTpark for each task that is to be performed by the tool 112, as stored in the task data 114. In some examples, each task (e.g., how to paint a part of a crosswalk) can be assigned an associated preferred vehicle pose with respect to the task (e.g., recommended vehicle location relative to the crosswalk). In other examples, a number of different feasible vehicle poses can be defined for each respective task. It is thus pre-established that if the vehicle is positioned at the recommended pose with respect to the task, the task will be reachable by the tool 112. In this pre-processing phase, the spatial transform function 138 can express the suggested (e.g., preferred) vehicle pose with respect to a spatial frame associated with the task, regardless of where the task is to be performed in the world. Asdescribed herein, the suggested / preferred vehicle pose(s) for performing each respective task is referred to herein as virtual a “parking spot,” which is stored in the virtual parking spot data. The spatial transform function 138 provides the transformtaskTpark to express a recommended parking spot (e.g., a vehicle reference frame) with respect to the task of interest. The transformtaskTpark may be precomputed and defined as constant for a respective task. The process for this computation can be fully automated or may be human interactive, for example, in response to user inputs. One way to establish a viable virtual parking spot is to move the target location for the task with respect to the vehicle and simulate if the task can be performed at the proposed vehicle pose. For example, the joint space evaluator 140 can determine if the joint space trajectory for performing the task (e.g., applying a selected marking on a surface) can complete the task at one or more candidate target locations when the vehicle is located at the virtual parking spot Through such evaluation (which may simply be trial-and-error), one can establish a preferred vehicle pose (virtual parking spot) with respect to the task frame,taskTpark.

[0039] Additionally, the task is associated with physical coordinates. For example, a desired crosswalk can be positioned relative to Earth, e.g., by pre- specifying geospatial coordinates (including orientation), or in coordinates relative to some local reference frame, or by interactively moving a virtual template to align with a target pose in a camera view. Associating a task with physical coordinates is equivalent to defining a transform for the coordinate frame associated with the task with respect to a coordinate frame associated with ground. (Ground coordinates may be absolute GPS, or they may be coordinates relative to some convenient frame tied to earth). The pose of the task with respect to ground isgroundTtask. If the task coordinates (with respect to ground) are established interactively (e.g., by aligning a symbol overlay on live video of the ground), then the operator may “lock” the aligned symbol to ground (e.g. using a control on a GUI), which establishesgrouildTtask at run time.

[0040] Further, the instructions 108 compute (and continuously update) the platform pose data 116 of the vehicle with respect to ground (e.g., using GPS and odometry provided by the localization system 126 and / or other sensors 130). A transform for the pose of the vehicle with respect to ground can be represented herein as:gIoundTVeh, which can be stored as part of the platform pose data 116 and / or included as part of the spatial transform function 138.

[0041] Having establishedtaskTpark andgroundTtask for a given task, it is desired that the vehicle (or other movable platform) be moved so that it is in the virtual parking spot, which is equivalent to attempting to establish:Equivalently, the objective for the driver is to achievegroundTveh =groundTpark.

[0042] This can be equivalent to saying the vehicle frame and parking-spot frame arc coincident (or spatially converging), namely, the vehicle is parked (e.g., precisely) in the virtual parking spot that had been determined by the joint space evaluator 140 for the respective task (e.g., defined by task data 114).

[0043] While spatial coordinates of a desired pose (virtual parking spot) may be known (computed, as described above), it nonetheless can be difficult to navigate the vehicle to the desired pose. To assist the operator in doing so, the guidance generator 136 is configured to determine and display a representation of the virtual parking spot relative to the pose of the vehicle. For example, the guidance generator 136 displays vehicle pose (e.g., based on current vehicle pose data 116) with respect to a desired virtual parking spot,parkTveh. As a further example, the guidance generator 136 can generate a visual display on the display device 132 that includes a graphical representation of the vehicle (e.g., abstracted to a simple icon or a faithful rendering of a top view of the vehicle or other platform) and a graphical representation the desired virtual parking spot.

[0044] As a further example, the display device 132 has its own coordinate system. Accordingly, the spatial transform functions 138 include additional transformsdlsplayTveh (a transform expressing the vehicle with respect to the display) anddlsplayTpark (a transform expressing the virtual parking spot with respect to the display). The additional transformsdlsplayTveh anddlsplayTpark can be updated (e.g., in real time or near’ real time) based on the platform pose data 116, which varies as the vehicle moves (i.e., as ground^ changes) based on location data from the localization system 126 and / or from other sensors 130. It is then the operator’s objective to align the graphical object representing the vehicle with another graphical object representing the virtual parking spot, such as described herein.

[0045] As further examples, the operator can be a driver in the vehicle or may be operating the vehicle remotely. The guidance generator 136 can generate an augmented reality (AR) visualization that is presented on the display device 132 to provide driver assistance (e.g., guidance) to position the vehicle favorably for performing the task. In the AR visualization presented on the display device 132, there are two possible display scenarios: the target pose (virtual parking spot) is displayed as moving with respect to the (stationary) vehicle, or the display shows the vehicle moving on a stationary visualization, such as a map that includes a stationary target pose. In either scenario, the driver is assisted in guiding the vehicle to the highlighted, preferred pose (position and orientation).

[0046] In some examples, the guidance generator 136 can provide additional guidance that can be presented in the AR visualization on the display device 132 such as to suggest a specific strategy to navigate to the target pose (virtual parking spot). This might include a graphical representation of a path(or trajectory) to follow from a current pose to the target destination pose. Also, or alternatively, the additional guidance can include an indication of distance (e.g., a graphical output, text, and / or audio) from the current pose to the target destination pose, in which the distance is updated (in real time) based on the current platform pose data and the virtual parking spot data. Directional indicators further can be included. Also, or alternatively, the additional guidance can include a display of steering angles to be executed en-route to the target destination pose. Also, or alternatively, the additional guidance can include forward / reverse motions, as necessary. More complex forms of guidance may be included, such as providing instructions or generating automatic controls to navigate a trailer to a target pose, or how to perform the equivalent of parallel parking from a current pose to the target destination pose. Additionally, in some examples, the virtual parking spot for a given task can be automatically updated based on adjustments implemented (at any time) by the operator (e.g., in response to a user input instruction specifying changes to the position, orientation, and / or size) for the current task or a future task being performed.

[0047] Advantageously, in some examples, the transform function 138 can compute (e.g., in real time) the transform between the vehicle’s current pose (based on the pose data 116) and the virtual parking spot (based on virtual parking spot data 118), such that the graphical outputs and graphical cues that are generated based on current and desired vehicle poses. Also, in some examples, the joint space evaluator 140 further evaluates the tool's reachability by checking joint-space trajectories based on the target pose for performing the task. For example, a user can interact with the GUI 124 through the user input device 122 to select or adjust one or more tasks, including the target pose, in response to which the transforms can be recomputed and guidance generated to facilitate correctly positioning the vehicle and its associated tool. The tool controller 134 can then execute the operation to move the tool through the joint space trajectories for the current task after determining that the vehicle has the correct position and orientation. Unlike existing AR-based parking or alignment aids, this systems and methods can integrate spatial transforms with on-vehicle tool reachability checks and selective tool activation to ensure precise, task-specific vehicle positioning for mounted-tool operations.

[0048] As a further example, the instructions 108 of the system 100 can include a site event coordination function 144 to provide location information (e.g., virtual parking space) for one or more auxiliary units during execution of one or more tasks described herein. The number and placement of virtual parking spots for auxiliary units can depend on the environment where the given task is being performed. For the example of tasks to apply surface markings to a road, considerations for placing auxiliary units can include the location of the target site for the task, the virtual parking spot for themovable platform (e.g., vehicle) carrying the tool, the size of the intersection, the location of the target site(s), the number of lanes, etc. Examples of auxiliary units for performing roadwork tasks can include truck mounted attenuators, towable attenuators, sleds, vehicles, barricades, barrels, cones, signage, and the like. Other types of auxiliary units (e.g., tools, vehicles, etc.) can depend on the particular task being performed, its location, and surrounding environment.

[0049] The location where each auxiliary unit is placed (its virtual parking spot) can be determined for each task manually (e.g., by visual inspection of the task environment), by automated methods, or a combination of manual and automated methods. The spatial coordinates for the virtual parking spot for each auxiliary unit can be stored in the memory 104 as part of the virtual parking spot data 118 for each respective task that is to be performed. Then as a given task is to be performed, the site coordinator function 144 can communicate the virtual parking spots for the movable platform (e.g., the tool platform or vehicle) and each auxiliary unit to each operator for placing respective units for a given task or series of tasks.

[0050] As an example, the site coordinator function 144 (or other instructions) can generate a graphical representation of the environment, the movable platform. The graphical representation can be displayed on the display devices (e.g., on display devices mounted in vehicles, on tablet computers, smart phones, etc.) in a coordinated manner for each stage of the task being performed. Also, or alternatively, a printout of the virtual parking spaces can be provided for operators associated with each auxiliary unit for each stage of the task. In some examples, the system can be configured to communicate the graphical representation (e.g., as an html document or other form) that is received by respective computing devices to provide a coordinated real-time display for the virtual parking spot for each auxiliary unit. The communication between the computing devices and the central computer can be bidirectional, such that a location (e.g., GPS coordinates and / or other location information) for each auxiliary unit can be communicated (e.g., as feedback) in real time to help ensure that each auxiliary unit has been placed at its respective virtual parking lot before beginning to perform the current (or next task). The virtual parking spaces for the auxiliary units helps to increase safety associated with performing tasks at desired target locations.

[0051] In view of the foregoing, the AR display can be visualized on the display device(s) 132 to provide guidance so the driver (or other operator) can more easily succeed in placing the vehicle in a pose from which the desired task can be performed at the desired target location. Such guidance might be particularly helpful if the desired location is not visually apparent, but the task nonetheless has known coordinates, such as underground pipes or wires, or pre-determined target locations for construction operations, such as placing fence posts, utility poles, foundations, etc. Additionally, in some situations, the desired location for the movable platform may actually be counterintuitive, such as requiring moving to one side when the surface treatment is to be applied through a joint space trajectory that extends to the opposite side.

[0052] In some examples, the display device can be mounted in a vehicle at a location within view of the driver (or other operator). In other implementations, the display device could be a “heads-up” display in which the graphics are projected onto the windshield or some transparent display. It is desirable that the driver can simultaneously see the physical surroundings and the virtual parking spot. An alternative view is to display the vehicle on a map, where the map can include recommended virtual parking spots overlaid on satellite images. With this augmented-reality driver assistance, operations involving human-driven mobile equipment can be more productive and effective.

[0053] In view of the foregoing description, the systems and methods described herein will be better appreciated with reference to AR displays of FIGS. 2a through 7. The AR displays of FIGS. 2a through 7 can be generated by the system 100 of FIG. 1 and output to one or more display devices 132. Accordingly, the description of FIGS. 2a through 7 also refers to FIG. 1. In the examples of FIGS. 2a through 7, the AR visualizations are associated with performing surface treatment tasks in a road environment, in which the tool 112 is described as a controllable system (e.g., robot having a defined joint space or other apparatus having a defined reachability) that is operative to position a dispensing apparatus (e.g., painthead, sealant or filler application tool, light source, heat source, etc.), a surface modification apparatus (e.g., grinder, saw, drill or the like), or to a combination thereof for performing a defined task relative to a target location. For example, the tool 112 can be configured to dispense any of a variety of materials on a surface at the target location, such as including paints, coatings, fillers, sealant and the like. In other examples, the tool can be implemented to perform other functions, which can vary according to application requirements and use environment, such as described herein.

[0054] FIGS. 2a and 2b show examples of AR displays 200 and 220 that can be generated (e.g., by guidance generator 136) for an example task is to apply a surface treatment (e.g., paint) a section of a large crosswalk. The AR display 200 provides an interactive display that includes GUI elements (e.g.,GUI 124) through which a user can interact through a user input device (e.g., a touchscreen or mouse) to input user instructions as described herein. The AR display 220 can provide a heads-up display that is generated based on the same data (e.g., data 106 as the AR display 200.

[0055] The interactive AR display 200 of FIG. 2a includes graphical features overlaid (e.g., superimposed) on a representation or image of the environment, shown at 202, which includes the road where the surface treatment is being applied. The AR display 200 includes a graphical representation of a virtual parking spot for the vehicle, shown as a white bounding box 203. The AR display 200 also includes a graphical element 204 representative of the vehicle pose (e.g., based on the platform pose data 116) and another graphical element 206 representative of the desired pose for the vehicle so the tool can perform the given task (e.g., based on the virtual parking space data 118) at the target location 208 on the surface. While the graphical elements 204 and 206 are shown as rectangles / squares, other shapes or types of fiducials (e.g., circles, crosses / cross-hairs, stars, etc.) and different colors (e.g., blue and green; black and white; etc.) can be used to visualize the graphical elements representing the respective poses, and selection of fiducial types and / or sizes can be configured in response to a user input, for example. The AR display 200 can also include a visualization for a reachable region 210 that provides an estimated zone of reachability for the tool (e.g., based on the joint space of the tool and the vehicle pose).

[0056] The other AR display 220 can provide a heads-up display to a driver of the vehicle and thus may not include interactive GUI features. In other examples, the AR display 220 can include some or all the same graphical features as the AR display 200 or, in other examples, it may be omitted. For example, the AR display 220 includes a graphical representation of the vehicle indicator 222 and a graphical representation of virtual parking spot 223. The virtual parking spot is shown as a rectangular bounding box sized (e.g., approximately the same or slightly larger than the representation of the vehicle) so the graphical representation of the vehicle can fit within the box when at the target pose for the current task. In other examples, other shapes or a graphical outline of the vehicle can be used to represent the virtual parking spot 223. Additionally, a first graphical element 224 is representative of the vehicle pose (e.g., based on the platform pose data 116) and another graphical element 226 is representative of the desired pose for the vehicle so the tool can perform the given task (e.g., based on the virtual parking space data 118) at the target location 208 on the surface. When the vehicle is positioned at the desired pose, the graphical elements 224 and 226 converge and the graphical representation of the vehicle indicator 222 fits (e.g., is shown parked) within the displayed virtual parking spot 223. In the AR display 220 additional guidance can be provided, such as including a directional arrow 228 for the vehicle and atext / graphics overlay 230 describing the distance between the current vehicle pose and the desired pose for the current task.

[0057] In the examples of FIGS. 2a and 2b, the vehicle has been placed within the virtual parking space for performing a first surface treatment task. This is demonstrated in FIG. 2a by graphical element 206 being located within the periphery of the graphical element 204. Similarly, the AR display 220 demonstrates a graphical feature 226 located within the periphery of the graphical element 224. Additionally, the text / graphics overlay 230 of the AR display 220 indicates that the target pose has been reached.

[0058] In the example of FIG. 2a, the target location 208 where the task is to be performed is shown as a solid gray rectangle, corresponding to a section of a “rail” of the crosswalk. The target location thus represents an area that is to be painted as the first task in a series of tasks for painting the crosswalk. The cross-hatched regions represent other areas that are to be painted in one or more subsequent (and / or previous) steps, such as defined by a respective task plan. Other types of indicators, coloring, and / or shading can be used to differentiate between different features provided in each of the AR displays 200 and 220. The regions to be painted are overlaid on the image 202 (e.g., a live video image), which is convenient when it is desirable to align the painting plan with prior (e.g., partially worn) markings. From the pose shown, the section of rail (horizontal, gray rectangle) is reachable by the tool (e.g., paintable by the robotically controlled painting apparatus that is carried by the vehicle), so the task is performed, automatically or in response to the operator activating the paint button 214. The operator activating the task can be local (e.g., located in or near the vehicle). Also, or alternatively, the operator activating the task user interface (paint interface element 214, e.g., a button) can be at a remote site (e.g., a supervisor located in an office or mobile location).

[0059] The interactive AR display 200 can also include an arrangement of control GUI elements 212 for entering user input instructions to the system. For example, the GUI elements 212 include a paint button 214, through which a user can initiate a painting task at the target location. As described herein, the paint button 214 can be selectively enabled or disabled depending on whether the vehicle has been positioned at the virtual parking spot (e.g., at the desired vehicle pose) for the current task. The GUI elements 212 can also include forward and reverse controls (shown as “forward” and “reverse” arrows) 216 through which a user can move forwards or backwards through each of the series of tasks that are to be performed at one or more locations. Other GUI elements 212 can include controls for the information being displayed, such as to adjust the opacity of the AR display on the image 202, the size of the information being displayed (e.g., zoom control). GUI elements 212 can also be informational, such asincluding an indicator for the volume of paint used, the number of features (e.g., bars) being treated, quantity of beads used, and the like. Additional interactive GUI elements 212 can include buttons or other features to initiate a sensor scan, clean the tip of the paint dispenser, cancel the current operation as well as perform other relevant operations.

[0060] FIGS. 3a, 3b, 3c, 3d, 3e, 3f, 3g, and 3h depict examples of graphical outputs showing AR displays 250 and 252 for a next task (referred to as task 2) to be performed following performance of the task (task 1) demonstrated with respect to FIGS. 2a and 2b. The same reference numbers are used in the AR displays 250 and 252 of FIGS. 3a, 3b, 3c, 3d, 3e, 3f, 3g, and 3h to refer to the same features of the AR displays 200 and 220, respectively, described with respect to FIGS. 2a and 2b. As an example, the next task (task 2) was rendered by clicking the “forward” arrow (activating GUI element 216) to advance to the next planned task. As described herein, a plan can include a number of predefined tasks, each of which can be performed by the tool when the vehicle’s pose matches the virtual parking spot(s) determined for the respective task. As shown in FIG. 3a, the original task (horizontal rectangle) is now displayed as a solid white rectangle 208, which is based on fresh paint shown in the image 202 captured by the camera (e.g., camera 128). Although the graphical outputs shown on the displays 250 and 252 of FIGS. 3a and 3b have advanced to the second task, the vehicle has not yet moved.

[0061] In the example of FIGS. 3a and 3b, task 2 consists of three vertical rectangles to be painted (e.g., an initial section of three rungs of a crosswalk). As shown in FIG. 3a, the rectangles to be painted are outside the highlighted semicircle, indicating they are not reachable by the tool from the current vehicle pose. Thus, attempting to paint these regions, as shown, might result in collision between the robot and the vehicle’s rear bumper. Therefore, the guidance generator generates the AR displays 250 and 252 to provide guidance indicating that it is necessary to move the vehicle to execute task 2. Specifically, the AR displays 250 and 252 each includes respective graphical elements 204, 224 representative of the vehicle pose (e.g., based on the platform pose data 116) and graphical elements 206, 226 representative of the desired pose (e.g., based on virtual parking spot data 118) that has been determined for the vehicle so the tool can perform task 2.

[0062] As shown in FIGS. 3a and 3b, the graphical elements inform the driver to move the vehicle forward to make the graphical element 204, 224 converge on the respective graphical element 206, 226. In this process, the representation of the symbols to be painted (e.g., crosswalk sections) are locked to ground (since it has already been established where the crosswalk belongs on the pavement based on task data 114). Thus, as the vehicle advances, the symbols will not move with respect to ground, but the vehicle indicator 222 will move with respect to ground. Additionally, on the AR displays 250 and 252,the graphical elements 204 and 224 will move closer to the graphical element 206 and 226 responsive to updated vehicle pose data (e.g., platform pose data 116) as the vehicle moves towards alignment with the assigned virtual parking space. In the AR display 252 of FIG. 3b, the directional arrow 228 for the vehicle and a text / graphics overlay 230 provide additional guidance to the user showing the direction and distance (e.g., 4’ 2” forward) that the vehicle needs to be moved to reach the virtual parking spot for task 2.

[0063] The process of the vehicle moving closer to and into alignment with the virtual parking space assigned for task 2 is illustrated in the AR displays of FIGS. 3c, 3d, 3e, 3f, 3g, and 3h, in which the graphical elements 204 and 224 move closer to the graphical element 206 and 226 as the vehicle’s location changes. FIGS. 3c, 3d, 3e, 3f, 3g, and 3h show snapshots of the display as the vehicle moves forward towards the virtual parking spot (representative of the target pose) determined for task 2. In the graphical displays of FIGS. 3c, 3d, 3e, 3f, 3g, and 3h, the graphical elements 206 and 226 are representative of the desired (target) vehicle pose determined for task 2 (e.g., based on virtual parking spot data 118 for task 2) and the graphical elements 204 and 224 square is representative of the current vehicle pose (e.g., based on platform pose data 116).

[0064] Initially, as shown in FIGS. 3a and 3b, the virtual parking spot for task 2 is 4’2” forward of the current vehicle pose. In FIGS. 3c and 3d, as shown by overlay 230, the vehicle is only 2’ 10” from its goal of reaching the virtual parking spot. Thus, the graphical elements 204, 224 (e.g., a blue square or other indicator) are closer to the respective graphical elements 206, 226 (e.g., a green square or other indicator for the virtual parking spot). In the example of FIGS. 3e and 3f, the vehicle is closer still to its goal, with the guidance overlay indicating only 11” forward to reach the virtual parking spot. FIGS. 3g and 3h show that the vehicle has converged on the recommended virtual parking spot for task 2. Thus, as shown, the graphical elements 204, 224 (e.g., a blue square or other indicator) have converged with the respective graphical elements 206, 226. From this pose, the three vertical rectangles to be painted for task 2 are entirely within the reachable region 210, and thus task 2 is executable by the tool from this vehicle pose. For example, the task can be initiated in response to an operator activating the paint interface element 214 through the touchscreen or other user interface of the device (e.g., via input device 122) where the AR display 250 is displayed. As described herein, the paint interface element 214 can be enabled in response to determining that the current vehicle pose matches the target pose for task 2. Until such a condition is met, the system (e.g., instructions 108) can be programmed to disable paint interface element 214 and / or operation of the tool 112 to prevent the tool from executing task 2. After executing task 2 (painting the 3 vertical bars), the next task can be displayed by clicking on the “advance” userinterface element (e.g., button) 216 with a user input device (e.g., device 122 such as a mouse or touchscreen).

[0065] FIGS. 4a, 4b, 4c, and 4d depict examples of graphical outputs showing AR displays 300 and 302 for a next task (referred to as task 3) to be performed following performance of the task (task 2) demonstrated with respect to FIGS. 3a-3h. The same reference numbers are used in the AR displays 300 and 302 of FIGS. 3a-3h to refer to the same features of the AR displays 250 and 252, respectively, described with respect to FIGS. 3a-3h. As shown, task 3 also requires painting three additional vertical bars of the crosswalk, and these are to be aligned with the previous 3 vertical bars, shown as 308 in the image 202, formed in performance of task 2 (e.g., further extending these three rungs of the crosswalk).

[0066] As shown in FIGS. 4a and 4b, task 3 cannot be performed from the current pose of the vehicle, which is where task 2 had been completed. In the AR display 302 of FIG. 4b, the directional arrow 228 for the vehicle and a text / graphics overlay 230 provide additional guidance to the user showing the computed direction and distance (e.g., 5’ 0” forward) that the vehicle needs to be moved to reach virtual parking spot for task 3. As shown in FIGS. 4a and 4b, the graphical elements further inform the driver to move the vehicle forward to make the graphical element 204, 224 converge on the respective graphical element 206, 226. In this process, as the vehicle advances, the symbols 208 and 308 and virtual parking spot indicators 203 and 223 will not move with respect to ground, but the vehicle indicator 222 will move with respect to ground. Additionally, on the AR displays 300 and 302, the graphical elements 204 and 224 will move closer to the graphical element 206 and 226 responsive to updated vehicle pose data (e.g., platform pose data 116) as the vehicle moves towards alignment with the assigned virtual parking space (e.g., defined by virtual parking space data 118).

[0067] FIGS. 4c and 4d shows an example of the AR displays 300 and 302 after the vehicle has been advanced to the pose corresponding to the virtual parking spot that has been determined for task 3. As shown in FIGS. 4c and 4d, task 3 is executable from this vehicle pose, such as in response to a user input activating the paint user interface element 214. The example crosswalk section can be completed after two more tasks: task 4 to finish painting 3 additional rungs of the crosswalk, and task 5 to paint the associated rail (horizontal rectangle) at the termination of the 3 rungs painted in task 4, for example. Similar to the preceding tasks, each of the tasks 4 and 5 can also loaded for execution and rendering in the AR displays responsive to clicking the “advance” button 216, and the operator / driver is informed where to place the vehicle for each additional task, which can be represented by virtual parking spots for each of the respective tasks 4 and 5, to make each of these tasks executable. For example, the task canbe initiated in response to an operator activating the paint interface element 214 through the touchscreen or other user interface of the device (e.g., via input device 122) where the AR display 250 is displayed.

[0068] FIGS. 5a, 5b, and 5c depict example AR displays 350, 380, and 390, respectively, for providing guidance and executing a task plan that includes a series of tasks (e.g., two tasks) to be performed by a tool that is carried by a vehicle. The tasks, which are defined by task data, include painting a turn arrow at a target location on a surface of a road. In this example, the turn arrow cannot be painted in a single task, but instead the stem is to be painted as a first task while the vehicle is at a first target pose and the turn-arrow head is to be painted as a second task while the vehicle is at a second pose.

[0069] FIG. 5a depicts an example of an interactive AR display 350 for the first part of the task plan for painting the turn arrow, namely for painting a stem of the turn arrow. The AR display 350 includes an arrangement of graphical features overlaid (e.g., superimposed) on a representation or image of the environment, shown at 352, which includes the road where the turn arrow is being applied. The AR display 350 also includes a graphical element 354 representative of the vehicle pose (e.g., based on the platform pose data 116) and another graphical element 356 representative of the desired pose for the vehicle so the tool can paint the stem of the turn arrow (e.g., based on the virtual parking space data 118) at the target location 358 on the surface. The AR display 350 also includes a graphical representation of a virtual parking spot for the vehicle, shown as at 360.

[0070] In the example of FIG. 5a, the vehicle is at the desired pose, defined by the virtual parking space data for the current task, such that the graphical elements 354 and 356 converge and the graphical representation of the vehicle fits (e.g., is shown parked) within the displayed virtual parking spot 360. From this pose, the target location 358 (e.g., for the stem of the turn arrow) to be painted for task 1 is entirely within the semicircular reachable region 362, and thus task 2 is executable by the tool from this vehicle pose. As expected, however, the graphical representation of the arrowhead of the turn arrow, shown at 364, does not fit within the reachable region 362. For example, the task 1 can be initiated in response to an operator activating the paint interface element 366 through the touchscreen or other user interface of the device (e.g., via input device 122) where the AR display 350 is displayed (e.g., locally or remotely from the target site). As described herein, the paint interface element 366 can be enabled in response to determining that the current vehicle pose matches the target pose for the current task 1. After executing task 1 (painting the stem), the next task (e.g., task 2 for painting the arrowhead) can be loaded and displayed by clicking on the “advance” user interface element (e.g., button) 368 with a user input device (e.g., device 122 such as a mouse or touchscreen). The interactive AR display 350 can also includea number of other informational text and / or GUI elements 370 for monitoring and / or controlling operating parameters of the vehicle and / or tool, such as described herein.

[0071] FIG. 5b illustrates an example of the AR display 380 showing the vehicle in a second pose after the vehicle has advanced straight forward in an attempt to locate the vehicle for executing task 2 for painting the arrowhead. FIG. 5b demonstrates that if the vehicle merely pulls forward, the arrowhead is not paintable. This is visualized in the AR display 380 by the vehicle pose graphical elements 354 not converging with the target pose graphical element 356. Additionally, the paint user interface element 366 is disabled and will be unresponsive to a user activation.

[0072] In order to advance the vehicle in the recommended virtual parking spot for task 2 (e.g., painting the arrowhead) from the pose shown in FIG. 5a, the driver should pull forward and slightly to the right, as shown in the AR display 390 of FIG. 5c. For example, the graphical elements 354 and 356 have converged for task 2, indicating that the vehicle is located in the virtual parking spot for task 2 so that the target site for the arrowhead 364 is reachable. As described herein, the paint interface element 366 can be rendered responsive based on detecting that the current vehicle pose matches the target pose for the current task 2. Therefore, the arrowhead can be painted by the robot tool (e.g., by executing a defined joint space while dispensing paint through dispensing tool) in response to an operator activating the paint interface element 366 through the touchscreen or other user interface of the device (e.g., via input device 122).

[0073] In some examples, it might seem counterintuitive that the vehicle should shift to the right in order to paint an object (e.g., like an arrowhead) that is positioned to the left. This is because there can be a region centered on the bumper that is hard for the robotic tool to reach. By shifting the vehicle slightly to the right, the tool can avoid the restriction of the hard-to-reach area. This might not be obvious to the driver, who would otherwise pull forward and be frustrated that the arrowhead is unreachable from the new pose. But with the virtual parking spot provided in the AR display, the driver is advised to shift slightly to the right while pulling forward. If the vehicle (or other movable platform) converges on the virtual parking spot, the current task will be reachable, thus avoiding frustration and improving productivity.

[0074] As described herein, the systems and methods (e.g., site event coordinator function 144) be configured to provide supplemental guidance in the form of virtual parking spots for one or more auxiliary units. The site coordination function can be implemented for the auxiliary units in combination with, or separately from, systems and methods that define a virtual parking spot for a movable platform that carries a tool for performing core tasks. The auxiliary units can be implemented as one or more oftruck mounted attenuators, towable attenuators, sleds, vehicles, barricades, barrels, cones, signage, or any other auxiliary objects that can support performing one or more tasks. The number and location of auxiliary objects and their virtual parking spaces can depend on the core task being performed, the location of the core task, and the environment near and around where the tool performs the core task. In the example of applying one or more surface treatments at desired target locations (e.g., on road surfaces), location information (e.g., virtual parking spots) for one or more auxiliary objects can be specified (e.g., by site event coordination function 144). For example, at large intersections, this could involve defining virtual parking spots for several auxiliary units that should be pre-positioned at strategic locations to appropriately block traffic coming from different approaches so as to keep the work area protected.

[0075] The virtual parking spots can be communicated to the auxiliary units directly or to their operators (e.g., to computing devices associated with the respective units). In examples where the auxiliary units are vehicles, a computing device and associated display can be mounted in a cab or other associated driving compartment. Also, or alternatively, the computing device can be a portable computing device (e.g., cell phone, tablet computer, notebook computer etc.) that can be associated with a respective auxiliary unit (or its operator). The virtual parking spaces for a given task can be visualized for the auxiliary units collectively on an AR display. Also, or alternatively, a separate AR display can be generated to visualize a virtual parking space separately for each auxiliary unit. The AR displays for the auxiliary units can be generated as static displays. Alternatively, the AR displays for the auxiliary units can be dynamic and vary based on localization information (e.g., vehicle pose data) that is generated for each (or some) of the auxiliary units to provide dynamic, real time guidance similar to the guidance described herein for performing respective tasks. The auxiliary units do not need to be positioned for a robot or other tool to reach a task, but the auxiliary units should be positioned per a task plan to execute an event coordination strategy for supporting one or more core tasks that are being performed. The AR displays generated for providing virtual parking spots for one or more auxiliary units also can include a graphical representation of a virtual parking spot for the tool and its associate movable platform.

[0076] By way of example, FIGS. 6 and 7 depict examples of AR displays 400 and 410 that can be generated (e.g., by site event coordination function 144), such as to implement a coordinated safety strategy for applying surface treatments to a road over one or more tasks, such as described herein. In the examples of FIGS. 6 and 7, the auxiliary units are demonstrated as truck mounted attenuators. In other examples, other types of auxiliary units can be used depending on the circumstances and tasks being performed. Also, or alternatively, the graphical representations shown in the AR display can begeneric objects (e.g., icons) or be adapted to match the type of auxiliary unit that is to be positioned in each respective virtual parking spot.

[0077] In the AR display 400 of FIG. 6, the AR display provides a graphical representation of an intersection, such as a map or satellite rendering of the intersection. The AR display 400 includes a graphical representation of task vehicle (or other movable platform) 402, which includes a tool for performing one or more tasks. For example, the vehicle 402 is shown in the AR display at a location corresponding to a virtual parking spot (e.g., based on virtual parking spot data 118) from which the tool is to perform a given task (e.g., painting a portion of a rectangular strip on the road). In the example of FIG. 6, the vehicle is oriented in the opposite direction of traffic flow to position the tool in a desired position behind the vehicle to perform the given task. The AR display 400 also includes a graphical representation of an auxiliary unit 404 shown at a location described by a virtual parking spot for the auxiliary unit, which is directly behind the virtual parking spot for the vehicle 402.

[0078] The AR display 410 of FIG. 7 also provides a graphical representation of an intersection. The AR display 410 includes a graphical representation of a task vehicle 412 at a corresponding virtual parking spot from which the associated tool (e.g., a robot) can perform a given task. The AR display 410 also includes graphical representations of a plurality of (e.g., three) auxiliary units 414, 416, and 418 at respective virtual parking spots strategically determined for supporting the task vehicle for performing the given task. In the example of FIG. 7, the virtual parking spots for the auxiliary units 414, 416, and 418 includes the directly opposing lane and adjacent orthogonal lanes that feed into the lane where the task vehicle is to perform the given task. As described herein, a different AR display can be provided for each task to be performed by the task vehicle, which can be automatically updated and displayed in a display device for each of the auxiliary units.Additional Examples

[0079] Several aspects of the present technology are set forth in the following numbered examples.Example 1. A system comprising: non-transitory memory that stores data and instructions, in which the data comprises: task data that defines a given task to be performed by a tool at a target location, in which the tool is carried by a moveable platform; pose data representative of a current location and orientation of the moveable platform;virtual parking space data represents at least one desired pose of the moveable platform from which the tool is capable to perform the given task at the target location; and a processor configured to access the memory and execute the instructions to cause the processor to at least: provide a guidance output based on the pose data and the virtual parking space data, whereby the guidance output facilitates placement of the moveable platform at the desired pose for performing the given task.Example 2. The system of example 1, wherein the task data comprises task instructions to control the tool for performing the given task.Example 3. The system of example 1 or 2, wherein the tool includes a robot having a joint space and the task instructions are programmed to control motion of the robot through a joint space trajectory to perform the given task.Example 4. The system of example 3, wherein the moveable platform is a vehicle and the given task comprises at least one of painting, grinding, digging, drilling, cleaning, cutting, sealing, cutting and / or milling along a pattern over the target location.Example 5. The system according to any one of the previous examples, wherein the guidance output includes a first graphical element representative of the pose based on the pose data relative a second graphical element representative of the desired pose based on the virtual parking space data.Example 6. The system of example 5, wherein the given task includes applying a marking on a surface and the guidance output further comprises a third graphical element representative of the marking to be applied on the surface.Example 7. The system according to example 5 or 6, further comprising a camera, fixed relative to the moveable platform, and configured to provide an image that includes at least a portion of a zone of reachability for the tool, wherein at least one of the first graphical element,the second graphical element, and the third graphical element are overlaid on at least a portion of the image.Example 8. The system according to any one of the preceding examples, wherein the instructions are further programmed to evaluate candidate joint space trajectories based on the pose of the moveable platform and the desired pose of the moveable platform for the given task to determine if the given task is reachable by the tool as to be successfully completed, to enable the tool to perform the given task responsive to determining that the given task is reachable based on the evaluation, and disable the tool from performing the given task responsive to determining that the given task is not reachable based on the evaluation.Example 9. The system according to any one of the preceding examples, wherein the moveable platform is a vehicle and the data further comprises plan data that defines a set of tasks to be performed by the tool at respective target locations based on task data for each respective task, the virtual parking spot data including a predefined vehicle pose associated with each task from which the tool can successfully complete each respective task, in which the plan is encoded as a sequence of virtual parking spots.Example 10. The system of example 9, further comprising a user interface element programmed to select a next task in the set of tasks responsive to a user input, in which the guidance is generated for the selected next task based on the task data, the virtual parking space data for the next task, and the vehicle pose data.Example 11. The system according to any one of the preceding examples, wherein the instructions are further programmed to cause the processor to lock the target location for a respective task with respect to the current pose of the moveable platform in response to user input instruction, such that a graphical representation of the task to be performed at the predefined target location is fixed with respect to the predefined target location in an image acquired by a camera carried by the moveable platform.Example 12. The system of example 7 or 11, wherein a graphical representation of the output guidance is overlaid on the image.Example 13. The system according to any one of the preceding examples, further comprising a localization system configured to provide the pose data to represent a current pose of the moveable platform.Example 14. The system according to any one of the preceding examples, wherein the instructions are further programmed to cause the processor to provide a graphical representation of a virtual parking spot for at least one auxiliary unit based on the virtual parking spot data for the given task.Example 15. A method, comprising: storing, in the non-transitory memory, pose data representative of a current location and orientation of a moveable platform, in which a tool is carried by the moveable platform; storing, in the non-transitory memory, virtual parking space data representative of at least one desired pose of a moveable platform from which the tool is capable to perform a given task at a target location; and providing, by a processor, a guidance output based on the pose data and the virtual parking space data, whereby the guidance output facilitates placement of the moveable platform at the desired pose for performing the given task.Example 16. The method of example 15, further comprising storing, in the non-transitory memory, task instructions to control the tool for performing the given task.Example 17. The method of example 16, wherein the tool includes a robot having a joint space and the task instructions are programmed to control motion of the robot through a joint space trajectory to perform the given task.Example 18. The method of example 18, wherein the moveable platform is a vehicle and the given task comprises at least one of painting, grinding, digging, drilling, cleaning, cutting, sealing, cutting and / or milling along a pattern over the target location.Example 19. The method according to any one of examples 15-18, wherein the guidance output includes a first graphical element representative of the pose based on the pose data relative a second graphical element representative of the desired pose based on the virtual parking space data.Example 20. The method of example 19, wherein the given task includes applying a marking on a surface and the guidance output further comprises a third graphical element representative of the marking to be applied on the surface.Example 21. The method according to example 19 or 20, further comprising: acquiring, by a camera that is fixed relative to the moveable platform, an image that includes at least a portion of a zone of reachability for the tool; and overlying at least one of the first graphical element, the second graphical element, and the third graphical element on at least a portion of the image.Example 22. The method according to any one of examples 15-21, further comprising: evaluating, by the processor, candidate joint space trajectories based on the pose of the moveable platform and the desired pose of the moveable platform for the given task to determine if the given task is reachable by the tool as to be successfully completed; enabling the tool to perform the given task responsive to determining that the given task is reachable based on the evaluation; and disabling the tool from performing the given task responsive to determining that the given task is not reachable based on the evaluation.Example 23. The method according to any one of examples 15-22 according to any one of the preceding examples, wherein the moveable platform is a vehicle and the method further comprises: storing plan data that defines a set of tasks to be performed by the tool at respective target locations based on task data for each respective task, the virtual parking spot data including a predefined vehicle pose associated with each task from which the tool can successfully complete each respective task, in which the plan is encoded as a sequence of virtual parking spots.Example 24. The method of example 23, further comprising: selecting a next task in a set of tasks responsive to a user input via a user interface element; and generating guidance for the selected next task based on the task data, the virtual parking space data for the next task, and the vehicle pose data.Example 25. The method according to any one of the examples 15-24, locking, by the processor, the target location for a respective task with respect to the current pose of the moveable platform in response to a user input instruction, such that a graphical representation of the task to be performed at the predefined target location is fixed with respect to the predefined target location in an image acquired by a camera carried by the moveable platform.Example 26. The method of example 21 or 25, further comprising: overlaying a graphical representation of the output guidance on the image.Example 27. The method according to any one of examples 15-26, further comprising receiving localization information from a localization system and the pose data is provided to represent a current pose of the moveable platform based on the localization information.Example 28. The method according to any one of examples 15-27, further comprising: providing, by the processor, a graphical representation of a virtual parking spot for at least one auxiliary unit based on the virtual parking spot data for the given task.Example 29. A method comprises providing substantially real-time guidance on a display based on a current pose of a moveable platform carrying an automated or semi- automated tool and a predetermined task to be performed by the tool at a predetermined target location, in which the guidance provides instructions on the display for placing the moveable platform at a corresponding pose to enable the tool to perform the predetermined task at the predetermined target location.Example 30. One or more non-transitory computer-readable media includes instructions which, when executed by a processor, are configured to cause the processor to perform the method according to any one of examples 15-29.

[0080] It should be understood that various aspects disclosed herein may be combined in different combinations than the combinations specifically presented in the description and accompanying drawings. It should also be understood that, depending on the example, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, may be added, merged, or left out altogether (e.g., all described acts or events may not be necessary to carry out the techniques). In addition, while certain aspects of this disclosure are described as being performed by a single module or unit for purposes of clarity, it should be understood that the techniques of this disclosure may be performed by a combination of units or modules associated with, for example, a robotically controlled painting tool, a medical device, or other type of tool.

[0081] In one or more examples, the described techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include non-transitory computer-readable media, which corresponds to a tangible medium such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0082] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” as used herein may refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements, any of which would constitute a processor as used herein.

[0083] It will be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Thus, a “first” element discussed below could also be termed a “second” element without departing from the teachings of the subject disclosure. The sequenceof operations (or steps) is not limited to the order presented in the claims or figures unless specifically indicated otherwise.

[0084] In this description, the term “couple” may cover connections, communications, or signal paths that enable a functional relationship consistent with this description. For example, if device A generates a signal to control device B to perform an action: (a) in a first example, device A is coupled to device B by direct connection; or (b) in a second example, device A is coupled to device B through intervening component C if intervening component C does not alter the functional relationship between device A and device B, such that device B is controlled by device A via the control signal generated by device A.

[0085] Additionally, in an effort to maintain clarity in the Figures, certain ones of duplicative components shown have not been specifically numbered, but one of ordinary skill in the art will realize, based upon the components that were numbered, the element numbers which should be associated with the unnumbered components; no differentiation between similar components is intended or implied solely by the presence or absence of an element number in the Figures. Any of the described structures and components could be integrally formed as a single unitary or monolithic piece or made up of separate sub-components, with either of these formations involving any suitable stock or bespoke components and / or any suitable material or combinations of materials; however, the chosen material(s) should be biocompatible for many applications. Any of the described structures and components could be disposable or reusable as desired for a particular use environment.

[0086] Modifications are possible in the described embodiments, and other embodiments are possible, within the scope of the claims.

[0087] All publications identified herein are incorporated by reference in their entireties.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. A system comprising: non-transitory memory that stores data and instructions, in which the data comprises: task data that defines a given task to be performed by a tool at a target location, in which the tool is carried by a moveable platform; pose data representative of a current location and orientation of the moveable platform; virtual parking space data represents at least one desired pose of the moveable platform from which the tool is capable to perform the given task at the target location; and a processor configured to access the memory and execute the instructions to cause the processor to at least: provide a guidance output based on the pose data and the virtual parking space data, whereby the guidance output facilitates placement of the moveable platform at the desired pose for performing the given task.2 The system of claim 1, wherein the task data comprises task instructions to control the tool for performing the given task.3 The system of claim 1 or 2, wherein the tool includes a robot having a joint space and the task instructions are programmed to control motion of the robot through a joint space trajectory to perform the given task.4 The system of claim 3, wherein the moveable platform is a vehicle and the given task comprises at least one of painting, grinding, digging, drilling, cleaning, cutting, sealing, cutting and / or milling along a pattern over the target location.5 The system according to any one of the previous claims, wherein the guidance output includes a first graphical element representative of the pose based on the pose data relative a second graphical element representative of the desired pose based on the virtual parking space data.

6. The system of claim 5, wherein the given task includes applying a marking on a surface and the guidance output further comprises a third graphical element representative of the marking to be applied on the surface.

7. The system according to claim 5 or 6, further comprising a camera, fixed relative to the moveable platform, and configured to provide an image that includes at least a portion of a zone of reachability for the tool, wherein at least one of the first graphical element, the second graphical element, and the third graphical element are overlaid on at least a portion of the image.8 The system according to any one of the preceding claims, wherein the instructions a e further programmed to evaluate candidate joint space trajectories based on the pose of the moveable platform and the desired pose of the moveable platform for the given task to determine if the given task is reachable by the tool as to be successfully completed, to enable the tool to perform the given task responsive to determining that the given task is reachable based on the evaluation, and disable the tool from performing the given task responsive to determining that the given task is not reachable based on the evaluation.9 The system according to any one of the preceding claims, wherein the moveable platform is a vehicle and the data further comprises plan data that defines a set of tasks to be performed by the tool at respective target locations based on task data for each respective task, the virtual parking spot data including a predefined vehicle pose associated with each task from which the tool can successfully complete each respective task, in which the plan is encoded as a sequence of virtual parking spots.10 The system of claim 9, further comprising a user interface element programmed to select a next task in the set of tasks responsive to a user input, in which the guidance is generated for the selected next task based on the task data, the virtual parking space data for the next task, and the vehicle pose data11 The system according to any one of the preceding claims, wherein the instructions a e furthex' programmed to cause the processor to lock the target location for a respective task with respect to the current pose of the moveable platform in response to user input instruction, such that a graphicalrepresentation of the task to be performed at the predefined target location is fixed with respect to the predefined target location in an image acquired by a camera carried by the moveable platform.

12. The system of claim 7 or 11, wherein a graphical representation of the output guidance is overlaid on the image.

13. The system according to any one of the preceding claims, further comprising a localization system configured to provide the pose data to represent a current pose of the moveable platform.

14. The system according to any one of the preceding claims, wherein the instructions are further programmed to cause the processor to provide a graphical representation of a virtual parking spot for at least one auxiliary unit based on the virtual parking spot data for the given task.

15. A method, comprising: storing, in the non-transitory memory, pose data representative of a current location and orientation of a moveable platform, in which a tool is carried by the moveable platform; storing, in the non-transitory memory, virtual parking space data representative of at least one desired pose of a moveable platform from which the tool is capable to perform a given task at a target location; and providing, by a processor, a guidance output based on the pose data and the virtual parking space data, whereby the guidance output facilitates placement of the moveable platform at the desired pose for performing the given task.

16. The method of claim 15, further comprising storing, in the non-transitory memory, task instructions to control the tool for performing the given task.

17. The method of claim 16, wherein the tool includes a robot having a joint space and the task instructions are programmed to control motion of the robot through a joint space trajectory to perform the given task.

18. The method of claim 18, wherein the moveable platform is a vehicle and the given task comprises at least one of painting, grinding, digging, drilling, cleaning, cutting, sealing, cutting and / or milling along a pattern over the target location.

19. The method according to any one of claims 15-18, wherein the guidance output includes a first graphical element representative of the pose based on the pose data relative a second graphical element representative of the desired pose based on the virtual parking space data.

20. The method of claim 19, wherein the given task includes applying a marking on a surface and the guidance output further comprises a third graphical element representative of the marking to be applied on the surface.

21. The method according to claim 19 or 20, further comprising: acquiring, by a camera that is fixed relative to the moveable platform, an image that includes at least a portion of a zone of reachability for the tool; and overlying at least one of the first graphical element, the second graphical element, and the third graphical element on at least a portion of the image.

22. The method according to any one of claims 15-21, further comprising: evaluating, by the processor, candidate joint space trajectories based on the pose of the moveable platform and the desired pose of the moveable platform for the given task to determine if the given task is reachable by the tool as to be successfully completed; enabling the tool to perform the given task responsive to determining that the given task is reachable based on the evaluation; and disabling the tool from performing the given task responsive to determining that the given task is not reachable based on the evaluation.

23. The method according to any one of claims 15-22 according to any one of the preceding claims, wherein the moveable platform is a vehicle and the method further comprises: storing plan data that defines a set of tasks to be performed by the tool at respective target locations based on task data for each respective task, the virtual parking spot data including apredefined vehicle pose associated with each task from which the tool can successfully complete each respective task, in which the plan is encoded as a sequence of virtual parking spots.

24. The method of claim 23, further comprising: selecting a next task in a set of tasks responsive to a user input via a user interface element; and generating guidance for the selected next task based on the task data, the virtual parking space data for the next task, and the vehicle pose data.

25. The method according to any one of the claims 15-24, locking, by the processor, the target location for a respective task with respect to the current pose of the moveable platform in response to a user input instruction, such that a graphical representation of the task to be performed at the predefined target location is fixed with respect to the predefined target location in an image acquired by a camera canned by the moveable platform.

26. The method of claim 21 or 25, further comprising: overlaying a graphical representation of the output guidance on the image.

27. The method according to any one of claims 15-26, further comprising receiving localization information from a localization system and the pose data is provided to represent a current pose of the moveable platform based on the localization information.

28. The method according to any one of claims 15-27, further comprising: providing, by the processor, a graphical representation of a virtual parking spot for at least one auxiliary unit based on the virtual parking spot data for the given task.

29. A method comprises providing substantially real-time guidance on a display based on a current pose of a moveable platform carrying an automated or semi- automated tool and a predetermined task to be performed by the tool at a predetermined target location, in which the guidance provides instructions on the display for placing the moveable platform at a corresponding pose to enable the tool to perform the predetermined task at the predetermined target location.

30. One or more non-transitory computer-readable media includes instructions which, when executed by a processor, are configured to cause the processor to perform the method according to any one of claims 15-29.

Images