Method and apparatus for indicating motion

Abstract

A method includes determining a first motion plan and a second motion plan based on an input, and determining that the first motion plan is prioritized relative to the second motion plan. The method further includes identifying one of the inputs as a sensitive input that prioritizes the first motion plan over the second motion plan; and presenting, using a display, information describing the first exercise plan. The information includes an interpretation indicating the sensitive input as a reason for the first motion plan to prioritize the second motion plan. The method also includes communicating and initiating a priority exercise plan.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 407,217, filed on September 16, 2022, the contents of which are hereby incorporated by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure relates generally to the field of intent indication for computing systems. Background Art

[0004] Some automated systems are able to make decisions and act upon those decisions, such as by moving within an environment. Users can gain insight into automated systems when the future actions of the system are communicated. Summary of the invention

[0005] A first aspect of the present disclosure is a method comprising determining a first motion plan and a second motion plan for a mobile electronic device based on an input. The method also comprises determining that the first motion plan is prioritized relative to the second motion plan. The method also comprises identifying one of the inputs as a sensitive input that makes the first motion plan take precedence over the second motion plan. The method also comprises presenting information describing the first motion plan using a display, wherein the information comprises an explanation indicating the sensitive input as a reason why the first motion plan takes precedence over the second motion plan. The method also comprises controlling the mobile electronic device using one of the first motion plan or the second motion plan.

[0006] In some specific implementations of the method according to the first aspect of the present disclosure, the explanation includes text determined based on the sensitive input. In some specific implementations of the method according to the first aspect of the present disclosure, the explanation includes an icon representing the sensitive input. In some specific implementations of the method according to the first aspect of the present disclosure, the explanation of why the first motion plan takes precedence over the second motion plan includes text determined based on the sensitive input. In some specific implementations of the method according to the first aspect of the present disclosure, the explanation of why the first motion plan takes precedence over the second motion plan includes an icon representing the sensitive input.

[0007] In some specific implementations of the method according to the first aspect of the present disclosure, the sensitive input is associated with occupant comfort. In some specific implementations of the method according to the first aspect of the present disclosure, the sensitive input is associated with a travel path plan. In some specific implementations of the method according to the first aspect of the present disclosure, the sensitive input is associated with travel time.

[0008] In some implementations of the method according to the first aspect of the disclosure, the sensitivity analysis includes modifying at least some of the inputs, and the modification of the sensitivity inputs reduces a first score for the first motion plan so that the first score is lower than a second score for the second motion plan. In some implementations of the method according to the first aspect of the disclosure, the first motion plan includes a first expected path of travel around an obstacle, and the second motion plan includes a second expected path of travel that is different from the first path of travel.

[0009] In some specific implementations of the method according to the first aspect of the present disclosure, presenting information describing the first motion plan includes displaying a first motion plan representation of the first motion plan and a second motion plan representation of the second motion plan. In some specific implementations of the method according to the first aspect of the present disclosure, the first motion plan representation of the first motion plan and the second motion plan representation of the second motion plan are displayed with different visual indications, such as at least one of different colors or different opacities. In some specific implementations, the input is obtained by a sensor of the mobile electronic device. The above features can be combined with each other.

[0010] A second aspect of the present disclosure is a non-transitory computer-readable storage device comprising program instructions that can be executed by one or more processors. The program instructions cause one or more processors to perform operations when executed. The operations include determining a first motion plan and a second motion plan for a mobile electronic device based on an input. The operations also include determining that the first motion plan is prioritized relative to the second motion plan. The operations also include identifying one of the inputs as a sensitive input that prioritizes the first motion plan over the second motion plan. The operations also include presenting information describing the first motion plan using a display, wherein the information includes an explanation indicating the sensitive input as a reason why the first motion plan prioritizes the second motion plan. The operations also include controlling the mobile electronic device using one of the first motion plan or the second motion plan.

[0011] The third aspect of the present disclosure is a device, which includes a memory and one or more processors, which are configured to execute instructions stored in the memory. When the instruction is executed, the one or more processors determine a first motion plan and a second motion plan for a mobile electronic device based on an input. When the instruction is executed, the one or more processors also cause: determining that the first motion plan is prioritized relative to the second motion plan; identifying one of the inputs as a sensitive input that prioritizes the first motion plan over the second motion plan; using a display to present information describing the first motion plan, wherein the information includes an explanation indicating the sensitive input as the reason why the first motion plan takes precedence over the second motion plan, and using one of the first motion plan or the second motion plan to control the mobile electronic device.

[0012] A fourth aspect of the present disclosure is a method comprising: at a mobile electronic device located in an environment, determining a motion plan for the mobile electronic device based on input, wherein the motion plan includes a motion manipulation including a planned path around an object; analyzing the input to identify features of the environment that cause the motion manipulation to be included in the motion plan; presenting, using a display, information describing the motion manipulation and identifying features of the environment; and controlling the mobile electronic device using the motion plan.

[0013] In some specific implementations of the method according to the fourth aspect of the present disclosure, analyzing the input to identify the characteristics of the environment that make the motion manipulation included in the motion plan includes identifying the characteristics of the environment by performing a sensitivity analysis. In some specific implementations of the method according to the fourth aspect of the present disclosure, the information describing the motion manipulation and identifying the characteristics of the environment includes outputting text that identifies the characteristics of the environment. In some specific implementations of the method according to the fourth aspect of the present disclosure, presenting to the user information describing the motion manipulation and identifying the characteristics of the environment includes displaying an icon representing the characteristics of the environment. In some specific implementations of the method according to the fourth aspect of the present disclosure, presenting to the user information describing the motion manipulation and identifying the characteristics of the environment includes displaying a motion plan representation representing the motion manipulation. In some specific implementations of the method according to the fourth aspect of the present disclosure, the characteristics of the environment are dynamic objects. In some specific implementations of the method according to the fourth aspect of the present disclosure, the characteristics of the environment are static objects. In some specific implementations, the input is obtained by a sensor of a mobile electronic device. The above features can be combined with each other.

[0014] A fifth aspect of the present disclosure is a non-transitory computer-readable storage device, the non-transitory computer-readable storage device including one or more processor executable program instructions, which when executed cause the one or more processors to perform operations. The operations include: at a mobile electronic device located in an environment, determining a motion plan for the mobile electronic device based on an input, wherein the motion plan includes a motion manipulation, the motion manipulation including a planned path around an object; analyzing the input to identify features of the environment that cause the motion manipulation to be included in the motion plan; presenting information describing the motion manipulation and identifying features of the environment using a display; and controlling the mobile electronic device using the motion plan.

[0015] The sixth aspect of the present disclosure is a device, the device comprising a memory and one or more processors, the one or more processors being configured to execute instructions stored in the memory. When the instructions are executed, the one or more processors: at a mobile electronic device located in an environment, determine a motion plan for the mobile electronic device based on an input, wherein the motion plan includes a motion manipulation, the motion manipulation includes a planned path around an object; analyze the input to identify features of the environment that cause the motion manipulation to be included in the motion plan; use a display to present information describing the motion manipulation and identifying features of the environment; and control the mobile electronic device using the motion plan.

[0016] The seventh aspect of the present disclosure is a method comprising: determining a motion plan for a mobile electronic device; and displaying an environment representation using a display, wherein the environment representation is a graphical representation of the environment surrounding the mobile electronic device. The method also includes displaying a motion plan representation overlaid on the environment representation to a user, the motion plan representation indicating an area of ​​the environment in which the mobile electronic device can travel within a time horizon, the motion plan representation extending from a first end corresponding to the current position of the mobile electronic device to a second end corresponding to the expected future position of the mobile electronic device at the end of the time horizon, wherein the motion plan representation is continuously updated to reflect changes in the current position of the mobile electronic device and changes in the motion plan. The method also includes using the motion plan to control the mobile electronic device.

[0017] In some specific implementations of the method according to the seventh aspect of the present disclosure, the environment representation includes a map. In some embodiments of the method according to the seventh aspect of the present disclosure, the end of the time range is determined by adding a fixed duration time interval to the current time. In some specific implementations of the method according to the seventh aspect of the present disclosure, the length of the motion plan representation between the first end of the motion plan representation and the second end of the motion plan representation represents the expected travel distance of the mobile electronic device during the time range, and the length of the motion plan representation is continuously updated to reflect the change of the current position of the mobile electronic device and the change of the motion plan. In some specific implementations of the method according to the seventh aspect of the present disclosure, the color of at least a part of the motion plan representation represents the acceleration of the mobile electronic device during the time range. In some specific implementations of the method according to the seventh aspect of the present disclosure, the motion plan representation is a first motion plan representation and the motion plan is a first motion plan, and the method also includes determining a second motion plan, and displaying a second motion plan representation of the second motion plan overlaid on the environment representation to the user. In some specific implementations of the method according to the seventh aspect of the present disclosure, the first motion plan representation of the first motion plan and the second motion plan representation of the second motion plan are displayed in at least one of different colors or different opacities. The above features can be combined with each other.

[0018] The eighth aspect of the present disclosure is a non-transitory computer-readable storage device, which includes program instructions executable by one or more processors, which when executed cause the one or more processors to perform operations. The operations include: determining a motion plan for a mobile electronic device; and displaying an environment representation using a display, wherein the environment representation is a graphical representation of the environment surrounding the mobile electronic device. The operations also include displaying a motion plan representation overlaid on the environment representation to a user, the motion plan representation indicating an area of ​​the environment in which the mobile electronic device can travel within a time range, the motion plan representation extending from a first end corresponding to the current position of the mobile electronic device to a second end corresponding to the expected future position of the mobile electronic device at the end of the time range, wherein the motion plan representation is continuously updated to reflect changes in the current position of the mobile electronic device and changes in the motion plan. The operations also include using the motion plan to control the mobile electronic device.

[0019] The ninth aspect of the present disclosure is a device, which includes a memory and one or more processors, which are configured to execute instructions stored in the memory. When the instruction is executed, the one or more processors determine a motion plan for the mobile electronic device and display an environment representation using a display, wherein the environment representation is a graphical representation of the environment surrounding the mobile electronic device. The instruction further causes the one or more processors to display a motion plan representation overlaid on the environment representation to the user, the motion plan representation indicating an area of ​​the environment in which the mobile electronic device can travel within a time range, the motion plan representation extending from a first end corresponding to the current position of the mobile electronic device to a second end corresponding to the expected future position of the mobile electronic device at the end of the time range, wherein the motion plan representation is continuously updated to reflect changes in the current position of the mobile electronic device and changes in the motion plan. The instruction further causes the one or more processors to control the mobile electronic device using the motion plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a graphic representation of the device.

[0021] Figure 2 is a block diagram illustrating the operation of the control system and the intent analyzer.

[0022] Figure 3 is a diagram showing the output of the intent analyzer.

[0023] Figures 4 to 6 is a block diagram of an exemplary process for intent indication.

[0024] Figure 7 is a block diagram of an exemplary computing device. DETAILED DESCRIPTION

[0025] Embodiments of the automated systems described herein may: analyze inputs to the control system to identify reasons why an action is being taken; or identify reasons why a first action is preferred over a second action to allow the system to present information to a user about expected future actions. In some implementations, the inputs are analyzed by a sensitivity analysis that perturbs the inputs over multiple iterations of the analysis to identify sensitivity inputs that are primarily responsible for selecting an action for a first action over a second action. By presenting information identifying an action to be taken by the automated system and by presenting information identifying reasons for taking an action, a person near the automated system will gain a higher degree of confidence about the automated system's actions.

[0026] Figure 1 102 is a diagram showing a device 100 operating in an environment 102. In the illustrated implementation, the device 100 includes a body 104 that defines an interior space 106 within the body 104. A passenger 108 is located in the passenger compartment 106 of the device 100. The passenger 108 may be referred to as a user. For example, when the device 100 transports the passenger 108 from a starting point (e.g., a first location) to a destination (e.g., a second location), the arrows represent the movement of the device 100 in the environment 102. The device 100 is located in the environment 102. The device 100 is a mobile electronic device. In some embodiments, the device 100 is a road-going vehicle supported by wheels and tires and is configured to carry passengers and / or cargo. Other objects may also be located in the environment, including static objects 103a (e.g., fixed obstacles, barriers, road surface defects, etc.) and dynamic objects 103b (e.g., people and vehicles).

[0027] The device 100 may include a control system 112, a sensor system 110, an actuator system 114, a human-machine interface device such as an interface 116, and an intent analyzer 118. These components may be attached to and / or form part of a body 104 or other physical structure of the device 100, and may be electrically interconnected to allow signals, data, commands, etc. to be sent between them via a wired connection (e.g., using a wired communication bus) or via a wireless data communication channel. Other components may be included in the device 100, such as conventional vehicle components, including chassis components, aesthetic components, suspension components, powertrain components, etc.

[0028] The sensor system 110 is configured to obtain information representing the state of the environment 102 and the device 100 for use by the control system 112. The sensor system 110 includes one or more sensor components configured to output information representing the characteristics (e.g., measurements, observations, etc.) of the device 100, the environment 102, the static objects 103a, and / or the dynamic objects 103b. Sensors that may be included as part of the sensor system 110 include, but are not limited to, imaging devices (e.g., visible spectrum static cameras, infrared spectrum static cameras, visible spectrum cameras, infrared spectrum cameras, etc.), three-dimensional sensors (e.g., lidar, radar sensors, depth cameras, structured light sensors, etc.), satellite positioning sensors, and inertial measurement units (e.g., outputting six degrees of freedom velocity and acceleration information). The information output by the sensors of the sensor system 110 may be in the form of sensor signals that can be interpreted as understanding the characteristics of the environment 102, the static objects 103a, and the dynamic objects 103b. The sensor signals obtained by the sensor system 110 may include a two-dimensional image and / or a three-dimensional scan (e.g., a point cloud, a depth image, etc.) of the environment. This information may be referred to as environmental information. Thus, the sensor system 110 may be configured to provide information to the control system 112 , such as observations of the environment 102 as sensed by the sensor system 110 and a current state of the device 100 as sensed by the sensor system 110 .

[0029] The control system 112 is configured to determine control decisions for the device 100 based on information from the sensor system 110 and / or other information. The control system 112 is configured to control the movement of the device 100 in an automatic control mode, and may implement other control modes, such as manual control and teleoperation control. In the automatic control mode, the control system 112 is configured to use information from the sensor system 110 and / or other information to make decisions about the movement of the device 100. In order to determine how to move the device 100, the control system 112 implements perception, motion planning, and control functions. These functions may be incorporated into hardware, firmware, and / or software systems.

[0030] The perception function of the control system 112 includes interpreting sensor outputs from the sensor system 110 to understand the environment 102 and objects in the environment 102, such as by generating a computer-interpretable representation of the environment 102 that can be used by the control system 112 during motion planning. The motion planning function of the control system 112 includes determining how to move the device 100 in order to achieve a goal, such as by moving along a route from a current location toward a destination location, as determined using the route planning function. The motion control function of the control system 112 includes determining actuator commands and sending the actuator commands to the actuator system 114 to cause the device 100 to move according to the decisions made by the motion planning function, such as by controlling the actuator system 114 in a manner that causes the device 100 to follow a trajectory determined by the motion planning function toward the destination location.

[0031] Various control algorithms now known or later developed may be used as the basis for the automatic control of the device 100 by the control system 112. The control system 112 may be implemented in the form of one or more computing devices that are provided with control software including computer program instructions that allow the control system 112 to perform the functions described above. In some specific implementations, the control system 112 uses the following reference Figure 7 The operation of the control system 112 will be further described herein.

[0032] The actuator system 114 is configured to cause movement of the device 100 and can be controlled by the control system 112 in an automatic control mode, for example, by sending actuator commands from the control system 112 to the actuator system 114 so that the actuator system 114 can operate according to the actuator commands. The actuator system 114 includes one or more actuator components that can affect the movement of the device 100. The actuator components can accelerate, decelerate, steer, or otherwise affect the movement of the device 100. These components can include suspension actuators, steering actuators, brake actuators, and propulsion actuators such as one or more electric motors.

[0033] The interface 116 is configured to present information to the user 108 (e.g., through an information display caused by the control system 112) and receive input from the user 108 (e.g., by sending a signal representing the input to the control system 112). The information presented to the user 108 through the interface 116 can be information about control decisions and / or other aspects of the operation of the control system 112. The input received by the control system 112 can be user input received from the user 108 for use by the control system 112. In order to present information and receive user input, the interface 116 includes components that allow the user to interact with various systems of the device 100, such as input devices and output devices. As examples, the interface 116 may include a display screen, a touch-sensitive interface, a gesture interface, an audio output device, a voice command interface, buttons, knobs, joysticks, control wheels, pedals, etc.

[0034] The intent analyzer 118 is configured to analyze the operation of the control system 112 and present information about the control system's operation of the device 100 to the user 108 using the interface 116. The operation of the intent analyzer 118 will be described further herein.

[0035] Figure 2 1 is a block diagram illustrating the operation of a system of the device 100 including the control system 112 and the intent analyzer 118. In the automatic control mode, the control system 112 uses input 220 to make control decisions. The input 220 includes information obtained from the sensor system 110, previously stored information obtained from a storage device 222 (e.g., a storage device) associated with the control system 112, and information representing user input obtained from the interface 116 due to the interaction of the user 108 with the interface 116. The control system 112 uses the input 220 to determine a motion plan 224 for the device 100. The motion plan 224 describes how the device 100 will move between a first position and a second position, such as between a current position and a position that the device 100 intends to be in the future (e.g., 10 to 20 seconds after the current time). The motion plan 224 may include a trajectory describing a path that the device 100 will take through the environment 102, and a velocity profile describing the speed at which the device 100 will move, such as by explicitly or implicitly describing the acceleration and deceleration of the device as the device 100 moves along the trajectory. Determination of motion plan 224 will be described further herein.

[0036] The input 220 obtained from the sensor system 110 can describe the environment 102, including the location of the static object 103a and the location and tracked motion of the dynamic object 103b. The input obtained from the interface 116 can include input from the user 108, such as the selection of the destination of the device 100. The input 220 obtained from the storage device 222 can include previously stored user preference information, for example, describing the comfort-related preferences of the user 108. The input 220 obtained from the storage device 222 can also include regulatory information input describing traffic rules. The input 220 obtained from the storage device 222 can also include navigation information input, such as map information, historical traffic conditions, and current traffic conditions. The input 220 can also include dynamic limitations for the device 100, such as the maximum speed and maximum acceleration (e.g., in up to six degrees of freedom) at which the device 100 can be operated by the control system 112, and this information can be obtained from the storage device 222 or otherwise available to the control system 112. The inputs 220 may also include factors such as cost factors that the control system 112 uses to determine a motion plan 224, calculate a score representing the compliance of the motion plan 224 with various criteria that may be used to judge the suitability of the motion plan 224, and compare two or more possible motion plans to determine which one is prioritized and will be used as the motion plan 224.

[0037] The control system 112 uses the input 220 as a basis for determining a motion plan 224 for the device. Determining the motion plan 224 may include determining the positions of the static objects 103a and the dynamic objects 103b by interpreting the output of the sensor system 110, and generating the motion plan in a manner consistent with traveling toward the destination location while complying with constraints. The constraints used in the generation of the motion plan 224 may include constraints defined by the input 220, such as obeying traffic regulations, moving the device 100 in a manner consistent with comfort-related preferences, and moving the device 100 in a manner that does not exceed dynamic limits.

[0038] The motion plan 224 can be determined by the control system 112 in a way that generates a score indicating how well the motion plan follows constraints and / or other performances. As an example, a function that gives a higher score for minimizing cost (e.g., travel time or fuel consumption), a higher score for increasing comfort, a lower score for violating constraints, etc. can be used to generate a score. As an example, the score can be determined as a function of multiple component scores, and each of the multiple component scores represents compliance with the desired conditions or compliance with the constraints (e.g., not violating the constraints). In some specific implementations, the desired conditions and constraints can be represented as factors such as cost factors. Therefore, a higher score can correspond to a motion plan that is considered to have priority over a motion plan with a lower score, thereby allowing the motion plan to be ranked. The motion plan 224 can be a motion plan in a plurality of alternative motion plans determined at a specific time point, and each motion plan in the plurality of motion plans can be associated with a score, thereby allowing a plurality of motion plans to be ranked, and allowing the highest score motion plan from a plurality of motion plans to be designated as a priority motion plan.

[0039] Intent analyzer 118 is configured to analyze motion plan 224 and input 220 and generate output 226 that can be displayed to user 108 via interface 116 (e.g., on a display screen associated with device 100 and accessible to user 108) to allow user 108 to understand motion plan 224 and the reasons for motion plan 224. Output 226 can include environment representation 228, motion plan representation 230, and intent indication 232.

[0040] Figure 3 226 is a schematic diagram of the output 226 being presented in graphical form on the display 316 of the interface 116. In the illustrated implementation, the environment representation 228, the movement plan representation 230, and the intention indication 232 are graphical elements that are combined and output for display. The vehicle indicator 334 is also presented on the display 316 (e.g., overlaid on the environment representation 228) at a location relative to the environment representation 228 that represents the current location of the device 100, so as to show the user 108 the location of the device 100 relative to the environment 102.

[0041] Output 226 may include an environment representation 228, which is a graphical representation of environment 102. Environment representation 228 is in a graphical form so that it can be combined with other graphical elements and presented to user 108, such as on a display screen included in interface 116 of device 100. The purpose of environment representation 228 is to provide a context for presenting information about motion plan 224 and the reasons for control decisions made by control system 112, and therefore, environment representation 228 may be in any suitable form that meets this purpose. Environment representation 228 may be generated using stored information (e.g., map information), images from an image sensor of sensor system 110, three-dimensional scans from a three-dimensional sensor of sensor system 110, information from other sources, or a combination thereof. As an example, environment representation 228 may be a map, which, for example, includes lines representing roads and / or lanes of travel in an area in which device 100 is traveling. As another example, environment representation 228 may be a three-dimensional representation of environment 102 surrounding device 100. As another example, environment representation 228 may be an image (eg, an image from a single camera or a composite image generated from multiple images obtained from multiple cameras) showing environment 102 surrounding device 100 .

[0042] The motion plan representation 230 is information describing the motion plan 224, and can be continuously updated during operation of the device 100 (e.g., at fixed time intervals) to reflect changes in the current location of the device 100 and changes in the motion plan 224. In the illustrated implementation, the motion plan representation is a graphical representation of the motion plan 224. The motion plan representation 230 can be a graphical indicator of the motion plan 224, which is overlaid on the environment representation 228 to show the position and range of the motion plan representation 230 relative to the environment representation 228. In order to allow the user to understand how the device 100 may move in the future, the motion plan representation 230 can have a shape and range corresponding to the expected movement of the device 100 according to the motion plan 224. As an example, the shape and range of the motion plan representation 230 can indicate an area of ​​the environment in which the device 100 can travel within a time range, and the motion plan representation can extend from a first end 331a corresponding to the current location of the device 100 to a second end 331b corresponding to the expected future location of the device 100 at the end of the time range.

[0043] The time range can be a fixed time interval that extends from the current time to a future time corresponding to the end of the fixed time interval. Therefore, the end of the time range can be determined by adding the fixed duration time interval to the current time. As an example, if the time range is eight seconds, the motion plan representation 230 will always show where the device 100 will be in the next eight seconds, and the second end 331b of the motion plan representation 230 corresponds to the location where the device 100 will be in the next eight seconds. Over time, the time range remains fixed, and the motion plan representation 230 will continue to represent the operation of the device 100 for the next eight seconds (or other fixed time interval).

[0044] By updating the motion plan representation 230 as the device 100 moves, the motion plan representation will always be updated to show where the device 100 will be at a future point in time corresponding to the end of the time range. The length of the motion plan representation 230 between the first end 331a of the motion plan representation and the second end 331b of the motion plan representation 230 represents the expected travel distance of the device 100 during the time range. Because the time range is a fixed duration time interval, the length of the motion plan representation 230 also varies according to the average speed of the device 100 during the time range. As an example, when the device 100 stops and will remain stopped for a period of time longer than the time range, the length of the motion plan representation 230 can be reduced until it reaches zero length or reaches a minimum length set to indicate that there is no movement during the time range. The length of the motion plan representation 230 will begin to increase before the device 100 resumes motion.

[0045] The motion plan representation 230 may include a graphical style for indicating information about the motion of the device 100 during the time period. The appearance of all or part of the motion plan representation 230 may be changed, such as by changing the color or by applying a dynamic graphic effect, so as to indicate the upcoming aspects of the motion of the device 100. As an example, a change in acceleration (e.g., longitudinal acceleration or lateral acceleration) within a time range may be indicated by changing the color of the motion plan representation 230 or by otherwise changing the appearance of the motion plan representation 230. In some implementations, the color of portion 331c of the graphical indicator may be changed to indicate the acceleration of the device 100 during the time range, wherein the range of portion 331c corresponds to the spatial range or time range over which the acceleration is expected. Thus, a short period of time in which the acceleration of the device 100 changes by more than a threshold value may be indicated by the color of portion 331c, and the color of portion 331c may also vary according to the magnitude of the acceleration.

[0046] Output 226 may include a second motion plan representation 331d corresponding to a second motion plan, which is determined by control system 112 as a substitute for motion plan 224 (which may be referred to as a first motion plan). As an example, a first motion plan may correspond to a first expected path of travel around an obstacle, and a second motion plan may correspond to a second expected path of travel different from the first path of travel. In the illustrated implementation, second motion plan representation 331d shows travel in different lanes of travel on a road compared to motion plan representation 230 (e.g., first motion plan representation). Second motion plan representation 331d may be equivalent to motion plan representation 230, but is presented in different colors, opacity or other graphic styles to distinguish it. In some implementations, interface 116 may be configured to receive an input from user 108 requesting use of a second motion plan corresponding to second motion plan representation 331d.

[0047] The intention indication 232 includes information indicating to the user 108 why the device 100 is taking an action. The intention indication 232 can be in the form of explanatory text, in the form of an icon, or in another form representing the cause of the action. In order to generate the intention indication 232, the intention analyzer 118 is configured to identify the reasons why certain actions are taken as part of the motion plan 224 and / or identify why the motion plan 224 takes precedence over an alternative motion plan (e.g., why the first motion plan takes precedence over the second motion plan). In a specific implementation, the intention analyzer 118 may search for nearby objects that may have affected the motion plan 224, such as static objects 103a and dynamic objects 103b, determine how the presence of these objects may affect the motion plan 224, and combine the information describing how the presence of these objects may affect the motion plan 224 into the intention indication 232. This can be performed, for example, using a rule-based method that considers the current position and current state of the device 100 relative to the current position and current state of the static objects 103a and dynamic objects 103b to determine possible explanations for the motion plan 224. In another specific implementation, intent analyzer 118 may search for conditions near device 100, such as current traffic conditions, detours, or construction activity that may have affected motion plan 224, determine those conditions may have affected motion plan 224, and incorporate information describing how those conditions may have affected motion plan 224 into intent indication 232. This may be performed, for example, using a rule-based approach that considers the current location and current state of device 100 relative to conditions that affect a transportation network (e.g., streets) near device 100.

[0048] In some implementations, the intent analyzer 118 is configured to perform a sensitivity analysis in order to identify reasons why certain actions are taken as part of the motion plan 224 and / or to identify why the motion plan 224 is preferred over an alternative motion plan (e.g., why a first motion plan is preferred over a second motion plan). The sensitivity analysis performed by the intent analyzer 118 is intended to determine which of the inputs 220 are sensitive inputs that have a significant impact on the motion plan 224. A sensitive input is an input that, if changed, would result in a significantly different outcome for the motion plan 224, such as changing the route that the device 100 is traveling, stopping as opposed to not stopping, accelerating as opposed to decelerating, changing lanes as opposed to staying in the current lane, turning as opposed to taking no action, and the like. As an example, an input may be identified as a sensitive input if changing the value of the input would result in a difference in the motion plan 224 (e.g., in acceleration rate or position) that exceeds a predetermined magnitude, or the input is a type that has been identified as corresponding to a sensitive (e.g., predetermined as a category that is considered to indicate a difference in a sensitive input).

[0049] The intent analyzer can use many known methods to perform sensitivity analysis. Some methods include changing one or more of inputs 220 to understand how inputs 220 affect motion plan 224 (e.g., how motion plan 224 will change if the inputs are different). If a non-sensitive input is changed, it will not result in a change in the motion plan, or it will result in a slight but insignificant difference in the motion plan (e.g., a tracking difference within a lane, a difference in acceleration rate or deceleration rate below a comfort threshold or perceptibility threshold, etc.).

[0050] In one specific implementation, the intent analyzer 118 may perform a sensitivity analysis by executing multiple iterations of a motion planning process for determining the motion plan 224. For each iteration of the motion planning process performed as part of the sensitivity analysis, the resulting motion plan is determined by changing one of the inputs 220 to determine whether the input is a sensitive input. As an example, an input may be identified as sensitive if changing the input changes the motion plan 224. The magnitude of the change may be quantified, such as by using a formula that assigns a numerical value to the difference between the motion plan 224 and the motion plan obtained from the sensitivity analysis. As an example, if the numerical value representing the magnitude of the change is above a threshold, one of the inputs 220 may be identified as sensitive. As another example, if the numerical value representing the magnitude of the change is greater than a value representing the magnitude of the change resulting from the analysis of other inputs in the input, one of the inputs 220 may be identified as sensitive.

[0051] The intention analyzer 118 can use sensitivity analysis to compare the motion plan 224 with an alternative motion plan, which can be referred to as a first motion plan and a second motion plan. The control system 112 determines the first motion plan and the second motion plan, and also determines a first score for the first motion plan and a second score for the second motion plan. The first score and the second score represent the suitability of the first motion plan and the second motion plan. The first score for the first motion plan is higher than the second score for the second motion plan, thereby indicating that the first motion plan takes precedence over the second motion plan. After determining the scores for the first motion plan and the second motion plan, the intention analyzer 118 performs sensitivity analysis to identify sensitive inputs that explain why the first motion plan takes precedence over the second motion plan. In multiple iterations of the sensitivity analysis, the input 220 used to determine the first motion plan is slightly changed, and scores are determined for each motion plan in the changed motion plan. In this example, the sensitive input makes the first score for the first motion plan higher than the second score for the second motion plan. The modification of the sensitive input can make the score for the modified version of the first motion plan lower than the score for the second motion plan, thereby identifying the sensitive input. Thus, by identifying sensitive inputs, the sensitivity analysis identifies one of the inputs 220 as a reason why the first motion plan is preferred over the second motion plan.

[0052] Based on the identification of the sensitive input, the intent analyzer 118 may generate an intent indication 232 such that it explains why the first motion plan is prioritized over the second motion plan, such as by generating text that identifies the sensitive input or the circumstances associated with the sensitive input as the reason why the first motion plan is prioritized over the second motion plan. As an example, the sensitive input may be associated with occupant comfort, and the intent indication 232 may include text or an icon indicating that the motion plan 224 is selected to increase occupant comfort. As another example, the sensitive input may be associated with avoiding a road defect (e.g., a road pit or other feature), and the intent indication 232 may include text or an icon indicating that the motion plan 224 is selected to travel around the road defect. As another example, the sensitive input may be associated with object avoidance, and the intent indication 232 may include text or an icon indicating that the motion plan 224 is selected to travel around the object. As another example, the sensitive input may be associated with travel time, and the intent indication 232 may include text or an icon indicating that the motion plan 224 is selected to reduce travel time.

[0053] In some cases, motion plan 224 includes motion maneuvers intended to circumvent features of environment 102 (referred to herein as environmental features, such as objects or road imperfections), and input 220 is analyzed by intent analyzer 118 to identify environmental features that cause the motion maneuvers to be included in motion plan 224. The analysis of input 220 may be a sensitivity analysis as previously described. Information describing the motion maneuvers and identifying the environmental features may then be included in intent indication 232, for example, in the form of explanatory text or icons.

[0054] The device 100 is configured to implement a process for intent indication, as will be explained herein with reference to example embodiments. The process described herein may be performed using a system implemented using one or more computing devices, such as a control system 112 of the device 100 and an intent analyzer 118, which may be used Figure 7 The computing device 760 of the present invention is implemented. As an example, the processes and operations described herein and the operations thereof may be implemented in the form of methods implemented using the device 100 and its various systems. As an example, the processes and operations described herein and the operations thereof may be implemented in the form of an apparatus including a memory and one or more processors configured to execute computer program instructions. The computer program instructions are executable by one or more computing devices so that the one or more computing devices perform functions corresponding to the steps of the process. As an example, the processes and operations described herein and the operations thereof may be implemented in the form of a non-transitory computer-readable storage device, which includes program instructions that can be executed by one or more processors, which when executed cause the one or more processors to perform operations corresponding to the steps of the process.

[0055] Figure 4 4 is a block diagram of a process 450 for intent indication. Process 450 may be performed by device 100, including through operation of control system 112 and intent analyzer 118 as previously described. Process 450 may be performed while device 100 is traveling from a current location to a destination location, and may include transporting user 108 by device 100. Figures 1 to 3 The described features may be combined in process 450 .

[0056] In operation 451, process 450 includes determining a motion plan 224 for device 100. Motion plan 224 may be determined in a manner consistent with travel from a current location toward a destination location of device 100. Motion plan 224 may be determined as described with respect to control system 112 and may be used to control actuator system 114 of device 100.

[0057] At operation 452, process 450 includes and displays to user 108 a graphical representation of environment 102, such as environment representation 228. Environment representation 228 may be output to display 316 or another display device associated with interface 116 of device 100. As examples, the environment representation may be or include a map representing environment 102, a three-dimensional rendering of environment 102 based on information from sensor system 110, or an image of environment 102 obtained from sensor system 110.

[0058] In operation 453, process 450 includes displaying to the user a graphical indication of a motion plan (such as motion plan representation 230) overlaid on a graphical representation of the environment (such as environment representation 228). In operation 453, motion plan representation 230 indicates an area of ​​environment 102 in which device 100 may travel within the time range and may extend from a first end 331a corresponding to a current position of device 100 to a second end 331b corresponding to an expected future position of device 100 at the end of the time range. Motion plan representation 230 is continuously updated to reflect changes to the current position of device 100 and changes to motion plan 224.

[0059] In operation 453, the end of the time range may be determined by adding a fixed duration time interval to the current time. The length of the motion plan representation 230 between the first end 331a and the second end 331b represents the expected travel distance of the device 100 during the time range, and the length of the motion plan representation 230 may be continuously updated to reflect changes to the current position of the device 100 and to reflect changes to the motion plan 224. In operation 453, the motion plan representation 230 may be output such that the color of at least a portion of the motion plan representation 230, such as portion 331c, represents the acceleration of the device 100 during the time range.

[0060] Some specific implementations of process 450 include determining a second motion plan in operation 451, and displaying a second graphical indication of the second motion plan (such as second motion plan representation 331d) overlaid on environment representation 228 to user 108 in operation 453. The motion plan representation 230 and the second motion plan representation 331d may be displayed in at least one of different colors or different opacities.

[0061] In operation 454, process 450 includes using the motion plan to control device 100. Operation 454 may include sending actuator commands and / or other information from control system 112 to actuator system 114 to cause actuator system 114 to operate actuators of device 100 in a manner that causes motion of device 100 to be consistent with the motion plan.

[0062] Figure 5 is a block diagram of a process 550 for intent indication. Process 550 may be performed by device 100, including through operation of control system 112 and intent analyzer 118 as previously described. Process 550 may be performed while device 100 is traveling from a current location to a destination location, and may include transporting user 108 by device 100. Figures 1 to 3 The described features may be combined in process 550 .

[0063] Operation 551 includes determining a first motion plan and a second motion plan for device 100 based on input 220. The first motion plan and the second motion plan are equivalent to motion plan 224 and can be determined in the manner described with respect to motion plan 224. Operation 552 includes determining to prioritize the first motion plan relative to the second motion plan. As an example, determining to prioritize the first motion plan relative to the second motion plan may include determining a first score for the first motion plan and a second score for the second motion plan. The first score represents the suitability of the first motion plan, and the second score represents the suitability of the second motion plan. In this example, when the first score is higher than the second score (which indicates that the first motion plan takes precedence over the second motion plan), it is determined to prioritize the first plan relative to the second plan.

[0064] Operation 553 includes identifying one of the inputs used in operation 551 to determine the first motion plan and the second motion plan as a sensitive input, which causes the first motion plan to be determined to take precedence over the second motion plan in operation 552. In order to identify the sensitive input, operation 553 may include performing a sensitivity analysis to identify one of the inputs 220 as a sensitive input that causes a first score for the first motion plan to be higher than a second score for the second motion plan. Operation 553 may be implemented in the manner described with respect to the intent analyzer 118. The sensitivity analysis may include modifying at least some of the inputs 220 and recalculating the first score for the first motion plan based on the modified input. The modification of the sensitive input may cause the first score for the first motion plan to be reduced so that it is lower than the second score for the second motion plan, thereby identifying one of the inputs as a sensitive input. As an example, the sensitive input may be associated with occupant comfort, road defect avoidance, obstacle avoidance, travel time, or other circumstances.

[0065] Operation 554 includes presenting information describing the first motion plan to the user 108 using a display such as the display 316, wherein the information includes an explanation indicating the sensitive input as a reason why the first motion plan takes precedence over the second motion plan. Operation 554 may include presenting information describing the first motion plan such as the motion plan representation 230, and including an indication such as the intent indication 232. The intent indication 232 is based on the sensitive input and explains why the first motion plan takes precedence over the second motion plan. The indication explaining why the first motion plan takes precedence over the second motion plan includes text determined based on the sensitive input, or may include an icon representing the sensitive input.

[0066] In operation 554, presenting information describing the first motion plan may include displaying a first graphical indicator of the first motion plan, such as motion plan representation 230, and a second graphical indicator of the second motion plan, such as second motion plan representation 331 d. The motion plan representation 230 and the second motion plan representation 331 d may be displayed with different visual characteristics, such as at least one of different colors or different opacities.

[0067] In operation 555, process 550 includes controlling device 100 using one of the first motion plan or the second motion plan. Operation 555 may include sending actuator commands and / or other information from control system 112 to actuator system 114 to cause actuator system 114 to operate actuators of device 100 in a manner that causes movement of device 100 to be consistent with the motion plan.

[0068] Figure 6 6 is a block diagram of a process 650 for intent indication. The process 650 may be performed by the device 100, including through the operation of the control system 112 and the intent analyzer 118 as previously described. The process 650 may be performed while the device 100 is traveling from a current location to a destination location, and may include transporting the user 108 by the device 100. Figures 1 to 3 The described features may be combined in process 650 .

[0069] Operation 651 includes determining a motion plan 224 for the device 100 based on the input 220, wherein the motion plan 224 includes a motion maneuver. As an example, the motion maneuver may include a planned motion of the device 100 that is intended to avoid contact with a road defect, obstacle, or other object. The motion plan 224 may be determined as previously described with respect to the control system 112.

[0070] Operation 652 includes analyzing input 220 to identify environmental features (e.g., features in environment 102 located around device 100) that cause the motion maneuver to be included in motion plan 224. As an example, the environmental feature may be one of static objects 103a or one of dynamic objects 103b. Analyzing input 220 to identify environmental features that cause the motion maneuver to be included in motion plan 224 may include identifying the environmental features by performing a sensitivity analysis in the manner previously described with reference to intent analyzer 118.

[0071] Operation 653 includes presenting information describing the motion maneuver and identifying the environmental features, such as the motion plan representation 230 and the indication of intent 232, to the user 108. The information describing the motion maneuver and identifying the environmental features may include text identifying the environmental features, which may be included as part of the indication of intent 232. The information describing the motion maneuver and identifying the environmental features may include icons representing the environmental features. The information describing the motion maneuver and identifying the environmental features may include a graphical indication, such as the motion plan representation 230, that represents the intended path of the motion maneuver.

[0072] In operation 654, process 650 includes controlling motion plan 224 of device 100. Operation 654 may include sending actuator commands and / or other information from control system 112 to actuator system 114 to cause actuator system 114 to operate actuators of device 100 in a manner that causes motion of device 100 to be consistent with the motion plan.

[0073] Figure 7 760 is a block diagram of a computing device 760 according to one example. The computing device 760 may be used as a basis for implementing the computer-based systems described herein, such as the control system 112 and the intent analyzer 118. In the illustrated example, the computing device 760 includes a processor 761, a memory 762, a storage device 763, and a communication device 764. The computing device 760 may include other components, such as input devices and output devices.

[0074] Processor 761 can be in the form of one or more conventional devices and / or one or more special-purpose devices configured to execute computer program instructions. Examples of processor 761 include one or more central processing units, one or more graphics processing units, one or more application-specific integrated circuits, and / or one or more field programmable gate arrays. Memory 762 can be a conventional short-term storage device, such as a random access memory module, that stores information used by processor 761. Storage device 763 is a non-volatile long-term storage device that can be used to store computer program instructions and / or other data, such as a flash memory module, a hard drive, or a solid-state drive. Communication device 764 allows communication with other systems using any manner of wired or wireless interface suitable for sending and receiving signals encoding data.

[0075] The computing device 760 is operable to store, load and execute computer program instructions. When executed by the computing device 760, the computer program instructions cause the computing device to perform operations. The computing device 760 can be configured to obtain information, such as by accessing information from a storage device, accessing information from a short-term memory, receiving a wired transmission or wireless transmission including information, receiving a signal representing a user input from an input device, and receiving a signal representing an observation performed by a sensor from a sensor. The computing device 760 can be configured to make a determination, such as by comparing a value with a threshold, comparing a state with a condition, using a formula to evaluate one or more input values, using an algorithm to evaluate one or more input values, and / or using any type of data to calculate. The computing device 760 can be configured to send information, such as by using a data bus to send information between components or using a wired data connection or a wireless data connection between systems. The computing device 760 can be configured to output a signal to control a component such as a sensor or an actuator. As an example, the signal can cause a sensor to obtain data and provide the data to the computing device 760. As another example, the signal can cause the movement of an actuator.

[0076] As described above, one aspect of the technology of the present invention is to collect and use data available from various sources to display robot intentions. Although the present invention does not require the use of personal information data, it should be noted that information such as those stored in the user profile and / or the user's intended destination can be used to benefit the user. For example, a user profile storing user preferences can be established, which controls the type of information presented to the user, the amount of information presented to the user, and the manner in which the information is presented. Therefore, the use of such personal information data enhances the user's experience. Within the scope of using personal information data, implementers should comply with a well-established privacy policy and / or privacy practice that is recognized to meet or exceed industry or government requirements to maintain the privacy and security of personal information data. The collection and / or sharing of personal information data should be carried out with the informed consent of the user, and the user should be allowed to opt out. In addition, measures should be taken to protect and ensure access to such stored information.

Claims

1. A method, comprising: determining a first motion plan and a second motion plan for the mobile electronic device based on the input; determining to prioritize the first motion plan relative to the second motion plan; identifying one of the inputs as a sensitive input that prioritizes the first motion plan over the second motion plan; presenting, using a display, information describing the first motion plan, wherein the information includes an explanation indicating the sensitive input as a reason why the first motion plan takes precedence over the second motion plan; as well as The mobile electronic device is controlled using one of the first motion plan or the second motion plan. 2 . The method of claim 1 , wherein the interpretation comprises text determined based on the sensitive input. The method of claim 1 , wherein the interpretation comprises an icon representing the sensitive input. The method of claim 1 , wherein the sensitive input is associated with occupant comfort. The method of claim 1 , wherein the sensitive input is associated with travel time. 6 . The method of claim 1 , wherein the first motion plan includes a first expected path of travel around an obstacle, and the second motion plan includes a second expected path of travel different from the first expected path of travel.

7. The method of claim 1, wherein presenting the information describing the first motion plan comprises displaying a first motion plan representation of the first motion plan and a second motion plan representation of the second motion plan.

8. A non-transitory computer readable storage device comprising program instructions executable by one or more processors, the program instructions when executed causing the one or more processors to perform operations comprising: determining a first motion plan and a second motion plan for the mobile electronic device based on the input; determining to prioritize the first motion plan relative to the second motion plan; identifying one of the inputs as a sensitive input that prioritizes the first motion plan over the second motion plan; presenting, using a display, information describing the first motion plan, wherein the information includes an explanation indicating the sensitive input as a reason why the first motion plan takes precedence over the second motion plan; as well as The mobile electronic device is controlled using one of the first motion plan or the second motion plan.

9. The non-transitory computer-readable storage device of claim 8, wherein the interpretation comprises text determined based on the sensitive input.

10. The non-transitory computer-readable storage device of claim 8, wherein the interpretation comprises an icon representing the sensitive input. 11 . The non-transitory computer readable storage device of claim 8 , wherein the sensitive input is associated with occupant comfort.

12. The non-transitory computer readable storage device of claim 8, wherein the sensitive input is associated with a travel time.

13. The non-transitory computer readable storage device of claim 8, wherein the first motion plan includes a first expected path of travel around an obstacle, and the second motion plan includes a second expected path of travel different from the first expected path of travel.

14. The non-transitory computer-readable storage device of claim 8, wherein presenting the information describing the first motion plan comprises displaying a first motion plan representation of the first motion plan and a second motion plan representation of the second motion plan.

15. A device, comprising: Memory; and one or more processors configured to execute instructions stored in the memory, wherein the instructions, when executed, cause the one or more processors to: determining a first motion plan and a second motion plan for the mobile electronic device based on the input; determining to prioritize the first motion plan relative to the second motion plan; identifying one of the inputs as a sensitive input that prioritizes the first motion plan over the second motion plan; presenting, using a display, information describing the first motion plan, wherein the information includes an explanation indicating the sensitive input as a reason why the first motion plan takes precedence over the second motion plan; as well as The mobile electronic device is controlled using one of the first motion plan or the second motion plan.

16. The apparatus of claim 15, wherein the interpretation comprises text determined based on the sensitive input. The apparatus of claim 15 , wherein the interpretation comprises an icon representing the sensitive input.

18. The apparatus of claim 15, wherein the sensitive input is associated with occupant comfort. The apparatus of claim 15 , wherein the sensitive input is associated with travel time.

20. The apparatus of claim 15, wherein the first motion plan includes a first expected path of travel around an obstacle, and the second motion plan includes a second expected path of travel different from the first expected path of travel.

21. The apparatus of claim 15, wherein presenting the information describing the first motion plan comprises displaying a first motion plan representation of the first motion plan and a second motion plan representation of the second motion plan.

Images