Method and apparatus for dynamically modifying the presentation of an autonomous robot

Abstract

A system for modifying a representation of a component is disclosed. The system implements a process for obtaining a first representation of the component and a second representation of the component. The system builds one or more intermediate representations of the component based on the component transitioning between the first representation and the second representation. The system performs a task, activity, movement, etc. based on the one or more intermediate representations. The system shares the one or more intermediate representations of the component with other components to avoid collisions with the other components. The system includes a networking program for streaming the representations to a computing device. The computing device includes the networking program for receiving the representations streamed from the system via the networking program and a user interface (UI) application for displaying the representations.

Description

[Technical Field]

[0001] The present disclosure relates generally to dynamic modification of a robot's representation, such as dynamic modification of an autonomous robot's contour. [Background technology]

[0002] Robots are used to perform and / or automate tasks. Robots contain several electronics that enable the robot to perform various actions. Computing devices communicate with the robot and cause the robot to perform tasks. The robot performs tasks based on a representation (e.g., a software representation) of the robot. Summary of the Invention

[0003] One aspect of the present disclosure is a method for modifying a representation of a robot. The method includes obtaining a first representation of the robot corresponding to a first shape of the robot before a motion transition. The method may further include obtaining a second representation of the robot corresponding to a predicted second shape of the robot after the motion transition. The method may further include determining a first bounding box for the first representation. The method may further include determining a second bounding box for the second representation. The method may further include determining a first intermediate representation having the same general shape as the second representation and scaled to fit within the first bounding box. The method may further include determining multiple additional intermediate representations that are incrementally scaled between the first intermediate representation and the second representation.

[0004] In another aspect of the present disclosure, the method further includes modifying the movement of the robot based on one or more of the intermediate representations.

[0005] In another aspect of the disclosure, the method further includes causing at least one of the first representation, the second representation, the first intermediate representation, or the intermediate representation of the plurality of additional intermediate representations to be displayed.

[0006] In another aspect of the disclosure, the method further includes causing a visualization to be displayed that identifies at least one of the transformation from the first representation to the first intermediate representation, the transformation from the first intermediate representation to a plurality of additional intermediate representations, or the transformation from the plurality of additional intermediate representations to the second representation.

[0007] In another aspect of the present disclosure, the first representation, the second representation, the first intermediate representation, and each of the plurality of additional intermediate representations comprises a software representation of a robot.

[0008] In another aspect of the present disclosure, the robot includes an autonomous robot.

[0009] In another aspect of the present disclosure, the first representation, the second representation, the first intermediate representation, and the plurality of additional intermediate representations each include a polygon.

[0010] In another aspect of the disclosure, the method further includes performing collision avoidance based on the first intermediate representation and the plurality of additional intermediate representations.

[0011] In another aspect of the disclosure, the method further includes performing collision avoidance based on the first intermediate representation, the plurality of additional intermediate representations, and the plurality of intermediate representations of the second robot.

[0012] In another aspect of the disclosure, the method further includes performing obstacle avoidance based on the first intermediate representation and the plurality of additional intermediate representations.

[0013] In another aspect of the present disclosure, the first representation is based on data associated with the robot and the second representation is based on data associated with the user computing device.

[0014] In another aspect of the present disclosure, the method further includes acquiring first expression data from the robot. The method may further include generating a first expression based on the first expression data. The method may further include acquiring second expression data from a user computing device. The method may further include generating a second expression based on the second expression data.

[0015] In another aspect of the present disclosure, the method further includes providing at least one of the first representation, the second representation, the first intermediate representation, or the intermediate representation of the plurality of additional intermediate representations to a second robot.

[0016] In another aspect of the present disclosure, the method further includes identifying one or more robots based on at least one of the locations of the one or more robots or the locations of the robots. The locations of the one or more robots may be within a certain vicinity of the location of the robot. The method may further include providing at least one of the first representation, the second representation, the first intermediate representation, or the intermediate representation of the plurality of additional intermediate representations to each of the one or more robots.

[0017] In another aspect of the disclosure, the method further includes identifying one or more robots based on at least one of the positions of the one or more robots or the locations of the robots. The method may further include obtaining at least a third representation from each of the one or more robots.

[0018] In another aspect of the present disclosure, the plurality of additional intermediate representations may include five intermediate representations.

[0019] In another aspect of the present disclosure, the method may further include identifying a time period. The method may further include periodically obtaining or providing a representation of the robot based on the time period.

[0020] In another aspect of the present disclosure, the method may further include identifying a first period for a transition from the first representation to the second representation. The method may further include identifying a second period. The second period indicates an intermediate representation period. The method may further include identifying a number of steps based on the first period and the second period. The method may further include generating one or more additional intermediate representations based on the number of steps.

[0021] In another aspect of the present disclosure, the method may further include scaling the first intermediate representation to generate a plurality of additional intermediate representations.

[0022] One aspect of the present disclosure is a method for modifying a representation of a component. The method may include obtaining a first representation and a second representation of the component. The method may further include determining a first bounding box of the first representation and a second bounding box of the second representation. The method may further include determining one or more intermediate representations between the first representation and the second representation based on the first bounding box and the second bounding box. The method may further include causing movement of the component based on the one or more intermediate representations.

[0023] One aspect of the present disclosure is a system capable of performing any of the elements of the methods described above.

[0024] One aspect of the present disclosure is a system for modifying a representation of a component. The system may include at least one processor and a memory storing computer-executable instructions. Execution of the computer-executable instructions by the at least one processor may cause the at least one processor to obtain a first representation and a second representation of the component. Execution of the instructions may further cause the at least one processor to determine a first bounding box of the first representation and a second bounding box of the second representation. Execution of the instructions may further cause the at least one processor to determine one or more intermediate representations between the first representation and the second representation based on the first bounding box and the second bounding box. Execution of the instructions may further cause the at least one processor to operate the component based on the one or more intermediate representations.

[0025] In another aspect of the present disclosure, execution of the instructions further causes the at least one processor to display at least one of the first representation, the second representation, or the intermediate representation of the one or more intermediate representations.

[0026] In another aspect of the present disclosure, execution of the instructions further causes the at least one processor to display a visualization identifying a transformation from the first representation to one or more intermediate representations and a transformation from the one or more intermediate representations to a second representation.

[0027] In another aspect of the present disclosure, the first representation, the second representation, and the one or more intermediate representations each include an outline or shape of the component.

[0028] In another aspect of the disclosure, the first representation, the second representation, and the one or more intermediate representations each include a software representation of a component.

[0029] In another aspect of the present disclosure, each of the first representation, the second representation, and the one or more intermediate representations mimics the physical shape of the component.

[0030] In another aspect of the present disclosure, the first representation, the second representation, and the one or more intermediate representations each include a polygon.

[0031] In another aspect of the present disclosure, the component includes a mobile robot.

[0032] In another aspect of the present disclosure, execution of the instructions further causes the at least one processor to perform collision avoidance based on the one or more intermediate representations.

[0033] In another aspect of the present disclosure, execution of the instructions further causes the at least one processor to perform collision avoidance of the component based on the one or more intermediate representations of the second component and the one or more second intermediate representations.

[0034] In another aspect of the disclosure, execution of the instructions may further cause the at least one processor to perform collision avoidance of the component based on the one or more second intermediate representations of the second component.

[0035] In another aspect of the disclosure, execution of the instructions further causes the at least one processor to perform obstacle avoidance based on the one or more intermediate representations.

[0036] In another aspect of the disclosure, the first representation, the second representation, and the one or more intermediate representations each include a representation of the component at a particular time period.

[0037] In another aspect of the present disclosure, the first representation includes a representation of the component in a first time period, and the second representation includes a representation of the component in a second time period, where the first time period may occur before the second time period.

[0038] In another aspect of the present disclosure, the first representation shows the current shape of the component and the second representation shows the modified shape of the component.

[0039] In another aspect of the present disclosure, the first representation is based on data associated with the component and the second representation is based on data associated with the user computing device.

[0040] In another aspect of the present disclosure, execution of the instructions may further cause the at least one processor to obtain first representation data from a component. Execution of the instructions may further cause the at least one processor to generate a first representation based on the first representation data. Execution of the instructions may further cause the at least one processor to obtain second representation data from a user computing device. Execution of the instructions may further cause the at least one processor to generate a second representation based on the second representation data.

[0041] In another aspect of the present disclosure, execution of the instructions further causes the at least one processor to provide at least one of the first representation, the second representation, or the intermediate representation of the one or more intermediate representations to a second component.

[0042] In another aspect of the present disclosure, execution of the instructions may further cause the at least one processor to identify one or more components, and may further cause the at least one processor to provide at least one of the first representation, the second representation, or the intermediate representation of the one or more intermediate representations to each of the one or more components.

[0043] In another aspect of the disclosure, execution of the instructions may further cause the at least one processor to identify one or more components based on at least one of the component locations or the component locations. Execution of the instructions may also cause the at least one processor to provide at least one of the first representation, the second representation, or the intermediate representation of the one or more intermediate representations for each of the one or more components.

[0044] In another aspect of the disclosure, execution of the instructions may further cause the at least one processor to identify one or more components based on at least one of the component locations or the component locations, where the component locations may be within a particular vicinity of the component location. Execution of the instructions may also cause the at least one processor to provide at least one of the first representation, the second representation, or the intermediate representation of the one or more intermediate representations to each of the one or more components.

[0045] In another aspect of the disclosure, execution of the instructions further causes the at least one processor to obtain a third representation from the second component.

[0046] In another aspect of the disclosure, execution of the instructions may further cause the at least one processor to identify one or more components. Execution of the instructions may further cause the at least one processor to obtain at least a third representation from each of the one or more components.

[0047] In another aspect of the disclosure, execution of the instructions may further cause the at least one processor to identify one or more components based on at least one of the location of the one or more components or the location of the components. Execution of the instructions may also cause the at least one processor to obtain at least a third representation from each of the one or more components.

[0048] In another aspect of the disclosure, execution of the instructions may further cause the at least one processor to identify the one or more components based on at least one of the locations of the one or more components or the locations of the components, where the locations of the one or more components may be within a particular vicinity of the location of the component. Execution of the instructions may further cause the at least one processor to obtain at least a third representation from each of the one or more components.

[0049] In another aspect of the present disclosure, the one or more intermediate representations include five intermediate representations.

[0050] In another aspect of the present disclosure, execution of the instructions further causes the at least one processor to periodically provide at least one of the first representation, the second representation, or the intermediate representation of the one or more intermediate representations to each of the one or more components.

[0051] In another aspect of the disclosure, execution of the instructions further causes the at least one processor to periodically obtain at least a third representation from each of the one or more components.

[0052] In another aspect of the present disclosure, execution of the instructions further causes the at least one processor to identify a time period. Execution of the instructions may further cause the at least one processor to periodically obtain or provide a representation of the component based on the time period.

[0053] In another aspect of the present disclosure, execution of the instructions further causes the at least one processor to identify a time period. The time period may be between approximately 50 milliseconds and 250 milliseconds. Executing the instructions may also cause the at least one processor to periodically obtain or provide a representation of the component based on the time period.

[0054] In another aspect of the present disclosure, execution of the instructions further causes the at least one processor to identify a first period for transitioning from the first representation to the second representation. Execution of the instructions may further cause the at least one processor to identify a second period. The second period indicates an intermediate representation period. Execution of the instructions may further cause the at least one processor to identify a number of steps based on the first period and the second period. Execution of the instructions may further cause the at least one processor to generate one or more intermediate representations based on the number of steps.

[0055] In another aspect of the disclosure, execution of the instructions further causes the at least one processor to scale the second representation to generate a first scaled representation. The first scaled representation may fit within a first bounding box. The first scaled representation may include a first intermediate representation of the one or more intermediate representations.

[0056] In another aspect of the disclosure, execution of the instructions may further cause the at least one processor to downscale the second representation to generate a first scaled representation. The first scaled representation may fit within a first bounding box. The first scaled representation may include a first intermediate representation of the one or more intermediate representations. Execution of the instructions may further cause the at least one processor to upscale the first scaled representation to generate at least one additional intermediate representation of the one or more intermediate representations.

[0057] In another aspect of the disclosure, execution of the instructions may further cause the at least one processor to downscale the second representation to generate a first scaled representation. The first scaled representation may fit within a first bounding box. The first scaled representation may comprise a first intermediate representation of the one or more intermediate representations. Execution of the instructions may further cause the at least one processor to upscale the first scaled representation to generate at least one additional intermediate representation of the one or more intermediate representations. A final intermediate representation of the one or more intermediate representations may comprise the second representation.

[0058] One aspect of the present disclosure is a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause at least one computing device to obtain a first representation and a second representation of a component. Execution of the instructions further causes the at least one computing device to determine a first bounding box of the first representation and a second bounding box of the second representation. Execution of the instructions may also cause the at least one computing device to determine one or more intermediate representations between the first representation and the second representation based on the first bounding box and the second bounding box. Execution of the instructions may also cause the at least one computing device to operate the component based on the one or more intermediate representations.

[0059] The foregoing summary is illustrative only and is not intended to be limiting. Other aspects, features, and advantages of the systems, devices, and methods described herein, and / or other subject matter, will become apparent in the teachings set forth below. The summary is provided to introduce a selection of concepts of the present disclosure. The summary is not intended to identify key or essential features of the subject matter described herein. [Brief explanation of the drawings]

[0060] Various examples are illustrated in the accompanying drawings for purposes of illustration and should not be construed as limiting the scope of the examples in any way. Various features of different disclosed examples can be combined to form additional examples that are part of this disclosure. [Figure 1] FIG. 1 illustrates an exemplary mobile robot in accordance with some embodiments. [Figure 2] FIG. 1 illustrates an exemplary environment including a computing device for modifying a representation of a robot, according to aspects of the present disclosure. [Figure 3] 1 illustrates a user interface (UI) associated with dynamically modifying a robot's representation, according to some embodiments of the present disclosure. [Figure 4] 1 illustrates a user interface (UI) associated with dynamically modifying a robot's representation, according to some embodiments of the present disclosure. [Figure 5] 1 illustrates a user interface (UI) associated with dynamically modifying a robot's representation, according to some embodiments of the present disclosure. [Figure 6] 1 is a flowchart of an exemplary routine for modifying a representation of a robot, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0061] Generally described, the present disclosure relates to modifying a representation (e.g., a software representation) of a component (e.g., a machine, device, robot, etc.) using a representation generation system. For example, the representation generation system communicates with the component and generates a representation of the component based on data provided by the component. The representation may be an approximation of the component's physical shape (e.g., outline). For example, the representation may be a software representation of the component's hardware. For example, the component may be a mobile robot including a base, one or more arms, one or more legs, one or more additional interactive elements, etc., and the component representation may be a representation of the physical shape of the base, one or more arms, one or more legs, one or more additional interactive elements, etc. In some cases, the component representation may be a one-dimensional, two-dimensional, three-dimensional, etc. representation of the component.

[0062] The representation generation system (or a separate system) may utilize the representation of the component to adjust the performance and / or behavior of the component. By modifying the representation of the component (e.g., periodically or aperiodically), the representation generation system can adjust the performance of the component (and / or other components that interact with the component). For example, the representation generation system may adjust how the component moves through an environment based on the modified representation of the component (e.g., for collision avoidance or navigation pathfinding). Thus, it may be important that the representation of the component is an accurate representation of the component, as errors in the representation may cause the component to perform undesired actions (e.g., hitting obstacles, getting stuck, getting lost, etc.).

[0063] The methods and apparatus described herein enable a representation generation system to initiate the process of generating a representation of a component. For example, the representation generation system generates an intermediate representation of the component that represents a representation of the component as the component transitions from a first shape associated with a first representation to a second shape associated with a second representation. A component may transition from a first shape to a second shape to navigate within an environment, manipulate and / or interact with one or more objects or other components, etc. Such a representation generation system can enable accurate identification of a component within a given environment. For example, the representation generation system can enable the system to identify how a component may interact with another component, the environment (e.g., obstacles in the environment), etc.

[0064] The representation generation system may initiate a process in response to user input received via a user computing device. For example, a user may provide user input via a user computing device, requesting a component to transition from a first state (e.g., a first shape) to a second state (e.g., a second shape). The representation generation system may determine a representation of the component as the component transitions from the first state to the second state to identify the component's probable state during a particular period of time. Various other types of inputs or triggers may change the shape or size of a component, causing the representation generation system to modify the component's representation. As an example, a robot may receive a command to perform a task, which may cause the robot to change shape or size (e.g., to extend or move an articulated arm or other component). The representation generation system may modify the robot's representation (or a component thereof) to track the movement of the articulated arm or other component of the robot. The modified representation may be used by the robot or other robots or systems to make decisions based on the size, shape, or position of the robot (or its component) represented by the determined representation.

[0065] In many conventional cases, implementing a generic representation system to identify a representation of a component may not provide satisfactory results for a particular situation or for a particular user. Such a generic process may include identifying a first representation of the component (e.g., a current representation of the component, e.g., representing a starting position or shape before a component movement) and a second representation of the component (e.g., a desired representation of the component, e.g., representing a finishing position or shape after a component movement) and automatically switching from the first representation to the second representation. The generic process may also include automatically switching from the first representation to the second representation for a particular period of time before, during, or after the robot transitions from the first shape to the second shape. For example, such a generic process may include switching from the first representation to the second representation based on determining that the component has initiated a process of changing shape from a first shape associated with the first representation to a second shape associated with the second representation. In another example, such a generic process may include switching from the first representation to the second representation based on determining that the component has completed a process of changing shape from the first shape to the second shape. Thus, the representation can transition directly and automatically from a first representation to a second representation.

[0066] Because the representation can transition directly and automatically from a first representation to a second representation, the representation of a component may not be an accurate representation of the component's shape. For example, if a typical system switches from a first representation to a second representation based on determining that the component has begun a process of switching from the first shape to the second shape, the second representation may not represent various intermediate shapes of the component as the component switches from the first shape to the second shape. In another example, if a typical system switches from a first representation to a second representation based on determining that the component has completed a process of switching from the first shape to the second shape, the first representation may not represent various intermediate shapes of the component as the component switches from the first shape to the second shape.

[0067] In some cases, a user may attempt to manually define different shapes for a component. For example, the user may define a first shape, a second shape, a third shape, a fourth shape, and a fifth shape, and transition the component from the first shape to the second shape, from the second shape to the third shape, from the third shape to the fourth shape, and from the fourth shape to the fifth shape, thereby transitioning the component from the first shape to the fifth shape. Some systems can generate representations of each of the first shape, second shape, third shape, fourth shape, and fifth shape, and transitions between each of the representations. However, such a process may be inefficient and impractical because it may require the user not only to define each of the various shapes, but also for the component to physically transition between each of the various shapes. Such a process may be power-intensive, time-consuming, and impractical.

[0068] The methods and apparatus described herein enable a system to generate a representation of a component that represents the component as it transitions from a first shape to a second shape, providing a more accurate representation of the component compared to typical systems that transition directly from the first representation associated with the first shape to the second representation associated with the second shape. For example, the system may generate an intermediate representation that identifies one or more shapes of the component as it transitions from the first shape to the second shape. By generating and utilizing an intermediate representation that represents the component as it transitions from the first shape to the second shape, the system can increase the accuracy and efficiency of the component. Thus, such a process can increase the efficiency and speed of the component. For example, because the system can identify a more accurate representation of the component compared to other processes, the component (or other device) can navigate an environment faster and more efficiently (e.g., by avoiding potential collisions).

[0069] As the number of components (e.g., mobile robots, computing devices, etc.) proliferates, there is an increasing demand for more accurate representations of the components. For example, a particular environment may include multiple components, each capable of maneuvering within the environment, and each component may attempt to avoid collisions with other components, obstacles, etc. within the environment. The present disclosure provides systems and methods that enable a reduction in collisions and potential collisions experienced by components. Furthermore, the present disclosure provides systems and methods that enable a reduction in the time and user interaction required to transition from a first shape to a second shape, compared to conventional embodiments, without significantly impacting the speed or power consumption of the device when adjusting from the first shape to the second shape. These advantages are provided by the embodiments described herein, specifically by implementation of a process that includes generating intermediate representations of the components.

[0070] As described herein, the process of generating a representation of a component may include obtaining a first representation and a second representation of the component. The first representation of the component may be a current representation of the component. For example, the first representation of the component may represent a current state of the component or a state prior to a movement. In some cases, the first representation of the component may represent a state of the component based on the component completing an action, task, etc. For example, the component is in the process of grasping an item with the component's arm, and the first representation of the component may represent a state of the component after the component has completed the process of grasping the item.

[0071] The second representation of the component may be a defined representation of the component (e.g., a desired representation of the component). The second representation of the component may represent a future state of the component. For example, the second representation of the component may represent a desired state, goal state, or state after a motion of the component. The second representation of the component may represent a state of the component after performance of a particular task by the component. In some cases, the first shape (and first representation) may be smaller than the second shape (and second representation). In other cases, the first shape (and first representation) may be larger than the second shape (and second representation).

[0072] In some cases, the first representation of the component and / or the second representation of the component may represent a user-defined state of the component. For example, the representation generation system may obtain information from a user computing device identifying a first state of the component and a second state of the component. The information may identify that the component transitions from the first state to the second state. In some cases, if the component is not configured according to the first state, to transition from the first state to the second state, the representation generation system may transition the component to the first state before transitioning from the first state to the second state.

[0073] Based on the first and second representations of the component, the representation generation system may generate one or more intermediate representations of the component. For example, the one or more intermediate representations of the component may identify one or more intermediate states of the component as the component transitions from a first state (associated with the first representation) to a second state (associated with the second representation). To generate the one or more representations of the component, the representation system may identify and / or generate bounding boxes (bounding box coordinates) associated with the first and / or second representations of the component.

[0074] The representation generation system may generate the first intermediate representation by scaling the second representation. For example, the representation generation system may downscale the second representation to fit within the bounding box of the first representation, such as when the second representation is larger than the first representation. In some cases, the second representation may be smaller than the first representation, and the representation generation system may upscale the second representation to fit within the bounding box of the first representation. Specifically, the representation generation system may upscale and / or downscale the second representation by generating a scaling factor between the first and second representations. In some cases, the representation generation system may upscale and / or downscale the second representation by matching one or more bounding box coordinates of the second representation to one or more bounding box coordinates of the first representation. For example, the representation generation system may perform scaling to match one or more bounding box coordinates identifying horizontal boundaries of the second representation to one or more bounding box coordinates identifying horizontal boundaries of the first representation. In some embodiments, the second representation may be scaled differently along different dimensions, such as when the bounding boxes of the first representation and the second representation have different aspect ratios. In some cases, the second representation may be upscaled in a first dimension and downscaled in a second dimension, which enables the representation generation system to generate the first intermediate representation.

[0075] The representation generation system may generate one or more additional intermediate representations by scaling a first intermediate representation. For example, the representation generation system may downscale or upscale the first intermediate representation to generate one or more additional intermediate representations. The representation generation system may iteratively scale a previous intermediate representation to generate a subsequent intermediate representation. For example, the representation generation system may scale the first intermediate representation to generate a second intermediate representation, scale the second intermediate representation to generate a third intermediate representation, and scale the third intermediate representation to generate a fourth intermediate representation. In this way, the representation generation system can generate intermediate representations.

[0076] In some cases, the representation generation system may generate the intermediate representations based at least in part on a step size. For example, the step size may be used to determine several intermediate representations between a first representation and a second representation. The representation generation system may identify the step size (e.g., which may be defined by a user in some embodiments). In some cases, the representation generation system may determine the number of steps based on a first period for a component to transition between a first state and a second state and a second period (e.g., step duration or frequency) that identifies how often the intermediate representations are updated. For example, the representation generation system may divide the first period by the second period to generate the number of intermediate steps. As described herein, the number of steps may be used to determine how much the representation changes at each intermediate step.

[0077] Based on the first representation, the second representation, and the one or more intermediate representations, the representation generation system may cause the components to perform one or more actions. For example, the representation generation system may communicate with one or more systems (e.g., navigation systems) to cause the performance of one or more actions. The one or more actions may include navigating within an environment, performing an obstacle and / or collision avoidance process, performing a task, etc. In some cases, the representation generation system may control or instruct the components to perform one or more actions. Because the representation generation system can cause the components to perform one or more actions based on the intermediate representations, the components may perform tasks more efficiently and more accurately compared to systems that rely on the first and second representations.

[0078] Mobile robot FIG. 1 illustrates an exemplary mobile robot 150 (which may be an example of the components described above), according to one embodiment. The mobile robot 150 includes one or more wheels 151. The mobile robot 150 includes one or more drive systems that can move the robot 150. The robot 150 includes one or more motors (e.g., electric motors) that can rotate the one or more wheels 151. In some embodiments, the robot 150 includes a braking system. The mobile robot 150 includes one or more movable components that can change the shape or footprint of the mobile robot 150. For example, the mobile robot 150 includes an articulated arm 152 that the mobile robot 150 can use to pick up, move, or otherwise manipulate objects. The arm 152 can move to a variety of different positions, and depending on the position of the arm 152, the entire mobile robot 150 can assume a variety of different shapes. The robot 150 may also include one or more actuators for moving the arm.

[0079] A robot, such as the mobile robot 150 described herein, can perform a variety of actions. As the robot performs the various actions, the shape of the robot may change. For example, the shape of the robot may change as the robot rotates, as the robot actuates one or more elements of the robot (e.g., the robot's arms, ports, etc.). Because the shape of the robot may change, it may be important for the associated system to utilize an accurate representation of the robot's given shape at a particular time period. If the associated system does not have an accurate representation of the robot's given shape, the robot may be more likely to stall, get stuck, or interact (e.g., bump into, collide, etc.) with obstacles and / or other robots.

[0080] System Overview 2 shows a schematic diagram of a system 200 including a representation generation system 202 according to some embodiments herein. In the illustrated embodiment, the system 200 includes the representation generation system 202, a component 204, and a user computing device 206. Any one or any combination of the components shown and described in FIG. 2 may each be implemented using one or more computing devices, such as, but not limited to, one or more servers, processors, computing devices, virtual machines, etc., and may communicate via a wired or wireless connection (e.g., via a network not shown in FIG. 1) to modify the representation of the component 204. For example, the representation generation system 202 and the user computing device 206 may communicate via a local area network or a wide area network such as the Internet.

[0081] Representation generation system 202 and / or user computing device 206 may include one or more computing systems or devices, which may include any network-equipped computing device, e.g., a desktop computer, laptop, smartphone, tablet, etc., or any non-network-equipped computing device. In some embodiments, representation generation system 202 and user computing device 206 may be incorporated into the same computing system (e.g., a laptop or smartphone, etc.). Furthermore, representation generation system 202 and user computing device 206 may be integrated into the same computer or program (e.g., such that a single data transfer link is used). In some embodiments, representation generation system 202 may be incorporated into component 204 (e.g., such that a single data transfer link is used). In some embodiments, user computing device 206 may be omitted. In some embodiments, a robot or other component 204 may include representation generation system 202. Component 204 can use representation generation system 202 to determine its own shape, and component 204 can use that information for collision avoidance, navigation, etc. In some implementations, information about the shape of component 204 can be transmitted to other devices (e.g., to other robots or components), and those other devices can use the information about the shape of component 204 for collision avoidance, navigation, etc.

[0082] The user computing device 206 may include one or more computing devices associated with a user. The user can utilize the user computing device 206 to provide representations (e.g., user-defined representations, desired representations, etc.) to the representation generation system 202. For example, the user can utilize the user computing device 206 to define the operation of the component 204 and / or the subsequent representation of the component 204 based on the execution of the operation. The user computing device 206 may include a networking utility 212 and an input / output device 214. In some embodiments, the user computing device 206 can be used to set parameters of the representation generation system 202. The user computing device 206 can be used to specify the number of intermediate steps and / or the interval time between each intermediate step. The user computing device 206 can be used to specify representation shapes for various configurations of the component 204. The representation generation system 202 then uses those specified representation shapes when the component has an associated configuration. As an example, a user specifies a first representation having a first or shorter length for a configuration in which the robot's arm is retracted, and a second representation having a second or longer length for a configuration in which the robot's arm is extended. In some embodiments, the user computing device 206 can be configured to receive commands from a user that change the shape of the component 204, such as a command for the robot to use its manipulator arm. In some embodiments, the user computing device can be incorporated into the component 204 (e.g., as a user interface on the robot).

[0083] Representation generation system 202 may include one or more computing devices for dynamically generating representations of components 204. Representation generation system 202 may include representation generation components 208, memory 209, and networking utilities 210. Representation generation system 202 can utilize representation generation components 208 to dynamically generate representations of components 204. Representation generation system 202 can communicate with components 204 via a wired or wireless connection, or can be incorporated into components 204 as described herein.

[0084] Component 204 may include one or more devices (e.g., a machine, a piece of equipment, a computing device, a robot, a mobile robot, or any other device). For example, component 204 may include mobile robot 150 as described in FIG. 1. Component 204 may include one or more elements that can have various states (shapes). For example, component 204 may include a particular element (e.g., an arm) that can have a shape that changes as one or more actions are performed. In some embodiments, the state of component 204 may be variable (e.g., as component 204 performs actions such as rotating, moving, etc.). In the example of FIG. 2, component 204 includes controller 216. Controller 216 may be a mechanical, electromechanical, or electronic device that causes an action to be performed based on received input (e.g., expression data) and transitions component 204 (or an element of component 204) from a first state to a second state.

[0085] The representation generation system 202 and / or the user computing device 206 may include a networking utility for data communication. For example, as shown in FIG. 1 , the representation generation system 202 includes a networking utility 210, and the user computing device 206 includes a networking utility 212. The networking utility 210 and the networking utility 212 can enable data communication between the representation generation system 202 and the user computing device 206. Furthermore, the networking utility 210 and the networking utility 212 enable data communication between the representation generation system 202 and the user computing device 206 over a network (e.g., the Internet). The network may include any suitable network, including an intranet, the Internet, a cellular network, a local area network, or any other such network or combination thereof. Protocols and components for communicating over the Internet or any of the other aforementioned types of communication networks are known to those skilled in the art of computer communications and need not be described in detail herein.

[0086] The representation generation system 202 and the component 204 communicate data via a streaming data connection (e.g., a data stream). For example, the representation generation system 202 streams actions, states, expressions, etc. to the component 204 via the representation generation component 208 via a data stream (e.g., at periodic intervals and / or as information is obtained by the representation generation system 202), and in response to the received actions, states, expressions, etc., the component 204 can transition from a first state to a second state. The representation generation system 202 and the user computing device 206 communicate data via a streaming data connection (e.g., a data stream).

[0087] The state or representation of the component 204 (e.g., the current state or representation of the component 204) is transmitted from the memory (e.g., volatile and / or non-volatile memory) of the component 204 to the memory (e.g., volatile and / or non-volatile memory) of the representation generation system 202. In some implementations, the state or representation of the component 204 is transmitted from the memory (e.g., volatile and / or non-volatile memory) of the representation generation system 202 to the memory (e.g., volatile and / or non-volatile memory) of the user computing device 206.

[0088] The user computing device 206 includes an input / output device 214 for obtaining input identifying a state and / or representation of the component 204 and / or for providing output identifying a representation or state of the component 204 (e.g., an intermediate representation generated by the representation generation system 202). For example, the input / output device 214 may include a display, a touchscreen, a keypad, a speaker, a microphone, or any other device for receiving input and / or providing output. Additionally, the input / output device 214 may be an application programming interface ("API"). The user computing device 206 communicates with the input / output device 214 to receive the state and / or representation of the component 204. In some cases, a user uses the user computing device 206 to drive the component 204 and / or cause the component 204 to perform various operations. For example, the user computing device 206 may include a joystick for manipulating the component 204 and / or a touchscreen for providing input identifying an operation. Additionally, the user computing device 206 communicates with the input / output device 214 to enable display of (e.g., cause of) a representation or state of the component 204. In some embodiments, the user computing device 206 may separately communicate the representation or state of the component 204 or may provide a Uniform Resource Identifier ("URI") (e.g., a Uniform Resource Locator ("URL")) that allows the system to access the representation or state of the component 204. In some embodiments, the component 204 may be an autonomous robot capable of operating without user input.

[0089] A user can provide an input command to the input / output device 214 of the user computing device 206 to initiate the transition of the component 204 from a first state to a second state. For example, the input command may include a request to perform a particular action (e.g., navigate to a particular location, perform a task, activate one or more elements of the component 204, etc.). In some cases, the user computing device 206 routes the input command to the component 204 through the representation generation system 202. In other cases, the user computing device 206 can communicate with the component 204 and communicate the input command directly to the representation generation system 202. In such cases, the user computing device 206 can separately communicate the input command to the representation generation system 202. In some embodiments, the component 204 (e.g., a robot) can make the decision to transition from the first state to the second state, such as without user input. The component 204 may be an autonomous or self-controlled robot.

[0090] Representation generation system 202 can receive input commands from user computing device 206 and current state data from component 204 via networking utility 210. In some cases, in response to receiving an input command from user computing device 206, representation generation system 202 can request state data from component 204. For example, representation generation system 202 can request data identifying the current state of the component. In some cases, representation generation system 202 may pause component 204's performance of a transition from a first state to a second state. For example, representation generation system 202 pauses performance of a transition from the first state to the second state and causes component 204 to continue performance once representation generation system 202 determines that a certain number of intermediate representations have been generated for the transition from the first state to the second state. In some embodiments, component 204 can maintain information about its current state and associated current representation. In some embodiments, component 204 itself may include representation generation system 202.

[0091] Representation generation system 202 can generate the representations. In some embodiments, based on receiving input commands from user computing device 206 and current state data from component 204, representation generation system 202 can utilize representation generation component 208 to generate a representation of component 204. To generate the representations, representation generation component 208 may generate a first representation of component 204 based on the current state data (e.g., representing a current representation of the component) and generate a second representation of component 204 based on the input commands (e.g., representing a desired or goal representation of component 204).

[0092] The representation generation component 208 may generate a scaling factor between the first representation and the second representation. In some cases, for the first representation and the second representation of component 204, the representation generation component 208 may identify and / or generate one or more bounding boxes associated with the representations. For example, the representation generation component 208 identifies one or more bounding box coordinates that identify a bounding box of the representation. A first bounding box is determined for the first representation (e.g., corresponding to an initial or pre-motion state). A second bounding box is determined for the second representation (e.g., corresponding to a target or post-motion state). Further, in some cases, the bounding box coordinates identify maximum and / or minimum values ​​in one or more axes for all or a portion of the representations. For example, the bounding box coordinates identify a minimum y-axis value, a maximum y-axis value, a minimum x-axis value, and a maximum x-axis value for each or a portion of the representations. The bounding boxes may be rectilinear. The bounding boxes may be two-dimensional rectangles with four corners, etc. The four corners of the bounding box can be defined by first and second (e.g., minimum and maximum) x-axis values ​​and first and second (e.g., minimum and maximum) y-axis values. In some embodiments, a three-dimensional bounding box can be used (e.g., using first and second z-axis coordinates).

[0093] The representation generation component 208 determines a first intermediate representation. To generate the first intermediate representation, the representation generation component 208 may scale the second representation based on the scaling factor and / or bounding box of the first representation. The representation generation component 208 may scale (e.g., downscale, upscale, etc.) the second representation to fit within and / or fill the bounding box of the first representation. For example, the representation generation component 208 may identify a factor by which to multiply one or more bounding box coordinates of the second representation (e.g., minimum and maximum y-axis values ​​and / or minimum and maximum x-axis values) such that one or more bounding box coordinates of the second representation match one or more bounding box coordinates of the first representation. In some cases, a separate scaling factor may be determined for each dimension of the bounding box (e.g., an x-axis scaling factor and a y-axis scaling factor in a 2D embodiment). The representation generation component 208 uses the identified coefficients to scale all or a portion of the bounding box coordinates of the second representation to generate a scaled second representation (e.g., a first intermediate representation). In some cases, all or a portion of the bounding box coordinates of the scaled second representation may match the bounding box coordinates of the first representation. In other cases, one or more of the bounding box coordinates of the scaled second representation may not match the bounding box coordinates of the first representation. For example, the x-axis bounding box coordinates of the scaled second representation may not match the x-axis bounding box coordinates of the first representation, and the y-axis bounding box coordinates of the scaled second representation may match the y-axis bounding box coordinates of the first representation. In another example, the x-axis bounding box coordinates of the scaled second representation may match the x-axis bounding box coordinates of the first representation, and the y-axis bounding box coordinates of the scaled second representation may not match the y-axis bounding box coordinates of the first representation. In some cases, all or some of the bounding box coordinates of the second representation may be scaled so that all or some of the bounding box coordinates approximate but do not match the bounding box coordinates of the first representation.

[0094] The representation generation process may include multiple intervals. In each interval of the multiple interval process, the representation generation component 208 obtains a pre-scaled representation and scales the pre-scaled representation. At or after a first time interval, the representation generation component 208 may scale a scaled second representation (e.g., a first intermediate representation) to generate a second intermediate representation. After the second time interval, the representation generation component 208 may scale the second intermediate representation to generate a third intermediate representation, and so on. It will be understood that the representation generation process may include any number of time intervals. For example, the representation generation process may include n intervals, where n may be any number.

[0095] In some cases, the number of intervals in the representation generation process is based on a step size or duration. The representation generation component 208 determines and / or generates several intermediate steps. For example, the representation generation component 208 may generate the number of steps based on a first period associated with a transition between a first state associated with a first representation and a second state associated with a second representation, and a second period associated with an interval between the intermediate representations. The representation generation component 208 identifies the first period based on historical data (e.g., the representation generation component 208 tracks the period for the transition between the first state and the second state), user-defined data (e.g., a user provides data indicating the first period for the transition), machine-learning-generated data (e.g., the representation generation component 208 utilizes one or more machine-learning models to determine the amount of time for the transition), etc. In some embodiments, the durations of various movements or transitions are stored in a computer-readable memory (e.g., in a lookup table or database). The representation generation component 208 identifies the second period based on system-defined data (e.g., a system-defined period for the interval between intermediate representations), user-defined data, user computing device capabilities (e.g., the processing speed of the user computing device 206), etc. The representation generation component 208 divides the first period by the second period to identify the number of steps. For example, for a first period of 10 seconds and a second period of 100 milliseconds, the representation generation component 208 identifies the number of steps, which is 100 steps. For each step, the representation generation component 208 generates a corresponding intermediate representation. For example, if the step size is 100, the representation generation component 208 generates 100 intermediate representations between the first representation and the second representation.The duration of each step can be about 10 ms, about 20 ms, about 30 ms, about 40 ms, about 50 ms, about 60 ms, about 70 ms, about 80 ms, about 90 ms, about 100 ms, about 110 ms, about 125 ms, about 150 ms, about 175 ms, about 200 ms, about 225 ms, about 250 ms, about 300 ms, about 350 ms, about 400 ms, about 450 ms, about 500 ms, or more, or any value therebetween, or any range between any of these values, although other embodiments outside these ranges are also possible.

[0096] The final representation of the intermediate representation corresponds to the second representation, and in some cases, to the intermediate representation immediately preceding the second representation.

[0097] Based on generating the intermediate representation, the representation generation component 208 causes the component 204 to initiate a transition from a first state (associated with the first representation) to a second state (associated with the second representation). When the component 204 transitions from the first state to the second state, the representation generation component 208 uses the intermediate representation to update the representation of the component 204. The representation generation component 208 synchronizes the intermediate representation such that when the component 204 initiates a transition from the first state to the second state, the representation generation component 208 uses the intermediate representation to initiate a transition from the first representation to the second representation.

[0098] The representation generation component 208 (or a separate system) utilizes the representation (e.g., synchronized with the state of the component 204) to identify how to cause the component 204 and / or other components to perform actions. For example, the representation generation component 208 may cause the component 204 to navigate an environment, perform obstacle avoidance, and perform collision avoidance (e.g., based on communication with a navigation system) based on a particular intermediate representation of the component 204 that identifies the state (e.g., current state) of the component 204 when the component 204 transitions from a first state to a second state. Because the representation generation component 208 has access to a representation of the environment of the component 204 (e.g., based on sensor data), it may be important for the representation generation component 208 to identify an accurate representation of the component 204. For example, if the representation of component 204 is erroneous (e.g., the representation indicates that the shape of component 204 is different from the actual shape of component 204), representation generation component 208 (or a separate navigation system relying on the representation) can cause component 204 to perform an action such as colliding with another component, colliding with an obstacle, getting stuck, damaging the component, etc. Component 204 operates a drive system based at least in part on one or more of the intermediate representations. Component 204 stops or slows down the robot using a drive system or a braking system based at least in part on one or more of the intermediate representations.

[0099] In some cases, representation generation component 208 obtains representations of other components in the environment of component 204. For example, an environment may include multiple components, each operating within the environment. Representation generation component 208 obtains representations of the multiple components (e.g., to avoid collisions between components, to avoid collisions with obstacles, etc.) and determines how to cause component 204 to perform an action based on the representations of the multiple components and the representation of component 204.

[0100] User interface for displaying a representation of a component 3 illustrates an exemplary user interface 300 that displays a visualization of a representation of a component. The exemplary user interface 300 illustrates an interface that a representation generation system (or a separate system) may generate (and present to a user) to identify a dynamically generated representation for a component. It will be understood that the user interface 300 is exemplary only, and the representation generation system may provide any type of user interface to enable the identification and display of a representation. In some embodiments, the representation is not displayed, and the component 204 or other device uses the representation to make decisions without presenting it to a user (e.g., for path planning or collision avoidance).

[0101] The user interface 300 may further include a first representation 302. The user interface 300 may further include more, fewer, or different interfaces. The first representation 302 may be a current representation of the component. In some cases, the first representation 302 may be an initial representation of the component. The representation 302 corresponds to a current or initial state of the component, such as before a transition. For example, the representation 302 identifies the state of the component during an initial period.

[0102] In some embodiments, the system may generate the first representation 302 and, in some cases, provide the first representation 302 for display. The system obtains state data from the component. For example, the system obtains sensor data from one or more sensors of the component, sensor data from one or more sensors in the component's environment, data related to one or more elements of the component (e.g., state data identifying the state of the component's arm, state data identifying the state of the component's legs or wheels, etc.). Based on the obtained state data, the system may generate the first representation 302 such that the first representation 302 represents the state of the component. In some embodiments, the component or system may know the current state or position of the component or a portion thereof (e.g., a movable arm) due to a previous command. After the component receives a command to move the arm to a first position, it determines that the arm is in the first position until a new command to move the arm to a second position is received.

[0103] The first representation 302 may be a visualization of a particular shape. For example, the first representation 302 may be a visualization of a polygon. In some cases, the first representation 302 may not directly correspond to the first state of the component. To generate the first representation 302, the system may generalize one or more aspects of the first state of the component. For example, the first state of the component may identify a component having m sides, where m may be any number. The system may generalize the component and generate the first representation 302 of the component with n sides, where n may be any number less than m. In some cases, the first state of the component may identify a component as a curve, a three-dimensional component, or any other component, and the system may generalize the component by representing the component using the first representation 302 that includes a two-dimensional polygon. Thus, the system may reduce the complexity of the component and generate the first representation 302. In some cases, the first representation 302 may be somewhat larger than the component in its first state, such as to provide a buffer.

[0104] The system that generates the first representation 302 may provide the first representation to an application for managing actions to be performed by the component. For example, the system may be a representation generation system. Further, the representation generation system or a separate system may implement an application for managing actions to be performed by the component based on the first representation 302 (e.g., for navigation or collision avoidance, etc.). A drive system of the component (e.g., a mobile robot) may operate based at least in part on the first representation 302.

[0105] In some cases, user interface 300 depicts visualizations of representations of multiple components. For example, user interface 300 depicts a visualization of a first representation of all or a portion of the components in a particular environment. In some embodiments, each component (e.g., a robot) may determine a representation of its own position and configuration, have sensors to identify other components (e.g., other robots) or other objects in the environment, and use that information for navigation and / or collision avoidance. In some embodiments, one component (e.g., a robot) does not receive or use representation information associated with other components (e.g., other robots). In other embodiments, decisions for navigation and / or collision avoidance can be made using representation information of multiple components. In some cases, components (e.g., robots) can broadcast or forward (e.g., wirelessly) their representation information to other components (e.g., other robots), so that those other components can use the representation information (e.g., for navigation and / or collision avoidance). In some cases, a single system can control the navigation and / or collision avoidance of multiple components (robots). In other embodiments, each component (eg, a robot) may determine its own navigation and / or collision avoidance decisions.

[0106] 3, the user interface 300 includes a first representation 302 showing a polygon. The first representation 302 includes a polygon having six sides. It will be understood that the first representation 302 may include more, fewer, or different sides. The first representation 302 is displayed against a graph (e.g., a graph based on the environment of the component) that specifies x-axis and y-axis coordinates.

[0107] 4 illustrates an exemplary user interface 400 that displays a visualization of a representation of a component. The exemplary user interface 400 illustrates an interface that a representation generation system (or a separate system) may generate (and present to a user) to identify dynamically generated representations for a component. It will be understood that the user interface 400 is exemplary only, and that the representation generation system may provide any type of user interface to enable the identification and display of representations. In some embodiments, the representations are not displayed.

[0108] The user interface 400 may further include a first representation 402 and a second representation 404. The first representation 402 may be the same as or similar to the first representation 302, as described above with respect to FIG. 3 . The user interface 400 may further include more, fewer, or different interfaces. As described above, the first representation 402 may be a current or initial representation of the component (e.g., corresponding to a current or initial state of the component), such as before a movement or transition. The second representation 404 may be a subsequent representation of the component (e.g., a representation of the component at a subsequent time period). The second representation 404 may correspond to a subsequent state of the component (e.g., after a movement or transition) compared to the current or initial state of the component. For example, the first representation 402 may identify the state of the component during an initial time period, and the second representation 404 may identify the state of the component during a subsequent time period.

[0109] In some embodiments, the system generates the first representation 402 and / or the second representation 404 and provides the first representation 402 and / or the second representation 404 for display. In some embodiments, the system uses the representation (e.g., by the component) without displaying the representation. As described above, the system obtains state data from the component and generates the first representation 402 such that the first representation 402 represents an initial state of the component. Additionally, the system obtains input (e.g., user input) that identifies one or more actions to be taken by the component. For example, the one or more actions may identify an element of the component to be actuated, a route to be navigated by the component, a task to be performed by the component (e.g., opening a door, rotating, grabbing, or picking up an item, taking a picture of an item, docking, etc.). The input may identify a particular state or representation of the component. For example, the input may identify a state for the component to operate. In some cases, the input may be an input command.

[0110] First representation 402 and second representation 404 may each be a visualization of a particular shape. For example, first representation 402 and second representation 404 may each be a visualization of a polygon. In some cases, one or more of first representation 402 and second representation 404 may not directly correspond to a particular state of the component, but may instead correspond to a generalized version of the component's state (e.g., a representation of reduced complexity compared to the component's state).

[0111] The representation generation system obtains the first representation 402 and the second representation 404. As described below, the representation generation system may utilize the first representation 402 and the second representation 404 to generate one or more intermediate representations of the component.

[0112] In some cases, user interface 400 depicts visualizations of first and second representations of multiple components, for example, user interface 400 depicts visualizations of first and / or second representations of all or some of the components in a particular environment.

[0113] 4, the user interface 400 includes a first representation 402 and a second representation 404, each of which represents a polygon. The first representation 402 includes a polygon having six sides, and the second representation 404 includes a polygon having eight sides. It will be understood that the first representation 402 and the second representation 404 may include more, fewer, or different sides. The first representation 402 and the second representation 404 are displayed relative to a graph (e.g., a graph based on the environment of the component) that specifies x-axis and y-axis coordinates.

[0114] 5 illustrates an exemplary user interface 500 that displays a visualization of a representation of a component, including an intermediate representation of the component. The exemplary user interface 500 illustrates an interface that a representation generation system (or a separate system) may generate (and present to a user) to identify a dynamically generated representation for a component. It will be understood that the user interface 500 is exemplary only, and that the representation generation system may provide any type of user interface to enable the identification and display of the representation. In some embodiments, the representation is not displayed. A component or system may use the representation for navigation and / or collision avoidance, etc., without displaying the representation to a user.

[0115] User interface 500 may further include first representation 502 and associated first bounding box 501, second representation 514 and associated second bounding box 503, and intermediate representations 504, 506, 508, 510, and 512. First representation 502 may be the same as or similar to first representation 302 described above with respect to FIG. 3 and / or first representation 402 described above with respect to FIG. 4. User interface 500 may further include more, fewer, or different interfaces. As described above, first representation 502 may be a current or initial representation of the component (e.g., corresponding to a current or initial state of the component), such as before a movement or transition. Second representation 514 may be a subsequent representation of the component (e.g., corresponding to a subsequent state of the component compared to the current or initial state of the component), such as after a movement or transition. As an example, the first representation 502 may outline the shape or footprint of the component when the arm is in a first position, and the second representation 514 may outline the shape or footprint of the component when the arm is in a second position.

[0116] All or some of the intermediate representations 504, 506, 508, 510, 512 may be intermediate representations between the first representation 502 and the second representation 514. The intermediate representations 504, 506, 508, 510, 512 may identify one or more intermediate states of the component as the component transitions from a first state associated with the first representation 502 to a second state associated with the second representation 514. For example, the intermediate representations may approximate the shape or footprint of the component as the arm moves from a first position to a second position.

[0117] To generate the intermediate representations 504, 506, 508, 510, 512, the representation generation system may obtain the first representation 502 and the second representation 514. The representation generation system may generate a scaling factor between the first representation 502 and the second representation 514. In some cases, the representation generation system may generate a first bounding box 501 for all or a portion of the first representation 502 and a second bounding box 503 for all or a portion of the second representation 514. The representation generation system may generate the first bounding box 501 for the first representation 502 and the second bounding box 503 for the second representation 514. The representation generation system may generate the first bounding box 501 and the second bounding box 503 using coordinate values ​​associated with the first representation 502 and the second representation 514. For example, the representation generation system may identify minimum x-axis values, maximum x-axis values, minimum y-axis values, and maximum y-axis values ​​of first representation 502 and minimum x-axis values, maximum x-axis values, minimum y-axis values, and maximum y-axis values ​​of first representation 502 and second representation 514. Based on the associated minimum x-axis values, maximum x-axis values, minimum y-axis values, and maximum y-axis values, the representation generation system may generate first bounding box 501 and second bounding box 503 such that first bounding box 501 includes a first side based on the minimum x-axis value of first representation 502, a second side based on the maximum x-axis value of first representation 502, a third side based on the minimum y-axis value of first representation 502, and a fourth side based on the maximum y-axis value of first representation 502. The second bounding box 503 may include a first side based on the minimum x-axis value of the second representation 514, a second side based on the maximum x-axis value of the second representation 514, a third side based on the minimum y-axis value of the second representation 514, and a fourth side based on the maximum y-axis value of the second representation 514. The bounding boxes 501 and 503 may be rectilinear. The bounding boxes 501 and 503 may be rectangular.

[0118] The representation generation system may scale the second representation 514 using a scaling factor and / or the first bounding box 501 and the second bounding box 503 (e.g., to generate the first intermediate representation 504). The representation generation system may scale the second representation 514 by scaling one or more coordinates (e.g., x-axis coordinate, y-axis coordinate, etc.) to match the coordinates of the first bounding box 501. For example, the representation generation system may identify a factor for scaling the coordinates of the second representation 514 so that the y-axis coordinate of the second representation 514 matches (e.g., is equal to) the y-axis coordinate of the first bounding box 501 and / or so that the x-axis coordinate of the second representation 514 matches (e.g., is equal to) the x-axis coordinate of the first bounding box 501. In some cases, the representation generation system may downscale or upscale the second representation 514. This may cause the representation generation system to scale the second representation 514 to fit within and / or fill the first bounding box 501.

[0119] The system may compare the first bounding box 501 to the second bounding box 503 to determine an x-axis scaling factor and a y-axis scaling factor (which may be different from the x-axis scaling factor, such as if the bounding boxes 501 and 503 have different aspect ratios). The second representation 514 may be scaled along the x-axis according to the x-axis scaling factor and scaled along the y-axis according to the y-axis scaling factor. As an example, the first representation 502 may have a bounding box 501 with an x-axis length of 100 and a y-axis length of 50 (e.g., for an aspect ratio of 2:1). The second representation 514 may have a bounding box 503 with an x-axis length of 125 and a y-axis length of 100 (e.g., for an aspect ratio of 1.25:1). The determined x-scaling factor may be 100 / 125 or 80%, and the determined y-scaling factor may be 50 / 100 or 50%. Thus, the first intermediate representation 504 may have the same general shape as the second representation 514, but is scaled 80% in the x-axis and 50% in the y-axis so that the first intermediate representation 504 fits inside the first bounding box 501. If the first bounding box 501 and the second bounding box 503 have the same aspect ratio, the first intermediate representation 504 may have a similar shape to the second representation 514. In the above example, the first intermediate representation 504 may have a shape that roughly corresponds to the second representation, but is more compressed along the y-axis than along the x-axis. Many variations are possible, such as based on different types of movement by different components (e.g., arms, trays, conveyors, etc.). In some embodiments, the second representation 514 is scaled in only one axis. In some embodiments, the second representation 514 is upscaled along a first axis and downscaled along a second axis.

[0120] The representation generation system may scale the second representation 514, as described herein, to generate the first intermediate representation 504. The first intermediate representation 504 may represent a first intermediate position of the component as the component transforms from a first shape associated with the first representation 502 to a second shape associated with the second representation 514. Because the first intermediate representation 504 and the first representation 502 are within the same bounding box (e.g., the first bounding box 501), changes from the first representation to the first intermediate representation 504 may be minimal.

[0121] To arrive at the second representation 514, the representation generation system may implement one or more subsequent intermediate representations. The number of intermediate representations may be based on a step size or number of steps. In the example of FIG. 5, the number of steps is five, and five intermediate representations 504, 506, 508, 510, and 512 are shown. It will be understood that more, fewer, or different steps may be utilized. The number of steps is based on the duration for the transition between the first state and the second state and the duration for the transition between each intermediate representation. A change to the first intermediate representation 504 may occur at the beginning of the transition, while changes to the other intermediate representations 506, 508, 510, and 512 and / or to the second representation 514 may occur at regular intervals. As an example, a transition from the first state to the second state may take 0.5 seconds. At time 0, the system may change from the first representation 502 to the first intermediate representation 504. At time 0.1, the system may change to the second intermediate representation 506. At time 0.2, the system may evolve to the third intermediate representation 508. At time 0.3, the system may evolve to the fourth intermediate representation 506. At time 0.4, the system may evolve to the fifth intermediate representation 506. At time 0.5, the system may evolve to the second representation 514.

[0122] To generate intermediate representations 506, 508, 510, and 512, the representation generation system may incrementally scale the first intermediate representation 504 toward the second representation 514. The representation generation system may determine how much to increase and / or decrease the scale of a shape based on the number of steps. For example, if the representation generation system determines that five steps should be used to transition from the first intermediate representation 504 to the second representation 514, the representation generation system may scale in 20% increments for each of the subsequent intermediate representations 506, 508, 510, 512, and the second representation 514.

[0123] As an example, the second intermediate representation 506 has a size and / or shape scaled by 20% from the first intermediate representation 504 to the second representation 514. The third intermediate representation 508 has a size and / or shape scaled by 40% from the first intermediate representation 504 to the second representation 514. The fourth intermediate representation 510 has a size and / or shape scaled by 60% from the first intermediate representation 504 to the second representation 514. The fifth intermediate representation 512 has a size and / or shape scaled by 80% from the first intermediate representation 504 to the second representation 514. The second intermediate representation 506 has a size and / or shape scaled by 20% from the first intermediate representation 504 to the second representation 514. The shape is scaled up to 100% for the second representation 514. If the transition includes n steps, the representation may be scaled in increments of 100% divided by n for each step from the first intermediate representation 504 to the second (eg, final) representation 514.

[0124] In some embodiments, the transition through the steps may be linear, with substantially the same amount of scaling (e.g., 20%) at each substantially equal time interval (e.g., 0.1 seconds). However, in some embodiments, the scaling may be nonlinear, such as to reflect nonlinear motion of the component. If the component starts moving slowly, then accelerates to a maximum velocity (e.g., midway through the transition), and then decelerates to relatively slow motion at the end of the transition, scaling via the intermediate representation may reflect that nonlinear motion. For example, rather than scaling the same incremental amount at each step, the system may change the scale by different amounts at different steps. For example, for a 1 second transition, at 0.1 seconds the scaling may be 2%, at 0.2 seconds the scaling may be 5%, at 0.3 seconds the scaling may be 10%, at 0.4 seconds the scaling may be 25%, at 0.5 seconds the scaling may be 50%, at 0.6 seconds the scaling may be 75%, at 0.7 seconds the scaling may be 90%, at 0.8 seconds the scaling may be 95%, at 0.9 seconds the scaling may be 98% and at 1 second the scaling may be 100% relative to the second representation.

[0125] 5, user interface 500 includes a first representation 502, intermediate representations 504, 506, 508, 510, and 512, each representing a polygon, and a second representation 514. Intermediate representations 504, 506, 508, 510, and 512 can identify transitions between first representation 502 and second representation 514. It will be understood that all or some of the representations can include more, fewer, or different edges. The representations are displayed with respect to a graph (e.g., a graph based on the environment of the component) that identifies x-axis and y-axis coordinates.

[0126] Controlling Component Structure Using Intermediate Representations 6 illustrates a method 600 performed by a representation generation system for generating an intermediate representation that identifies a transition between a first representation that identifies a first state of a component and a second representation that identifies a second state of the component, according to some examples of the disclosed technology. The representation generation system may be similar to representation generation system 202 described above, and may include, for example, a representation generation component similar to representation generation component 208, a memory similar to memory 209, and / or a networking utility similar to networking utility 210.

[0127] In block 602, the representation generation system obtains a first representation and a second representation of a component. The component may be a device. For example, the component may be a mobile robot. The first representation may identify an initial or current representation of the component (e.g., before a transition or movement), and the second representation may identify a subsequent representation of the component (e.g., showing a predicted size and / or shape of the component after the transition or movement). For example, the first representation may show the current shape of the component, and the second representation may show a modified shape of the component. The first representation may include a representation of the component at a first time period, and the second representation may include a representation of the component at a second time period following the first time period.

[0128] The first representation may be generated based on data associated with the component (e.g., sensor data), and the second representation may be generated based on data associated with a user computing device (e.g., input commands). The representation generation system may obtain first representation data (e.g., sensor data) from the component and second representation data (e.g., input commands) from the user computing device and generate the first and second representations. In some embodiments, the first representation may be based on a known state of the component based on a previous command or action. In some embodiments, the second representation may be based on a command or decision to move the component. The second representation may correspond to a predicted future size or shape of the component based on a command or decision to perform a movement or other transition. In some embodiments, the component may be an autonomous robot capable of making movement decisions without input from a user.

[0129] In block 604, the representation generation system determines a first bounding box for the first representation and a scaling factor between the first representation and the second representation. In some cases, the representation generation system may determine a second bounding box for the second representation. The representation generation system may determine the first bounding box and / or the second bounding box based on coordinates (e.g., x-axis coordinates, y-axis coordinates, etc.) of the first representation and the second representation.

[0130] At block 606, the representation generation system determines one or more intermediate representations between the first representation and the second representation. The representation generation system may determine the one or more intermediate representations based on one or more of a scaling factor, a first bounding box, and / or a second bounding box. To generate the first intermediate representation, the representation generation system may scale (e.g., downscale) the second representation to generate a first scaled representation (e.g., the first intermediate representation) that fits within the first bounding box. The representation generation system may scale (e.g., upscale) the first scaled representation to generate the second intermediate representation. The representation generation system may iteratively scale each intermediate representation to generate subsequent intermediate representations based on a ratio to arrive at a final intermediate representation corresponding to the second representation.

[0131] The representation generation system may generate one or more intermediate representations based on a step size or number of steps. To generate the number of steps, the representation generation system may identify a first period for a transition from the first representation to the second representation, identify a second period indicating the intermediate representation period or duration, and identify the number of steps by dividing the first period by the second period. For example, the step size may indicate that five intermediate representations will be generated.

[0132] At block 608, the representation generation system causes movement of the component based on the one or more intermediate representations. In some cases, the representation generation system may not cause movement of the component based on the one or more intermediate representations. In some examples, the representation generation system may actively instruct the component not to move or to stop or slow down. In other cases, the representation generation system may cause the component to perform one or more actions based on the one or more intermediate representations. For example, the representation generation system may cause the component to perform collision avoidance and / or obstacle avoidance based on the one or more intermediate representations of the component. A drive system of a component may operate, such as to stop, slow down, or proceed along a path, based at least in part on one or more intermediate representations disclosed herein. A braking system may be activated at least in part on one or more intermediate representations. For example, if the component identifies an object that poses a risk of collision with the intermediate representation, the component may apply brakes to slow down or stop. In some embodiments, the movement of an arm or other moving part of a component (e.g., a robot) may be controlled at least in part on one or more intermediate representations. For example, an arm or other moving component may be stopped, such as when a risk of collision is determined based on one or more intermediate representations.

[0133] In some embodiments, the representation generation system (or navigation or other system) may obtain representations (e.g., intermediate representations) of other components and cause the component to perform one or more actions based on the one or more intermediate representations of the component and / or the representations of the other components. For example, the representation generation system may cause the component to perform collision avoidance and / or obstacle avoidance based on the one or more intermediate representations of the component and / or the representations of the other components.

[0134] In some cases, the representation generation system may provide one or more intermediate representations of a component to other components (e.g., to perform actions such as obstacle avoidance, collision avoidance, etc.) For example, the representation generation system may identify one or more components (e.g., based on the component and the location of the one or more components such that the location of the one or more components is within a certain vicinity of the location of the component), provide one or more intermediate representations to the one or more components, and / or obtain representations of the one or more components from the one or more components.

[0135] In some cases, the representation generation system may periodically obtain and / or provide representations from other components. For example, the representation generation system may periodically obtain and / or provide representations to other components based on an identified period (e.g., a 100 millisecond period).

[0136] The first representation, the second representation, and / or the one or more intermediate representations may include an outline of a component, a shape (e.g., a polygon) of a component, a two-dimensional representation of a component, etc. In some embodiments, the first representation, the second representation, and / or the one or more intermediate representations may be a software-based representation of a component (e.g., hardware). For example, the first representation, the second representation, and / or the one or more intermediate representations may mimic the shape (e.g., physical shape) of a component at a particular time period.

[0137] In some cases, the representation generation system may cause at least one of the first representation, the second representation, or the one or more intermediate representations to be displayed. For example, the representation generation system may cause the representation to be displayed via a display of a user computing device. In some embodiments, the representation generation system may cause the display of a visualization identifying the transformation from the first representation to the one or more intermediate representations and the transformation from the one or more intermediate representations to the second representation.

[0138] Although the various examples herein relate to two-dimensional representations (e.g., having X and Y dimensions), in other embodiments the representation may be three-dimensional (e.g., having X, Y, and Z dimensions). The bounding box may be three-dimensional, and scaling may be performed in three dimensions rather than two.

[0139] Additional Information Conditional language such as "can," "could," "might," or "may," unless expressly stated otherwise or understood otherwise within the context in which it is used, is generally intended to convey that a particular example includes or does not include certain features, elements, and / or steps. Thus, such conditional language generally does not imply that the features, elements, and / or steps are somehow required in one or more examples.

[0140] Several illustrative examples that modify the representation of components are disclosed. Although the present disclosure has been described with respect to certain illustrative examples and uses, other examples and uses are within the scope of the present disclosure, including examples and uses that do not provide all of the features and advantages described herein. Components, elements, features, operations, or steps may be arranged or performed differently than described, and components, elements, features, operations, or steps may be combined, merged, added, or omitted in various examples. All possible combinations and subcombinations of the elements and components described herein are intended to be included in the present disclosure. No single feature or group of features is required or essential.

[0141] Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as acting in a particular combination and initially claimed as such, one or more features from a claimed combination may, in some cases, be cut from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0142] Any portion of any of the steps, processes, structures, and / or devices disclosed or illustrated in one example of the present disclosure may be combined with or used with (or in place of) any other portion of the steps, processes, structures, and / or devices disclosed or illustrated in a different example or flowchart. The examples described herein are not intended to be distinct and separate from one another. Combinations, variations, and implementations of the disclosed features are within the scope of the present disclosure.

[0143] Although operations may be shown in the figures or described herein in a particular order, such operations need not be performed in the particular order shown, or in sequential order, to achieve desirable results, and not all operations need be performed. Other operations not shown or described may be incorporated into the example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the described operations. Moreover, in some implementations, operations may be rearranged or reordered. Also, the separation of various components in the above implementations should not be understood to require such separation in all implementations, and it should be understood that the described components and systems may generally be integrated together in a single product or packaged in multiple products. Furthermore, some implementations are within the scope of the present disclosure.

[0144] Furthermore, although illustrative examples have been described, any examples having equivalent elements, modifications, omissions, and / or combinations are also within the scope of this disclosure. Moreover, although certain aspects, advantages, and novel features have been described herein, not all such advantages are necessarily achieved in accordance with any particular example. For example, some examples within the scope of this disclosure may achieve one advantage or group of advantages taught herein without necessarily achieving other advantages taught or suggested herein. Furthermore, some examples may achieve advantages different from those taught or suggested herein.

[0145] Some examples are described in connection with the accompanying drawings. While the drawings are drawn and / or shown to scale, such scale should not be limiting, and dimensions and proportions other than those shown are contemplated and within the scope of the disclosed invention. Distances, angles, and the like are merely illustrative and do not necessarily bear an exact relationship to the actual dimensions and layout of the devices shown. Components may be added, removed, and / or rearranged. Furthermore, any particular features, aspects, methods, properties, characteristics, qualities, attributes, elements, etc. disclosed herein with respect to various examples may be used in all other examples described herein. Furthermore, any method described herein may be practiced using any device suitable for performing the recited steps.

[0146] For purposes of summarizing the disclosure, certain aspects, advantages, and features of the invention have been described herein. Not all, or any, such advantages may necessarily be achieved in accordance with any particular example of the invention disclosed herein. No aspect of the disclosure is essential or required. In many examples, devices, systems, and methods may be configured differently than shown in the figures or descriptions herein. For example, various functions provided by the illustrated modules may be combined, rearranged, added, or deleted. In some implementations, additional or different processors or modules may perform some or all of the functions described in the figures and with reference to the illustrated examples. Many implementation variations are possible. Any of the features, structures, steps, or processes disclosed herein may be included in any example.

[0147] In some embodiments, the methods, techniques, microprocessors, and / or controllers described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hardwired to execute the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) permanently programmed to execute the techniques, or may include one or more general-purpose hardware processors programmed to execute the techniques in response to program instructions in firmware, memory, other storage, or a combination thereof. The instructions may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of non-transitory computer-readable storage medium. Such special-purpose computing devices may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, server computer systems, portable computer systems, handheld devices, networking devices, or any other device or combination of devices incorporating wired and / or program logic to implement the techniques.

[0148] The microprocessors or controllers described herein may be coordinated by operating system software, such as iOS, Android, Chrome OS, Windows, Unix, Linux, SunOS, Solaris, Blackberry OS, VxWorks, or other compatible operating systems. In other embodiments, the computing device may be controlled by its own operating system. A traditional operating system controls and schedules computer processes for execution, performs memory management, provides file system, networking, I / O services, and provides user interface functionality such as a graphical user interface ("GUI"), among other things.

[0149] The microprocessors and / or controllers described herein may implement the techniques described herein using customized hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that renders the microprocessor and / or controller a dedicated machine. According to one embodiment, portions of the techniques disclosed herein are performed by the controller in response to execution of one or more sequential instructions contained in a memory. Such instructions may be read into the memory from another storage medium, such as a memory storage device. Execution of the sequences of instructions contained in the memory causes the processor or controller to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0150] Furthermore, the various illustrative logic blocks and modules described in connection with the embodiments disclosed herein may be implemented or performed by a machine, such as a processor device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. A processor device may be a microprocessor, but in alternative examples, a processor device may be a controller, a microcontroller, or a state machine, combinations thereof, etc. A processor device may include electrical circuitry configured to process computer-executable instructions. In another embodiment, a processor device includes an FPGA or other programmable device that performs logical operations without processing computer-executable instructions. A processor device may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. While described herein primarily with respect to digital technology, a processor device may also include primarily analog components. For example, some or all of the technology described herein may be implemented with analog circuitry or mixed analog and digital circuitry.

[0151] In summary, various examples of modifying the expression of components are disclosed. The disclosure extends beyond the specifically disclosed examples to other alternatives and / or other uses of the examples, as well as specific modifications and equivalents thereof. Moreover, the disclosure expressly contemplates that various features and aspects of the disclosed examples can be combined with or substituted for one another. Thus, the scope of the disclosure should not be limited by the specific disclosed examples described above.

Claims

1. 1. A method for modifying a representation of a robot, comprising: obtaining a first representation of the robot corresponding to a first shape of the robot before a movement transition; obtaining a second representation of the robot corresponding to a predicted second shape of the robot after the motion transition; determining a first bounding box of the first representation; determining a second bounding box for the second representation; determining a first intermediate representation having the same general shape as the second representation and scaled to fit within the first bounding box; determining a plurality of additional intermediate representations that are incrementally scaled between the first intermediate representation and the second intermediate representation; A method comprising:

2. The method of claim 1 , comprising modifying a movement of the robot based on one or more of the intermediate representations.

3. 10. The method of claim 1, further comprising generating a display of at least one of the first representation, the second representation, the first intermediate representation, or an intermediate representation of the plurality of additional intermediate representations.

4. 10. The method of claim 1, further comprising: generating a visualization display identifying at least one of a transformation from the first representation to the first intermediate representation, a transformation from the first intermediate representation to the plurality of additional intermediate representations, or a transformation from the plurality of additional intermediate representations to the second representation.

5. The method of claim 1 , wherein the first representation, the second representation, the first intermediate representation, and the plurality of additional intermediate representations each comprise a software representation of the robot.

6. The method of claim 1 , wherein the robot comprises an autonomous robot.

7. The method of claim 1 , wherein the first representation, the second representation, the first intermediate representation, and the plurality of additional intermediate representations each comprise a polygon.

8. The method of claim 1 , further comprising: performing collision avoidance based on the first intermediate representation and the plurality of additional intermediate representations.

9. The method of claim 1 , further comprising: performing collision avoidance based on the first intermediate representation, the plurality of additional intermediate representations, and a plurality of intermediate representations of a second robot.

10. The method of claim 1 , further comprising: performing obstacle avoidance based on the first intermediate representation and the plurality of additional intermediate representations.

11. The method of claim 1 , wherein the first representation is based on data associated with the robot and the second representation is based on data associated with a user computing device.

12. acquiring first expression data from the robot; generating the first representation based on the first representation data; obtaining second expression data from the user computing device; generating the second representation based on the second representation data; The method of claim 1 further comprising:

13. 10. The method of claim 1, further comprising providing at least one of the first representation, the second representation, the first intermediate representation, or an intermediate representation of the plurality of additional intermediate representations to a second robot.

14. identifying one or more robots based on at least one of the locations of the one or more robots or the locations of the robots, wherein the locations of the one or more robots are within a certain vicinity of the location of the robot; providing at least one of the first representation, the second representation, the first intermediate representation, or the intermediate representation of the plurality of additional intermediate representations to each of the one or more robots; The method of claim 1 further comprising:

15. identifying one or more robots based on at least one of the locations of one or more robots or the locations of the robots; obtaining at least a third representation from each of the one or more robots; The method of claim 1 further comprising:

16. The method of claim 1 , wherein the plurality of additional intermediate representations comprises five intermediate representations.

17. identifying a time period; periodically obtaining or providing a representation of the robot based on the time period; The method of claim 1 further comprising:

18. identifying a first period for a transition from the first representation to the second representation; identifying a second period, said second period indicating an intermediate representation period; identifying a number of steps based on the first period and the second period; generating a plurality of additional intermediate representations of the one based on the number of steps; The method of claim 1 further comprising:

19. The method of claim 1 , further comprising scaling the first intermediate representation to generate the plurality of additional intermediate representations.

20. 1. A system for modifying a representation of a robot, comprising: one or more processors; and a computer-readable memory containing instructions, the instructions causing the one or more processors to: obtaining a first representation of the robot corresponding to a first shape of the robot before a motion transition; obtaining a second representation of the robot corresponding to a predicted second shape of the robot after the motion transition; determining a first bounding box of the first representation; determining a second bounding box for the second representation; determining a first intermediate representation having the same general shape as the second representation and scaled to fit within the first bounding box; determining a plurality of additional intermediate representations that are incrementally scaled between the first intermediate representation and the second intermediate representation; The system is configured as follows:

Images