System and method for real-time control of a robot using a robot animation system

Abstract

Various systems and methods are disclosed for controlling a target device. For example, the system includes a user computing device with a user interface, a user motion database communicatively coupled to the user computing device, a controller, a controller motion database communicatively coupled to the controller, and a target device communicatively coupled to the controller. The user computing device can be configured to connect the user motion database and the controller motion database to share corresponding sets of motion instructions in real time in response to receiving a synchronization indication from the user computing device. The target device can be configured to implement the corresponding sets of motion instructions on the target device in real time.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority under 35 U.S.C. §119(e) to earlier U.S. Provisional Patent No. 63 / 278,793, filed November 12, 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0002] Field The present disclosure relates to control systems, and more particularly to control systems that operate in real time using database connectivity. [Background technology]

[0003] background Currently, various control systems exist with individualized sets of capabilities that are not integrated into one system and designed to work together as a single system. For example, a traditional control system workflow may enable motion editing through a system integrated with a programmable logic controller (PLC), which requires exporting and importing motion editing data. These traditional control systems do not enable real-time editing and may require multiple software packages to provide the target device with the ability to edit the necessary motion parameters. Additionally, traditional control systems do not provide the ability to visualize and interact with the target device in real time. Even if the subsystems can work together, the additional work required to connect the subsystems slows the workflow, and the time to iterate can take hours or longer. Summary of the Invention [Means for solving the problem]

[0004] The present disclosure addresses these challenges and issues, whereby a control system may include two databases with the ability to synchronize during programming to enable real-time visualization, interactivity, and kinematic modeling of target devices.

[0005] overview In accordance with one or more embodiments, a robot animation system for controlling a robot with real-time control is provided. The system includes a user computing device having a user interface, a user motion database communicatively coupled to the user computing device, a controller, a controller motion database communicatively coupled to the controller, and a target device communicatively coupled to the controller. The user computing device can be configured to connect the user motion database and the controller motion database to share corresponding sets of motion instructions in real time in response to receiving a synchronization instruction from the user computing device. The target device can be configured to implement the corresponding sets of motion instructions on the target device in real time.

[0006] According to another embodiment, a method for controlling a robot in real time is provided. The method can include receiving a synchronization instruction at a user motion database from a user computing device, the user computing device including a user interface. The method can also include connecting the user motion database and the controller motion database to share a corresponding set of motion instructions in real time. The method can further include implementing the corresponding set of motion instructions shared as a result of the connecting on a target device via the controller in real time.

[0007] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims, and drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure. The present specification also provides, for example, the following items: (Item 1) a user computing device having a user interface; a user motion database communicatively coupled to the user computing device; A controller; a controller motion database communicatively coupled to the controller; a target device communicatively coupled to the controller; A system comprising: the user computing device, configured to connect the user motion database and the controller motion database to share corresponding sets of motion instructions in real time in response to receiving a synchronization instruction from the user computing device; the target device is configured to implement the corresponding set of motion instructions on the target device in real time. system. (Item 2) Item 10. The system of item 1, wherein a doll is displayed on the user interface of the user computing device, and the doll is configured to perform movements corresponding to a corresponding set of implemented motion instructions in real time. (Item 3) Item 10. The system of item 1, wherein the user computing device is configured to receive user input and update the corresponding set of motion instructions in real time in response to the synchronization instruction being sent from the user computing device. (Item 4) the user computing device, obtaining the set of motion instructions from the user motion database, the set of motion instructions having been previously uploaded and stored in the user motion database; and generating and displaying a plurality of user-selectable user interface elements on the user interface of the user computing device corresponding to each motion in the set of motion instructions; Item 1. The system according to item 1, configured to: (Item 5) Item 10. The system of item 1, wherein the target device is configured to implement the corresponding set of motion instructions on the target device in real time in response to receiving play instructions from the user computing device. (Item 6) Item 6. The system of item 5, wherein the corresponding set of motion instructions is implemented on the target device by the user computing device sending a set of frames to the target device in a predetermined order. (Item 7) Item 10. The system of item 1, wherein each motion in the set of motion instructions comprises: (i) an instruction to perform a movement via the doll; and (ii) an instruction to perform a movement on the target device. (Item 8) Item 10. The system of item 1, wherein the target device comprises a plurality of actuators for performing movements on the target device. (Item 9) Item 10. The system of item 1, wherein the synchronization indication is transmitted in response to a selection of a user-selectable interface element displayed on the user interface of the user computing device. (Item 10) Item 1, a system according to item 1, wherein the controller receives the set of motion instructions from the controller motion database, and the controller selects the set of motion instructions related to instructions for performing movements on the target device and transmits the set of motion instructions to the target device. (Item 11) receiving a synchronization indication at a user motion database from a user computing device, the user computing device comprising a user interface; connecting the user motion database and the controller motion database to share corresponding sets of motion instructions in real time; implementing the corresponding set of motion instructions shared as a result of said connecting in real time on the target device via the controller; A method comprising: (Item 12) displaying a doll on the user interface of the user computing device; performing, in real time, movements corresponding to the corresponding set of implemented motion instructions using the doll; Item 12. The method of item 11, further comprising: (Item 13) Item 12. The method of item 11, further comprising: receiving user input at the user computing device; and updating the corresponding set of motion instructions in real time in response to the synchronization indication being sent from the user computing device. (Item 14) obtaining the set of motion instructions from the user motion database, the set of motion instructions having been previously uploaded and stored in the user motion database; and generating and displaying a plurality of user-selectable user interface elements on the user interface of the user computing device corresponding to each motion in the set of motion instructions; Item 12. The method of item 11, further comprising: (Item 15) Item 12. The method of item 11, further comprising implementing the corresponding set of motion instructions on the target device in real time in response to receiving play instructions from the user computing device. (Item 16) Item 16. The method of item 15, wherein the corresponding set of motion instructions is implemented on the target device by the user computing device sending a set of frames to the target device in a predetermined order. (Item 17) Item 12. The method of item 11, wherein each motion in the set of motion instructions comprises: (i) an instruction to perform a movement via the doll; and (ii) an instruction to perform a movement on the target device. (Item 18) Item 12. The method of item 11, wherein the target device comprises a plurality of actuators for performing movements on the target device. (Item 19) Item 12. The method of item 11, wherein the synchronization indication is sent in response to selection of a user-selectable interface element displayed on the user interface of the user computing device. (Item 20) Item 12. The method of item 11, wherein the controller receives the set of motion instructions from the controller motion database, and the controller selects the set of motion instructions related to instructions for performing movements on the target device to send to the target device. [Brief explanation of the drawings]

[0008] The present disclosure will become more fully understood from the detailed description and the accompanying drawings, wherein:

[0009] [Figure 1] FIG. 1 is an example user interface of a robot animation system according to some embodiments.

[0010] [Figure 2] FIG. 2 is a block diagram of an exemplary computing device according to some embodiments.

[0011] [Figure 3] FIG. 3 is an exemplary functional block diagram of a robot animation system according to some embodiments.

[0012] [Figure 4] FIG. 4 is an exemplary user interface for controlling the motion of a robot through a robot animation system according to some embodiments.

[0013] [Figure 5] FIG. 5 is an exemplary flowchart of a method of operating a robotic animation system according to some embodiments.

[0014] In the drawings, reference numbers may be reused to identify similar and / or identical elements. DETAILED DESCRIPTION OF THE INVENTION

[0015] Detailed Description A control system, such as a robot animation system, enables lifelike motion to be created on a robot through a series of interconnected subsystems. More specifically, the robot animation system implements multiple databases, including a user motion database and a controller motion database, and allows user modifications to the robot instructions before synchronizing the databases and then providing lifelike, continuous motion of the robot. Furthermore, the robot animation system enables an improved workflow that is not only iterative but also fast.

[0016] That is, the method of providing a user computing device with a user interface communicatively coupled to a user motion database allows a user or animator to add, edit, or otherwise update controller instructions to the intended motion of a robot and have those changes immediately reflected in the robot's user interface and the robot itself. A robot animation system may include multiple modules or subsystems with various parameters, such as, for example, the type, timing, and speed of motion by the robot, which a user can modify on the user computing device to adjust or change any robot motion to a desired appearance, angle, form, etc.

[0017] While this disclosure is directed to a specific system for controlling the motion of a robot, robot animation systems are generally directed to gathering information from a user, allowing that information to be modified in real time, and then applying that information to a robot by implementing motions on the robot. Thus, any target device other than a robot that uses a real-time data stream with configurable parameters that can be inserted can be used.

[0018] Furthermore, the robot animation system is designed to program robot motions in real time. As described above, the robot animation system includes at least two databases that are kept synchronized during programming. These databases include joint-space motion data for animated figures (e.g., robots). A controller motion database communicatively coupled to the controller provides or stores instructions for operation. The instructions indicate which motions to perform, in what sequence, and at what point in time, allowing the robot to operate smoothly without being affected by any potential delays on the user's computing device. In various implementations, a user can select a specific motion for a set of motions, such as a robot head movement or a robot arm movement, or the user can select a corresponding item or part of the robot along with the timing for adding the motion to the motion sequence. Furthermore, the user can order various motions, such as arm movements or head movements, along a timeline and drag and drop motions before or after other motions.

[0019] For example, at the beginning of an operating or programming session, a user may connect to the controller motion database by synchronizing with the controller motion database and connecting to multiple modules or stored data for operating a real-time system. Synchronizing with the controller motion database opens a bidirectional communication path that operates in real time between the user system and the robot system. The user may make edits to either the robot motion commands or instructions on a user interface of the robot's motion profile displayed on the user computing device of the robot. The user interface provides multiple user interface elements that are user selectable and designed to correspond to various types and angles of motion for various parts or components of the robot. The user interface may also include user interface elements corresponding to the speed of a robotic motion or a timeline of a sequence of robotic motions. For example, a user may select a user interface element corresponding to a particular motion to adjust which part of the robot performs the motion or at what point in time a particular motion occurs relative to other motions.

[0020] Thereafter, when a user desires a motion to be reflected or implemented by the robot, the user may select a user interface element on the robot animation application or program to instantly synchronize the user motion database with the controller motion database, the synchronization being achieved using packed data or at least one packet data or command message across a distributed communication system or distributed network. These commands may include frame numbers corresponding to specific motions at specific times along a timeline of a sequence of motions. Once the two databases are synchronized, various control modes become available for implementation to move the robot. The control modes may include a play mode, a go to specific point mode, and others, as described further below. The various control modes are user-selectable as icons on the user interface of the robot animation application and target specific points in the motion profile of the command. The control modes may also target the robot and / or the program including implementing the motion at full speed on the robot on the user interface.

[0021] When using the play mode, the user interface can send commands to the controller, which sends actual motion commands to the robot, starts playing or implementing the received motion commands, and starts listening for the current frame number. When the controller receives the play command, the controller commands the robot to start performing the animation and sends the current frame number to the user interface, which depicts the robot's actual real-time motion to the user. By receiving the current frame number, the user interface can display the robot's actual motion in synchronization with the controller, which commands the robot to perform the motion. When using the go-to mode, the user interface commands the controller in real time which frame to implement. That is, a user operating a user computing device can command the robot in real time to perform a specific motion related to a specific frame through the user interface of the robot animation application.

[0022] A user interface model of the robot is displayed on the user interface of the user computing device to display the real-time motion of the robot to the user as well as provide the user with a way to program the motion of the robot, for example by defining specific position or motion parameters for different frames. The robot animation system may have a variety of different user interfaces on the robot animation application with a variety of different features provided to the user for robot control and robot display.

[0023] In various implementations, the robot animation application provides a three-dimensional visual representation of the robot (e.g., as shown in FIG. 1) and allows a rig to be built within the robot animation application, with the rig providing a method for maneuvering the robot. Additionally or alternatively, the user interface may be simpler, with capabilities such as recording data points of position information that may include few or many different position parameters. The robot animation system is specifically targeted to providing a user with a way to view robot motion and to direct the motions the robot should perform, with the user interface designed to be separate from other parts of the robot animation system.

[0024] In various implementations, the same model is created for use in a real-time controller. These two models allow the system to be programmed into the space used in the user interface, and then that space is replayed and converted into the space used by the robot's actuators, which implement the motions the robot is commanded to move by the controller. The model implemented in the real-time controller converts data between the spaces at the same rate that the controller is operating.

[0025] The robot animation system may also include a set of real-time controller settings that affect the robot's motion. These settings are variables used in equations to create the motion of the robotic figure on the user interface in various modes. Exemplary settings may include a gain multiplier in the motion proportional-integral (PI) loop, an initial homing direction, and an expected count during homing. For example, more than 100 individual settings exist for each motor. These settings can be modified to individually change the behavior of each axis, even while the robot is in motion (see, for example, FIG. 4). These settings enable shaping each actuator's response to a motion profile. The described settings can be edited not only through the controller's application programming interface (API) but also through the user interface. That is, for example, the controller may include multiple robot control options as well as an interface for manipulation by an operator. Thus, the described API may enable the robot's settings to be modified during motion to accommodate various needs from the robot.

[0026] In various embodiments, the robot animation system provides an external visualization tool that is used to view and potentially control the robot. Some existing systems, such as Maya, Cinema4D, and Blender, naturally support this. Others (such as lighting systems) do not have this support, and therefore a separate tool, such as Sim Mechanics, Unity, or Unreal, must be used to provide external visualization. However, external visualization is integrated into all aspects of the robot animation system, allowing anyone on a network (which may be local or over a distributed communication system) to visualize the locations of all of the robot's joints in real time. In general, an accurate representation of the doll that can be viewed from any angle is very useful, especially when access to the doll is limited. In various implementations, the robot animation system uses network communication along with packed data to send the position of each joint across the network to user computing devices and potentially other networked devices as the position is changed.

[0027] The robot animation system is configured to allow interactivity between the robot and user input, not just pre-programmed data. In various implementations, the robot animation system includes a feature that allows external data streams to be used with each motion. That is, external data can be used to influence the robot's motion. For example, motion data is presented to the robot animation system as position data, where the position data is represented as a 32-bit floating-point number between 0 and 1. These data streams can be received from various sources, such as joysticks, artificial intelligence (AI), game systems, and so on. In implementations with interactive systems, sensor data can be used to create on-the-fly motion profiles, allowing this data to be blended or transitioned into and out of the motion profile data.

[0028] As described above, the method of storing, synchronizing, and using the received and stored data implemented in the user and controller motion databases is unique compared to any existing system. For example, the use of exact copies of data in two places (databases) that can be instantly synchronized and then used to keep the user interface robot and the robot moving simultaneously is new and unique to a robot animation system. As described above, synchronization is initiated by the user at any time the user chooses (e.g., by selecting a button or user interface element labeled "synchronize"). For example, a user may implement synchronization when they have finished editing on the user interface of the robot animation application and want to see the new edited motion of the robot.

[0029] Due to the speed and ease with which a user can implement synchronization, synchronization allows even small changes to be immediately visible. The ability to synchronize is part of a robot animation system that provides the ability to produce the best possible performance on a robot. Furthermore, a robot animation system can implement synchronization to provide visualization of the robot via a puppet and on the robot, allowing interactivity to be easily integrated into the robot animation system and further allowing motion parameters to be edited in real time by a user and an API. It will be apparent to those skilled in the art that the robot animation system of the present disclosure has advantages over other systems used to control robots. For example, the robot animation system allows a single system to control and display a robot in real time through synchronization between a user and a controller motion database.

[0030] Furthermore, the robot animation system's supporting workflow allows a significant amount of the underlying technology to be varied while achieving the same results. For example, the user motion database could be implemented using Autodesk Maya (Blender or a proprietary program could be substituted) as the primary data store, and the functionality of the robot animation system would still be the same.

[0031] The robot animation system defines a workflow that allows users to work with familiar tools, with features that enable them to quickly and efficiently perform the tasks necessary to adjust or update the robot's motion. As described above, the capabilities of the robot animation system include the ability to instantly view edits on the puppet and robot in the user interface, the ability to move to frames with a single click, and the ability to scrub the timeline live on the puppet. When the creative workflow is complete, the data in the robot animation system is already updated and the puppet is ready to play. This is because the user motion database and controller motion database are updated during the synchronization step. The robot animation system also captures all the parameters necessary to create the robot's motion and puts them into a single, easy-to-use tool. Furthermore, additional tools such as visualization, interactivity, and real-time kinematic modeling and models are omitted from other systems.

[0032] 1, an exemplary user interface 100 of a robot animation system according to some embodiments is shown. As depicted, the exemplary user interface 100 shows a puppet robot 101 in an exemplary modification application being operated on a user computing device. That is, the modification application is visible to the user, and in the modification application, the user can view the real-time motion of the robot via the puppet robot 101, and by using various toolbars (e.g., toolbar 102, etc.) on the exemplary user interface 100, the user can modify and update any desired motions to command the robot.

[0033] Referring to Figure 2, a block diagram of an exemplary computing device 200 is shown in accordance with some embodiments. Figure 2 illustrates an example computing device 200 that can be used by the disclosed system or to perform the disclosed methods. A computing device 200, such as a user computing device or controller of Figure 3, can, for example, implement one or more of the functions described herein. However, it should be understood that other computing device configurations are possible.

[0034] The computing device 200 may include one or more processors 201, memory 202, one or more input / output devices 203, a transceiver 204, one or more communication ports 207, and a display 206, all operatively coupled to one or more data buses 208. The data bus 208 enables communication between various devices. The data bus 208 may include a wired or wireless communication channel. The data bus 208 connects one or more devices.

[0035] Processor 201 may comprise one or more distinct processors, each having one or more cores. Each distinct processor may have the same or different architecture. Processor 201 may comprise one or more central processing units (CPUs), one or more graphics processing units (GPUs), application specific integrated circuits (ASICs), digital signal processors (DSPs), and the like.

[0036] The processor 201 may be configured to perform a function or operation by executing code, which is code stored in an instruction memory and that implements the function or operation. For example, the processor 201 may be configured to perform one or more of any of the functions, methods, or operations disclosed herein.

[0037] The memory 202 may comprise an instruction memory that may store instructions that may be accessed (e.g., read) and executed by the processor(s) 201. For example, the instruction memory may be a non-transitory computer-readable storage medium such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a removable disk, a CD-ROM, any non-volatile memory, or other suitable memory. For example, the instruction memory may store instructions that, when executed by the one or more processors 201, cause the one or more processors 201 to perform one or more of the functions of the image reconstruction system.

[0038] The memory 202 may also comprise a working memory. The processor 201 may store data in the working memory and read data from the working memory. For example, the processor 201 may store a working set of instructions, such as instructions loaded from an instruction memory, in the working memory. The processor 201 may also use the working memory to store dynamic data created during operation of the computing device 200. The working memory may be, for example, a random access memory (RAM), such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), or other suitable memory.

[0039] Input / output device 203 may comprise any suitable device that allows for data input or output. For example, input / output device 203 may comprise one or more of a keyboard, touchpad, mouse, stylus, touchscreen, physical buttons, speakers, microphone, or other suitable input or output device.

[0040] The communications port 207 may comprise, for example, a serial port, such as a universal asynchronous receiver-transmitter (UART) connection, a universal serial bus (USB) connection, or other suitable communications port or connection. In some examples, the communications port 207 allows for programming executable instructions into an instruction memory. In some examples, the communications port 207 allows for the transfer of data (e.g., uploading or downloading).

[0041] The display 206 may display a user interface 205. The user interface 205 may enable user interaction with the computing device 200. In some examples, a user may interact with the user interface 205 by engaging with the input / output device 203. In some examples, the display 206 may be a touchscreen, and the user interface 205 may be displayed on the touchscreen.

[0042] The transceiver 204 may enable communication with a network, such as a Wi-Fi network, an Ethernet network, a cellular network, or other suitable communication network. For example, the transceiver 204 is configured to enable communication with the cellular network when operating on the cellular network. The processor 201 is operable to receive data from the network or send data to the network via the transceiver 204.

[0043] Referring to FIG. 3 , an exemplary functional block diagram of a robot animation system 300 according to some embodiments is shown. The robot animation system 300 includes a user computing device 304, such as a handheld computing device, laptop, tablet, desktop, etc., capable of operating a robot animation application through which a user can modify and view the motion of a target device 306, such as the robot illustrated in FIG. 3 . As previously mentioned, the target device 306 includes multiple actuators located at multiple locations 308a-308f throughout the target device 306 to perform or execute movements. While only six actuators and locations are shown, it can be understood that multiple actuators can be located anywhere on the target device 306 that can be commanded to move and are not limited to the example shown in FIG. 3 .

[0044] The user computing device 304 updates the user motion database 310 with any changes to the motion information for commanding the target device 306. As a result of the synchronization, two-way communication begins between the user motion database 310 and the controller motion database 312 (e.g., a robot motion database, etc.), allowing the motion parameter module 316, the motion data parameter module 320, the visualization module 324, the interactivity module 328, and the control command module 332 to also be synchronized between the user computing device 304 and the controller 336 (e.g., a robot controller, etc.). The user computing device 304 and the controller 336 synchronize in response to a user selection to enable real-time visualization of the doll and the target device 306 on a user interface (e.g., user interface 205, etc.) of the user computing device 304, the visualization resulting from the controller 336 outputting control signals to the target device 306 as instructed in the user motion database 310 as a result of the synchronization.

[0045] For example, the control command module 332 may include specific information regarding which frames to follow with specific motion commands from the motion parameter module 316 and / or the motion data parameter module 320, which dictate which motions to implement. In various implementations, the visualization module 324 enables real-time feedback to the user computing device 304 for the user to visualize the motion of the target device 306. Additionally, the interactivity module 328 enables real-time user input via the user computing device 304 (e.g., through the input / output device 203 and / or the display 206) to control the target device 306.

[0046] Referring now to FIG. 4 , an exemplary user interface 400 for controlling robot motion through a robot animation system according to some embodiments is shown. For example, the exemplary user interface 400 depicts an example robot animation application through which the motion parameters of the robot 401 can be viewed and set. A user can adjust various motions at various points on the robot and at various times. The user interface 400 can also provide indications 402 of variations between the robot's 401's commanded motion settings (e.g., position and velocity) relative to the robot's 401's actual motion. In some embodiments, the user interface 400 can also include fault indications 403. These fault indications 403 can comprise faults to or from the robot 401 and / or other components of the robot animation system (e.g., the user motion database 310 or the controller motion database 312). The user interface 400 can also include play interface elements 404 to allow a user to start and stop the robot 401.

[0047] 5, a flowchart of an exemplary method 500 of operating the robotic animation system 300 is shown according to some embodiments. The method 500 begins at step 502. At step 504, a synchronization instruction is received at the user motion database 310 from the user computing device 304, which may further comprise the user interface 100. At step 506, the user motion database 310 and the controller motion database 312 are connected to share corresponding sets of motion instructions in real time. At step 508, the target device 306 implements the corresponding sets of motion instructions shared as a result of the connecting in real time via the controller 336. The method 500 ends at step 510.

[0048] In some embodiments, the method 500 can comprise displaying the doll 101 on the user interface 100 of the user computing device 304 and performing movements corresponding to the corresponding set of implemented motion instructions in real time using the doll 101. In further embodiments, the method 500 can comprise receiving user input at the user computing device 304 and updating the corresponding set of motion instructions in real time in response to a synchronization instruction being sent from the user computing device 304. In some embodiments, the method 500 can comprise retrieving the set of motion instructions from the user motion database 310, where the set of motion instructions was previously uploaded and stored in the user motion database 310. The method 500 can then generate and display user-selectable user interface elements (e.g., elements found in the toolbar 102) on the user interface 100 of the user computing device 304 corresponding to each motion in the set of motion instructions. In some embodiments, the method 500 can further comprise implementing the corresponding set of motion instructions on the target device 306 in real time in response to receiving a play instruction from the user computing device 304.

[0049] The system includes a user computing device having a user interface, a user motion database communicatively coupled to the user computing device, a controller, a controller motion database communicatively coupled to the controller, and a target device communicatively coupled to the controller. The user computing device is configured to connect the user motion database and the controller motion database to share corresponding sets of motion instructions in real time in response to receiving a synchronization instruction from the user computing device. The target device is configured to implement the corresponding sets of motion instructions on the target device in real time.

[0050] In some embodiments, a doll is displayed on a user interface of a user computing device, the doll being configured to perform movements in real time corresponding to a corresponding set of implemented motion instructions.

[0051] In some embodiments, the user computing device is configured to receive user input and update the corresponding set of motion instructions in real time in response to synchronization instructions being sent from the user computing device.

[0052] In some embodiments, the user computing device is configured to retrieve the set of motion instructions from a user motion database, the set of motion instructions having been previously uploaded and stored in the user motion database, and the user computing device may generate and display a plurality of user-selectable user interface elements on a user interface of the user computing device corresponding to each motion in the set of motion instructions.

[0053] In some embodiments, the target device is configured to implement a corresponding set of motion instructions on the target device in real time in response to receiving play instructions from the user computing device.

[0054] In some embodiments, the corresponding set of motion instructions is implemented on the target device by the user computing device sending a set of frames to the target device in a predetermined order.

[0055] In some embodiments, each motion in the set of motion instructions comprises (i) an instruction to perform the motion via the doll and (ii) an instruction to perform the motion on the target device.

[0056] In some embodiments, the target device comprises multiple actuators for performing movements on the target device.

[0057] In some embodiments, the synchronization indication is sent in response to a selection of a user-selectable interface element displayed on a user interface of the user computing device.

[0058] In some embodiments, the controller receives a set of motion instructions from a controller motion database, and the controller selects a set of motion instructions to send to the target device that relate to instructions to perform a movement on the target device.

[0059] In some embodiments, the method includes receiving a synchronization instruction at a user motion database from a user computing device, the user computing device including a user interface. The method also includes connecting the user motion database and the controller motion database to share corresponding sets of motion instructions in real time. The method further includes implementing the corresponding sets of motion instructions shared as a result of the connecting on the target device via the controller in real time.

[0060] In some embodiments, the method comprises displaying a doll on a user interface of a user computing device and performing, in real time, movements corresponding to a corresponding set of implemented motion instructions using the doll.

[0061] In some embodiments, the method comprises receiving user input at a user computing device and updating a corresponding set of motion instructions in real time in response to a synchronization indication being sent from the user computing device.

[0062] In some embodiments, the method comprises obtaining a set of motion instructions from a user motion database, the set of motion instructions having been previously uploaded and stored in the user motion database, and generating and displaying a plurality of user-selectable user interface elements on a user interface of a user computing device corresponding to each motion in the set of motion instructions.

[0063] In some embodiments, the method comprises implementing a corresponding set of motion instructions in real time on the target device in response to receiving play instructions from the user computing device.

[0064] In some embodiments of the method, the corresponding set of motion instructions is implemented on the target device by the user computing device sending a set of frames to the target device in a predetermined order.

[0065] In some embodiments of the method, each motion in the set of motion instructions comprises (i) an instruction to perform the motion via the doll and (ii) an instruction to perform the motion on the target device.

[0066] In some embodiments of the method, the target device comprises a plurality of actuators for performing movements on the target device.

[0067] In some embodiments of the method, the synchronization indication is sent in response to a selection of a user-selectable interface element displayed on a user interface of the user computing device.

[0068] In some embodiments of the method, the controller receives a set of motion instructions from a controller motion database, and the controller selects a set of motion instructions related to instructions to perform a movement on the target device, the set of motion instructions for transmission to the target device.

[0069] Although the methods described above involve reference to flowcharts, it will be understood that many other ways of performing the acts involved in the methods can be used. For example, the order of some operations may be changed, and some of the operations described may be optional.

[0070] Additionally, the methods and systems described herein may be embodied, at least in part, in the form of computer-implemented processes and apparatuses for practicing those processes. The disclosed methods may also be embodied, at least in part, in the form of a tangible, non-transitory, machine-readable storage medium encoded with computer program code. For example, the method steps may be embodied in hardware, executable instructions (e.g., software) executed by a processor, or a combination of the two. The medium may comprise, for example, RAM, ROM, CD-ROM, DVD-ROM, BD-ROM, hard disk drive, flash memory, or other non-transitory, machine-readable storage medium. When the computer program code is loaded into a computer and executed by the computer, the computer becomes an apparatus for practicing the method. The methods may also be embodied, at least in part, in the form of a computer into which the computer program code is loaded or executed, thereby making the computer a special-purpose computer for practicing the method. When implemented on a general-purpose processor, the computer program code segments configure the processor to create specific logic circuits. Alternatively, the methods may be embodied, at least in part, in an application-specific integrated circuit for performing the method.

[0071] In this application with the definitions below, the term "module" or the term "controller" may be substituted for the term "circuitry." The term "module" may refer to, be part of, or comprise processor hardware (shared, specialized, or a group) that executes code and storage hardware (shared, specialized, or a group) that stores code that is executed by the processor hardware.

[0072] The module may include one or more interface circuits. In some examples, the interface circuit may implement a wired or wireless interface to connect to a local area network (LAN) or a wireless personal area network (WPAN). Examples of a LAN are the IEEE (Institute of Electrical and Electronics Engineers) standard 802.11-2016 (also known as the WIFI wireless networking standard) and the IEEE standard 802.3-2015 (also known as the ETHERNET wired networking standard). Examples of a WPAN are the BLUETOOTH wireless networking standard from the Bluetooth Special Interest Group and IEEE standard 802.15.4.

[0073] Modules may communicate with other modules using interface circuits. While modules may be depicted in this disclosure as logically communicating directly with other modules, in various implementations, modules may actually communicate through a communications system. A communications system may comprise physical and / or virtual networking equipment, such as hubs, switches, routers, and gateways. In some implementations, a communications system may connect to or route through a wide area network (WAN), such as the Internet. For example, a communications system may comprise multiple LANs connected to each other through the Internet or point-to-point leased lines using technologies including Multiprotocol Label Switching (MPLS) and Virtual Private Networks (VPNs).

[0074] In various implementations, the functionality of a module may be distributed among multiple modules connected via a communication system. For example, multiple modules may implement the same functionality that is distributed by a load balancing system. In a further example, the functionality of a module may be divided between a server module (also known as a remote or cloud) and a client (or user) module.

[0075] The foregoing is provided for purposes of illustrating, explaining, and describing embodiments of these disclosures. Modifications and adaptations to these embodiments will be apparent to those skilled in the art and can be made without departing from the scope or spirit of these disclosures.

Claims

1. a user computing device having a user interface; a user motion database communicatively coupled to the user computing device; A controller; a controller motion database communicatively coupled to the controller; a target device communicatively coupled to the controller; A system comprising: the user computing device, configured to connect the user motion database and the controller motion database to share corresponding sets of motion instructions in real time in response to receiving a synchronization instruction from the user computing device; the target device is configured to implement the corresponding set of motion instructions on the target device in real time. system.

2. 10. The system of claim 1, wherein a doll is displayed on the user interface of the user computing device, the doll being configured to perform movements in real time corresponding to a corresponding set of implemented motion instructions.

3. 2. The system of claim 1, wherein the user computing device is configured to receive user input and update the corresponding set of motion instructions in real time in response to the synchronization indication being sent from the user computing device.

4. the user computing device, obtaining the set of motion instructions from the user motion database, the set of motion instructions having been previously uploaded and stored in the user motion database; and generating and displaying a plurality of user-selectable user interface elements on the user interface of the user computing device corresponding to each motion in the set of motion instructions; The system of claim 1 configured to:

5. The system of claim 1 , wherein the target device is configured to implement the corresponding set of motion instructions on the target device in real time in response to receiving play instructions from the user computing device.

6. The system of claim 5 , wherein the corresponding set of motion instructions is implemented on the target device by the user computing device sending a set of frames to the target device in a predetermined order.

7. 3. The system of claim 2, wherein each motion in the set of motion instructions comprises: (i) an instruction to perform a movement via the doll; and (ii) an instruction to perform a movement on the target device.

8. The system of claim 1 , wherein the target device comprises a plurality of actuators for performing movements on the target device.

9. The system of claim 1 , wherein the synchronization indication is transmitted in response to a selection of a user-selectable interface element displayed on the user interface of the user computing device.

10. 2. The system of claim 1, wherein the controller receives the set of motion instructions from the controller motion database, and the controller selects the set of motion instructions to send to the target device that relate to instructions to perform a movement on the target device.

11. receiving a synchronization indication at a user motion database from a user computing device, the user computing device comprising a user interface; connecting the user motion database and the controller motion database to share corresponding sets of motion instructions in real time; implementing the corresponding set of motion instructions shared as a result of said connecting in real time on the target device via the controller; A method comprising:

12. displaying a doll on the user interface of the user computing device; performing, in real time, movements corresponding to the corresponding set of implemented motion instructions using the doll; The method of claim 11 further comprising:

13. 12. The method of claim 11, further comprising: receiving user input at the user computing device; and updating the corresponding set of motion instructions in real time in response to the synchronization indication being sent from the user computing device.

14. obtaining the set of motion instructions from the user motion database, the set of motion instructions having been previously uploaded and stored in the user motion database; and generating and displaying a plurality of user-selectable user interface elements on the user interface of the user computing device corresponding to each motion in the set of motion instructions; The method of claim 11 further comprising:

15. The method of claim 11 , further comprising: implementing the corresponding set of motion instructions in real time on the target device in response to receiving play instructions from the user computing device.

16. The method of claim 15 , wherein the corresponding set of motion instructions is implemented on the target device by the user computing device sending a set of frames to the target device in a predetermined order.

17. The method of claim 12 , wherein each motion in the set of motion instructions comprises: (i) an instruction to perform a movement via the doll; and (ii) an instruction to perform a movement on the target device.

18. The method of claim 11 , wherein the target device comprises a plurality of actuators for performing movements on the target device.

19. The method of claim 11 , wherein the synchronization indication is sent in response to selection of a user-selectable interface element displayed on the user interface of the user computing device.

20. 12. The method of claim 11, wherein the controller receives the set of motion instructions from the controller motion database, and the controller selects the set of motion instructions to send to the target device that relate to instructions to perform a movement on the target device.

Images