Computer Automation Script Electronic Actor Control

Abstract

A computer-implemented system and method uses a story to automatically control and coordinate electronic actors. The story can be written in a natural language (such as English) rather than a programming language or other language specifically designed to control machines. An embodiment of the invention can parse the story into a script and generate cue signals based on the story. The cue signals are provided to the electronic actors, which can perform actions based on their interpretation of those cue signals. As a result, the story, script, and cue signals issued to the electronic actors do not deterministically dictate the actions to be performed by the electronic actors.

Description

[Background technology]

[0001] There are various techniques for controlling animatronics, toys, and other devices (all of which are examples of "electronic actors" as the term is used herein) to perform actions described or implied by a script. Such techniques generally require a great deal of control and effort by a human programmer and / or operator.

[0002] For example, one or more human operators may directly control the electronic actors live, such as using the Sphero app (for controlling Sphero robots) or the Anduril Lattice platform. While this type of control is effective for controlling electronic actors to perform desired actions, human operators must manually control the electronic actors to perform such actions, which is tedious, time-consuming, and expensive.

[0003] Trigger-based control systems contain rules, and each rule contains conditions and actions. When the rule's conditions are met, the rule's actions are executed. An example would be sounding an alarm (action) at a specified time (condition). Examples of such trigger-based control systems are the Ift service, the Philips Hue app, and Pharos Designer 2, which perform a specified action (That) when a certain trigger (This) occurs. Such systems can be used to control electronic actors by creating rules for the actions that the electronic actors should perform. Although trigger-based control systems introduce a degree of automation compared to live manual control, rules in trigger-based control systems must be programmed manually, carefully anticipating all the relevant conditions that may arise and their corresponding actions. This can be tedious, time-consuming, and error-prone. Furthermore, trigger-based control systems are fixed in the sense that the same action is always executed when the same conditions are met. Thus, such systems are not suitable for situations where a variety of actions are required.

[0004] Programming-based control systems are more flexible than trigger-based control systems, allowing human programmers to create actor control programs that contain more sophisticated logic and action combinations than trigger-based control systems. Examples of programming-based control systems include block programming systems (e.g., Lego Mindstorms, Scratch, Blockly, and MIT App Inventor) and coding-based systems (e.g., Mu Editor). Such systems have the advantage of being able to control electronic actors in a more flexible and sophisticated way than trigger-based control systems, and of not requiring live control, but they require a significant amount of time to learn how to use and program them, and they control electronic actors deterministically. Thus, such systems are not suitable for situations where a greater variety of actions is required.

[0005] What is needed, therefore, is a system for automating the control of electronic actors that can be used with minimal human effort and that can generate a variety of actions based on the same human input. Summary of the Invention

[0006] A computer-implemented system and method automatically controls and coordinates electronic actors using stories. The stories can be written in a natural language (e.g., English) rather than a programming language or other language specifically designed to control machines. An embodiment of the invention parses the stories into scripts and generates cue signals based on the stories. The cue signals are provided to the electronic actors, which can perform actions based on their interpretation of those cue signals. As a result, the stories, scripts, and cue signals issued to the electronic actors do not deterministically dictate the actions performed by the electronic actors.

[0007] Other features and advantages of various aspects and embodiments of the invention will become apparent from the following description and from the claims. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram of a computer-implemented system for automatically controlling one or more electronic actors using a story, according to one embodiment of the present invention. [Diagram 2] 2 is a flowchart of a method performed by the system of FIG. 1 according to one embodiment of the present invention. [Diagram 3] FIG. 1 is a diagram of a system for loading scripts according to one embodiment of the present invention. [Figure 4] FIG. 1 is a diagram of a system for playing back (executing) a script according to one embodiment of the present invention. [Diagram 5] FIG. 1 illustrates a system for processing scripts according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A computer-implemented system and method automatically controls and coordinates electronic actors using stories. The stories may be written in a natural language (such as English) rather than a programming language or other language specifically designed to control machines. An embodiment of the invention may parse the stories into scripts and generate cue signals based on the stories. The cue signals are provided to the electronic actors, which may interpret the cue signals and perform actions based on their interpretation of the cue signals. As a result, the stories, scripts, and cue signals issued to the electronic actors do not deterministically dictate the actions to be performed by the electronic actors.

[0010] Referring to Figure 1, a diagram of a computer-implemented system 100 for automatically controlling one or more electronic actors 124 using a story 102 is shown in accordance with one embodiment of the present invention. Referring to Figure 2, a flowchart of a method 200 performed by the system 100 of Figure 1 in accordance with one embodiment of the present invention is shown.

[0011] The system 100 includes a story 102. The story 102 may include, for example, any combination of one or more of the following: Text data (e.g., text data representing text written in a natural language such as English). Audio data (e.g. audio data representing natural speech and / or synthetic speech). Image data (for example: image data representing natural and / or synthetic images). Video data (for example: video data representing natural video and / or synthetic video).

[0012] As one particular example, a story 102 may include text data and may not include audio data. As another example, a story 102 may include text data and may not include audio data, image data, or video data. For example, a story 102 may include only text data, which may include or consist of text data representing data written in, for example, a natural language.

[0013] A story 102 may include, for example, data describing a story involving one or more characters. A story 102 may include, for example, data describing one or more of the following in any combination: words spoken by a character, actions taken by a character, emotions of a character, and locations within a story. A story 102 may include, for example, natural language text in the form of a screenplay or script. A story 102 may include, for example, only text written in a natural language or may include text written in a natural language as well as text and other data that is not written in a natural language (e.g., text written in a programming language or scripting language).

[0014] The system 100 also includes an analysis module 104. The analysis module 104 receives the story 102 as input ( FIG. 2 , operation 202). For example, the story 102 may be stored in a file in a file system, and the analysis module 104 may receive the story 102 as input by opening and reading the file. As another example, a human user (not shown) may type, speak, or otherwise manually input the story 102 into the analysis module 104 using one or more input devices (such as a keyboard, touch screen, microphone, etc.) and associated software (such as a text editing application or a voice recognition application).

[0015] More generally, components of system 100 (e.g., one or more electronic actors 124) may receive input from one or more human users without the input being intentionally provided by the human users. For example, system 100 may use any of a variety of sensors (e.g., one or more cameras, microphones, and / or physiological sensors) to obtain sensor data (e.g., image data, video data, audio data, and / or physiological data, any combination thereof) from one or more human users, regardless of whether the input was intentionally provided by the one or more human users. For example, system 100 may obtain sensor data from a human user as the human user naturally interacts with system 100 or simply resides within system 100's environment without intentionally interacting with system 100. Such sensor data is an example of the term "input" as used herein. Thus, it should be understood that functions disclosed herein as being performed on "input" may be performed on any input, such as input received from one or more human users. This includes input intentionally provided and / or input not intentionally provided by one or more human users. Just one example of a function that system 100 may perform in response to such input is to derive an estimate of the emotional state of a human user based in whole or in part on the input received from that human user.

[0016] The analysis module 104 generates a script 106 based on the story 102 ( FIG. 2 , operation 204). The script 106 may include data representing, for example, one or more lines 108, one or more characters 112, and one or more locations 116 in any combination. As described in more detail below, the analysis module 104 may generate the lines 108, the characters 112, and / or the locations 116 based on the story 102 in any of a variety of ways. In general, the analysis module 104 may identify content (e.g., text data, audio data, image data, and / or video data) within the story 102 that represents the lines 108 and generate the dialogue 108 based on that content. Similarly, the analysis module 104 may identify content within the story 102 that represents the characters 112 and generate the characters 112 based on that content. Similarly, the analysis module 104 may identify content within the story 102 that represents the locations 116 and generate the locations 116 based on that content.

[0017] As described in more detail below, embodiments of the present invention may, but need not, include all of the elements illustrated in Figure 1 and may, but need not, perform all of the operations illustrated in Figure 2. By way of example only, some embodiments of the present invention may not include story 102 and / or analysis module 104 or may not perform operations 202 and 204. Such embodiments may, for example, include or receive script 106 but may not generate script 106. For example, such embodiments may not generate script 106 based on story 102.

[0018] As another example, functionality disclosed herein as applied to script 106 may additionally or alternatively be applied to story 102. For example, embodiments of the present invention may omit analysis module 104 and script 106, and embodiments of the present invention may not perform operation 204, in which case embodiments of the present invention may perform other functionality disclosed herein (e.g., operation 206) on story 102 rather than script 106. As this illustrates, functionality such as operation 206 may be performed directly on story 102 without generating script 106 based on story 102. These are merely examples, and other embodiments are within the scope of the present invention.

[0019] In Figure 1, rows 108 are shown to include rows 110a-i, where i is any number. Similarly, in Figure 1, characters 112 are shown to include characters 114a-j, where j is any number. Similarly, in Figure 1, locations 116 are shown to include locations 118a-k, where k is any number. The values ​​of i, j, and k may be the same or different from one another and need not bear any relationship to one another.

[0020] Script 106 may include data (not explicitly shown in FIG. 1 ) that represents relationships between lines 108, characters 112, and locations 116. For example, script 106 may include data indicating that a particular one of lines 108 is spoken by a particular one of characters 112. As another example, script 106 may include data indicating that a particular one of lines 108 is spoken at a particular one of locations 116. As yet another example, script 106 may include data indicating that a particular one of characters 112 is located at a particular one of locations 116.

[0021] The rows 108 may take any of a variety of forms and may include any of a variety of data. For example, each line may be any of the following types: narration, dialogue, and non-dialogue. For example, dialogue may include text, such as text representing words intended to be spoken by one or more characters 112. As another example, dialogue may include audio data, such as audio data representing words intended to be spoken by one or more characters 112. As another example, dialogue may include a description of an action to be performed by one of the characters 112. Such an action description may define the action at any level of generality, by describing a specific physical action to be performed by the character (e.g., move right arm to left), or by describing a general action to be performed by the character without specifying a specific physical action to perform that general action (e.g., move both feet to left), or by describing an emotion to be expressed (e.g., joy or surprise).

[0022] The analysis module 104 may generate the dialogue 108 based on the story 102 in any of a variety of ways. For example, the analysis module 104 may copy dialogue from the story 102 verbatim, such as where a dialogue included in the story 102 is copied into one of the dialogues 108. However, some or all of the dialogue 108 in the script 106 may not explicitly appear in the story 102. For example, if the story 102 includes content that describes and / or displays a character expressing happiness, the analysis module 104 may generate one or more dialogues 108 based on the content that include specific text expressing happiness (e.g., "I feel happy") and / or specify an action expressing happiness (e.g., smiling).

[0023] When generating dialogue 108 based on the story 102, the analysis module 104 can distinguish between actionable dialogue (e.g., dialogue describing an action that one of the electronic actors 124 can perform) and non-actionable dialogue (e.g., dialogue describing an action that cannot be performed by any of the electronic actors 124). Actionable dialogue includes, for example, action verbs related to movement (e.g., "walk", "fly", "jump"), non-verbal communication (e.g., "bark", "grumble"), and change of state (e.g., "light up"). Examples of text in the story 102 that may cause the analysis module 104 to generate actionable (non-dialogue) lines 108 associated with a character 112 in the script 106 include, "Victor jumped up when he heard the news", "The dog began to bark", and "Lily's face lit up".

[0024] When the analysis module 104 encounters a non-actionable line in the story 102, the analysis module 104 may, for example, include the line in the script 106 as a line of narration. Such a line of narration may or may not be associated with one of the characters 112. For example, a line of narration may not be associated with any character 112 and may instead be spoken in the voice of a non-character narrator. An example of text in the story 102 that the analysis module 104 may interpret as a non-action line and include as a non-dialogue line in the script 106 is, "Victor was going to the store, but decided not to."

[0025] Characters 112 may take any of a variety of forms and may include any of a variety of data, for example, each character 112 may include any one or more of the following in any combination: text representing the character's name, an image and / or video depicting the character, a unique identifier for the character, and a description of the character.

[0026] The analysis module 104 may generate characters 112 based on the story 102 in any of a variety of ways. For example, if the story 102 is in the form of a screenplay, script, or other text that specifically assigns lines or actions to characters by name (e.g., "Mary: Welcome home") or descriptor (e.g., "The postman"), the analysis module 104 may generate characters with names or descriptors included in the story 102. As another example, if the story includes one or more images or videos of characters, the analysis module 104 may use image recognition techniques to identify characters in those images or videos and create corresponding characters 112 in the script 106.

[0027] The locations 118a-k may take any of a variety of forms and include any of a variety of data. For example, a location may include data specifying an absolute location (e.g., GPS coordinates) or a relative location (e.g., 2 feet to the right). A location may include data specifying a location, a place specified by a name or other description (e.g., "home," "office," "red brick building," "park," etc.), or a character (to indicate the location of the character).

[0028] The analysis module 104 can generate locations 116 based on the story 102 in any of a variety of ways. For example, if the story 102 is in the form of a screenplay, script, or other text that includes stage directions or other directions associated with characters, the analysis module 104 can generate one or more locations that correspond to those stage directions. As another example, if the story 102 includes text that describes locations, the analysis module 104 can generate locations that include verbatim copies of that text. As another example, if the story 102 includes text that describes locations without specifying coordinates or another location, the analysis module 104 can generate locations that specify specific coordinates or another location based on that text. As another example, if the story 102 includes an image or video of a location, the story 102 can generate locations that include the image or video and / or text that describes the location that appears in the image or video.

[0029] One way that the script 106 can indicate that the location of a character 112 may change over time is by storing data representing an association that associates a particular line 108 with a particular character 112 and a particular location 116, thereby indicating that the specified character is in the specified location when speaking the specified line. The script 106 can also include data representing an association of the same character with a different line and location, thereby indicating that the specified character is in the different location when speaking a different line.

[0030] The script 106 may, but need not, be stored in a human-readable format, such as a traditional screenplay or script. For example, the script 106 may include text in a format such as "<character name>:<dialogue>" (e.g., "Victor: Good morning"). Even if the script 106 is not stored in such a human-readable format, the system 100 may still represent the script 106 in such a format for a human user to view on any output device, such as a display screen.

[0031] The system 100 also includes an electronic actor coordination module 120 that generates one or more cue signals 122 based on the script 106 ( FIG. 2 , operation 206). As described in more detail below, the electronic actor coordination module 120 generates the cue signals 122 that coordinate one or more electronic actors 124 to perform actions consistent with the script 106 (or, if the script 106 was generated based on or associated with the story 102, consistent with the story 102).

[0032] The cue signal 122 may be or include, for example, any combination of one or more of the following: Text, such as written text in a natural language. This can represent, for example, audio (e.g., part or all of a conversation). Audio cues, e.g. text, instructions for generating audio data, or the audio data itself. Data specifying the audio. Lighting cues. For example, cues that specify a particular exterior hue, brightness, atmospheric lighting, and / or directional lighting. Movement cues.

[0033] The electronic actor adjustment module 120 is, for example, physically distinct from one or more of the electronic actors 124. For example, the electronic actor adjustment module 120 may be physically distinct from some or all of the electronic actors 124 (e.g., does not include the electronic actor 124, is contained within the electronic actor 124, is not attached to the electronic actor 124), and the electronic actor adjustment module 120 may communicate with some or all of the electronic actors 124 (e.g., provides cue signals 122 to the electronic actors 124 and receives feedback 130 from the electronic actors 124) via wired and / or wireless network connections. Alternatively, for example, one or more of the electronic actors 124 may include the electronic actor adjustment module 120. For example, two or more electronic actors 124 may each include their own instance of the electronic actor adjustment module 120. As this suggests, the electronic actor adjustment module 120 shown in FIG. 1 may be implemented using multiple electronic actor adjustment modules.

[0034] The electronic actor coordination module 120 provides (e.g., transmits) a cue signal 122 to an electronic actor 124 ( FIG. 2 , operation 208), and in response, the electronic actor 124 performs one or more actions based on the cue signal 122 ( FIG. 2 , operation 210). Although there may be multiple electronic actors 124, the description herein may refer to only a single electronic actor 124 for simplicity of explanation. It should be understood that such description is equally applicable to multiple electronic actors.

[0035] The electronic actor coordination module 120 may provide the cue signals 122 to the electronic actors 124 in any of a variety of ways. For example, the electronic actor coordination module 120 may provide the cue signals to the electronic actors by transmitting the cue signals over a network. Such transmitting may include, for example, addressing the cue signals to the electronic actors. The network may include, for example, a local area network (LAN) and / or a wide area network (WAN) (such as the Internet) in any combination. The network may be any combination of wired and / or wireless. Such transmitting may include, for example, using Bluetooth and / or near field communication (NFC) to transmit the cue signals to the electronic actors. Such transmitting may include, for example, transmitting the cue signals (e.g., over a network) to a device other than the electronic actor 124 (e.g., a server), which may transmit the cue signals directly or indirectly to the electronic actors 124.

[0036] The cue signal 122 may provide enough information for a corresponding electronic actor 124 to determine how to act (or not act) based on the cue signal 122. A cue signal 122 for a particular action may include, for example, the original "stem" action verb from the script 106 and / or story 102 and its hierarchy of hypernyms (more general words that group a particular set of words together in meaning). Consider the following example:

[0037] Example Cue Signal #1: The original line of script 106 is "Victor ran into the house as soon as he heard the explosion." The resulting cue signal can be specified as "Run into (30, 6, 10)." In this example, "run" is the stem action verb used in the original line. "Move" is a hypernym for "run." "In" indicates that the action has a target. (30, 6, 10) is the target of the action (which could be another electronic actor).

[0038] Example Cue Signal #2: The original line of script 106 is "Tom smiled when he heard the news." The resulting cue signal can be specified as smile(frown, tell, interact, move). In this example, "smile" is the stem action verb used in the original line. "frown", "tell", "interact", and "move" are its hypernyms.

[0039] Now consider three robots (examples of electronic actors). Robot #1 has legs, arms, and a static face (like a toy soldier). Robot #2 has wings and a dynamic face (like a flying monkey). Robot #3 has no ability to move, no face, and only one LED (like a phone).

[0040] For cue signal #2 (smile (frown, communication, interaction, movement)): Robot #1 only knows the action verb "move". However, since there is no target, the robot performs a generic routine of flinching its arms and legs. Robot #2 recognizes "smile" and displays a smile by activating the "smile" routine in the face module. Robot #3 cannot interpret "smile" or "frown". Instead, it has a general routine for "communication", which involves blinking a single LED.

[0041] The cue signal 122 may, but need not, specify a particular action to be performed by the electronic actors 124. For example, the cue signal 122 may specify that one of the electronic actors 124 is to perform a particular physical action, and in response, that electronic actor may perform that particular physical action. Alternatively, for example, the cue signal 122 may represent a cue signal defined at a higher level of abstraction. For example, the cue signal 122 may represent a cue signal to one of the particular electronic actors 124 to move left without specifying a particular physical action for moving left. Upon receiving such a cue signal 122, the electronic actor 124 may select and perform a particular physical action for moving left, such as rotating one or more wheels or moving a robotic leg.

[0042] As described above, the electronic actor coordination module 120 can provide cue signals 122 to the electronic actors 124. For the first line of the script 106, the following processes occur in the electronic actor coordination module 120: Generate a first cue signal corresponding to the first line based on the first line; Identify a first character (e.g., from among the characters 112) associated with the first line; Identify a first electronic actor (e.g., from among the electronic actors 124) associated with the first character; Generate a first cue signal representing the first line.

[0043] The term "first line" and "first" elsewhere in this specification are merely labels and do not imply any temporal, spatial, or other ordering. For example, a "first line" is not necessarily the line that occurs at the beginning of lines 108 of a script 106. In fact, if lines 108 are ordered (in whole or in part), then a "first line" may occur anywhere within that ordering. The term "first line" is simply used to distinguish the first line from other lines such as "second line," "third line," etc. More generally, the same is true of the term "first" as an adjective modifying other terms such as "character," "location," "electronic actor," etc.

[0044] The electronic actor adjustment module 120 may, for example, repeat any of the operations disclosed herein in relation to the first line in relation to any additional lines in the script 106. For example, the electronic actor adjustment module 120 may repeat any of such operations in relation to the second, third, etc. lines within lines 108 in the script. For example, the electronic actor adjustment module 120 may loop through some or all of the lines 108 in the script and perform, in relation to each line, any of the operations disclosed herein in relation to the first line.

[0045] Different cue signals (e.g., corresponding to different lines of the script 106) may be received, interpreted, and processed by different electronic actors 124, which may perform different actions in response. For example, a first cue signal 122 may be received by a first electronic actor 124, which may interpret the first cue signal and perform a first action based on its interpretation. A second cue signal 122 may be received by a second electronic actor 124, which may interpret the second cue signal and perform a second action (which may be different from the first action) based on its interpretation.

[0046] As another example, multiple cue signals may be received by the same electronic actor, which interprets the cue signals and performs different actions based on the multiple cue signals. More specifically, a first one of the cue signals 122 is received by a first one of the electronic actors 124, which interprets the first cue signal and performs a first action based on the interpretation. A second one of the cue signals 122 is received by the first one of the electronic actors 124, which interprets the second cue signal and performs a second action (which may be different from the first action) based on the interpretation.

[0047] As these examples show, the electronic actor 124 receives the cue signal 122, interprets the cue signal 122, selects one or more actions to perform based on the cue signal 122, and executes the selected action. The cue signal 122 may or may not describe the selected action. Furthermore, a particular electronic actor may or may not deterministically select an action to perform. For example, the process of selecting an action to perform may include the use of a pseudorandom number generator, and the electronic actor may select one or another action to perform based on the output of the pseudorandom number generator. As this example shows, the electronic actor may receive the same control signal twice, but each time select and execute an action to perform. The same is true even if a pseudorandom number generator is not involved. For example, the electronic actor's selection of an action to perform may be influenced by inputs in addition to the cue signal 122, such as time, location, user input, audio input, image / video input, specific features of the electronic actor, data received over a network, etc. As a result, the same cue signal 122 may cause the electronic actor to perform different actions based on features of the cue signal 122 and other inputs.

[0048] The electronic actor 124 can interpret the cue signal 122 in a variety of ways. For example, the electronic actor can interpret the cue signal to identify an action to perform based on any combination of one or more of the following: · Electronic actor features. · The internal state of the electronic actor (e.g., the first state 126 and / or the second state 128). · Logic stored in or accessible to the electronic actor (e.g., computer program instructions stored in or accessible to the electronic actor that can be executed by the electronic actor to interpret the cue signal). · Input received from at least one sensor in the environment of the electronic actor (such as one or more cameras, microphones, infrared sensors, pressure sensors, and / or piezoelectric sensors). Such sensors may, for example, be included in or coupled to the electronic actor.

[0049] Moreover, the same cue signal 122 may be provided to two electronic actors 124 (at the same time or at different times) and, in response, the two electronic actors 124 may select and perform different actions. For example, assume that one of the electronic actors 124 has wheels and the other electronic actor 124 has legs. Upon receiving a cue signal 122 representing a cue signal to move left, the first electronic actor 124 may select and perform an action to roll its wheels to move the electronic actor to the left, and the second electronic actor 124 may select and perform an action to walk to the left using its legs.

[0050] When the electronic actor coordination module 120 sends cue signals 122, it may address some or all of those cue signals 122 to a particular electronic actor 124. For example, the electronic actor coordination module 120 may address a first cue signal 122 to a first electronic actor 124 and a second cue signal 122 to a second electronic actor 124.

[0051] Alternatively, for example, one or more of the cue signals 122 may not be addressed to any of the electronic actors 124. For example, when the electronic actor coordination module 120 sends, transmits, or otherwise outputs one of the cue signals 122, it may send, transmit, or otherwise output the cue signal without addressing it to any of the electronic actors 124, such as by broadcasting the cue signal. One or more of the electronic actors 124 may receive such a cue signal, associate the received cue signal with itself, interpret the received cue signal, perform an action based on the interpretation, or perform any of the operations disclosed herein in connection with the received cue signal. A second one of the electronic actors 124 may receive the same cue signal and not associate the received cue signal with itself or interpret the received cue signal.

[0052] As yet another example, the electronic actor coordination module 120 can send one of the cue signals 122 to one of the electronic actors 124. A second of the electronic actors 124 can receive the cue signal, interpret it in any of the manners disclosed herein, and perform an action based on that interpretation in any of the manners disclosed herein. In other words, the electronic actor that receives, interprets, and performs an action based on the cue signal may not be the electronic actor to which the cue signal was sent.

[0053] Many other possibilities are within the scope of the present invention. For example, the electronic actor coordination module 120 may address one of the cue signals 122 to one of the electronic actors 124. A first of the electronic actors may receive the cue signal and take no action based thereon, and may forward the cue signal (e.g., in any of the manners disclosed herein in connection with the electronic actor coordination module 120 sending the cue signal 122) to a second electronic actor, which may receive, interpret, and take action based thereon in any of the manners disclosed herein. Alternatively, a first of the electronic actors may receive the cue signal, interpret it in any of the manners disclosed herein, and take action based thereon in any of the manners disclosed herein, and may forward the cue signal to a second electronic actor, which may receive, interpret, and take action based thereon in any of the manners disclosed herein. As this example suggests, one or more of the electronic actors 124 may receive, interpret, and perform an action based on one of the cue signals 122, where two or more electronic actors may perform the same or different actions from one another based on the same cue signal 122.

[0054] The electronic actor 124 can perform any of a number of actions in response to receiving the cue signal 122. In general, the electronic actor 124 can transition, for example, from a first state 126 to a second state 128 in response to receiving the cue signal 122. Such state transitions can include, for example, any combination of one or more of the following: A change in external state, e.g., performing a physical movement, generating audio output, generating video output (e.g., text, images, and / or video), receiving data and / or cue signals from another device (e.g., another one of the electronic actors 124), and / or sending data and / or control signals to another device (e.g., another one of the electronic actors 124). · Changing internal state, such as generating data, saving data, and / or modifying stored data.

[0055] One example of an audio output that an electronic actor may generate based on a cue signal is an audio output based on the text of one or more lines 108 of the script 106. Such an audio output may be, for example, a voice representing the text, e.g., a pre-recorded or synthesized voice generated based on the text.

[0056] Examples of physical movements that an electronic actor may perform based on a cue signal include one or more of the following, in any combination: - Moving electronic actors in the direction specified by a cue signal. Moving an electronic actor to a target specified by a cue signal, where the target is, for example, another electronic actor or a location within the first electronic actor's environment.

[0057] The electronic actor 124 can provide feedback 130 to the electronic actor coordination module 120 ( FIG. 2 , operation 212). The electronic actor 124 can generate the feedback 130 in a variety of ways, and the feedback 130 can have a variety of content. As an example, the electronic actor 124 can include one or more sensors that sense one or more parameter values, generate a sensor output based on those parameter values, and provide the sensor output (or data derived from the sensor output) as the feedback 130. As another example, a user can provide manual input to the electronic actor 124, and the electronic actor 124 can generate the feedback 130 based on the manual input.

[0058] The electronic actor coordination module 120 may generate the above-mentioned cue signal 122 based on a portion of the script 106, such as the beginning of the script 106 (e.g., the first clause, sentence, or other section of the script 106). If the electronic actor coordination module 120 has not yet reached the end of the script 106 ( FIG. 2 , operation 214), the electronic actor coordination module 120 repeats operations 206 and 208 for the next portion of the script 106 (e.g., the next clause, sentence, or other section of the script 106), and the electronic actor 124 may perform one or more actions based on the subsequent cue signal received from the electronic actor coordination module 120. The generation of the subsequent cue signal by the electronic actor coordination module 120 may be influenced by any of the feedback 130 previously received from the electronic actor 124. This process continues until the electronic actor coordination module 120 determines that the end of the script 106 has been reached in operation 214.

[0059] One or more of the electronic actors 124 may be, include, or be part of any one of the following: a robot, a toy, a digital character, a wearable technology / object with embedded technology, a computer, and any combination thereof. Two or more of the electronic actors 124 may be the same or different from one another in any combination. For example, two or more of the electronic actors 124 may be instances of the same model of device (e.g., the same model of robot). As another example, two or more of the electronic actors may be instances of different models of the same type of device. For example, one of the electronic actors 124 may be an instance of a first model of a robot and a second of the electronic actors 124 may be an instance of a second model of a robot. As another example, two or more of the electronic actors 124 may be instances of different types of devices. Just to give one example of this, one of the electronic actors 124 may be a robot and another of the electronic actors may be a computer (e.g., a desktop computer, a laptop computer, or a smartphone).

[0060] The stories 102 may be in any of a variety of formats, such as screenplays, essays, illustrations (such as comics or paintings that are parsed by the analysis module 104 into a script 106), etc. The stories 102 may include one or more stories of any kind, such as descriptions of historical events, training guides for physical procedures, descriptions of routines, descriptions of locations for use in walking tours, descriptions of presentations using stories, recipes, and physical props for use in escape games or ad-hoc interactive environments.

[0061] As described above, the process of generating a script 106 based on a story 102 may include, for example, parsing the story 102 into one or more characters 112, one or more lines 108, and one or more locations 116. Each line 108 may be or include, for example, dialogue, movement, sound effects, and lighting cues. As this illustrates, the resulting lines 108 may include multiple lines 110a-i of different types, such as one line and one line showing a movement. Each character 112 may be associated with a respective electronic actor 124. For each character 112 and corresponding line, the electronic actor coordination module 120 may provide a cue signal 122 representing the line to the electronic actor 124 associated with that character.

[0062] If the line 108 is dialogue, the electronic actor coordination module 120 can use a text-to-speech engine to convert the text in the script 106 that represents the dialogue into speech. The cue signals 122 are signals that control the electronic actors 124 to generate audio outputs that represent the dialogue.

[0063] The story 102 and / or the script 106 may be written in a manner that is partially or completely independent of the electronic actors 124. For example, the script 106 and / or the story 102 may describe a state change in general terms that do not specify a particular action to be performed. The electronic actor coordination module 120 may generate cue signals 122 that cause the electronic actors 124 to perform one or more actions not explicitly described by the state change description in the script 106, based on that state change description. As a particular example, the story 102 and / or the script 106 may include text describing an emotion, but not explicitly describing a particular action to convey that emotion. The electronic actor coordination module 120 may generate cue signals 122 that cause the electronic actors 124 to perform one or more actions that convey the described emotion, based on the emotion description text.

[0064] Embodiments of the present invention have various advantages, including allowing the user to focus on the desired experience, rather than the mechanics of how it works. This allows for a more expressive and intuitive organization. Another advantage of embodiments of the present invention is that the story 102 can be interpreted multiple times in different ways. For example, the electronic actor coordination module 120 can interpret the script 106 a first time to generate a first set of cue signals 122 for controlling a first set of electronic actors 124 to perform a first set of actions, and then interpret the script 106 a second time to generate a second set of cue signals 122 for controlling a second set of electronic actors to perform a second set of actions. This can be useful, for example, when the first set of electronic actors and the second set of electronic actors have different capabilities.

[0065] The various components of the system 100 may be implemented in any of a variety of devices and may communicate with each other in any of a variety of ways. For example, the analysis module 104 and the electronic actor adjustment module 120 may be implemented in whole or in part in a computing device such as a desktop computer, a laptop computer, a smartphone, or a tablet computer. Each of the electronic actors 124 may be implemented in whole or in part as a computing device or other electronic device such as a toy or robot, for example. The analysis module 104 and / or the electronic actor adjustment module 120 may be implemented in one or more devices that are physically distinct from the electronic actors 124, such as when the electronic actor adjustment module 120 is implemented in a computing device and the electronic actors 124 are implemented in multiple toys or robots that are physically distinct from the computing devices. The electronic actor adjustment module 120 may communicate with the electronic actors 124 by wired and / or wireless communication (e.g., Bluetooth).

[0066] The electronic actor coordination module 120 can be distributed across multiple computing devices. For example, the analysis module 104 can perform the script analysis functions disclosed herein on a local computing device and offload the speech recognition functions disclosed herein to a remote speech recognition server with which the local computing device communicates over the Internet.

[0067] Referring to FIG. 3, a diagram of a system 300 for loading a story (i.e., story 102) according to one embodiment of the present invention is shown. A story includes multiple lines, and each line may include any combination of one or more of the following: plain text (e.g., one or more words, clauses, and / or sentences), structured text (e.g., text marked up with XML / HTML tags), hyperlinks, audio content, image content, and video content. A story execution engine (which may be implemented as part of the analysis module 104 and / or electronic actor coordination module 120 of FIG. 1) processes the story to create characters and lines. For example, the story execution engine may identify multiple characters identified in the story, such as by identifying the name of the character at the beginning of a line in the screenplay. The story execution engine can create lines by identifying dialogue, stage directions, audio content, image content, video content, hyperlinks, and other content in the story.

[0068] The story execution engine may create an association between each character and a corresponding line based on the story. For example, if the story contains text in the format "<character name>:<dialogue>", the story execution engine may create a character named <character name>, create a line containing the text <dialogue>, and create an association between the character named <character name> and the line containing the text <dialogue>.

[0069] The story execution engine can assign each character to a corresponding electronic actor 124. These assignments are used by the system 300 to provide cue signals 122 to the electronic actors 124. For those electronic actors that have been assigned a corresponding character, the story execution engine can provide information about that character (such as a name) to the electronic device, and the electronic device can receive and store the character information.

[0070] The story execution engine may identify lines that require the transmission of large amounts of data, such as audio files, video files, etc. For such lines, the story execution engine may transmit the corresponding data to the electronic actors, who may receive and store the data.

[0071] 4, a diagram of a system 400 for playing (executing) a script 106 according to one embodiment of the present invention is shown. System 400 may be executed after system 300 of FIG. 3 has been executed.

[0072] The story execution engine reads the previously created lines in order and for each line it can do one or more of the following: · Create queues based on rows. Identify the character associated with a line. ·Identify the electronic actor associated with that character. Create cue signals based on one or more of lines, characters, and electronic actors. Sending a cue signal to the identified electronic actor. Decide whether to wait for feedback from the electronic actor. If you wait for feedback, the story execution engine waits for feedback from the electronic actor. Otherwise, the story execution engine proceeds to the next line and repeats the above steps.

[0073] When an electronic actor receives a cue signal for a particular row, the electronic actor interprets the cue and performs a corresponding action in any of the manners disclosed herein. When appropriate, the electronic actor sends feedback to the electronic actor coordination module 120 in the manner described above in connection with Figures 1 and 2. The references to Bluetooth Low Energy in Figures 3 and 4 are merely examples and do not constitute limitations of the invention.

[0074] It has been noted above that the story execution engine can process the script 106 line by line. Figure 5 illustrates a system 500 for performing such processing in accordance with one embodiment of the present invention. The story execution engine extracts actors and locations from the script 106. For example, the story execution engine may use NLP to extract all entities from the text of the script 106 and then filter the extracted entities (e.g., by frequency, salience, known nouns, and / or proper nouns) to include only important entities. The result is a list of actors and locations.

[0075] The story execution engine pre-processes the sentences in the script 106. For example, the story execution engine replaces pronouns in the script 106 with common / proper nouns to clarify the speakers of dialogue in the script 106. The result is a modified version of the script 106 with the sentences pre-processed. The remainder of the system of Figure 5 may operate on the modified script and the extracted actors and locations.

[0076] The story execution engine generates the line and cue signals. For example, the story execution engine may read each sentence in the script 106 in order, (1) match the sentence against a dialogue template, or (2) use NLP to obtain a syntax tree for the sentence, extract verb / noun / object pairs from the sentence, normalize the verbs using NLP stemming, and filter the pairs to leave only known verbs and action verbs. The story execution engine then generates the line and / or cue signals.

[0077] In some embodiments, the techniques described herein relate to a method performed by at least one computer processor executing computer program instructions stored on at least one non-transitory computer-readable medium, the method including: (A) for a first row of a plurality of rows, generating a first cue signal corresponding to the first row based on the first row; (B) identifying a first character associated with the first row; and (C) identifying a first electronic actor associated with the first character.

[0078] The method may further include, prior to (A), (D) receiving a story, the story including at least one of text data, audio data, image data, and video data; and (E) generating a script based on the story, the script representing a plurality of lines and a plurality of characters, where at least a portion of the plurality of lines is associated with at least a portion of the plurality of characters.

[0079] A story may include any combination of one or more of audio data, image data, video data, and text data (e.g., dialogue). Text data may include, for example, text written in a natural language.

[0080] The first electronic actor may, for example, be, include, or be contained within a robot.

[0081] The method may further include (D) transmitting the first cue signal to the first electronic actor. The method may further include (E) interpreting, at the first electronic actor, the first cue signal to identify a first action to perform, and (F) performing, at the first electronic actor. Operation (E) may include interpreting the first cue signal to identify the first action to perform based on any one or more of at least one function of the first electronic actor, an internal state of the first electronic actor, logic stored in the first electronic actor, and input received from at least one sensor in the first electronic actor's environment, in any combination.

[0082] Performing the first action may include moving the first electronic actor based on the first cue signal. Moving the first electronic actor may include, for example, moving the first electronic actor in a direction specified in the first row and / or moving the first electronic actor to a target specified in the first row, in any combination. The target may include, for example, a location in the environment of the first electronic actor. The target may include, for example, a second electronic actor.

[0083] The first line may include, for example, text, and interpreting the first cue signal may include, for example, generating, at the first electronic actor, audio data based on the text of the first line.

[0084] The method may be performed, for example, by a computing device physically distinct from the first electronic actor, the method further comprising: (G) providing a feedback signal to a computing device at the first electronic actor. The method may further comprise, prior to (G), (H) obtaining sensor data via a sensor at the first electronic actor, where (G) includes providing the sensor data to the computing device. The method may further comprise, prior to (G), obtaining input from a user via an input device at the first electronic actor, where (G) includes providing the input to the computing device.

[0085] The method may further include (E) for a second row of the plurality of rows, generating a second cue signal corresponding to the second row based on the second row, (F) identifying a second character associated with the second row, (G) identifying a second electronic actor associated with the second character, and (H) transmitting the second cue signal to the second electronic actor.

[0086] The method is performed by a computing device that is physically distinct from the first electronic actor and the second electronic actor, and may further include: (I) interpreting, at the first electronic actor, a first cue signal to identify a first action to perform, (J) performing, at the first electronic actor, the first action, (K) interpreting, at the second electronic actor, a second cue signal to identify a second action to perform, and (L) performing, at the second electronic actor, a second action, where the first electronic actor is physically distinct from the second electronic actor, and the first action is distinct from the second action.

[0087] The method may further include (D) transmitting the first cue signal, (E) receiving the first cue signal at the first electronic actor, (F) associating the first cue signal with the first electronic actor, and (G) interpreting the first cue signal at the first electronic actor to identify a first action to perform, and (H) performing the first action at the first electronic actor. The method may further include (I) receiving the first cue signal at the second electronic actor, and (J) not associating the first cue signal with the second electronic actor.

[0088] The method may further include (D) transmitting a first cue signal, where the first cue signal is directed to the first electronic actor; (E) interpreting, at the second electronic actor, the first cue signal to identify the first actor to perform; and (F) performing, at the second electronic actor, the first action.

[0089] The method may further include (D) for a second row of the plurality of rows, generating a second cue signal corresponding to the second row based on the second row, (E) identifying a second character associated with the second row, and (F) providing a second cue signal representative of the second row to the first electronic actor.

[0090] The first cue signal corresponding to the first row may include text, the text may include text written in a natural language and representing a conversation.

[0091] The first cue signal may include an audio cue. The first cue signal may specify a sound. The first cue signal may include a motion cue.

[0092] The first electronic actor is capable of executing (A).

[0093] The method may further include (D) transmitting, at the first electronic actor, the first cue signal to the second electronic actor.

[0094] In some embodiments, the technology described herein relates to a system including a script including a plurality of lines, an electronic actor coordination module, and a first electronic actor, the electronic actor coordination module including computer program instructions for performing a method including: (A) for a first line of the plurality of lines in the script, generating a first cue signal corresponding to the first line based on the first line, (B) identifying a first character associated with the first line, (C) identifying a first electronic actor associated with the first character, and (D) generating a first cue signal representative of the first line.

[0095] Although the present invention has been described above with respect to certain embodiments, those skilled in the art will appreciate that the foregoing embodiments are provided for illustrative purposes only and are not intended to limit or define the scope of the present invention. Various other embodiments are also within the scope of the claims, including but not limited to the following. For example, the elements and components described herein may be further divided into additional components or combined to form fewer components for performing the same functions.

[0096] Any of the functions disclosed herein may be implemented using means for performing those functions, including, but not limited to, any of the components disclosed herein, such as computer-related components described below.

[0097] The techniques described above may be implemented, for example, in hardware, one or more computer programs tangibly stored on one or more computer-readable media, firmware, or any combination thereof. The techniques described above may be implemented in one or more computer programs running on (or programmable by) a programmable computer including a processor, a storage medium (including, for example, volatile and non-volatile memory and / or storage elements) readable and / or writable by the processor, any number and combination of input devices, and output devices. The program code may be applied to input entered using the input devices to perform the described functions and to generate output using the output devices.

[0098] Embodiments of the present invention include functionality that is implementable and / or executable only using one or more computers, computer processors, and / or other elements of a computer system. Such functionality is impossible or impractical to implement mentally and / or manually. For example, embodiments of the present invention automatically parse a high-level description of a story to automatically identify instructions to be provided to one or more machines (e.g., robots), and then automatically interpret those instructions to automatically perform one or more actions. Such functionality is inherently rooted in computer technology and cannot be performed mentally or manually.

[0099] Inventions herein that positively require a computer, processor, memory, or similar computer-related elements are intended to require such elements and should not be construed as such elements not being present in or not required by such invention. Such inventions are not intended to cover methods and / or systems lacking the recited computer-related elements, and should not be construed as such. For example, method claims herein that recite a claimed method being performed by a computer, processor, memory, and / or similar computer-related elements are intended to encompass, and should only be construed as such, methods performed by the recited computer-related elements. Such method inventions should not be construed as encompassing, for example, methods that are performed mentally or manually (e.g., using pencil and paper). Similarly, product inventions herein that recite a claimed product that includes a computer, processor, memory, and / or similar computer-related elements are intended to encompass, and should only be construed as such, products that include the recited computer-related elements. Such product inventions should not be construed as encompassing, for example, products that do not include the recited computer-related elements.

[0100] Each computer program within the scope of the present invention below may be implemented in any programming language, such as assembly language, machine language, a high-level procedural programming language, or an object-oriented programming language, for example, a compiled or interpreted programming language.

[0101] Each such computer program may be implemented in a computer program product tangibly embodied in a machine-readable storage device for execution by a computer processor. The method steps of the present invention may be executed by one or more computer processors executing a program tangibly embodied in a computer-readable medium to perform the functions of the present invention by manipulating inputs to generate outputs. Suitable processors include, by way of example, both general purpose and special purpose microprocessors. In general, a processor receives (reads) instructions and data from a memory (such as a read-only memory and / or a random access memory) and writes (stores) instructions and data to a memory. Suitable storage devices for tangibly embodying computer program instructions and data include all forms of non-volatile memory, such as, for example, semiconductor memory devices such as EPROMs, EEPROMs, flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROMs. Any of the above may be supplemented by, or incorporated in, specially designed ASICs (Application Specific Integrated Circuits) or FPGAs (Field Programmable Gate Arrays). A computer can typically receive (read) and write (store) programs and data from a non-transitory computer-readable storage medium, such as an internal disk (not shown) or a removable disk. These elements may also be found in conventional desktop or workstation computers, and other computers suitable for running computer programs implementing the methods described herein, including in combination with any digital printing or marking engine, display monitor, or other raster output device capable of producing color or grayscale pixels on paper, film, a display screen, or other output medium.

[0102] Data disclosed herein may be embodied in one or more data structures tangibly stored on a non-transitory computer-readable medium, for example, and embodiments of the present invention may store such data in and read such data from such data structures.

[0103] Any step or act disclosed herein as being performed or capable of being performed by a computer or other machine may be automatically performed by a computer or other machine, whether or not expressly disclosed herein. An automatically performed step or act is performed solely by a computer or other machine, without human intervention. An automatically performed step or act may, for example, operate solely based on input received from a computer or other machine, and not from a human. An automatically performed step or act may, for example, be initiated by a signal received from a computer or other machine, and not from a human. An automatically performed step or act may, for example, provide output to a computer or other machine, and not to a human.

[0104] The terms "A or B," "at least one of A or / and B," "at least one of A and B," "at least one of A or B," or "one or more of A or / and B," as used in various embodiments of the present disclosure, include any and all combinations of the words listed therein. For example, "A or B," "at least one of A and B," or "at least one of A or B" means (1) including at least one A, (2) including at least one B, (3) including either A or B, or (4) including both at least one A and at least one B.

Claims

1. 1. A method performed by at least one computer processor executing computer program instructions stored on at least one non-transitory computer readable medium, comprising: The method is characterized by comprising the following steps (A) to (C). (A) for a first row of a plurality of rows, generating a first cue signal corresponding to the first row based on the first row; (B) identifying a first character associated with the first line; and (C) identifying a first electronic actor associated with the first character;

2. In claim 1, The method further comprises the following steps before the step (A): (D) receiving a story, the story including at least one of text data, audio data, image data, and video data; (E) generating a script representing a plurality of lines and a plurality of characters based on the story; At least a portion of the plurality of lines are associated with at least a portion of the plurality of characters.

3. In claim 2, The method, wherein the story includes audio data.

4. In claim 2, The method, wherein the story includes video data.

5. In claim 1, The method of claim 1, wherein the first electronic actor comprises a robot.

6. In claim 1, (D) transmitting the first cue signal to the first electronic actor.

7. In claim 6, (E) interpreting, at the first electronic actor, the first cue signal to identify a first action to perform; (F) performing, with the first electronic actor, the first action.

8. In claim 7, The method of claim 1, wherein step (E) includes interpreting the first cue signal to identify the first action to perform based on at least one function of the first electronic actor.

9. In claim 7, and (E) interpreting the first cue signal based on input received from at least one sensor in an environment of the first electronic actor to identify the first action to perform.

10. In claim 7, 11. The method of claim 10, wherein performing the first action includes moving the first electronic actor based on the first cue signal.

11. In claim 7, The method is performed by a computing device that is physically distinct from the first electronic actor; The method further comprising the step of: (G) providing, at the first electronic actor, a feedback signal to the computing device.

12. In claim 11, The method further includes, prior to step (G), a step of (H) obtaining, at the first electronic actor, an input from a user via an input device; The method, wherein step (G) includes providing an input to the computing device.

13. In claim 6, (E) for a second row of the plurality of rows, generating a second cue signal corresponding to the second row based on the second row; (F) identifying a second character associated with the second line; (G) identifying a second electronic actor associated with the second character; and (H) transmitting said second cue signal to said second electronic actor. The method further comprising:

14. In claim 13, the method being performed by a computing device that is physically distinct from the first electronic actor and the second electronic actor; The method comprises: (I) interpreting, at the first electronic actor, the first cue signal to identify a first action to perform; (J) executing, with the first electronic actor, the first action; (K) interpreting, at the second electronic actor, the second cue signal to identify a second action to perform; and (L) performing the second action with the second electronic actor; the first electronic actor is physically distinct from the second electronic actor; The method, wherein the first action is different from the second action.

15. In claim 1, (d) transmitting the first cue signal; (E) receiving, at the first electronic actor, the first cue signal; (F) associating, at the first electronic actor, the first cue signal with the first electronic actor; (G) interpreting, at the first electronic actor, the first cue signal to identify a first action to perform; and (H) executing the first action with the first electronic actor. The method further comprising:

16. In claim 1, (D) transmitting the first cue signal, the first cue signal being directed to the first electronic actor; (E) interpreting, at the second electronic actor, the first cue signal to identify a first action to perform; and (F) executing the first action with the second electronic actor. The method further comprising:

17. In claim 1, The method of claim 1, wherein the first electronic actor performs step (A).

18. In claim 17, (D) at the first electronic actor, transmitting the first cue signal to a second electronic actor.

19. A script containing multiple lines, an electronic actor coordination module; a first electronic actor; A system including: The system, wherein the electronic actor coordination module includes computer program instructions for executing a method including the steps of: (A) detecting a plurality of electronic actors; (A) for a first line of the plurality of lines in the script, generating a first cue signal corresponding to the first line based on the first line; (B) identifying a first character associated with the first line; (C) identifying a first electronic actor associated with the first character; and (D) generating the first cue signal representative of the first row.

Images