Narrative Text and Narrative Voice Game User Interfaces

Abstract

A narrative engine receives object metadata describing the content of a user interface from a user application providing the user interface via an input / output device of a computing device. An extended description of the user interface is generated, the extended description including a description of the surroundings of the user interface and a list of actions to be performed on the user interface. The extended description is presented using an output device. User input requesting one of the actions is processed. The extended description is updated based on the user input.

Description

[Technical field]

[0001] Cross-reference to related applications This application claims the benefit of U.S. Provisional Application No. 63 / 265,697, filed December 19, 2021, the disclosure of which is incorporated herein by reference in its entirety. [Background technology]

[0002] Aspects of the present disclosure relate to user interfaces that interpret displayed or stored computer data as narrative prose. Further aspects relate to computer input collected through the interface from text or voice-to-text input. Further aspects relate to computer interfaces that are accessible to players who are physically disabled or blind. Summary of the Invention

[0003] In one or more example embodiments, the system includes a computing device with an input device and an output device, the computing device being programmed to execute a narrative engine, receiving object metadata describing content of the user interface via the input device and the output device from a user application providing the user interface, the augmented description including a description of a situation within the user interface and a list of actions to be performed on the user interface, generating an augmented description of the user interface, presenting the augmented description using the output device, processing user input requesting one of the actions, and updating the augmented description (or explanation) based on the user input.

[0004] In one or more exemplary embodiments, the method includes receiving, via input and output devices of a computing device, object metadata describing content of the user interface from a user application providing the user interface; generating an extended description of the user interface, the extended description including a description of an environment of the user interface and a list of actions to be performed on the user interface; presenting the extended description using the output device; processing user input requesting one of the actions; and updating the extended description based on the user input.

[0005] In one or more example embodiments, the non-transitory computer-readable medium includes narrative engine instructions that, when executed by one or more processors of the computing device, cause the computing device to perform operations including receiving, on the computing device, object metadata describing content of the user interface from a user application providing a user interface via input and output devices of the computing device; receiving the object metadata from the user application using an application programming interface (API) of the narrative engine, the API including extensions for each type of user interface supported to enable access to the object metadata for the particular user interface type; filtering the object metadata using properties of the object metadata, determining associated objects in the object metadata, and generating an extended description of the user interface using the associated objects, the extended description including a description of an environment of the user interface and a list of actions to be performed on the user interface; presenting, using the input and output devices, the extended description one or more of as an overlay interposed on the user interface or as computer-generated audio; processing user input requesting one of the actions; updating the extended description based on the user input; and presenting the updated extended description using the output devices. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 illustrates an example of a system that includes a computing device that implements a narrative interface for operating a user application. [Diagram 2] FIG. 2 shows further details of an example implementation of a narrative interface. [Diagram 3] FIG. 3 illustrates an example of the use of a narrative engine in the user interface of a 2D game. [Figure 4]FIG. 4 illustrates an example of the use of a narrative engine in the user interface of a 2D application. [Diagram 5] FIG. 5 illustrates an example of the use of a narrative engine in the user interface of a 3D game. [Figure 6] FIG. 6 illustrates an example of the use of a narrative engine in the user interface of a store application. [Figure 7] FIG. 7 is a diagram illustrating an example of object metadata for a wallet item shown in the store user interface of FIG. [Figure 8] FIG. 8 is an example process illustrating the main interface loop for the operation of the narrative engine. [Figure 9] FIG. 9 illustrates an example process for a narrative engine responding to user input. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely exemplary, and that other embodiments may take various alternative forms. The figures are not necessarily to scale, and some features may be exaggerated or minimized to show the details of a particular component. Therefore, specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to employ the embodiments in various ways. As will be understood by those skilled in the art, various features illustrated and described with reference to any one of the figures can be combined with features illustrated in one or more other figures to create an embodiment not explicitly illustrated or described. The combination of the illustrated features provides a representative embodiment for a typical application. However, various combinations and modifications of features consistent with the teachings of the present disclosure may be desired for a particular application.

[0008] Aspects of the disclosure relate to an approach for interpreting and conveying computer user interface information as narrative expository prose that is displayed and spoken aloud by a text-to-speech engine. As an example, a video game player can control or trigger events within a computer game through natural voice or text input. A blind player can use an interface with voice output and text or voice input. A deaf player can use an interface with text and / or graphic output and text or voice input. The speech-to-text and text-to-speech utilized can be found in modern smartphones and personal computers.

[0009] As one example, narrative interfaces are effective when used in turn-based computer games. In another example, narrative interfaces are effective when used in 2D applications, such as word processors or websites. In yet another example, narrative interfaces are effective when used with 3D applications, such as the metaverse or 3D video games.

[0010] 1 illustrates an exemplary system 100 including a computing device 102 for implementing a narrative engine 122 for operation of a user application 118. The computing device 102 may be various types of devices, such as a smartphone, tablet, desktop computer, smart watch, video game console, smart television (TV), virtual reality (VR) headset, augmented reality (AR) glasses, etc. Regardless of form, the computing device 102 includes a processor 104 operably connected to storage 106, a network device 108, an output device 114, and an input device 116. It should be noted that this is merely exemplary and that a computing device 102 having more, fewer, or different components may be used.

[0011] The processor 104 may include one or more integrated circuits implementing the functionality of a central processing unit (CPU) and / or a graphics processing unit (GPU). In some examples, the processor 104 is a system-on-chip (SoC) that integrates the functionality of a CPU and a GPU. The SoC may optionally include other components, such as, for example, storage 106 and a network device 108, in a single integrated device. In other examples, the CPU and GPU are connected to each other via a peripheral connection device, such as a Peripheral Component Interconnect (PCI) Express or other suitable peripheral data connection. In one embodiment, the CPU is a commercially available central processing unit that implements an instruction set of, for example, one of the following instruction set families: x86, ARM, Power, or microprocessor without interlocked pipeline stage (MIPS). It should be noted that while only one processor 104 is shown, in many discussions, the computing device 102 may include multiple processors 104 with various interconnected functions.

[0012] Storage 106 may include both non-volatile and volatile memory devices. Non-volatile memory includes solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the system is deactivated or loses power. Volatile memory includes static and dynamic random access memory (RAM), which stores program instructions and data during operation of system 100.

[0013] Each of the network devices 108 may include any of a variety of devices that enable the computing device 102 to transmit and / or receive data from external devices. Examples of suitable network devices 108 include an Ethernet interface, a Wi-Fi transceiver, a cellular transceiver, a BLUETOOTH or BLUETOOTH Low Energy (BLE) transceiver, or other network adapter or peripheral interconnect device that receives data from other computers or external data storage devices.

[0014] In one example, the network device 108 may enable the computing device 102 to access one or more remote servers 110 or other devices via a communication network 112. The communication network 112 may include one or more interconnected communication networks, such as the Internet, a cable television distribution network, a satellite link network, a local area network, and a telephone network, as some non-limiting examples. The remote server 110 may include devices configured to provide various cloud services to the computing device 102, such as speech-to-text conversion, database access, downloading of applications and / or data files, Internet searching, etc.

[0015] Output device 114 may include a graphical or visual display device, such as an electronic display screen, a projector, a printer, or other suitable device for reproducing a graphical display. As another example, output device 114 may include an audio device, such as a loudspeaker or headphones. As yet another example, output device 114 may include a tactile device, such as a Braille keyboard, or a mechanical device that a user can touch and approach, configured to display Braille or other physical output. In the case of a system that includes a GPU, GPU processor 104 may include hardware and software for displaying at least two-dimensional (2D), and optionally three-dimensional (3D), graphics on output device 114.

[0016] The input devices 116 may include any of a variety of devices that enable the computing device 102 to receive control input from a user. Examples of suitable input devices 116 for receiving human interface input may include a keyboard, a mouse, a trackball, a touch screen, a microphone, a headset, a graphics tablet, and the like.

[0017] During operation, the processor 104 executes stored program instructions retrieved from the storage 106. The stored program instructions thus comprise software for controlling the operation of the processor 104 and for performing the operations described herein. This software may include, for example, one or more user applications 118 and a narrative engine 122.

[0018] User applications 118 may include various types of software applications executable by processor 104 having a defined user interface 120. As some examples, user applications 118 may be video games, websites, stores, productivity applications, metaverse components, etc.

[0019] User interface 120 refers to the manner in which a user interacts with system 100 through the use of input devices 116 and output devices 114. In some examples, user application 118 may define a 2D interface, such as a website or a word processor. In other examples, user application 118 may define a 3D interface, such as a first-person video game or a metaverse application. In yet another example, user application 118 may define a text interface, such as a command line application or a text adventure. Additionally, in some examples, user interface 120 may be presented via output device 114 in a bi-modal manner, such as a 2D display screen. In other examples, user interface 120 may be presented via output device 114 in a 3D manner, such as using a VR or AR headset. In yet another example, user interface 120 may be presented via output device 114 using an audio interface.

[0020] The narrative engine 122 is configured to allow binding software actions or sequences of actions in the user interface 120 to be used in natural language through the API 124 , thereby allowing the user an increased level of control over the user application 118 .

[0021] 2 illustrates further aspects of the narrative engine 122. As shown, the narrative engine 122 can receive object metadata 202 from the user interface 120 via the API 124. The narrative engine 122 can use an attention filter 204 to filter the object metadata 202 into a set of related objects 206 that are relevant to the user. The related objects 206 are provided to an object interpreter 208, which generates an interface model 210. The interface model 210 can describe properties 212 and available actions 214 of the related objects 206. A description creator 216 uses the interface model 210, text templates 217, and user settings 220 to generate and provide an augmented description 218 to the user interface 120 via the API 124. This may include, for example, providing the augmented description 218 to the user interface 120 as text using an overlay generator 222 and / or providing the augmented description 218 audibly to the user interface 120 using a text-to-speech engine 224. Additionally, the narrative engine 122 may be configured to receive user input 226 from the user interface 120 via the API 124. This user input 226 may be provided to a command executor 228 and processed by the user application 118. The user input 226 may also be provided to a speech-to-text engine 230, which may use a command recognizer 234 to identify actions within the interface model 210 and provide them to the command executor 228 for processing (e.g., via the API 124 or otherwise).

[0022] Although an exemplary modularization of the narrative engine 122 is described herein, it should be noted that the components of the narrative engine 122 may be incorporated into fewer components, combined into fewer components, or combined into a single component. For example, while each of the object interpreter 208, description generator 216, overlay generator 222, text-to-speech engine 224, command executor 228, speech-to-text engine 230, command recognizer 234, and command executor 228 are described separately, these components may be implemented separately or in combination with one or more controllers, in hardware, and / or a combination of software and hardware.

[0023] Object metadata 202 may refer to published or otherwise available information that defines aspects of interface elements in user interface 120. For 2D interfaces, these interface elements may refer to 2D elements, such as windows, dialog boxes, buttons, sliders, text boxes, web page links, etc. For 3D interfaces, these interface elements may refer to 3D mesh objects in a 3D scene, such as models of trees, houses, avatars, vehicles, etc. For text-based interfaces, interface elements may refer to blocks of text, such as user prompts or other text-based information, such as responses to help commands used to display available text commands.

[0024] The API 124 may include computer code used to enable the narrative engine 122 to receive object metadata 202 from the user interface 120. As an example, in the case of a 3D scene, such as one rendered in Unity or other 3D engine, each object rendered may have object metadata 202. This object metadata 202 may be accessed by the narrative engine 122 via the API 124. In another example, in the case of a 2D web page, the HyperText Transfer Protocol (HTTP) markup of the web page may include or otherwise define object metadata 202 that may be read by the narrative engine 122 via the API 124. In yet another example, in the case of a Windows application, the window position, text, and other attributes may be captured by the API 124 via enumeration of windows on the desktop and / or via other operating system (OS) level interface functions. In yet another example, in the case of a console application, the console buffer text may be read by the narrative engine 122 via the API 124. In some examples, the API 124 may require a shim or extension to be created for each new user interface 120 type that is supported, allowing the narrative engine 122 to access the object metadata 202 for that particular user interface 120 type. For example, if rendered Java applications are supported, a shim or extension may be added to the API 124 so that rendered Java control information is exposed to the narrative engine 122.

[0025] The attention filter 204 may be configured to filter the object metadata 202 to relevant objects 206. For example, the attention filter 204 may simply allow processing of all object metadata 202. However, this may not be practical in complex interfaces or crowded 3D scenes. Additionally, it may be desirable to limit the range of interface elements considered based on criteria related to the user's attention, such as, for example, the user's location in the 3D scene, the location of the mouse pointer in a 2D interface, the current task being performed by the user, etc. In one example, the attention filter 204 may filter the object metadata 202 based on properties of the object metadata 202. Continuing with the 3D location example, the attention filter 204 may limit the object metadata 202 to objects that are within a predefined distance from the user or the user's avatar and / or objects that are within the user's field of view. In a 2D example, the attention filter 204 may limit the object metadata 202 to controls that are within a predefined 2D distance from the mouse cursor and / or interface elements that are enabled.

[0026] The object interpreter 208 is configured to receive the related objects 206 and compile an interface model 210 based on the received related objects 206. As an example, the object interpreter 208 may generate the interface model 210 as including the properties 212 and available actions 214 of the related objects 206 filtered by the attention filter 204. In doing so, the object interpreter 208 may create a set of information that can be used to both augment the content in the user interface 120 and to improve the user's command selection.

[0027] In an example 2D interface, the object metadata 202 may include property 212 information such as control properties 212 (e.g., name, owner, screen location, text, button identifier (ID), link reference ID, etc.). The object metadata 202 may also include available actions 214 such as pressing or activating a button, scrolling to a location, receiving text, deleting text, etc. In an example 3D interface, the object metadata 202 may include property 212 information (e.g., mesh name, creator ID, model ID, color, shading, texture, size, position, etc.). The available actions 214 may include aspects such as moving an object, opening a door, starting a car, adjusting the car's speed or direction, etc. In an example text interface, the object metadata 202 may include property 212 information such as prompt text. The available actions 214 may include text commands exposed by a command line. For example, issuing a help command may display available text commands.

[0028] The description generator 216 may be configured to generate an augmented description 218 of the interface model 210 to enhance the user interface 120. In one example, the description generator 216 may generate a natural language description that describes the properties 212 of the related objects 206. In another example, the description generator 216 may generate a natural language description that describes the available actions 214 of the related objects 206.

[0029] In one example, the description generator 216 may provide natural language descriptions based on the metadata of the related objects 206 using text templates 217. Each template 217 may include natural language text with one or more placeholders for values ​​of the properties 212 or available actions 214 of the related objects 206 being described. The template 217 may be applied to the related object 206 or set of related objects 206 if the value placeholders are specified by the metadata of the related objects 206. As shown in the examples herein, the names of the properties 212 and available actions 214 are specified within square brackets in the template 217, however, this is merely an example and other approaches for parameterizing text may be used (e.g., using AI techniques to generate natural language text from prompt information, etc.).

[0030] In one example, to generate information describing the environment, the description generator 216 can utilize templates 217 such as "You are using [application name]," or "You are near [object name]," or "You are facing [direction]," or "You have [object name] nearby that is [attribute]," etc. For example, if one of the related objects 206 in the interface model 210 has an application name property 212 specified, the template 217 "You are using [application name]" may be used.

[0031] In another example, to generate the list of available actions 214, the description generator 216 can utilize a template 217 such as, for example, “From here you can [list of available actions 214 formatted into a comma-separated list]”, where each of the available actions 214 may be listed based on metadata such as command name, tooltip text, attribute names, etc. Aspects of the creation of the extended description 218 may also be based on user settings 220. For example, the user settings 220 may indicate a level of verbosity for the generation of the extended description 218 (e.g., use a template 217 that is a complete sentence versus a concise list of attributes).

[0032] The overlay generator 222 may be configured to visually present the augmented description 218 to a user via the output device(s) 114 of the user interface 120. In one example, the overlay generator 222 may present the augmented description 218 as textual information (e.g., in high contrast colors and / or fonts) on top of an existing display.

[0033] The text-to-speech engine 224 may be configured to audibly provide the augmented description 218 to a user via the output device 114 of the user interface 120. As an example, the text-to-speech engine 224 may convert normal language text into speech using any of a variety of speech synthesis techniques, which can be played via speakers, headphones, or other audio output device 114.

[0034] In some examples, the user settings 220 may further indicate how the description generator 216 should provide the augmented description 218 to the user. These user settings 220 may be based on the level or type of disability of the user. For example, if the user is visually impaired, the user settings 220 may indicate that the augmented description 218 is spoken to the user via the text-to-speech engine 224. Alternatively, if the user is hearing impaired, the user settings 220 may indicate that the augmented description 218 is displayed to the user via the overlay generator 222. It should be noted that these settings may be used in situations other than when the user is disabled. For example, these settings may be used to enable use of the application in a loud room by using the overlay generator 222 to explain information that may be inaudible due to the noise level.

[0035] The command executor 228 may be configured to cause the narrative engine 122 to execute an available action 214 requested by a user. The command executor 228 may receive user input 226 from one or more input devices 116 of the user interface 120. In some examples, the user input 226 may include an action that the user application 118 may understand without processing by the narrative engine 122. For example, the user input 226 may include pressing a control mapped to one of the available actions 214. In such examples, the command executor 228 of the narrative engine 122 may simply pass the user input 226 to the user application 118 for processing.

[0036] In other examples, the user input 226 may be instructions to execute a command indicated by the augmented description 218, but in a manner that the user application 118 may not be able to process. For example, the augmented description 218 may indicate that the user is to vocalize a particular command to be executed; however, the user application 118 may lack speech support. Thus, the user input 226 may be further provided to a speech-to-text engine 230 of the narrative engine 122, which may process the user input 226 into a textual representation, referred to herein as recognized text 232.

[0037] The command recognizer 234 may receive the recognized text 232 and process the recognized text 232 to identify which, if any, of the available actions 214 to execute. For example, the command recognizer 234 may scan the recognized text 232 for action words (e.g., names of available actions 214 in the interface model 210). In another example, the command recognizer 234 may scan for predefined verbs such as "help" or other actions. If such available actions 214 are found, the command recognizer 234 may instruct the command executor 228 to execute the spoken available action 214.

[0038] 3 illustrates an example of the use of the narrative engine 122 for a 2D game user interface 120. The example shows a dynamically created text block (also called a dynamic text block) 302, which contains an extended description 218, and is displayed in the user interface 120 along with the 2D game user application 118.

[0039] As shown, the user interface 120 includes various objects that are presented to the screen output device 114 by the game user application 118. Each object may expose various object metadata 202 that may be accessed by the narrative engine 122 via the API 124. For example, the API 124 may be configured to allow the narrative engine 122 to enumerate the game objects of the user interface 120. Based on this received data, the narrative engine 122 may construct an augmented description 218. The augmented description 218 may be displayed in a dynamic text block 302 and displayed on the display output device 114.

[0040] In many examples, the dynamic text block 302 may first include a description of the user's surroundings, followed by the available actions 214. Each element of the dynamic text block 302 references the location of the player avatar 310, a game object 312 that is within the line of sight 308 of the player avatar 310, or a description of an audio event. For example, the dynamic text block 302 begins with the phrase 303, "You are standing in a room from which you cannot escape." The text of this phrase 303 may be taken from a description of an audio event that occurs at the user's location. The phrase 304, "There is a key nearby," in the dynamic text block 302 refers to a key game object 316 that is within the area marked as the line of sight 308 of the player avatar 310. The phrase 306, "There is an exit to the north," in the dynamic text block 302 refers directly to a door game object 314 that is within the area marked as the line of sight 308 of the player avatar 310.

[0041] The attention filter 204 may receive the position of the player avatar 310 and use the player avatar 310 and / or line of sight 308 to determine related objects 206 from the object metadata 202. In one example, the attention filter 204 may define the line of sight 308 to include as related objects 206 any interface elements that may have object metadata 202 indicating that the element is in the same room as the current room position of the player avatar 310 (e.g., door game object 314, key game object 316). These related objects 206 may be included in the interface model 210 by the object interpreter 208. Other objects, such as keys to other rooms or doorways to other rooms, are not relevant and are not included in the extended description 218.

[0042] The text of the extended description 218 may be compiled using text templates 217 that match the properties 212 of the associated objects 206 in the interface model 210. For example, the template 217 "Nearby is [object name]" may be utilized for a key game object 316 when that object has an object name property 212 and is within the line of sight 308 of the player avatar 310.

[0043] Although not shown in the dynamic text block 302, the interface model 210 may further include one or more available actions 214. These may be available as commands that can be invoked by the user. For example, a key game object 316 may specify a pick-up method, which may be added to the available actions 214 of the interface model 210, such that when a user speaks a command that includes a key and a pick-up action, the command recognizer 234 may identify the requested command and send it to the command executor 228 for processing.

[0044] 4 illustrates an example of the use of the narrative engine 122 with a 2D application user interface 120. In this example, a dynamic text block 402 containing an extended description 218 is shown, which is displayed in the user interface 120 along with the 2D user application 118.

[0045] As shown, the dynamic text block 402 includes various information describing the 2D user application 118. For example, the dynamic text block 402 may include a phrase 404 indicating the application name, which may be generated using the name of the in-focus application obtained from the related object 206 and applied to a template 217 that receives the application name, such as "You are using [application name]."

[0046] Additional elements in dynamic text block 402 may reference potential user actions represented by related objects 206 in the software (e.g., as shown in phrase 404) or frequently used menu items or functions (e.g., as shown in phrase 406). Phrase 408 in dynamic text block 402, "Your 'pinned' notes are 'Shopping' and 'To-Do'," may be prioritized and placed earlier in dynamic text block 402 because the user has pinned those items as shown in user interface 120 by element 410, indicating that those notes are relatively important.

[0047] In many examples, the dynamic text block 402 may first include a description of the user's situation, followed by available actions 214, where available actions 214 may include menu commands available in the user interface 120, such as creating a new note, searching for notes, selecting a note by title, etc. This order is merely an example, and other orders of properties 212 and available actions 214 may be used.

[0048] 5 illustrates an example of the use of the narrative engine 122 for a 3D game user interface 120. This example shows a dynamic text block 502 containing an extended description 218, which is displayed in the user interface 120 along with the user application 118 of the 3D application.

[0049] As shown, the dynamic text block 502 includes various information describing the 3D user application 118. For example, the dynamic text block 502 may include a phrase 503 that indicates the user's location within the 3D application. This may be selected based on the associated object 206 that is closest to the user's location. Here, a house object 510 is closest to the user. In some examples, the section of the map where the user is located may be marked with a property 212 such as a map area, the selected object may be marked with a property 212 such as a landmark object, and the narrative engine 122 may use a template 217 such as "You are near [landmark object] in [map area]."

[0050] In another example, the dynamic text block 502 may include a phrase 508 describing the number of other users included in the interface model 210. For example, a template 217 may be used such as, "There are [number] [object type] here," where "object type" is the type or types of properties 212 of the related objects 206 in the interface model 210, and "number" is the number of those related objects 206 that have that same type.

[0051] The dynamic text block 502 may include context-aware information regarding an ongoing interaction that a user is having with the user application 118. In this example, one of the users (Danny) has been selected and a menu of commands relevant to that user is available in the user interface 120. Thus, the dynamic text block 502 may include a phrase 504 that describes the context in which the interaction with the user Danny is being coordinated. Additionally, a phrase 506 may be provided that includes a list of available actions 214, such as, for example, "From here, you can [list of available actions 214 formatted into a comma-separated list]", where each of the available actions 214 may be listed based on the method metadata of the selected Danny's associated object 206. Thus, again, the extended description 218 includes a description of the user's context first, followed by the available actions 214, but may be in other orders.

[0052] 6 illustrates an example of the use of the narrative engine 122 for a store application user interface 120. The store may allow a user to shop for items such as a wallet as shown in the example. In some examples, the store user interface 120 may be presented to a user via a web application or a mobile app. In another example, the store user interface 120 may be presented as part of a 3D user interface 120, such as a metaverse store. In the metaverse example, a user may enter the store level and navigate to a store item, for example, by setting the store as a destination using a voice command to a virtual assistant.

[0053] A user may provide a command in natural speech or text, such as requesting a particular brand of wallet. A user interface 120 may be provided in response to the command. As shown, the name 602 of the wallet is presented along with a wallet mesh 604, a wallet description 606, and a list of various styles 608. Each of the styles 608 may include a texture 610 and a price 612 that corresponds to that style 608. The user interface 120 may also include item size 614 information, such as height, depth, width, weight, shoulder strap length, etc.

[0054] Figure 7 shows an example of object metadata 202 for a wallet item shown in the store user interface 120 of Figure 6. For example, the object metadata 202 may specify a name 602 for the wallet, a mesh 604 corresponding to the wallet, a wallet description 606, and a set of wallet styles 608, where each style 608 may include a respective texture 610 and price 612. Additionally, the currently selected texture 610 may be specified with a selected texture tag to describe how the mesh 604 is textured.

[0055] The object metadata 202 may be used to render the user interface 120 itself. Additionally, the object metadata 202 may be received from the user interface 120 via the API 124 and compiled into an interface model 210 by the attention filter 204 and object interpreter 208 to enable the narrative engine 122 to provide additional accessible functionality to the presentation of the store user interface 120. It should be noted that while the object metadata 202 is shown in JavaScript Object Notation (JSON), this is merely an example and various formats for the object metadata 202 may be used.

[0056] For example, the narrative engine 122 receiving the user interface 120 may use the attention filter 204 to filter the object metadata 202 to the related objects 206 available in the purse portion of the store, while generating an interface model 210 of the related objects 206 using the object interpreter 208. In response to the user interface 120 being displayed, the narrative engine 122 may construct an extended description 218. In one example, the extended description 218 may include an extended description 218 showing, in natural language, the name 602 of the purse, a description 606 of the purse, and a list 608 of different styles. The narrative engine 122 may begin speaking the extended description 218 using the text-to-speech engine 224.

[0057] In an example dialogue, the user may pause before the complete extended description 218 is read by the narrative engine 122 and say, "Do you have brown leather?" In response to receiving the user input 226, the narrative engine 122 may convert the user input 226 into recognized text 232 using a speech-to-text engine 230. The command recognizer 234 may use the recognized text 232 to identify available actions 214. In one example, a list of styles 608 may be compiled into available actions 214 of the interface model 210 that supports selection from the styles 608. The available actions 214 may include a single style 608 that includes the words "brown leather." The narrative engine 122 may construct a response that reads, "The style includes 'brown leather exterior, tan lambskin interior.' This style is priced at $5,200."

[0058] In a further example interaction, the user may ask, "What's the size?" Again, the narrative engine 122 may use the speech-to-text engine 230 to convert the user input 226 into recognized text 232. The command recognizer 234 may use the recognized text 232 to recognize that the interface model 210 has a size property 212 and build a phase to say the size 614 of the wallet, e.g., "The wallet has a height of 21 cm, a depth of 11 cm, a width of 27 cm, a weight of 0.6 kg, and a shoulder strap drop of 54.5 cm." Importantly, the answer to the question may be obtained from the interface model 210 without any additional knowledge of the wallet object by the narrative engine 122.

[0059] 8 illustrates an example process 800 illustrating a main interface loop for the operation of the narrative engine 122. In one example, the process 800 may be performed by a computing device 102 executing the narrative engine 122 and the user application 118, as described in detail herein.

[0060] In operation 802, the narrative engine 122 receives object metadata 202. In one example, the narrative engine 122 captures or otherwise receives the object metadata 202 from the user interface 120 using the API 124. In one example, in the case of a 3D scene, such as one rendered in Unity or other 3D engine, each rendered object may have object metadata 202 that may be captured by the API 124. In another example, in the case of a 2D web page, the HTTP markup of the web page may contain or otherwise define object metadata 202 that may be read by the narrative engine 122 via the API 124. In yet another example, window position, text, and other attributes may be captured by the API 124 via enumeration of windows on the desktop and / or using other OS-level interface functions. In yet another example, in the case of a console application, the console buffer text may be read by the narrative engine 122 via the API 124.

[0061] In operation 804, the narrative engine 122 describes the user's surroundings, which may include filtering the object metadata 202 using an attention filter 204 to determine related objects 206, constructing an interface model 210 using an object interpreter 208, and generating an extended description 218 based on properties 212 of the related objects 206 using a description generator 216.

[0062] In one example, the attention filter 204 of the narrative engine 122 may filter the object metadata 202 received in operation 802 to relevant objects 206. In the case of a video game, this object metadata 202 may include game objects 312, however defined, that are within the line of sight 308 or are otherwise near the user. In the case of a 2D application (e.g., a word processor, other productivity application, web pages, etc.), the object metadata 202 may refer to windows, dialog boxes, buttons, sliders, text boxes, web page links, etc. that make up the user interface 120. In the case of a console application, the object metadata 202 may include text that is displayed on the console. In some examples, the attention filter 204 may simply enable the processing of all object metadata 202. In other examples, to narrow the context to a more relevant environment, the attention filter 204 may filter the object metadata 202 based on properties 212 of the object metadata 202, for example, limiting the object metadata 202 to objects within a predefined distance from the user and / or within the user's field of view, limiting the object metadata 202 to controls within a predefined 2D distance from the mouse cursor, and / or limiting the object metadata 202 to interface elements that are enabled.

[0063] The description generator 216 may generate natural language describing the properties 212 of the related object 206. In another example, the description generator 216 may generate natural language describing the available actions 214 of the related object 206. The description generator 216 may provide a natural language description based on the metadata of the related object 206 using text templates 217. Each template 217 may include natural language text with one or more placeholders for the values ​​of the properties 212 or available actions 214 of the related object 206 being described. For example, to generate information describing an environment, the description generator 216 may use templates 217 such as "You are using [application name]", or "You are located near [object name]", or "You are facing [direction]", or "There is [object name] nearby that is [attribute]".

[0064] In operation 806, the narrative engine 122 lists the interactive objects in the user interface 120. Similar to operation 804, the narrative engine 122 may again use the description generator 216 to generate an extended description 218 based on the properties 212 of the associated objects 206. However, in this example, the available actions 214 may be used to build a list of available commands that may be executed by the user in the user interface 120. For example, a phrase may be provided that includes a list of available actions 214, such as "From here, you can [list of available actions 214 formatted into a comma-separated list]," where each of the available actions 214 may be listed based on the method metadata of the selected associated object 206. For example, if a key game object 316 has a pickable action 214 in the interface model 210, the description generator 216 may add a sentence or phrase to the extended description 218 indicating that a command to pick up the key is available.

[0065] At operation 810, the narrative engine 122 presents the augmented description 218 in the user interface 120. In one example, the narrative engine 122 may use a text-to-speech engine 224 to convert the augmented description 218 into a simulated human voice and provide the voice to an audio output device 114, such as a speaker or headphones. In another example, the narrative engine 122 may use an overlay generator 222 to create a visual textual representation of the augmented description 218 and provide it over the existing context of the user interface 120 via the display output device 114. The user settings 220 may be utilized to determine whether the augmented description 218 is presented visually, audibly, both, or in some other manner. For example, the user settings 220 may determine how to present the augmented description 218 based on the user's level or type of disability.

[0066] In operation 812, the narrative engine 122 processes the user input 226. This processing may include receiving the user input 226 from the user interface 120 via the API 124, providing the user input 226 to a speech-to-text engine 230 to generate recognized text 232, and using a command recognizer 234 to identify actions within the interface model 210 that are provided to a command executor 228 for processing (e.g., via the API 124 or otherwise). Further aspects of the processing of the user input 226 are described in more detail with respect to process 900.

[0067] At operation 814, the narrative engine 122 updates based on the user input 226. In one example, the user input 226 at operation 812 may include execution of one or more commands that may change the state of the user interface 120. This may cause the narrative engine 122 to return to operation 802 to again accept the object metadata 202, update the interface model 210, generate a new extended description 218, etc. In some examples, control may be passed to operation 802 based on other conditions, such as the narrative engine 122 detecting a change in the user interface 120 that is not the result of the user input 226, or based on the expiration of a periodic timeout during which the narrative engine 122 performs an update.

[0068] 9 illustrates an example of a process 900 by which the narrative engine 122 responds to user input 226. Like the process 800, the process 900 may be performed by a computing device 102 executing the narrative engine 122 and a user application 118, as discussed in detail herein.

[0069] At operation 902, the narrative engine 122 receives user input 226. The user input 226 may be received at the computing device 102 via one or more input devices 116. The user input 226 may be provided by the computing device 102 to a user application 118. The user input 226 may also be provided to the narrative engine 122 for further processing to facilitate operation of the narrative interface.

[0070] At operation 904, the narrative engine 122 determines whether the user input 226 includes voice or text. If the user input 226 is voice input, for example received from a microphone, control passes to operation 906. If not, control passes to operation 908.

[0071] At operation 906, the narrative engine 122 converts the speech to the recognized text 232. In one example, the narrative engine 122 uses the speech-to-text engine 230 to parse the user input 226 into a textual representation as the recognized text 232. After operation 906, control proceeds to operation 908.

[0072] In operation 908, the narrative engine 122 parses the recognition text 232. In one example, the command recognizer 234 may receive the recognition text 232 and process the recognition text 232 to identify which of the available actions 214 to perform. For example, the command recognizer 234 may scan the recognition text 232 to search for action words, such as names of available actions 214 in the interface model 210. In another example, the command recognizer 234 may scan for predefined verbs, such as "help," or other actions.

[0073] At operation 910, the narrative engine 122 determines whether an action exists. If such an available action 214 is found, control passes to operation 912. If not, control passes to operation 914. At operation 912, the narrative engine 122 determines whether the action is executable. In one example, the narrative engine 122 may verify that the action can occur within the architecture of the user application 118. If not, control passes to operation 914.

[0074] At operation 914, the narrative engine 122 describes the error that occurred. In one example, the error may indicate that no action was found in the recognized text 232. In such an example, the error may indicate that no available action 214 was found in the recognized text 232. In another example, the error may indicate that the available action 214 cannot be performed on the indicated related object 206. As an example, the recognized text 232 "pick up the car" may not be able to be performed even though the action "pick up" is available for other objects, such as keys. In such an example, an error may be displayed stating that "cars do not support the action pick up." In some examples, the error may be returned to the user via the text-to-speech engine 224 or via the overlay generator 222.

[0075] At operation 916, the narrative engine 122 performs the action. For example, the narrative engine 122 can instruct the command recognizer 234 to instruct the command executor 228 to perform the spoken available action 214. After operation 916, control returns to operation 902.

[0076] Although processes 800-900 are shown in a looped sequence, in many embodiments, processes 800-900 may be performed sequentially. Also, as shown in processes 800-900, one or more operations of processes 800-900 may be performed simultaneously and / or out of order.

[0077] Thus, the narrative engine 122 may evaluate information in the user application 118 and present it to the user as text and / or as audio. The narrative engine 122 then processes user input 226, such as text or audio, from the user. This input may be used to trigger application functionality.

[0078] The processes, methods, or algorithms disclosed herein can be delivered to and executed by a processor, controller, or computer, including any existing programmable or dedicated electronic control unit. Similarly, the processes, methods, or algorithms can be stored in many forms, including but not limited to information permanently stored in a non-writable storage medium, such as a read-only memory (ROM) device, and information mutably stored in a writable storage medium, such as a floppy disk, magnetic tape, compact disk (CD), RAM device, and other magnetic and optical media, as data and instructions executable by a controller or computer. The processes, methods, or algorithms can also be implemented in software executable objects. Alternatively, the processes, methods, or algorithms can be embodied in whole or in part using suitable hardware components, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0079] Although exemplary embodiments have been described above, it is not intended that these embodiments describe all possible forms encompassed by the scope of the claims. The words used herein are words of description rather than limitation, and it is understood that various modifications are possible without departing from the spirit and scope of the present disclosure. As previously described, features of various embodiments can be combined to form further embodiments of the present invention that may not be explicitly described or illustrated. Although various embodiments may be described as offering advantages or being preferred over other or prior art embodiments with respect to one or more desired characteristics, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system attributes that depend on the particular application and embodiment. These attributes may include, but are not limited to, strength, durability, life cycle, marketability, appearance, packaging, size, serviceability, weight, manufacturability, ease of assembly, and the like. Thus, to the extent that any embodiment is described as less desirable than other or prior art embodiments with respect to one or more characteristics, these embodiments may be desirable for the particular application, not outside the scope of the present disclosure.

[0080] With respect to processes, systems, methods, heuristics, etc. described herein, steps of such processes, etc. have been described as occurring according to a certain order, but it should be understood that such processes may be practiced with the described steps performed in orders other than the order described herein. Furthermore, it should be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the description of the steps herein is provided for the purpose of illustrating particular embodiments and should not be construed as limiting the scope of the claims in any way.

[0081] Therefore, it should be understood that the above description is illustrative and not restrictive. Many embodiments and applications other than the examples provided will be apparent upon reading the above description. The scope should not be determined with reference to the above description, but instead with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technology discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In summary, it should be understood that this application is capable of modification and variation.

[0082] All terms used in the claims are intended to be given their broadest reasonable interpretation and ordinary meaning as understood by those skilled in the art described herein, unless expressly indicated to the contrary in the specification. In particular, the use of singular articles such as "a," "the," "said," etc., should be read as describing one or more of the indicated elements, unless an express limitation to the contrary is recited in the claim.

[0083] The Abstract of the Disclosure is provided to enable the reader to quickly grasp the nature of the technical disclosure. The Abstract of the Disclosure is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Moreover, in the foregoing Detailed Description, it will be seen that various features have been grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Accordingly, the following claims are hereby incorporated into this specification, with each claim standing on its own as separately claimed subject matter.

[0084] Although exemplary embodiments have been described above, it is not intended that these embodiments describe all possible forms of the invention. Rather, the words used herein are words of description rather than limitation, and it will be understood that various changes may be made without departing from the spirit and scope of the invention. Moreover, features of various embodiments may be combined to form further embodiments of the invention.

Claims

1. a computing device including an input device and an output device, The computing device executes a narrative engine to receiving object metadata describing content of a user interface from a user application providing the user interface via the input device and the output device; generating an augmented description of the user interface, the augmented description including a description of an environment of the user interface and a list of actions to be performed on the user interface; presenting the augmented description using the output device; Processing a user input requesting one of said actions; and updating the extended description based on the user input; The system is programmed to do so.

2. the computing device further comprising: The system of claim 1, wherein the object metadata is received from the user application using an application programming interface (API), the API including extensions for each type of user interface supported, and the narrative engine is programmed to access the object metadata for a particular user interface type.

3. The system of claim 2 , wherein the user interface is a 3D user interface rendered by a 3D engine, elements of the 3D user interface being rendered according to the object metadata, and the object metadata being accessed by a narrative engine via the API.

4. the user interface is a web user interface rendered by a web browser; elements of the web user interface are rendered according to the object metadata contained in HyperText Transfer Protocol (HTTP) markup; The system of claim 2 , wherein the object metadata is accessed from the HTTP markup by the narrative engine via the API.

5. the user interface is a console application user interface; The system of claim 2 , wherein the object metadata is accessed by the narrative engine via an API from a console text buffer of the console application.

6. The system of claim 1 , wherein the extended description is presented as an overlay superimposed on the user interface.

7. The system of claim 1 , wherein the augmented description is audibly presented as a computer-generated voice.

8. The system of claim 1 , wherein the narrative engine includes user settings that define how to present the extended description based on a level or type of disability of a user of the narrative engine.

9. The narrative engine further comprises: filtering the object metadata using properties of the object metadata with an attention filter to determine related objects in the object metadata; limiting the object metadata to elements of the user interface that are within a predefined distance of a user's avatar; restricting the object metadata to the elements of the user interface that are within the user's field of view; or 2. The system of claim 1, further programmed to include one or more of: limiting the object metadata to elements of the user interface that are within a predetermined 2D distance from a mouse cursor; and limiting the object metadata to elements of the user interface that are enabled.

10. The narrative engine further comprises: constructing an interface model that describes the properties and available actions of the related objects; 10. The system of claim 9, further comprising: a description generator programmed to generate a description of the environment based on the properties of the related objects and to generate a list of available actions based on the available actions of the related objects.

11. 11. The system of claim 10, wherein the description creator is configured to generate the extended description using a template that includes natural language text and placeholders for values ​​of the properties or available actions of the related object being described.

12. The narrative engine further comprises: Converting the user input into recognized text using a speech-to-text engine; scanning the recognized text for names of the available actions in the interface model; 11. The system of claim 10, further programmed to prompt the user application to perform the spoken available action.

13. receiving object metadata describing the content of the user interface from a user application providing the user interface via input and output devices of a computing device; - generating an augmented description of the user interface, the augmented description including a description of an environment in the user interface and a list of actions to be performed on the user interface; presenting the augmented description using the output device; processing a user input requesting one of said actions; and updating the extended description based on the user input.

14. 14. The method of claim 13, further comprising receiving the object metadata from the user application using an application programming interface (API), the API including extensions for each type of user interface supported to enable access to the object metadata for each particular type of user interface.

15. The method of claim 13 , wherein the extended description is presented as an overlay superimposed on the user interface.

16. The method of claim 13 , wherein the augmented description is aurally presented as a computer-generated voice.

17. 14. The method of claim 13, further comprising presenting the extended description according to a user setting indicating a level or type of a user's disability.

18. filtering the object metadata using properties of the object metadata to determine related objects in the object metadata; The step comprises: restricting the object metadata to elements of the user interface that are within a predefined distance of a user's avatar; restricting the object metadata to elements of the user interface that are within the user's field of view; or limiting the object metadata to elements of the user interface that are within a predetermined 2D distance from a mouse cursor; and limiting the object metadata to elements of the user interface that are enabled; The method of claim 13, comprising one or more of the following:

19. constructing an interface model that describes the properties and available actions of the related objects; 20. The method of claim 18, further comprising: using a description generator to generate a description of the environment based on the properties of the related objects, and to generate the list of actions based on the available actions of the related objects.

20. 20. The method of claim 19, further comprising generating the extended description using a template that includes natural language text and placeholders for values ​​of the properties or available actions of the associated object being described.

21. converting the user input into recognized text using a speech-to-text engine; scanning the recognized text for names of the available actions in the interface model; 20. The method of claim 19, further comprising: prompting the user application to perform the spoken available action.

22. 1. A non-transitory computer-readable medium comprising narrative engine instructions that, when executed by one or more processors of a computing device, cause the computing device to perform operations, the operations including: receiving object metadata describing the content of a user interface from a user application providing the user interface via input and output devices of the computing device, the object metadata including receiving the object metadata from the user application using an API of the narrative engine, the API including extensions for each type of user interface supported allowing access to the object metadata for a particular user interface type; filtering the object metadata using properties of the object metadata to determine related objects in the object metadata; generating an extended description of the user interface using the associated objects, the extended description including a description of an environment of the user interface and a list of actions to be performed on the user interface; presenting, with the output device, one or more of the following: as an overlay on the user interface, or audibly as a computer-generated sound; processing a user input requesting one of the actions; updating the extended description based on the user input; presenting the updated extended description using the output device; A non-transitory computer readable medium comprising:

Images