Operation command boundary

Abstract

A method for managing commands utilizing command boundaries includes establishing a first command boundary for a first intelligent virtual assistant operating on a first electronic device. The method also includes identifying portions of the audio from the multiple sources within the first command boundary in response to receiving audio from the multiple sources within a vicinity of the first electronic device. The method also includes sending a command to the first intelligent virtual assistant on the first electronic device in response to determining that a command is identifiable in the portion of the audio from the multiple sources within the first command boundary.

Description

Technical Field

[0001] The present disclosure generally relates to intelligent virtual assistants, and more particularly to establishing an operating command boundary for devices employing intelligent virtual assistants.

[0002] An intelligent virtual assistant (IVA) represents a software agent that can perform various tasks for a user based on oral commands or questions provided by the user, and the IVA operates on an electronic device. An Internet of Things (IoT) device is an example of an electronic device having an IVA, which can connect to other devices and systems via the Internet to exchange data with them and process any oral command or question. The user providing the oral command or question is usually located in the vicinity of the electronic device having the IVA, and the microphone on the electronic device can capture the oral command or question. However, there are cases where the user's surrounding environment may include various interfering noises such as personal unrelated conversations, which can affect the way the electronic device having the IVA captures the oral command or question. The lack of distinction between near-field sound and far-field sound affects the way the IVA accurately captures the oral command or question.

[0003] A user's surrounding environment can also include multiple IVAs operating on multiple electronic devices, and verbal commands or questions can be intercepted by more than one of the multiple IVAs sharing the surrounding environment. Currently, a user can identify a specific IVA operating on a specific electronic device within the shared environment and direct commands or questions to it, but the user needs to remember an identifier word or phrase for that specific IVA operating on that specific electronic device. The identifier word or phrase may not necessarily be known by an individual who is not familiar with the surrounding environment and the multiple IVAs operating within it. In some cases, an IVA may mistakenly receive a verbal command directed to another IVA in the surrounding environment due to a user misidentifying two IVAs. Therefore, when multiple IVAs are operating in an environment with multiple rooms, a malicious individual could potentially eavesdrop on confidential information at an IVA that mistakenly receives a verbal command, resulting in a potential security breach. In addition to eavesdropping on confidential information, a malicious individual could access confidential information through an IVA by verbal commands or questions. [Overview of the project]

[0004] Embodiments of the present invention also disclose a method, a computer program product, and a computer system for managing commands using a command boundary, the method, the computer program product, and the computer system being able to establish a first command boundary for a first intelligent virtual assistant operating on a first electronic device. The method, the computer program product, and the computer system being able to identify portions of audio from multiple sources within the first command boundary in response to receiving audio from multiple sources within the vicinity of the first electronic device. The method, the computer program product, and the computer system being able to transmit a command to the first intelligent virtual assistant on the first electronic device in response to determining that a command is identifiable in portions of audio from multiple sources within the first command boundary.

[0005] Embodiments of the present invention can establish a second command boundary for a second intelligent virtual assistant operating on a second electronic device, where the first command boundary does not overlap with the second command boundary. Embodiments of the present invention can establish a first set of command rules for a first intelligent virtual assistant operating on a first electronic device, and a second set of command rules for a second intelligent virtual assistant operating on a second electronic device.

[0006] Embodiments of the present invention can analyze commands with respect to personal information. In response to determining that a command is related to personal information, embodiments of the present invention can send an instruction to a first intelligent virtual assistant on a first electronic device based on a personal information command boundary for the command. Embodiments of the present invention can classify commands that can be executed by the first intelligent virtual assistant on the first electronic device. In response to determining, based on the classification, that a command is related to personal information, embodiments of the present invention can query the user to establish a personal information command boundary for the command, where the volume level response to the first intelligent virtual assistant on the first electronic device is associated with the personal information command boundary.

[0007] Embodiments of the present invention classify commands related to personal information and a second portion of commands related to personal information, classifying commands as those containing personal information or those that produce an audible response containing personal information by a first intelligent virtual assistant on a first electronic device. Embodiments of the present invention classify a command as having personal information in its first portion, and since the command produces an audible response containing personal information by a first intelligent virtual assistant on a first electronic device, it can be determined that the command is related to personal information.

[0008] Embodiments of the present invention disclose a method, a computer program product, and a computer system for managing commands using a command boundary, the method, the computer program product, and the computer system being able to establish a near-field radius for an intelligent virtual assistant operating on a first electronic device, where the near-field radius is a command boundary separating near-field and far-field speech. The method, the computer program product, and the computer system being able to identify portions of speech from multiple sources within the near-field radius in response to receiving speech from multiple sources within the vicinity of the first electronic device. The method, the computer program product, and the computer system being able to instruct an intelligent virtual assistant to execute a command in response to determining that a command is identifiable in portions of speech from multiple sources within the near-field radius. [Brief explanation of the drawing]

[0009] [Figure 1] This is a functional block diagram showing a distributed data processing environment according to one embodiment of the present invention.

[0010] [Figure 2] A flowchart of a command boundary program for managing intelligent virtual assistant commands using a near-field command boundary, according to one embodiment of the present invention, is shown.

[0011] [Figure 3] A flowchart of a command boundary program for managing intelligent virtual assistant commands using a command boundary for multiple intelligent virtual assistants operating in a shared environment, according to one embodiment of the present invention, is shown.

[0012] [Figure 4] A flowchart of a command boundary program that analyzes commands directed to an intelligent virtual assistant for personal information, according to one embodiment of the present invention, is shown.

[0013] [Figure 5] This is a block diagram of the components of a computer system, such as the server computer shown in Figure 1, according to one embodiment of the present invention.

[0014] [Figure 6] This shows a cloud computing environment related to one embodiment of the present invention.

[0015] [Figure 7] An abstraction model layer relating to one embodiment of the present invention is shown. [Modes for carrying out the invention]

[0016] Figure 1 is a functional block diagram showing a distributed data processing environment, designated as 100 in total, according to one embodiment of the present invention. As used herein, the term “distributed” refers to a computer system comprising multiple physically separate devices operating together as a single computer system. Figure 1 provides only an example of one implementation and does not imply any limitation with respect to environments in which different embodiments may be implemented. Many modifications to the illustrated environment can be made by those skilled in the art without departing from the scope of the present invention enumerated by the claims.

[0017] The distributed data processing environment includes a server computer 102, a client device 104, an augmented reality (AR) device 106, and an Internet of Things (IoT) device 108, all interconnected via a network 110. The server computer 102 may be a standalone computing device, a management server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, transmitting, and processing data. In another embodiment, the server computer 102 may represent a server computing system that utilizes multiple computers as a server system, such as in a cloud computing environment. In yet another embodiment, the server computer 102 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a smartphone, or any programmable electronic device capable of communicating with the client device 104, the AR device 106, the IoT device 108, and other computing devices (not shown) in the distributed data processing environment via the network 110. In another embodiment, the server computer 102 represents a computing system that utilizes clustered computers and components (e.g., a database server computer, an application server computer, etc.) that function as a single pool of seamless resources when accessed within a distributed data processing environment. The server computer 102 includes a command boundary program 112 and a database 114. The server computer 102 may include internal and external hardware components, as will be shown and described in more detail with respect to Figure 5.

[0018] The client device 104 may be a laptop computer, tablet computer, smartphone, smartwatch, smart speaker, or any programmable electronic device capable of communicating with various components and devices in a distributed data processing environment (e.g., a server computer 102, an AR device 106, and an IoT device 108) via the network 110. The client device 104 may be a wearable computer. A wearable computer is a small electronic device that can be worn by a wearer under, with, or over clothing, and in or attached to glasses, a hat, or other accessories. Wearable computers are particularly useful in applications that require more complex computational support than simply hardware-coded logic. Generally, the client device 104 represents one or more programmable electronic devices, or a combination of programmable electronic devices, capable of executing machine-readable program instructions and communicating with other computing devices (not shown) in a distributed data processing environment via a network such as the network 110. In one embodiment, the client device 104 represents one or more devices associated with a user. The client device 104 includes an instance of a user interface 122 for interacting with the command boundary program 112 on the server computer 102.

[0019] AR device 106 represents a user-wearable augmented reality device (e.g., electronic contact lenses, wearable electronic headset) having a miniature integrated circuit capable of displaying content to the user. AR device 106 includes a microcontroller 116, a display 118, and a microphone 120, and a command boundary program 112 may receive verbal commands or questions from a user wearing AR device 106 for execution by one or more IoT devices 108. The microcontroller 116 may include a display control circuit for the display 118, a communication and power conversion circuit for communicating via the network 110 and managing the integrated power supply, and a sensor readout and control circuit for monitoring the eye movements of the user wearing AR device 106. The display 118 allows the user of AR device 106 to view various notifications and queries from the command boundary program 112. In one example where AR device 106 is an electronic contact lens, the display 118 is a translucent display and microlens array integrated into AR device 106 for viewing content. The AR device 106 may also include a power storage module, a solar cell module for charging the power storage module, a biosensor module for data acquisition (e.g., tracking eye movements), and a communication and power module for communicating with the server computer 102, client device 104, and IoT device 108 via the network 110. The microphone 120 captures verbal commands or questions directed to the IoT device 108, and the microcontroller 116 transmits the captured verbal commands or questions to the command boundary program 112 on the server computer 102.

[0020] IoT device 108 represents an electronic device connected to network 110 that can receive and execute voice commands provided by the user of client device 104 and / or AR device 106. Each IoT device 108 includes an instance of an intelligent virtual assistant (IVA), which represents a software agent capable of performing various tasks on behalf of the user based on commands or questions provided by the user. IoT device 108 may include, but is not limited to, electrical appliances, home security systems, health monitoring devices, factory equipment, wireless inventory trackers, biometric security scanners, and any other electronic devices embedded with sensors and software to connect to and exchange data with other devices and systems via the internet (e.g., network 110).

[0021] Network 110 may be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of these three, and may include wired, wireless, or fiber optic connections. Network 110 may include one or more wired and / or wireless networks capable of receiving and transmitting data, voice, and / or video signals, including multimedia signals including voice, data, and video information. Generally, Network 110 may be any combination of connections and protocols that support communication between a server computer 102, a client device 104, an AR device 106, an IoT device 108, and other computing devices (not shown) in a distributed data processing environment.

[0022] The command boundary program 112 can manage verbal commands directed to IoT devices 108 operating within a shared environment using a command boundary, and allows the administrator user of the command boundary program 112 to consent to one or more of the functions described herein to ensure that the administrator user's privacy is maintained. The command boundary program 112 manages verbal commands directed to IoT devices 108 by establishing a near-field radius for each IoT device 108. The command boundary program 112 receives audio from multiple sources within the vicinity of IoT device 108 and identifies audio from multiple sources within the near-field radius of IoT device 108. The command boundary program 112 analyzes the identified audio from multiple sources within the near-field radius of IoT device 108 for commands that can be executed by IoT device 108. If the command boundary program 112 determines that the identified audio does not contain an identifiable command, the command boundary program 112 notifies the user, via client device 104, AR device 106, and / or IoT device 108, that the command is identifiable. If the command boundary program 112 determines that the identified voice contains an identifiable command, the command boundary program 112 determines whether an AR device (e.g., AR device 106) is associated with the command. If the command boundary program 112 determines that the AR device is not associated with the command, the command boundary program 112 instructs the IoT device 108 to execute the command. If the command boundary program 112 determines that the AR device is associated with the command (i.e., AR device 106), the command boundary program 112 displays the identified command in AR device 106, and the user associated with AR device 106 can make modifications to the identified command. If the command boundary program 112 determines that no modifications to the identified command have been received, the command boundary program 112 instructs the IoT device 108 to execute the command.When the command boundary program 112 determines that a modification to the identified command has been received, the command boundary program 112 instructs the IoT device 108 to execute the modified command.

[0023] The command boundary program 112 can also manage spoken commands directed to IoT devices 108 operating within a shared environment by using the command boundary by establishing an operating command boundary for each IoT device 108. The command boundary program 112 identifies a plurality of IoT devices 108 within a defined area and establishes an operating command boundary for each of the plurality of IoT devices 108 to reduce the possibility that an unintended IoT device 108 inadvertently captures a spoken command or question provided by a user. The command boundary program 112 further establishes command rules for each of the plurality of IoT devices 108 to ensure that the security integrity is maintained if an unauthorized user attempts to extract information through the plurality of IoT devices 108. The command boundary program 112 receives audio with commands executable by at least one of the plurality of IoT devices 108 and analyzes the audio commands for personally identifiable information. The command boundary program 112 identifies the distance of the audio source associated with the audio commands for each of the plurality of IoT devices 108.

[0024] If the command boundary program 112 determines that the voice source is not within the command boundary of a specific IoT device 108, the command boundary program 112 ignores the voice with the command. If the command boundary program 112 determines that the voice source is within the command boundary of a specific IoT device 108, the command boundary program 112 determines whether the voice source command rule is satisfied. If the command boundary program 112 determines that the voice source command rule is not satisfied, the command boundary program 112 ignores the voice with the command. If the command boundary program 112 determines that the voice source command rule is satisfied, the command boundary program 112 determines whether command detection is disabled. If the command boundary program 112 determines that command detection is disabled, the command boundary program 112 ignores the voice with the command. If the command boundary program 112 determines that command detection is enabled, the command boundary program 112 sends the command to the appropriate IoT device 108.

[0025] The command boundary program 112 analyzes the command for personal information by classifying the commands executable by the IoT device 108 and determining whether the command is related to personal information. If the command boundary program 112 determines that the command is not related to personal information, the command boundary program 112 identifies the distance of the voice source for each of the plurality of IoT devices 108. If the command boundary program 112 determines that the command is related to personal information, the command boundary program 112 queries the user to establish a personal information command boundary for the commands executable by the plurality of IoT devices 108. The command boundary program 112 receives and stores the personal information command boundary for the commands executable by the plurality of IoT devices 108.

[0026] The database 114 is a repository that stores various data, including floor plans (e.g., individual residences, manufacturing floors, warehouse floors), near-field radius boundaries for each IoT device 108, operational command boundaries for each IoT device 108, voice source command rules for each IoT device 108, personal information command boundaries for each IoT device 108, and any other data that the command boundary program 112 uses to manage verbal commands directed to the IoT devices 108. In the shown embodiment, the database 114 resides on the server computer 102. In another embodiment, the database 114 may reside on the client device 104 or elsewhere in a distributed data processing environment, provided that the command boundary program 112 has access to the database 114. The database is an organized collection of data, and the database 114 may be implemented on any type of storage device capable of storing data and configuration files that can be accessed and used by the command boundary program 112, such as a database server, a hard disk drive, or flash memory.

[0027] The user interface 122 allows the user to make requests or issue commands to the server computer 102, client device 104, AR device 106, and IoT device 108 via the network 110. The user interface 122 also allows the user to respond to and receive information and instructions on the client device 104 via the network 110. In one embodiment, the user of the client device 104 accesses the user interface 122 via voice commands in natural language. In one embodiment, the user interface 122 may be a graphical user interface (GUI) or a web user interface (WUI) that can display text, documents, web browser windows, user options, application interfaces, and instructions for operation, and includes information (graphics, text, and sound, etc.) that the program presents to the user, and control sequences that the user employs to control the program. The user interface 122 allows the user of the client device 104 to interact with a command boundary program 112 running on the server computer 102.

[0028] Figure 2 shows a flowchart of a command boundary program for managing intelligent virtual assistant commands using a near-field command boundary, according to one embodiment of the present invention.

[0029] The command boundary program 112 establishes a near-field radius for IoT devices (202). The command boundary program 112 can distinguish between near-field and far-field voice, and processes voice commands received within the near-field voice radius when identifying commands that can be executed by IoT devices. The command boundary program 112 may establish a near-field radius for IoT devices by querying an administrative user through associated client devices and / or AR devices. Alternatively, the command boundary program 112 may utilize a predetermined radius (i.e., boundary) that separates near-field and far-field voice, where voice captured within the predetermined radius is near-field voice and voice captured outside the predetermined radius is far-field voice. To establish a near-field radius by querying an administrative user, the command boundary program 112 may send a request to the client devices and / or AR devices associated with the administrative user to input a radius (e.g., 10 feet (3.05 meters), 3.5 meters) that separates near-field and far-field voice. When an AR device is used to input a near-field radius, the command boundary program 112 can train the IoT device by instructing the user to provide sample commands along the desired field radius in order to establish a near-field radius for the IoT device. The command boundary program 112 can use the sample commands provided during the training of the IoT device to compare with future commands received by the IoT device and identify which of the future commands are located within the near-field radius of the IoT device.

[0030] The command boundary program 112 receives audio from multiple sources within the vicinity of the IoT device (204). Following the establishment of a near-field radius for the IoT device, the microphone on the IoT device captures a wake word, also called a trigger word, to activate the IoT device and capture audio within its vicinity. The IoT device may capture audio for a predetermined duration (e.g., 30 seconds) or for a duration in which the audio level (e.g., a decibel measurement) exceeds a predetermined threshold. The audio captured by the IoT device may come from multiple sources within its vicinity, and these sources may include various individuals participating in multiple conversations, as well as any ambient noise that may be present in the surrounding environment (e.g., construction, music, vehicles). The IoT device captures audio from multiple sources within its vicinity and transmits the captured audio to the command boundary program 112.

[0031] The command boundary program 112 identifies audio from multiple sources within the near-field radius of the IoT device (206). The command boundary program 112 analyzes the audio received from multiple sources and removes any ambient noise from the surrounding environment that may have been captured by the IoT device through signal processing. Subsequently, the command boundary program 112 distinguishes between near-field and far-field audio by utilizing audio level (e.g., average decibel measurement), acoustic cues (e.g., sound frequency), and speech clarity (e.g., peak decibel measurement). In some embodiments, the command boundary program 112 can utilize data received from AR devices located near the IoT device to identify audio located within the near-field radius. For example, the command boundary program 112 can identify AR devices operating in the vicinity of the IoT device if each AR device is connected to a local network shared with the IoT device. The command boundary program 112 uses the location information of the identified AR devices to determine whether any of the AR devices are located within the near-field radius of the IoT device. Furthermore, the command boundary program 112 can query the portion of the AR device located within the near-field radius of the IoT device and determine whether each microphone in that portion of the AR device has captured the audio previously received by the command boundary program 112 in (204).

[0032] The command boundary program 112 analyzes identified voices from multiple sources within the near-field radius of the IoT device to determine commands that can be executed by the IoT device (208). The command boundary program 112 analyzes the voices identified within the near-field radius of the IoT device to identify commands that can be executed by the IoT device, which may be questions or instructions for the IoT device to perform an action. The command boundary program 112 uses natural language processing (NLP) to identify commands that can be executed by the IoT device. The command boundary program 112 determines whether the command is identifiable (decision 210). If the command boundary program 112 determines that the command is not identifiable (branch to "no", decision 210), the command boundary program 112 notifies the user that the command is not identifiable (214). If the command boundary program 112 determines that the command is identifiable (branch to "yes", decision 210), the command boundary program 112 determines whether an AR device is associated with the command (decision 216).

[0033] The command boundary program 112 notifies the user that the command is unidentifiable (214). In this embodiment, the command boundary program 112 instructs the IoT device to respond with an audible response indicating that the command is unidentifiable. The command boundary program 112 may also provide one or more recommendations in the audible response to help the individual provide an identifiable command. In one example, the command boundary program 112 provides a recommendation to instruct the individual to move closer to the IoT device before providing the wake word and command combination. In another example, the command boundary program 112 provides a recommendation to instruct the individual to increase the near-field radius relative to the IoT radius. In another embodiment, the command boundary program 112 notifies the user that the command is unidentifiable by sending a notification to client devices and / or AR devices associated with the voice within the near-field radius of the IoT device.

[0034] The command boundary program 112 determines whether the AR device is associated with a command (decision 216). As mentioned above, the command boundary program 112 can identify voices located within the near-field radius by utilizing data received from an AR device located near the IoT device. If the command boundary program 112 determines that the AR device is associated with a command (branch to "yes", decision 216), the command boundary program 112 displays the command in the AR device (218). If the command boundary program 112 determines that the AR device is not associated with a command (branch to "no", decision 216), the command boundary program 112 instructs the IoT device to execute the command.

[0035] The command boundary program 112 displays the command within the AR device (218). The command boundary program 112 displays commands directed to an IoT device within the AR device associated with the command, enabling the user to provide modifications to the command and any subcommands. In one embodiment, the command boundary program 112 has previously identified a command by associating the word with a wake word or phrase for the IoT device. The command boundary program 112 displays a partial command (i.e., a subcommand) within the AR device using visual indicators. A first unique indicator for a subcommand (i.e., color, font, highlighting) may indicate the current command that the IoT device should execute, and a second unique indicator for another subcommand in the queue that the IoT device should subsequently execute. The command boundary program 112 enables the user to modify each subcommand, enabling the user to modify the order in which the IoT device should execute each subcommand of the identified command.

[0036] In one example, the command boundary program 112 identifies a command from an individual with an associated AR device located within the near-field radius of the IoT device, stating, "AB, turn off the living room lights and turn on the hallway lights." The command boundary program 112 displays the identified command within the AR device associated with the command, with a first indicator highlighting "AB" as the wake word, a second indicator highlighting "Turn off the living room lights" as the first subcommand, and a third indicator highlighting "Turn on the hallway lights" as the second subcommand. Alternatively, the command boundary program 112 displays the identified command within the AR device associated with the command, with a first indicator (e.g., green font color) highlighting "Turn off the living room lights" as the first subcommand, and a second indicator (e.g., red font color) highlighting "Turn on the hallway lights" as the second subcommand, so that the first subcommand is executed by the IoT device before the second subcommand. The command boundary program 112 allows the user to modify the order of subcommands executed by the IoT device by changing the indicators for the first and second subcommands. As a result of the modification, the first indicator highlights "Turn on the hallway lights" as the first subcommand, and the second indicator highlights "Turn off the living room lights." The command boundary program 112 may accept any modifications to the subcommands via audible commands provided by the user of the AR device associated with the identified command. Alternatively, the command boundary program 112 may accept any modifications to subcommands via physical input on a client device (e.g., a mobile phone) paired with the AR device associated with the identified command.

[0037] In another example, command boundary program 112 identifies a command from an individual with an associated AR device located within the near-field radius of the IoT device, stating, "AB, set an alarm...and remind me to contact my manager at noon." Command boundary program 112 displays the identified command in the AR device using a first indicator that highlights "remind me to contact my manager at noon" as the first subcommand. However, command boundary program 112 cannot fully capture the second subcommand, "set an alarm...", and by displaying the identified command in the AR device, the user can modify the command to include any missing parts of the second subcommand. Command boundary program 112 may accept modifications to the second subcommand by repeating the second subcommand as a whole (e.g., "set an alarm for 11:55 a.m.") and assigning a second indicator to the second subcommand. Command boundary program 112 allows the user to later change the order of the first and second subcommands of the identified command.

[0038] The command boundary program 112 determines whether a command modification has been received (decision 220). If the command boundary program 112 determines that a modification has been received (branch to "yes", decision 220), the command boundary program 112 instructs the IoT device to execute the modified command (222). If the command boundary program 112 determines that a modification has not been received (branch to "no", decision 220), the command boundary program 112 instructs the IoT device to execute the command (224).

[0039] Figure 3 shows a flowchart of a command boundary program that manages intelligent virtual assistant commands using a command boundary for multiple intelligent virtual assistants operating in a shared environment, according to one embodiment of the present invention.

[0040] The command boundary program 112 identifies multiple IoT devices within a defined area (302). The command boundary program 112 identifies multiple IoT devices within a defined area as each IoT device connects to the local network. In one example, multiple IVAs running on multiple IoT devices are connected to a local network in a private home, and the command boundary program 112 identifies each of the multiple IoT devices as they connect to the local network in the private home. In another example, multiple IVAs running on multiple IoT devices are connected to a local network in an office environment, and the command boundary program 112 identifies each of the multiple IoT devices as they connect to the local network in the office environment. The command boundary program 112 has the ability to command each of the multiple IoT devices to perform an audio capture test to determine whether one of the multiple IoT devices can capture an audible command. For example, the command boundary program 112 commands a first IoT device connected to the local network to generate a unique voice at a determined frequency, and commands the remaining IoT devices to listen for a unique voice at the determined frequency. If a unique voice is captured by at least one of the remaining IoT devices, the command boundary program 112 determines that the IoT device generating the unique voice and the at least one IoT device that captured the unique voice share an operational command area. The operational command area represents an area where audible commands generated by a user can be captured by IoT devices, in this case by at least two IoT devices. The command boundary program 112 may perform voice capture tests on each IoT device to determine which command boundaries of the multiple IoT devices overlap with each other.

[0041] The command boundary program 112 establishes an operational command boundary for each of the multiple IoT devices (304). In this embodiment, the command boundary program 112 queries the management user to establish an operational command boundary for each of the multiple IoT devices. The management user has the ability to establish each operational command boundary as a radius surrounding each of the multiple IoT devices (e.g., 10 feet (3.05 meters), 3.5 meters). The command boundary program 112 may receive operational command boundaries for each of the multiple IoT devices via user input or verbal commands on an AR device or client device associated with the management user. In another embodiment, the command boundary program 112 establishes an operational command boundary for each of the multiple IoT devices based on additional voice capture tests performed on the multiple IoT devices. The command boundary program 112 may instruct each IoT device to generate unique voices at various volume levels in order to determine which of the remaining IoT devices among the multiple IoT devices will capture unique voices at various volume levels. For example, the first IoT device generates a unique voice at five different volume levels, and the second IoT device captures the unique voice at the maximum volume level of the five different volume levels. The command boundary program 112 uses a predetermined voice capture distance for each of the five different volume levels to determine the distance between the first IoT device and the second IoT device, based on the second IoT device capturing the unique voice at the maximum volume level generated by the first IoT device. Based on the predetermined distance, the command boundary program 112 establishes an operational command boundary for the first IoT device, and the radius of the operational command boundary for the first IoT device is smaller than the predetermined distance to the unique voice captured at maximum volume. The command boundary program 112 executes a unique voice at various volume levels for each of the multiple devices in order to establish each of the operational command boundaries.

[0042] The command boundary program 112 establishes command rules for each of the multiple IoT devices (306). The command boundary program 112 may establish command rules for each of the multiple IoT devices based on constraints provided by the user. Alternatively, the command boundary program 112 has the ability to learn command patterns for each IoT device and establish command rules for each IoT device based on the learned command patterns. In one example, the command rules indicate which individuals can interact with each IAV operating on each IoT device, based on one or more of the user's established utterance patterns and / or one or more devices (e.g., client devices, AR devices) associated with the user providing the commands. In another example, the command rules indicate whether a user can provide commands to an IoT device when the user is located outside the operational command boundary. If a user is located outside the operational command boundary of an IoT device and is providing commands to the IoT device via an intermediate device (e.g., a client device, AR device), the command boundary program 112 may ignore the command for security reasons. If a user is located within the operational command boundary of an IoT device and is providing commands to the IoT device via an intermediate device (e.g., a client device, an AR device), the command boundary program 112 accepts the commands.

[0043] The command boundary program 112 receives voice with a command that can be executed by an IoT device (308). The command boundary program 112 receives voice with a command that can be executed by an IoT device from one of several IoT devices operating in a shared environment. The command boundary program 112 analyzes the command for personal information (310). The command boundary program 112 analyzes the received voice with a command for personal information and determines whether action is required to ensure security integrity. The analysis of the command for personal information by the command boundary program 112 is discussed in more detail with respect to Figure 4.

[0044] The command boundary program 112 identifies the distance of the audio source to each of the multiple IoT devices (312). In one embodiment, the command boundary program 112 has the ability to determine the distance of the audio source to each of the multiple IoT devices by utilizing the captured audio levels of the multiple IoT devices at the time the audio with the command is received. The command boundary program 112 may also compare the audio levels of the multiple IoT devices at the time the audio with the command is received with the results of additional audio capture tests performed on the multiple IoT devices at various volume levels in (304). In another embodiment, the command boundary program 112 determines that the received audio with the command is associated with an AR device, and the command boundary program 112 queries the AR device for location information. The command boundary program 112 uses the captured location information of the AR device and the known locations of each of the multiple IoT devices in the shared environment to identify the distance of the audio source to each of the multiple IoT devices.

[0045] The command boundary program 112 determines whether the voice source is located within the command boundary of the IoT device (decision 314). If the command boundary program 112 determines that the voice source is located within the command boundary of the IoT device (branch to "yes", decision 314), the command boundary program 112 determines whether the voice source command rules are satisfied (decision 316). If the command boundary program 112 determines that the voice source is not located within the command boundary of the IoT device (branch to "no", decision 314), the command boundary program 112 ignores the voice with a command (322). Even if at least one of the multiple IoT devices could capture the voice with a command, the command boundary program 112 ignores the voice with a command because the voice source was located outside the operational boundary of the multiple IoT devices.

[0046] The command boundary program 112 determines whether the voice source command rules are met (decision 316). If the command boundary program 112 determines that the voice source command rules are met (branch to "yes", decision 316), the command boundary program 112 determines whether command detection is enabled (decision 318). If the command boundary program 112 determines that the voice source command rules are not met (branch to "no", decision 316), the command boundary program 112 ignores the voice with the command (decision 322).

[0047] The command boundary program 112 determines whether command detection is enabled (decision 318). The command boundary program 112 allows the administrator to temporarily disable command detection for multiple IoT devices, and the wake word for multiple IoT devices is temporarily disabled (for example, for 1 hour). If the command boundary program 112 determines that command detection is enabled (branch "yes", decision 318), the command boundary program 112 sends a command to the appropriate IoT device (320). If the command boundary program 112 determines that command detection is not enabled (branch "no", decision 318), the command boundary program 112 ignores the voice with the command (decision 322).

[0048] The command boundary program 112 transmits a command to the appropriate IoT device (320). In this embodiment, if the command relates to personal information, the command boundary program 112 transmits commands and instructions for action to ensure security integrity to the appropriate IoT device. The appropriate IoT device represents a previously identified IoT device among a plurality of IoT devices, which includes an audio source located within the corresponding command boundary. In one example, the instruction may include the use of a specific volume level when an IVA operating on the appropriate IoT device responds to a command relating to personal information, the specific volume level being lower than an operational volume level previously established by the user. The specific volume level of the audible response by the IVA on the IoT device is associated with a personal information command boundary for the IoT device, which ensures that the volume level of the response is low enough so that an individual outside the personal information command boundary cannot hear the audible response by the IVA on the IoT device. In another example, the instruction may include the IVA operating on the appropriate IoT device transmitting a text response to a client device and / or AR device associated with the management user. In yet another example, the instruction may include an IVA operating on a suitable IoT device sending a text response to a client device and / or AR device associated with the received audio, along with a command that can be executed by the IoT device.

[0049] Figure 4 shows a flowchart of a command boundary program that analyzes commands directed to an intelligent virtual assistant for personal information, according to one embodiment of the present invention.

[0050] The command boundary program 112 classifies commands that can be executed by IoT devices (402). The command boundary program 112 can identify and classify commands that can be executed by IoT devices using a user feedback-based iterative training model. The command boundary program 112 uses natural language processing to classify parts of a command as containing personal information or not containing personal information, and identifies parts of a command that contain personal information. In one example, the command boundary program 112 receives a command that says, "AB, please call my credit card company about my credit card ending in ~1234," and classifies the part "my credit card ending in ~1234" as containing personal information. In another example, the command boundary program 112 receives a command that says, "AB, what was the account number used for our last shipment from Company C?" and does not classify any part of the command as containing personal information. The command contains the word "account number," but does not contain any "number" within the command. The command boundary program 112 also classifies commands that can be executed by IoT devices as those that contain personal information or that produce an audible response containing personal information. From the aforementioned example in which the command boundary program 112 receives a command stating, "AB, please call my credit card company about my credit card ending in ~1234," the command boundary program 112 classifies this command as a command that contains personal information and can be executed by an IoT device. From the aforementioned example in which the command boundary program 112 receives a command stating, "AB, what was the account number used for our last shipment from Company C?", the command boundary program 112 classifies this command as a command that produces an audible response containing personal information.

[0051] The command boundary program 112 determines whether a command is related to personal information (decision 404). A command is related to personal information if at least a portion of the command is classified as having personal information, and if the command produces an audible response that contains personal information. If the command boundary program 112 determines that a command is related to personal information (branch "yes", decision 404), the command boundary program 112 queries the user to establish a personal information command boundary for commands that can be executed by IoT devices (406). If the command boundary program 112 determines that a command is not related to personal information (branch "no", decision 404), the command boundary program 112 proceeds to identify the distance of the voice source to each of the multiple IoT devices (Figure 3, 312).

[0052] The command boundary program 112 queries the user to establish a personal information command boundary for commands that can be executed by the IoT device (406). In this embodiment, the command boundary program 112 queries the administrative user to establish a personal information command boundary associated with a specific volume level when responding to a command related to personal information, the specific volume level being lower than the operating volume level previously established by the user. The command boundary program 112 receives personal information command boundaries from the user in a similar manner to how the command boundary program 112 establishes operating command boundaries for each of the multiple devices in (304) of Figure 3. The radius of the personal information command boundary is smaller than the radius of the operating command boundary for the IoT device. If the command boundary program 112 does not receive a personal information command boundary from the user, or if the command boundary program 112 determines that the received radius for the personal information command boundary exceeds a predetermined level, the command boundary program 112 sends an additional notification warning of a potential security issue associated with the radius of the personal information command boundary exceeding the predetermined level.

[0053] The command boundary program 112 stores personal information command boundaries for commands that can be executed by IoT devices (408). The command boundary program 112 stores personal information command boundaries for commands that can be executed by IoT devices and sends the personal information command boundary as an instruction along with the command to the appropriate IoT device. The appropriate IoT device utilizes the personal information command boundary if it provides an audible response to the command at a specific volume level.

[0054] Figure 5 shows a computer system 500, where the server computer 102 is an example of the computer system 500, including a command boundary program 112. The computer system includes a processor 504, a cache 516, memory 506, persistent storage 508, a communication unit 510, an input / output (I / O) interface 512, and a communication fabric 502. The communication fabric 502 provides communication between the cache 516, memory 506, persistent storage 508, the communication unit 510, and the input / output (I / O) interface 512. The communication fabric 502 can be implemented using any architecture designed to pass data and / or control information between processors (such as microprocessors, communication and network processors), system memory, peripheral devices, and any other hardware components in the system. For example, the communication fabric 502 can be implemented using one or more buses or crossbar switches.

[0055] Memory 506 and persistent storage 508 are computer-readable storage media. In this embodiment, memory 506 includes random access memory (RAM). Generally, memory 506 may include any suitable volatile or non-volatile computer-readable storage media. Cache 516 is a high-speed memory that enhances the performance of processor 504 by holding recently accessed data and data adjacent to recently accessed data from memory 506.

[0056] Program instructions and data used to implement embodiments of the present invention may be stored in persistent storage 508 and memory 506 for execution by one or more of the respective processors 504 via cache 516. In one embodiment, persistent storage 508 includes a magnetic hard disk drive. As an alternative to, or in addition to, a magnetic hard disk drive, persistent storage 508 may include a solid-state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0057] The media used by persistent storage 508 may also be removable. For example, a removable hard drive may be used for persistent storage 508. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into the drive for transfer onto another computer-readable storage medium, which is also part of persistent storage 508.

[0058] In these examples, the communication unit 510 provides communication with other data processing systems or devices. In these examples, the communication unit 510 includes one or more network interface cards. The communication unit 510 may provide communication through the use of either or both physical communication links and wireless communication links. Program instructions and data used to implement embodiments of the present invention may be downloaded to persistent storage 508 through the communication unit 510.

[0059] The I / O interface 512 enables data input and output with other devices that may be connected to each computer system. For example, the I / O interface 512 may provide connection to an external device 518 such as a keyboard, keypad, touchscreen and / or some other suitable input device. The external device 518 may also include portable computer-readable storage media such as a thumb drive, portable optical or magnetic disk and memory card. Software and data used to practice embodiments of the present invention may be stored on such portable computer-readable storage media and may be loaded onto persistent storage 508 via the I / O interface 512. The I / O interface 512 also connects to a display 520.

[0060] The display 520 provides a mechanism for displaying data to the user, and could be, for example, a computer monitor.

[0061] The programs described herein are identified based on the applications in which they are implemented in specific embodiments of the present invention. However, any particular program names used herein are for convenience only, and it should be understood that the present invention should not be limited to use only in any specific applications identified and / or suggested by such names.

[0062] The present invention may be a system, method, and / or computer program product at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or multiple mediums) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0063] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any preferred combination of those described above. A non-exclusive list of more specific examples of computer-readable storage media includes, namely, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved raised structures on which instructions are recorded, and any preferred combination of those described above. The computer-readable storage medium used herein should not be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0064] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers them for storage in a computer-readable storage medium within each computing / processing device.

[0065] The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® or C++, and procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may run as a standalone software package in whole on the user's computer, in part on the user's computer, in part on the user's computer and in part on a remote computer, or in whole on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized by executing computer-readable program instructions by utilizing state information of computer-readable program instructions in order to perform an aspect of the present invention.

[0066] Aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0067] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing device to generate a machine, thereby creating means for instructions executed via the processor of a computer or other programmable data processing device to implement functions / operations specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing device and / or other device to function in a particular manner, thereby comprising a product in which a computer-readable storage medium storing instructions therein includes instructions that implement modes of functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0068] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device to perform a series of operational steps on the computer, other programmable device, or other device, thereby generating a computer implementation process in which the instructions executed on the computer, other programmable device, or other device implement the functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0069] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may be performed in an order different from the order shown in the figures. For example, two consecutively shown blocks may actually be implemented as a single step, executed simultaneously, substantially simultaneously, partially or entirely, with overlapping timelines, or blocks may be executed in reverse order depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by application-specific hardware-based systems that perform a specified function or operation, or combinations of application-specific hardware instructions and computer instructions.

[0070] While this disclosure includes a detailed description of cloud computing, it should be understood that the implementations of the teachings enumerated herein are not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment currently known or to be developed in the future.

[0071] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and deployed with minimal management effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0072] The characteristics are as follows:

[0073] On-demand self-service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers.

[0074] Broad network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, and PDAs®).

[0075] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with various physical and virtual resources dynamically allocated and reallocated according to demand. Consumers generally do not have control or knowledge of the exact location of the resources provided, but there is location independence in that they may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0076] Rapid scalability: Capabilities are provisioned quickly and flexibly, sometimes automatically, allowing for instant scaling out or rapid release and instant scaling in. To consumers, the available capacity for provisioning often appears unlimited and can be purchased at any time in any quantity.

[0077] Measurement Services: Cloud systems automatically control and optimize resource usage by leveraging measurement capabilities appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts) at a certain level of abstraction. Resource usage may be monitored, controlled, and reported, thereby providing transparency to both service providers and consumers.

[0078] The service model is as follows:

[0079] Software as a Service (SaaS): The capability offered to consumers is the use of a provider's applications running on a cloud infrastructure. These applications are accessible from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with the conceivable exception of limited, user-specific application configuration settings.

[0080] Platform as a Service (PaaS): The capability offered to consumers is the ability to deploy applications they have created or acquired, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they do have control over the deployed applications and, in some cases, the configuration of the application hosting environment.

[0081] Infrastructure as a Service (IaaS): The ability provided to consumers is to provision processing, storage, networking, and other basic computing resources, allowing consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they have control over the operating system, storage, and deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0082] The deployment model is as follows:

[0083] Private Cloud: Cloud infrastructure is operated solely for an organization. It may be managed by that organization or a third party and may reside on-premises or off-premises.

[0084] Community Cloud: Cloud infrastructure is shared by multiple organizations to support a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). It may be managed by the organization or a third party and may reside on-premises or off-premises.

[0085] Public cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0086] Hybrid Cloud: This cloud infrastructure is a complex of two or more clouds (private, community, or public) that remain separate entities but are joined together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing across clouds).

[0087] Cloud computing environments are service-oriented, emphasizing statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing lies an infrastructure that includes a network of interconnected nodes.

[0088] Referring here to Figure 5, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an automotive computer system 54N, can communicate. The nodes 10 can communicate with each other. They may be grouped physically or virtually within one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above in this specification (not shown). This enables the cloud computing environment 50 to provide infrastructure, platforms, and / or software as a service to cloud consumers, without requiring them to maintain resources on their local computing devices. It should be understood that the types of computing devices 54A to 54N shown in Figure 5 are intended for illustrative purposes only, and that the computing node 10 and the cloud computing environment 50 may communicate with any type of computerized device (e.g., using a web browser) through any type of network and / or network addressable connection.

[0089] Referring now to Figure 6, a set of functional abstraction layers provided by the cloud computing environment 50 (Figure 5) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 6 are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As shown in the figure, the following layers and corresponding functions are provided:

[0090] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include a mainframe 61; a RISC (Reduced Instruction Set Computer) architecture-based server 62; a server 63; a blade server 64; a storage device 65; and network and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0091] The virtualization layer 70 provides an abstraction layer that may provide examples of virtual entities, namely, virtual servers 71, virtual storage 72, virtual networks 73 including virtual private networks, virtual applications and operating systems 74, and virtual clients 75.

[0092] In one example, the management layer 80 may provide the following functions: Resource provisioning 81 provides dynamic procurement of computing and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 82 provides cost tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification of cloud consumers and tasks, as well as protection of data and other resources. The user portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management to ensure that required service levels are met. Service level agreement (SLA) planning and execution 85 provides advance preparation and procurement of cloud computing resources where future requirements are anticipated in accordance with the SLA.

[0093] The workload layer 90 provides examples of functions that can be utilized in a cloud computing environment. Examples of workloads and functions that can be provided from this layer include mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis processing 94; transaction processing 95; and command boundary programs 112.

[0094] The programs described herein are identified based on the applications in which they are implemented in specific embodiments of the present invention. However, any particular program names used herein are for convenience only, and it should be understood that the present invention should not be limited to use only in any specific applications identified and / or suggested by such names.

[0095] The present invention may be a system, method, and / or computer program product at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or multiple mediums) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0096] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any preferred combination of those described above. A non-exclusive list of more specific examples of computer-readable storage media includes, namely, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved raised structures on which instructions are recorded, and any preferred combination of those described above. The computer-readable storage medium used herein should not be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0097] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers them for storage in a computer-readable storage medium within each computing / processing device.

[0098] The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® or C++, and procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may run as a standalone software package in whole on the user's computer, in part on the user's computer, in part on the user's computer and in part on a remote computer, or in whole on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized by executing computer-readable program instructions by utilizing state information of computer-readable program instructions in order to perform an aspect of the present invention.

[0099] Aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0100] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing device to generate a machine, thereby creating means for instructions executed via the processor of a computer or other programmable data processing device to implement functions / operations specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing device and / or other device to function in a particular manner, thereby comprising a product in which a computer-readable storage medium storing instructions therein includes instructions that implement modes of functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0101] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device to perform a series of operational steps on the computer, other programmable device, or other device, thereby generating a computer implementation process in which the instructions executed on the computer, other programmable device, or other device implement the functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0102] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in a block may be performed in an order different from that shown in the figure. For example, two consecutively shown blocks may actually be implemented as a single step, executed simultaneously, substantially simultaneously, partially or entirely, with overlapping timelines, or blocks may be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, may be implemented by an application-specific hardware-based system that performs a specified function or operation, or a combination of application-specific hardware instructions and computer instructions. 。 [Item 1] A step of establishing a first command boundary for a first intelligent virtual assistant operating on a first electronic device; A step of identifying a portion of the audio from the plurality of sources within the first command boundary in response to receiving audio from a plurality of sources within the vicinity of the first electronic device; and Steps to transmit the command to the first intelligent virtual assistant on the first electronic device in response to determining that the command is identifiable in the portion of the audio from the plurality of sources within the first command boundary. A computer implementation method comprising the above. [Item 2] A step of establishing a second command boundary for a second intelligent virtual assistant operating on a second electronic device, wherein the first command boundary does not overlap with the second command boundary. A computer implementation method as described in item 1, further comprising the features described above. [Item 3] Steps to establish a first set of command rules for the first intelligent virtual assistant operating on the first electronic device, and a second set of command rules for the second intelligent virtual assistant operating on the second electronic device. A computer implementation method as described in item 2, further comprising the above. [Item 4] The stage of analyzing the aforementioned commands regarding personal information; Based on the analysis step, in response to determining that the command relates to personal information, a step of transmitting an instruction to the first intelligent virtual assistant on the first electronic device based on the personal information command boundary for the command. A computer implementation method described in any of the above items 1 to 3, further comprising the above. [Item 5] The stage in analyzing the aforementioned commands regarding personal information is: A step of classifying the commands that can be executed by the first intelligent virtual assistant on the first electronic device; In response to determining that the command relates to personal information based on the classification, the user is queried to establish the personal information command boundary for the command, wherein the volume level response to the first intelligent virtual assistant on the first electronic device is associated with the personal information command boundary; and Steps to store the personal information command boundary for the aforementioned command. The computer implementation method described in item 4, further comprising the above. [Item 6] The step of classifying the commands that can be executed by the first intelligent virtual assistant on the first electronic device is: The step of classifying the command as relating to personal information, or the first part of the command as relating to personal information; and The step of classifying the command as containing personal information or causing an audible response containing personal information by the first intelligent virtual assistant on the first electronic device. The computer implementation methods described in item 5, further including the above. [Item 7] The first portion of the command is classified as containing personal information, and the command causes the first intelligent virtual assistant on the first electronic device to produce an audible response containing personal information, thus determining that the command is related to personal information. A computer implementation method as described in item 6, further comprising the features described therein. [Item 8] One or more computer-readable storage media, and program instructions collectively stored on the one or more computer-readable storage media. The stored program instructions are executable by one or more computer processors, and the stored program instructions are: Program instructions for establishing a first command boundary for a first intelligent virtual assistant operating on a first electronic device; Program instructions for identifying portions of the audio from multiple sources within the first command boundary in response to receiving audio from multiple sources within the vicinity of the first electronic device; and A program instruction for sending the command to the first intelligent virtual assistant on the first electronic device in response to determining that the command is identifiable in the portion of the audio from the plurality of sources within the first command boundary. A computer program product that has [certain characteristics]. [Item 9] The stored program instructions are: A program instruction for establishing a second command boundary for a second intelligent virtual assistant operating on a second electronic device, wherein the first command boundary does not overlap with the second command boundary. A computer program product as described in item 8, further comprising the above. [Item 10] The stored program instructions are: Program instructions for establishing a first set of command rules for the first intelligent virtual assistant operating on the first electronic device, and a second set of command rules for the second intelligent virtual assistant operating on the second electronic device. A computer program product as described in item 8, further comprising the above. [Item 11] The stored program instructions are: Program instructions for analyzing the aforementioned command regarding personal information; In response to determining, based on the analysis, that the command relates to personal information, a program instruction to send a command to the first intelligent virtual assistant on the first electronic device, based on the personal information command boundary for the command. A computer program product described in any of the preceding items 8 to 10, further comprising: [Item 12] The program instructions for analyzing the aforementioned command regarding personal information are: Program instructions for classifying the commands that can be executed by the first intelligent virtual assistant on the first electronic device; In response to determining that the command relates to personal information based on the classification, a program instruction for querying the user to establish the personal information command boundary for the command, wherein the volume level response to the first intelligent virtual assistant on the first electronic device is associated with the personal information command boundary; and Program instructions for storing the personal information command boundary for the aforementioned command The computer program products listed in item 11, further including the following. [Item 13] The program instructions for classifying the commands that can be executed by the first intelligent virtual assistant on the first electronic device are: Program instructions for classifying the aforementioned command as relating to personal information, or the first part of the aforementioned command as relating to personal information; and Program instructions for classifying the command as containing personal information or causing an audible response containing personal information from the first intelligent virtual assistant on the first electronic device. The computer program products listed in item 12, further including the following. [Item 14] The stored program instructions are: A program instruction for determining that a command is related to personal information, since the first portion of the command is classified as having personal information and the command produces the audible response containing personal information from the first intelligent virtual assistant on the first electronic device. A computer program product as described in item 13, further comprising the above. [Item 15] One or more computer processors; One or more computer-readable storage media; and Program instructions stored on the one or more computer-readable storage media to be executed by at least one of the one or more computer processors The program instructions are: Program instructions for establishing a first command boundary for a first intelligent virtual assistant operating on a first electronic device; Program instructions for identifying portions of the audio from multiple sources within the first command boundary in response to receiving audio from multiple sources within the vicinity of the first electronic device; and A program instruction for sending the command to the first intelligent virtual assistant on the first electronic device in response to determining that the command is identifiable in the portion of the audio from the plurality of sources within the first command boundary. A computer system having the following features. [Item 16] The stored program instructions are: A program instruction for establishing a second command boundary for a second intelligent virtual assistant operating on a second electronic device, wherein the first command boundary does not overlap with the second command boundary. A computer system as described in item 15, further comprising: [Item 17] The stored program instructions are: Program instructions for establishing a first set of command rules for the first intelligent virtual assistant operating on the first electronic device, and a second set of command rules for the second intelligent virtual assistant operating on the second electronic device. A computer system as described in item 15, further comprising: [Item 18] The stored program instructions are: Program instructions for analyzing the aforementioned command regarding personal information; In response to determining, based on the analysis, that the command relates to personal information, a program instruction to send a command to the first intelligent virtual assistant on the first electronic device, based on the personal information command boundary for the command. A computer system as described in any of the preceding items 15 to 17, further comprising: [Item 19] The program instructions for analyzing the command with respect to personal information, and the stored program instructions are: Program instructions for classifying the commands that can be executed by the first intelligent virtual assistant on the first electronic device; In response to determining that the command relates to personal information based on the classification, a program instruction for querying the user to establish the personal information command boundary for the command, wherein the volume level response to the first intelligent virtual assistant on the first electronic device is associated with the personal information command boundary; and Program instructions for storing the personal information command boundary for the aforementioned command A computer system as described in item 18, further comprising: [Item 20] A step of establishing a near-field radius for an intelligent virtual assistant operating on a first electronic device, where the near-field radius is a command boundary that separates near-field and far-field speech; A step of identifying a portion of the sound from the multiple sources within the near-field radius in response to receiving sound from multiple sources within the vicinity of the first electronic device; and The step of determining that the command is identifiable in the portion of the audio from the multiple sources within the near-field radius, and instructing the intelligent virtual assistant to execute the command. A computer implementation method comprising the above. [Item 21] A step of analyzing the portion of the sound from the plurality of sources within the near-field radius for the command that can be executed by the first electronic device. A computer implementation method as described in item 20, further comprising the features described therein. [Item 22] A step of displaying the command within the second electronic device in response to determining that the second electronic device is associated with the command; and In response to receiving a modification to the command from the second electronic device; the step of instructing the intelligent virtual assistant to execute the command with the modification. A computer implementation method as described in item 20 or 21, further comprising the above. [Item 23] A computer implementation method according to any of the preceding items 20 to 22, wherein the first indicator is assigned to the first subcommand of the command, and the second indicator is assigned to the second subcommand of the command. [Item 24] The step of receiving the modification to the command from the second electronic device, wherein the modification assigns the first indicator to the second subcommand of the command, and the second indicator to the first subcommand of the command. A computer implementation method as described in item 23, further comprising the above. [Item 25] The computer implementation method according to item 24, wherein the first indicator instructs the intelligent virtual assistant to execute the second subcommand of the command before the second indicator executes the first subcommand of the command.

Claims

1. A step of establishing a first command boundary for a first intelligent virtual assistant operating on a first electronic device; A step of identifying a first audio from the plurality of sources within the first command boundary among the received audio from the plurality of sources, based on the average decibel measurement of the audio, the frequency of the audio, the peak of the decibel measurement of the audio, or data received from an electronic device located near the first electronic device, in response to receiving audio from a plurality of sources within the vicinity of the first electronic device; and Steps to transmit the command to the first intelligent virtual assistant on the first electronic device in response to determining that the command is identifiable in the first audio from the plurality of sources within the first command boundary. A computer implementation method comprising the above.

2. The computer implementation method according to claim 1, wherein the step of establishing a first command boundary for a first intelligent virtual assistant operating on a first electronic device includes the step of establishing the first command boundary based on the result of a query to a user of the electronic device associated with the first electronic device, or the result of a voice capture test to determine whether at least one electronic device among a plurality of electronic devices including the first electronic device can capture an audible command.

3. The step of establishing a second command boundary for a second intelligent virtual assistant operating on a second electronic device, wherein the first command boundary does not overlap with the second command boundary. The computer implementation method according to claim 1, further comprising the following:

4. Steps to establish a first set of command rules for the first intelligent virtual assistant operating on the first electronic device, and a second set of command rules for the second intelligent virtual assistant operating on the second electronic device. The computer implementation method according to claim 3, further comprising the above.

5. A step of establishing a first command boundary for a first intelligent virtual assistant operating on a first electronic device; A step of identifying, in response to receiving sound from a plurality of sources in the vicinity of the first electronic device, a first sound from the plurality of sources within the first command boundary from among the sound received from the plurality of sources; A step of transmitting the command to the first intelligent virtual assistant on the first electronic device in response to determining that the command is identifiable in the first audio from the plurality of sources within the first command boundary; The stage of analyzing the aforementioned commands regarding personal information; Based on the analysis step, in response to determining that the command relates to personal information, a step of transmitting a command to the first intelligent virtual assistant on the first electronic device based on the personal information command boundary for the command. A computer implementation method comprising the above.

6. The stage in analyzing the aforementioned commands regarding personal information is: A step of classifying the commands that can be executed by the first intelligent virtual assistant on the first electronic device; In response to determining that the command relates to personal information based on the classification, the user is queried to establish the personal information command boundary for the command, wherein the volume level response to the first intelligent virtual assistant on the first electronic device is associated with the personal information command boundary; and Steps to store the personal information command boundary for the aforementioned command. The computer implementation method according to claim 5, further comprising the above.

7. The step of classifying the commands that can be executed by the first intelligent virtual assistant on the first electronic device is: The step of classifying the command as relating to personal information, or the first part of the command as relating to personal information; and The step of classifying the command as containing personal information or causing an audible response containing personal information by the first intelligent virtual assistant on the first electronic device. The computer implementation method according to claim 6, further comprising:

8. The first portion of the command is classified as containing personal information, and the command causes the first intelligent virtual assistant on the first electronic device to produce an audible response containing personal information, thus determining that the command is related to personal information. The computer implementation method according to claim 7, further comprising the following:

9. One or more computer processors, A procedure for establishing a first command boundary for a first intelligent virtual assistant operating on a first electronic device; A procedure for identifying a first audio from a plurality of sources within a first command boundary among the received audio from the plurality of sources, based on the average decibel measurement of the audio, the frequency of the audio, the peak of the decibel measurement of the audio, or data received from an electronic device located near the first electronic device, in response to receiving audio from a plurality of sources within the vicinity of the first electronic device; and A procedure for transmitting the command to the first intelligent virtual assistant on the first electronic device in response to determining that the command is identifiable in the first audio from the plurality of sources within the first command boundary. A computer program designed to execute something.

10. The one or more computer processors: A procedure for establishing a second command boundary for a second intelligent virtual assistant operating on a second electronic device, wherein the first command boundary does not overlap with the second command boundary. A computer program according to claim 9 for further execution of the above.

11. The one or more computer processors: A procedure for establishing a first set of command rules for the first intelligent virtual assistant operating on the first electronic device, and a second set of command rules for the second intelligent virtual assistant operating on the second electronic device. A computer program according to claim 10 for further execution of the above.

12. One or more computer processors, A procedure for establishing a first command boundary for a first intelligent virtual assistant operating on a first electronic device; A procedure for identifying, in response to receiving sound from a plurality of sources in the vicinity of the first electronic device, a first sound from the plurality of sources within the first command boundary from among the received sound from the plurality of sources; A procedure for transmitting the command to the first intelligent virtual assistant on the first electronic device in response to determining that the command is identifiable in the first audio from the plurality of sources within the first command boundary; Procedures for analyzing the aforementioned commands regarding personal information; and A procedure to transmit an instruction to the first intelligent virtual assistant on the first electronic device based on the personal information command boundary for the command, in response to determining that the command relates to personal information based on the analysis described above. A computer program designed to execute something.

13. The procedure for analyzing the aforementioned command regarding personal information is as follows: A procedure for classifying the commands that can be executed by the first intelligent virtual assistant on the first electronic device; A procedure to query the user to establish a personal information command boundary for the command in response to determining that the command relates to personal information based on the classification, wherein the volume level response to the first intelligent virtual assistant on the first electronic device is associated with the personal information command boundary; and Procedure for storing the personal information command boundary for the aforementioned command The computer program according to claim 12, further comprising:

14. The procedure for classifying the commands that can be executed by the first intelligent virtual assistant on the first electronic device is: A procedure for classifying the command as relating to personal information, or the first part of the command as relating to personal information; and A procedure for classifying the aforementioned command as containing personal information or causing an audible response containing personal information by the first intelligent virtual assistant on the first electronic device. The computer program according to claim 13, further comprising:

15. The one or more computer processors: A procedure for determining that a command is related to personal information, since the first portion of the command is classified as having personal information, and the command produces an audible response containing personal information from the first intelligent virtual assistant on the first electronic device. A computer program according to claim 14 for further execution of the above.

16. One or more computer processors; One or more computer-readable storage media; and Program instructions stored on the one or more computer-readable storage media to be executed by at least one of the one or more computer processors The program instructions are: Program instructions for establishing a first command boundary for a first intelligent virtual assistant operating on a first electronic device; A program instruction for identifying a first audio from the plurality of sources within the first command boundary among the received audio from the plurality of sources, based on the average decibel measurement of the audio, the frequency of the audio, the peak of the decibel measurement of the audio, or data received from an electronic device located near the first electronic device, in response to receiving audio from a plurality of sources within the vicinity of the first electronic device; and A program instruction for sending the command to the first intelligent virtual assistant on the first electronic device in response to determining that the command is identifiable in the first audio from the plurality of sources within the first command boundary. A computer system having the following features.

17. The stored program instructions are: A program instruction for establishing a second command boundary for a second intelligent virtual assistant operating on a second electronic device, wherein the first command boundary does not overlap with the second command boundary. The computer system according to claim 16, further comprising:

18. The stored program instructions are: Program instructions for establishing a first set of command rules for the first intelligent virtual assistant operating on the first electronic device, and a second set of command rules for the second intelligent virtual assistant operating on the second electronic device. The computer system according to claim 17, further comprising:

19. One or more computer processors; One or more computer-readable storage media; and Program instructions stored on the one or more computer-readable storage media to be executed by at least one of the one or more computer processors The program instructions are: Program instructions for establishing a first command boundary for a first intelligent virtual assistant operating on a first electronic device; A program instruction for identifying a first audio from a plurality of sources within a first command boundary among the received audio from the plurality of sources, in response to receiving audio from a plurality of sources in the vicinity of the first electronic device; A program instruction for sending the command to the first intelligent virtual assistant on the first electronic device in response to determining that the command is identifiable in the first audio from the plurality of sources within the first command boundary; Program instructions for analyzing the aforementioned command regarding personal information; In response to determining, based on the analysis, that the command relates to personal information, a program instruction to send a command to the first intelligent virtual assistant on the first electronic device, based on the personal information command boundary for the command. A computer system having the following features.

20. The program instructions for analyzing the aforementioned command regarding personal information are: Program instructions for classifying the commands that can be executed by the first intelligent virtual assistant on the first electronic device; In response to determining that the command relates to personal information based on the classification, a program instruction for querying the user to establish the personal information command boundary for the command, wherein the volume level response to the first intelligent virtual assistant on the first electronic device is associated with the personal information command boundary; and Program instructions for storing the personal information command boundary for the aforementioned command The computer system according to claim 19, further comprising:

21. A step of establishing a near-field radius for an intelligent virtual assistant operating on a first electronic device, where the near-field radius is a command boundary that separates near-field and far-field speech; A step of identifying, in response to receiving sound from multiple sources within the vicinity of the first electronic device, the near-field sound from within the near-field radius among the sound from the multiple sources that have been received; In response to determining that the command is identifiable in the near-field audio, the intelligent virtual assistant is instructed to execute the command; A step of displaying the command in the second electronic device in response to determining that the second electronic device is associated with the command; and Steps to instruct the intelligent virtual assistant to execute the command with the modifications in response to receiving a modification to the command from the second electronic device. A computer implementation method comprising the above.

22. The computer implementation method according to claim 21, wherein the first indicator is assigned to the first subcommand of the command, and the second indicator is assigned to the second subcommand of the command.

23. The step of receiving the modification to the command from the second electronic device, wherein the modification assigns the first indicator to the second subcommand of the command, and the second indicator to the first subcommand of the command. The computer implementation method according to claim 22, further comprising the above.

24. The computer implementation method according to claim 23, wherein the first indicator instructs the intelligent virtual assistant to execute the second subcommand of the command before the second indicator executes the first subcommand of the command.

Images