Augmented reality-based selective inclusion of ambient sounds while executing certain voice commands

Abstract

A method, system, and apparatus for generating augmented voice commands, comprising: identifying a plurality of sounds from a respective plurality of transducers to a smart speaker device; generating a visualization of the sounds using an augmented reality device, wherein one or more of the sounds can be selected using the visualization; and generating an augmented voice command for the smart speaker device, wherein the augmented voice command includes one or more sounds selected using the visualization of the augmented reality device.

Description

[Technical Field]

[0001] The present invention relates in one embodiment to a method, apparatus, and system for selectively including sounds, and more particularly, but not by way of limitation, to a method, apparatus, and system for selectively including sounds from the environment in an AR (Augmented Reality) based manner while executing some voice command. [Background technology]

[0002] AI (artificial intelligence) voice assistant systems such as AMAZON ALEXA, GOOGLE HOME, etc. are becoming increasingly popular. For example, the global voice assistant market is expected to reach unprecedented levels. One of the driving factors includes the increasing demand for improved customer experience, and another is the increasing use cases in the healthcare market (patient engagement).

[0003] In such a system, a user can send a voice command, and in response, the voice command is executed.

[0004] Multiple people can send voice commands to a single AI voice assistant system, which can recognize the voice and execute the user-specific voice commands.

[0005] However, sometimes a user sends a voice command, and other users around them give further suggestions or additional feedback to the first user's voice command. When this happens, the original user may not want to accept some of the other users' suggestions. Currently, AI voice assistant systems are unable to understand which voice commands to execute and which can be ignored. Summary of the Invention

[0006] Therefore, there is a need to have an AI voice assistant system that can handle more complex inputs, for example, a method and system that allows a user to selectively specify which spoken and unspoken content from their surroundings should be considered while the user is issuing a voice command.

[0007] In view of the foregoing and other problems, disadvantages, and drawbacks of the aforementioned background art, exemplary aspects of the disclosed invention provide methods, apparatus, and systems for AR-based selective inclusion of ambient sounds while executing certain voice commands.

[0008] According to one embodiment of the present invention, a method for generating an augmented voice command includes identifying a plurality of sounds from a respective plurality of transducers to a smart speaker device; generating a visualization of the sounds using an augmented reality device, where one or more of the sounds can be selected using the visualization; and generating an augmented voice command for the smart speaker device, where the augmented voice command includes one or more sounds selected using the visualization of the augmented reality device.

[0009] According to another embodiment of the present invention, a system for generating augmented voice commands includes a memory storing computer instructions and a processor, the processor being configured to execute the computer instructions to identify a plurality of sounds from a respective plurality of transducers to a smart speaker device, generate a visualization of the sounds using an augmented reality device, select one or more of the sounds using the visualization, and generate an augmented voice command for the smart speaker device, the augmented voice command including the one or more sounds selected using the visualization of the augmented reality device.

[0010] According to yet another embodiment of the present invention, a computer program product including a computer-readable storage medium having program instructions embodied thereon includes a computer program product including a computer-readable storage medium having program instructions embodied thereon, the computer-readable and executable program instructions causing a computer to perform a method including identifying a plurality of sounds from a respective plurality of transducers to a smart speaker device and generating a visualization of the sounds using an augmented reality device, wherein the visualization can be used to generate an augmented voice command for the smart speaker device to select one or more of the sounds, the augmented voice command including the one or more sounds selected using the visualization of the augmented reality device.

[0011] Certain embodiments of the invention have thus been outlined rather broadly in order that the detailed description of the invention herein may be better understood, and in order that the contribution to the art may be better appreciated. Of course, there are additional embodiments of the invention that will be described below and that will form the subject matter of the claims appended hereto.

[0012] It is to be understood that the invention is not limited in its application to the details of construction and the arrangement of components set forth in the following specification or illustrated in the drawings. The invention is capable of embodiments in addition to those described and can be practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein, as well as the abstract, are for the purpose of description and should not be regarded as limiting.

[0013] As such, those skilled in the art will appreciate that the conception upon which this disclosure is based may readily be utilized as a basis for the designing of other structures, methods and systems for carrying out the several purposes of the present invention. It is important, therefore, that the claims be regarded as including such equivalent constructions insofar as they do not depart from the scope of the present invention.

[0014] Exemplary aspects of the present invention will be better understood from the following detailed description of exemplary embodiments of the invention with reference to the drawings. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a configuration diagram of an embodiment of the present invention; [Figure 2] 1 is a diagram illustrating a system according to an embodiment of the present invention. [Figure 3] FIG. 1 is a process flow / workflow diagram of one embodiment of the present invention. [Figure 4] FIG. 4 illustrates command validation by the system of FIG. 3. [Figure 5] FIG. 4 illustrates the detection of sound location by the system of FIG. 3. [Figure 6] FIG. 4 is a diagram showing display commands on the AR glasses of FIG. 3. [Figure 7] 4A and 4B are diagrams illustrating selection commands on the AR glasses in one embodiment of the present invention in FIG. 3. [Figure 8] FIG. 4 illustrates the AI ​​voice assistant executing the command from the AR selection in FIG. 3. [Figure 9] Figure 3 shows the feedback loop to the historical corpus. [Figure 10] FIG. 1 shows a high-level diagram summary of one embodiment of the present invention. [Figure 11] FIG. 1 illustrates an exemplary hardware / information handling system for incorporating exemplary embodiments of the present invention therein. [Figure 12] 1 illustrates a signal-bearing storage medium for storing machine-readable instructions of a program implementing a method according to an exemplary embodiment of the present invention. [Figure 13] FIG. 1 illustrates a cloud computing node according to an exemplary embodiment of the present invention. [Figure 14] FIG. 1 illustrates a cloud computing environment in accordance with an exemplary embodiment of the present invention. [Figure 15] FIG. 1 illustrates abstraction model layers in accordance with an exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] The present invention will now be described with reference to the drawings. Like reference numerals refer to like parts throughout. It is emphasized that, according to common practice, the various features of the drawings are not necessarily drawn to scale. Rather, dimensions of the various features may be arbitrarily increased or decreased for clarity. Example embodiments are provided below for illustrative purposes and do not limit the scope of the claims. Also, note that any of the steps may be performed in different orders or may be combined or simultaneous. Additionally, any of the illustrated structures and embodiments may be modified or combined.

[0017] As described above, when a user sends a voice command, other users in the vicinity may provide further suggestions or additional feedback to the first user's voice command. When this occurs, the original user may not want to accept some of the other users' voice command suggestions. Currently, AI voice assistant systems are unable to understand which voice commands to execute and which can be ignored. The present invention provides a solution as follows:

[0018] FIG. 1 shows a configuration diagram of one embodiment of the present invention.

[0019] The present invention provides a method and system that allows a user 10 to selectively specify which spoken and non-spoken content from their surroundings should be considered while they are issuing voice commands.

[0020] While the user 10 is sending a voice command, people around them (users 12, 14, 16) are simultaneously providing additional suggestions to the voice command, so the user 10 can selectively include one or more sounds from their surroundings.

[0021] For example, user B12 gives a voice prompt for "milk," user C14 gives a voice prompt for "apple," and user D16 gives a voice prompt for "toy." Meanwhile, main user A10 sends a voice command such as "Please order rice A, biscuit B, and a cold drink to system 100."

[0022] An AI (artificial intelligence) voice assistant system 30 receives a voice command as "Please order rice, biscuits, a cold drink, milk, and an apple." The user, by using the AR glasses, sees the voice command and other sounds in the AR surroundings and selects "milk" 22 and "apple" 24, while not selecting "toy" 26 in the AR selection 18, and also selects the command "Please order rice, biscuits, and a cold drink" 20, giving a final output of "Please order rice, biscuits, a cold drink with milk, and an apple." The selection of toy 26 is cancelled.

[0023] 2 illustrates an exemplary system of one embodiment of the present invention. User A 10, user B 12, user C 14, and user D 16 can communicate with system 100 via paired devices, including an AR system 40, an AI assistant 30, and an Internet of Things (IoT) device 50, which are described further below. System 100 can include a system computer 102 including a processor 64 and memory 66. A historical corpus 60 can be included in memory 66, or the historical corpus can be located separately in memory device 62. Additionally, AR system 40 includes a processor 42 along with memory 44, while AI assistant 30 includes a processor 32 along with memory 34. Multiple IoT devices 50 can be paired with system 100. AR system 40 can include augmented reality glasses 46 (or can be a separate IoT device 50). IoT device 50 can include a processor 52 along with memory 54 and a transducer 56, such as a microphone and speaker. IoT device 50 may be multiple devices, each also including a display 58 and multiple sensors 57. Sensors 57 may detect biometric data or other information, such as in a smartwatch or other device.

[0024] Alternative embodiments may include any one or all of the following systems of the AR system 40 and the AI ​​assistant 30 being incorporated into the system computer 102. Additionally, the AR glasses 46 may be a separate IoT device 50 that is paired with the AR system 40.

[0025] Figure 3 illustrates a process flow / workflow diagram of one embodiment of the present invention. Referring to Figures 1 and 2, in a first section 110 for device pairing and user identification, a primary user (e.g., user A10 in Figures 1 and 2) is selected and identified by the module 112. User 10 is selected by the present invention, and historical corpora 60, 62 of user 10's speech are initiated. The corpora 60, 62 at 130 collect, via path 134, the user's command intonation, the user's command frequency, and contextual analysis of the user's commands from the system 100.

[0026] Then, at 114, the system 100 is paired with an AR (Augmented Reality) system 40, an AI (Artificial Intelligence) assistant 30, and an IoT (Internet of Things) device 50. The AI ​​voice assistant system 30 is paired with augmented reality glasses 46. The AI ​​voice assistant system 30 recognizes the user 10 based on their voiceprint, and the augmented reality system 40 recognizes the user based on retinal scan IoT authentication.

[0027] If the user 10, 12, 14, 16 is wearing another IoT device 50, such as a smartwatch, biometrics collected from the user can identify the user 10, 12, 14, 16 based on proximity to the AR system 40, the AI ​​voice assistant 30, and other biometric data.

[0028] Pairing can be done via any type of communication protocol, such as BLUETOOTH, Wi-Fi, a proprietary protocol, etc.

[0029] The primary user 10 then speaks a command, followed by the surrounding members 116. If any activation command is sent, the AI ​​voice assistant system 30 then wakes up and receives the voice command.

[0030] The user 10 is sending a voice command, whereupon the AI ​​voice assistant system 30 remains awake and receives the voice command.

[0031] Activation of the AR glasses 46 is as follows: When the AI ​​voice assistant 30 is awakened, it triggers the augmented reality glasses 46 to "wake up" and listen for other sounds (if the AR glasses 46 contain a microphone 48).

[0032] The system 100 then confirms the voice command and captures the exact command along with its location in space 118. While the user is sending the voice command, the AI ​​voice assistant system 30 then captures other sounds being generated in the surroundings.

[0033] FIG. 4 shows the validation of a command by the system of FIG.

[0034] 4, 3, and 2, command confirmation 118a is as follows: The AR system 40 (if it has a microphone 48) also receives sounds from the surroundings and confirms secondary commands from the surroundings. This can be done by analyzing the difference in sound waves and making sure they are within the same threshold.

[0035] Using the Fourier transform 154, the audio files can be scaled to identify similar files as follows:

[0036]

number

[0037] If the scaled equation is within a certain range ε>0, then the two sound waves are corroborated156.

[0038] Further corroboration can be done with text analysis 158. Commands registered by the AI ​​assistant system 30 are compared with commands from the augmented reality system 40. This can be done, for example, by comparing the words detected by each system 30, 40 using a Bag of Words (BOW) algorithm to detect command similarities.

[0039] The authentication of the IoT 50 is as follows: Other IoT devices 50 connected to the system 100 can verify the information.

[0040] Figure 5 shows the detection of the sound location by the system 100 at time 118 in Figure 3. With reference to Figures 5, 2 and 3, the detection of the sound location 118b is as follows: The AR glasses 46 also recognize the sound and who is making the sound (e.g., users 10, 12, 14, 16, etc.).

[0041] Identification may occur through social media integration 204. Users (e.g., users 10, 12, 14, and 16) may be identified as such if they are identified as "friends" on a social media platform or by the user's contact list.

[0042] Human object detection 206 in the space may be performed by the AR system 40 to identify the person tagging the sound (e.g., user 10, 12, 14, 16, etc.).

[0043] The sound direction 208 may be approximated by the direction of the sound wave containing the maximum amplitude.

[0044] IoT corroboration 210 is as follows: other microphone-enhanced paired IoT devices 50 can help triangulate the location of the command. IoT devices 50 may include those present in social media integration 204 (i.e., "friend" devices on the same network can be leveraged to determine location).

[0045] The combination of these parameters 204, 206, 208, and 210 localizes the person who said what was commanded.

[0046] However, referring back to Figure 3, in the second section 122, non-commands are ignored 120. Ignoring any sounds includes ignoring sounds that are not detected as being commanded, which may include, but are not limited to, faint noises, or other types of noises, etc.

[0047] The commands are then displayed on the AR screen in the AR system 40 and the primary user 10 selects 124 the final command.

[0048] Figure 6 shows a display 124a of a command on the AR glasses of Figure 3. The detected command is displayed on the AR glasses 46.

[0049] Which object issued a particular command or sound is displayed 220 on the AR glasses 46 .

[0050] The AR glasses 46 indicate the actual location and direction of the ambient sound along with the voice command within the AR surroundings 222 .

[0051] While the voice command is being sent, the AR glasses show real-time sounds generated from the surroundings and are displayed within the AR glasses.224 The user can also visualize their voice command being converted into text within the AR glasses, as well as the sounds generated from the surroundings.226

[0052] FIG. 7 illustrates a selection command for the AR glasses 124b in one embodiment of the present invention of FIG.

[0053] The user 10 can use finger gestures, eye contact, voice commands, or facial expressions along with any of the ambient sounds to allow those sounds to be included in the voice command. Based on the user's selection action, one or more sounds from the surroundings are considered and allowed to be added 230 along with the original voice command.

[0054] The user 10 can selectively select a sound from the sounds and add that sound accordingly 232 with the voice command sent by the user 10 .

[0055] An optional sound may be included with the voice command, which may be included while the voice command is being given by the user or after the voice command is given 234.

[0056] Selection of a sound portion 236 is as follows: While selecting one or more sounds from the surroundings in a selective manner, the user can select a sound portion from any sound source within the surroundings and add that sound accordingly to the original voice command 236.

[0057] Selection 238 based on location instead of text is as follows: While selecting some sound from the surroundings, the user 10 can select 238 the location of the original voice command where the sound will be added along with the voice command.

[0058] Figure 8 shows the AI ​​voice assistant executing the command from the AR selection in Figure 3.

[0059] 2, 3, and 8, the AI ​​voice assistant then executes 126 the command from the AR selection. Final voice command compilation and execution 242 is as follows: the proposed system 100 compiles the audio selected from the surroundings and combines it with the original voice command sent 244 by the user 10. The AR display of the AR system 40 shows the final voice command to the user 10, or the AI ​​assistant 30 states 246 the final voice command for execution. The system 100 then executes 250 the voice command after verification 248.

[0060] Figure 9 shows the feedback loop for the historical corpus of Figure 3.

[0061] 2, 3, and 9, the scenario is then captured and fed back into the historical corpus 132. The feedback loop to the historical corpus is as follows: Historically, the AI ​​voice assistant system 30 and AR glasses 46 have learned how ambient sounds are contextually associated with the voice commands sent and have identified which sounds are appropriate and can be added.

[0062] While sending the voice command, if any sound or sounds are generated from the surroundings, the proposed system 100 will determine which sounds should be included along with the voice command 252 .

[0063] Based on historical learning, the proposed system shows a visual diagram showing where the generated sounds will be included in the voice command, so the user has an understanding of where the sounds will be added 254.

[0064] Social media interaction 256 is as follows: if the secondary user 12, 14, or 16 says many useful command add-ons, they are more likely to be added when the primary user gives a voice command, and vice versa (if the secondary user adds many useless commands, they will not be added to future voice commands) 256.

[0065] Referring back to FIG. 3, the historical corpus 130 is built based on user input from paths 134 and user interactions with the system 100, as well as captured scenarios 132.

[0066] Additional embodiments may include a family member mode, where family members' voices are recognized and have a low threshold for passing voice commands. Alternatively, the voice assistant 30 may also be built into the AR system 40.

[0067] Referring back to Figure 2, a manufacturing use case might be as follows: Someone working in a manufacturing plant and equipped with AR system 40 is given various commands about what work needs to be done. Those commands appear on the screen, and the user can choose which command is needed next and which is just background noise. A module in AR system 40 compiles those commands and provides them as the final command for user 10.

[0068] FIG. 10 shows a high-level diagrammatic summary of one embodiment of the present invention.

[0069] Some of the features of the present invention are summarized as follows: First, in step 302, the system 100 converts a voice command to the AR system 40. While sending any voice command, the AI ​​voice assistant system 40 recognizes individual sounds based on the voiceprint. Thus, the AI ​​voice assistant system 40 identifies which commands come from the users 10, 12, 14, 16 and which are generated from the environment. Furthermore, the sent voice command is displayed on the augmented reality glasses 46 to selectively include one or more sounds in the original voice command.

[0070] The system 100 then selects an external command within the AR space 304. During or after sending the voice command, the user 10 can selectively include one or more spoken contents visualized in the augmented reality system 40, which are therefore taken into account and acted upon within the voice command.

[0071] The system 100 then performs execution of the voice command within the AR space 306. During any voice command sent by any user 10, 12, 14, 16, the augmented reality glasses 46 show real-time sound generation from the surroundings, so that based on the interaction of fingers, facial expressions, and eye contact, the user 10 can selectively include one or more sounds in the voice command, and accordingly, the AI ​​voice assistant system 40 executes the complete voice command.

[0072] The user 10 can select complete or partial spoken content from any audio source while sending any voice command, and thus the selected portion of the sound will be included with the voice command and the complete voice command will be executed.While selectively including one or more sounds from the environment with the voice command, the user can select a position on the voice command where an external sound can be selectively included to complete the voice command.

[0073] Therefore, the system 100 provides a historical corpus for selecting a command 308. Using historical learning about the selective content of one or more sounds along with the voice command being sent, the proposed system 100 recommends which sounds should be included along with the voice command and also shows a visual diagram to recommend where the sounds should be added along with the original voice command.

[0074] Figures 11 through 15 below provide additional hardware configurations that can be implemented. Different features shown in different figures of Figures 1 through 15 may be combined, changed, or switched between different examples.

[0075] FIG. 11 shows another hardware configuration of the system 100, which includes an information handling / computer system 1100 according to the present invention, preferably having at least one processor or central processing unit (CPU) 1110 capable of implementing the techniques of the present invention.

[0076] The CPU 1110 is interconnected via a system bus 1112 to random access memory (RAM) 1114, read-only memory (ROM) 1116, an input / output (I / O) adapter 1118 (for connecting peripheral devices such as a disk unit 1121 and a tape drive 1140 to the bus 1112), a user interface adapter 1122 (for connecting a keyboard 1124, a mouse 1126, speakers 1128, a microphone 1132, or other user interface devices, or a combination thereof, to the bus 1112), a communications adapter 1134 for connecting the information handling system to a data processing network, the Internet, an intranet, a personal area network (PAN), etc., and a display adapter 1136 for connecting the bus 1112 to a display device 1138, a printer 1139, or both (e.g., a digital printer, etc.).

[0077] In addition to the hardware / software environments described above, different aspects of the present invention include computer-implemented methods for executing the methods described above. As an example, the methods may be implemented in the specific environments described above.

[0078] Such methods may be implemented, for example, by operating a computer, such as embodied by a digital data processing device, to execute a series of machine-readable instructions, which may be in various types of signal-bearing media.

[0079] This aspect of the invention is therefore directed to a programmed product including a signal-bearing storage medium tangibly embodying a program of machine-readable instructions executable by a digital data processor incorporating CPU 1110 and the above-described hardware to perform the method of the invention.

[0080] This signal-bearing storage medium may include, for example, RAM housed within CPU 1110, as represented by fast-access storage.

[0081] Alternatively, the instructions may be contained within another signal-bearing storage medium 1200, such as a magnetic data storage diskette 1210, or an optical storage diskette 1220 (FIG. 12), that is directly or indirectly accessible by the CPU 1210.

[0082] The instructions may be stored on a variety of machine-readable data storage media, whether contained on a diskette 1210, an optical disk 1220, the computer / CPU 1210, or elsewhere.

[0083] Thus, the present invention may be a system, a method, and / or a computer program product, which may include a computer-readable storage medium having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0084] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge-in-groove structures that store instructions, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a signal that is itself ephemeral, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0085] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium within the respective computing / processing device for storage.

[0086] Computer-readable program instructions for carrying out the operations of the present invention may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk or C++, conventional procedural programming languages ​​such as the "C" programming language, or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0087] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0088] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to create a machine such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, generate means for performing the specified functions / acts in one or more blocks of the flowcharts and / or block diagrams.

[0089] These computer-readable program instructions, which may direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, may also be stored on a computer-readable storage medium such that the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture containing instructions that implement particular functional / actual aspects of one or more blocks of the flowcharts and / or block diagrams.

[0090] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions, executing on the computer, other programmable apparatus, or other device, implement the specified functions / acts in one or more blocks of the flowcharts and / or block diagrams.

[0091] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a particular logical function. In some alternative implementations, the illustrated functions may occur out of the order shown in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, depending on the functionality involved, or in some cases, the blocks may be executed in the reverse order. It will also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that implements a particular function or function or executes a combination of dedicated hardware and computer instructions.

[0092] 13, a schematic diagram 1400 of an example cloud computing node is shown. Cloud computing node 1400 is merely one example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the invention described herein. Nevertheless, cloud computing node 1400 may implement and / or perform any of the functionality described above.

[0093] Cloud computing node 1400 includes computer system / server 1412 that is operable with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system / server 1412 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0094] Computer system / server 1412 may be described in the general context of computer system-executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system / server 1412 may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0095] 13, computer system / server 1412 in cloud computing node 1400 is shown in the form of a general-purpose computing device. Components of computer system / server 1412 may include, but are not limited to, one or more processors or processing units 1416, a system memory 1428, and a bus 1418 that couples various system components including the system memory 1428 to the processor 1416.

[0096] Bus 1418 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0097] Typically, computer system / server 1412 includes a variety of computer system-readable media. Such media can be any available media that can be accessed by computer system / server 1412 and includes both volatile and nonvolatile media, removable and non-removable media.

[0098] System memory 1428 may include computer-readable media in the form of volatile memory, such as random access memory (RAM) 1430 and / or cache memory 1432. Computer system / server 1412 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 1434 may be provided to read from and write to non-removable, non-volatile magnetic media (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided to read from and write to removable, non-volatile magnetic disks (e.g., "floppy disks"), and an optical disk drive may be provided to read from and write to removable, non-volatile optical disks, such as CD-ROMs, DVD-ROMs, or other optical media. In such an example, each may be connected to bus 1418 by one or more data media interfaces. As further illustrated and described below, memory 1428 may include at least one program product having a set (e.g., at least one) of program modules configured to perform functions of embodiments of the present invention.

[0099] By way of example and not limitation, a program / utility 1440 having a set (at least one) of program modules 1442 may be stored in memory 1428, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include implementing a networking environment. Generally, the program modules 1442 implement the functionality and / or methodology of embodiments of the present invention as described herein.

[0100] The computer system / server 1412 may also communicate with one or more external devices 1414, such as a keyboard, pointing device, display 1424, etc., one or more devices that allow a user to interact with the computer system / server 1412, or any device (e.g., a network card, modem, etc.) that allows the computer system / server 1412 to communicate with one or more other computing devices, or a combination thereof. Such communication may occur via an input / output (I / O) interface 1422. Furthermore, the computer system / server 1412 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 1420. As shown, the network adapter 1420 communicates with other components of the computer system / server 1412 via a bus 1418. It should be understood that other hardware and / or software components, not shown, may be used with the computer system / server 1412. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archiving storage systems.

[0101] Referring now to FIG. 14, an exemplary cloud computing environment 1550 is illustrated. As shown, the cloud computing environment 1550 includes one or more cloud computing nodes 1400 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 1554A, a desktop computer 1554B, a laptop computer 1554C, or an automotive computer system 1554N, or combinations thereof, can communicate. The nodes 1400 can communicate with each other. The nodes 1400 may be physically or virtually grouped together in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, as described hereinabove (not shown). This enables the cloud computing environment 1550 to provide infrastructure, platform, and / or software that does not require cloud consumers to maintain resources on their local computing devices. The types of computing devices 1554A-N shown in FIG. 14 are intended to be illustrative only, and it is understood that computing node 1400 and cloud computing environment 1550 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0102] Referring now to Figure 15, a set of functional abstraction layers provided by cloud computing environment 1550 (Figure 14) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 15 are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0103] The hardware and software layer 1660 includes hardware and software components. Examples of hardware components include mainframes (e.g., IBM® zSeries® systems), RISC (reduced instruction set computer) architecture-based servers (e.g., IBM® pSeries® systems), IBM® xSeries® systems, IBM® BladeCenter® systems, storage devices, networks, and networking components. Examples of software components include network application server software (e.g., IBM® WebSphere® application server software) and database software (e.g., IBM® DB2® database software). (IBM®, zSeries, pSeries, xSeries, BladeCenter, WebSphere, and DB2 are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide.)

[0104] The virtualization layer 1662 provides an abstraction layer from which the following examples of virtual entities may result: virtual servers, virtual storage, virtual networks including virtual private networks, virtual applications and operating systems, and virtual clients.

[0105] In one example, the management layer 1664 can provide the functions described below. Resource provisioning provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing provides cost management as resources are utilized within the cloud computing environment, as well as billing and invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. A user portal provides access to the cloud computing environment for consumers and system administrators. Service level management provides cloud computing resource allocation and management so that required service levels are met. Service level agreement (SLA) planning and fulfillment provides advance arrangement and procurement of cloud computing resources in anticipation of future requirements according to SLAs.

[0106] The workload layer 1666 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include mapping and navigation, software development and lifecycle management, virtual classroom instruction delivery, data analytics processing, transaction processing, and, more particularly to the present invention, functions such as APIs and runtime system components for generating search autocomplete suggestions based on contextual input.

[0107] The many features and advantages of the present invention are apparent from the detailed description, and thus, it is intended by the appended claims to cover all such features and advantages of the invention that are within the true scope of the invention. Further, because numerous modifications and variations will readily occur to those skilled in the art, it is not desired to limit the invention to the exact construction and operation as illustrated and described, and therefore, all suitable modifications and equivalents may be resorted to that are within the scope of the invention.

[0108] It is to be understood that the invention is not limited in its application to the details of construction and the arrangement of components set forth in the following specification or illustrated in the drawings. The invention is capable of embodiments in addition to those described and can be practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein, as well as the abstract, are for the purpose of description and should not be regarded as limiting. As such, those skilled in the art will appreciate that the conception upon which this disclosure is based may readily be utilized as a basis for the designing of other structures, methods and systems for carrying out the several purposes of the present invention. It is important, therefore, that the claims be regarded as including such equivalent constructions insofar as they do not depart from the scope of the present invention.

Claims

1. 1. A method for generating an augmented voice command, comprising: Identifying the plurality of sounds from each of the plurality of transducers to the smart device; an augmented reality device may be used to generate a visualization of the sounds, and the visualization may be used to select one or more of the sounds; generating the enhanced voice commands for the smart device; wherein the augmented voice command comprises the one or more sounds selected using the visualization of the augmented reality device.

2. The method of claim 1 , further comprising automatically selecting the one or more sounds based on a history of selection and deselection of historical sounds from one or more of the transducers.

3. each of the plurality of transducers includes a speaker, and the smart device includes a smart speaker device; The method of claim 1 , wherein the sounds include spoken or non-spoken content selected within an augmented reality space.

4. The method of claim 1 , further comprising translating the voice command into the generation of the visualization of the sound using an augmented reality device.

5. The method of claim 1 , further comprising: executing the augmented voice command by selection of one or more sounds by an augmented reality system.

6. during or after sending a voice command, selecting one or more spoken contents to be visualized on the augmented reality device and to be taken into account in the augmented voice command for execution accordingly; selecting the one or more sounds based on a history of selection and deselection of historical sounds from one or more of the transducers stored in a historical corpus; The method of claim 1 further comprising:

7. The method of claim 1 , wherein the method is implemented within a cloud infrastructure.

8. 1. A system for generating augmented voice commands, comprising: a memory for storing computer instructions; a processor, the processor comprising: Identifying the plurality of sounds from each of the plurality of transducers to the smart device; an augmented reality device may be used to generate a visualization of the sounds, and the visualization may be used to select one or more of the sounds; generating the augmented voice command for the smart device, the augmented voice command including the one or more sounds selected using the visualization of the augmented reality device; A system configured to execute the computer instructions.

9. The system of claim 8 , further comprising automatically selecting the one or more sounds based on a history of selection and deselection of historical sounds from one or more of the transducers.

10. each of the plurality of transducers includes a speaker, and the smart device includes a smart speaker device; The system of claim 8 , wherein the sounds include spoken or non-spoken content selected within an augmented reality space.

11. The system of claim 8 , further comprising translating the voice command into the generation of the visualization of the sound using an augmented reality device.

12. executing the augmented voice command by selecting one or more sounds by an augmented reality system; during or after sending a voice command, selecting one or more spoken contents to be visualized on the augmented reality device and to be taken into account in the augmented voice command for execution accordingly; selecting the one or more sounds based on a history of selection and deselection of historical sounds from one or more of the transducers stored in a historical corpus; The system of claim 8 further comprising:

13. A computer program product that causes a computer to carry out the method according to any one of claims 1 to 7.

Images