Resolving conflicting commands using hierarchy

Abstract

To provide a method for controlling devices that use a hierarchy to resolve conflicting commands, and a voice control device.SOLUTION: An IoT device receives a first command from a first user, receives a second command from a second user, and determines, by a first trained machine learning model, that the second command conflicts with the first command. In response to determining that the second command conflicts with the first command, the IoT device determines, by a second trained machine learning model, a physical world hierarchy between the first user and the second user, determines a response to the second command using the physical world hierarchy, and executes the response.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to computer security, and more particularly, to the resolution of conflicting commands using hierarchies.

Background Art

[0002] The development of the EDVAC system in 1948 is often cited as the beginning of the computer age. Since then, computer systems have evolved into extremely complex devices. Today's computer systems typically include combinations of sophisticated hardware and software components, application programs, operating systems, processors, buses, memories, input / output devices, and the like. As advancements in semiconductor processing and computer architecture have pushed performance higher, increasingly advanced computer software has evolved to take advantage of the higher performance benefits of those capabilities, resulting in today's computer systems that are far more powerful than just a few years ago.

[0003] One application of this new capability is the "Internet of Things," or IoT. Generally, IoT refers to an ad-hoc network of IoT devices, such as devices, vehicles, signs, and other objects, that have network connectivity in addition to incorporating electronics, software, sensors, actuators, or combinations thereof. Network connectivity can enable these objects to collect and exchange data with other IoT devices, computer systems, or combinations thereof. IoT allows objects to be sensed or remotely controlled via existing network infrastructure. This creates an opportunity to integrate the physical world more directly with computer-based systems, resulting in reduced human intervention, improved efficiency and accuracy, and increased economic benefits. When IoT is enhanced with sensors and actuators, this technology becomes an example of a more general class of cyber-physical systems, encompassing technologies such as smart grids, virtual power plants, smart homes, intelligent transportation, and smart cities. Each object in such a system may be uniquely identifiable via an embedded computing system and interoperable within existing internet infrastructure. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] A part of this disclosure relates to computer security, and a more specific part relates to resolving conflicting commands using a hierarchy. [Means for solving the problem]

[0005] Embodiments of the present disclosure provide a method for controlling a device, comprising the steps of: the device receiving a first command from a first user; the device receiving a second command from a second user; and a first trained machine learning model determining whether the second command conflicts with the first command. Depending on the determination that the second command conflicts with the first command, some embodiments may use a second trained machine learning model to determine the hierarchy of the physical world between the first user and the second user, and use the hierarchy of the physical world to determine a response to the second command. The method may further comprise the step of performing the response. Some embodiments may further comprise the step of using the second trained machine learning model to identify a higher-level user selected from a group consisting of the first user and the second user.

[0006] Embodiments of the present disclosure present a voice control device comprising a processing unit and a memory coupled to the processing unit. The memory may include program instructions executable by the processing unit for causing the processing unit to receive a first command from a first user, a second command from a second user, and for causing a first trained machine learning model to determine whether the second command conflicts with the first command. The memory may further include program instructions for causing a second trained machine learning model to determine the hierarchy of the physical world between the first user and the second user, and to use the hierarchy of the physical world to determine a response to the second command, in response to the determination that the second command conflicts with the first command. The memory may further include program instructions for causing the response to be executed and for causing a second trained machine learning model to identify a higher-ranking user selected from the group consisting of the first user and the second user.

[0007] Embodiments of the present disclosure provide a computer program product for a voice-controlled device, comprising a computer-readable storage medium having program instructions embodied therein. The program instructions may be executable by the processor to cause the processor to receive a first command from a first user, a second command from a second user, and to cause a first trained machine learning model to determine whether the second command conflicts with the first command. The program instructions may further cause the processor, in response to determining whether the second command conflicts with the first command, to cause a second trained machine learning model to determine the hierarchy of the physical world between the first user and the second user, and to use the hierarchy of the physical world to determine a response to the second command. The program instructions may further cause the processor to execute the response. The program instructions may further cause the processor, by the second trained machine learning model, to identify a higher-level user selected from the group consisting of the first user and the second user.

[0008] The above summary is not intended to describe each of the exemplary embodiments or all implementations of this disclosure. [Brief explanation of the drawing]

[0009] The drawings included in this application are incorporated herein and form part of this specification. They illustrate embodiments of the disclosure and, together with the description, help to illustrate the principles of the disclosure. The drawings illustrate only specific embodiments and do not limit the disclosure.

[0010] [Figure 1] The following are representative key components of a computer system that may be used in accordance with embodiments of this disclosure.

[0011] [Figure 2]This disclosure illustrates a cloud computing environment according to an embodiment of this disclosure.

[0012] [Figure 3] An abstract model layer according to an embodiment of this disclosure is shown.

[0013] [Figure 4] This document presents an exemplary embodiment of a voice command IoT system that conforms to several embodiments.

[0014] [Figure 5A] This flowchart shows one method for operating a voice command IoT system, conforming to several embodiments. [Figure 5B] This flowchart shows one method for operating a voice command IoT system, conforming to several embodiments. [Figure 5C] This flowchart shows one method for operating a voice command IoT system, conforming to several embodiments.

[0015] [Figure 6] This flowchart shows one method for performing several actions using a trained machine learning model, tailored to several embodiments.

[0016] [Figure 7] This flowchart shows one method for training a machine learning model, tailored to several embodiments.

[0017] [Figure 8] This flowchart shows one method for updating a machine learning model, tailored to several embodiments.

[0018] The present disclosure can be modified into various modified forms or alternative forms, the specific details of which are shown as examples in the drawings and will be described in detail. However, it should be understood that the intention is not to limit the present invention to the specific embodiments described. On the contrary, it is intended to include all modifications, equivalents, and alternatives included in the spirit and scope of the present invention.

Embodiments for Carrying Out the Invention

[0019] Aspects of the present disclosure relate to computer security, and more specific aspects relate to using hierarchies to resolve competing commands. The present disclosure is not necessarily limited to such applications, but various aspects of the present disclosure can be understood through consideration of various examples using this context.

[0020] Many IoT devices are deployed in an environment for the purpose of being able to provide multiple inputs by multiple parties almost simultaneously and / or are adapted to receive commands via voice control from anyone near the IoT device. As a result, the IoT device may need to manage competing commands. For example, when a first user issues a command to play classical music on a smart speaker (i.e., a general IoT device) and shortly thereafter a second user issues a command to play rock music on the smart speaker, a conflict may occur.

[0021] One possible approach to this problem is to resolve conflicts by processing all commands, i.e., always executing commands issued later. However, this approach can pose the problem that in a real-world hierarchy, commands from higher-ranking individuals can be overridden in the real world by lower-ranking individuals in the hierarchy. As an example of such a hierarchy-based conflict, a parent may request that educational content be played on a smart speaker, and then, after that content has started, a child may issue a command to play a jingle on the smart speaker. In the "process all commands" approach, the smart speaker will immediately process the second command, which will result in stopping the educational content. This solution can frustrate the parent, and if it occurs continuously, the parent may turn off the smart speaker. This reaction, in turn, may anger the child since nothing will be playing anymore.

[0022] Some conflicts can occur even between commands issued at clearly different times. For example, in a manufacturing situation, a senior engineer may issue a command that a particular switch must remain closed during a maintenance process. Next, when a junior engineer issues a command to open the same switch, a conflict with that command will occur. That is, even if the second command can occur several hours later, it violates the range set by the first command and will thus be treated as a conflict.

[0023] Accordingly, IoT systems in some embodiments of this disclosure include logic that can determine how to identify and resolve conflicting commands using the context of the commands and the hierarchical relationships in the physical world between the users who issued those commands. In some embodiments, an IoT device may receive a first command from a first user. For example, the IoT device may receive a voice command via a microphone communicatively coupled to the IoT device. After receiving the first command, the IoT device may receive a second command from a second user, for example, via the microphone. An IoT system (e.g., an IoT device, a cloud processing server, or a combination thereof) may analyze the commands and, taking into account the context of the commands, determine whether the commands are conflicting. The IoT system may then resolve any identified conflicts, at least in part, based on the hierarchy in the physical world between the users.

[0024] In some embodiments, an IoT system may first analyze an incoming command using, for example, natural language processing (NLP) that can convert the command into a string of machine instructions. The IoT system may then analyze the string of machine instructions to determine whether the string of machine instructions conflicts, taking into account the context of the two commands (for example, if the associated IoT device cannot execute both commands simultaneously, or if the second command reverses or cancels the first command). If a conflict is found, the IoT system may determine a response to the second command based at least partially on the hierarchy of the physical world of the user who issued the command. The response may include executing the second command, ignoring the second command, or performing an alternative response. An alternative response may include seeking approval from a higher-level user or partially executing the second command.

[0025] In some embodiments, a conflict may be determined by examining the context of a command. For example, a first command may instruct an IoT device to add an item to a grocery list, and a second command may add another item to the same list without conflict. In some embodiments, the context may be determined by examining the user's past command history. Continuing with the grocery list example, if a second command is given by a child to add candy to the grocery list, the first user may override that command, mark it as a conflict, or do a combination of both. After one or more such events, some embodiments may learn to treat the second command as a conflict even if the same two commands do not conflict with respect to the other user. Similarly, past history may be used to help determine whether duplicate entries are conflicts.

[0026] In some embodiments, the past history of commands can be used to define fuzzy contextual boundaries around future commands. For example, in industry, a chief engineer initializes a process with a set of instructions. When an employee first gives a command that conflicts with those initial instructions, the IoT system may ask the supervisor to approve the conflicting command. If the supervisor frequently and routinely approves that particular conflicting command, the IoT system will consequently begin to treat the command as potentially non-conflicting (e.g., notifying the supervisor but not requiring approval before execution), and then as ultimately non-conflicting (e.g., executing the command as usual). In this way, some embodiments can establish dynamic guardrails around the chief engineer's initial instructions. In these embodiments, since the initial instructions may be drafted relatively strictly and then dynamically modified / laid off as the organization develops expertise in the process, it may also be desirable to proactively identify incorrect, inappropriate, or combinations thereof. In other words, the organization's normal practices can be dynamically integrated into the level of authority for its members / employees.

[0027] In some embodiments, different audio can be tagged with one or more attributes to help establish context. In some embodiments, these attributes can be established as follows: (i) If the speaker's voice identification information is unknown to the system, the IoT system may use physical information (e.g., where commands are given in the context of the IoT device) and past history with the IoT system (e.g., past commands from this voice, past interactions between voices, etc.) that provides a hierarchical context (e.g., one party calling the other party "boss") to map the voice to a hierarchical persona (child, parent, employee, manager, etc.) or a combination thereof. (ii) If voice identification information is known to the system, for example from an external authentication mechanism, the attributes may be retrieved from one or more external data stores, such as an enterprise hierarchy data store in the business world. These attributes may be used to construct a systematic physical world hierarchy within a given physical world context, such as home, school, office, etc. Once the user hierarchy is established, the relative position in the hierarchy may be used to provide conflict resolution for conflicting commands, for example, to comply with commands of a higher-level speaker in the hierarchy.

[0028] In some embodiments, one or more machine learning models may be used to dynamically generate user hierarchies. For example, in a home environment, an IoT device adapted to voice commands may analyze the user's voice and determine that, in a home environment, a user with a voice that is perceived as an adult's voice will be given a higher position than a user with a voice that is perceived as a child's voice. Thus, commands issued by a user perceived as an adult may take precedence over commands issued by a user perceived as a child.

[0029] Furthermore, machine learning models can dynamically generate customized hierarchies for specific projects, specific machines, etc., using one or more external data sources, such as corporate directories. This may include querying corporate directories to determine explicit hierarchies, extracting one or more relevant job roles, inferring relevant job experience, or a combination thereof. The derivation of dynamic hierarchies may then involve mapping hierarchies, previous job roles, and previous job experience to the specific context of commands. In this way, dynamic, customized hierarchies can be generated for each specific task and employee pair. For example, for a programming task, a machine learning model may rank an employee with prior programming experience higher than someone whose past experience is almost entirely in sales, even if a salesperson may hold a higher title in the corporate directory. Similarly, if a developer and a technical lead are working collaboratively on a project using a voice-enabled integrated development environment (IDE) (e.g., pair programming), the two employees may debug code using a set of voice commands, and conflicting execution scenarios may exist. The IDE's voice command component can recognize the voices of both employees (for example, because they are authenticated to the corporate network). The voice command component can then use the corporate directory to recognize that the lead engineer is more senior than the two and / or more skilled in this particular area, and therefore can prioritize the lead engineer's commands over the developer's commands.

[0030] In another example, a parent might ask a voice-command-enabled IoT device to play educational content, and once the content has started, the child might give another command to play a jingle. In this application, the IoT device can detect and understand the parent-child hierarchy in the home, and therefore, if both the parent and child give conflicting voice inputs, the system can recognize that it should continue to follow the parent's input. Thus, in some embodiments, the system may continue playing the educational content. However, if the child gives a non-conflicting voice input, such as increasing or decreasing the volume, then in some embodiments, the system may then follow that command.

[0031] In another example involving a parent and child, a parent asks a voice-command-enabled IoT device to add items to a shopping list where orders will be automatically placed. Later that day, the child asks the IoT device to add candy and chocolate to the shopping list. In this example, a similar situation has occurred in the past that resulted in the additional items being ordered, and the parent has subsequently instructed, trained, or combined to instruct the IoT device not to accept shopping list management commands from anyone other than themselves. In this example, the IoT device may learn that these commands conflict and will not add candy or chocolate to the shopping list in the future.

[0032] Another example involves a voice-activated IoT thermostat that controls the temperature of a house, a homeowner whose voice the IoT thermostat recognizes from past interactions, and a guest whose voice the thermostat does not recognize. In this example, the host feels cold and issues a voice command to raise the temperature. The IoT thermostat complies with this request. The guest disagrees and tries to lower the temperature, and therefore attempts to issue a voice command to lower the temperature. The IoT thermostat may not comply with this request and may provide feedback to the guest that the host recently set this temperature and the guest cannot override that setting.

[0033] Those skilled in the art will understand that embodiments of the present disclosure are not limited to IoT devices. For example, a voice-activated autonomous vehicle may be occupied by two people: one person sitting in the owner / driver seat and the other sitting in the passenger seat. In this example, the autonomous vehicle may be able to detect where each voice is emanating from and be configured to treat the person in the driver seat as having a higher position in the hierarchy than the person in the passenger seat. In this example, the passenger commands the autonomous vehicle to drive to a nearby market. If the driver has not given a command regarding the destination, the autonomous vehicle may accept this command and begin driving. Along the way, the driver may give a command to go to a restaurant. The autonomous vehicle may also accept this command and therefore change the destination. Next, if the passenger objects and commands the autonomous vehicle to continue driving to the market, the autonomous vehicle may explain that this last command will not be obeyed because the driver has set a destination and has not yet reached it.

[0034] Another example involves a co-robotics manufacturing line. In this example, several collaborative robots (cobots) are supervised by a worker-supervisor pair. Voices from all employees have already been recognized and previously registered in the voice command module of the cobot control program. In this example, one of the workers detects a defect and gives a verbal command to stop the manufacturing line. This command may be initially accepted by the voice command module, and the manufacturing line may be stopped. Upon hearing the command, the supervisor may determine that, at the end of the manufacturing process, the product should be tagged for rework. The supervisor may instruct the voice command module to continue despite the defect. This command conflicts with the preceding stop command, but the supervisor's command has higher priority. Therefore, the voice command module may accept the supervisor's command.

[0035] In some embodiments, if a conflict is detected in a command from a lower-level person in the real-world hierarchy, a conflict resolution process may be initiated that requires resolution from a higher-level person in the hierarchy. Continuing with the co-robotics manufacturing line example described above, consider a scenario where one worker detects a defect and is instructed to continue the manufacturing line. The command processing system may use historical information to evaluate the command and determine that the identified defect could cause serious damage. The conflict handling system may respond by automatically requesting supervisor confirmation and providing feedback before work can continue.

[0036] In some embodiments, the hierarchy may be contextual, so as to be dynamically established based on the context. For example, an autonomous vehicle may be remotely commanded by a parent to pick up a child from school and bring the child home. While inside the vehicle, the child may ask the vehicle to stop at a snack aisle. Since this command conflicts with a command issued by the parent, the autonomous vehicle in some embodiments may not fulfill the child's command and may continue on its way home. Later, if the parent issues a second remote command to the autonomous vehicle asking it to detour and pick them up on the way home, the autonomous vehicle may determine that it should detour and pick up the parent on the way, even if this third command conflicts with the original command.

[0037] Another example involves communication between IoT devices, where conflict resolution occurs at the intelligent orchestration layer. In this example, a human user has numerous IoT devices (e.g., a smart refrigerator, a smart washing machine, etc.) that can add items to a grocery list. In this example, the smart refrigerator might determine that the family is out of ice cream and issue a command to add it to the grocery list. However, the family may have a new fitness goal in which they have agreed to limit such high-calorie foods. The intelligent orchestration layer might determine that the command to add ice cream conflicts with the fitness goal and then either reject the command to add ice cream or flag the command for further review. [Data Processing System]

[0038] Figure 1 shows one embodiment of a data processing system (DPS) 100a, 100b (collectively referred to as DPS100 herein) that conforms to several embodiments. Figure 1 shows only the main components typical of DPS100, and the individual components may have a higher complexity than those shown in Figure 1. In some embodiments, DPS100 may be implemented as a processor, smart device, or any other suitable type of electronic device, embedded in a portable computer such as a personal computer, server computer, laptop or notebook computer, PDA (Personal Digital Assistant), tablet computer, or smartphone, or in a larger device such as an automobile, aircraft, teleconferencing system, or electrical appliance. Furthermore, there may be components other than or additional to those shown in Figure 1, and the number, type, and configuration of such components may vary.

[0039] The data processing system 100 in Figure 1 may comprise a number of processing units 110a to 110d (collectively referred to as processors 110 or CPUs 110) that can be connected by a system bus 122 to main memory 112, a mass storage interface 114, a terminal / display interface 116, a network interface 118, and an input / output ("I / O") interface 120. In this embodiment, the mass storage interface 114 may connect the system bus 122 to one or more mass storage devices such as direct access storage devices 140, USB drives 141, read / write optical disk drives 142, or a combination thereof. One or more direct access storage devices 140 may be logically organized into a RAID array and consequently managed by a RAID controller 115, for example, by software running on the processors 110 in the mass storage interface 114, or a combination of both. The network interface 118 may allow DPS 100a to communicate with other DPS 100b through a network 106. The main memory 112 may include an operating system 124, multiple application programs 126, and program data 128.

[0040] Embodiments of the DPS100 in Figure 1 may be general-purpose computing devices. In these embodiments, the processor 110 may be any device capable of executing program instructions stored in the main memory 112, and may itself be constructed of one or more microprocessors, integrated circuits, or combinations thereof. In some embodiments, the DPS100 may include a number of processors or processing cores or combinations thereof, as is typical in larger, more functional computer systems. However, in other embodiments, the computing system 100 may include only a single-processor system or a single processor or combination thereof, designed to emulate a multiprocessor system. Furthermore, the processor 110 may be implemented using a number of heterogeneous data processing systems 100 in which the main processor 110 resides together with secondary processors on a single chip. As another example, the processor 110 may be a symmetric multiprocessor system including a number of processors 110 of the same type.

[0041] When the DPS100 is started, the associated processor 110 may first execute program instructions that constitute the operating system 124. The operating system 124 may then manage the physical and logical resources of the DPS100. These resources may include main memory 112, mass storage interface 114, terminal / display interface 116, network interface 118, and system bus 122. Similar to processor 110, several embodiments of the DPS100 may utilize a number of system interfaces 114, 116, 118, 120 and bus 122, and as a result, each of these may include its own separate, fully programmed microprocessor.

[0042] Instructions for the operating system 124 or application programs 126 or combinations thereof (collectively referred to as “program code,” “computer-readable program code,” or “computer-readable program code”) may first be located in a mass storage device communicating with the processor 110 via the system bus 122. In different embodiments, the program code may be embodied on different physical or tangible computer-readable media, such as memory 112 or the mass storage device. In the example of Figure 1, instructions may be stored in a functional form in persistent storage on a direct-access storage device 140. These instructions may then be loaded into main memory 112 for execution by the processor 110. However, the program code may also be located in a functional form on a computer-readable medium, such as the direct-access storage device 140 or a read / write optical disk drive 142, which may be selectively removable in some embodiments. The program code may be loaded into or transferred to the DPS 100 for execution by the processor 110.

[0043] Continuing with the reference to Figure 1, the system bus 122 may be any device that facilitates communication between the processor 110, the main memory 112, and interfaces 114, 116, 118, and 120. Furthermore, while the system bus 122 in this embodiment is a relatively simple single bus structure that provides direct communication paths between system buses 122, other bus structures, including, but not limited to, point-to-point links in hierarchical, star-shaped, or web-shaped configurations, multiple hierarchical buses, parallel and redundant paths, etc., are consistent with this disclosure.

[0044] The main memory 112 and the mass storage device 140 can work together to store the operating system 124, application programs 126, and program data 128. In some embodiments, the main memory 112 may be a random-access semiconductor memory device ("RAM") capable of storing data and program instructions. While Figure 1 conceptually shows the main memory 112 as a single monolithic entity, in some embodiments, the main memory 112 may have a more complex configuration, such as a hierarchy of caches and other memory devices. For example, the main memory 112 may exist with multiple levels of caches, which may be further divided by function, such that one cache holds instructions while another holds non-instruction data used by the processor 110. The main memory 112 may also be distributed and associated with different processors 110 or sets of processors 110, as is known in any of the various so-called unequal memory access (NUMA) computer architectures. Furthermore, some embodiments may utilize a virtual addressing mechanism that allows the DPS100 to behave as if it had access to a single large storage entity instead of accessing a number of smaller storage entities (such as the main memory 112 and the mass storage device 140).

[0045] Although the operating system 124, application program 126, and program data 128 are shown in Figure 1 as being contained within the main memory 112 of the DPS 100a, in some embodiments, some or all of them may be physically located on a different computer system (e.g., DPS 100b) and be accessed remotely, for example, via a network 106. Furthermore, the operating system 124, application program 126, and program data 128 are not necessarily all contained within the same physical DPS 100a at the same time, and may even reside in the physical or virtual memory of another DPS 100b.

[0046] In some embodiments, system interface units 114, 116, 118, and 120 may support communication with a variety of storage and I / O devices. The mass storage interface 114 may support the installation of one or more mass storage devices 140, which may include a rotating magnetic disk drive storage device, a solid-state storage device (SSD) that uses an integrated circuit assembly as memory for persistently storing data, typically using flash memory, or a combination of the two. Furthermore, the mass storage device 140 may also include other devices and assemblies, such as an array of disk drives configured to appear to the host as a single large storage device (commonly called a RAID array), or archival storage media such as hard disk drives, tapes (e.g., MiniDV), writable compact discs (e.g., CD-R and CD-RW), digital multipurpose discs (e.g., DVD, DVD-R, DVD+R, DVD+RW, DVD-RAM), holographic storage systems, Blue LaserDiscs, IBM Millipede devices, and similar devices, or combinations thereof. The I / O interface 120 may support the connection of one or more I / O devices, such as a keyboard 181, a mouse 182, a modem 183, or a printer (not shown).

[0047] The terminal / display interface 116 may be used to directly connect one or more displays 180 to the data processing system 100. These displays 180 may be non-intelligent (i.e., dumb) terminals such as LED monitors, or they may be fully programmable workstations that enable IT administrators and users to communicate with the DPS 100. However, it should be noted that while the display interface 116 may be provided to support communication with one or more displays 180, the computer system 100 does not necessarily require the displays 180, as all necessary interactions with users and other processes can occur via the network 106.

[0048] Network 106 may be any suitable network or combination of networks and may support any suitable protocol suitable for the communication of data or code or combinations thereof to and from a number of DPS 100s. Thus, the network interface 118 may be any device that facilitates such communication, whether or not the network connection is made using current analog or digital techniques or a combination thereof, or via some future networking mechanism. Suitable networks 106 include, but are not limited to, networks implemented using one or more of the “Infiniband” or IEEE 802.3x “Ethernet®” specifications, cellular transmission networks, radio networks implemented using one of the IEEE 802.11x, IEEE 802.16, General Purpose Packet Radio Service ("GPRS"), FRS (Family Radio Service), or Bluetooth® specifications, ultra-wideband ("UWB") technology as described in FCC 02-48, or similar. Those skilled in the art will understand that many different network and transport protocols may be used to implement network 106. The Transmission Control Protocol / Internet Protocol ("TCP / IP") suite includes suitable network and transport protocols. [Cloud Computing]

[0049] Figure 2 shows one embodiment of a suitable cloud environment for providing NLP services including machine learning models. While this disclosure includes a detailed description of cloud computing, it should be understood that the implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented in combination with any other type of computing environment that is currently known or may be developed in the future.

[0050] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0051] The characteristics are as follows: • On-demand self-service: Cloud users can unilaterally provision computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers. • Broad network access: Capabilities are available over the network and accessed through standard mechanisms (e.g., mobile phones, laptops, and PDAs) that facilitate use by heterogeneous thin or thick client platforms. • Resource Pool: A provider's computing resources are pooled and served to multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated as needed. Generally, consumers have no control or knowledge of the exact location of the resources provided, but location independence means they may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center). • Rapid Flexibility: In some cases, capacity can be provisioned rapidly and flexibly for rapid scale-out, and rapidly released for rapid scale-in, automatically. To consumers, the capacity available for provisioning often appears unlimited and can be purchased in any quantity at any time. • Measured Services: Cloud systems automatically control and optimize resource usage by leveraging measurement capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0052] The service model is as follows: Software as a Service (SaaS): The ability offered to consumers is the use of a provider's applications running on cloud infrastructure. These applications are accessible from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with the exception of limited, user-specific application configuration settings. Platform as a Service (PaaS): The ability provided to consumers is to deploy applications they have created or acquired, written using programming languages ​​and tools supported by the provider, onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and, in some cases, the applications hosting the environment configuration. Infrastructure as a Service (IaaS): The ability offered to consumers is the provisioning of processing, storage, networking, and other fundamental computing resources that enable consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do control the operating system, storage, and deployed applications, and potentially have limited control over selected networking components (e.g., host firewalls).

[0053] The deployment model is as follows: • Private Cloud: The cloud infrastructure operates solely for the organization. It can be managed by the organization or a third party and can reside on-premises or off-premises. • Community Cloud: Cloud infrastructure is shared by several organizations and supports a specific community with shared interests (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can reside on-premises or off-premises. • Public cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by organizations that sell cloud services. • Hybrid Cloud: A combination of two or more clouds (private, community, or public) that remain separate entities but are joined together by standardization or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0054] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is infrastructure, including a network of interconnected nodes.

[0055] Referring here to Figure 2, an exemplary cloud computing environment 50 is shown. As illustrated, the cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud users, such as a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, or an automotive computer system 54N, or a combination thereof, can communicate. The nodes 10 can communicate with each other. They may be physically or virtually grouped (not shown) in one or more networks, or a combination thereof, such as private, community, public, or hybrid clouds as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, or software, or a combination thereof, as a service, without requiring cloud users to maintain resources on their local computing devices. The types of computing devices 54A-N shown in Figure 2 are for illustrative purposes only, and it should be understood that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of computerized device via any type of network or network addressable connection, or a combination thereof (for example, using a web browser).

[0056] Referring now to Figure 3, a set of functional abstraction layers provided by the cloud computing environment 50 (Figure 2) is shown. It should be understood in advance that the components, layers, and functionalities shown in Figure 3 are for illustrative purposes only, and embodiments of the present invention are not limited thereto. As illustrated, the following layers and their corresponding functionalities are provided:

[0057] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include a mainframe 61, RISC (Reduced Instruction Set Computer) architecture-based servers 62, 63, blade servers 64, storage devices 65, and network and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0058] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities may be provided, namely virtual servers 71, virtual storage 72, virtual networks 73 including virtual private networks, virtual applications and operating systems 74, and virtual clients 75.

[0059] In one example, the management layer 80 may provide the following functions: Resource provisioning 81 provides dynamic procurement of computing and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 82 provides cost tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud users and tasks, and protection for data and other resources. The user portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides allocation and management of cloud computing resources to ensure that required service levels are met. Service Level Agreement (SLA) planning and implementation 85 provides pre-arrangements and procurement of cloud computing resources where future requirements are anticipated according to the SLA.

[0060] Workload layer 90 provides examples of functions that can be utilized in a cloud computing environment. Examples of workloads and functions that can be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analytics processing 94, transaction processing 95, and NLP services 96. [Voice-controlled IoT device]

[0061] Figure 4 shows an exemplary embodiment of the IoT system 400, adapted to several embodiments. The IoT system 400 includes a voice command-enabled IoT device 405 having a memory 410, a processor 415, and a network interface 418. The memory 410 may store data such as user settings, user usage history, user commands, and user profile data. The processor 415 may perform functions such as receiving voice commands, presenting voice commands to an NLP service 96 for analysis, and executing a series of commands resulting from the NLP service 96. In some embodiments, the processor 415 may execute computer-readable instructions to perform one or more of the methods described herein. Furthermore, in some embodiments, the IoT device 405 may include one or more audio indicators or visual displays 411 that may provide information such as how a command is being handled. For example, an audio indicator may beep twice if a command is ignored, and an LED may emit light of a specific color when a command is executed or queued, etc.

[0062] The IoT device 405 may communicate with an array of devices such as an NLP service 96 and one or more electronic devices 455 via one or more networks 460 and a network interface 418. Each electronic device 455 may include an application programming interface (API) that, as a result, allows the IoT device 405 to control their operation. The network 460 may be implemented using any number of any suitable communication medium. For example, the network 460 may be a wide area network (WAN), a local area network (LAN), the internet, or an intranet. In some embodiments, the NLP service 96, electronic devices 455, the IoT device 405, or a combination thereof may be communicably coupled using one or more networks and / or one or more local connections, or a combination thereof. For example, electronic devices 455 may be hardwired to the IoT device 405 (e.g., connected by an Ethernet® cable), while the NLP service 96 may communicate with the IoT device 405 using the network 460 (e.g., via the internet).

[0063] In some embodiments, portions of the network 460 can be implemented within a cloud computing environment 50 or using one or more cloud computing services. To suit various embodiments, the cloud computing environment 50 may include network-based distributed data processing systems (DPS) 100 that provide one or more cloud computing services, including NLP services 96. Furthermore, the cloud computing environment 50 may include many DPS 100 (e.g., hundreds or thousands of computers, or more) that are located in one or more data centers and configured to share resources across the network 460.

[0064] In some embodiments, the NLP service 96 may include one or more command analysis models 490 and a competition model 492. The NLP service 96 may also be adapted to query one or more data sources such as a voice-to-identity (V2ID) mapping data store 417, a corporate directory 437, a device / application-specific corporate data store 438, and a historical knowledge data corpus 439. These data sources may be provided in whole or in part by virtual storage 72.

[0065] Figures 5A to 5C are flowcharts of one method 500 for operating an IoT device 405, which is adapted to several embodiments. In operation 505, the IoT device 405 may receive two or more voice commands substantially simultaneously. Here, the amount of time considered "substantially the same" may be a configurable fixed amount of time, or it may depend on the nature of the particular voice command (e.g., to any length). An example of the latter may include two or more voice commands, where the second voice command is received before the first voice command is processed, or the result of the first voice command is still presented, for example, on an attached display device, or the first command is maintained until it is explicitly discarded.

[0066] After IoT device 405 receives and identifies that two or more voice commands have been received substantially simultaneously (as defined above), IoT device 405 may verify whether these voice commands are conflicting or not in operation 510. Examples of conflicting voice commands include situations where an adult asks IoT device 405 to play educational content and a child then asks IoT device 405 to play a jingle, or where a parent asks for the volume to be lowered and the child then asks for the volume to be raised. In contrast, an example of non-conflicting voice commands is when an adult asks IoT device 405 to play educational content and a child then asks for the volume to be raised or lowered. As will be illustrated in more detail with reference to Figures 5A–8, this operation may include an NLP service 96 that translates voice commands into strings of machine commands.

[0067] If voice commands do not conflict, IoT device 405 may process them in the order they are received (not shown). However, if IoT device 405 identifies that a large number of voice commands are received substantially simultaneously and there are conflicting voice commands, IoT device 405 may determine which voice commands to prioritize. In some embodiments, this may include IoT device 405 mapping the received voice to the identification information of the person who provided the command in operation 515. In some embodiments, IoT device 405 may perform this mapping using V2ID 417. Additionally or alternatively, IoT device 405 may use authentication attributes (e.g., user ID) that the individual presents before using IoT device 405. For example, a voice command-enabled video conferencing system may be linked to a corporate authentication system, which may require all employees to authenticate (e.g., log in) before using the system. In some embodiments, the identification information in V2ID 417 may have limited attributes due to functional or privacy considerations. For example, an IoT device 405 adapted for use with a consumer-grade home automation system may only recognize the individual who previously used the device 405, and may not require the user to explicitly authenticate themselves.

[0068] In operation 525, IoT device 405 may verify whether persona attributes for all speakers can be identified so that a hierarchy can be established. If all persona attributes are identified, the system proceeds to operation 535. However, if not all relevant persona attributes are identified, and therefore IoT device 405 cannot establish a hierarchy, IoT device 405 may stop performing hierarchy-based command execution and, in operation 590, execute commands sequentially.

[0069] Depending on the mapping within V2ID417, IoT device 405 may or may not be able to map one or more voices to corresponding speaker identification information. If IoT device 405 cannot identify an individual via V2ID417, IoT device 405 may, in operation 520, use / utilize one or more alternative corpora to map voices to personas. In some embodiments, the alternative corpus may have a number of personas specifically tailored to group voices into hierarchical scenarios. Examples of such personas may include a persona of a child, a persona of an adult, a persona of a non-parent adult, a persona of a supervisor, etc. In these embodiments, IoT device 405 may utilize one or more external data stores to match a unique device identifier associated with IoT device 405 with a persona in order to identify whether the persona is a known persona, such as a social media login, or an unknown persona.

[0070] If IoT device 405 validates persona attributes for all speakers, a hierarchy can be established in operation 525. If all persona attributes are identified, the system proceeds to operation 535. However, if not all relevant persona attributes are identified, and therefore the system cannot establish a hierarchy, IoT device 405 may stop performing hierarchy-based command execution and proceed to execute commands sequentially in operation 590.

[0071] If IoT device 405 has established a mapping from voice to identification information in operation 515, IoT device 405 may, in operation 530, retrieve speaker attributes by searching a predefined data store such as a corporate directory 437 or a business data store 438 specific to another device / application. Examples of such attributes may include corporate hierarchical relationships, skill mappings, and team position related to the context of command execution. For example, if remote operations are performed by a team of engineers through a robotic device that operates via voice commands, IoT device 405 may use the corporate directory 437 to identify the engineers' hierarchy based on their individual experience, the number of remote operations performed, other relevant attributes, or a combination thereof.

[0072] In operation 535, the IoT device 405 may compute additional attribute relationships from past interaction experiences and system interactions (for example, based on the historical knowledge data corpus 439), which may be further complemented by the attribute list generated in operation 530. For example, in some embodiments, the IoT device 405 may use the historical data corpus to identify that one speaker has a known association with the IoT device 405, while another speaker does not have a past / historical association with the IoT device 405. Thus, the IoT device 405 may utilize this information through operations 540 and 545 (described below). In some embodiments, the IoT device 405 may skip this operation if no such additional attribute or hierarchical relationship mapping is identified from the historical knowledge data corpus 439.

[0073] In operation 540, the IoT device 405 may compile a list of all known static and dynamic attributes, as well as their respective values, to help determine the hierarchy of each speaker. Static attributes may be obtained from one or more external sources, such as the corporate directory 437 or the corporate data store 438, as described in more detail with reference to operation 530. Dynamic attributes may then be obtained from one or more sources, such as the historical interaction experience corpus 439.

[0074] Next, in operation 545, IoT device 405 may obtain attribute weighting and attribute overrides from an externalized configuration. One example of weighting applied is when IoT device 405 receives a number of voice commands from two adult personas; known voices may be given a higher weight than unknown voices. Another example of weighting applied is when IoT device 405 receives a number of known voice commands from two personas, one adult and the other child; IoT device 405 may give a higher weight to the adult persona than to the child persona. An example of override applied is when IoT device 405 receives a number of voices, some known and some unknown, but when IoT device 405 digs further into the personas, known voices may be identified as being associated with children and unknown voices as being associated with adults. In such a scenario, IoT device 405 may utilize additional override rules depending on specific user preferences, such as overriding commands provided by adults with commands provided by children.

[0075] In operation 550, IoT device 405 may remove any attribute that could cause bias, such as gender / race, from the final list of attributes. During this operation, filtering of any other attributes may also be performed (if applicable). Next, in operation 555, IoT device 405 may verify whether there are enough attributes and their corresponding values ​​that would allow IoT device 405 to derive a hierarchy. If there are not enough attributes, IoT device 405 may stop executing hierarchy-based commands in operation 590 and proceed to execute commands sequentially. If there are enough attributes, IoT device 405 may perform attribute matching and comparison in operation 560 to derive a hierarchy among speakers. Hierarchy verification checks may take into account any specific overrides established and attribute weightings.

[0076] In operation 565, the IoT device 405 may establish a hierarchy between speakers of different commands. Based on the established hierarchy, the IoT device 405 may identify which machine command should be executed in operation 570.

[0077] Based on an established hierarchy, IoT device 405 may execute a selected machine command in operation 575. Command execution can result in different actions depending on the context of the input, and results that allow IoT device 405 to determine whether or not to execute the command. Optionally, IoT device 405 may provide feedback on why the command was executed (or not executed) and what hierarchical decision logic is behind it. For example, in a scenario where an adult commands IoT device 405 to play an educational story and a child provides another command to play a verse, if the result of the first command (i.e., playing the story) is not completed, IoT device 405 may utilize the hierarchical logic and not follow the command provided by the child (i.e., not play the verse). In some applications, the functionality of IoT device 405 may not depend on the completion of a command. For example, in a remote operation where multiple commands are provided, and different engineers are located in different places and connected together via remote conferencing to execute the provided commands via robotics, the IoT device 405 may perform hierarchical execution to determine which commands should be executed.

[0078] Next, in operation 580, IoT device 405 may extract hierarchical learning from this interaction (weighting / overriding) along with the attributes and preferences used, and add them to the historical corpus. IoT device 405 may then stop executing hierarchical-based commands and execute commands sequentially. IoT device 405 may then stop when further operation is required.

[0079] Figure 6 is a flowchart of a method 600 that performs an operation 510 by machine learning models 490, 492, which are adapted to several embodiments. Method 600 is initiated by an IoT device 405 that receives first and second commands. For example, the first command may be to play educational content, and the second command may be to play a jingle. This is shown in operation 605. Method 600 may continue with the IoT device 405 forwarding the first and second commands to an NLP service 96 in a cloud environment 50. This is shown in operation 610. The command analysis model 490 may then analyze the first command in operation 615 and output a specific machine instruction of the first sequence. For example, the command analysis model 490 may convert the first command into a machine instruction to play content at a first uniform resource locator (URL) and a machine instruction to continue playback until the content is finished. Next, in operation 620, the server DPS100a may use the command analysis model 490 to analyze a second voice command and output a specific machine instruction for the second sequence. For example, the command analysis model 490 may convert the second command into a machine instruction to play content at the first uniform resource locator (URL) and a machine instruction to continue playback until the content is finished.

[0080] Next, the server DPS100a may pass the machine instructions of the first and second sequences to the trained race model 492 in the NLP service 96. This is shown in operation 630. The trained race model may analyze the machine instructions of the two sequences and determine whether the second sequence can be executed without modifying the machine instructions of the first sequence. This is shown in operation 635. [Model training]

[0081] The NLP service 96 may, in some embodiments, include one or more machine learning models 490,492, which may be any software systems that recognize patterns. In some embodiments, the machine learning models 490,492 may comprise a plurality of artificial neurons interconnected via connection points called synapses. Each synapse may encode the strength of the connection between the output of one neuron and the input of another neuron. The output of each neuron may then be determined by the total input received from other neurons connected to it, and therefore by the outputs of these “upstream” connected neurons and the strength of the connection, which is determined by synaptic weights.

[0082] In some embodiments, machine learning models 490,492 can be trained to solve specific problems (e.g., determining whether two sequences of commands can both be executed without conflict) by adjusting the synaptic weights so that a particular class of inputs produces a desired output. In these embodiments, this weight adjustment procedure is known as "learning". Ideally, these adjustments, during the learning process, result in a pattern of synaptic weights that converges to an optimal solution for a given problem based on some cost function.

[0083] In some embodiments, artificial neurons can be organized into layers. The layer that receives external data is the input layer. The layer that produces the final result is the output layer. Some embodiments include hidden layers between the input and output layers, and typically include hundreds of such hidden layers.

[0084] Figure 7 is a flowchart of a method 700 for training machine learning models 490,492, adapted to several embodiments. The system manager may start in operation 710 by loading training vectors. The vectors may include recordings from several different users reading specially prepared transcripts. Some transcripts may contain conflicting sets of commands. Other transcripts may contain non-conflicting sets of commands.

[0085] In operation 712, the system manager may select a desired output (e.g., a series of commands conflict). In operation 714, the training data may be prepared to reduce sources of bias, typically including deduplication, normalization, and randomization of order. In operation 716, the initial gate weights for the machine learning model may be randomized. In operation 718, machine learning models 490,492 may be used to predict an output using a set of input data vectors, and their predictions are compared to labeled data. Errors (e.g., differences between the predicted values ​​and the labeled data) are then used in operation 720 to update the gate weights. This process may be repeated, updating the weights in each iteration, until the training data is exhausted or the machine learning models 490,492 reach an acceptable level of accuracy, confidence, or a combination thereof. In operation 722, the resulting model may optionally be compared to previously unevaluated data to validate and test its performance. In operation 724, the resulting model can be loaded into the NLP service 96, which is used to analyze user commands.

[0086] Figure 8 is a flowchart of one method 800 for updating machine learning models 490,492, adapted to several embodiments. In operation 805, the current gate weights for machine learning models 490,492 may be loaded. In operation 810, the IoT device 405 may send two voice commands to an NLP service 96 capable of processing two voices, using method 500 as described with reference to Figures 5A-5C and the current gate weights. The IoT device 405 may then, in operation 815, execute the commands as recommended by the NLP service 96. In operation 820, a higher-level user may issue a corrective command to the IoT device 405 in response to the executed commands.

[0087] Voice commands and modifications may be stored in the command history data store in operation 830. Periodically (e.g., daily), in operation 840, the NLP service 96 may re-execute the history through machine learning models 490,492, and modified commands may be used to generate error signals. These error signals may then be used in operation 850 to update gate weights. The resulting model may then be loaded into the NLP service 96 and used to analyze future user commands. [General content]

[0088] While embodiments of this disclosure are described with reference to voice-command-enabled IoT devices, other systems and applications are within the scope and intent. For example, some embodiments may be adapted for use in gesture-controlled devices that can receive commands via user gestures (e.g., physical movements that can be interpreted by motion sensors) in addition to, or instead of, voice commands. Similarly, some embodiments may be adapted to receive one or both commands via conventional I / O systems such as keyboards, mice, touchscreens, or combinations thereof. Conflict resolution may be initiated by a command initiated by the device (e.g., in response to IoT-to-IoT communication). In these embodiments, conflict resolution may occur at an intelligent orchestration layer.

[0089] The present invention may be a system, method, or computer program product or a combination thereof in an integration of any possible level of technical detail. The computer program product may include a computer-readable storage medium (or a plurality of computer-readable storage media) having computer-readable program instructions for causing a processor to perform an aspect of the present invention.

[0090] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any preferred combination of those described above. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures with recorded instructions in grooves, and any preferred combination of those described above. The computer-readable storage media used herein should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0091] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers such instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0092] The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® or C++ or similar, and procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), and the connection may be to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized by executing computer-readable program instructions by utilizing state information of computer-readable program instructions in order to perform aspects of the present invention.

[0093] Aspects of the present invention are described herein with reference to flowcharts or block diagrams, or combinations thereof, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or combination thereof, and any combination of blocks in a flowchart or block diagram, or combination thereof, can be implemented by computer-readable program instructions.

[0094] These computer-readable program instructions may be provided to a computer processor or other programmable data processing device to generate a machine, which in turn may be executed by the computer processor or other programmable data processing device to form means for implementing functions / operations specified in a flowchart or block diagram or a combination thereof, in blocks or multiple blocks. These computer-readable program instructions may also be stored on a computer-readable storage medium, which can instruct a computer, a programmable data processing device, or other device or a combination thereof to function in a particular manner, and as a result, a computer-readable storage medium having the stored instructions may include a manufactured article containing instructions for implementing functions / operations specified in a flowchart or block diagram or a combination thereof, in blocks or multiple blocks.

[0095] Computer-readable program instructions may also be loaded onto a computer, other programmable device, or other device to perform a series of operational steps on the computer, other programmable device, or other device to generate a computer implementation process, and as a result, the instructions executed on the computer, other programmable device, or other device implement a function / operation specified in a flowchart and / or block diagram, block or more blocks.

[0096] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown within a block may occur in an order other than that shown in the figure. For example, two consecutively shown blocks may actually be implemented as a single step, executed simultaneously or substantially simultaneously in a partially or entirely overlapping manner, or the blocks may be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, or a combination thereof, and any combination of blocks in a block diagram or flowchart, or a combination thereof, may be implemented by an application-specific hardware-based system that performs a specified function or operation, or executes a combination of application-specific hardware and computer instructions.

[0097] The descriptions of the various embodiments of this disclosure are presented for illustrative purposes only and are not intended to be exhaustive or to limit the scope to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the embodiments described. The terminology used herein has been chosen to describe the principles, practical applications, or technical improvements to the technologies available on the market, or to enable other those skilled in the art to understand the embodiments disclosed herein.

[0098] Therefore, it is desirable that the embodiments described herein be considered illustrative rather than restrictive in all respects, and that reference be made to the appended claims to determine the scope of the invention.

Claims

1. In the device, there is a stage where the device receives a first command from a first user, The device includes the step of receiving a second command from a second user, A step in which a first trained machine learning model determines that the second command conflicts with the first command, In response to determining that the second command conflicts with the first command: A step of identifying the context for one of the first commands and one of the second commands; The step of compiling the attribute lists of the first user and the second user; The step of mapping the aforementioned context to the aforementioned attribute list; A second trained machine learning model determines, based on the attribute list, the hierarchy of the physical world between the first user and the second user; A step of determining the response to the second command using the hierarchy of the physical world; and, The step of performing the aforementioned response A device control method comprising:

2. The method according to claim 1, further comprising the step of using the second trained machine learning model to identify higher-ranking users selected from a group consisting of the first user and the second user.

3. The stage of receiving corrections from higher-level users as described above, A stage to maintain the history of the first command, the second command, and modifications, The steps include updating the second trained machine learning model using the aforementioned history, The method according to claim 2, further comprising:

4. The attribute list is based on querying the corporate directory for a first profile associated with the first user and a second profile associated with the second user, The second trained machine learning model further uses the mapping to dynamically determine the hierarchy of the physical world between the first and second users in the context. The method according to claim 1.

5. The method according to claim 3, further comprising the step of identifying an incorrect instruction using the history and the updated second trained machine learning model.

6. The method according to claim 4, wherein the first profile includes the first user's previous relevant experiences, and the second profile includes the second user's previous relevant experiences.

7. The method according to any one of claims 1 to 6, wherein the device is a voice control device, and the first command and the second command include voice commands.

8. A gesture control device comprising the steps of receiving a first command including a gesture command from a first user, The gesture control device includes the step of receiving a second command, including a gesture command, from a second user, A step in which a first trained machine learning model determines that the second command conflicts with the first command, In response to determining that the second command conflicts with the first command: A second trained machine learning model determines the hierarchy of the physical world between the first user and the second user; A step of determining the response to the second command using the hierarchy of the physical world; and, The step of performing the aforementioned response A method for controlling a gesture control device, comprising the following:

9. Processing unit and A memory to which the processing unit is coupled, wherein the memory is connected to the processing unit, It receives the first command from the first user, It will receive a second command from the second user, The first trained machine learning model is used to determine that the second command conflicts with the first command. In response to determining that the second command conflicts with the first command: Identify the context for one of the first commands and one of the second commands; Compile the attribute lists of the first user and the second user; Map the aforementioned context to the attribute list; A second trained machine learning model is used to determine the hierarchy of the physical world between the first user and the second user based on the attribute list; The response to the second command is determined using the hierarchy of the physical world; and, To execute the aforementioned response, A memory containing program instructions executable by the processing unit, A voice-controlled device equipped with the following features.

10. The program instruction further instructs the processing unit, The second trained machine learning model identifies higher-ranking users selected from the group consisting of the first and second users. The voice control device according to claim 9.

11. The program instruction further instructs the processing unit, We will receive corrections from higher-ranking users than those mentioned above. Maintain the history of the first command, the second command, and modifications. The history is used to update the second trained machine learning model. The voice control device according to claim 10.

12. The attribute list is based on querying the corporate directory for a first profile associated with the first user and a second profile associated with the second user, The second trained machine learning model further uses the mapping to dynamically determine the hierarchy of the physical world between the first and second users in the context. The voice control device according to claim 9.

13. The voice control device according to claim 11, wherein the program instructions cause the processing unit to further identify incorrect instructions using the history and the updated second trained machine learning model.

14. The voice control device according to claim 12, wherein the first profile includes the first user's previous relevant experiences, and the second profile includes the second user's previous relevant experiences.

15. To the processor The first user will receive the first command, The second user will receive a second command. The first trained machine learning model is used to determine that the second command conflicts with the first command. In response to determining that the second command conflicts with the first command: Identify the context for one of the first commands and one of the second commands; Compile the attribute lists of the first user and the second user; Map the aforementioned context to the attribute list; A second trained machine learning model is used to determine the hierarchy of the physical world between the first user and the second user based on the attribute list; The response to the second command is determined using the hierarchy of the physical world; and, To execute the aforementioned response, A computer program for a voice-controlled device, comprising program instructions executable by the aforementioned processor.

16. The aforementioned program instruction further instructs the processor, The second trained machine learning model identifies higher-ranking users selected from the group consisting of the first and second users. The computer program according to claim 15.

17. The aforementioned program instruction further instructs the processor, We will receive corrections from higher-ranking users than those mentioned above. Maintain the history of the first command, the second command, and modifications. The history is used to update the second trained machine learning model. The computer program according to claim 16.

18. The attribute list is based on querying the corporate directory for a first profile associated with the first user and a second profile associated with the second user, The second trained machine learning model further uses the mapping to dynamically determine the hierarchy of the physical world between the first and second users in the context. The computer program according to claim 15.

19. The computer program according to claim 17, wherein the program instructions cause the processor to further identify incorrect instructions using the history and the updated second trained machine learning model.

20. The computer program according to claim 18, wherein the first profile includes the previous relevant experiences of the first user, and the second profile includes the previous relevant experiences of the second user.

21. A processing unit and A memory to which the processing unit is coupled, wherein the memory is connected to the processing unit, The system receives a first command from the first user, which includes a gesture command. The system receives a second command from the second user, including a gesture command. The first trained machine learning model is used to determine that the second command conflicts with the first command. In response to determining that the second command conflicts with the first command: A second trained machine learning model is used to determine the hierarchy of the physical world between the first user and the second user; The hierarchy of the physical world is used to determine the response to the second command; and, To execute the aforementioned response, A memory containing program instructions executable by the processing unit, A gesture control device equipped with [a specific feature].

22. The processor The system receives a first command from the first user, which includes a gesture command. The system receives a second command from the second user, including a gesture command. The first trained machine learning model is used to determine that the second command conflicts with the first command. In response to determining that the second command conflicts with the first command: A second trained machine learning model is used to determine the hierarchy of the physical world between the first user and the second user; The hierarchy of the physical world is used to determine the response to the second command; and, To execute the aforementioned response, A computer program for a gesture control device, comprising program instructions executable by the aforementioned processor.

Images