Multi-model architecture for classification and intent determination
Through the use of multi-model structure and convex coefficient machine learning rules, the information sharing problem of traditional SLU systems when identifying intentions in multiple fields is solved, and the accurate identification and rapid training of multiple intentions in a single discourse is achieved, which improves the system's Efficiency and accuracy.
Patent Information
- Application Number
- CN202080028839.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-12-27
- Filing Date
- 2020-03-20
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-03-20
AI Technical Summary
Traditional SLU systems lack information sharing and integration capabilities when identifying multiple domain intentions, resulting in the inability to correctly identify multiple intentions from a single discourse.
Using a multi-model structure, iterative adjustment is performed by generating and sharing convex coefficients, and using local and global loss metrics to train domain-specific models, the collaboration and information integration between multiple models is achieved to identify multiple intentions in a single discourse.
Improve the accuracy and speed of identifying multiple intentions, reduce training time, reduce dependence on backpropagation, avoid gradient vanishing problems, and achieve faster convergence and higher prediction accuracy.
Smart Images

Figure CN113692595B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to machine learning and, more particularly, to multi-model structures generated using machine learning. Background Art
[0002] Machine learning systems are trained rather than explicitly pre-programmed to perform a given task. Given a task or class of tasks, machine learning systems learn to perform the task based on experience. Therefore, the performance of a machine learning system, no matter how it is measured, improves with experience. Machine learning systems learn in the sense that the system can generate rules (e.g., classify utterances or images) based on the input to the system and the expected results based on the input. In particular, machine learning systems can discover statistical structure in example inputs and generate rules for automating the performance of a given task based on that structure. Machine learning systems have a wide range of applications in various fields such as natural language understanding (NLU) and computer vision. Summary of the invention
[0003] Technical Solution
[0004] In one or more embodiments, a method includes generating a plurality of convex coefficients based on outputs generated by a plurality of computer-implemented machine learning models in response to receiving training data. The method also includes iteratively adjusting one or more of the plurality of convex coefficients using a convex coefficient machine learning rule using computer hardware.
[0005] In one or more embodiments, a system includes a processor configured to initiate operations. The operations include generating outputs from each of a plurality of domain-specific models in response to received inputs, wherein the domain-specific models are simultaneously trained machine learning models trained using a corresponding local loss metric for each domain-specific model and a global loss metric for the plurality of domain-specific models. The operations also include determining the presence or absence of an intent corresponding to each domain-specific model by classifying the output of each domain-specific model.
[0006] In one or more embodiments, a method includes: generating, using computer hardware, an output from each of a plurality of domain-specific models in response to a received input, wherein the domain-specific models are simultaneously trained machine learning models trained using a corresponding local loss metric for each domain-specific model and a global loss metric for the plurality of domain-specific models. The method also includes: determining, using the computer hardware, the presence or absence of an intent corresponding to each domain-specific model by classifying the output of each domain-specific model.
[0007] In one or more embodiments, a computer program product includes a computer-readable storage medium having instructions stored thereon. The instructions are executable by a processor to initiate operations. The operations include: generating outputs from each of a plurality of domain-specific models in response to received inputs, wherein the domain-specific models are simultaneously trained machine learning models trained using a corresponding local loss metric for each domain-specific model and a global loss metric for the plurality of domain-specific models. The operations also include: determining the presence or absence of an intent corresponding to each domain-specific model by classifying the output of each domain-specific model.
[0008] This summary is provided only to introduce certain concepts, without identifying any key or essential features of the claimed subject matter. Other features of the inventive arrangement will be apparent from the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] The arrangement of the present invention is shown by way of example in the accompanying drawings. However, the accompanying drawings should not be interpreted as limiting the arrangement of the present invention to only the specific embodiments shown. Various aspects and advantages will become clear by reading the following detailed description and referring to the accompanying drawings.
[0010] Figure 1 An example communication environment is described in which a system implementing a multiple model architecture operates in accordance with an embodiment.
[0011] Figure 2 An example user device interacting with a system implementing a multiple model architecture over a communication network according to an embodiment is described.
[0012] Figure 3 Networked computing nodes implementing a system execution of a multiple model architecture according to an embodiment are described.
[0013] Figure 4 Certain operational features of a multi-model architecture for determining user intent based on utterances according to an embodiment are schematically illustrated.
[0014] Figure 5 Certain aspects of using machine learning to train a multi-model architecture according to an embodiment are schematically illustrated.
[0015] Figure 6 Some operational features of a multi-model structure for determining conversational intents of multiple users according to an embodiment are schematically illustrated.
[0016] Figure 7 is a flowchart of a method for determining user intent based on speech according to an embodiment.
[0017] Figure 8is a flowchart of a method for training a multi-model structure for determining user intent based on utterances according to an embodiment. DETAILED DESCRIPTION
[0018] Although the present disclosure is summarized with claims defining novel features, it is believed that the various features described in the present disclosure will be better understood by considering the description in conjunction with the accompanying drawings. The processes, machines, manufactures, and any variations thereof described herein are provided for illustrative purposes. The specific structural and functional details described in the present disclosure should not be interpreted as limiting, but merely as a basis for the claims and a representative basis for teaching those skilled in the art to employ the features described in substantially any appropriate detailed structure in different ways. In addition, the terms and phrases used in the present disclosure are not intended to be limiting, but rather to provide an understandable description of the described features.
[0019] The present disclosure relates to machine learning, and more particularly, to a multi-model architecture generated using machine learning. In various embodiments, the multi-model architecture can be implemented in a system for determining multiple intents from a single utterance (text or speech converted to text by a speech-to-text processor) and / or relationships between multiple objects within an image.
[0020] In certain embodiments, the multi-model architecture is the basis for implementing a spoken language understanding (SLU) system. SLU combines elements of signal processing, pattern recognition, machine learning, and artificial intelligence. The goal of SLU is not just to recognize speech, but to extract "meaning" from speech by targeting specific SLU tasks within a specific domain. For example, in the field of flight booking, a SLU system may be responsible for discerning intent from user speech (e.g., "I want to fly from New York to Boston next week"). Therefore, the target understanding task of the system is to extract task-specific parameters in a given frame-based semantic representation, where the semantic representation includes a destination and a departure date.
[0021] Many SLU systems rely most heavily on machine learning, which may reflect the robustness of statistical-based approaches relative to rule-based approaches. Thus, for SLU tasks (such as the one described with respect to flight booking), understanding or extracting intent from an utterance requires a specific machine learning model that has been trained to classify utterances within a specific domain. Traditional SLU systems are trained for a specific task about a single domain, determining a single intent from an utterance. However, systems based on a multi-model architecture can identify multiple intents from multiple domains based on a single utterance.
[0022] Traditional SLU systems operate independently of each other. Therefore, even if multiple traditional SLU systems (each using a separately trained model for a different domain) are combined, it is impossible for the combination of traditional SLU systems to correctly recognize every intent from each individual domain because no system is able to share information with any other system. In contrast, by leveraging collaborative information generated by and shared between multiple models, an SLU system based on a multi-model structure can recognize multiple intents even from a single utterance.
[0023] As defined herein, "intention" is a user goal. The goal can be expressed in user speech or images. As defined herein, "speech" is one or more words contained in text or in speech converted to text by a speech-to-text processor. An example of a speech is a sentence in which if the speech is spoken, the speech is followed by a pause, or if the speech is written, the speech is followed by a period or other punctuation (e.g., a dash). For example, a clear pause after utterance, such as "I'm interested in flying to New York next week", can indicate that the speech is completed with the word "week". If the same speech is written, the speech will end with a period (full stop). A period represents a complete speech, in contrast, for example, a comma represents that the words after the comma are just to clarify the words before the comma. Therefore, in the scene of speech, a clear pause between two separate speech sequences can represent two different sentences, each containing different speech. Another example of distinguishing different speech is an interactive exchange between a user and another entity (human or machine). When the other entity starts speaking, the user's utterance stops, and when the user continues speaking, a new utterance begins. This type of turn-taking during a conversation may occur in scenarios such as interactive voice response (IVR) systems that allow users to interact with a computer by using voice utterances.
[0024] For example, the user intent may be to book a flight, rent a car, or cancel a hotel room, the intent expressed in the user utterance comprising a spoken word or text entered into a device (e.g., a smartphone or other device). The intent in the image is expressed by processing the image using a machine learning model, wherein the machine learning model is trained to recognize images of text and / or images of discrete objects (e.g., airplanes, cars, or hotels) identified by text. Thus, the user intent to book a flight, rent a car, or cancel a hotel room may also be discerned, for example, from text and / or images of objects in a travel brochure, which images may be captured by a camera integrated into a smartphone or other device.
[0025] The example embodiments disclosed herein provide methods, systems, and computer program products that can extract different intents in different domain scenarios from a single utterance. In addition, one or more of the multiple intents may be implicit rather than explicitly expressed, where the unexpressed intent is inferred from other intents explicitly expressed in the single utterance. One of the additional aspects of the disclosed embodiments is to simultaneously determine the relative importance of each of the multiple intents.
[0026] The disclosed methods, systems, and computer program products are also applicable to other fields. For example, in some embodiments, the multi-model structure can provide the basis for a system for detecting relationships between multiple objects within a given image. In other embodiments, the multi-model structure is used to determine one or more mutual intentions based on a conversation between multiple users communicating with each other using separate devices.
[0027] According to one or more embodiments, a multi-model structure can be created by training multiple models (e.g., neural networks) using machine learning. The multi-model structure is created by generating multiple convex coefficients based on outputs generated by models implemented by multiple computers in response to training data sets. One or more of the convex coefficients can be iteratively adjusted using convex coefficient machine learning rules. The parameters of the multiple models are determined based on model-specific local loss metrics in combination with the global loss metric defined herein. The global loss metric can be determined based on the difference between the correct classification of the training data and the estimated classification of the training data generated by multiple computer-implemented models (e.g., neural networks). The estimated classification of the training data can be the sum of the outputs generated by multiple computer-implemented models, each of which is weighted by a corresponding convex coefficient.
[0028] The multi-model structure can be implemented in various task systems, such as performing SLU functions related to human-computer interaction. The SLU function can be, for example, to determine the user's intention based on the user's voice utterance and classify the utterance as part of a call center application or a flight reservation system. The system application involves various types of human-computer interaction, covering a wide range of complex tasks from voice summarization to voice search and voice retrieval. The system can also involve tasks related to human-to-human interaction, such as retrieving calls with a person on a specific topic.
[0029] Such a system relying on a multi-model structure can generate an output from each of a plurality of models in response to a received user input (e.g., an utterance). The system can determine a plurality of user intents based on the user input by classifying the input, which is represented by a feature vector generated by the system based on the input. The system can perform classification by combining the outputs generated by the plurality of models using a multi-model structure. In some embodiments, each of the plurality of models can include an intent-specific neural network. The system can combine the outputs by generating a weighted sum of the outputs generated by the plurality of models, wherein each output is weighted by a convex coefficient. The system can determine each convex coefficient by generating an initial plurality of coefficients and iteratively adjusting one or more of the plurality of coefficients based on a convex coefficient machine learning rule, wherein the convex coefficient machine learning rule is determined using a global loss metric.
[0030] In other embodiments, a system implementing a multi-model structure may be assigned a task, such as identifying relationships between objects appearing in an image. In other embodiments, the system may be tasked with determining the intent of multiple users based on input from multiple devices of the multiple users. Thus, to determine the intent of multiple users, multiple models incorporated into the multi-model structure may each include a user-specific model, and the multi-model classifier may determine the intent of multiple users based on conversations between multiple users performed using multiple devices.
[0031] The different arrangements described herein relate to computer technology and provide improvements to computer technology. For example, the presented arrangements improve the speed and accuracy of computer technology involving determining user intent from relationships between objects within speech utterances and / or images. A system based on a multi-model structure can determine multiple intents from multiple fields based on a single utterance. In addition, the multi-model structure is trained as a single structure, rather than as different models that must all be trained separately. This increases the speed of training using machine learning. It is worth noting that many existing systems rely on neural networks, which are generally believed to require longer training times relative to other machine learning models. Therefore, the presented arrangements can significantly reduce training time by training a single entity (i.e., a multi-model structure comprising multiple neural networks) rather than training multiple entities separately.
[0032] Various aspects of the multi-model architecture are faster training time, higher prediction accuracy, and less susceptibility to the vanishing gradient problem. Regarding training time, despite the predictive power of the models, traditional neural networks are widely regarded as slow to train. The slowness may stem, at least in part, from the conventional training of neural networks using backpropagation. Backpropagation is based on gradient descent optimization, which in the context of neural network training involves calculating the gradient of a nonlinear activation function. In contrast, the multi-model architecture reduces the reliance on backpropagation. Although backpropagation is used to train the multi-model architecture, this use is limited and is largely replaced by iterative adjustment of convex coefficients and a combination of local and global loss metrics. The convex coefficients are adjusted according to the convex coefficient machine learning rules described here. Backpropagation involves calculating gradients, and the updates to the convex coefficients are based on linear difference equations that provide relatively fast convergence.
[0033] Regarding the vanishing gradient problem, conventional training using gradient descent (e.g., using the chain rule to compute partial derivatives) involves updating the parameters of the neural network in proportion to the partial derivatives of the cost function. In some cases, the gradients (partial derivatives) can be very small or "almost zero," which can effectively prevent the update of the neural network parameters or terminate training entirely. In contrast, the multi-model architecture training described in this article relies on a global loss metric as well as a local loss metric. The global loss metric (a convex combination of functions) is itself a convex function. Faster convergence when updating convex coefficients reduces the likelihood of vanishing gradients.
[0034] Other aspects of the embodiments described in the present disclosure are described in more detail with reference to the following drawings. For simplicity and clarity of illustration, the elements shown in the drawings are not necessarily drawn to scale. For example, for clarity, the size of some elements may be exaggerated relative to other elements. In addition, where deemed appropriate, reference numerals are repeated in the drawings to indicate corresponding, similar or similar features.
[0035] Figure 1 An example communication environment 100 according to an embodiment is described in which a system including a multi-model structure operates. The communication environment 100 illustratively includes a user device 102, which can receive a speech utterance 106 and other input (e.g., text) from a user 104. The user device 102 can be a smart phone, a virtual assistant, or other such device. The communication environment 100 also illustratively includes a server 108. A system 110 is executed on the server 108, as described in more detail below, and the system 110 implements a multi-model structure for determining intent from the utterance of the user 104 (text or speech converted to text by a speech-to-text processor) and / or the relationship between multiple objects within an image captured by the user device 102.
[0036] The user device 102 communicates with the server 108 (illustratively a cloud-based server) via a data communication network 112. The data communication network 112 can provide communication links between various devices and data processing systems. The communication link can include a connection, such as a wired communication link, a wireless communication link, or a fiber optic cable. The communication link can be implemented as or include one or more different communication technologies (or any combination), such as a wide area network (WAN), a local area network (LAN), a wireless network (e.g., a wireless WAN and / or a wireless LAN), a mobile or cellular network, a virtual private network (VPN), the Internet, a public switched telephone network (PSTN), etc. Devices that can be coupled to the data communication network 112 via wired and / or wireless communication links can include smart phones, virtual assistants, personal computers, portable computing or communication devices, network computers, tablet computers, mobile phones, etc.
[0037] Figure 2 A user device 200 is depicted, which represents a device such as Figure 1 The user device 200 illustratively includes a memory 202, one or more processors 204 (e.g., an image processor, a digital signal processor, a data processor), and an interface circuit 206. The memory 202, the processor 204, and / or the interface circuit 206 are implemented as separate components in some embodiments. In other embodiments, the memory 202, the processor 204, and / or the interface circuit 206 are integrated in one or more integrated circuits. The various components of the user device 200 can be coupled, for example, via one or more communication buses or signal lines (e.g., interconnects and / or wires). In one embodiment, the memory 202 is coupled to the interface circuit 206 via a memory interface (not shown).
[0038] User device 200 illustratively includes a camera subsystem 208 that may be coupled to an optical sensor 210. Optical sensor 210 may be implemented using any of a variety of technologies. Examples of optical sensor 210 include a charge coupled device (CCD), a complementary metal-oxide semiconductor (CMOS) optical sensor, etc. Camera subsystem 208 and optical sensor 210 may be used to implement camera functions, such as recording images and / or video clips.
[0039] Communication functionality may be implemented through one or more wireless communication subsystems 212. The wireless communication subsystems 212 may include radio frequency receivers and transmitters, optical (e.g., infrared) receivers and transmitters, etc. The specific design and implementation of the wireless communication subsystems 212 may depend on the specific type of user device 200 implemented and / or the type of communication network on which the user device 200 is intended to operate.
[0040] For the purpose of illustration, the wireless communication subsystem 212 can be designed to operate on one or more mobile networks (e.g., GSM, GPRS, EDGE), Wi-Fi networks that can include WiMax networks, short-range wireless networks (e.g., Bluetooth networks), and / or any combination of the foregoing. The wireless communication subsystem 212 can implement a hosting protocol so that the user device 200 can be configured as a base station for other wireless devices.
[0041] The audio subsystem 214 can be coupled to a speaker 216 and a microphone 218 to implement voice-enabled functions such as voice recognition, voice replication, digital recording, audio processing, and telephony functions. The audio subsystem 214 represents a subsystem used with the embodiments described below to receive spoken utterances of a device user. The audio subsystem 214 can generate various types of audio data.
[0042] Input / output (I / O) devices 220 may be coupled to interface circuitry 206. Examples of I / O devices 220 include, for example, a display device, a touch-sensitive display device, a trackpad, a keyboard, a pointing device, a communication port (e.g., a USB port), a network adapter, a button or other physical control, etc. Touch-sensitive devices such as display screens and / or display panels are configured to detect contact, movement, interruption of contact, etc. using any of a variety of touch-sensitive technologies. Example touch-sensitive technologies include, for example, capacitive, resistive, infrared, and surface acoustic wave technologies, other proximity sensor arrays, or other elements for determining one or more points of contact with a touch-sensitive device, etc. One or more I / O devices 220 may be adapted to control functionality of sensors, subsystems, etc. of user device 200.
[0043] The user device 200 also includes a power supply 240. The power supply 240 is capable of providing power to various elements of the user device 200. In one embodiment, the power supply 240 is implemented as one or more batteries. The battery can be implemented using any of a variety of different battery technologies, whether disposable (e.g., replaceable) or rechargeable. In another embodiment, the power supply 240 is configured to obtain power from an external power source and provide power (e.g., DC power) to the elements of the user device 200. In the case of a rechargeable battery, the power supply 240 may also include a circuit capable of charging one or more batteries when coupled to an external power source.
[0044] The memory 202 may include random access memory (e.g., volatile memory) and / or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, flash memory, etc. The memory 202 may store an operating system 252, such as LINUX, UNIX, a mobile operating system, an embedded operating system, etc. The operating system 252 may include instructions for handling system services and for performing hardware-dependent tasks.
[0045] The memory 202 may additionally store other program code 254. Examples of other program code 254 may include: instructions for implementing communication with one or more additional devices, one or more computers, and / or one or more servers; graphical user interface processing; processing instructions for implementing sensor-related functions; phone-related functions; electronic messaging-related functions; web browsing-related functions; media processing-related functions; GPS and navigation-related functions; security functions; camera-related functions, including web camera and / or web video functions; etc. The memory 202 may also store one or more other applications 262.
[0046] The memory 202 may store voice interaction program code 256. The voice interaction program code 256 may be used, for example, by a user to audibly interact with a network-based system (e.g., an airline, car, and / or hotel reservation system). The user device 200 may be configured to communicate with the user through, for example, Figure 1 A data communication network such as the data communication network 112 of FIG. 1 is communicatively coupled to such a system. Thus, the user device 200 can interact with the network-based system via the data communication network through spoken utterances.
[0047] The various types of instructions and / or program codes described are provided for purposes of illustration and not limitation. The program code may be implemented as a separate software program, process, or module. The memory 202 may include additional instructions or fewer instructions. In addition, the various functions of the user equipment 200 may be implemented in hardware and / or software, including implementation in one or more signal processing and / or application specific integrated circuits.
[0048] The program code stored in the memory 202 and any data used, generated and / or operated by the user device 200 are functional data structures that impart functionality to the device when used as part of the device. Other examples of functional data structures include, for example, sensor data, data obtained through user input, data obtained by querying external data sources, baseline information, etc. The term "data structure" refers to the physical implementation of a data model to organize data in a physical memory. Thus, a data structure is formed by specific electrical or magnetic structural elements within the memory. The data structure physically organizes the data stored in the memory for use by the processor.
[0049] Depending on the particular type of system implemented, user equipment 200 may include more than Figure 2 Fewer components than those shown or including in addition to Figure 2Additional components other than those shown. In addition, the specific operating system and / or application and / or other program code included may also vary depending on the system type. In addition, one or more illustrative components may be incorporated into another component or otherwise form a part of another component. For example, a processor may include at least some memory.
[0050] User device 200 is provided for purposes of illustration and not limitation. Devices and / or systems configured to perform the operations described herein may have Figure 2 . The architecture may be a simplified version of the user device 200 and may include a processor and a memory storing instructions. Example implementations of the user device 200 include, for example, a smartphone or other mobile device, a wearable computing device (e.g., a smart watch), a computer (e.g., a desktop, a laptop, a tablet computer, a virtual assistant, a smart TV, other data processing systems), or other electronic devices. It should be understood that the embodiments may be deployed as a standalone device or as multiple devices in a distributed client-server networking system. For example, in some embodiments, a smart watch may be operably coupled to a mobile device (e.g., a phone). The mobile device may or may not be configured to interact with a remote server and / or computer system.
[0051] Such as Figure 2 A user device such as user device 200 may be communicatively coupled to a networked server or other computing node via a data communications network. A system executed on a networked computing node may implement a multi-model structure for determining intent from a user's utterance (spoken or text-based) (e.g., voice interaction code 256) and / or relationships between multiple objects within an image captured by the user device as described below (e.g., one or more other applications 262).
[0052] Figure 3 An example computing node 300 is illustrated. In one or more embodiments, computing node 300 is an exemplary cloud-based computing node (e.g., a cloud-based server). Computing node 300 can perform any functions described in this disclosure and is not intended to imply any limitation on the scope of use or functionality of the embodiments of the invention described herein.
[0053] Computing node 300 illustratively includes computer system 312, which can operate with many other general or special purpose computing system environments or configurations. Examples of known computing systems, environments, and / or configurations that may be suitable for computer system 312 include, but are not limited to, smartphones, virtual assistants, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems or devices, etc.
[0054] Computer system 312 can be described in the general context of a computer system executing computer system executable instructions, such as program modules. In general, program modules can include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer system 312 can be practiced in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located in local and remote computer system storage media, including memory storage devices.
[0055] like Figure 3 As shown, computer system 312 is shown in the form of a general-purpose computing device (e.g., a server). Components of computer system 312 may include, but are not limited to, one or more processors 316, memory 328, and a bus 318 coupling various system components including memory 328 to processor 316. The processor includes a hardware circuit configured to execute instructions. The hardware circuit may be an integrated circuit. Examples of processors include, but are not limited to, a central processing unit (CPU), an array processor, a vector processor, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic array (PLA), an application specific integrated circuit (ASIC), a programmable logic circuit, and a controller.
[0056] Bus 318 represents one or more of a variety of types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example only and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, a Peripheral Component Interconnect (PCI) bus, and a PCI Express (PCIe) bus.
[0057] Computer system 312 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system 312, and can include volatile and nonvolatile media, removable and non-removable media.
[0058] The memory 328 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 330 and / or cache memory 332. The computer system 312 may also include other removable / non-removable, volatile / non-volatile computer system storage media. For example, a storage system 334 may be provided for reading from and writing to a non-removable, non-volatile magnetic media and / or a solid-state drive (not shown and generally referred to as a "hard drive"). Although not shown, a magnetic drive for reading from and writing to a removable, non-volatile disk (e.g., a "floppy drive") and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided, where the optical disk is, for example, a CD-ROM, DVD-ROM, or other optical media. In this case, each medium may be connected to the bus 318 via one or more data media interfaces. As will be further depicted and described below, the memory 328 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of an embodiment of the present invention.
[0059] By way of example and not limitation, a program / application 340 having a set (at least one) of program modules 342 can be stored in memory 328 together with an operating system, one or more application programs, other program modules, and program data. Each or some combination of the operating system, one or more application programs, other program modules, and program data can include an implementation of a networking environment. Program modules 342 generally perform the functions and / or methods of embodiments of the present invention as described herein. In particular, one or more program modules can implement a system for determining intent from a user's speech utterance and / or a relationship between multiple objects within an image based on one or more multi-model structures described herein.
[0060] Program / application 340 is executable by processor 316. Program / application 340 and any data items used, generated, and / or manipulated by computer system 312 are functional data structures that impart functionality when used by computer system 312.
[0061] The computer system 312 may also communicate with one or more external devices 314, such as a keyboard, pointing device, display 324, etc.; one or more devices that enable a user to interact with the computer system 312; and / or any device that enables the computer system 312 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur through an I / O interface 322. However, the computer system 312 may communicate with one or more networks (e.g., a LAN, a general WAN, and / or a public network (e.g., the Internet)) through a network adapter 320. As shown, the network adapter 320 communicates with other components of the computer system 312 via a bus 318. It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with the computer system 312. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archiving storage systems, etc.
[0062] Although computing node 300 is used to illustrate an example of a cloud computing node, it should be understood that the use of Figure 3 Computer systems with the same or similar architectures as described may also be used in non-cloud computing implementations to perform the various operations described herein. In this regard, the example embodiments described herein are not intended to be limited to cloud computing environments. Computing node 300 is an example of a data processing system. As defined herein, a "data processing system" refers to one or more hardware systems configured to process data, each hardware system including at least one processor and memory programmed to initiate operations.
[0063] Computing node 300 is an example of computer hardware. Computing node 300 may include Figure 3 Fewer components shown or Figure 3 Additional components not shown in the drawings may be included, depending on the specific type of device and / or system implemented. The specific operating system and / or applications included may vary depending on the device and / or system type, as may the types of I / O devices included. In addition, one or more of the illustrative components may be incorporated into or otherwise form part of another component. For example, a processor may include at least some memory.
[0064] Figure 4 Schematically illustrates certain operational features of a multi-model architecture 400 according to one embodiment. The multi-model architecture 400 can be implemented in a system running on a server or other networked computing node. Based on the multi-model architecture 400, the system can determine user intent based on user utterances transmitted to the system over a data communication network.
[0065] Intent determination can be relevant in many different contexts. For example, in the scenario of an automated reservation system, a user utterance can express multiple intents. For example, a user utterance conveyed via a smartphone or virtual assistant might take the form "Can you book me a flight from New York to San Francisco on May 1, stay at the Hilton for two days, and rent a car using Hertz? And can you tell me the price?". This user utterance expresses four separate intents: book a flight; rent a car; book a hotel room; and determine the price of each. Two intents are only implicit, referencing Hilton instead of hotels and Hertz instead of car rentals. The last intent only implicitly refers to the separate prices for flights, car rentals, and hotels, referencing only the price in the singular. The multi-model structure 400 operates to discern each intent from separate domains (flights, car rentals, and hotels) as well as shared features common to each domain, and determine the price.
[0066] Typically, the multi-model structure 400 generates output y from k models, namely model 402a, model 402b and model 402k respectively. 1 ,y 2 , …, y k . Although only models 402a, 402b, and 402k are shown, it should be understood that any number of models may be incorporated into the multi-model structure 400. Each model 402a-402k corresponds to a specific domain (e.g., flights, car rentals, hotel reservations, prices). Each output is generated in response to a user utterance (spoken or text-based), which the system preprocesses into a feature vector x. The feature vector x is input as a whole into each domain-specific model 402a-402k. The output generated by each domain-specific model 402a-402k based on the input of the feature vector x may be a real number between 0 and 1, reflecting the probability that the corresponding intent exists in the user utterance represented by the feature vector x. The output y 1 ,y 2 , …, y k is input to the intention determiner 404. The intention determiner 404 converts each output y 1 ,y 2 , …, y k Multiply by the corresponding coefficient α 1 , α 2 , …, α k , these coefficients are determined as described in detail below. In certain embodiments described in detail below, these coefficients comprise a convex set of coefficients.
[0067] In one or more embodiments, intent determiner 404 is a multi-input binary classifier that classifies the output of each domain-specific model into one of two states: intent present or intent not present. i y iIf the feature vector x is greater than a predetermined threshold (e.g., 0.5), the intent determiner 404 determines that the corresponding intent is likely to be expressed in the user utterance represented by the feature vector x. Therefore, the intent determiner 404 generates a k-dimensional vector [I ] with 0s and 1s based on each output of each domain-specific model 402a-402k. 1 , I 2 ,…,I k ], 1 indicates that there is an intent in the user utterance. For example, in the scenario of an automatic booking system, if the multi-model structure 400 models the domain "flights, car rentals, hotel reservations, flight prices, rental prices, hotel prices", then a user utterance feature vector that generates a vector [1, 0, 1, 1, 0, 0] indicates that the user may intend to book both a flight and a hotel, but is only requesting a flight price. Once the intent is determined based on the multi-model structure 400, the automatic booking system performs downstream tasks such as tagging, filling in empty seats, and extracting the details required to complete the booking request.
[0068] One aspect of the multi-model architecture 400 is to use collective information from multiple domain-specific models to create a single architecture, i.e., a multi-model architecture, using machine learning. The multi-model architecture provides enhanced training speed. The multi-model architecture converges in fewer training steps or periods, and once trained, provides similar or better prediction (or classification) accuracy than a single trained model. The multi-model architecture provides other advantages described below.
[0069] Figure 5 Schematically illustrates certain aspects of using machine learning to create a multi-model structure 500 according to an embodiment. Illustratively, the multi-model structure 500 includes domain-specific models 502a, 502b, and 502k. Although only three domain-specific models are shown, it should be understood that any number of domain-specific models can be incorporated into the multi-model structure 500. Each domain-specific model 502a-502k is trained using feedback based on a local loss metric, each local loss metric corresponding to one of the domain-specific models. Training is also based on a global loss metric, which iteratively adjusts a set of convex coefficient estimators 504. During training using a set of example inputs (feature vectors), the output (value) generated by the model is compared to the true value or target value corresponding to the example input. Based on the difference between the value generated by the model and the true value or target value, the error is determined, and the error is used to iteratively update the parameters and convex coefficients 504 of each domain-specific model 502a-502k. Now, the training process is described in the context of each domain-specific model containing a neural network. However, in other embodiments, different machine learning models can be used to create the multi-model structure 500.
[0070] A neural network (or artificial neural network) is an algorithmic or circuit-based approach that provides a robust method for approximating or "learning" a real-valued, discrete-valued, or vector-valued target function based on a set of training examples. Neural network structures typically consist of layers consisting of interconnected nodes and activation functions. Patterns are presented to the neural network at an input layer, which communicates with one or more hidden layers that perform processing through a weighted system of connections. The hidden layers are linked to an output layer that produces the results of the processing. Operationally, the output is a mapping of input data to predictions. The loss function compares the predictions to the target values, generating a loss value, which is a measure of how closely the predictions match the correct value for the example. Backpropagation neural networks use gradient descent to adjust the parameters of the network to best fit a set of input-output pairs (x, y), where the input x is an n-dimensional vector and y is the output generated by the neural network based on the input values.
[0071] exist Figure 5 In the example, models 502a, 502b and 502k are neural networks N 1 (w 1 ,x)、N 2 (w 2 , x) and N k (w k , x). Although only three such neural networks are shown, it should be understood that any number of neural networks may be incorporated into the multi-model architecture 500. Each N i (w i , x) classifies the input vector x by computing the value y, which represents the label assigned to x. y is based on the weights w i Determined. Weight w i It is determined based on machine learning rules, and its initial value can be set arbitrarily. According to the machine learning rules, the weights are adjusted iteratively until N i (w i , x) can predict the output with an acceptable level of accuracy. i is determined using a set of training examples containing input-output data pairs (x, y), where each input x corresponds to the label y correctly assigned to the input. In successive iterations, the neural network N i (w i , x) determines the output The local loss metric determines the model's predictions. The closeness to the target y, and the weight w is adjusted based on machine learning rules i A separate test data set was used to determine the i (w i , x) accuracy.
[0072] During training using machine learning, Yes i estimates, and is the i-th neural network N i (w i , x) is the predicted output. Therefore,
[0073]
[0074] Weight w i The output error can be t is trained (iteratively conditioned) by backpropagation of t, where t is the index of the step or training epoch. Based on machine learning using backpropagation, That is, as the number of steps or training epochs increases (approaching infinity), the predicted output for each input x Close to the true value y t .
[0075] Instead of using each prediction output To predict the output y t , but use The combination of (called convex combination) as y at each step t t Estimates:
[0076]
[0077] in if The i-th coefficient (for i=1,...,k) is the i-th target convexity coefficient α i The estimate at step t, where α i The following conditions are met:
[0078]
[0079] Target coefficient α 1 , α 2 , …, α k Through the convex coefficient estimator The set 504 is estimated. Convex coefficients are coefficients that satisfy the following convexity criteria:
[0080]
[0081] In addition, based on convexity, the estimated convex coefficient The convex domain is constrained at step t as follows:
[0082]
[0083] Where k is the neural network Ni (w i , the number of x).
[0084] The multi-model architecture 500 is trained using two different loss functions, which are used simultaneously to train each model 502a-502k simultaneously (illustratively, the neural network N i (w i ,x), i=1,2,...,k). A loss function is a local loss metric Another loss function is the global loss metric g t For each model 502a-502k, there is a corresponding loss function. The local loss metric is defined as i (w i , x)’s output cross entropy:
[0085]
[0086] in is the predicted probability of the jth label of the ith neural network at step t, and m is the output y i The total number of categories (labels) (in the case of a binary classifier, as above, there is only one category).
[0087] The global loss metric g t Defined as the true or correct value (label) y t and predicted value (label) The difference between:
[0088]
[0089] Based on the convexity constraint, the global loss metric can be rewritten as follows:
[0090]
[0091] in is the i-th model output at step t And will It is defined as the difference between the target output and the i-th model output at step t:
[0092]
[0093] The difference between the i-th neural network (or other model) loss function and the last (k-th) neural network (or other model) loss function is It is defined as follows:
[0094]
[0095] The loss function derived from Equation 8 can be further simplified as follows:
[0096]
[0097] Among them will (for m classes and k neural networks or other machine learning models) is defined as And define the convex coefficient vector as
[0098] Local loss metrics and global loss metrics are used to train neural networks (or other machine learning models) simultaneously. Local loss metrics is used to adapt the weights by back-propagating the error term Adjust to train each individual neural network The global loss metric g t And the derived global loss metric matrix are used together to simultaneously train the convex coefficient vector
[0099] Use the exported global loss metric matrix Training the convex coefficients provides significant advantages as described below. Multiplying both sides of Equation 11 by This produces:
[0100]
[0101] The machine learning rule for convex coefficients is derived from Equation 12:
[0102]
[0103] in Therefore, for the convex coefficient The machine learning rules are as follows:
[0104]
[0105] Therefore, as described, using the local loss metric (e.g. cross entropy) by back propagation for each neural network N i (w i , x) for training, the i-th local loss metric corresponds to the i-th neural network. A global loss metric is generated based on the output of each of the multiple neural networks (k-1×m matrix), and the global loss metric is used to train the convex coefficients The convex coefficients satisfy the condition at each iteration step t As mentioned above, local loss metrics and global loss metrics are used to train neural networks (or other machine learning models) simultaneously. Global loss metric provides the basis for the convex coefficient machine learning rules. During training, the global loss metric Determine how close the value predicted by the multi-model architecture is to the true value or target value of the training example, and adjust the multi-model architecture parameters accordingly. The step-wise approach iteratively adjusts model parameters until the model achieves an acceptable level of predictive accuracy when applied to a test dataset.
[0106] The following pseudocode implements an algorithm for training a multi-model architecture according to the aspects described:
[0107]
[0108] The backpropagation used to train each individual domain-specific model (e.g., neural network) is based on gradient descent optimization, which involves computing the gradient of a nonlinear activation function: a relatively slow process that requires more time than adjusting the convex coefficients. The convex coefficient machine learning rule (Equation 15) that updates the convex coefficients is a linear difference equation that provides relatively fast convergence. Once the convex coefficients converge, the multi-model architecture trained according to the described process produces model prediction outputs that are close to the true target value. The global loss metric is the function is a convex combination of , and hence a convex function. Thus, some advantages of the multi-model architecture are faster learning and a greater likelihood of avoiding the vanishing gradient problem due to faster convergence.
[0109] At each training step, the estimated convex coefficients satisfy the constraints Therefore, only k-1 convex coefficients need to be calculated at each step t.
[0110] It is important to note that in order to form a convex hull based on the outputs of multiple domain-specific models, the number of models k needs to be at least 1 greater than the number of output categories or labels n. That is, k ≥ n. Although for the above binary classifier, n is 1, so the constraint is easily satisfied. In general, although the constraint seems to increase the total number of required parameters, in practice, as verified by simulations, for multiple neural networks, a smaller number of hidden layers can be used with the multi-model structure. In addition, as verified by simulations, even better performance can be achieved using a smaller number of hidden layers. Even if the constraints are not met, the multi-model structure can still be trained, but there may be no real-valued solutions, only numerical approximations, which will have an adverse effect on performance.
[0111] Global loss metric One aspect of is to generate collaborative information based on integrating the information generated by separate domain-specific models during training. For example, a SLU system using a multi-model architecture can be trained by leveraging collaborative information to identify multiple intents in a single utterance. The global loss metric This enables the SLU system to overcome the limitations of traditional SLU systems, which operate independently of each other even when combined, and therefore cannot integrate or combine information from one model with information from any other model. The collaborative information of the multi-model structure is reflected in the set of convex coefficients (α 1 , α 2 , ..., α k ), as mentioned above, the convex coefficients are used to quantize the output y of each of the k domain-specific models in the multi-model architecture separately. 1 ,y 2 , …, y k Each product term α i y i Determine whether it is possible to express the corresponding intent in a single user utterance represented by the feature vector x.
[0112] Thus, for example, according to Figure 4 In the context of the embodiment described above, if α i y i If y is greater than a predetermined threshold (e.g., 0.5), then the utterance represented by the feature vector x expresses an intention corresponding to the i-th domain. However, this determination is not based solely on the i-th model determining y i , but rather the coefficient α determined based on the i-th model and trained based on the global loss metric i The global loss metric incorporates the influence of other models of the multi-model structure. Therefore, the multiple models of the multi-model structure are not independent, but each model jointly reflects the information embodied by each of the other models. In the context of the SLU system, this aspect of the multi-model structure enables the SLU system containing the structure to overcome the limitations of traditional SLU systems, which operate independently of each other even when combined and cannot integrate or combine information from multiple models.
[0113] In various embodiments, the multi-model architecture may be extended to determine one or more intents of multiple users based on utterances expressed (via speech or text) during a conversation between two or more users. Figure 6The diagram illustrates certain operational features of a multi-model architecture 600 according to an embodiment, wherein the multi-model architecture 600 operates jointly between multiple devices (e.g., smartphones) of multiple users to provide conversational intent detection. The multi-model architecture 600 illustratively includes user-specific models 602a, 602b, 602k corresponding to different users each using a separate device, and a multi-model intent determiner 604 running on a networked server that is communicatively coupled to each separate device via a data communications network. Although only three models corresponding to three users are shown, it should be understood that the number of models can be any number greater than one. Each model 602a-602k can, for example, be operably coupled to a virtual assistant running on a corresponding device. Based on the above process, the multi-model architecture 600 can determine one or more intents from a user utterance. 1 ,I 2 and I k , these intents are communicated separately to a virtual assistant running on the user device - collectively referred to as virtual assistant 606.
[0114] A conversation between two or more users can trigger a virtual assistant running on each device of the users participating in the conversation. Each such virtual assistant is customized for a specific user and contains that person's preferences (e.g., phone information, web browsing information), which can be considered when making mutual decisions involving each user participating in the conversation. The multi-model structure 600 can determine the intent of each user based on the output of the user-specific models 602a, 602b, and / or 602k generated in response to the corresponding user's utterance. For example, user Joan may ask a virtual assistant running on her device (e.g., a smartphone): "Can you help me and Amy book a flight from New York to San Francisco next Saturday? Book Hertz and stay at Hilton for two days. Can you give me a quote?" Based on the conversation, the multi-model structure 600 determines the intent related to Amy and provides the intent to Joan's virtual assistant, which may contact Amy using personalized phone information available from Joan's device.
[0115] The processing of Joan's utterance depends on Amy's consent. If the multi-model structure 600 determines that Amy's intention is to join Joan on the proposed trip, then processing continues with Joan's originally expressed intention. Otherwise, if Amy responds negatively, processing terminates. For example, if Amy responds by saying, for example, "I'd love to go, but I have a scheduling conflict," the multi-model structure 600 determines that Amy's intention is not to join Joan on the trip, and no further processing occurs based on the intention determined by the model. In contrast, if Amy responds to the virtual assistant's inquiry by saying "Yes, I'm free those days and would like to go," then processing continues. Thus, the multi-model structure 600 determines Amy's intention, which confirms that she wishes to travel with Joan to the location indicated on the indicated data, and in response, the multi-model structure 600 determines from Joan's utterance that Joan's intention is to book a flight for herself and Amy, book a rental car (although this is not explicitly stated), book a hotel room for two days starting on Saturday (although this is also not explicitly stated), and obtain the price of each item (inferred from the single reference to the price). The user's virtual assistant 606 can complete actions (booking tickets, reserving rooms, renting cars, and providing pricing information) based on mutual decisions made in response to the user's intent.
[0116] In other embodiments, a multi-model structure can be used to determine one or more user intents based on an image. The multi-model structure can determine one or more user intents from an image based on outputs generated by multiple domain-specific models (e.g., deep learning neural networks). Deep learning neural networks are often used to classify camera images. Using backpropagation, neural networks can learn various objective functions for classifying images. For example, images with a resolution of 120×128 and each image pixel described by a grayscale intensity between 0 (black) and 255 (white) have long been used to train neural networks to classify images based on facial expressions, identify different people in images, determine the direction in which the person in the picture is looking, identify whether the person is wearing sunglasses, and even classify people by gender. The image can be encoded as a 3-D array and labeled with a corresponding 1-D array classifier label. The multi-model structure for determining intent based on an image can use multiple domain-specific deep learning neural networks trained according to convex coefficient machine learning rules using a local loss metric and a global loss metric as described above.
[0117] For example, a user may capture an image from a travel brochure using, for example, a camera integrated into the user's smartphone or other device. The device may be communicatively coupled to a network server via a data communications network that hosts a system implementing a multi-model structure. The multi-model structure may be trained to identify one or more intents based on outputs from, for example, multiple domain-specific deep learning neural networks trained according to the above process. The device may transmit an image to a system hosted on a network server. The deep learning neural network may, for example, identify from image text describing an island resort and an image of a hotel near a beach. The multi-model structure may determine one or more intents from the image. For example, the multi-model structure may be trained to determine a user intent to obtain more information about the quality of a resort and a nearby beach (e.g., via the Internet). For example, the multi-model structure may be trained to determine a user intent to obtain pricing information about ticket prices for an island and room rentals at a resort. In various embodiments, the multi-model structure may be trained to determine various other intents based on various other types of images.
[0118] Figure 7 700 is a flowchart of an example method 700 for determining user intent from speech (voice or text) according to an embodiment. The method 700 may be performed by referring to Figures 1 to 6 The system may be performed by a system the same or similar to the system described in the foregoing. Illustratively, the system generates outputs from a plurality of domain-specific models in response to received inputs at block 702. The domain-specific models are simultaneously trained machine learning models trained using a corresponding local loss metric for each domain-specific model and a global loss metric for the plurality of domain-specific models. At block 704, the system determines the presence or absence of an intent corresponding to each domain-specific model, the determination being based on classifying the output of each domain-specific model.
[0119] The domain-specific models can be trained simultaneously by iteratively adjusting the parameters of each domain-specific model based on a series of error terms, each error term being determined by the difference between the model prediction value and the true value. Each model prediction value can be a sum of products, each product being the output of the domain-specific model multiplied by a convex coefficient corresponding to the domain-specific model.
[0120] The domain-specific model may be trained simultaneously by generating an initial plurality of convex coefficients and iteratively adjusting one or more of the plurality of coefficients based on a convex coefficient machine learning rule determined using a global loss metric.
[0121] In some embodiments, each domain-specific model is a neural network. In some embodiments, the received input includes an utterance. The utterance may be text or speech converted to text using a speech-to-text processor. In other embodiments, the received input may include one or more images.
[0122] In other embodiments, the received input may include input from multiple devices of multiple users. To determine the intent of multiple users based on utterances expressed (by voice or text) during a conversation between two or more users, the domain-specific model may include a user-specific model. The multi-model structure may determine one or more user intents based on a conversation between multiple users performed using multiple devices.
[0123] Figure 8 is a flowchart of an example method for training a multi-model structure according to an embodiment. Figures 1 to 6 The method 800 may be performed by a system that is the same as or similar to the system described above. The system generates a plurality of convex coefficients at block 802 based on outputs generated by a machine learning model implemented by a plurality of computers in response to receiving training data. At block 804, the system iteratively adjusts one or more of the plurality of convex coefficients using a convex coefficient machine learning rule. The convex coefficient machine learning rule is determined based on a global loss metric, as derived from equations 12-14 above.
[0124] Each machine learning model can be trained using a corresponding local loss metric and a global loss metric. The training of the machine learning model can be performed simultaneously with the iterative adjustment of one or more of the multiple convex coefficients.
[0125] A global loss metric may be determined based on a difference between a correct classification of the training data and an estimated classification of the training data generated by the plurality of computer-implemented models. The estimated classification of the training data may include a sum of outputs generated by the plurality of computer-implemented machine learning models, each output weighted by a respective one of the plurality of convex coefficients.
[0126] The plurality of computer-implemented models may include neural networks. Each neural network may be trained by back-propagation of an error calculated based on a local loss metric, the error being the difference between a correct classification of the training data and an estimated classification of the training data generated by each neural network. Training of each neural network may be performed concurrently with iteratively adjusting one or more of the plurality of convex coefficients. The local loss metric may be a cross entropy metric.
[0127] Once trained, the accuracy of the plurality of machine learning models may be determined based on classifying a test data set. Elements of the test data set may be classified based on a weighted sum of outputs generated by the machine learning models, where each output is weighted by a corresponding one of a plurality of convex coefficients.
[0128] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. Nevertheless, several definitions are now provided throughout the document.
[0129] As defined herein, singular forms of terms are intended to include plural forms as well, unless the context clearly indicates otherwise.
[0130] As defined herein, "another" means at least a second or more.
[0131] As defined herein, unless expressly stated otherwise, "at least one," "one or more," and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," and "A, B, and / or C" refers to A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.
[0132] As defined herein, "automatically" means without user intervention.
[0133] As defined herein, "computer-readable storage medium" refers to a storage medium containing or storing program code for use by or in conjunction with an instruction execution system, device or apparatus. "Computer-readable storage medium" as defined herein is not itself a transient, propagating signal. A computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. As described herein, a memory element is an example of a computer-readable storage medium. A non-exhaustive list of more specific examples of computer-readable storage media may include: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a digital versatile disk (DVD), a memory stick, a floppy disk, etc.
[0134] As defined herein, “including,” “comprising,” and / or “having” specifies the presence of recited features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.
[0135] As defined herein, “if” means “when” or “upon” or “in response to” or “in response to,” depending on the context. Thus, the phrase “if it is determined” or “if [stated condition or event] is detected” may be interpreted as “upon determining” or “in response to determining” or “upon detecting [stated condition or event]” or “in response to detecting [stated condition or event]” or “in response to detecting [stated condition or event],” depending on the context.
[0136] As defined herein, "one embodiment," "an embodiment," "in one or more embodiments," "in a specific embodiment," or similar language means that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment described in the present disclosure. Thus, appearances of the foregoing phrases and / or similar language throughout the present disclosure may, but do not necessarily, all refer to the same embodiment.
[0137] As defined herein, "output" means stored in a physical memory element (eg, a device), written to a display or other peripheral output device, sent or transmitted to another system, exported, and the like.
[0138] As defined herein, a "processor" refers to at least one hardware circuit configured to execute instructions contained in a program code. The hardware circuit may be an integrated circuit. Examples of processors include, but are not limited to, a central processing unit (CPU), an array processor, a vector processor, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic array (PLA), an application specific integrated circuit (ASIC), a programmable logic circuit, and a controller.
[0139] As defined herein, "response" and "in response to" refer to being susceptible to responding or reacting to an action or event. Thus, if a second action is performed "in response to" or "in response to" a first action, there is a causal relationship between the occurrence of the first action and the occurrence of the second action. The terms "response" and "in response to" indicate a causal relationship.
[0140] As defined herein, "substantially" means that the recited features, parameters or values need not be achieved precisely, but deviations or changes including, for example, tolerances, measurement errors, measurement accuracy limitations, and other factors known to those skilled in the art may occur in an amount that does not interfere with the effect intended to be provided by the described features.
[0141] As defined herein, the terms "user" and "individual" refer to a human being.
[0142] The terms first, second, etc. may be used herein to describe various elements. These elements should not be limited by these terms, as these terms are only used to distinguish one element from another, unless otherwise stated or the context clearly indicates otherwise.
[0143] Various embodiments of the aspects of the invention disclosed herein can be implemented in a system, as a method and / or in a computer program product at any possible level of integrated technical detail. A computer program product may include a computer-readable storage medium (or medium) having computer-readable program instructions thereon, and the computer-readable program instructions are used to cause a processor to perform various aspects of the embodiments disclosed herein. In the present disclosure, "program code" and "computer-readable program instructions" can be used interchangeably. The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device or downloaded to an external computer or external storage device via a network, such as the Internet, a LAN, a WAN and / or a wireless network. The network may include a copper transmission cable, an optical transmission fiber, a wireless transmission, a router, a firewall, a switch, a gateway computer and / or an edge device including an edge server. The network adapter line or network interface in each computing / processing device receives a computer-readable program instruction from the network and forwards the computer-readable program instruction to be stored in a computer-readable storage medium using a corresponding computing / processing device.
[0144] The computer-readable program instructions for performing the operation of the invention arrangement disclosed herein can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages (including object-oriented programming languages and / or process programming languages). The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or executed completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including LAN or WAN, or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In certain embodiments, the electronic circuit including, for example, a programmable logic circuit, FPGA or PLA can be executed by customizing the electronic circuit using the state information of the computer-readable program instructions to execute the computer-readable program instructions, so as to perform the various aspects of the invention arrangement described herein.
[0145] Some aspects of the arrangement of the present invention are described herein with reference to the flow chart and / or block diagram of the method, device (system) and computer program product of the embodiment arranged according to the present invention. It should be understood that each block of the flow chart and / or block diagram and the combination of the blocks in the flow chart and / or block diagram can be realized by computer-readable program instructions (e.g., program code).
[0146] These computer-readable program instructions can be provided to a processor of a computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a means for implementing the functions / actions specified in the flow chart and / or block diagram blocks or modules. The processor is operably coupled to the program code instructions to transform the processor's machine into a special-purpose machine for executing the instructions of the program code. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other device to operate in a particular manner, so that the computer-readable storage medium having the instructions stored therein includes a manufactured article, and the manufactured article includes instructions for implementing various aspects of the functions / actions specified in the flow chart and / or block diagram blocks or modules.
[0147] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device to perform a series of operational steps on the computer, other programmable device, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable device, or other device implement the functions / actions specified in the flowchart and / or block diagram or module.
[0148] The flow charts and block diagrams in the accompanying drawings show the architecture, functions and operations of the possible implementation schemes of the systems, methods and computer program products according to the various embodiments of the invention arrangement described herein. In this regard, each block in the flow chart or block diagram can represent a part of a module, segment, or instruction, including one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions marked in the block may not occur in the order marked in the figure. For example, depending on the functions involved, the two blocks displayed continuously can actually be performed as a step, at the same time, substantially at the same time, in a way that part or all of the time overlaps, or sometimes these blocks can be performed in reverse order. It should also be noted that each block of the block diagram and / or flow chart and the combination of the blocks in the block diagram and / or flow chart can be implemented by a system based on dedicated hardware that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.
[0149] The description of various embodiments provided herein is presented for the purpose of illustration and is not intended to be exhaustive or to limit the disclosed forms and examples. The terms used herein are selected to explain the principles of the inventive arrangement, practical applications, or technical improvements to the technology found in the market and / or to enable a person of ordinary skill in the art to understand the embodiments disclosed herein. Modifications and variations will be clear to a person of ordinary skill in the art without departing from the scope and spirit of the described inventive arrangement. Therefore, reference should be made to the following claims rather than the foregoing disclosure as an indication of the scope of these features and embodiments.
Claims
1. A system, include: A processor is configured to initiate an operation, the operation comprising: generating an output from each of a plurality of domain-specific models in response to the received input, wherein the domain-specific models are machine learning models trained simultaneously using a respective local loss metric for each domain-specific model and a global loss metric for the plurality of domain-specific models; and Determine the presence or absence of the intent corresponding to each domain-specific model by classifying the output of each domain-specific model, wherein the received input includes at least one of speech or an image, and Wherein, the output generated from each of the plurality of domain-specific models reflects a probability that an intent corresponding to a user goal is present in the received input.
2. The system of claim 1 , wherein the domain-specific model is simultaneously trained by generating an initial plurality of convex coefficients and iteratively adjusting one or more of the plurality of convex coefficients based on a convex coefficient machine learning rule determined using the global loss metric.
3. A system according to claim 2, wherein each corresponding local loss metric is determined by a series of error terms, each error term being determined by the difference between a model predicted value and a true value.
4. The system of claim 3, wherein the model prediction value is a sum of products, and wherein each product is the product of an output of a domain-specific model multiplied by a convex coefficient corresponding to the domain-specific model.
5. The system of claim 1, wherein each of the plurality of domain-specific models is a neural network.
6. The system of claim 1, wherein the received input comprises input from a plurality of devices of a plurality of users, and wherein the at least one domain-specific model comprises a user-specific model.
7. A method, include: generating, with computer hardware, an output from each of a plurality of domain-specific models in response to received input, wherein the domain-specific models are machine learning models trained simultaneously using a respective local loss metric for each domain-specific model and a global loss metric for the plurality of domain-specific models; and determining, using the computer hardware, the presence or absence of an intent corresponding to each domain-specific model by classifying the output of each domain-specific model, wherein the received input includes at least one of speech or an image, and Wherein, the output generated from each of the plurality of domain-specific models reflects a probability that an intent corresponding to a user goal is present in the received input.
8. The method of claim 7, wherein the domain-specific model is simultaneously trained by generating an initial plurality of convex coefficients and iteratively adjusting one or more of the plurality of convex coefficients based on a convex coefficient machine learning rule determined using the global loss metric.
9. A method according to claim 8, wherein each corresponding local loss metric is determined by a series of error terms, each error term being determined by the difference between the model predicted value and the true value.
10. The method of claim 9, wherein the model prediction value is a sum of products, and wherein each product is the product of an output of a domain-specific model multiplied by a convex coefficient corresponding to the domain-specific model.
11. The method of claim 7, wherein each of the plurality of domain-specific models is a neural network.
12. The method of claim 7, wherein the received input comprises input from a plurality of devices of a plurality of users, and wherein the at least one domain-specific model comprises a user-specific model.
13. A computer program product comprising one or more computer-readable storage media storing program instructions together, in, The program instructions are executable by one or more processors to cause the one or more processors to perform the method according to any one of claims 7 to 12.
