Electronic device and control method thereof

By dynamically managing the natural language understanding model in electronic devices, and acquiring and updating the model based on usage frequency and running time, the problems of storage capacity and network dependence are solved, and efficient natural language understanding on the device and instant processing of user voice input are achieved.

CN118915993APending Publication Date: 2024-11-08SAMSUNG ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410908906.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2020-04-06
Filing Date
2020-06-28
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the prior art, the storage and acquisition of natural language understanding models have problems such as increasing storage capacity and network dependence, especially on smart devices, resulting in insufficient storage capacity of the device and delays or interruptions in model acquisition.

Method used

By configuring processors and memory in electronic devices, acquiring and managing natural language understanding models, dynamically obtaining and updating models based on application usage frequency and runtime, synchronizing models with servers using communication interfaces, providing switching between individual and public models, and Caches the model when there is no network connection.

Benefits of technology

It realizes efficient natural language understanding on devices with limited storage capacity, reduces dependence on external networks, ensures instant processing and understanding of user voice input, and improves the natural language understanding performance of the device.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915993A_ABST
    Figure CN118915993A_ABST
Patent Text Reader

Abstract

An electronic device is provided. The electronic device includes: a memory configured to store at least one instruction; and a processor configured to execute the at least one instruction to obtain usage information about an application installed in the electronic device, obtain a natural language understanding model corresponding to the application among a plurality of natural language understanding models based on the usage information, and execute the at least one instruction to execute the at least one instruction. Natural language understanding of a user voice input related to an application is performed based on a natural language understanding model corresponding to the application, and an operation of the application is performed based on the performed natural language understanding.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of a patent application with an application date of June 28, 2020, application number 202010597082.8, and invention name “Electronic device and its control method”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application is based on and claims the benefit of priority under 35 U.S.C. §119 to Korean Patent Application No. 10-2019-0179747 filed in the Korean Intellectual Property Office on December 31, 2019 and Korean Patent Application No. 10-2020-0041470 filed in the Korean Intellectual Property Office on April 6, 2020, and to U.S. Provisional Patent Application No. 62 / 866,716 filed in the United States Patent and Trademark Office on June 26, 2019 and U.S. Provisional Patent Application No. 62 / 889,229 filed in the United States Patent and Trademark Office on August 20, 2019, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0004] The present disclosure relates to an electronic device for acquiring a natural language understanding model and a control method thereof. Background Art

[0005] The natural language understanding model is a model that analyzes user speech and grasps the user's intention included in the user speech. This natural language understanding model can exist separately for each different application.

[0006] However, in order to store natural language understanding models for all applications installed on an electronic device such as a smartphone, it may be necessary to increase the storage capacity of the electronic device.

[0007] In addition, when a natural language understanding model is received from a server and used as necessary, a period of time must pass to receive the natural language understanding model. In addition, when the network is not connected, the natural language understanding model cannot be received. Summary of the invention

[0008] The present disclosure provides an electronic device for acquiring a natural language understanding model corresponding to operations of applications frequently used by a user and a control method thereof.

[0009] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0010] According to one aspect of the present disclosure, an electronic device includes: a memory configured to store at least one instruction; and a processor configured to execute the at least one instruction to: obtain usage information about an application installed in the electronic device, obtain a natural language understanding model corresponding to the application among multiple natural language understanding models based on the usage information, perform natural language understanding of user voice input related to the application based on the natural language understanding model corresponding to the application, and perform operations of the application based on the performed natural language understanding.

[0011] The electronic device also includes: a communication interface, and wherein the processor is further configured to execute the at least one instruction to: obtain usage information including at least one of the number of applications running and the running time of the application, control the communication interface to send the obtained usage information to an external server, receive a natural language understanding model corresponding to the application from among the multiple natural language understanding models corresponding to the multiple applications from the external server via the communication interface, and store the natural language understanding model in the memory.

[0012] And, based on the number of times the application included in the usage information is run being greater than or equal to a threshold number, or based on the running time of the application being greater than or equal to a threshold time, a natural language understanding model corresponding to the application is received from the external server.

[0013] Furthermore, the application includes resource information that matches information about a predetermined voice utterance and operation information corresponding to the predetermined voice utterance; and the processor is also configured to run the at least one instruction to generate a natural language understanding model corresponding to the application based on the information about the predetermined voice utterance and the operation information corresponding to the predetermined voice utterance, based on the usage information about the application satisfying a predetermined condition.

[0014] And the processor is also configured to execute the at least one instruction to request a first user voice input corresponding to the operation of the application, and the application is determined to be executed greater than or equal to a threshold number of times or executed greater than or equal to a threshold time based on the usage information.

[0015] And the processor is also configured to run the at least one instruction to: determine whether the first user voice input conflicts with the user voice input stored in the memory based on the first user voice input being input according to the request; request a second user voice input different from the first user voice input based on determining that the first user voice input conflicts with the stored user voice input; and store the first user voice input corresponding to the operation of the application in the memory based on determining that the first user voice input does not conflict with the stored user voice input.

[0016] And the processor is further configured to execute the at least one instruction to request a natural language understanding model corresponding to the application from an external server based on the application being installed in the electronic device.

[0017] And wherein, the natural language understanding model includes at least one of a personal model generated based on the voice utterances of a user using the electronic device and a public model generated based on the voice utterances of a public user; and the processor is also configured to run the at least one instruction to: perform natural language understanding of the user voice input in the personal model, based on unsuccessful natural language understanding of the user voice input in the personal model, perform natural language understanding of the user voice input in the public model, and based on unsuccessful natural language understanding of the user voice input in the public model, send the user voice input to an external server to perform natural language understanding of the user voice input.

[0018] And the processor is also configured to execute the at least one instruction to remove the obtained natural language understanding model based on that the user voice input has not been input for greater than or equal to a threshold number of times or greater than or equal to a threshold time period.

[0019] According to another aspect of the present disclosure, a control method for an electronic device includes: obtaining usage information about an application installed in the electronic device; based on the usage information, obtaining a natural language understanding model corresponding to the application from among multiple natural language understanding models; based on the natural language understanding model corresponding to the application, performing natural language understanding of user voice input related to the application; and performing operations of the application based on the performed natural language understanding.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable recording medium has recorded thereon instructions executable by at least one processor to perform a control method for an electronic device, the control method comprising: obtaining usage information about an application installed in the electronic device; based on the usage information, obtaining a natural language understanding model corresponding to the application from among a plurality of natural language understanding models; based on the natural language understanding model corresponding to the application, performing natural language understanding of user voice input related to the application; and performing operations of the application based on the performed natural language understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other aspects, features, and advantages of certain embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings, in which:

[0022] Figure 1 is a diagram for describing a configuration of an electronic system according to an embodiment;

[0023] Figure 2 is a diagram showing a flowchart of a process of acquiring a natural language understanding model to perform natural language understanding according to an embodiment;

[0024] Figure 3A is a block diagram of a case where a natural language understanding model is received from a server according to an embodiment;

[0025] Figure 3B is a diagram for describing a process of receiving a natural language understanding model from a server according to an embodiment;

[0026] Figure 4A is a block diagram of a case of generating a natural language understanding model according to another embodiment;

[0027] Figure 4B is a diagram for describing a process of generating a natural language understanding model by an electronic device according to another embodiment;

[0028] Figure 5 is a block diagram showing an electronic device and a server according to an embodiment;

[0029] Figure 6 is a view for describing a sequence of natural language understanding model operations according to an embodiment; and

[0030] Figure 7 is a sequence diagram for describing an example in which an electronic device and a server work together to execute a function of an application according to a user's voice according to an embodiment. DETAILED DESCRIPTION

[0031] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

[0032] Figure 1 is a diagram for describing a configuration of an electronic system 1000 according to an embodiment.

[0033] refer to Figure 1 , the electronic system 1000 according to the embodiment includes an electronic device 100 and a server 200 .

[0034] The electronic device 100 may be implemented or configured to perform natural language understanding of a user's voice using an artificial intelligence model (or a neural network model, a learning or training network model, etc.), and to execute the functions of an application based on the results of the performed natural language understanding. Here, natural language understanding refers to grasping or understanding the user's intention included in a natural language that is conventionally used, rather than just following the rules in which the wording is set, unlike an artificial language such as a programming language. Artificial intelligence models that perform such natural language understanding are collectively referred to as natural language understanding models.

[0035] The functions related to artificial intelligence according to the embodiments are operated by a processor (e.g., at least one processor) and a memory (e.g., at least one memory). The processor may be configured as one or more processors. At this time, one or more processors may be general-purpose processors such as a central processing unit (CPU), an application processor (AP), a digital signal processor (DSP), a graphics-specific processor such as a graphics processing unit (GPU) or a visual processing unit (VPU), or an artificial intelligence-specific processor such as a neural processing unit (NPU). One or more processors perform control to process input data according to predefined operating rules or artificial intelligence models stored in the memory. Alternatively, when one or more processors are artificial intelligence-specific processors, the artificial intelligence-specific processors may be designed to be a hardware architecture dedicated to processing a specific artificial intelligence model.

[0036] The predefined operating rules or artificial intelligence models are characterized in that they are created (or updated) by learning (or training). Here, creation by learning means learning (or training) a basic artificial intelligence model using a plurality of learning data (or training data) by a learning algorithm (or training algorithm), thereby creating a predefined operating rule or artificial intelligence model that is set to perform the desired characteristics (or purpose). This learning can be performed in the device itself in which the artificial intelligence according to the embodiment is performed, or it can also be performed by a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but the learning algorithm is not limited to the above examples.

[0037] The artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and the neural network operation is performed by operating between the operation result of the previous layer and the multiple weights. The multiple weight values ​​of the multiple neural network layers can be optimized by the learning results of the artificial intelligence model. For example, multiple weight values ​​can be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network may include a deep neural network (DNN), a convolutional neural network (CNN), a recursive neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recursive deep neural network (BRDNN) or a deep Q network, but is not limited thereto.

[0038] According to an embodiment, assuming that the artificial intelligence model is a natural language understanding model, the electronic device 100 includes a natural language understanding model that grasps the user's intention, and can also provide a virtual secretary function (or operation) through it. The electronic device 100 can be implemented in various forms, such as a smart phone, a tablet personal computer (PC), a mobile phone, an image phone, an e-book reader, a desktop personal computer (PC), a laptop personal computer (PC), a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a medical device, a camera, a wearable device, etc.

[0039] This natural language understanding model can exist separately according to the application, and there may be one or more natural language understanding models corresponding to one application. Because the operations performed for each of the different applications may be different, the content of the user voice utterance input for each application may also be different. Therefore, the natural language understanding model can be a model dedicated to the corresponding application. In addition, the natural language understanding model can be divided into a personal model and a public model. Here, the personal model is a model generated or learned based on the user voice utterance of the electronic device 100, and can be a model personalized for the user of the electronic device 100. In addition, the public model can be a model generated or learned based on the voice utterances of multiple users (e.g., multiple public users).

[0040] The server 200 is a device for managing a plurality of natural language understanding models, and may be implemented as a central server or a cloud server.

[0041] According to an embodiment, the server 200 may send a corresponding natural language understanding model to the electronic device 100 based on a request of the electronic device 100. Specifically, the server 200 may send a pre-learned public model to the electronic device 100 based on public user data, and may also send a personal model to the electronic device 100.

[0042] According to another embodiment, a natural language understanding model may be generated on the electronic device 100 based on resource information included in an application installed on the electronic device 100. Here, the resource information may include information about a predetermined voice utterance, operation information corresponding to the predetermined voice utterance, condition information for generating a natural language understanding model, function information of the application corresponding to the predetermined voice utterance, and the like.

[0043] The electronic device 100 may generate or learn a natural language understanding model based on the resource information received from the server 200. In addition, the electronic device 100 may receive an operation list and a slot list from the server 200 and learn a public model as a personal model. This will be described in detail below with reference to the accompanying drawings.

[0044] According to various embodiments, when (or based on) a natural language understanding model is acquired on the electronic device 100, an operation corresponding to the user voice may be performed based on the acquired natural language understanding model.

[0045] Furthermore, according to various embodiments, since the electronic device 100 acquires (or obtains) a natural language understanding model corresponding to the operation of an application frequently used by a user, it is not necessary to store all natural language understanding models in the electronic device 100. Therefore, even on an electronic device with a small memory capacity, natural language understanding of a user's voice can be efficiently performed.

[0046] Furthermore, since the natural language understanding model is pre-acquired in the electronic device 100 according to the usage frequency of the application, there is no need to download the natural language understanding model from an external server in real time, so that natural language understanding of the user's voice can be efficiently performed even when there is no network connection.

[0047] Figure 2 is a diagram illustrating a flowchart of a process of acquiring a natural language understanding model to execute a function (or operation) of an application according to an embodiment.

[0048] refer to Figure 2 , the electronic device 100 may acquire usage information about an application installed on the electronic device 100 (operation S210). Here, the usage information may include at least one of user input information related to the application and usage information of the application according to the user input. For example, the usage information may include at least one of information about the amount of user input for running the application, information about an operation running in the application, time information when the application is run, etc.

[0049] According to an embodiment, the user input may include an input by a pen or a user touch. For example, the user input may be a user touch input for an application icon or an icon that executes a function included in an application, so as to run the application or the function included in the application.

[0050] According to another embodiment, the user input may include a user input inputted by a virtual secretary. As an example, when the virtual secretary service is running, an operation corresponding to the user input may be performed according to the user input by user voice, user touch, pen touch, etc.

[0051] Here, the virtual secretary service refers to a service for performing voice recognition on the input user voice using an artificial intelligence model, grasping (or determining) the user's intention, and providing response information corresponding to the user's voice as a result. In addition, the virtual secretary service can be provided as a dedicated program that is distinguished from other software (SW), and can be implemented to run the AI ​​model through a separate AI dedicated processor. When the virtual secretary service (or virtual assistant service) uses multiple artificial intelligence models, some artificial intelligence models can be implemented to run through an AI dedicated processor.

[0052] As another example, the electronic device 100 may acquire information about an application operation executed according to a user input. At this time, the information about the application operation may include information about the application operation type, information about the execution time, information about the execution quantity, and the like.

[0053] The usage information monitoring module included in the electronic device 100 may monitor usage information based on a user input and store the acquired usage information in a memory.

[0054] The electronic device 100 may acquire a natural language understanding model corresponding to the installed application from among the plurality of natural language understanding models based on the usage information (operation S220). Specifically, if the number of applications running is greater than or equal to a threshold number (e.g., a preset or predetermined threshold number), and / or if the running time is greater than or equal to a threshold time (e.g., a preset or predetermined threshold time), the electronic device 100 may acquire a natural language understanding model corresponding to the installed application. In other words, the electronic device 100 may acquire a natural language understanding model for an application frequently used by the user, and may efficiently process the user's voice input.

[0055] According to an embodiment, the electronic device 100 may receive and acquire a natural language understanding model corresponding to an application identified based on usage information from the server 200. Figure 3B Describe in detail.

[0056] According to another embodiment, the electronic device 100 may also generate a natural language understanding model corresponding to an application identified based on the usage information about the electronic device 100. Figure 4B Describe in detail.

[0057] In addition, the electronic device 100 can obtain resource information corresponding to the application operation based on the number of runs or the running time of the application operation and the number of runs or the running time of the application, and can obtain a natural language understanding model learned (or trained) based on the obtained resource information. Specifically, if the number of runs of the application operation is greater than or equal to the threshold number, and / or if the running time is greater than or equal to the threshold time, the electronic device 100 can obtain resource information corresponding to the running application operation, and obtain a natural language understanding model learned based on the obtained resource information.

[0058] When (or based on) a user voice related to an application is input, the electronic device 100 may perform natural language understanding on the user voice based on a natural language understanding model corresponding to the application (operation S230). Here, the natural language understanding model corresponding to the application may be the model acquired in operation S220.

[0059] Specifically, the natural language understanding module can grasp the user's intention by performing syntactic analysis or semantic analysis on the user's voice. Therefore, the natural language understanding model can obtain the domain, intention or parameter (or slot) required to represent the intention of the user input. Here, the domain can refer to a category classified according to the subject of the user's voice or the type of control command corresponding to the user's voice, and can be, for example, the type or type of application targeted by the user's voice. Such a domain can be identified by a domain classifier model. As an example, if the user's voice is input, the domain classifier model can identify the application corresponding to the user's voice. For example, when "turn on the selfie shooting mode of the A camera application" is input via the user's voice, the domain classifier model can identify the "A camera application" input by the user's voice as an area representing the domain. That is, the domain classifier model can identify the domain of the corresponding user voice as "A camera application". As another example, if the user's voice is input, the domain classifier model can identify a broad category related to the user's intention. For example, the domain classifier model can identify whether the category related to the user's intention is related to a schedule, a message, or a reminder.

[0060] Thereafter, the natural language understanding model may perform natural language understanding included in the user speech based on the identified domain.

[0061] The electronic device 100 may execute the function of the application based on the result of the performed natural language understanding (operation S240). For example, when (or based on) the result of the natural language understanding model is to run the camera application "A", the electronic device 100 may run the A camera application.

[0062] Figure 3A 2 is a block diagram of a case where a natural language understanding model is received from the server 200 according to an embodiment.

[0063] The electronic device 100 may include a memory 110 (eg, at least one memory), a communication interface 120 (eg, at least one communication interface), and a processor 130 (eg, at least one processor).

[0064] The memory 110 may be electrically connected to the processor 130 and store data used in various embodiments (eg, instructions executable by the processor 130 ).

[0065] According to an embodiment, the memory 110 may store a usage information monitoring module (eg, instructions) and a voice conflict confirmation module (eg, instructions). The memory 110 may also store a natural language understanding model sent by the server 200. In addition, the memory 110 may also include a domain classifier model.

[0066] Here, the usage information monitoring module may be a module that monitors user input information related to the application and usage information of the application according to the user input. In addition, the usage information monitoring module may monitor intent information related to the user's voice and usage information about the domain acquired through the artificial intelligence model of the virtual secretary service.

[0067] According to an example, the usage information monitoring module can monitor usage information about an application. Specifically, the usage information monitoring module can monitor the number of times a corresponding operation is performed and the time when the application is run by touching an application icon or an icon that executes a function included in the application with a pen or a user's body (e.g., a finger).

[0068] According to another example, the usage information monitoring module can monitor the usage information about the virtual secretary. For example, if the user inputs a voice, the speech recognition model in the artificial intelligence model of the virtual secretary service can convert the user's voice into text. In addition, if the user's voice is converted into text, at least one domain related to the user's voice can be identified by the domain classifier model included in the natural language understanding model in the artificial intelligence model of the virtual secretary service. Here, the domain can be information related to the application. For example, the domain can refer to the application itself or the category of the application. That is, the domain of the A application related to music can be the A application itself or the music category.

[0069] The usage information monitoring module may monitor the number of times a specific domain is recognized by the artificial intelligence model of the virtual secretary service, and when (or based on) the number of times a domain is recognized is greater than or equal to a threshold number, the identification information about the corresponding domain may be sent to the server 200. The identification information about the domain may refer to information that can distinguish a domain from other domains, and may include at least one of the name (title), identification (ID) information, identifier, etc. of the domain. Therefore, the electronic device 100 may receive a natural language understanding model corresponding to the corresponding domain from the server 200. Alternatively, the server 200 may request the electronic device 100 to send information about the number of identifications of the domain in a predetermined period of time. In this case, even when the number of identifications of the domain is less than the threshold number, the electronic device 100 may send information about the number of identifications of the domain to the server 200, and the server 200 may accumulate the periodically received information about the number of identifications of the domain to determine whether the number of identifications of the domain is the threshold number or more. When (at this time, in response to or based on) determining that the number of identifications of a specific domain is the threshold number or more, the server 200 may send the natural language understanding model corresponding to the corresponding domain to the electronic device 100.

[0070] As another example, after identifying at least one domain related to the user voice through the domain classifier model, natural language understanding of the text corresponding to the user voice can be performed through the natural language understanding model in the artificial intelligence model of the virtual secretary service to obtain user intent information corresponding to the user voice.

[0071] The usage information monitoring module can monitor the number of times a specific intent information is recognized by the artificial intelligence model of the virtual secretary service, and when the number of times an intent information is recognized is greater than or equal to a threshold number, identification information about the corresponding intent information can be sent to the server 200. The identification information about the intent information may refer to information that can identify an intent information to distinguish it from other intent information, and may include at least one of the name (title), identification (ID) information, identifier, etc. of the intent information. Therefore, the electronic device 100 can receive a natural language understanding model corresponding to the corresponding intent from the server 200. Alternatively, the server 200 may request the electronic device 100 to send usage information about the intent information during a predetermined period or based on a predetermined event. In this case, even when the number of times the intent information is recognized is less than the threshold number, the electronic device 100 may send the identification information about the intent information to the server 200, and the server 200 may accumulate the identification information about the intent information received periodically to determine whether the number of recognitions of the intent information is greater than or equal to the threshold number. When it is determined that the number of recognitions of the intent information is greater than or equal to the threshold number, the server 200 may send the natural language understanding model corresponding to the corresponding intent information to the electronic device 100.

[0072] In other words, even before the application is run, the usage information monitoring module can obtain the usage information about the domain and intent information identified in the virtual secretary service. As the usage information about the domain and intent information is sent to the server 200, a natural language understanding model corresponding to the domain and intent information can be received from the server 200 regardless of the running of the application.

[0073] As another example, the usage information monitoring module can monitor the number of applications running or the number of functions included in the application running through the virtual secretary. For example, the virtual secretary model can store usage information based on user input in the memory 110 or in a separate memory connected to the virtual secretary, and the usage information monitoring module can obtain the stored usage information.

[0074] Meanwhile, in the above-mentioned embodiments, the usage information monitoring module has been described as monitoring based on domain information or intent information, but this is only an example. As another example, the usage information monitoring module can monitor the operation information of the application. That is, the usage information monitoring module can monitor the number of times the operation information of the application corresponding to the user input is identified, and when the number of times an operation information is identified is greater than or equal to the threshold number, the identification information about the corresponding operation information and the identification information about the application executing the corresponding operation information can be sent to the server 200. The identification information about the operation information can refer to information that can identify an operation to distinguish it from other operations, and can include at least one of the name (title) of the operation information, identification (ID) information, slot information included in the operation, etc. Therefore, the electronic device 100 can receive the resource information and application information corresponding to the corresponding operation from the server 200, and can learn or generate a natural language understanding model based on the resource information and application information received from the server 200. Alternatively, the server 200 can learn or generate a natural language understanding model based on the identification information of the operation received from the electronic device 100 and the identification information of the application, and send the learned or generated natural language understanding model to the electronic device 100.

[0075] Alternatively, the server 200 may request the electronic device 100 to send usage information about the operation information during a predetermined period or based on a predetermined event. In this case, even when the number of identifications of the operation information is less than a threshold number, the electronic device 100 may send identification information about the operation information to the server 200, and the server 200 may accumulate the identification information about the operation information received periodically to determine whether the number of identifications of the operation information is greater than or equal to the threshold number. When it is determined that the number of identifications of the operation information is greater than or equal to the threshold number, the server 200 may send resource information or a natural language understanding model corresponding to the corresponding operation information to the electronic device 100.

[0076] Meanwhile, although the usage information monitoring module is described above as being included in the electronic device 100, the usage information monitoring module may also be included in the server 200. In particular, when natural language understanding of a user's voice is performed in the server 200, the server 200 may also monitor usage information about an application through the usage information monitoring module.

[0077] Specifically, when the user voice is input to the electronic device 100, the electronic device 100 can obtain the text corresponding to the user voice through the voice recognition model. At this time, the text corresponding to the user voice can be obtained through the voice recognition model existing in the electronic device 100, but this is only an example, and the text can be obtained through the voice recognition model of another server.

[0078] In addition, the electronic device 100 may send the text acquired for natural language understanding to the server 200. The server 200 may use the acquired text as input data of the domain classifier model included in the natural language understanding model. The server 200 may acquire domain or intent information related to the text acquired by the domain classifier model. However, the server 200 is not limited thereto, and may receive a user voice from the electronic device 100, and acquire a text corresponding to the user voice through the speech recognition model provided in the server 200 as described above. At the same time, the electronic device 100 sends the user voice or the text corresponding to the user voice to the server 200 as follows.

[0079] As an example, assume that the domain classifier model of the electronic device 100 does not recognize a domain from the text corresponding to the user voice input to the electronic device 100. In this case, the electronic device 100 may send the user voice or the text corresponding to the user voice to the server 200. When the domain of the user voice or text received from the electronic device 100 is recognized by the domain classifier model of the server 200, the usage information monitoring module of the server 200 may monitor the number of recognitions for each domain, that is, usage information about the domain for each domain. In addition, when intent information about the user voice or text is recognized after domain recognition, the usage information monitoring module of the server 200 may monitor the number of recognitions for each intent information, that is, usage information about the intent information for each intent information.

[0080] As another example, when a domain is recognized from a text corresponding to a user voice input to the electronic device 100 by a domain classifier model of the electronic device 100, but the intent information is not recognized, the electronic device 100 may send the recognized domain information and the user voice or text to the server 200. The server 200 may recognize the intent information about the user voice based on the received domain information and the user voice or text. In addition, the usage information monitoring module of the server 200 may monitor the number of recognitions for each intent information, that is, the usage information about the intent information for each intent information.

[0081] As described above, when the domain or intent information is identified in the server 200, the usage information monitoring module of the server 200 may acquire the usage information about the domain or intent information. In this case, the usage information monitoring module of the server 200 may aggregate the usage information previously received from the electronic device 100 and the usage information acquired from the server 200. Therefore, the server 200 may identify a natural language understanding model corresponding to the aggregated usage information and transmit the identified natural language understanding model to the electronic device 100.

[0082] Because the electronic device 100 does not include a natural language understanding model related to the user voice or the text corresponding to the user voice, as described above, the electronic device 100 may request the server 200 to identify the domain or intent information related to the user voice (or text). In addition, the electronic device 100 may include a natural language understanding model corresponding to the user voice (text) received from the server 200. In this case, when the user voice (or text) related to the natural language understanding model received from the server 200 is input to the electronic device 100, the electronic device 100 may identify the domain or intent information related to the user voice, and therefore may not request the server 200 separately for identification of the domain or intent information.

[0083] In other words, the electronic device 100 can identify the domain related to the user voice or the text corresponding to the user voice through the domain classifier model of the electronic device 100, and when the intent information within the domain, that is, the information about the specific function of the application is identified, the electronic device 100 may not request the identification of the domain or intent information separately from the server 200, and may send the acquired usage information to the server 200 when a predetermined condition is met or at the request of the server 200.

[0084] The voice conflict confirmation module is a module that confirms whether there is a conflict based on the similarity with the previously registered user voice or text corresponding to the user voice when the user voice corresponding to the predetermined operation of the application or the text corresponding to the user voice is newly registered. In other words, the voice conflict confirmation module is a module that can confirm whether there is a conflict between the text and the voice. This will be described in detail below.

[0085] The communication interface 120 includes a circuit, and is a component capable of communicating with the server 200 .

[0086] The communication interface 120 may transmit the usage information to the server 200 through wired or wireless communication, and receive a natural language understanding model corresponding to the application identified based on the usage information from the server 200. In this case, the communication interface 120 may transmit the identification information of the electronic device 100 and the usage information under the control of the processor 130. Here, the identification information may include at least one of an Internet Protocol (IP) address, a model name, a device name, and a serial number of the electronic device 100. The server 200 may determine a target for transmitting the natural language understanding model based on the transmitted identification information.

[0087] The processor 130 is electrically connected to the memory 110 and controls the overall operation of the electronic device 100. The processor 130 generally uses various instructions or programs stored in the memory 110 to control the operation of the electronic device 100. In particular, according to an embodiment, the main CPU can copy the program to the random access memory (RAM) according to the instructions stored in the read-only memory (ROM), and access the RAM to run the program. Here, the program may include a natural language understanding model, etc.

[0088] The processor 130 may load the usage information monitoring module, the natural language understanding model, the voice conflict confirmation module, and the domain classifier model stored in the memory 110 into the processor 130. For example, the processor 130 may load the usage information monitoring module, the natural language understanding model, the voice conflict confirmation module, and the domain classifier model stored in the memory 110 outside the processor 130 into the memory inside the processor 130. Figure 3B A process in which the electronic device 100 receives a natural language understanding model from the server 200 based on usage information about an application is described in detail.

[0089] Figure 3B is a diagram for describing a process of receiving a natural language understanding model from a server according to an embodiment of the present disclosure.

[0090] The application may be installed in the electronic device 100 (operation S310). The application may be installed when the electronic device 100 is manufactured, such as a built-in application, or may be installed by user selection.

[0091] Thereafter, the usage information monitoring module may monitor usage information about the installed application (operation S320). In addition, the usage information monitoring module may monitor usage information about domain and intent information related to user voice acquired through the artificial intelligence model of the virtual secretary service.

[0092] The usage information monitoring module can monitor user input information related to the application and usage information of the application according to the user input. For example, the usage information monitoring module can monitor the number of times the food delivery application "B" is run according to the user input. The case of running the B food delivery application according to the user input may include the case of selecting and running the B food delivery application on the display by user touch or pen, and the case of running the B food delivery application by user voice. In this case, the domain classifier model can identify the information that the domain included in the user voice is the B application. For example, "B application" may be directly included in the user's voice, or may not be directly included in the user's voice, but may include words about delivery (such as "I want to order food for delivery" or "Please order food for delivery"). In the latter case, the domain classifier model can identify the domain of the user's voice as the B application based on the "delivery" included in the user's voice.

[0093] In addition, the usage information monitoring module can monitor the number of times a specific domain is recognized by the artificial intelligence model of the virtual secretary service. For example, when a user voice for "B application" is input through the virtual secretary, the user voice for "B application" can be converted into text by the voice recognition model. The domain classifier model in the artificial intelligence model of the virtual secretary service can use the converted text as input data to identify the domain related to the user voice as "B application". The usage information monitoring module can monitor the number of domains recognized as described above, that is, the usage information about the domain.

[0094] Alternatively, the usage information monitoring module may monitor the number of recognitions of the intention information. For example, the user's intention information may be recognized as selecting a Chinese menu of "B application" as a food delivery application. The usage information monitoring module may monitor the number of times the intention information is recognized as described above, that is, the usage information about the intention information.

[0095] Alternatively, the usage information monitoring module may monitor the number of recognitions of the operation information. For example, when the user's operation information is a photo shooting operation of the A camera application, the usage information monitoring module may identify "shooting a photo" and "A camera application" as information about the application running the operation as the operation information. In addition, the usage information monitoring module may monitor the number of times the operation information is recognized as described above, that is, the usage information about the operation information.

[0096] The usage information monitoring module may store the acquired usage information in the memory 110. The usage information monitoring module may store the changed (or updated) usage information in the memory 110 whenever the usage information changes, or may store the usage information in the memory 110 periodically.

[0097] The usage information monitoring module may identify whether the monitored usage information satisfies a predetermined condition (operation S330).

[0098] If the usage information satisfies the predetermined condition (yes in operation S330), the usage information monitoring module may send the usage information to the server 200 through the communication interface 120 (operation S340). For example, when the B application is run three times or more, or when the cumulative time that the B application is run is 1 hour or more, the usage information monitoring module may send the usage information about the B application to the server 200. Here, the usage information may include at least one of the information of the application to be run, the number of runs of the B application, the running time of the B application, etc. For example, information about the number of runs of 5 and identification information that the run application is B may be sent to the server 200 together. In addition, the usage information monitoring module may also send the identification information of the electronic device 100 to the server 200. Here, the identification information may include at least one of the IP address, model name, device name, and serial number of the electronic device 100. The server 200 may determine the target for sending the natural language understanding model based on the sent identification information.

[0099] As another example, when a domain related to the B application is identified three or more times, or when intent information for selecting a specific function in the B application is identified three or more times, the usage information monitoring module may send identification information about the domain or intent information to the server 200.

[0100] In other words, the predetermined condition may be a condition related to usage information about an application or usage information about domains and intent information in the virtual secretary service. However, the predetermined condition is not limited thereto, and may be the occurrence of a predetermined event or the arrival of a predetermined time. For example, the usage information monitoring module may send the acquired usage information to the server 200 every 50 hours. In addition, the server 200 accumulates and stores the received usage information, and when the number of application runs, the number of domain recognitions, or the number of intent information recognitions is greater than or equal to a threshold number based on the accumulated usage information, the server 200 may also send a natural language understanding model corresponding thereto to the electronic device 100.

[0101] As another example, when it is identified that the operation of the C application is executed three or more times, the usage information monitoring module may transmit the operation information and information about the application that executes the operation to the server 200 .

[0102] Meanwhile, if the usage information does not satisfy the predetermined condition (No in operation S330), the usage information monitoring module may continue to monitor the usage information about the application or the usage information about the domain and intent information in the virtual secretary.

[0103] Meanwhile, the usage information monitoring module periodically confirms whether the usage information satisfies a predetermined condition, or the server 200 may also send a request signal to the electronic device 100 to confirm whether the usage information satisfies a predetermined condition.

[0104] The server 200 may identify whether an application running on the electronic device 100 is included among one or more applications managed by the server 200 based on the usage information. Specifically, the server 200 identifies whether an application identified by the domain classifier model of the electronic device 100 is an application managed by the server 200. If it is identified that the application running on the electronic device 100 is managed by the server 200, the server 200 may send a natural language understanding model corresponding to the corresponding application from among a plurality of natural language understanding models to the electronic device 100. For example, if a natural language understanding model corresponding to the B application is identified from among a plurality of natural language understanding models, the server 200 may send the identified natural language understanding model to the electronic device 100.

[0105] As an example, in the natural language understanding model corresponding to the corresponding application, the server 200 may transmit a natural language understanding model corresponding to a specific functional operation (or function) to the electronic device 100 based on the usage information transmitted from the electronic device 100. For example, when the A camera application is run according to the user input and selfies are frequently captured through the A camera application, the server 200 may transmit a natural language understanding model for the selfie operation of the A camera application to the electronic device 100 based on the usage information including such usage history.

[0106] The processor 130 may receive a natural language understanding model sent from the server 200 through the communication interface 120 (operation S350). At this time, the received natural language understanding model may be a public model. The received natural language public understanding model may include information about a predetermined voice utterance and operation information corresponding to the voice utterance. For example, the received natural language public understanding model may include information about voice utterances such as "open A application", "A application", etc., and operation information for running A application in response thereto. At this time, the processor 130 may receive the natural language understanding model itself, but may also obtain resource information for learning or generating a natural language understanding model.

[0107] The voice conflict confirmation module can confirm whether there is a conflict (or contradiction) between the information about the voice utterance included in the natural language understanding model and the previously registered user voice information (operation S360). Here, each of the information about the voice utterance and the previously registered user voice information includes a voice or a text corresponding to the voice.

[0108] For example, when a text corresponding to a voice utterance and a text corresponding to a previously registered user voice are acquired through a voice recognition model, the voice conflict confirmation module may confirm whether there is a conflict between the text corresponding to the voice utterance and the text corresponding to the previously registered user voice.

[0109] For example, in the case where the previously registered user voice is similar to the pronunciation of "A application" or the same name but different application A exists on the electronic device 100, different operations may correspond to the voice "A application" or the text corresponding to the voice. In this way, if the voice conflict is confirmed (yes in operation S360), the voice conflict confirmation module may request another voice utterance corresponding to "operation of running A application" (operation S370). Therefore, when another voice utterance is newly input to run A application, the voice conflict confirmation module may reconfirm the newly input voice or the text corresponding to the voice to confirm whether the input voice utterance conflicts with the previously registered user voice. If it is recognized that there is no voice conflict, the voice conflict confirmation module may register information about the newly input user voice or the text corresponding to the user voice as a voice or text mapped to "operation of running A application". At this time, the processor 130 may display a user interface (UI) for registering another voice utterance, and may register information about another voice utterance input through the UI.

[0110] According to another example, if the number of applications running is greater than or equal to a threshold number, or the running time of the application is greater than or equal to a threshold time, that is, if a predetermined condition is satisfied, the electronic device 100 may request a user voice corresponding to a specific operation of the application. When a user voice is input according to the request, the voice conflict confirmation module may identify whether the input user voice conflicts with a previously registered user voice stored in the memory 110, request a voice different from the input user voice when the input user voice conflicts with the previously registered user voice, and when no previously registered voice conflicting with the input user voice is recognized, map the input user voice to the operation of the application and store it in the memory 110.

[0111] Meanwhile, if it is recognized that the information about the voice utterance included in the natural language understanding model does not conflict with the previously registered user voice information (No in operation S360), the voice conflict confirmation module may register the information about the voice utterance (operation S380).

[0112] However, it should be understood that one or more other embodiments are not limited thereto. For example, according to another embodiment of sending a natural language understanding model from the server 200, whether there is a voice conflict is confirmed on the server 200 in advance, and the operation of separately confirming the voice conflict on the electronic device 100 can also be omitted.

[0113] The processor 130 may store the natural language understanding model in the memory 110 (operation S390). At this time, the stored natural language understanding model may be a personal model specified by the user. Thereafter, if the user's voice is input, the processor 130 may load the natural language understanding model stored in the memory 110 into the processor 130 to grasp the user's intention included in the user's voice based on the natural language understanding model.

[0114] Meanwhile, according to another embodiment, if an application is installed on the electronic device 100, the electronic device 100 may request a natural language understanding model corresponding to the application from an external server. In other words, if the application is installed on the electronic device 100, regardless of the usage information about the application or the information about the number of recognitions of a specific domain or specific intent information, the electronic device 100 may request a natural language understanding model corresponding to the installed application from an external server. In addition, and as an example, if the usage information about the application does not satisfy a predetermined condition (e.g., the number of uses or the number of user voice inputs within a predetermined time period after installation), the natural language understanding model may be deleted from the electronic device 100.

[0115] Figure 4A is a block diagram of a case of generating a natural language understanding model according to another embodiment.

[0116] If the application is installed on the electronic device 100, the application may be stored in the memory 110. The processor 130 may generate a natural language understanding model based on resource information included in the application. Figure 4A As shown, the resource information may include at least one of voice utterance information, operation information, information about predetermined conditions, etc. Here, the voice utterance information may be a representative utterance defined by a developer, and the voice utterance information may include slot information. The slot information may be used as a parameter for running a function. The operation information may refer to an event to be monitored. The information about a predetermined condition may refer to a natural language understanding module corresponding to an application or a condition for registering a predetermined utterance. In addition, the resource information may further include running function information, and the running function information may store information about the function of the application to be connected to the corresponding utterance.

[0117] The resource information may be matched with information about a predetermined voice utterance, operation information corresponding to the predetermined voice utterance, condition information for generating a natural language understanding model, and function information of an application corresponding to the predetermined voice utterance, and stored in a template form. For example, as shown in Table 1 below, the resource information may be stored in the memory 110.

[0118]

Table 1

[0119]

[0120] When installing an application, resource information may be included in the application data. In addition, the memory 110 may store an operation list and a slot list based on the actions and slots (or entities) defined in the public model to expand the coverage of the public model. As shown in Table 2 below, the operation list may be stored by matching at least one representative utterance corresponding to the operation.

[0121]

Table 2

[0122]

[0123] In addition, the slot list may store at least one common noun that may be included in one slot, as shown in Table 3 below.

[0124]

Table 3

[0125] building Hotel A, Hotel B, Building C, High-rise D, … car XXX, YYY, ZZZ, ...

[0126] Meanwhile, the resource information, operation list and slot list mentioned above may be stored in the server 200, and the electronic device 100 may receive and store the resource information, operation list and slot list from the server 200. In this case, when installing the application, the electronic device 100 may receive and store the resource information, operation list and slot list from the server 200, and when the predetermined condition is met, the resource information, operation list and slot list may be received and stored from the server 200. In addition, the electronic device 100 may automatically generate the resource information by analyzing the application data. Specifically, when installing an application without resource information in the form of a template, the electronic device 100 may generate the field value of the template by analyzing the application data. More specifically, the electronic device 100 may obtain the operation information by extracting the operation description information from the application data. In addition, the electronic device 100 may obtain the representative speech corresponding to the operation information based on the operation list received from the server 200. Alternatively, the electronic device 100 may obtain the representative speech corresponding to the operation information based on the natural language generation model stored in the electronic device 100. In addition, the electronic device 100 may obtain information about the predetermined condition based on the resource information in the form of different templates. As described above, the electronic device 100 may acquire resource information in the form of a template based on the acquired representative utterance, operation information, and information about a predetermined condition.

[0127] Specifically, the processor 130 may load the resource information, the usage information monitoring module, the natural language understanding model, the voice conflict confirmation module, and the domain classifier model stored in the memory 110 into the processor 130 and generate a natural language understanding model. Figure 4B Describe in detail the process of building a natural language understanding model.

[0128] Figure 4B is a diagram for describing a process of generating a natural language understanding model by the electronic device 100 according to another embodiment.

[0129] The electronic device 100 may install an application in the electronic device 100 (operation S410). The application may be installed when the electronic device 100 is manufactured, such as a built-in application, or may be installed by user selection. Installing an application in the electronic device 100 may mean that information about the application is stored in the memory 110, and when an application run command is input, the information about the application is loaded into the processor 130 so that various functions included in the application can be run. Here, the information about the application may include resource information. As described above, the resource information may include at least one of information about a predetermined voice utterance, operation information corresponding to a predetermined voice utterance, condition information for generating a natural language understanding model, function information of the application corresponding to the predetermined voice utterance, and the like.

[0130] Meanwhile, the usage information monitoring module may monitor usage information about the installed applications (operation S420).

[0131] In addition, the usage information monitoring module can identify whether the acquired usage information satisfies a predetermined condition included in the resource information (operation S430). For example, if the predetermined condition included in the resource information is a condition for running the application three or more times, and the usage information satisfies such a condition (yes in operation S430), the processor 130 can generate a natural language understanding model corresponding to the corresponding application (operation S440). At this time, the natural language understanding model can be a public model generated based on the resource information of the application. However, the predetermined condition included in the resource information is not limited thereto, and may be a specific domain of an artificial intelligence model of a virtual secretary service, specific intent information, or a specific operation recognition number may be a predetermined number or more.

[0132] Specifically, the processor 130 can generate (or learn) a natural language understanding model based on mapping the voice utterance information included in the resource information to the corresponding operation information. For example, the processor 130 can generate or learn a natural language understanding model based on the voice utterance information of "open A camera" included in the resource information and the operation of running the A camera application. Therefore, when the voice utterance "open A camera" is recognized, the domain classifier model recognizes that the domain included in the user's voice is "A camera application", and a natural language understanding model corresponding to "A camera application" can be generated to recognize that the intention included in the user's voice is to run the operation of the A camera application.

[0133] As another example, the processor 130 may generate operation plan information by analyzing the operation information of the application, and acquire representative utterances based on the generated operation plan information. In addition, the processor 130 may generate a natural language understanding model based on the operation information and the representative utterances.

[0134] Specifically, the processor 130 may generate an operation plan list by analyzing the operation of the application in advance. That is, the processor 130 may generate an operation plan list by recording the operation of the application when the user runs the application. In addition, the processor 130 may send the operation plan list to the server 200 to obtain a representative utterance of the natural language understanding model corresponding to the corresponding application. Alternatively, the processor 130 may analyze the meaning of the operation plan list through a translation module and translate the analyzed meaning into a representative utterance. Specifically, because the operation plan is a plurality of commands, the processor 130 may obtain a plurality of words through the operation plan. For example, if (1) [activity: execution] [attribute: Baemin], (2) [activity: search] [attribute: Chinese food], and (3) [activity: display] [attribute: search results] as an action plan are obtained, the processor 130 may obtain (1) in Baemin, (2) search for Chinese food, and (3) display the search results as a plurality of words. In addition, the processor 130 may generate a plurality of words into a single natural language sentence through a natural language generation module. At this time, the processor 130 may obtain a natural language sentence by assigning a weight value to each operation. Specifically, the processor 130 may assign a high weight value to an operation including a named entity, and assign a low weight value to a subsequent operation. For example, the processor 130 may obtain a natural language sentence such as "search and display Chinese food in Baemin" or "search for Chinese food in Baemin" by inputting the above-mentioned multiple words into the natural language generation module. At this time, the named entity included in the application name or operation may be obtained based on the slot list. In addition, the processor 130 may generate or learn a natural language understanding model based on the obtained operation plan and natural language sentence.

[0135] If the usage information does not satisfy the predetermined conditions for generating a natural language understanding model (No in S430), the usage information monitoring module may continuously monitor usage information about the application or usage information about a specific domain or specific intent information through an artificial intelligence model of the virtual secretary service.

[0136] Meanwhile, the voice conflict confirmation module may confirm whether there is a conflict between the information about the voice utterance included in the resource information and the previously registered user voice information (operation S450). Alternatively, the voice conflict confirmation module may confirm whether there is a conflict between the text corresponding to the voice utterance and the text corresponding to the previously registered user voice.

[0137] Specifically, the voice conflict confirmation module can confirm whether there is a conflict based on the similarity between the voice utterance information (or the text information corresponding thereto) included in the resource information and the previously registered user voice (or the text corresponding thereto). For example, suppose there is a voice input of "open A camera" in the previously registered user's voice, and the corresponding operations include running the A camera application and shooting with the rear camera. In this case, because different operations correspond to the same voice input "open A camera", the voice conflict confirmation module can recognize that a voice conflict has occurred.

[0138] As such, if the voice conflict is confirmed (Yes in operation S450), the voice conflict confirmation module may request a voice utterance different from the voice utterance information included in the resource information (operation S460).

[0139] For example, the processor 130 may control the speaker to output a command, “Please provide a voice input other than turning on the A camera,” or may control the display to display a UI window including “Please provide a voice input other than 'turn on the A camera'.”

[0140] Therefore, when another voice utterance is newly input to run the A camera application, the voice conflict confirmation module can reconfirm whether the input voice utterance conflicts with the previously registered user voice. If it is recognized that there is no voice conflict, the voice conflict confirmation module can generate a natural language understanding module corresponding to the A camera application in a state where the information about the newly input user voice is registered as a voice mapped to the “operation of running the A camera application”. As an example, a user voice input such as “Please run A camera” that is different from “Turn on A camera” can be input and registered as a voice (or voice input) corresponding to the operation of executing the A camera application. Therefore, based on the generated natural language understanding model, when “Please run A camera” is input, the operation of running the A camera application is executed, and when “Turn on A camera” is input, the operation of shooting through the rear camera by running the A camera application is executed, and they can be distinguished from each other.

[0141] As an example, a case is described where the voice utterance information included in the resource information is the same as the previously registered user voice, but even if the voice utterance information included in the resource information is similar to the previously registered user voice and the natural language understanding model is likely to misunderstand the user's intention, the voice conflict confirmation module may request a voice utterance different from the voice utterance information included in the resource information.

[0142] Meanwhile, if it is recognized that the information about the voice utterance included in the natural language understanding model does not conflict with the previously registered user voice information (No in operation S450), the voice conflict confirmation module may register the information about the voice utterance as a voice corresponding to the operation of running the A camera application.

[0143] The processor 130 may store the natural language understanding model in which the voice of the user is registered in the memory 110 (operation S470). At this time, the stored natural language understanding model may be a personal model.

[0144] Thereafter, if a user voice is input, the processor 130 may load the natural language understanding model stored in the memory 110 into the processor 130 to grasp (or determine) the user intention included in the user voice based on the natural language understanding model.

[0145] Meanwhile, according to the above-described embodiment, the natural language understanding model may be acquired by a method received from the server 200, or generated on the electronic device 100 based on the usage information. In addition, when a predetermined condition is satisfied (for example, if the user input corresponding to the acquired natural language understanding model has not been input a threshold number of times or more within a predetermined time period), the electronic device 100 may remove (or delete) the acquired, generated, or stored natural language understanding model from the electronic device 100. If the user input has not been input a threshold number of times or more indicating that the usage frequency of the application corresponding to the natural language understanding model is low, the electronic device 100 may remove the natural language understanding model having a low usage frequency from the electronic device 100.

[0146] Figure 5 is a block diagram illustrating an electronic device 100 and a server 200 according to an embodiment.

[0147] refer to Figure 5 , the electronic device 100 includes a memory 110 (e.g., at least one memory), a communication interface 120 (e.g., at least one communication interface) and a processor 130 (e.g., at least one processor), and the server 200 includes a memory 210 (e.g., at least one memory), a communication interface 220 (e.g., at least one communication interface) and a processor 230 (e.g., at least one processor).

[0148] The memory 110 may be electrically connected to the processor 130 and store data used in various embodiments.

[0149] Depending on the data storage purpose, the memory 110 may be implemented in the form of a memory embedded in the electronic device 100, or may also be implemented in the form of a memory that is attachable to and detachable from the electronic device 100. For example, data for driving the electronic device 100 may be stored in the memory embedded in the electronic device 100, and data for the extended functions of the electronic device 100 may be stored in the memory that is attachable to and detachable from the electronic device 100. Meanwhile, the memory embedded in the electronic device 100 may be implemented as at least one of a volatile memory (e.g., a dynamic random access memory (DRAM), a static RAM (SRAM), a synchronous dynamic RAM (SDRAM), etc.) or a nonvolatile memory (e.g., a one-time programmable read-only memory (OTPROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory (e.g., an AND / OR flash memory, or a non-flash memory, etc.), a hard disk drive, or a solid state drive (SSD)). The memory that can be attached to and detached from the electronic device 100 can be implemented in the form of a memory card (e.g., Compact Flash (CF), Secure Digital (SD), Micro Secure Digital (Micro-SD), Mini Secure Digital (Mini-SD), Extreme Digital (xD), MultiMedia Card (MMC)), etc.), an external memory that can be connected to a USB port (e.g., USB memory), etc.

[0150] According to an embodiment, the memory 110 may store a usage information monitoring module, a voice conflict confirmation module, a natural language understanding model, and a domain classifier model.

[0151] The communication interface 120 includes a circuit, and is a component capable of communicating with the server 200 .

[0152] The communication interface 120 may include a Wi-Fi module, a Bluetooth module, an infrared (IR) module, a local area network (LAN) module, an Ethernet module, etc. Here, each communication module can be implemented in the form of at least one hardware chip. In addition to the above-mentioned communication method, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication protocols (such as Zigbee, Universal Serial Bus (USB), Mobile Industrial Processor Interface Camera Serial Interface (MIPI CSI), third generation (3G), third generation partnership project (3GPP), long term evolution (LTE), advanced LTE (LTE-A), fourth generation (4G), fifth generation (5G), etc.). However, this is only an example, and the communication interface 120 can use at least one communication module in various communication modules. In addition, the communication interface 120 can communicate with the server through wired communication.

[0153] exist Figure 5 In the illustrated configuration, the memory 210 and the communication interface 220 included in the server 200 perform similar functions to the memory 110 and the communication interface 120 included in the electronic device 100 , and a redundant description thereof is omitted below.

[0154] At the same time, the processor 130 may send usage information about the application to the server 200 through the communication interface 120. The processor 230 of the server 200 may receive the usage information through the communication interface 220, and identify whether the application running on the electronic device 100 is an application stored in the memory 210 based on the received usage information. If it is identified that the application executed on the electronic device 100 is an application stored in the memory 210, the processor 230 may identify a natural language understanding model corresponding to the corresponding application among a plurality of natural language understanding models on the memory 210. Thereafter, the processor 230 may control the communication interface 220 to send the identified natural language understanding model to the electronic device 100. The processor 130 may store the natural language understanding model sent from the server 200 in the memory 110.

[0155] Figure 6 is a view for describing a sequence of natural language understanding model operations according to an embodiment.

[0156] refer to Figure 6 , the electronic device 100 may receive a user voice (e.g., a user voice input) (operation S610). If the user voice is input to the electronic device 100, the electronic device 100 may acquire text corresponding to the user voice through a voice recognition model. However, it should be understood that one or more other embodiments are not limited thereto, and voice recognition of the user voice may be performed by an external device, an external server, etc., and the electronic device 100 may also receive text data corresponding to the user voice from the outside. In addition, a domain classifier model included in the electronic device 100 may identify a domain corresponding to the acquired text. The electronic device 100 may identify a natural language understanding model associated with the identified domain. In this case, the electronic device 100 may identify whether it is possible to perform natural language understanding of the user voice through a personal model in the natural language understanding model associated with the identified domain (operation S620).

[0157] Here, the personal model is a model generated or learned based on a user voice utterance of the electronic device 100, and may be a model personalized for the user of the electronic device 100. For example, the personal model may be a personalized model based on a natural language understanding model received from the server 200, or a personalized model generated in the electronic device 100 based on usage information acquired by a usage information monitoring module. In other words, compared to the public model described below, the personal model may be a natural language understanding model related to a specific application frequently used by the user.

[0158] If it is identified in the personal model that natural language understanding of the user's voice can be performed (yes in operation S620), the electronic device 100 can perform natural language understanding through the corresponding personal model (operation S630), and can execute the function of the application based on the result of the natural language understanding (operation S680).

[0159] Meanwhile, if it is not possible to perform natural language understanding of the user's voice in the personal model (no in operation S620), the electronic device 100 may identify whether natural language understanding of the user's voice can be performed in the public model. Here, the public model is a model generated or learned based on the voice utterances of a public user (e.g., a different user or a user who is a public member), and may be a model for understanding the natural language included in the general voice of the public user. For example, the public model may be a natural language understanding model associated with a built-in application that is basically installed in the electronic device 100 when the electronic device 100 is manufactured, or a natural language understanding model generated based on resource information of the application.

[0160] If it is identified in the public model that natural language understanding of the user's voice can be performed (yes in operation S640), the electronic device 100 can perform natural language understanding through the corresponding public model (operation S650), and can execute the function of the application based on the result of the natural language understanding (operation S680).

[0161] Meanwhile, if it is impossible to perform natural language understanding of the user voice in the common model (No in operation S640), the electronic device 100 may transmit the user voice or text corresponding to the user voice to the server 200 (operation S660). Since a natural language understanding model capable of performing natural language understanding of the user voice is not provided in the electronic device 100, the electronic device 100 may transmit the user voice or text corresponding to the user voice to the server 200 including a plurality of natural language understanding models in order to perform natural language understanding of the user voice.

[0162] Thereafter, the domain classifier model included in the server 200 may identify the domain corresponding to the received user voice or the text corresponding to the user voice. The server 200 may identify a natural language understanding model associated with the identified domain, and perform natural language understanding of the user voice (or the text corresponding to the user voice) through the identified natural language understanding model.

[0163] The electronic device 100 may receive information about the result of the natural language understanding from the server 200 (operation S670). Thereafter, the electronic device 100 may execute a function of the application based on the result of the natural language understanding (operation S680).

[0164] Figure 7 is a sequence diagram for describing an example in which the electronic device 100 and the server 200 work together to execute a function of an application according to a user voice according to an embodiment.

[0165] refer to Figure 7 , the electronic device 100 may receive a user voice (e.g., a user voice input) (operation S710). If the user voice is input to the electronic device 100, the electronic device 100 may obtain text corresponding to the user voice through a voice recognition model. However, it should be understood that one or more other embodiments are not limited thereto, and voice recognition of the user voice may be performed by an external device, an external server, etc., and the electronic device 100 may also receive text data corresponding to the user voice from the outside.

[0166] The electronic device 100 may classify the domain corresponding to the user voice (operation S720). Specifically, the electronic device 100 may identify the domain corresponding to the acquired text using a domain classifier model included in the electronic device 100.

[0167] The electronic device 100 may determine whether there is an application corresponding to the classified domain (operation S730). That is, the electronic device 100 may determine whether there is a natural language understanding model corresponding to the application corresponding to the classified domain in the electronic device 100. If there is no application corresponding to the classified domain (no in operation S730), the electronic device 100 may send information about the user voice to the server 200 (operation S735).

[0168] If it is determined that there is an application corresponding to the classified domain (Yes in operation S730), the electronic device 100 may determine whether to map the operation information corresponding to the text (operation S740). If the operation information corresponding to the text is not mapped in the electronic device 100 (No in operation S745), the electronic device 100 may send information about the user voice and information about the application to the server 200 (operation S745).

[0169] If the operation information corresponding to the text is mapped in the electronic device 100 (Yes in operation S745), the electronic device 100 may acquire the operation information based on the natural language understanding model (operation S750). In addition, the electronic device 100 may execute the function of the application according to the acquired operation information (operation S770).

[0170] Alternatively, the server 200 may acquire operation information using a natural language understanding model stored in the server 200 based on information about a user voice (e.g., text information) or information about an application received from the electronic device 100 (operation S760). In addition, the server 200 may transmit the acquired operation information to the electronic device 100 (operation S765), and the electronic device 100 may execute a function of the application according to the acquired operation information (operation S770).

[0171] It should be understood that the method according to one or more embodiments disclosed above may be implemented in the form of an application that can be installed on an electronic device.

[0172] Furthermore, the method according to one or more embodiments disclosed above may be implemented by upgrading only software or hardware of the related art electronic device.

[0173] In addition, one or more of the above embodiments may also be executed by an embedded server included in the electronic device or at least one external server of the electronic device.

[0174] At the same time, one or more embodiments described above may be implemented by software including instructions stored in a machine (e.g., computer) readable storage medium. The machine is a device that calls the stored instructions from a storage medium and can operate according to the called instructions, and may include an electronic device according to the disclosed embodiments. When the instructions are executed by the processor, the processor may perform the functions corresponding to the instructions directly or using other components under the control of the processor. The instructions may include code generated or run by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" means that the storage medium does not include a signal and is tangible, but does not distinguish whether the data is stored in the storage medium semi-permanently or temporarily. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0175] In addition, the method according to one or more embodiments described above may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or through an application store (e.g., PLAYSTORETM ) Online distribution. In the case of online distribution, at least a part of the computer program product may be at least temporarily stored in a storage medium such as a memory of a manufacturer's server, an application store's server, or a relay server, or may be temporarily generated.

[0176] In addition, the above one or more embodiments can be implemented in a computer or similar device readable recording medium using software, hardware or a combination thereof. In some cases, the embodiment can be implemented by the processor itself. According to software implementation, the above process and function can be implemented as a separate software module. Each software module can perform the above one or more functions and operations.

[0177] Meanwhile, computer instructions for executing the processing operation of the device according to one or more embodiments described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in the non-transitory computer-readable medium, when executed by a processor of a specific device, allow the specific device to execute the processing operation of the device according to one or more embodiments described above.

[0178] The non-transitory computer-readable medium is not a medium that stores data for a short period of time, such as a register, a cache, a memory, etc., but refers to a machine-readable medium that stores data semi-permanently. Specific examples of the non-transitory computer-readable medium may include a compact disk (CD), a digital versatile disk (DVD), a hard disk, a Blu-ray disk, a universal serial bus (USB), a memory card, a read-only memory (ROM), etc.

[0179] In addition, each component (e.g., module or program) according to the above-described embodiments may include a single entity or multiple entities, and some of the above-described subcomponents may be omitted, or other subcomponents may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into one entity to perform the same or similar functions performed by the various components before integration. According to various embodiments, the operations performed by modules, programs, or other components may be performed in a sequential, parallel, iterative, or heuristic manner, or at least some operations may be performed or omitted in a different order, or other operations may be added.

[0180] Although the embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the above specific embodiments, but various modifications can be made by those skilled in the art to which the present disclosure belongs without departing from the gist of the present disclosure as disclosed at least in the attached claims. These modifications should also be understood to fall within the scope and spirit of the present disclosure.

[0181] Meanwhile, currently widely used common terms may be terms used in the description of the above embodiments in consideration of the functions in the present disclosure, but may be changed according to the intention of those skilled in the art or the emergence of judicial precedents, new technologies, etc. In addition, in certain cases, there may be terms arbitrarily selected by the applicant. In this case, the meanings of these terms may be mentioned in detail in the corresponding description parts of the present disclosure. Therefore, the terms used in the present disclosure should be defined based on the meanings of the terms and the content throughout the present disclosure, rather than the simple names of the terms.

[0182] Singular expressions include plural expressions unless the context clearly indicates otherwise. It should also be understood that the terms "comprising" or "consisting of" used in this application specify the presence of features, numbers, steps, operations, components, parts or combinations thereof mentioned in the specification, but do not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0183] Expressions such as "at least one of A and / or B" should be understood to mean "A" or "B" or "A and B".

[0184] The expressions “first”, “second”, etc. used in the present disclosure may indicate various components regardless of the order and / or importance of the components, and are only used to distinguish one component from other components and do not otherwise limit the corresponding components.

[0185] When it is mentioned that any component (e.g., a first component) is coupled (operably or communicatively) with / to another component (e.g., a second component) or connected to another component (e.g., a second component), it should be understood that any component is directly coupled with / to another component, or may be coupled with / to another component through other components (e.g., a third component).

[0186] In the present disclosure, a "module" or "device / machine" can perform at least one function or operation and can be implemented by hardware or software, or by a combination of hardware and software. In addition, multiple "modules" or multiple "devices / machines" can be integrated in at least one module and implemented by at least one processor, except for the "modules" or "devices / machines" that need to be implemented by specific hardware. In the specification, the term user may refer to a person using a terminal device (or terminal device) or a device using a terminal device (e.g., an artificial intelligence electronic device).

[0187] Although the embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the above specific embodiments, but various modifications can be made by those skilled in the art to which the present disclosure belongs without departing from the gist of the present disclosure disclosed at least in the attached claims. These modifications should also be understood to fall within the scope and spirit of the present disclosure.

Claims

1. An electronic device, include: a memory configured to store at least one instruction and a plurality of applications installed on the electronic device; and A processor configured to execute the at least one instruction to: obtaining usage information about the plurality of applications, obtaining a natural language understanding model among a plurality of natural language understanding models corresponding to an application identified among the plurality of applications based on the usage information, Based on the natural language understanding model corresponding to the application, performing natural language understanding of user voice input related to the application, and The operation of the application is performed based on the natural language understanding performed.

2. The electronic device according to claim 1, further comprising: include: Communication interface, The processor is further configured to execute the at least one instruction to: obtaining the usage information, wherein the usage information includes at least one of the number of executions of the application and the execution time of the application, controlling the communication interface to send the obtained usage information to an external server, receiving, from the external server via the communication interface, the natural language understanding model corresponding to the application among the plurality of natural language understanding models corresponding to a plurality of applications, and The natural language understanding model is stored in the memory.

3. The electronic device according to claim 2, in, Based on the number of executions of the application included in the usage information being greater than or equal to a threshold number, or based on the execution time of the application being greater than or equal to a threshold time, the natural language understanding model corresponding to the application is received from the external server.

4. The electronic device according to claim 1, in: The application includes resource information matching information about a predetermined voice utterance and operation information corresponding to the predetermined voice utterance; as well as The processor is also configured to run the at least one instruction to generate the natural language understanding model corresponding to the application based on the information about the predetermined voice utterance and the operation information corresponding to the predetermined voice utterance, based on the usage information about the application satisfying a predetermined condition.

5. The electronic device according to claim 1, in, The processor is also configured to execute the at least one instruction to request a first user voice input corresponding to an operation of the application, the application being determined based on the usage information to be executed greater than or equal to a threshold number of times or executed greater than or equal to a threshold time.

6. The electronic device according to claim 5, in, The processor is further configured to execute the at least one instruction to: determining whether the first user voice input conflicts with a user voice input stored in the memory based on the first user voice input being input according to the request; requesting a second user voice input different from the first user voice input based on determining that the first user voice input conflicts with a stored user voice input; as well as Based on determining that the first user voice input does not conflict with a stored user voice input, the first user voice input corresponding to the operation of the application is stored in the memory.

7. The electronic device according to claim 1, in, The processor is further configured to execute the at least one instruction to request a natural language understanding model corresponding to the application from an external server based on the application being installed in the electronic device.

8. The electronic device according to claim 1, in: The natural language understanding model includes at least one of a personal model generated based on a voice utterance of a user using the electronic device and a public model generated based on a voice utterance of a public user; and The processor is further configured to execute the at least one instruction to: performing the natural language understanding of the user speech input in the personal model, performing the natural language understanding of the user voice input in the public model based on the natural language understanding of the user voice input in the personal model being unsuccessful, and Based on the unsuccessful natural language understanding of the user voice input in the public model, the user voice input is sent to an external server to perform the natural language understanding of the user voice input.

9. The electronic device according to claim 1, in, The processor is also configured to execute the at least one instruction to remove the obtained natural language understanding model based on the user voice input not being input for greater than or equal to a threshold number of times or greater than or equal to a threshold time period.

10. A control method for an electronic device, the control method include: obtaining usage information about a plurality of applications installed in the electronic device; obtaining a natural language understanding model among a plurality of natural language understanding models corresponding to an application identified among the plurality of applications based on the usage information; Based on the natural language understanding model corresponding to the application, perform natural language understanding of user voice input related to the application; as well as The operation of the application is performed based on the natural language understanding performed.

11. The control method according to claim 10, in: The usage information includes at least one of the number of executions of the application and the execution time of the application; as well as Wherein, obtaining the natural language understanding model includes: Send the obtained usage information to an external server, receiving, from the external server, the natural language understanding model corresponding to the application among the plurality of natural language understanding models corresponding to a plurality of applications, and The natural language understanding model is stored in the electronic device.

12. The control method according to claim 11, in, Based on the number of executions of the application included in the usage information being greater than or equal to a threshold number, or based on the execution time of the application being greater than or equal to a threshold time, the natural language understanding model corresponding to the application is received from the external server.

13. The control method according to claim 10, in: The application includes resource information matching information about a predetermined voice utterance and operation information corresponding to the predetermined voice utterance; as well as Obtaining the natural language understanding model includes: based on the usage information about the application satisfying a predetermined condition, generating the natural language understanding model corresponding to the application according to the information about the predetermined voice utterance and the operation information corresponding to the predetermined voice utterance. 14 . The control method according to claim 10 , further comprising requesting a first user voice input corresponding to an operation of the application, the application being determined to be executed greater than or equal to a threshold number of times or executed greater than or equal to a threshold time based on the usage information.

15. The control method according to claim 10, further comprising: include: determining whether the first user voice input conflicts with a user voice input stored in the electronic device based on the first user voice input being input according to the request; requesting a second user voice input different from the first user voice input based on determining that the first user voice input conflicts with a stored user voice input; as well as Based on determining that the first user voice input does not conflict with a stored user voice input, the first user voice input corresponding to the operation of the application is stored in the electronic device.